Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis

2024/06/28 by Marchisio, Kelly, Wei-Yin Ko, Ko, Wei-Yin +6 · 12 citations
Health Professions · Arts and Humanities · #Interpreting and Communication in Healthcare #Translation Studies and Practices

paper · pdf · doi:10.48550/arxiv.2406.20052

Abstract

This repository contain datasets and results for the paper: Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis Github repository for the code: Quantifying Language Confusion GitHub repo DATA include the following datasets: i) raw language graphs and ii) the calculated language similarities from the language graphs, iii) MTEI: the files from the experimental results of multilingual inversion attacks, and calculated language confusion entropy from the data; iv) LCB: the files from the language confusion benchmark and calculated language confusion entropy from the data Results include aggregated results for further analysis: i) inversion_language_confusion: results from MTEI ii) prompting_language_confusion: results from LCB

Cited by

• Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languages
• Uncovering the Potential Risks in Unlearning: Danger of English-only Unlearning in Multilingual LLMs
• The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
• Conditions for Catastrophic Forgetting in Multilingual Translation
• Language Confusion Gate: Language-Aware Decoding Through Model Self-Distillation
• Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models
• MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages
• One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning
• The role of synthetic data in Multilingual, Multi-cultural AI systems: Lessons from Indic Languages
• SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages
• ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance
• Culture is Everywhere: A Call for Intentionally Cultural Evaluation

Related

• Controlling Language Confusion in Multilingual LLMs
• Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models
• Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis
• How Reliable is Multilingual LLM-as-a-Judge?
• Language Confusion Gate: Language-Aware Decoding Through Model Self-Distillation