2025/05/20 by Somnath Banerjee, Banerjee, Somnath, Chatterjee, Pratyush +10
Computer Science · #Natural Language Processing Techniques
paper · pdf · doi:10.48550/arxiv.2505.14469
While LLMs appear robustly safety-aligned in English, we uncover a catastrophic, overlooked weakness: attributional collapse under code-mixed perturbations. Our systematic evaluation of open models shows that the linguistic camouflage of code-mixing -- ``blending languages within a single conversation'' -- can cause safety guardrails to fail dramatically. Attack success rates (ASR) spike from a benign 9% in monolingual English to 69% under code-mixed inputs, with rates exceeding 90% in non-Western contexts such as Arabic and Hindi. These effects hold not only on controlled synthetic datasets but also on real-world social media traces, revealing a serious risk for billions of users. To explain why this happens, we introduce saliency drift attribution (SDA), an interpretability framework that shows how, under code-mixing, the model's internal attention drifts away from safety-critical tokens (e.g., ``violence'' or ``corruption''), effectively blinding it to harmful intent. Finally, we propose a lightweight translation-based restoration strategy that recovers roughly 80% of the safety lost to code-mixing, offering a practical path toward more equitable and robust LLM safety.