SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs

2025/07/20 by Boyi Deng, Deng, Boyi, Yu Wan +9
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.14894

openalex publication_date 2025/07/20 · openalex created_date 2025/12/17 · openalex updated_date 2026/07/29

Abstract

Large Language Models (LLMs) have impressive multilingual capabilities, but they suffer from unexpected code-switching, also known as language mixing, which involves switching to unexpected languages in the model response. This problem leads to poor readability and degrades the usability of model responses. However, existing work on this issue lacks a mechanistic analysis and shows limited effectiveness. In this paper, we first provide an in-depth analysis of unexpected code-switching using sparse autoencoders and find that when LLMs switch to a language, the features of that language exhibit excessive pre-activation values. Based on our findings, we propose Sparse Autoencoder-guided Supervised Finetuning (SASFT), which teaches LLMs to maintain appropriate pre-activation values of specific language features during training. Experiments on five models across three languages demonstrate that SASFT consistently reduces unexpected code-switching by more than 50% compared to standard supervised fine-tuning, with complete elimination in one case. Moreover, SASFT maintains or even improves the models' performance on six multilingual benchmarks, showing its effectiveness in addressing code-switching while preserving multilingual capabilities. The code and data are available at https://github.com/Aatrox103/SASFT.

Citations

• Controllable LLM Reasoning via Sparse Autoencoder-Based Steering
• Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models
• Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causes
• Qwen3 Technical Report
• Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
• Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training
• Gemma 3 Technical Report
• I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders
• Route Sparse Autoencoder to Interpret Large Language Models
• Qwen2.5 Technical Report
• Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models
• P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs
• Can sparse autoencoders be used to decompose and interpret steering vectors?
• Code-Switching Curriculum Learning for Multilingual Transfer in LLMs
• Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Models
• Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders
• Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
• Gemma 2: Improving Open Language Models at a Practical Size
• The Llama 3 Herd of Models
• PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment
• Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
• External Invariants: A Cryptographic Trust Architecture for Institutional AI Inference
• MINERS: Multilingual Language Models as Semantic Retrievers
• A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers
• WildChat: 1M ChatGPT Interaction Logs in the Wild
• Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
• HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Code-Switched Language Identification is Harder Than You Think
• Instruction-Following Evaluation for Large Language Models
• GlotScript: A Resource and Tool for Low Resource Writing System Identification
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• PolyLM: An Open Source Polyglot Large Language Model
• Multilingual Large Language Models Are Not (Yet) Code-Switchers
• Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languages
• The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges
• Language Models are Multilingual Chain-of-Thought Reasoners
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
• Measuring Massive Multitask Language Understanding
• HellaSwag: Can a Machine Really Finish Your Sentence?

Related

• Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
• Causal Language Control in Multilingual Transformers via Sparse Feature Steering
• Controlling Language Confusion in Multilingual LLMs
• Multilingual Large Language Models Are Not (Yet) Code-Switchers
• Efficiently Adapting Pretrained Language Models To New Languages