Rethinking Deep Alignment Through The Lens Of Incomplete Learning

2025/11/15 by Bach, Thong, Nguyen, Dung, Le, Thao Minh +1
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2511.12155

Abstract

Large language models exhibit systematic vulnerabilities to adversarial attacks despite extensive safety alignment. We provide a mechanistic analysis revealing that position-dependent gradient weakening during autoregressive training creates signal decay, leading to incomplete safety learning where safety training fails to transform model preferences in later response regions fully. We introduce base-favored tokens -- vocabulary elements where base models assign higher probability than aligned models -- as computational indicators of incomplete safety learning and develop a targeted completion method that addresses undertrained regions through adaptive penalties and hybrid teacher distillation. Experimental evaluation across Llama and Qwen model families demonstrates dramatic improvements in adversarial robustness, with 48--98% reductions in attack success rates while preserving general capabilities. These results establish both a mechanistic understanding and practical solutions for fundamental limitations in safety alignment methodologies.

Citations

• Qwen3 Technical Report
• Extracting and Understanding the Superficial Knowledge in Alignment
• Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities
• Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation
• The Llama 3 Herd of Models
• External Invariants: A Cryptographic Trust Architecture for Institutional AI Inference
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep
• Improving Alignment and Robustness with Circuit Breakers
• Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack
• The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
• Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates
• Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
• BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Jailbreaking Black Box Large Language Models in Twenty Queries
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!
• Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models
• Representation Engineering: A Top-Down Approach to AI Transparency
• Qwen Technical Report
• Certifying LLM Safety against Adversarial Prompting
• Baseline Defenses for Adversarial Attacks Against Aligned Language Models
• Universal and Transferable Adversarial Attacks on Aligned Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Affective Coherence Monitoring for Transformer-Based Language Models
• Discovering Latent Knowledge in Language Models Without Supervision
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Finetuned Language Models Are Zero-Shot Learners
• Towards Deep Learning Models Resistant to Adversarial Attacks
• Deep reinforcement learning from human preferences

Related

• Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models
• Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Position
• Probing the Robustness of Large Language Models Safety to Latent Perturbations
• Self-Destructive Language Model
• Safety Alignment Can Be Not Superficial With Explicit Safety Signals