The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features

2025/09/16 by Ferrao, Jeremias, Matthijs van der Lende, Ilija Lichkovski +4
Computer Science · #Artificial Intelligence (cs.AI) #Constraint Satisfaction and Optimization #FOS: Computer and information sciences

paper · pdf · doi:10.48550/arxiv.2509.12934

openalex publication_date 2025/09/16 · openalex created_date 2025/10/18 · openalex updated_date 2026/07/28

Abstract

Prevailing alignment methods induce opaque parameter changes, obscuring what models truly learn. To address this, we introduce Feature Steering with Reinforcement Learning (FSRL), a framework that trains a lightweight adapter to steer model behavior by modulating interpretable sparse features. First, we theoretically demonstrate that this mechanism is expressive enough to approximate the behavioral shifts of post-training processes. We then apply FSRL to preference optimization and perform a causal analysis of the learned policy. Our analysis reveals a crucial insight: the model learns to reward stylistic presentation as a proxy for quality, disproportionately relying on features related to style and formatting over those tied to alignment concepts like honesty. By effectively optimizing the preference objective, FSRL serves as a transparent proxy for observing the alignment process. Overall, FSRL offers an interpretable control interface and a practical way to diagnose how preference optimization pressures manifest at the feature level.

Citations

• Automating Steering for Safe Multimodal Large Language Models
• Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMs
• Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders
• Steering Large Language Model Activations in Sparse Spaces
• Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers
• BatchTopK Sparse Autoencoders
• Improving Steering Vectors by Targeting Sparse Autoencoder Features
• Automatically Interpreting Millions of Features in Large Language Models
• Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2
• Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
• Transcoders Find Interpretable LLM Feature Circuits
• Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
• InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance
• Steering Llama 2 via Contrastive Activation Addition
• The Expressive Power of Low-Rank Adaptation
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Steering Language Models With Activation Engineering
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Discovering Language Model Behaviors with Model-Written Evaluations
• Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals
• Toy Models of Superposition
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• TruthfulQA: Measuring How Models Mimic Human Falsehoods
• LoRA: Low-Rank Adaptation of Large Language Models
• Measuring Massive Multitask Language Understanding
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Proximal Policy Optimization Algorithms

Related

• On-the-fly Preference Alignment via Principle-Guided Decoding
• Alignment is Localized: A Causal Probe into Preference Layers
• Constrain Alignment with Sparse Autoencoders
• Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
• Beyond RLHF: A Unified Theoretical Framework of Alignment