Representation Engineering for Large-Language Models: Survey and Research Challenges

2025/02/24 by Łukasz Bartoszcze, Bartoszcze, Lukasz, Bryan Sukidi +13 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Semantic Web and Ontologies #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2502.17601

openalex publication_date 2025/02/24 · openalex created_date 2025/10/15 · openalex updated_date 2026/07/28

Abstract

Large-language models are capable of completing a variety of tasks, but remain unpredictable and intractable. Representation engineering seeks to resolve this problem through a new approach utilizing samples of contrasting inputs to detect and edit high-level representations of concepts such as honesty, harmfulness or power-seeking. We formalize the goals and methods of representation engineering to present a cohesive picture of work in this emerging discipline. We compare it with alternative approaches, such as mechanistic interpretability, prompt-engineering and fine-tuning. We outline risks such as performance decrease, compute time increases and steerability issues. We present a clear agenda for future research to build predictable, dynamic, safe and personalizable LLMs.

Cited by

• ReflCtrl: Controlling LLM Reflection via Representation Engineering
• SteeringSafety: A Systematic Safety Evaluation Framework of Representation Steering in LLMs
• Steering Information Utility in Key-Value Memory for Language Model Post-Training
• Internalizing LLM Reasoning via Discovery and Replay of Latent Actions
• FOCUS: Decoupling Expert Personas in LLMs to Enhance Domain Expert Capabilities

Related

• Taxonomy, Opportunities, and Challenges of Representation Engineering for Large Language Models
• Adversarial Representation Engineering: A General Model Editing Framework for Large Language Models
• A Timeline and Analysis for Representation Plasticity in Large Language Models
• Aligning Large Language Models with Human Preferences through Representation Engineering
• Challenges and Contributing Factors in the Utilization of Large Language Models (LLMs)