Internal Value Alignment in Large Language Models through Controlled Value Vector Activation

2025/07/15 by He Jin, Jin, Haoran, Meng Li +10
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2507.11316

openalex publication_date 2025/07/15 · openalex created_date 2025/10/08 · openalex updated_date 2026/07/28

Abstract

Aligning Large Language Models (LLMs) with human values has attracted increasing attention since it provides clarity, transparency, and the ability to adapt to evolving scenarios. In this paper, we introduce a Controlled Value Vector Activation (ConVA) method that directly aligns the internal values of LLMs by interpreting how a value is encoded in their latent representations and modifies relevant activations to ensure consistent values in LLMs. To ensure an accurate and unbiased interpretation, we propose a context-controlled value vector identification method. To consistently control values without sacrificing model performance, we introduce a gated value vector activation method for effective and minimum degree of value control. Experiments show that our method achieves the highest control success rate across 10 basic values without hurting LLM performance and fluency, and ensures target values even with opposite and potentially malicious input prompts. Source code and data are available at~ https://github.com/hr-jin/ConVA.

Citations

• Controlling Large Language Models Through Concept Activation Vectors
• The Llama 3 Herd of Models
• CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses
• Evaluating Large Language Model Biases in Persona-Steered Generation
• High-Dimension Human Value Representation in Large Language Models
• Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?
• Steering Llama 2 via Contrastive Activation Addition
• Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Values
• Moral Foundations of Large Language Models
• Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learning
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Representation Engineering: A Top-Down Approach to AI Transparency
• Emergent Linear Representations in World Models of Self-Supervised Sequence Models
• Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Training Socially Aligned Language Models on Simulated Social Interactions
• Adam-Smith at SemEval-2023 Task 4: Discovering Human Values in Arguments with Ensembles of Transformer-based Models
• Toxicity in ChatGPT: Analyzing Persona-assigned Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Discovering Latent Knowledge in Language Models Without Supervision
• Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP
• Self-critiquing models for assisting human evaluators
• Training language models to follow instructions with human feedback
• Ethical and social risks of harm from Language Models
• StereoSet: Measuring stereotypical bias in pretrained language models
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Dimensionalizing Cultures: The Hofstede Model in Context
• A Coefficient of Agreement for Nominal Scales
• Uncovering Safety Risks of Large Language Models through Concept Activation Vector
• MaxMin-RLHF: Alignment with Diverse Human Preferences

Related

• Multi-Value Alignment for LLMs via Value Decorrelation and Extrapolation
• Large Language Model Alignment: A Survey
• Value FULCRA: Mapping Large Language Models to the Multidimensional Spectrum of Basic Human Values
• PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimization
• Prompt-Based Value Steering of Large Language Models