Evaluating Steering Techniques using Human Similarity Judgments

2025/05/25 by Zach Studdiford, Timothy T. Rogers, Studdiford, Zach +5 · 2 citations
Engineering · Psychology · #Artificial Intelligence (cs.AI) #Autonomous Vehicle Technology and Safety #FOS: Computer and information sciences #Human-Automation Interaction and Safety #I.2.7

paper · pdf · doi:10.48550/arxiv.2505.19333

openalex publication_date 2025/05/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Current evaluations of Large Language Model (LLM) steering techniques focus on task-specific performance, overlooking how well steered representations align with human cognition. Using a well-established triadic similarity judgment task, we assessed steered LLMs on their ability to flexibly judge similarity between concepts based on size or kind. We found that prompt-based steering methods outperformed other methods both in terms of steering accuracy and model-to-human alignment. We also found LLMs were biased towards 'kind' similarity and struggled with 'size' alignment. This evaluation approach, grounded in human cognition, adds further support to the efficacy of prompt-based steering and reveals privileged representational axes in LLMs prior to steering.

Citations

• AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
• Strong and weak alignment of large language models with human values
• Aligning Large Language Models with Human Preferences through Representation Engineering
• Steering Llama 2 via Contrastive Activation Addition
• In-Context Learning Creates Task Vectors
• Getting aligned on representational alignment
• The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets
• Sparse Autoencoders Find Highly Interpretable Features in Language Models
• Steering Language Models With Activation Engineering
• Improving neural network representations using human similarity judgments
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
• Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex
• Human alignment of neural network representations
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
• On the control of automatic processes: A parallel distributed processing account of the Stroop effect.
• Generalized Procrustes Analysis
• An Integrative Theory of Prefrontal Cortex Function

Cited by

• Uncovering the Computational Ingredients of Human-Like Representations in LLMs
• On the Alignment of Large Language Models with Global Human Opinion

Related

• Uncovering the Computational Ingredients of Human-Like Representations in LLMs
• On the Limitations of Steering in Language Model Alignment
• Investigating Context Effects in Similarity Judgements in Large Language Models
• Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments
• Aligning Black-box Language Models with Human Judgments