Frictional Agent Alignment Framework: Slow Down and Don't Break Things

2025/05/26 by Abhijnan Nath, Carine Graff, Nath, Abhijnan +5 · 1 voice · 4 citations
Computer Science · Mathematics · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Modeling, Simulation, and Optimization #Multi-Agent Systems and Negotiation #cs.CL

paper · pdf · doi:10.48550/arxiv.2505.19428

openalex publication_date 2025/05/26 · arxiv published 2025/05/26 · arxiv updated 2025/05/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

AI support of collaborative interactions entails mediating potential misalignment between interlocutor beliefs. Common preference alignment methods like DPO excel in static settings, but struggle in dynamic collaborative tasks where the explicit signals of interlocutor beliefs are sparse and skewed. We propose the Frictional Agent Alignment Framework (FAAF), to generate precise, context-aware "friction" that prompts for deliberation and re-examination of existing evidence. FAAF's two-player objective decouples from data skew: a frictive-state policy identifies belief misalignments, while an intervention policy crafts collaborator-preferred responses. We derive an analytical solution to this objective, enabling training a single policy via a simple supervised loss. Experiments on three benchmarks show FAAF outperforms competitors in producing concise, interpretable friction and in OOD generalization. By aligning LLMs to act as adaptive "thought partners" -- not passive responders -- FAAF advances scalable, dynamic human-AI collaboration. Our code and data can be found at https://github.com/csu-signal/FAAF_ACL.

Citations

• Dynamic Epistemic Friction in Dialogue
• TRACE: Real-Time Multimodal Common Ground Tracking in Situated Collaborative Dialogues
• Any Other Thoughts, Hedgehog? Linking Deliberation Chains in Collaborative Dialogues
• Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both
• Modulating Language Model Experiences through Frictions
• Toward Optimal LLM Alignments Using Two-Player Games
• Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
• Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs
• Robust Preference Optimization through Reward Model Distillation
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• From r to Q^*: Your Language Model is Secretly a Q-Function
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
• Common Ground Tracking in Multimodal Dialogue
• Human Alignment of Large Language Models through Online Preference Optimisation
• ORPO: Monolithic Preference Optimization without Reference Model
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
• West-of-N: Synthetic Preferences for Self-Improving Reward Models
• Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss
• LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• How Good is Automatic Segmentation as a Multimodal Discourse Annotation Aid?
• QLoRA: Efficient Finetuning of Quantized LLMs
• CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
• Aligning Language Models with Preferences through f-divergence Minimization
• Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
• Robust Speech Recognition via Large-Scale Weak Supervision
• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
• Controllable Dialogue Simulation with In-Context Learning
• On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting
• OPT: Open Pre-trained Transformer Language Models
• Reframing Human-AI Collaboration for Generating Free-Text Explanations
• DeliData: A dataset for deliberation in multi-party problem solving
• MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation
• Measuring Association Between Labels and Free-Text Rationales
• Learning to summarize from human feedback
• Sparse Text Generation
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset
• The Curious Case of Neural Text Degeneration
• Decoupled Weight Decay Regularization
• Proximal Policy Optimization Algorithms
• The Communicative Accomplishment of Collaboration Failure
• Evidence and plausibility in neighborhood structures
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• A Survey of Imitation Learning Methods, Environments and Metrics

Cited by

• Learning "Partner-Aware" Collaborators in Multi-Party Collaboration
• Let's Roleplay: Examining LLM Alignment in Collaborative Dialogues
• Dynamic Epistemic Friction in Dialogue
• Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetry

Discussions

• Your #AI system should be slower. Our paper "Frictional Agent Alignment Framework" will appear at #ACL2025 in Vienna this summer, but you can preview it on arXiv! [bsky, 3 points, 0 comments]

Related

• Let's Roleplay: Examining LLM Alignment in Collaborative Dialogues
• Quantifying Misalignment Between Agents: Towards a Sociotechnical Understanding of Alignment
• AI Alignment Dialogues: An Interactive Approach to AI Alignment in Support Agents
• The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process
• Dynamic Alignment for Collective Agency: Toward a Scalable Self-Improving Framework for Open-Ended LLM Alignment