POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

2025/10/17 by Chen, Yizhuo, Xin Liu, Ruijie Wang +13
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.17881

openalex publication_date 2025/10/17 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.

Citations

• A Survey of Reinforcement Learning for Large Reasoning Models
• Co-persona: Leveraging LLMs and Expert Collaboration to Understand User Personas through Social Media Data Analysis
• NextQuill: Causal Preference Modeling for Enhancing LLM Personalization
• Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
• PersonaBOT: Bringing Customer Personas to Life with LLMs and RAG
• HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation
• LoRe: Personalizing LLMs via Low-Rank Reward Modeling
• A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
• A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
• From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment
• One Size doesn't Fit All: A Personalized Conversational Tutoring Agent for Mathematics Instruction
• Test-Time Alignment via Hypothesis Reweighting
• RALI@TREC iKAT 2024: Achieving Personalization via Retrieval Fusion in Conversational Search
• End-to-end Training for Recommendation with Language-based User Profiles
• PersonalSum: A User-Subjective Guided Personalized Summarization Dataset for Large Language Models
• Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
• SumRecom: A Personalized Summarization Approach by Learning from Users' Feedback
• The Llama 3 Herd of Models
• PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences
• Aligning to Thousands of Preferences via System Message Generalization
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems
• Long-context LLMs Struggle with Long In-context Learning
• ORPO: Monolithic Preference Optimization without Reference Model
• Language-Based User Profiles for Recommendation
• Transparent and Scrutable Recommendations Using Natural Language User Profiles
• Personalized Language Modeling from Personalized Human Feedback
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Group Preference Optimization: Few-Shot Alignment of Large Language Models
• Large Language Models for Information Retrieval: A Survey
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• SLiC-HF: Sequence Likelihood Calibration with Human Feedback
• RRHF: Rank Responses to Align Language Models with Human Feedback without tears
• On the creativity of large language models
• Affective Coherence Monitoring for Transformer-Based Language Models
• Training language models to follow instructions with human feedback
• Deep reinforcement learning from human preferences
• Matrix Factorization Techniques for Recommender Systems
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• Aligning Large Language Models with Human: A Survey

Related

• Orchestrating LLMs with Different Personalizations
• Personalized Language Modeling from Personalized Human Feedback
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Learning to summarize user information for personalized reinforcement learning from human feedback
• Language Model Personalization via Reward Factorization