Human-in-the-loop: Real-time Preference Optimization

2025/06/02 by Wang, Wenbin, Xu, Wenjie, Jones, Colin N. · 2 citations
#FOS: Mathematics #Optimization and Control (math.OC)

paper · doi:10.48550/arxiv.2506.02225

Abstract

Optimization with preference feedback is an active research area with many applications in engineering systems where humans play a central role, such as building control and autonomous vehicles. While most existing studies focus on optimizing a static user utility, few have investigated its closed-loop behavior that accounts for system transients. In this work, we propose an online feedback optimization controller that can optimize user utility using pairwise comparison feedback with both optimality and closed-loop stability guarantees. By adding a random exploration signal, the controller estimates the gradient based on the binary utility comparison feedback between two consecutive time steps. We analyze its closed-loop behavior when interacting with a nonlinear plant and show that, under mild assumptions, the controller converges to the optimal point without inducing instability. Theoretical findings are further validated through numerical experiments.

Cited by

• Preference-based optimization from noisy pairwise comparisons
• Regularized GLISp for sensor-guided human-in-the-loop optimization

Related

• Preference-based optimization from noisy pairwise comparisons
• Sequential Preference-Based Optimization
• Persistently Exciting Online Feedback Optimization Controller with Minimal Perturbations
• Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game
• Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization