Connections between reinforcement learning with feedback,test-time scaling, and diffusion guidance: An anthology

2025/09/04 by Jiao, Yuchen, Chen, Yuxin, Li, Gen
#FOS: Computer and information sciences #FOS: Mathematics #General Literature (cs.GL) #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Statistics Theory (math.ST)

paper · doi:10.48550/arxiv.2509.04372

Abstract

In this note, we reflect on several fundamental connections among widely used post-training techniques. We clarify some intimate connections and equivalences between reinforcement learning with human feedback, reinforcement learning with internal feedback, and test-time scaling (particularly soft best-of-N sampling), while also illuminating intrinsic links between diffusion guidance and test-time scaling. Additionally, we introduce a resampling approach for alignment and reward-directed diffusion models, sidestepping the need for explicit reinforcement learning techniques.

Citations

• Learning to Reason without External Rewards
• Soft Best-of-n Sampling for Model Alignment
• Scalable Best-of-N Selection for Large Language Models via Self-Certainty
• s1: Simple test-time scaling
• Reward-Directed Score-Based Diffusion Models via q-Learning
• A Score-Based Density Formula, with Applications in Diffusion Generative Models
• Classifier-Free Guidance is a Predictor-Corrector
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
• Unfamiliar Finetuning Examples Control How Language Models Hallucinate
• A Survey of Reinforcement Learning from Human Feedback
• DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models
• Training Diffusion Models with Reinforcement Learning
• Classifier-Free Diffusion Guidance
• Training language models to follow instructions with human feedback
• Diffusion Models Beat GANs on Image Synthesis
• Fine-Tuning Language Models from Human Preferences
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Related

• Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
• Diffusion Models for Reinforcement Learning: A Survey
• Plug-and-Play Prompt Refinement via Latent Feedback for Diffusion Model Alignment
• Reinforcement Learning from Human Feedback
• Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning