2025/07/17 by Chongli Qin, Jost Tobias Springenberg, Qin, Chongli +1 · 8 voices · 13 citations
Computer Science · #Control (management) #Imitation #Neural Networks and Applications #Quality (philosophy) #Reinforcement learning #Stability (learning theory) #Structured prediction #Supervised learning #Upper and lower bounds #cs.AI #cs.LG
paper · pdf · doi:10.48550/arxiv.2507.12856
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/07/17 · openalex created_date 2025/10/18 · openalex updated_date 2026/08/05
Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.