Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective

2024/12/18 by Zhiyuan Zeng, Qinyuan Cheng, Zeng, Zhiyuan +15 · 19 voices · 7 citations
Decision Sciences · Computer Science · #Auction Theory and Applications #Open Source Software Innovations #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2412.14135

Abstract

OpenAI o1 represents a significant milestone in Artificial Inteiligence, which achieves expert-level performances on many challanging tasks that require strong reasoning ability.OpenAI has claimed that the main techinique behinds o1 is the reinforcement learining. Recent works use alternative approaches like knowledge distillation to imitate o1's reasoning style, but their effectiveness is limited by the capability ceiling of the teacher model. Therefore, this paper analyzes the roadmap to achieving o1 from the perspective of reinforcement learning, focusing on four key components: policy initialization, reward design, search, and learning. Policy initialization enables models to develop human-like reasoning behaviors, equipping them with the ability to effectively explore solution spaces for complex problems. Reward design provides dense and effective signals via reward shaping or reward modeling, which is the guidance for both search and learning. Search plays a crucial role in generating high-quality solutions during both training and testing phases, which can produce better solutions with more computation. Learning utilizes the data generated by search for improving policy, which can achieve the better performance with more parameters and more searched data. Existing open-source projects that attempt to reproduce o1 can be seem as a part or a variant of our roadmap. Collectively, these components underscore how learning and search drive o1's advancement, making meaningful contributions to the development of LLM.

Cited by

• DiRL: An Efficient Post-Training Framework for Diffusion Language Models
• Offline-Online Curriculum RL for Multimodal Reasoning
• Multilingual Medical Reasoning for Question Answering with Large Language Models
• Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks
• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models
• Trust but Verify! A Survey on Verification Design for Test-time Scaling
• KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration

Discussions

• A path to O1 open source [hn, 133 points, 80 comments]
• “this paper analyses the roadmap to achieving o1 from the perspective of reinforcement learning, focusing on four key components: policy initialisation, reward design, search, and learning” arxi [bsky, 4 points, 0 comments]
• So a group of Chinese researchers think they already figured out how OpenAI’s O1 learning algorithm works and published a 50 page roadmap paper breaking it down with links to like 30 other papers that [bsky, 3 points, 0 comments]
• Researchers from China explore the Secret Behind OpenAI o1's Success - o1 marks shift toward RL - and a shift from scaling solely training computation to scaling both training & inference - o1 archite [bsky, 2 points, 0 comments]
• Scaling of Search and Learning [hn, 2 points, 0 comments]
• A path to O1 open source view on hacker news [bsky, 1 points, 0 comments]
• Enhancing LLM Reasoning with RL An RL framework enhances large language models' reasoning, boosting performance on complex tasks via policy gradient techniques and reward shaping. Source [arXiv]: [bsky, 1 points, 0 comments]
• Two papers that imho should have made the great collection presented by the authors: - Scaling of Search and Learning: Roadmap to Reproduce o1 arxiv.org/abs/2412.14135 - Mulberry: Empowering MLLM with [bsky, 1 points, 1 comments]
• Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective arxiv.org/abs/2412.14135 [bsky, 1 points, 1 comments]
• A path to O1 open source (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• Look at the research paper arxiv.org/abs/2412.14135 [bsky, 0 points, 1 comments]
• A path to O1 open source https://arxiv.org/abs/2412.14135 (https://news.ycombinator.com/item?id=42590322) [bsky, 0 points, 0 comments]
• Paper on arXiv: arxiv.org/abs/2412.14135 [bsky, 0 points, 0 comments]
• A path to O1 open source [bsky, 0 points, 0 comments]
• A path to O1 open source #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]
• A path to O1 open source https://arxiv.org/abs/2412.14135 https://news.ycombinator.com/item?id=42590322 [bsky, 0 points, 0 comments]
• A path to O1 open source https://arxiv.org/abs/2412.14135 [comments] [17 points] [bsky, 0 points, 0 comments]
• A path to O1 open source https://arxiv.org/abs/2412.14135 (https://news.ycombinator.com/item?id=42590322) [bsky, 0 points, 0 comments]
• A path to O1 open source https://arxiv.org/abs/2412.14135 [bsky, 0 points, 0 comments]

Related

• Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
• Enhancing LLM Reasoning with Reward-guided Tree Search
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
• Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
• A Comparative Study on Reasoning Patterns of OpenAI's o1 Model