Automatic Chain of Thought Prompting in Large Language Models

2022/10/07 by Zhuosheng Zhang, Aston Zhang, Zhang, Zhuosheng +5 · 68 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2210.03493

openalex publication_date 2022/10/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like "Let's think step by step" to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstrations one by one. We show that such manual efforts may be eliminated by leveraging LLMs with the "Let's think step by step" prompt to generate reasoning chains for demonstrations one by one, i.e., let's think not just step by step, but also one by one. However, these generated chains often come with mistakes. To mitigate the effect of such mistakes, we find that diversity matters for automatically constructing demonstrations. We propose an automatic CoT prompting method: Auto-CoT. It samples questions with diversity and generates reasoning chains to construct demonstrations. On ten public benchmark reasoning tasks with GPT-3, Auto-CoT consistently matches or exceeds the performance of the CoT paradigm that requires manual designs of demonstrations. Code is available at https://github.com/amazon-research/auto-cot

Cited by

• Post-Training Quantization of OpenPangu Models for Efficient Deployment on Atlas A2
• Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding
• Auto-Prompting with Retrieval Guidance for Frame Detection in Logistics
• PortAgent: LLM-driven Vehicle Dispatching Agent for Port Terminals
• Reasoning Relay: Evaluating Stability and Interchangeability of Large Language Models in Mathematical Reasoning
• Asynchronous Reasoning: Training-Free Interactive Thinking LLMs
• Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving
• Rethinking Chain-of-Thought Reasoning for Videos
• Prompting-in-a-Series: Psychology-Informed Contents and Embeddings for Personality Recognition With Decoder-Only Models
• Cross Domain Evaluation of Multimodal Chain-of-Thought Reasoning of different datasets into the Amazon CoT Framework
• VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
• Fairness in Multi-modal Medical Diagnosis with Demonstration Selection
• Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges
• Knots: A Large-Scale Multi-Agent Enhanced Expert-Annotated Dataset and LLM Prompt Optimization for NOTAM Semantic Parsing
• Beyond Elicitation: Provision-based Prompt Optimization for Knowledge-Intensive Tasks
• Towards Trustworthy Dermatology MLLMs: A Benchmark and Multimodal Evaluator for Diagnostic Narratives
• Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models
• DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning
• Automated Prompt Generation for Code Intelligence: An Empirical study and Experience in WeChat
• QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code
• The Sequential Edge: Inverse-Entropy Voting Beats Parallel Self-Consistency at Matched Compute
• Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation
• When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought
• Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark
• DTKG: Dual-Track Knowledge Graph-Verified Reasoning Framework for Multi-Hop QA
• EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
• Improving Human Verification of LLM Reasoning through Interactive Explanation Interfaces
• Can Language Models Compose Skills In-Context?
• How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation
• Vision-Based Mistake Analysis in Procedural Activities: A Review of Advances and Challenges
• DelvePO: Direction-Guided Self-Evolving Framework for Flexible Prompt Optimization
• DAG-Math: Graph-of-Thought Guided Mathematical Reasoning in LLMs
• A Theoretical Study on Bridging Internal Probability and Self-Consistency for LLM Reasoning
• Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction
• Schema for In-Context Learning
• LLM Reasoning for Machine Translation: Synthetic Data Generation over Thinking Tokens
• Chain-of-Thought Reasoning in Streaming Full-Duplex End-to-End Spoken Dialogue Systems
• Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras
• Verifying Chain-of-Thought Reasoning via Its Computational Graph
• MOSAIC: Multi-agent Orchestration for Task-Intelligent Scientific Coding
• Towards Label-Free Biological Reasoning Synthetic Dataset Creation via Uncertainty Filtering
• MITS: Enhanced Tree Search Reasoning for LLMs via Pointwise Mutual Information
• MedCritical: Enhancing Medical Reasoning in Small Language Models via Self-Collaborative Correction
• ReviewScore: Misinformed Peer Review Detection with Large Language Models
• Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Data
• Citrus-V: Advancing Medical Foundation Models with Unified Medical Image Grounding for Clinical Reasoning
• Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
• Solving Math Word Problems Using Estimation Verification and Equation Generation
• Mitigating Strategy-Selection Bias in Reasoning for More Effective Test-Time Scaling
• OnePiece: Bringing Context Engineering and Reasoning to Industrial Cascade Ranking System
• MCP: A Control-Theoretic Orchestration Framework for Synergistic Efficiency and Interpretability in Multimodal Large Language Models
• Reasoning with Preference Constraints: A Benchmark for Language Models in Many-to-One Matching Markets
• Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity
• From Implicit Exploration to Structured Reasoning: Leveraging Guideline and Refinement for LLMs
• Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial
• DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training
• PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs
• Do LLMs Adhere to Label Definitions? Examining Their Receptivity to External Label Definitions
• LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents
• Reasoning Vectors: Transferring Chain-of-Thought Capabilities via Task Arithmetic
• Tailored Teaching with Balanced Difficulty: Elevating Reasoning in Multimodal Chain-of-Thought via Prompt Curriculum
• Cognitive Decision Routing in Large Language Models: When to Think Fast, When to Think Slow
• Online Anti-sexist Speech: Identifying Resistance to Gender Bias in Political Discourse
• Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understanding
• Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges
• Cognitive Chain-of-Thought: Structured Multimodal Reasoning about Social Situations
• BadReasoner: Planting Tunable Overthinking Backdoors into Large Reasoning Models for Fun or Profit
• Decoupling Knowledge and Reasoning in LLMs: An Exploration Using Cognitive Dual-System Theory

Related

• Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
• Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models
• Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
• Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models
• ChainLM: Empowering Large Language Models with Improved Chain-of-Thought Prompting