Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2

2021/03/24 by Gregor Betz, Kyle Richardson, Betz, Gregor +4 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2103.13033

arxiv created 2021/03/24 · openalex publication_date 2021/03/24 · arxiv updated 2021/03/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Thinking aloud is an effective meta-cognitive strategy human reasoners apply to solve difficult problems. We suggest to improve the reasoning ability of pre-trained neural language models in a similar way, namely by expanding a task's context with problem elaborations that are dynamically generated by the language model itself. Our main result is that dynamic problem elaboration significantly improves the zero-shot performance of GPT-2 in a deductive reasoning and natural language inference task: While the model uses a syntactic heuristic for predicting an answer, it is capable (to some degree) of generating reasoned additional context which facilitates the successful application of its heuristic. We explore different ways of generating elaborations, including fewshot learning, and find that their relative performance varies with the specific problem characteristics (such as problem difficulty). Moreover, the effectiveness of an elaboration can be explained in terms of the degree to which the elaboration semantically coheres with the corresponding problem. In particular, elaborations that are most faithful to the original problem description may boost accuracy by up to 24%.

Citations

• e-SNLI: Natural Language Inference with Natural Language Explanations
• The Curious Case of Neural Text Degeneration
• Can neural networks understand monotonicity reasoning?
• Language Models as Knowledge Bases?
• Multi-hop Question Answering via Reasoning Chains
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• How Context Affects Language Models' Factual Predictions
• Language Models are Few-Shot Learners
• Critical Thinking for Language Models
• Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering

Cited by

• DesignFromX: Empowering Consumer-Driven Design Space Exploration through Feature Composition of Referenced Products

Related

• Innate Reasoning is Not Enough: In-Context Learning Enhances Reasoning Large Language Models with Less Overthinking
• Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Models
• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
• Thinking Before You Speak: A Proactive Test-time Scaling Approach
• Boosting Theory-of-Mind Performance in Large Language Models via Prompting