Teaching Small Language Models to Learn Logic through Meta-Learning

2025/05/20 by Leonardo Bertolazzi, Bertolazzi, Leonardo, Manuel Vargas Guzmán +7 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Intelligent Tutoring Systems and Adaptive Learning

paper · pdf · doi:10.48550/arxiv.2505.14313

openalex publication_date 2025/05/20 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) are increasingly evaluated on reasoning tasks, yet their logical abilities remain contested. To address this, we study LLMs' reasoning in a well-defined fragment of logic: syllogistic reasoning. We cast the problem as premise selection and construct controlled datasets to isolate logical competence. Beyond evaluation, an open challenge is enabling LLMs to acquire abstract inference patterns that generalize to novel structures. We propose to apply few-shot meta-learning to this domain, thereby encouraging models to extract rules across tasks rather than memorize patterns within tasks. Although meta-learning has been little explored in the context of logic learnability, our experiments show that it is effective: small models (1.5B-7B) fine-tuned with meta-learning demonstrate strong gains in generalization, with especially pronounced benefits in low-data regimes. These meta-learned models outperform GPT-4o and o3-mini on our syllogistic reasoning task.

Citations

• MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
• Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
• A Closer Look at the Self-Verification Abilities of Large Language Models in Logical Reasoning
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context
• Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
• QLoRA: Efficient Finetuning of Quantized LLMs
• Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
• Large Language Models Can Be Easily Distracted by Irrelevant Context
• Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
• MetaICL: Learning to Learn In Context
• Meta-learning via Language Model In-context Tuning
• ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language
• COGS: A Compositional Generalization Challenge Based on Semantic Interpretation
• Transformers as Soft Reasoners over Language
• Prototypical Networks for Few-shot Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Adam: A Method for Stochastic Optimization

Cited by

• Hybrid Models for Natural Language Reasoning: The Case of Syllogistic Logic

Related

• Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
• LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
• Meta Reasoning for Large Language Models
• Do Large Language Models Understand Logic or Just Mimick Context?
• Exploring Self-supervised Logic-enhanced Training for Large Language Models