Syntactic Blind Spots: How Misalignment Leads to LLMs Mathematical Errors

2025/10/02 by Williamson, Dane, Yangfeng Ji, Ji, Yangfeng +2
Computer Science · #Computation and Language (cs.CL) #Dependency (UML) #FOS: Computer and information sciences #I.2.0 #I.2.7 #Intelligent Tutoring Systems and Adaptive Learning #Mathematics, Computing, and Information Processing #Metric (unit) #Parsing #Selection (genetic algorithm) #Semantics (computer science) #Syntax #Visual reasoning

paper · pdf · doi:10.48550/arxiv.2510.01831

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/10/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) demonstrate strong mathematical problem-solving abilities but frequently fail on problems that deviate syntactically from their training distribution. We identify a systematic failure mode, syntactic blind spots, in which models misapply familiar reasoning strategies to problems that are semantically straightforward but phrased in unfamiliar ways. These errors are not due to gaps in mathematical competence, but rather reflect a brittle coupling between surface form and internal representation. To test this, we rephrase incorrectly answered questions using syntactic templates drawn from correct examples. These rephrasings, which preserve semantics while reducing structural complexity, often lead to correct answers. We quantify syntactic complexity using a metric based on Dependency Locality Theory (DLT), and show that higher DLT scores are associated with increased failure rates across multiple datasets. Our findings suggest that many reasoning errors stem from structural misalignment rather than conceptual difficulty, and that syntax-aware interventions can reveal and mitigate these inductive failures.

Citations

• MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
• ProcessBench: Identifying Process Errors in Mathematical Reasoning
• Does Prompt Formatting Have Any Impact on LLM Performance?
• Gemma 2: Improving Open Language Models at a Practical Size
• Qwen2 Technical Report
• Granite Code Models: A Family of Open Foundation Models for Code Intelligence
• A Careful Examination of Large Language Model Performance on Grade School Arithmetic
• Unfamiliar Finetuning Examples Control How Language Models Hallucinate
• Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap
• How Likely Do LLMs with CoT Mimic Human Reasoning?
• Explore Spurious Correlations at the Concept Level in Language Models for Text Classification
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• Deductive Verification of Chain-of-Thought Reasoning
• LLaMA: Open and Efficient Foundation Language Models
• Data Contamination: From Memorization to Exploitation
• Training Verifiers to Solve Math Word Problems
• Are NLP Models really able to Solve Simple Math Word Problems?
• A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers
• Language Models are Few-Shot Learners
• Solving General Arithmetic Word Problems
• Linguistic complexity: locality of syntactic dependencies
• Remindings and their effects in learning a cognitive skill

Related

• Large Language Models and Mathematical Reasoning Failures
• No Need for Explanations: LLMs can implicitly learn from mistakes in-context
• Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning
• Exploring LLM Reasoning Through Controlled Prompt Variations
• Exploring the Hidden Reasoning Process of Large Language Models by Misleading Them