Implementing Derivations of Definite Logic Programs with Self-Attention Networks

2024/10/15 by Phan Thi Thuy, Akihiro Yamamoto, Thuy, Phan Thi Thanh +1
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Formal Methods in Verification #Logic, Reasoning, and Knowledge #Logic, programming, and type systems

paper · pdf · doi:10.48550/arxiv.2410.11396

openalex publication_date 2024/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper we propose that a restricted version of logical inference can be implemented with self-attention networks. We are aiming at showing that LLMs (Large Language Models) constructed with transformer networks can make logical inferences. We would reveal the potential of LLMs by analyzing self-attention networks, which are main components of transformer networks. Our approach is not based on semantics of natural languages but operations of logical inference. %point of view. We show that hierarchical constructions of self-attention networks with feed forward networks (FFNs) can implement top-down derivations for a class of logical formulae. We also show bottom-up derivations are also implemented for the same class. We believe that our results show that LLMs implicitly have the power of logical inference.

Related

• Is logical analysis performed by transformers taking place in self-attention or in the fully connected part?
• A Implies B: Circuit Analysis in LLMs for Propositional Logical Reasoning
• Learning Language Representations with Logical Inductive Bias
• Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer
• From Neural Networks to Logical Theories: The Correspondence between Fibring Modal Logics and Fibring Neural Networks