LLMs' morphological analyses of complex FST-generated Finnish words

2024/07/11 by Anssi Moisio, Moisio, Anssi, Mathias Creutz +3
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2407.08269

openalex publication_date 2024/07/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.

Related

• Evaluating Morphological Compositional Generalization in Large Language Models
• Neural Morphology Dataset and Models for Multiple Languages, from the Large to the Endangered
• A Character-Word Compositional Neural Language Model for Finnish
• Falling Through the Gaps: Neural Architectures as Models of Morphological Rule Learning
• Unsupervised Inflection Generation Using Neural Language Modeling