Analyzing constrained LLM through PDFA-learning

2024/06/12 by Carrasco, Matías, Mayr, Franz, Yovine, Sergio +4 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Formal Languages and Automata Theory (cs.FL) #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2406.08269

Abstract

We define a congruence that copes with null next-symbol probabilities that arise when the output of a language model is constrained by some means during text generation. We develop an algorithm for efficiently learning the quotient with respect to this congruence and evaluate it on case studies for analyzing statistical properties of LLM.

Cited by

• Minimality of Random Moore Automata under Prefix-Dependent Congruences

Related

• Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference
• LLM Inference Unveiled: Survey and Roofline Model Insights
• Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws
• Compute-Optimal LLMs Provably Generalize Better With Scale
• Textual Bayes: Quantifying Uncertainty in LLM-Based Systems