Validating Large Language Models with ReLM

2022/11/21 by Michael Kuchnik, Virginia Smith, Kuchnik, Michael +3 · 12 citations
Computer Science · Psychology · #Artificial intelligence #Cognitive psychology #Computer science #Expression (computer science) #Memorization #Natural Language Processing Techniques #Natural language #Psychology #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2211.15458

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2022/11/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Although large language models (LLMs) have been touted for their ability to generate natural-sounding text, there are growing concerns around possible negative effects of LLMs such as data memorization, bias, and inappropriate language. Unfortunately, the complexity and generation capacities of LLMs make validating (and correcting) such concerns difficult. In this work, we introduce ReLM, a system for validating and querying LLMs using standard regular expressions. ReLM formalizes and enables a broad range of language model evaluations, reducing complex evaluation rules to simple regular expression queries. Our results exploring queries surrounding memorization, gender bias, toxicity, and language understanding show that ReLM achieves up to 15x higher system efficiency, 2.5x data efficiency, and increased statistical and prompt-tuning coverage compared to state-of-the-art ad-hoc queries. ReLM offers a competitive and general baseline for the increasingly important problem of LLM validation.

Cited by

• Sharp Two-Round Adaptivity and Round Hierarchies for Semantic Regular Expressions
• Imprompt: A Language Framework for Prompt Programming
• Serve Programs, Not Prompts
• CliniBench: A Clinical Outcome Prediction Benchmark for Generative and Encoder-Based Language Models
• REFINESTAT: Efficient Exploration for Probabilistic Program Synthesis
• Automated Validation of LLM-based Evaluators for Software Engineering Artifacts
• Fast and Accurate Contextual Knowledge Extraction Using Cascading Language Model Chains and Candidate Answers
• The Curious Language Model: Strategic Test-Time Information Acquisition
• Earley-Driven Dynamic Pruning for Efficient Structured Decoding
• DINGO: Constrained Inference for Diffusion LLMs
• XGrammar-2: Dynamic and Efficient Structured Generation Engine for Agentic LLMs
• Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling

Related

• Replicating ReLM Results: Validating Large Language Models with ReLM
• Evaluating Large Language Models: A Comprehensive Survey
• A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations
• RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit
• LLMEval-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models