Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models

2025/05/05 by Matthew Dahl, Dahl, Matthew · 5 voices
Social Sciences · #Artificial Intelligence in Law #Comparative and International Law Studies

paper · pdf · doi:10.48550/arxiv.2505.02763

Abstract

Legal practice requires careful adherence to procedural rules. In the United States, few are more complex than those found in The Bluebook: A Uniform System of Citation. Compliance with this system's 500+ pages of byzantine formatting instructions is the raison d'etre of thousands of student law review editors and the bete noire of lawyers everywhere. To evaluate whether large language models (LLMs) are able to adhere to the procedures of such a complicated system, we construct an original dataset of 866 Bluebook tasks and test flagship LLMs from OpenAI, Anthropic, Google, Meta, and DeepSeek. We show (1) that these models produce fully compliant Bluebook citations only 69%-74% of the time and (2) that in-context learning on the Bluebook's underlying system of rules raises accuracy only to 77%. These results caution against using off-the-shelf LLMs to automate aspects of the law where fidelity to procedure is paramount.

Citations

• Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering
• Does Prompt Formatting Have Any Impact on LLM Performance?
• LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios
• Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries
• LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
• The Llama 3 Herd of Models
• SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain
• Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
• In-Context Learning with Long-Context Models: An In-Depth Exploration
• Make Your LLM Fully Utilize the Context
• RULER: What's the Real Context Size of Your Long-Context Language Models?
• Long-context LLMs Struggle with Long In-context Learning
• Re-evaluating GPT-4’s bar exam performance
• Caveat Lector: Large Language Models in Legal Practice
• What's in a Name? Auditing Large Language Models for Race and Gender Bias
• Promises and pitfalls of artificial intelligence for legal applications
• Task Contamination: Language Models May Not Be Few-Shot Anymore
• BLT: Can Large Language Models Handle Basic Legal Text?
• How is ChatGPT's behavior changing over time?
• Lost in the Middle: How Language Models Use Long Contexts
• Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence
• Can GPT-3 Perform Statutory Reasoning?
• The Legal Argument Reasoning Task in Civil Procedure
• Training language models to follow instructions with human feedback
• When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
• Lawma: The Power of Specialization for Legal Annotation

Discussions

• New paper: Matthew Dahl, Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models arxiv.org/pdf/2505.02763 #LegalWriting #Citation #LegalTech [bsky, 9 points, 1 comments]
• For what it’s worth, there’s a recent paper that benchmarks Bluebook citation formatting on frontier models and finds compliant/accurate citations only ~70% of the time. arxiv.org/abs/2505.02763 [bsky, 8 points, 1 comments]
• Ai "produces fully compliant bluebook citations only 69-74% of the time" bro that's far more than any human does and far far more than one ought to aspire to, this is a brief for immediately putting i [bsky, 6 points, 0 comments]
• Someone posted this link over at the other place— study on LLM ability to master the blue book, which is . . . Not great. Interesting. arxiv.org/pdf/2505.02763 [bsky, 3 points, 2 comments]
• ChatGPT for Bluebooking? Study shows that frontier models get citations right less than 75% of the time, raising questions on ability of LLMs to automate procedural tasks arxiv.org/abs/2505.02763 [bsky, 0 points, 0 comments]

Related

• Caveat Lector: Large Language Models in Legal Practice
• BLT: Can Large Language Models Handle Basic Legal Text?
• Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey
• Large Language Models and Explainable Law: a Hybrid Methodology
• When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance