When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset

2021/04/18 by Lucia Zheng, Zheng, Lucia, Neel Guha +7 · 28 citations
Computer Science · Social Sciences · #Artificial Intelligence in Law #Computation and Language (cs.CL) #FOS: Computer and information sciences #Legal Education and Practice Innovations #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2104.08671

openalex publication_date 2021/04/18 · openalex created_date 2021/04/26 · openalex updated_date 2026/07/28

Abstract

While self-supervised learning has made rapid advances in natural language processing, it remains unclear when researchers should engage in resource-intensive domain-specific pretraining (domain pretraining). The law, puzzlingly, has yielded few documented instances of substantial gains to domain pretraining in spite of the fact that legal language is widely seen to be unique. We hypothesize that these existing results stem from the fact that existing legal NLP tasks are too easy and fail to meet conditions for when domain pretraining can help. To address this, we first present CaseHOLD (Case Holdings On Legal Decisions), a new dataset comprised of over 53,000+ multiple choice questions to identify the relevant holding of a cited case. This dataset presents a fundamental task to lawyers and is both legally meaningful and difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second, we assess performance gains on CaseHOLD and existing legal NLP datasets. While a Transformer architecture (BERT) pretrained on a general corpus (Google Books and Wikipedia) improves performance, domain pretraining (using corpus of approximately 3.5M decisions across all courts in the U.S. that is larger than BERT's) with a custom legal vocabulary exhibits the most substantial performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12% improvement on BERT) and consistent performance gains across two other legal tasks. Third, we show that domain pretraining may be warranted when the task exhibits sufficient similarity to the pretraining corpus: the level of performance increase in three legal tasks was directly tied to the domain specificity of the task. Our findings inform when researchers should engage resource-intensive pretraining and show that Transformer-based architectures, too, learn embeddings suggestive of distinct legal language.

Citations

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• ERNIE: Enhanced Representation through Knowledge Integration
• The Cost of Training NLP Models: A Concise Overview
• On the Dangers of Stochastic Parrots

Cited by

• Mining Legal Arguments to Study Judicial Formalism
• EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruning
• WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue
• POLIS-Bench: Towards Multi-Dimensional Evaluation of LLMs for Bilingual Policy Tasks in Governmental Scenarios
• ClauseLens: Clause-Grounded, CVaR-Constrained Reinforcement Learning for Trustworthy Reinsurance Pricing
• Operationalizing Data Minimization for Privacy-Preserving LLM Prompting
• MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources
• Large Language Models Meet Legal Artificial Intelligence: A Survey
• Hybrid Topic-Semantic Labeling and Graph Embeddings for Unsupervised Legal Document Clustering
• Legal Zero-Days: A Novel Risk Vector for Advanced AI Systems
• LLMs for Law: Evaluating Legal-Specific LLMs on Contract Understanding
• SmartLLMs Scheduler: A Framework for Cost-Effective LLMs Utilization
• DeepSieve: Information Sieving via LLM-as-a-Knowledge-Router
• A Primer in Post-Training Reasoning Data: What We Know About How It Works
• TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law
• Exploring the use of LLMs in the Italian legal domain: A survey on recent applications
• Legal knowledge infusion for large language models: A survey
• LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training
• An Integrated Framework of Prompt Engineering and Multidimensional Knowledge Graphs for Legal Dispute Analysis
• When Large Language Models Meet Law: Dual-Lens Taxonomy, Technical Advances, and Ethical Governance
• A Survey of Classification Tasks and Approaches for Legal Contracts
• The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasure
• Legal text summarization via judicial syllogism with large language models
• Are manual annotations necessary for statutory interpretations retrieval?
• RAFT: A Real-World Few-Shot Text Classification Benchmark
• Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?
• sudoLLM: On Multi-role Alignment of Language Models
• AUTOLAW: Enhancing Legal Compliance in Large Language Models via Case Law Generation and Jury-Inspired Deliberation

Related

• Legal Transformer Models May Not Always Help
• Comparing the Performance of NLP Toolkits and Evaluation measures in Legal Tech
• CaseEncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encoding
• Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law
• Continual Pre-Training is (not) What You Need in Domain Adaption