2022/11/05 by Leonard Bongard, Bongard, Leonard, Lena Held +3 · 1 citation
Social Sciences · #Artificial Intelligence in Law #Legal Education and Practice Innovations #Comparative and International Law Studies
paper · pdf · doi:10.48550/arxiv.2211.02950
We present a new NLP task and dataset from the domain of the U.S. civil procedure. Each instance of the dataset consists of a general introduction to the case, a particular question, and a possible solution argument, accompanied by a detailed analysis of why the argument applies in that case. Since the dataset is based on a book aimed at law students, we believe that it represents a truly complex task for benchmarking modern legal language models. Our baseline evaluation shows that fine-tuning a legal transformer provides some advantage over random baseline models, but our analysis reveals that the actual ability to infer legal arguments remains a challenging open research question.