Document-Level Machine Translation with Large Language Models

2023/04/05 by Longyue Wang, Chenyang Lyu, Wang, Longyue +11 · 11 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2304.02210

openalex publication_date 2023/04/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of ChatGPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs.We release our data and annotations at https://github.com/longyuewangdcu/Document-MT-LLM.

Cited by

• AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress
• Foundation of Intelligence: Review of Math Word Problems from Human Cognition Perspective
• All You Need is One: Capsule Prompt Tuning with a Single Vector
• BiMax: Bidirectional MaxSim Score for Document-Level Alignment
• The machine translator’s visibility
• Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG
• EvoEmpirBench: Dynamic Spatial Reasoning with Agent-ExpVer
• OpenWHO: A Document-Level Parallel Corpus for Health Translation in Low-Resource Languages
• Leveraging Large Language Models for Accurate Sign Language Translation in Low-Resource Scenarios
• Do Biased Models Have Biased Thoughts?
• Discerning minds or generic tutors? Evaluating instructional guidance capabilities in Socratic LLMs

Related

• Adapting Large Language Models for Document-Level Machine Translation
• A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models
• Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Models
• Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions
• Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation