ALOPE: Adaptive Layer Optimization for Translation Quality Estimation using Large Language Models

2025/08/10 by Archchana Sindhujan, Sindhujan, Archchana, Shenbin Qian +6
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2508.07484

openalex publication_date 2025/08/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) have shown remarkable performance across a wide range of natural language processing tasks. Quality Estimation (QE) for Machine Translation (MT), which assesses the quality of a source-target pair without relying on reference translations, remains a challenging cross-lingual task for LLMs. The challenges stem from the inherent limitations of existing LLM-based QE systems, which are pre-trained for causal language modelling rather than regression-specific tasks, further elevated by the presence of low-resource languages given pre-training data distribution. This paper introduces ALOPE, an adaptive layer-optimization framework designed to enhance LLM-based QE by restructuring Transformer representations through layer-wise adaptation for improved regression-based prediction. Our framework integrates low-rank adapters (LoRA) with regression task heads, leveraging selected pre-trained Transformer layers for improved cross-lingual alignment. In addition to the layer-specific adaptation, ALOPE introduces two strategies-dynamic weighting, which adaptively combines representations from multiple layers, and multi-head regression, which aggregates regression losses from multiple heads for QE. Our framework shows improvements over various existing LLM-based QE approaches. Empirical evidence suggests that intermediate Transformer layers in LLMs provide contextual representations that are more aligned with the cross-lingual nature of the QE task. We make resultant models and framework code publicly available for further research, also allowing existing LLM-based MT frameworks to be scaled with QE capabilities.

Citations

• When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages
• Multilingual Large Language Models: A Systematic Survey
• MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
• What do Large Language Models Need for Machine Translation Evaluation?
• Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation
• UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages
• Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languages
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Task
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
• LoRA: Low-Rank Adaptation of Large Language Models
• TransQuest: Translation Quality Estimation with Cross-lingual\n Transformers
• OpenKiwi: An Open Source Framework for Quality Estimation

Related

• When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages
• Tailoring Domain Adaptation for Machine Translation Quality Estimation
• An Exploratory Analysis of Multilingual Word-Level Quality Estimation with Cross-Lingual Transformers
• MTUncertainty: Assessing the Need for Post-editing of Machine Translation Outputs by Fine-tuning OpenAI LLMs
• MDQE: A More Accurate Direct Pretraining for Machine Translation Quality Estimation