TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation

2026/03/31 by Samah Fodeh, Linhai Ma, Ganesh Puthiaraju +5
Computer Science · #cs.CL

paper · pdf · doi:10.48550/arxiv.2603.00025

arxiv created 2026/08/03 · arxiv updated 2026/08/04

Abstract

Direct Preference Optimization (DPO) is effective for offline alignment but poorly matched to ontology-driven structured prediction, where preferred and rejected JSON often differ by only a few schema-defining tokens. In this low-edit-distance regime, sequence-level DPO spreads gradient mass across non-critical serialization tokens (gradient dilution) and can reduce likelihood on rare preferred schema tokens (token erosion). To address these limitations, we first develop a confusion-aware preference-construction strategy combining expert-curated ambiguity patterns with validation-set SFT structured-error modes, producing minimally perturbed, schema-valid negatives for ontology-level decision errors. We then introduce Token-Adaptive Barrier Preference Optimization (TAB-PO), a post-SFT objective for token-critical structured generation with a confidence-gated token-level barrier that anchors under-confident schema tokens. On SciERC, with Llama/Qwen models, TAB-PO improves ontology-critical semantic-label and relational-linking metrics by 11.59% relative to SFT, wins 100% of comparisons against strongest token-level/sequence-level DPO variants, and surpasses strongest frontier baselines on these metrics by 14.71% relative while improving textual grounding.

Citations

• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
• Token-Importance Guided Direct Preference Optimization
• Token Weighting for Long-Range Language Modeling
• Grammar-constrained decoding for structured information extraction with fine-tuned generative models applied to clinical trial abstracts
• Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
• Qwen2.5 Technical Report
• MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations
• The Llama 3 Herd of Models
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• Lessons from the Trenches on Reproducible Evaluation of Language Models
• Token-level Direct Preference Optimization
• FinBen: A Holistic Financial Benchmark for Large Language Models
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
• A Simple but Effective Approach to Improve Structured Language Model Output for Information Extraction
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Instruction Fine-Tuning: Does Prompt Loss Matter?
• Zephyr: Direct Distillation of LM Alignment
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• Guideline Learning for In-context Information Extraction
• GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction
• Large Language Models Cannot Self-Correct Reasoning Yet
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Better Zero-Shot Reasoning with Role-Play Prompting
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• QLoRA: Efficient Finetuning of Quantized LLMs
• Self-Refine: Iterative Refinement with Self-Feedback
• Reflexion: Language Agents with Verbal Reinforcement Learning
• A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
• Affective Coherence Monitoring for Transformer-Based Language Models
• NLP-based Automated Compliance Checking of Data Processing Agreements against GDPR
• Large Language Models are Zero-Shot Reasoners
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialogues
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
• Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
• MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generation
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Class-Balanced Loss Based on Effective Number of Samples
• Correcting Length Bias in Neural Machine Translation
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Inter-Coder Agreement for Computational Linguistics