Private Direct Preference Optimization for LLM Alignment

2026/08/05 by Yangfan Jiang, Fei Wei, Ergute Bao +3
Computer Science · #cs.CR

paper · pdf

accepted for publication at CCS 2026, extended version

arxiv created 2026/08/05 · arxiv updated 2026/08/06

Abstract

Direct preference optimization (DPO) is now a standard method for aligning large language models (LLMs) using human preference data. Each DPO example contains a prompt and a pair of candidate model responses. While prompts and responses are often public or model-generated, the relative preference between responses reflects subjective judgments and can reveal sensitive attributes of annotators or end users. Off-the-shelf privacy-preserving approaches are not well matched to this structure, leading to unnecessary noise injection and biased updates in training. In this paper, we formalize preference privacy, a label-DP-style privacy notion for DPO that protects only the relative preference between candidate responses, assuming an adversary who already knows the prompt and responses. We then design PrivDPO, a DPO variant that enforces preference privacy while remaining compatible with large-scale LLM training. Our main observation is that, for neighboring examples differing only in their preference signal, the gradient difference lies on a one-dimensional preference axis determined solely by the text; all preference information flows through this axis. PrivDPO adds calibrated randomness only along this axis via an unbiased randomized rescaling of the DPO objective, avoiding per-example gradient operations. Our experiments on three alignment benchmarks and three LLM families show that PrivDPO consistently achieves strong privacy-utility trade-offs compared with privacy-preserving baselines.

Citations

• Extracting alignment data in open models
• VaultGemma: A Differentially Private Gemma Model
• Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
• Private Federated Learning using Preference-Optimized Synthetic Data
• Differentially Private Steering for Large Language Model Alignment
• Measuring memorization in language models via probabilistic extraction
• Qwen2.5-Coder Technical Report
• The Llama 3 Herd of Models
• Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignment
• SimPO: Simple Preference Optimization with a Reference-Free Reward
• Private Fine-tuning of Large Language Models with Zeroth-order Optimization
• UltraFeedback: Boosting Language Models with Scaled AI Feedback
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Fine-Tuning Language Models with Just Forward Passes
• PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
• A Note on Normalized Emergence Timing (in Pythia Language Model Evaluations)
• Sparks of Artificial General Intelligence: Early experiments with GPT-4
• LLaMA: Open and Efficient Foundation Language Models
• Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
• Training language models to follow instructions with human feedback
• Differentially Private Fine-tuning of Language Models
• Large Language Models Can Be Strong Differentially Private Learners
• Towards Understanding and Mitigating Social Biases in Language Models
• Antipodes of Label Differential Privacy: PATE and ALIBI
• Deep Learning with Label Differential Privacy
• Extracting Training Data from Large Language Models
• Differentially Private Learning Needs Better Features (or Much More Data)
• Learning to summarize from human feedback
• Rényi Differential Privacy of the Sampled Gaussian Mechanism
• Collecting and Analyzing Multidimensional Data with Local Differential Privacy
• Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoising
• The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
• Scalable Private Learning with PATE
• Proximal Policy Optimization Algorithms
• Membership Inference Attacks against Machine Learning Models
• Deep Learning with Differential Privacy
• Minimax Optimal Procedures for Locally Private Estimation
• Trust Region Policy Optimization
• Adam: A Method for Stochastic Optimization
• RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response
• Recovering Low-Rank Matrices From Few Coefficients in Any Basis
• Calibrating Noise to Sensitivity in Private Data Analysis
• Revealing information while preserving privacy
• Randomized Response: A Survey Technique for Eliminating Evasive Answer Bias
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS