2024/07/11 by Ding Chen, Haochen Luo, Chen, Ding +3
Computer Science · #Machine Learning and ELM #Advanced Decision-Making Techniques
paper · pdf · doi:10.48550/arxiv.2407.08233
Current differentially private learning paradigms face a severe utility bottleneck: DP-SGD degrades performance through noise accumulation over training steps, while aggregation-based approaches such as PATE suffer from data inefficiency due to disjoint data partitioning. We propose Differentially Private Decoupled Training (DP-DT), a framework that decouples representation learning from privacy enforcement. DP-DT confines noise injection to the weight aggregation stage and employs auxiliary models, continuously synchronized with a global model, to perform noise-free feature extraction on private data shards. We prove that DP-DT converges globally to a limit point under non-convex objectives, via a Lyapunov potential analysis combined with the Kurdyka--Łojasiewicz property. Under the Hidden State Assumption, where adversaries observe only the final published model, we further prove that DP-DT's privacy loss can converge to a constant bound rather than accumulating with iterations. Empirical results across vision and language benchmarks confirm that DP-DT significantly mitigates utility degradation, achieving state-of-the-art privacy-utility trade-offs.