Confidence Optimization for Probabilistic Encoding

2025/07/22 by Yihao Huang, Xia, Pengjiu, Huang, Yidian +3
Biochemistry, Genetics and Molecular Biology · Computer Science · #Algorithms and Data Compression #Artificial Intelligence (cs.AI) #DNA and Biological Computing #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2507.16881

openalex publication_date 2025/07/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Probabilistic encoding introduces Gaussian noise into neural networks, enabling a smooth transition from deterministic to uncertain states and enhancing generalization ability. However, the randomness of Gaussian noise distorts point-based distance measurements in classification tasks. To mitigate this issue, we propose a confidence optimization probabilistic encoding (CPE) method that improves distance reliability and enhances representation learning. Specifically, we refine probabilistic encoding with two key strategies: First, we introduce a confidence-aware mechanism to adjust distance calculations, ensuring consistency and reliability in probabilistic encoding classification tasks. Second, we replace the conventional KL divergence-based variance regularization, which relies on unreliable prior assumptions, with a simpler L2 regularization term to directly constrain variance. The method we proposed is model-agnostic, and extensive experiments on natural language classification tasks demonstrate that our method significantly improves performance and generalization on both the BERT and the RoBERTa model.

Citations

• Structured Probabilistic Coding
• Probabilistic Embeddings for Cross-Modal Retrieval
• TweetEval: Unified Benchmark and Comparative Evaluation for Tweet\n Classification
• Denoising Diffusion Probabilistic Models
• Data Uncertainty Learning in Face Recognition
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Probabilistic Face Embeddings
• Modeling Uncertainty with Hedged Instance Embedding
• Bag of Tricks for Efficient Text Classification
• Word Representations via Gaussian Embedding
• Representation Learning: A Review and New Perspectives
• Representation Learning: A Review and New Perspectives
• Long Short-Term Memory
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Distance-based Confidence Score for Neural Network Classifiers
• Structured Probabilistic Coding
• Structural Entropy Guided Probabilistic Coding
• Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• Regularizing Neural Networks by Penalizing Confident Output Distributions