Quality over Quantity: An Effective Large-Scale Data Reduction Strategy Based on Pointwise V-Information

2025/06/19 by Fei Chen, Chen, Fei, Wenchi Zhou +1 · 1 citation
Computer Science · Mathematics · #Advanced Data Compression Techniques #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Rough Sets and Fuzzy Logic #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2507.00038

openalex publication_date 2025/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In order to increase the effectiveness of model training, data reduction is essential to data-centric Artificial Intelligence (AI). It achieves this by locating the most instructive examples in massive datasets. To increase data quality and training efficiency, the main difficulty is choosing the best examples rather than the complete datasets. In this paper, we propose an effective data reduction strategy based on Pointwise V-Information (PVI). To enable a static method, we first use PVI to quantify instance difficulty and remove instances with low difficulty. Experiments show that classifier performance is maintained with only a 0.0001% to 0.76% decline in accuracy when 10%-30% of the data is removed. Second, we train the classifiers using a progressive learning strategy on examples sorted by increasing PVI, accelerating convergence and achieving a 0.8% accuracy gain over conventional training. Our findings imply that training a classifier on the chosen optimal subset may improve model performance and increase training efficiency when combined with an efficient data reduction strategy. Furthermore, we have adapted the PVI framework, which was previously limited to English datasets, to a variety of Chinese Natural Language Processing (NLP) tasks and base models, yielding insightful results for faster training and cross-lingual data reduction.

Citations

• Qwen3 Technical Report
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models
• Understanding Transformers via N-gram Statistics
• A Survey on Data Quality Dimensions and Tools for Machine Learning
• Sexism Detection on a Data Diet
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• An In-Depth Analysis of Data Reduction Methods for Sustainable Deep Learning
• Data-Efficient Contrastive Language-Image Pretraining: Prioritizing Data Quality over Quantity
• Dataset Difficulty and the Role of Inductive Bias
• Learning Sample Difficulty from Pre-trained Models for Reliable Prediction
• Cross-Entropy Loss Functions: Theoretical Analysis and Applications
• Natural Language Reasoning, A Survey
• Statistical Dataset Evaluation: Reliability, Difficulty, and Validity
• Training Compute-Optimal Large Language Models
• Understanding Dataset Difficulty with V-Usable Information
• Deep Learning Through the Lens of Example Difficulty
• OCNLI: Original Chinese Natural Language Inference
• Characterising Bias in Compressed Models
• Estimating Example Difficulty Using Variance of Gradients
• Learning from Noisy Labels with Deep Neural Networks: A Survey
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• A Theory of Usable Information Under Computational Constraints
• Estimating Training Data Influence by Tracing Gradient Descent
• CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese
• CrossWeigh: Training Named Entity Tagger from Imperfect Annotations
• Natural Adversarial Examples
• Do ImageNet Classifiers Generalize to ImageNet?
• XNLI: Evaluating Cross-lingual Sentence Representations
• Annotation Artifacts in Natural Language Inference Data
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference

Cited by

• MobileLLM-R1: Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes

Related

• Data Efficacy for Language Model Training
• Scaling Laws Revisited: Modeling the Role of Data Quality in Language Model Pretraining
• Oversampling and Downsampling with Core-Boundary Awareness: A Data Quality-Driven Approach
• Rethinking Data Selection at Scale: Random Selection is Almost All You Need
• Is Training Data Quality or Quantity More Impactful to Small Language Model Performance?