AutoEval Done Right: Using Synthetic Data for Model Evaluation

2024/03/09 by Pierre Boyeau, Boyeau, Pierre, Anastasios N. Angelopoulos +7 · 7 citations
Decision Sciences · #Simulation Techniques and Applications

paper · pdf · doi:10.48550/arxiv.2403.07008

Abstract

The evaluation of machine learning models using human-labeled validation data can be expensive and time-consuming. AI-labeled synthetic data can be used to decrease the number of human annotations required for this purpose in a process called autoevaluation. We suggest efficient and statistically principled algorithms for this purpose that improve sample efficiency while remaining unbiased. These algorithms increase the effective human-labeled sample size by up to 50% on experiments with GPT-4.

Cited by

• ReSAGE-PAR: Representational Similarity Assessment for Generative Expansion in Pedestrian Attribute Recognition
• An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
• How to Correctly Report LLM-as-a-Judge Evaluations
• Extending Prediction-Powered Inference through Conformal Prediction
• Statistical Inference Leveraging Synthetic Data with Distribution-Free Guarantees
• Statistical Methods in Generative AI
• Towards a rigorous evaluation of RAG systems: the challenge of due diligence

Related

• Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• Probably Approximately Correct Labels
• CAME: Contrastive Automated Model Evaluation
• Self-Taught Evaluators
• A Little Human Data Goes A Long Way