2025/06/30 by Zhiyuan Zhao, Bin Wang, Linke Ouyang +5
Computer Science · #Adaptive sampling #Benchmarking #Class (philosophy) #Closing (real estate) #Data collection #Data modeling #Loop (graph theory) #Machine Learning and Algorithms #Machine Learning in Healthcare #Topic Modeling #Training set
paper · pdf · doi:10.1109/icme59968.2025.11208956
openalex publication_date 2025/06/30 · openalex created_date 2025/10/30 · openalex updated_date 2026/08/05
In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop iteration, the MLLM-DataEngine first analyzes the weakness of the model based on the evaluation results, then generates a proper incremental dataset for the next training iteration, and enhances the model capability iteratively. Compared with previous instruction fine-tuning dataset collection methods which are separate from the benchmarking, MLLM-DataEngine shows better targeting and can improve MLLMs’s capabilities more effectively. Firstly, we propose an Adaptive Bad-case Sampling module, which can effectively analyze model weakness based on the benchmarking results and adjust the generation of incremental datasets flexibly. Secondly, in order to ensure high-quality data for specific capability types, the most representative in-context examples and abundant information are provided to GPT-4, which helps GPT-4 fully comprehend the model’s weakness and further guarantees high-quality generated data. Through extensive experiments, we find MLLM-DataEngine could boost the MLLMs capability in a targeted and automatic manner without human participants. We hope MLLM-DataEngine could be a general solution for the following MLLMs data curation. Code, data, and model are available at https://github.com/opendatalab/MLLM-DataEngine.