CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

2021/06/15 by Ningyu Zhang, Zhang, Ningyu, Mosha Chen +43 · 7 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence (cs.AI) #Biomedical Text Mining and Ontologies #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2106.08087

openalex publication_date 2021/06/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually changing medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling. Our benchmark is released at \urlhttps://tianchi.aliyun.com/dataset/dataDetail?dataId=95414&lang=en-us.

Cited by

• MedBench v4: A Robust and Scalable Benchmark for Evaluating Chinese Medical Language Models, Multimodal Models, and Intelligent Agents
• 47B Mixture-of-Experts Beats 671B Dense Models on Chinese Medical Examinations
• Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications
• Enabling Doctor-Centric Medical AI with LLMs through Workflow-Aligned Tasks and Benchmarks
• A Unified Biomedical Named Entity Recognition Framework with Large Language Models
• AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field
• Exploring Stability-Plasticity Trade-offs for Continual Named Entity Recognition

Related

• PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain
• Building Chinese Biomedical Language Models via Multi-Level Text Discrimination
• MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models
• MedBench: A Comprehensive, Standardized, and Reliable Benchmarking System for Evaluating Chinese Medical Large Language Models
• ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences