QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation

2025/05/08 by Mengze Hong, Wailing Ng, Hong, Mengze +5 · 2 citations
Computer Science · Medicine · #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.05225

openalex publication_date 2025/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The rapid advancement of Chinese LLMs underscores the need for vertical-domain evaluations to ensure reliable applications. However, existing benchmarks often lack domain coverage and provide limited insights into the Chinese working context. Leveraging qualification exams as a unified framework for expertise evaluation, we introduce QualBench, the first multi-domain Chinese QA benchmark dedicated to localized assessment of Chinese LLMs. The dataset includes over 17,000 questions across six vertical domains, drawn from 24 Chinese qualifications to align with national policies and professional standards. Results reveal an interesting pattern of Chinese LLMs consistently surpassing non-Chinese models, with the Qwen2.5 model outperforming the more advanced GPT-4o, emphasizing the value of localized domain knowledge in meeting qualification requirements. The average accuracy of 53.98% reveals the current gaps in domain coverage within model capabilities. Furthermore, we identify performance degradation caused by LLM crowdsourcing, assess data contamination, and illustrate the effectiveness of prompt engineering and model fine-tuning, suggesting opportunities for future improvements through multi-domain RAG and Federated Learning.

Citations

• A Survey on Large Language Model Benchmarks
• EraRAG: Efficient and Incremental Retrieval Augmented Generation for Growing Corpora
• Bridge the Domains: Large Language Models Enhanced Cross-domain Sequential Recommendation
• Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements
• Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements
• Dial-In LLM: Human-Aligned LLM-in-the-loop Intent Clustering for Customer Service Dialogues
• Rethinking LLM Memorization through the Lens of Adversarial Compression
• GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
• FinGPT: Large Generative Models for a Small Language
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning
• Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset
• Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
• Evaluating the Performance of Large Language Models on GAOKAO Benchmark
• AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
• ChatGPT outperforms crowd workers for text-annotation tasks
• Large Language Models Encode Clinical Knowledge
• Large language models encode clinical knowledge
• PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
• Language Models are Few-Shot Learners
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

Cited by

• Contextualized Token Discrimination for Speech Search Query Correction
• Understanding How Value Neurons Shape the Generation of Specified Values in LLMs

Related

• OpenEval: Benchmarking Chinese LLMs across Capability, Alignment and Safety
• TencentLLMEval: A Hierarchical Evaluation of Real-World Capabilities for Human-Aligned LLMs
• ZhuJiu: A Multi-dimensional, Multi-faceted Chinese Benchmark for Large Language Models
• Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmark
• Benchmarking for Domain-Specific LLMs: A Case Study on Academia and Beyond