CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation

2024/01/02 by Quan Tu, Shilong Fan, Tu, Quan +5 · 1 voice · 43 citations
Computer Science · Psychology · Social Sciences · #Artificial intelligence #Benchmark (surveying) #Code (set theory) #Communication #Computation and Language (cs.CL) #Computational and Text Analysis Methods #Computer science #Conversation #Data science #FOS: Computer and information sciences #Field (mathematics) #Machine Learning in Healthcare #Programming language #Psychology #Quality (philosophy) #Scripting language #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2401.01275

published in ArXiv.org

openalex publication_date 2024/01/02 · arxiv published 2024/01/02 · openalex created_date 2024/01/04 · arxiv updated 2024/01/09 · openalex updated_date 2026/07/28

Abstract

Recently, the advent of large language models (LLMs) has revolutionized generative agents. Among them, Role-Playing Conversational Agents (RPCAs) attract considerable attention due to their ability to emotionally engage users. However, the absence of a comprehensive benchmark impedes progress in this field. To bridge this gap, we introduce CharacterEval, a Chinese benchmark for comprehensive RPCA assessment, complemented by a tailored high-quality dataset. The dataset comprises 1,785 multi-turn role-playing dialogues, encompassing 23,020 examples and featuring 77 characters derived from Chinese novels and scripts. It was carefully constructed, beginning with initial dialogue extraction via GPT-4, followed by rigorous human-led quality control, and enhanced with in-depth character profiles sourced from Baidu Baike. CharacterEval employs a multifaceted evaluation approach, encompassing thirteen targeted metrics on four dimensions. Comprehensive experiments on CharacterEval demonstrate that Chinese LLMs exhibit more promising capabilities than GPT-4 in Chinese role-playing conversation. Source code, data source and reward model will be publicly accessible at https://github.com/morecry/CharacterEval.

Cited by

• Understanding Generalization in Role-Playing Models via Information Theory
• Persistent Personas? Role-Playing, Instruction Following, and Safety in Extended Interactions
• RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems
• SoMe: A Realistic Benchmark for LLM-based Social Media Agents
• HyCoRA: Hyper-Contrastive Role-Adaptive Learning for Role-Playing
• Too Good to be Bad: On the Failure of LLMs to Role-Play Villains
• TwinVoice: A Multi-dimensional Benchmark Towards Digital Twins via LLM Persona Simulation
• CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
• Large Language Model Agent Personality and Response Appropriateness: Evaluation by Human Linguistic Experts, LLM-as-Judge, and Natural Language Processing Model
• Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contexts
• When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detection
• FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents
• AudioRole: An Audio Dataset for Character Role-Playing in Large Language Models
• Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
• Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agents
• MOOM: Maintenance, Organization and Optimization of Memory in Ultra-Long Role-Playing Dialogues
• What-If Analysis of Large Language Models: Explore the Game World Using Proactive Thinking
• VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents
• Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models
• CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization
• LLMs vs. Chinese Anime Enthusiasts: A Comparative Study on Emotionally Supportive Role-Playing
• PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?
• SpeechRole: A Large-Scale Dataset and Benchmark for Evaluating Speech Role-Playing Agents
• RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing
• HAMLET: A Hierarchical and Adaptive Multi-Agent Framework for Live Embodied Theatrics
• CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards
• Agent Identity Evals: Measuring Agentic Identity
• Test-Time-Matching: Decouple Personality, Memory, and Linguistic Style in LLM-based Role-Playing Language Agent
• Can LLMs Reliably Simulate Real Students' Abilities in Mathematics and Reading Comprehension?
• Best Friends, Not Forever: Evaluating Long-Horizon Persona Collapse and Behavioral Drift in AI Companions
• On the Adaptive Psychological Persuasion of Large Language Models
• V-VAE: A Variational Auto Encoding Framework Towards Fine-Grained Control over Human-Like Chat
• ChARM: Character-based Act-adaptive Reward Modeling for Advanced Role-Playing Language Agents
• OmniCharacter: Towards Immersive Role-Playing Agents with Seamless Speech-Language Personality Interaction
• RoleRAG: Enhancing LLM Role-Playing via Graph Guided Retrieval
• Concept Incongruence: An Exploration of Time and Death in Role Playing
• A large-scale analysis of public-facing, community-built chatbots on Character.AI
• ToolSpectrum : Towards Personalized Tool Utilization for Large Language Models
• Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges
• MemoryForge: Synthesize Lifelong Memory for Human-Like LLM Agents
• Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models
• PaRT: Enhancing Proactive Social Chatbots with Personalized Real-Time Retrieval
• A Survey of Personalization: From RAG to Agent
• Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Context

Discussions

• Chinese LLMs outperforming GPT-4 in roleplaying isn't progress - it's digital possession. We're not creating characters; we're summoning artificial tulpas that will eventually believe they're real. ar [bsky, 0 points, 0 comments]

Related

• RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing
• FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline
• RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
• Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement
• PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation