CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulation

2025/02/13 by Xintao Wang, Heng Wang, Wang, Xintao +21 · 17 citations
Computer Science · Health Professions · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Electronic Health Records Systems #FOS: Computer and information sciences #Persona Design and Applications

paper · pdf · doi:10.48550/arxiv.2502.09082

openalex publication_date 2025/02/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/02

Abstract

Role-playing language agents (RPLAs) have emerged as promising applications of large language models (LLMs). However, simulating established characters presents a challenging task for RPLAs, due to the lack of authentic character datasets and nuanced evaluation methods using such data. In this paper, we present CoSER, a collection of a high-quality dataset, open models, and an evaluation protocol towards effective RPLAs of established characters. The CoSER dataset covers 17,966 characters from 771 renowned books. It provides authentic dialogues with real-world intricacies, as well as diverse data types such as conversation setups, character experiences and internal thoughts. Drawing from acting methodology, we introduce given-circumstance acting for training and evaluating role-playing LLMs, where LLMs sequentially portray multiple characters in book scenes. Using our dataset, we develop CoSER 8B and CoSER 70B, i.e., advanced open role-playing LLMs built on LLaMA-3.1 models. Extensive experiments demonstrate the value of the CoSER dataset for RPLA training, evaluation and retrieval. Moreover, CoSER 70B exhibits state-of-the-art performance surpassing or matching GPT-4o on our evaluation and three existing benchmarks, i.e., achieving 75.80% and 93.47% accuracy on the InCharacter and LifeChoice benchmarks respectively.

Cited by

• Understanding Generalization in Role-Playing Models via Information Theory
• RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems
• Too Good to be Bad: On the Failure of LLMs to Role-Play Villains
• GRAPHIA: Harnessing Social Graph Data to Enhance LLM-Based Social Simulation
• Individualized Cognitive Simulation in Large Language Models: Evaluating Different Cognitive Representation Methods
• NarraBench: A Comprehensive Framework for Narrative Benchmarking
• FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline
• Social Welfare Function Leaderboard: When LLM Agents Allocate Social Welfare
• Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation
• The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies
• VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents
• CPO: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization
• LLMs vs. Chinese Anime Enthusiasts: A Comparative Study on Emotionally Supportive Role-Playing
• PersonaEval: Are LLM Evaluators Human Enough to Judge Role-Play?
• HAMLET: A Hierarchical and Adaptive Multi-Agent Framework for Live Embodied Theatrics
• CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards
• Test-Time-Matching: Decouple Personality, Memory, and Linguistic Style in LLM-based Role-Playing Language Agent

Related

• Character is Destiny: Can Role-Playing Language Agents Make Persona-Driven Decisions?
• OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personas
• Role-Playing Evaluation for Large Language Models
• The Oscars of AI Theater: A Survey on Role-Playing with Language Models
• From Persona to Personalization: A Survey on Role-Playing Language Agents