Role-playing Prompt Framework: Generation and Evaluation

2024/06/02 by Liu, Xun, Ni, Zhengwei
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2406.00627

Abstract

Large language models (LLMs) exhibit impressive proficiency in natural language generation, understanding user instructions, and emulating human-like language use, which has led to significant interest in their application to role-playing scenarios. However, the manual collection of role-specific script data and the evaluation of model performance are resource-intensive processes. This paper introduces a prompt-based framework designed to leverage GPT's capabilities for the generation of role-playing dialogue datasets and the evaluation of role-playing performance. To validate the effectiveness of the GPT-based generation and evaluation, we further incorporate the recall-oriented Rouge-L metric, providing an additional quantitative measure of performance.

Related

• Role-Playing Evaluation for Large Language Models
• RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models
• PingPong: A Benchmark for Role-Playing Language Models with User Emulation and Multi-Model Evaluation
• RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following
• RMTBench: Benchmarking LLMs Through Multi-Turn User-Centric Role-Playing