Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Models

2025/10/14 by Ziliang Qiu, Qiu, Ziliang, Renfen Hu +1
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.12110

openalex publication_date 2025/10/14 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

The evaluation of LLMs' creativity represents a crucial research domain, though challenges such as data contamination and costly human assessments often impede progress. Drawing inspiration from human creativity assessment, we propose PACE, asking LLMs to generate Parallel Association Chains to Evaluate their creativity. PACE minimizes the risk of data contamination and offers a straightforward, highly efficient evaluation, as evidenced by its strong correlation with Chatbot Arena Creative Writing rankings (Spearman's ρ= 0.739, p < 0.001) across various proprietary and open-source models. A comparative analysis of associative creativity between LLMs and humans reveals that while high-performing LLMs achieve scores comparable to average human performance, professional humans consistently outperform LLMs. Furthermore, linguistic analysis reveals that both humans and LLMs exhibit a trend of decreasing concreteness in their associations, and humans demonstrating a greater diversity of associative patterns.

Citations

• We're Different, We're the Same: Creative Homogeneity Across LLMs
• Probing the contents of semantic representations from text, behavior, and brain data using the psychNorms metabase
• Does ChatGPT Have a Poetic Style?
• CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints
• Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
• Are Large Language Models Capable of Generating Human-Level Narratives?
• Benchmarking Language Model Creativity: A Case Study on Code Generation
• Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• Large Language Models are Inconsistent and Biased Evaluators
• How AI Ideas Affect the Creativity, Diversity, and Evolution of Human Ideas: Evidence From a Large, Dynamic Experiment
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset
• When and How Artificial Intelligence Augments Employee Creativity
• Putting GPT-3's Creativity to the (Alternative Uses) Test
• Advances in Pre-Training Distributed Word Representations
• Word Translation Without Parallel Data
• Sample Size Requirements for Estimating Pearson, Kendall and Spearman Correlations
• The associative basis of the creative process.
• AI Benchmark Half-Life in Recursive Corpora: A Theory of Validity Decay under Semantic Leakage and Regeneration

Cited by

• ConlangCrafter: Constructing Languages with a Multi-Hop LLM Pipeline

Related

• Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variability
• CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
• Do LLMs Agree on the Creativity Evaluation of Alternative Uses?
• Large Language Models show both individual and collective creativity comparable to humans
• Divergent Creativity in Humans and Large Language Models