2025/09/05 by Navya Bhat, Bhat, Ninad, Kieran Browne +3 · 1 citation
Business, Management and Accounting · Computer Science · #Artificial Intelligence (cs.AI) #Big Data and Business Intelligence #Computation and Language (cs.CL) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Open Source Software Innovations
paper · pdf · doi:10.48550/arxiv.2509.09702
openalex publication_date 2025/09/05 · openalex created_date 2025/09/29 · openalex updated_date 2026/08/01
We introduce Creativity Benchmark, an evaluation framework for large language models (LLMs) in marketing creativity. The benchmark covers 100 brands (12 categories) and three prompt types (Insights, Ideas, Wild Ideas). Human pairwise preferences from 678 practising creatives over 11,012 anonymised comparisons, analysed with Bradley-Terry models, show tightly clustered performance with no model dominating across brands or prompt types: the top-bottom spread is Δθ≈ 0.45, which implies a head-to-head win probability of 0.61; the highest-rated model beats the lowest only about 61% of the time. We also analyse model diversity using cosine distances to capture intra- and inter-model variation and sensitivity to prompt reframing. Comparing three LLM-as-judge setups with human rankings reveals weak, inconsistent correlations and judge-specific biases, underscoring that automated judges cannot substitute for human evaluation. Conventional creativity tests also transfer only partially to brand-constrained tasks. Overall, the results highlight the need for expert human evaluation and diversity-aware workflows.