On Recipe Memorization and Creativity in Large Language Models: Is Your Model a Creative Cook, a Bad Cook, or Merely a Plagiator?

2025/06/30 by Jan Kvapil, Kvapil, Jan, Martin Fajčík +1
Agricultural and Biological Sciences · Computer Science · #AI in Service Interactions #Computation and Language (cs.CL) #Culinary Culture and Tourism #FOS: Computer and information sciences #Sentiment Analysis and Opinion Mining

paper · pdf · doi:10.48550/arxiv.2506.23527

openalex publication_date 2025/06/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This work-in-progress investigates the memorization, creativity, and nonsense found in cooking recipes generated from Large Language Models (LLMs). Precisely, we aim (i) to analyze memorization, creativity, and non-sense in LLMs using a small, high-quality set of human judgments and (ii) to evaluate potential approaches to automate such a human annotation in order to scale our study to hundreds of recipes. To achieve (i), we conduct a detailed human annotation on 20 preselected recipes generated by LLM (Mixtral), extracting each recipe's ingredients and step-by-step actions to assess which elements are memorized--i.e., directly traceable to online sources possibly seen during training--and which arise from genuine creative synthesis or outright nonsense. We find that Mixtral consistently reuses ingredients that can be found in online documents, potentially seen during model training, suggesting strong reliance on memorized content. To achieve aim (ii) and scale our analysis beyond small sample sizes and single LLM validation, we design an ``LLM-as-judge'' pipeline that automates recipe generation, nonsense detection, parsing ingredients and recipe steps, and their annotation. For instance, comparing its output against human annotations, the best ingredient extractor and annotator is Llama 3.1+Gemma 2 9B, achieving up to 78% accuracy on ingredient matching. This automated framework enables large-scale quantification of memorization, creativity, and nonsense in generated recipes, providing rigorous evidence of the models' creative capacities.

Citations

• The Llama 3 Herd of Models
• OLMES: A Standard for Language Model Evaluations
• Monte Carlo Tree Search for Recipe Generation using GPT-2
• Scalable Extraction of Training Data from (Production) Language Models
• "Mango Mango, How to Let The Lettuce Dry Without A Spinner?": Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner
• Large Language Models as Sous Chefs: Revising Recipes with GPT-3
• The Authenticity Gap in Human Evaluation
• On the Dangers of Stochastic Parrots
• The Standard Definition of Creativity
• A Coefficient of Agreement for Nominal Scales

Related

• The Digital Sous Chef -- A Comparative Study on Fine-Tuning Language Models for Recipe Generation
• Large Language Models as Sous Chefs: Revising Recipes with GPT-3
• Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenario
• Large Language Models for Ingredient Substitution in Food Recipes using Supervised Fine-tuning and Direct Preference Optimization
• Culinary Class Wars: Evaluating LLMs using ASH in Cuisine Transfer Task