论大型语言模型中的食谱记忆与创造力：你的模型是创意厨师、糟糕厨师，还是纯粹的抄袭者？
On Recipe Memorization and Creativity in Large Language Models: Is Your Model a Creative Cook, a Bad Cook, or Merely a Plagiator?
AI总结：
本文研究大型语言模型生成食谱时的记忆、创造力与无意义内容，通过人工标注评估Mixtral的成分重用现象，并设计“LLM作为评判者”流水线实现大规模自动化评估，最佳模型在成分匹配上达78%准确率。
摘要：
这项进行中的工作调查了大型语言模型（LLMs）生成的烹饪食谱中的记忆、创造力和无意义内容。具体而言，我们的目标是利用一小批高质量的人类判断来分析LLMs中的记忆、创造力和无意义内容，并评估自动化此类人类标注的潜在方法，以便将我们的研究扩展到数百个食谱。为实现目标，我们对LLM（Mixtral）生成的20个预选食谱进行了详细的人类标注，提取每个食谱的成分和逐步操作，以评估哪些元素是被记忆的——即可以直接追溯到可能在训练期间见过的在线来源——哪些是源于真正的创造性综合或彻头彻尾的无意义内容。我们发现，Mixtral持续重用可以在在线文档中找到的成分，这些文档可能在模型训练期间被见过，这表明模型严重依赖记忆的内容。为实现目标并将我们的分析扩展到小样本量和单一LLM验证之外，我们设计了一个“LLM作为评判者”的流水线，用于自动化食谱生成、无意义内容检测、成分和食谱步骤的解析及其标注。例如，将其输出与人类标注进行比较，最佳的成分提取器和标注器是Llama 3.1+Gemma 2 9B，在成分匹配上达到了高达78%的准确率。这一自动化框架能够对生成食谱中的记忆、创造力和无意义内容进行大规模量化，为模型的创造力提供了严格的证据。
英文摘要：
This work-in-progress investigates the memorization, creativity, and nonsense found in cooking recipes generated from Large Language Models (LLMs). Precisely, we aim (i) to analyze memorization, creativity, and non-sense in LLMs using a small, high-quality set of human judgments and (ii) to evaluate potential approaches to automate such a human annotation in order to scale our study to hundreds of recipes. To achieve (i), we conduct a detailed human annotation on 20 preselected recipes generated by LLM (Mixtral), extracting each recipe's ingredients and step-by-step actions to assess which elements are memorized--i.e., directly traceable to online sources possibly seen during training--and which arise from genuine creative synthesis or outright nonsense. We find that Mixtral consistently reuses ingredients that can be found in online documents, potentially seen during model training, suggesting strong reliance on memorized content. To achieve aim (ii) and scale our analysis beyond small sample sizes and single LLM validation, we design an ``LLM-as-judge'' pipeline that automates recipe generation, nonsense detection, parsing ingredients and recipe steps, and their annotation. For instance, comparing its output against human annotations, the best ingredient extractor and annotator is Llama 3.1+Gemma 2 9B, achieving up to 78% accuracy on ingredient matching. This automated framework enables large-scale quantification of memorization, creativity, and nonsense in generated recipes, providing rigorous evidence of the models' creative capacities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
