arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.09007cs.CL
收藏

基准评估语言模型创造力:以代码生成为例的案例研究

Benchmarking Language Model Creativity: A Case Study on Code Generation

University of Notre Dame(圣母大学);Johns Hopkins University(约翰斯·霍普金斯大学)

Yining Lu, Dixuan Wang, Tianjian Li, Dongwei Jiang, Sanjeev Khudanpur, Meng Jiang, Daniel Khashabi

AI总结:

本文提出DENIAL PROMPTING方法和NEOGAUGE指标,在Codeforces数据集上量化评估LLM的聚合与发散思维,发现即使GPT-4也缺乏类人创造力,并发布NEOCODER数据集供复现。

摘要:

随着大语言模型(LLMs)日益普及,思考这些模型能够具有多大程度的“创造力”成为一个有趣的问题。从认知科学的角度来看,创造力至少包含两个关键特征:聚合思维(为实现给定目标而具有的目的性)和发散思维(探索新环境或新约束的适应性)。在这项工作中,我们提出了一个用于量化LLM创造力的框架,该框架包含两个设计要素:(1)我们引入了DENIAL PROMPTING(拒绝提示)方法,通过在前一解决方案上逐步施加新约束,推动LLM为给定问题开发更具创造性的解决方案,从而迫使LLM采用新策略。(2)我们定义了NEOGAUGE这一度量指标,用于量化LLM生成的创造性响应中的聚合思维和发散思维。我们在Codeforces题目上测试了所提出的框架,这些题目既是编码任务的自然数据集,也是先前人类解决方案的集合。我们为多种专有模型和开源模型量化了NEOGAUGE,发现即使是最具创造力的模型GPT-4,也仍然未能展现出类似人类的创造力。我们还对高级推理策略(MCTS、自我修正等)进行了实验,观察到创造力没有显著提升。作为我们分析的一个副产品,我们发布了NEOCODER数据集,用于在未来模型上复现我们的结果。

英文摘要:

As LLMs become increasingly prevalent, it is interesting to consider how ``creative'' these models can be. From cognitive science, creativity consists of at least two key characteristics: \emph{convergent} thinking (purposefulness to achieve a given goal) and \emph{divergent} thinking (adaptability to explore new environments or constraints) \citep{runco2003critical}. In this work, we introduce a framework for quantifying LLM creativity that incorporates the two design ingredients: (1) We introduce DENIAL PROMPTING which pushes LLMs to develop more creative solutions to a given problem by incrementally imposing new constraints on the previous solution, compelling LLMs to adopt new strategies. (2) We define NEOGAUGE, a metric that quantifies both convergent and divergent thinking in the generated creative responses by LLMs. We test the proposed framework on Codeforces problems, which serve as both a natural dataset for coding tasks and a collection of prior human solutions. We quantify NEOGAUGE for various proprietary and open-source models and find that even the most creative model, GPT-4, still falls short of demonstrating human-like creativity. We also experiment with advanced reasoning strategies (MCTS, self-correction, etc.) and observe no significant improvement in creativity. As a by-product of our analysis, we release NEOCODER dataset for reproducing our results on future models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑