As LLMs become increasingly prevalent, it is interesting to consider how ``creative'' these models can be. From cognitive science, creativity consists of at least two key characteristics: \emph{convergent} thinking (purposefulness to achieve a given goal) and \emph{divergent} thinking (adaptability to explore new environments or constraints) \citep{runco2003critical}. In this work, we introduce a framework for quantifying LLM creativity that incorporates the two design ingredients: (1) We introduce DENIAL PROMPTING which pushes LLMs to develop more creative solutions to a given problem by incrementally imposing new constraints on the previous solution, compelling LLMs to adopt new strategies. (2) We define NEOGAUGE, a metric that quantifies both convergent and divergent thinking in the generated creative responses by LLMs. We test the proposed framework on Codeforces problems, which serve as both a natural dataset for coding tasks and a collection of prior human solutions. We quantify NEOGAUGE for various proprietary and open-source models and find that even the most creative model, GPT-4, still falls short of demonstrating human-like creativity. We also experiment with advanced reasoning strategies (MCTS, self-correction, etc.) and observe no significant improvement in creativity. As a by-product of our analysis, we release NEOCODER dataset for reproducing our results on future models.
CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challenges
Southern University of Science and Technology(南方科技大学);Tsinghua University(清华大学);Peking University(北京大学);University of Bristol(布里斯托大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Xi’an Jiaotong University(西安交通大学)
AGC-Bench: Measuring Artificial General Creativity
Pennsylvania State University(宾夕法尼亚州立大学);University of Massachusetts Lowell(马萨诸塞大学洛厄尔分校);University of Amsterdam(阿姆斯特丹大学);Amazon AGI(亚马逊AGI);Dartmouth College(达特茅斯学院)
Benchmarking Language Models for Code Syntax Understanding
University of Maryland, College Park(马里兰大学帕克分校);Google Research(谷歌研究院);Washington University in St. Louis(圣路易斯华盛顿大学);University of California, Berkeley(加州大学伯克利分校)
Introducing multiplex semantic networks as multifaceted representations of creative associative knowledge across multilingual samples
CogNosco Lab, Department of Psychology and Cognitive Science, University of Trento(CogNosco实验室,心理学与认知科学系,特伦托大学);Department of Science Education, University of Education Upper Austria(科学教育系,上奥地利教育大学);Department of Psychology, The Pennsylvania State University(心理学系,宾夕法尼亚州立大学);Department of Psychology, National University of Singapore(心理学系,新加坡国立大学)
Decoupled Analysis-Judging: An Automated Creativity Evaluator Using LLMs in Complex Multi-step Creativity Tasks
East China Normal University(华东师范大学);Shanghai Institute of Artificial Intelligence for Education(上海人工智能教育研究院);School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Yangzhou University(扬州大学);School of Education and Intelligent Education Research Center, Yangzhou University(扬州大学教育学院与智能教育研究中心);School of Data Science and Engineering, East China Normal University(华东师范大学数据科学与工程学院)
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Raffles Institution(莱佛士书院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院);Lee Kong Chian School of Medicine, Nanyang Technological University(南洋理工大学李光前医学院);Centre of AI in Medicine (C-AIM), Nanyang Technological University(南洋理工大学人工智能医学中心)
Playing with Words, Improving with Rewards: Training Language Models for Creative Association
University of Massachusetts Lowell(马萨诸塞大学洛市分校);Dartmouth College(达特茅斯学院);University of Amsterdam(阿姆斯特丹大学);Pennsylvania State University(宾夕法尼亚州立大学);Amazon AGI(亚马逊人工智能研究院)
Divergent Creativity in Humans and Large Language Models
Université de Montréal(蒙特利尔大学);Concordia University(康考迪亚大学);Mila (Quebec AI research Institute)(米拉(魁北克人工智能研究所));University of Toronto Mississauga(多伦多大学密西沙加分校);UNIQUE Center (Quebec Neuro-AI research Center)(UNIQUE中心(魁北克神经人工智能研究中心))
GAUSS: Benchmarking Structured Mathematical Skills for Large Language Models
Hyperbolic Labs;California Institute of Technology(加州理工学院);University of California, Berkeley(加州大学伯克利分校);Stanford University(斯坦福大学);Nvidia(英伟达);University of Washington(华盛顿大学);University of Hong Kong(香港大学)
Characterising the Creative Process in Humans and Large Language Models
Max Planck Institute for Biological Cybernetics(马克斯·普朗克生物控制论研究所);University of Tübingen(蒂宾根大学);Max Planck School of Cognition(马克斯·普朗克认知科学学院);University of Amsterdam(阿姆斯特丹大学)
CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
The Hong Kong Polytechnic University(香港理工大学);University of Edinburgh(爱丁堡大学);UCL(伦敦大学学院);CUHK (SZ)(香港中文大学(深圳));SUTD(新加坡科技设计大学);HKUST(香港科技大学);CUHK(香港中文大学);HKUST (GZ)(香港科技大学(广州))
DependEval: Benchmarking LLMs for Repository Dependency Understanding
Nanyang Technological University(南洋理工大学);Tsinghua University(清华大学);Beihang University(北京航空航天大学);ShanghaiTech University(上海科技大学);The University of Hong Kong(香港大学)
Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition
New York University Abu Dhabi(纽约大学阿布扎比分校);University of Stuttgart(斯图加特大学);Johns Hopkins University(约翰斯·霍普金斯大学);Stanford University(斯坦福大学);Qatar Computing Research Institute(卡塔尔计算研究所);Microsoft Research India(微软印度研究院)
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems
National University of Singapore(新加坡国立大学);The University of Hong Kong(香港大学);Nanyang Technological University(南洋理工大学);Hong Kong Baptist University(香港浸会大学)
Functional Consistency of LLM Code Embeddings: A Self-Evolving Data Synthesis Framework for Benchmarking
School of Computer Science and Engineering, Sun Yat-Sen University(中山大学计算机科学与工程学院);School of Software Engineering, Sun Yat-Sen University(中山大学软件工程学院);School of Artificial Intelligence, Sun Yat-Sen University(中山大学人工智能学院);Key Laboratory of Sustainable Tourism Smart Assessment Technology, Ministry of Culture and Tourism of China(文化和旅游部可持续旅游智能评估技术重点实验室);Department of Computer Science, City University of Hong Kong(香港城市大学计算机科学系)
CODEMENV: Benchmarking Large Language Models on Code Migration
Provable Responsible AI and Data Analytics (PRADA) Lab(可证明责任人工智能与数据分析实验室);King Abdullah University of Science and Technology(国王 Abdullah 科学与技术大学);Peking University(北京大学);South China University of Technology(华南理工大学)