Assessing and Understanding Creativity in Large Language Models
University of Science and Technology of China（中国科学技术大学）；Chinese Academy of Sciences（中国科学院）；Shanghai Innovation Center for Processor Technologies（上海处理器技术创新中心）；University of Chinese Academy of Sciences（中国科学院大学）
In the field of natural language processing, the rapid development of large language model (LLM) has attracted more and more attention. LLMs have shown a high level of creativity in various tasks, but the methods for assessing such creativity are inadequate. The assessment of LLM creativity needs to consider differences from humans, requiring multi-dimensional measurement while balancing accuracy and efficiency. This paper aims to establish an efficient framework for assessing the level of creativity in LLMs. By adapting the modified Torrance Tests of Creative Thinking, the research evaluates the creative performance of various LLMs across 7 tasks, emphasizing 4 criteria including Fluency, Flexibility, Originality, and Elaboration. In this context, we develop a comprehensive dataset of 700 questions for testing and an LLM-based evaluation method. In addition, this study presents a novel analysis of LLMs' responses to diverse prompts and role-play situations. We found that the creativity of LLMs primarily falls short in originality, while excelling in elaboration. Besides, the use of prompts and the role-play settings of the model significantly influence creativity. Additionally, the experimental results also indicate that collaboration among multiple LLMs can enhance originality. Notably, our findings reveal a consensus between human evaluations and LLMs regarding the personality traits that influence creativity. The findings underscore the significant impact of LLM design on creativity and bridges artificial intelligence and human creativity, offering insights into LLMs' creativity and potential applications.
Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation
Natural Language Learning Group (NLLG) School of Business Informatics and Mathematics University of Mannheim（曼海姆大学自然语言学习组（NLLG）商学院信息与数学学院）；University of Technology Nuremberg (UTN), Department Engineering（纽伦堡技术大学（UTN）工程系）；University of Gent, Department of Translation, Interpreting and Communication（根特大学翻译、口译与传播系）；University of Aberdeen, Department of Computing Science（阿伯丁大学计算科学系）；Natural Language Learning and Generation (NLLG) Lab（自然语言学习与生成（NLLG）实验室）
Question: I have five fingers but I am not alive. What am I? Answer: a glove. Answering such a riddle-style question is a challenging cognitive process, in that it requires complex commonsense reasoning abilities, an understanding of figurative language, and counterfactual reasoning skills, which are all important abilities for advanced natural language understanding (NLU). However, there are currently no dedicated datasets aiming to test these abilities. Herein, we present RiddleSense, a new multiple-choice question answering task, which comes with the first large dataset (5.7k examples) for …
LLMs can Realize Combinatorial Creativity: Generating Creative Ideas via LLMs for Scientific Research
Boston University（波士顿大学）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Delaware（特拉华大学）；Acaciawood Preparatory Academy（阿卡西亚伍德预备学院）
Automated Creativity Evaluation of Language Models Across Open-Ended Tasks
Raffles Institution（莱佛士书院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Lee Kong Chian School of Medicine, Nanyang Technological University（南洋理工大学李光前医学院）；Centre of AI in Medicine (C-AIM), Nanyang Technological University（南洋理工大学人工智能医学中心）
Probing and Inducing Combinational Creativity in Vision-Language Models
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Yuanpei College, Peking University（北京大学元培学院）；State Key Laboratory of General Artificial Intelligence, BIGAI（人工智能通用基础理论国家重点实验室）；Center on Frontiers of Computing Studies, School of Computer Science, Peking University（计算机科学学院前沿计算研究中心）
Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports
Indian Institute of Technology Patna（印度理工学院帕纳布分校）；Sardar Patel Institute of Technology（萨达尔·帕特尔技术学院）；Universitas Gadjah Mada（加查马大学）；King Mongkut’s Institute of Technology Ladkrabang（拉差班国王技术学院）；Shenzhen Technology University（深圳技术大学）；Université de Toulouse（图卢兹大学）
As large-scale, pre-trained language models achieve human-level and superhuman accuracy on existing language understanding tasks, statistical bias in benchmark data and probing studies have recently called into question their true capabilities. For a more informative evaluation than accuracy on text classification tasks can offer, we propose evaluating systems through a novel measure of prediction coherence. We apply our framework to two existing language understanding benchmarks with different properties to demonstrate its versatility. Our experimental results show that this evaluation framew…
Leveraging Large Language Models for NLG Evaluation: Advances and Challenges
Peking University（北京大学）；Institute of Information Engineering, CAS（中国科学院信息工程研究所）；AAII, FEIT, University of Technology Sydney（悉尼科技大学）；UCLA（加利福尼亚大学洛杉矶分校）；The Open University of China（国家开放大学）；Beihang University（北京航空航天大学）
VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院（AIR））；Tencent Robotics X（腾讯Robotics X实验室）；University of Toronto（多伦多大学）；Zhili College, Tsinghua University（清华大学志道书院）
CAREBench: Evaluating LLMs' Emotion Understanding by Assessing Cognitive Appraisal Reasoning
Department of Informatics（信息学院）；King’s College London（伦敦国王学院）；Institute of Psychology（心理学研究所）；University of Tartu（塔尔图大学）；Department of Psychology（心理学系）；Stanford University（斯坦福大学）；The Alan Turing Institute（艾伦·图灵研究所）
Can MLLMs Decode the Creative Leap? Introducing C4 for Cross-Concept Understanding
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；School of Computer Science and Engineering, Shandong University of Science and Technology（山东科技大学计算机科学与工程学院）
StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
Tsinghua University（清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；Beijing University of Posts and Telecommunications（北京邮电大学）
Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understanding
Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, Zhejiang University（浙江可感知智能系统重点实验室，浙江大学）；Alibaba Group（阿里巴巴集团）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and DataSecurity（杭州高新技术区（滨江）区块链与数据安全研究院）
Recent advances in NLP have been attributed to the emergence of large-scale pre-trained language models. GPT-2, in particular, is suited for generation tasks given its left-to-right language modeling objective, yet the linguistic quality of its generated text has largely remain unexplored. Our work takes a step in understanding GPT-2's outputs in terms of discourse coherence. We perform a comprehensive study on the validity of explicit discourse relations in GPT-2's outputs under both organic generation and fine-tuned scenarios. Results show GPT-2 does not always generate text containing valid…
A Model-Internal Protocol for Assessing Multimodal Models as Integrated Systems
Hangzhou Institute for Advanced Study（杭州高等研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Computer Network Information Center（计算机网络信息中心）；Chinese Academy of Sciences（中国科学院）
Evaluating the Creativity of LLMs in Persian Literary Text Generation
Tehran Institute for Advanced Studies, Khatam University, Iran（德黑兰高级研究学院，卡坦大学，伊朗）；School of Electrical and Computer Engineering, College of Engineering, University of Tehran, Tehran, Iran（电气与计算机工程学院，工程学院，德黑兰大学，德黑兰，伊朗）
ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic
MBZUAI（穆罕默德·本·扎耶德人工智能大学）；Prince Sattam bin Abdulaziz University（萨塔姆·本·阿卜杜勒阿齐兹王子大学）；King Fahd University of Petroleum and Minerals（法赫德国王石油与矿业大学）；Core42；New York University Abu Dhabi（纽约大学阿布扎比分校）；The University of Melbourne（墨尔本大学）
University of Tokyo（东京大学）；University College London（伦敦大学学院）；Tohoku University（东北大学）；RIKEN Center for AIP（理化学研究所人工智能中心）；National Institute of Informatics（国立信息学研究所）
Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems. However, the capabilities of datasets are not assessed for benchmarking language understanding precisely. We propose a semi-automated, ablation-based methodology for this challenge; By checking whether questions can be solved even after removing features associated with a skill requisite for language understanding, we evaluate to what degree the questions do not require the skill. Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a strong baseline mod…
