一种因果感知的多模态大语言模型创造力评估范式
A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Models
AI总结：
本文提出因果感知的LoTbench框架，基于Oogiri游戏评估多模态LLM创造力，量化性能并可视化思维过程，发现与认知基准强相关，优于传统指标。
摘要：
近年来，大量基准测试被开发出来，用于评估大语言模型（LLMs）的逻辑推理能力。然而，由于创造力具有主观性、多样性和数据稀缺性，评估同样重要的LLM创造能力颇具挑战性，尤其是在多模态场景中。在本文中，我们考虑了评估多模态LLM创造力的完整流程，重点关注合适的评估平台和方法。首先，我们发现Oogiri游戏，这是一种由创造力驱动的任务，需要幽默感、联想思维以及产生对文本、图像或两者的意外回应的能力。该游戏与现代多模态LLM的输入-输出结构高度契合，并受益于丰富的高质量、人工标注的创意回应库，使其成为研究LLM创造力的理想平台。其次，除了使用Oogiri游戏进行标准评估（如排序和选择）之外，我们提出了LoTbench，一个交互式、因果感知的评估框架，以进一步解决标准评估中的一些固有风险，如信息泄露和有限的可解释性。所提出的LoTbench不仅更有效地量化了LLM的创造力，而且可视化了潜在的创造性思维过程。我们的结果表明，尽管大多数LLM表现出受限的创造力，但LLM与人类之间的性能差距并非不可逾越。此外，我们观察到多模态认知基准MMMU与LoTbench的结果之间存在强相关性，但与传统的创造力指标仅存在弱关联。这表明LoTbench更符合人类认知理论，强调认知是创造力早期阶段的关键基础，并能够连接不同概念。https://lotbench.github.io
英文摘要：
Recently, numerous benchmarks have been developed to evaluate the logical reasoning abilities of large language models (LLMs). However, assessing the equally important creative capabilities of LLMs is challenging due to the subjective, diverse, and data-scarce nature of creativity, especially in multimodal scenarios. In this paper, we consider the comprehensive pipeline for evaluating the creativity of multimodal LLMs, with a focus on suitable evaluation platforms and methodologies. First, we find the Oogiri game, a creativity-driven task requiring humor, associative thinking, and the ability to produce unexpected responses to text, images, or both. This game aligns well with the input-output structure of modern multimodal LLMs and benefits from a rich repository of high-quality, human-annotated creative responses, making it an ideal platform for studying LLM creativity. Next, beyond using the Oogiri game for standard evaluations like ranking and selection, we propose LoTbench, an interactive, causality-aware evaluation framework, to further address some intrinsic risks in standard evaluations, such as information leakage and limited interpretability. The proposed LoTbench not only quantifies LLM creativity more effectively but also visualizes the underlying creative thought processes. Our results show that while most LLMs exhibit constrained creativity, the performance gap between LLMs and humans is not insurmountable. Furthermore, we observe a strong correlation between results from the multimodal cognition benchmark MMMU and LoTbench, but only a weak connection with traditional creativity metrics. This suggests that LoTbench better aligns with human cognitive theories, highlighting cognition as a critical foundation in the early stages of creativity and enabling the bridging of diverse concepts. https://lotbench.github.io
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
