FLIP推理挑战
FLIP Reasoning Challenge
AI总结：
针对AI推理能力不足的问题，本文提出FLIP数据集作为多模态推理基准，评估显示现有最优模型零样本准确率远低于人类，集成模型或结合字幕模型可提升性能，凸显FLIP对推动推理研究的价值。
摘要：
过去几年，人工智能（AI）的进步展示了其在解决图像分类、文本生成等感知与生成任务上的能力，但推理仍是一大挑战。本文介绍FLIP数据集，这是一个基于Idena区块链上人类验证任务的AI推理能力评估基准。FLIP挑战向用户呈现4张图像的两种排序，要求识别逻辑连贯的一种。通过强调序列推理、视觉叙事和常识，FLIP为多模态AI系统提供了独特的测试平台。我们的实验评估了最先进的模型，同时利用视觉语言模型（VLMs）和大型语言模型（LLMs）。结果显示，即使是最佳的开源和闭源模型，在零样本设置下的最高准确率分别为75.5%和77.9%，而人类表现为95.3%。字幕模型通过提供图像的文本描述帮助推理模型，相比直接使用原始图像取得了更好的结果，Gemini 1.5 Pro的准确率从69.6%提升至75.2%。将15个模型的预测结果集成后，准确率提升至85.2%。这些发现凸显了现有推理模型的局限性，以及对FLIP这类稳健多模态基准的需求。完整代码库和数据集将在https://github.com/aplesner/FLIP-Reasoning-Challenge上提供。
英文摘要：
Over the past years, advances in artificial intelligence (AI) have demonstrated how AI can solve many perception and generation tasks, such as image classification and text writing, yet reasoning remains a challenge. This paper introduces the FLIP dataset, a benchmark for evaluating AI reasoning capabilities based on human verification tasks on the Idena blockchain. FLIP challenges present users with two orderings of 4 images, requiring them to identify the logically coherent one. By emphasizing sequential reasoning, visual storytelling, and common sense, FLIP provides a unique testbed for multimodal AI systems. Our experiments evaluate state-of-the-art models, leveraging both vision-language models (VLMs) and large language models (LLMs). Results reveal that even the best open-sourced and closed-sourced models achieve maximum accuracies of 75.5% and 77.9%, respectively, in zero-shot settings, compared to human performance of 95.3%. Captioning models aid reasoning models by providing text descriptions of images, yielding better results than when using the raw images directly, 69.6% vs. 75.2% for Gemini 1.5 Pro. Combining the predictions from 15 models in an ensemble increases the accuracy to 85.2%. These findings highlight the limitations of existing reasoning models and the need for robust multimodal benchmarks like FLIP. The full codebase and dataset will be available at https://github.com/aplesner/FLIP-Reasoning-Challenge.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
