大型语言模型作为自动化对齐器用于基准测试视觉语言模型
Large Language Models as Automated Aligners for benchmarking Vision-Language Models
AI总结：
本文提出Auto-Bench框架，利用大型语言模型作为自动化对齐器，通过自动生成问答推理三元组和作为评判者进行评估，全面衡量视觉语言模型与人类智能及价值观的对齐程度。
摘要：
随着大型语言模型（LLMs）的进步，视觉语言模型（VLMs）已达到新的复杂度水平，在执行复杂认知和推理任务方面展现出显著能力。然而，现有的评估基准主要依赖僵化的人工制作数据集来衡量特定任务性能，在评估这些日益拟人化的模型与人类智能的对齐程度时面临重大局限。在这项工作中，我们通过Auto-Bench解决了这些局限，该框架深入探索将LLMs作为熟练的对齐器，通过自动数据整理和评估来衡量VLMs与人类智能及价值观的对齐程度。具体而言，在数据整理方面，Auto-Bench利用LLMs（例如GPT-4）通过对视觉符号表示（例如字幕、对象位置、实例关系等）进行提示，自动生成大量问答-推理三元组。由于LLMs中嵌入的广泛世界知识，整理的数据与人类意图紧密匹配。通过此流程，共整理了28.5K个人工验证和3,504K个未过滤的问答-推理三元组，涵盖4项主要能力和16项子能力。随后，我们引入GPT-3.5等LLMs作为评判者，实施定量和定性的自动化评估，以促进对VLMs的全面评估。我们的验证结果显示，LLMs在评估数据整理和模型评估方面都很熟练，平均一致率达到85%。我们设想Auto-Bench作为一个灵活、可扩展且全面的基准，用于评估不断演进的复杂VLMs。
英文摘要：
With the advancements in Large Language Models (LLMs), Vision-Language Models (VLMs) have reached a new level of sophistication, showing notable competence in executing intricate cognition and reasoning tasks. However, existing evaluation benchmarks, primarily relying on rigid, hand-crafted datasets to measure task-specific performance, face significant limitations in assessing the alignment of these increasingly anthropomorphic models with human intelligence. In this work, we address the limitations via Auto-Bench, which delves into exploring LLMs as proficient aligners, measuring the alignment between VLMs and human intelligence and value through automatic data curation and assessment. Specifically, for data curation, Auto-Bench utilizes LLMs (e.g., GPT-4) to automatically generate a vast set of question-answer-reasoning triplets via prompting on visual symbolic representations (e.g., captions, object locations, instance relationships, and etc.). The curated data closely matches human intent, owing to the extensive world knowledge embedded in LLMs. Through this pipeline, a total of 28.5K human-verified and 3,504K unfiltered question-answer-reasoning triplets have been curated, covering 4 primary abilities and 16 sub-abilities. We subsequently engage LLMs like GPT-3.5 to serve as judges, implementing the quantitative and qualitative automated assessments to facilitate a comprehensive evaluation of VLMs. Our validation results reveal that LLMs are proficient in both evaluation data curation and model assessment, achieving an average agreement rate of 85%. We envision Auto-Bench as a flexible, scalable, and comprehensive benchmark for evaluating the evolving sophisticated VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
