arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.11522cs.CV
收藏

FIRE:多模态模型反馈整合与优化评估数据集

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models

Beijing Institute of Technology(北京理工大学);BIGAI(北京智源人工智能研究院);Peking University(北京大学);Monash University(莫纳什大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Tsinghua University(清华大学)

Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun Zhu, Qing Li

AI总结:

针对视觉语言模型(VLMs)的反馈优化能力评估需求,本文构建包含110万轮对话的FIRE数据集及FIRE-Bench基准,开发的FIRE-LLaVA模型在该基准上性能优于未训练模型50%,提升了用户-智能体交互效率。

摘要:

视觉语言模型(VLMs)在各类应用中取得了显著进展,已成为主流研究方向。本文构建了FIRE——一个反馈优化数据集,包含源自27个源数据集的110万轮多轮对话,能让VLMs基于用户在不同任务中的反馈自发优化响应。为扩大数据收集规模,FIRE分为两部分:FIRE-100K和FIRE-1M,其中FIRE-100K由GPT-4V生成,FIRE-1M由在FIRE-100K上训练的模型自动生成。此外,我们构建了FIRE-Bench基准,用于全面评估VLMs的反馈优化能力,包含11000条反馈优化对话作为测试数据、两种评估设置及一个为VLMs提供反馈的模型。我们通过在FIRE-100K和FIRE-1M上微调LLaVA开发了FIRE-LLaVA模型,该模型在FIRE-Bench上表现出卓越的反馈优化能力,比未训练的VLMs性能提升50%,实现了更高效的用户-智能体交互,凸显了FIRE数据集的重要性。

英文摘要:

Vision language models (VLMs) have achieved impressive progress in diverse applications, becoming a prevalent research direction. In this paper, we build FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn conversations that are derived from 27 source datasets, empowering VLMs to spontaneously refine their responses based on user feedback across diverse tasks. To scale up the data collection, FIRE is collected in two components: FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a benchmark to comprehensively evaluate the feedback-refining capability of VLMs, which contains 11K feedback-refinement conversations as the test data, two evaluation settings, and a model to provide feedback for VLMs. We develop the FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows remarkable feedback-refining capability on FIRE-Bench and outperforms untrained VLMs by 50%, making more efficient user-agent interactions and underscoring the significance of the FIRE dataset.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑