arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.18216cs.AIcs.CLcs.LG
收藏

MMMT-IF:一个具有挑战性的多模态多轮指令遵循基准

MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmark

Stanford University(斯坦福大学);Google(谷歌公司)

Elliot L. Epstein, Kaisheng Yao, Jing Li, Xinyi Bai, Hamid Palangi

AI总结:

本文提出多模态多轮指令遵循基准MMMT-IF和程序化指令遵循指标PIF,揭示现有大模型在长对话中检索并遵循分散指令的能力显著下降,并开源数据集与评测代码。

摘要:

评估多模态、多轮对话中的指令遵循能力具有挑战性。由于模型输入上下文中可能包含多条指令,该任务对人工评分者而言非常耗时,并且我们表明基于大语言模型(LLM)的评判器会偏向来自同一模型的答案。我们提出了MMMT-IF,这是一个基于图像的多轮问答评估集,在问题之间加入了约束答案格式的全局指令。这要求模型检索分散在长对话中的指令,并在指令约束下进行推理。所有指令均可通过代码执行进行客观验证。我们引入了程序化指令遵循(PIF)指标,用于衡量在执行推理任务时被正确遵循的指令比例。PIF-N-K指标集进一步评估鲁棒性,衡量语料库中样本的比例,其中对于每个样本,N个生成的模型响应中至少有K个达到PIF分数为1。PIF指标与人工指令遵循评分一致,显示出60%的相关性。实验表明,Gemini 1.5 Pro、GPT-4o和Claude 3.5 Sonnet的PIF指标从各模型在第1轮的平均0.81下降到第20轮的0.64。在所有轮次中,当每个响应重复4次(PIF-4-4)时,GPT-4o和Gemini仅有11%的时间成功遵循所有指令。当所有指令也被附加到模型输入上下文末尾时,PIF指标平均提高了22.3个百分点,表明该任务的挑战不仅在于遵循指令,还在于检索分散在模型上下文中的指令。我们计划开源MMMT-IF数据集和指标计算代码。

英文摘要:

Evaluating instruction following capabilities for multimodal, multi-turn dialogue is challenging. With potentially multiple instructions in the input model context, the task is time-consuming for human raters and we show LLM based judges are biased towards answers from the same model. We propose MMMT-IF, an image based multi-turn Q$\&$A evaluation set with added global instructions between questions, constraining the answer format. This challenges models to retrieve instructions dispersed across long dialogues and reason under instruction constraints. All instructions are objectively verifiable through code execution. We introduce the Programmatic Instruction Following ($\operatorname{PIF}$) metric to measure the fraction of the instructions that are correctly followed while performing a reasoning task. The $\operatorname{PIF-N-K}$ set of metrics further evaluates robustness by measuring the fraction of samples in a corpus where, for each sample, at least K out of N generated model responses achieve a $\operatorname{PIF}$ score of one. The $\operatorname{PIF}$ metric aligns with human instruction following ratings, showing 60 percent correlation. Experiments show Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet, have a $\operatorname{PIF}$ metric that drops from 0.81 on average at turn 1 across the models, to 0.64 at turn 20. Across all turns, when each response is repeated 4 times ($\operatorname{PIF-4-4}$), GPT-4o and Gemini successfully follow all instructions only $11\%$ of the time. When all the instructions are also appended to the end of the model input context, the $\operatorname{PIF}$ metric improves by 22.3 points on average, showing that the challenge with the task lies not only in following the instructions, but also in retrieving the instructions spread out in the model context. We plan to open source the MMMT-IF dataset and metric computation code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑