超越任务性能：利用上下文学习评估并减少大型多模态模型的缺陷
Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
AI总结：
本文系统评估了10个开源大型多模态模型在幻觉、弃权、组合性、可解释性和指令遵循五个维度的缺陷，并提出多种免训练的上下文学习变体以有效缓解这些问题。
摘要：
继大型语言模型（LLMs）取得成功之后，大型多模态模型（LMMs），如Flamingo模型及其后续竞争者，已开始作为迈向通用智能体的自然步骤而出现。然而，与近期LMMs的交互揭示了当前评估基准难以捕捉的重大局限。事实上，仅凭任务性能（例如VQA准确率）并不能提供足够的线索来理解它们的真实能力、局限性，以及这些模型在多大程度上与人类期望保持一致。为了深化对这些缺陷的理解，我们偏离了当前的评估范式，并（1）在5个不同维度上评估了10个近期开源LMMs，参数规模从3B到80B不等；这些维度包括幻觉、弃权（不执行）、组合性、可解释性和指令遵循。我们在这些维度上的评估揭示了LMMs中的重大缺陷。虽然当前对齐这些模型的主流解决方案基于训练，例如指令微调或RLHF，但我们更倾向于（2）探索免训练的上下文学习（ICL）作为解决方案，并研究它如何影响这些局限性。基于我们的ICL研究，（3）我们进一步推进ICL，并提出了新的多模态ICL变体，例如：Multitask-ICL、Chain-of-Hindsight-ICL和Self-Correcting-ICL。我们的发现如下：（1）尽管LMMs取得了成功，但它们仍存在仅靠规模扩展无法解决的缺陷。（2）ICL对LMMs缺陷的影响是微妙的；尽管ICL在改善可解释性和答案弃权（不执行）方面有效，但它仅略微改善了指令遵循，并未提高组合能力，实际上甚至放大了幻觉。（3）所提出的ICL变体作为事后方法，在高效解决其中一些缺陷方面具有前景。代码可在以下地址获取：https://github.com/mshukor/EvALign-ICL。
英文摘要：
Following the success of Large Language Models (LLMs), Large Multimodal Models (LMMs), such as the Flamingo model and its subsequent competitors, have started to emerge as natural steps towards generalist agents. However, interacting with recent LMMs reveals major limitations that are hardly captured by the current evaluation benchmarks. Indeed, task performances (e.g., VQA accuracy) alone do not provide enough clues to understand their real capabilities, limitations, and to which extent such models are aligned to human expectations. To refine our understanding of those flaws, we deviate from the current evaluation paradigm, and (1) evaluate 10 recent open-source LMMs from 3B up to 80B parameter scale, on 5 different axes; hallucinations, abstention, compositionality, explainability and instruction following. Our evaluation on these axes reveals major flaws in LMMs. While the current go-to solution to align these models is based on training, such as instruction tuning or RLHF, we rather (2) explore the training-free in-context learning (ICL) as a solution, and study how it affects these limitations. Based on our ICL study, (3) we push ICL further and propose new multimodal ICL variants such as; Multitask-ICL, Chain-of-Hindsight-ICL, and Self-Correcting-ICL. Our findings are as follows. (1) Despite their success, LMMs have flaws that remain unsolved with scaling alone. (2) The effect of ICL on LMMs flaws is nuanced; despite its effectiveness for improved explainability, answer abstention, ICL only slightly improves instruction following, does not improve compositional abilities, and actually even amplifies hallucinations. (3) The proposed ICL variants are promising as post-hoc approaches to efficiently tackle some of those flaws. The code is available here: https://github.com/mshukor/EvALign-ICL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
