arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.16133cs.CVcs.AIcs.CLcs.LG
收藏

揭示并解决统一视觉-语言模型中的跨任务不一致性

Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Models

University of North Carolina, Chapel Hill(北卡罗来纳大学教堂山分校);University of California, Los Angeles(加利福尼亚大学洛杉矶分校);Allen Institute for AI(艾伦人工智能研究所)

Adyasha Maharana, Amita Kamath, Christopher Clark, Mohit Bansal, Aniruddha Kembhavi

AI总结:

本文提出基准数据集CocoCon和基于秩相关的辅助训练目标,用于衡量并改善统一视觉-语言模型在异质任务间的预测一致性,在不损失下游任务准确率的前提下显著减少跨任务不一致行为。

摘要:

随着通用视觉模型在广泛任务上变得越来越有效,它们在其支持的任务之间保持一致变得至关重要。不一致的AI模型被人类用户认为是脆弱且不可信的,并且更难以整合到依赖其输出的大型系统中。在可能包含不同模态输出的高度异质任务之间衡量一致性具有挑战性,因为很难确定预测是否彼此一致。作为解决方案,我们引入了一个基准数据集CocoCon,在该数据集中,我们通过对多个任务的测试实例进行微小但在语义上有意义的修改来创建对比集,以改变金标准标签,并概述了通过跨任务对原始实例和扰动实例进行排序来衡量模型是否一致的指标。我们发现,最先进的视觉-语言模型在跨任务上存在令人惊讶的高度不一致行为,尤其是对于更异质的任务。为了缓解这一问题,我们提出了一种基于秩相关的辅助训练目标,该目标在大规模自动创建的跨任务对比集上计算,提高了大型统一模型的多任务一致性,同时保持其在下游任务上的原始准确率。

英文摘要:

As general purpose vision models get increasingly effective at a wide set of tasks, it is imperative that they be consistent across the tasks they support. Inconsistent AI models are considered brittle and untrustworthy by human users and are more challenging to incorporate into larger systems that take dependencies on their outputs. Measuring consistency between very heterogeneous tasks that might include outputs in different modalities is challenging since it is difficult to determine if the predictions are consistent with one another. As a solution, we introduce a benchmark dataset, CocoCon, where we create contrast sets by modifying test instances for multiple tasks in small but semantically meaningful ways to change the gold label and outline metrics for measuring if a model is consistent by ranking the original and perturbed instances across tasks. We find that state-of-the-art vision-language models suffer from a surprisingly high degree of inconsistent behavior across tasks, especially for more heterogeneous tasks. To alleviate this issue, we propose a rank correlation-based auxiliary training objective, computed over large automatically created cross-task contrast sets, that improves the multi-task consistency of large unified models while retaining their original accuracy on downstream tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑