GUIDE：用语言指令引导内部证据

中文摘要 AI 辅助

大型多模态模型会遵循关于生成内容的指令，但未必遵循关于依赖何种证据的指令。因此，即便指令另有要求，模型仍可能继续依赖与捷径相关的线索。我们提出GUIDE，这是一种通过语言指令控制内部证据使用的框架。GUIDE将分组参数高效适配与指令条件门控相结合，以在推理和生成过程中调节多模态证据通路。我们还提出了一种通路级评估框架，该框架通过依赖敏感性、受控扰动分析、通路调节和自回归解码动态来表征指令条件下的证据调节。在多模态推理、分类和生成任务中，GUIDE实现了结构化且与指令对齐的证据依赖重新分配，同时基本保留了任务行为。在GQA、TextVQA、MM-IMDb、CREMA-D、RAVDESS和Flickr30K上的实验表明，GUIDE提升了目标证据扰动下的鲁棒性，并能在多种多模态设置中实现可控调节。这表明多模态指令跟随可从输出控制扩展到调节不同证据源对模型预测的贡献。

英文摘要

Large multimodal models follow instructions about what to generate, but not necessarily about what evidence to rely on. Hence, models may continue to depend on shortcut-associated cues even when instructions suggest otherwise. We introduce GUIDE, a framework for controlling internal evidence usage through language instructions. GUIDE combines grouped parameter-efficient adaptation with instruction-conditioned gating to modulate multimodal evidence pathways during reasoning and generation. We further introduce a pathway-level evaluation framework that characterizes instruction-conditioned evidence modulation through reliance sensitivity, controlled perturbation analysis, pathway modulation, and autoregressive decoding dynamics. Across multimodal reasoning, classification, and generation, GUIDE induces structured and instruction-aligned redistribution of evidence reliance while largely preserving task behavior. Experiments on GQA, TextVQA, MM-IMDb, CREMA-D, RAVDESS, and Flickr30K show that GUIDE improves robustness under targeted evidence perturbations and enables controllable modulation across diverse multimodal settings. This suggests that multimodal instruction following can extend beyond output control toward regulating how different evidence sources contribute to model predictions.

发表机构

• The University of Melbourne（墨尔本大学）
• Korea Institute of Science and Technology（韩国科学技术研究院）
• Korea University（高丽大学）
• University of Science and Technology（科学技术大学）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

发表信息

EMNLP 2026