FiLoRA: 专注与忽略LoRA用于可控的特征依赖
FiLoRA: Focus-and-Ignore LoRA for Controllable Feature Reliance
AI总结：
FiLoRA通过指令条件门控实现对多模态模型内部特征依赖的可控调节，提升模型在虚假特征干预下的鲁棒性。
摘要：
多模态基础模型整合跨模态的异质信号，但仍然不清楚其预测如何依赖于特定的内部特征组，以及这种依赖性是否可以被有意控制。现有对捷径和虚假行为的研究大多依赖于事后分析或特征移除，提供有限的见解，表明在不改变任务语义的情况下，依赖性是否可以被调节。我们介绍了FiLoRA（专注与忽略LoRA），一种指令条件的、参数高效的适应框架，使可以在保持预测目标不变的情况下显式控制内部特征依赖性。FiLoRA将适应分解为特征组对齐的LoRA模块，并应用指令条件的门控，使自然语言指令可以作为计算层面的控制信号，而不是任务重新定义。在文本-图像和音频-视觉基准测试中，我们显示，指令条件的门控在内部计算中引起一致且因果的转变，有选择地放大或抑制核心和虚假特征组，而不会修改标签空间或训练目标。进一步分析表明，FiLoRA在虚假特征干预下显示出改进的鲁棒性，揭示了一种调节依赖性的原理机制，超越了相关驱动学习。
英文摘要：
Multimodal foundation models integrate heterogeneous signals across modalities, yet it remains unclear whether their predictions can be controlled by explicitly modulating reliance on different internal feature pathways. Existing approaches to shortcut and spurious behavior primarily rely on post hoc analysis or data-level interventions, offering limited ability to directly intervene on how models use information. We introduce FiLoRA (Focus-and-Ignore LoRA), an instruction-conditioned, parameter-efficient adaptation framework that enables controllable modulation of feature reliance while keeping the task and predictive objective fixed. FiLoRA decomposes adaptation into feature-aligned low-rank modules and applies instruction-conditioned gating, allowing natural language instructions to act as computation-level control signals over internal representations. We evaluate FiLoRA across both controlled classification settings and generative multimodal tasks, and under a range of instruction types, including natural and compositional instructions. Results show that FiLoRA induces consistent and interpretable shifts in feature reliance, selectively amplifying or suppressing different feature groups in accordance with the instruction, without altering task semantics. Our findings suggest that instruction-conditioned parameter adaptation can serve as a practical mechanism for intervening on internal model behavior, providing a new perspective on controllability and analysis of multimodal systems beyond output-level prompting or post hoc interpretation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
