arXiv 2407.086722024-07-11cs.CV

NODE-Adapter：用于提升视觉-语言推理性能的神经常微分方程方法

NODE-Adapter: Neural Ordinary Differential Equations for Better Vision-Language Reasoning

Harbin Institute of Technology（哈尔滨工业大学）；Southern University of Science and Technology（南方科技大学）；University of Cambridge（剑桥大学）

Yi Zhang, Chun-Wun Cheng, Ke Yu, Zhihai He, Carola-Bibiane Schönlieb, Angelica I. Aviles-Rivero

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对基于原型的视觉-语言推理现有方法的资源需求高、参数多、单模态微调等问题，提出NODE-Adapter方法，通过两阶段跨模态原型构建与神经ODE优化，在三类任务上性能超越现有最优水平。

摘要：

本文研究基于原型的视觉-语言推理问题。我们发现现有方法面临三大核心挑战：1）资源需求持续攀升且训练时间不断延长；2）可学习参数数量过多；3）仅基于单一模态进行微调。这些挑战会制约视觉-语言模型（Vision-Language Models, VLMs）适配下游任务的能力。受这一关键发现启发，我们提出一种名为NODE-Adapter的新方法，利用神经常微分方程优化视觉-语言推理效果。为充分利用视觉和文本两种模态、更高效准确地估计类别原型，我们将方法分为两个阶段：跨模态原型构建，以及基于神经常微分方程的跨模态原型优化。具体而言，我们利用VLM将人工设计的提示编码为文本特征，将少样本支持图像编码为视觉特征；随后分别对文本特征和视觉特征取平均，估计得到文本原型和视觉原型，再将二者自适应融合以构建跨模态原型。为缓解原型偏差，我们进一步将原型优化过程建模为神经ODE（Neural ODEs）的初值问题，以此估计连续梯度流。我们在少样本分类、域泛化、人机交互视觉推理三类任务上开展了大量实验，结果表明所提方法的性能显著优于现有最优方法。

英文摘要：

In this paper, we consider the problem of prototype-based vision-language reasoning problem. We observe that existing methods encounter three major challenges: 1) escalating resource demands and prolonging training times, 2) contending with excessive learnable parameters, and 3) fine-tuning based only on a single modality. These challenges will hinder their capability to adapt Vision-Language Models (VLMs) to downstream tasks. Motivated by this critical observation, we propose a novel method called NODE-Adapter, which utilizes Neural Ordinary Differential Equations for better vision-language reasoning. To fully leverage both visual and textual modalities and estimate class prototypes more effectively and accurately, we divide our method into two stages: cross-modal prototype construction and cross-modal prototype optimization using neural ordinary differential equations. Specifically, we exploit VLM to encode hand-crafted prompts into textual features and few-shot support images into visual features. Then, we estimate the textual prototype and visual prototype by averaging the textual features and visual features, respectively, and adaptively combine the textual prototype and visual prototype to construct the cross-modal prototype. To alleviate the prototype bias, we then model the prototype optimization process as an initial value problem with Neural ODEs to estimate the continuous gradient flow. Our extensive experimental results, which cover few-shot classification, domain generalization, and visual reasoning on human-object interaction, demonstrate that the proposed method significantly outperforms existing state-of-the-art approaches.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。