X-MIC：面向第一人称动作泛化的跨模态实例条件化方法
X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization
AI总结：
本文提出 X-MIC 跨模态适配框架，通过解耦时间建模与冻结视觉编码器，将文本嵌入对齐到第一人称视频，在三个数据集上显著提升了跨数据集动作泛化能力。
摘要：
近年来，由于视觉-语言模型（VLMs）在零样本识别中的成功，人们对其在图像和第三人称视频分类中的适配越来越感兴趣。然而，将这些模型适配到第一人称视频的研究在很大程度上尚未被探索。为了填补这一空白，我们提出了一个简单而有效的跨模态适配框架，称之为 X-MIC。通过使用视频适配器，我们的流程学习直接在共享嵌入空间中将冻结的文本嵌入对齐到每个第一人称视频。我们的新型适配器架构通过解耦可学习的时间建模与冻结的视觉编码器，保留并提升了预训练 VLMs 的泛化能力。这增强了对每个第一人称视频的文本嵌入对齐，从而显著改善了跨数据集泛化性能。我们在 Epic-Kitchens、Ego4D 和 EGTEA 数据集上对细粒度跨数据集动作泛化进行了评估，证明了我们方法的有效性。代码可在 https://github.com/annusha/xmic 获取。
英文摘要：
Lately, there has been growing interest in adapting vision-language models (VLMs) to image and third-person video classification due to their success in zero-shot recognition. However, the adaptation of these models to egocentric videos has been largely unexplored. To address this gap, we propose a simple yet effective cross-modal adaptation framework, which we call X-MIC. Using a video adapter, our pipeline learns to align frozen text embeddings to each egocentric video directly in the shared embedding space. Our novel adapter architecture retains and improves generalization of the pre-trained VLMs by disentangling learnable temporal modeling and frozen visual encoder. This results in an enhanced alignment of text embeddings to each egocentric video, leading to a significant improvement in cross-dataset generalization. We evaluate our approach on the Epic-Kitchens, Ego4D, and EGTEA datasets for fine-grained cross-dataset action generalization, demonstrating the effectiveness of our method. Code is available at https://github.com/annusha/xmic
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
