使用多模态大语言模型进行活动时序定位
Temporal Grounding of Activities using Multimodal Large Language Models
AI总结：
本文提出结合图像与文本LLM的两阶段时序活动定位方法，并通过指令微调提升小型多模态LLM，在Charades-STA上优于现有视频LLM。
摘要：
活动时序定位，即在更大的事件上下文中识别动作的特定时间间隔，是视频理解中的一项关键任务。多模态大语言模型（LLM）的最新进展为增强时序推理能力提供了新机遇。本文中，我们评估了在两阶段方法中结合基于图像和基于文本的大语言模型（LLM）进行时序活动定位的有效性。我们证明，该方法优于现有的基于视频的LLM。此外，我们探究了指令微调对较小多模态LLM的影响，结果表明，优化其处理动作查询的能力可产生更具表达力和信息量的输出，从而提升其识别活动特定时间间隔的性能。我们在Charades-STA数据集上的实验结果凸显了该方法在推进时序活动定位和视频理解领域方面的潜力。
英文摘要：
Temporal grounding of activities, the identification of specific time intervals of actions within a larger event context, is a critical task in video understanding. Recent advancements in multimodal large language models (LLMs) offer new opportunities for enhancing temporal reasoning capabilities. In this paper, we evaluate the effectiveness of combining image-based and text-based large language models (LLMs) in a two-stage approach for temporal activity localization. We demonstrate that our method outperforms existing video-based LLMs. Furthermore, we explore the impact of instruction-tuning on a smaller multimodal LLM, showing that refining its ability to process action queries leads to more expressive and informative outputs, thereby enhancing its performance in identifying specific time intervals of activities. Our experimental results on the Charades-STA dataset highlight the potential of this approach in advancing the field of temporal activity localization and video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
