COLT: 通过持续工具使用增强视频大语言模型
COLT: Enhancing Video Large Language Models with Continual Tool Usage
AI总结：
COLT通过持续工具使用增强视频大语言模型，解决工具数据持续变化的问题，提升视频理解性能。
摘要：
大型语言模型（LLMs）的成功显著推动了视频理解的研究。为了利用训练良好的专家模型（即工具）的好处，视频LLMs优先探索工具使用能力。现有方法要么提示闭源LLMs，要么采用指令微调范式进行工具使用微调。然而，这些方法假设已有的固定工具库，难以泛化到现实环境中，其中工具数据持续变化并不断流式传输。为此，我们提出通过持续工具使用（称为COLT）增强开源视频LLMs，从而在连续的工具流中自动获取工具使用能力，而不会遭受'灾难性遗忘'。具体而言，我们的COLT结合了可学习的工具代码本作为工具特定的记忆系统。然后根据用户指令与代码本中工具特征之间的相似性动态选择相关工具。为了释放视频LLMs的工具使用潜力，我们收集了一个以视频为中心的工具使用指令微调数据集VideoToolBench。在先前的视频LLM基准和专门针对工具使用的VideoToolBench数据集上的广泛实验表明，我们提出的COLT达到了最先进的性能。
英文摘要：
The success of Large Language Models (LLMs) has significantly propelled the research of video understanding. To harvest the benefits of well-trained expert models (i.e., tools), video LLMs prioritize the exploration of tool usage capabilities. Existing methods either prompt closed-source LLMs or employ the instruction tuning paradigm for tool-use fine-tuning. These methods, however, assume an established repository of fixed tools and struggle to generalize to real-world environments where tool data is perpetually evolving and streaming in. To this end, we propose to enhance open-source video LLMs with COntinuaL Tool usage (termed COLT), which automatically acquires tool-use ability in a successive tool stream without suffering 'catastrophic forgetting' of the past learned tools. Specifically, our COLT incorporates a learnable tool codebook as a tool-specific memory system. Then relevant tools are dynamically selected based on the similarity between user instruction and tool features within the codebook. To unleash the tool usage potential of video LLMs, we collect a video-centric tool-use instruction tuning dataset VideoToolBench. Extensive experiments on both previous video LLM benchmarks and the tool-use-specific VideoToolBench dataset demonstrate the state-of-the-art performance of our proposed COLT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
