将查询分解为工具调用以进行长视频关键帧检索
Decomposing Queries into Tool Calls for Long-Video Keyframe Retrieval
AI总结：
提出ToolMerge方法，通过LLM规划器将查询分解为工具调用并使用布尔运算符合并排名，实现长视频关键帧检索，在字幕检索任务上优于其他方法5%。
摘要：
关键帧选择是为长视频问答（QA）提供可验证视觉证据的直接方式。查询所需的内容各不相同，找到正确的帧取决于知道要查找什么。现有的关键帧选择器要么根据单个查询对每一帧进行评分，要么将查询分解为由单个视觉工具评估的固定模式。我们提出ToolMerge，一种基于分解和合并的关键帧检索方法：基于大语言模型（LLM）的规划器将查询分解为工具调用，并指定如何使用布尔运算符合并每个工具的排名。为了直接评估检索，我们构建了Molmo-2 Moments（M2M）基准，其中每个问题通过构造锚定到特定的时间间隔。在QA、问题检索和字幕检索中，ToolMerge与先前的关键帧选择器具有竞争力，尤其是在字幕检索上，优于其他方法5%。代码和数据可在https://github.com/michalsr/ToolMerge找到。
英文摘要：
Keyframe selection is a direct way to provide verifiable visual evidence for long-video question answering (QA). Queries differ in what they require, and finding the right frames depends on knowing what to look for. Existing keyframe selectors either score every frame against a single query, or decompose the query into a fixed schema evaluated by a single visual tool. We propose ToolMerge, a keyframe retrieval method based on decomposition and merging: an Large Language Model (LLM) based planner decomposes the query into tool calls and specifies how their per-tool rankings are merged using boolean operators. To evaluate retrieval directly, we construct Molmo-2 Moments (M2M), a benchmark in which every question is anchored to a specific time interval by construction. Across QA, question retrieval, and caption retrieval, ToolMerge is competitive with prior keyframe selectors, most notably on caption retrieval, outperforming other methods by 5%. Code and data can be found at https://github.com/michalsr/ToolMerge .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
