TAMA: 面向程序性活动理解的工具增强多模态智能体
TAMA: Tool-Augmented Multimodal Agent for Procedural Activity Understanding
AI总结：
针对程序性活动理解系统开发不足的问题，提出无需训练的工具增强多模态智能体框架TAMA，通过多媒体返回工具实现交错推理，在ProMQA-Assembly数据集上提升了视觉语言模型性能，验证了两大核心特征的有效性。
摘要：
程序性活动助手有望在多种场景中为人类提供支持，涵盖烹饪、组装板式家具等日常生活场景，以及制造、生物实验等专业场景。尽管该方向具备潜在应用价值，但针对这类助手的系统开发研究仍显不足。本文提出一种名为TAMA（工具增强多模态智能体）的新型框架，用于程序性活动理解。TAMA无需训练，通过利用返回多媒体内容的工具，实现交错式多模态推理。我们在多模态程序性问答数据集ProMQA-Assembly上的实验结果表明，该方法能够提升视觉语言模型的性能，尤其是GPT-5和MiMo-VL的表现。此外，消融实验为框架的两大核心特征——返回多媒体内容的工具与智能体式灵活工具选择——的有效性提供了实证支持。我们认为，所提出的框架与实验结果不仅能推动程序性活动助手的发展，还能促进视频及多模态任务中“用图像思考”范式的发展。
英文摘要：
Procedural activity assistants potentially support humans in a variety of settings, from our daily lives, e.g., cooking or assembling flat-pack furniture, to professional situations, e.g., manufacturing or biological experiments. Despite its potential use cases, the system development tailored for such an assistant is still underexplored. In this paper, we propose a novel framework, called TAMA, a Tool-Augmented Multimodal Agent, for procedural activity understanding. TAMA enables interleaved multimodal reasoning by making use of multimedia-returning tools in a training-free setting. Our experimental result on the multimodal procedural QA dataset, ProMQA-Assembly, shows that our approach can improve the performance of vision-language models, especially GPT-5 and MiMo-VL. Furthermore, our ablation studies provide empirical support for the effectiveness of two features that characterize our framework, multimedia-returning tools and agentic flexible tool selection. We believe our proposed framework and experimental results facilitate the thinking with images paradigm for video and multimodal tasks, let alone the development of procedural activity assistants.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
