Spacewalk-18：多模态和长形式程序视频理解的基准测试，用于新领域
Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding in Novel Domains
AI总结：
Spacewalk-18是一个用于多模态和长形式程序视频理解的新领域基准测试，通过步骤识别和视频问答任务评估模型在新领域和长时序上下文中的泛化能力。
摘要：
从（程序性）视频学习已逐渐成为体现智能体从人类示范中获取技能的途径。为此，视频理解模型必须能够获得结构化的理解，例如将演示时间分割为动作和技能序列，并且能够将这些理解泛化到新的环境、任务和问题领域。为实现这一目标，我们引入了Spacewalk-18基准测试，包含两个任务：（1）步骤识别和（2）视频问答，基于国际空间站空间行走记录中时间分割和标记的任务数据集。同时，这两个任务量化了模型的能力：（1）泛化到新领域；（2）利用长时间时序上下文和多模态（例如视觉和语音）信息。我们广泛的实验分析突显了Spacewalk-18的挑战，但也提出了领域泛化和长形式理解的最佳实践。值得注意的是，我们发现通过总结技术可以实现显著的性能提升，而无需模型微调。Spacewalk-18基准测试已发布在https://brown-palm.github.io/Spacewalk-18/。
英文摘要：
Learning from (procedural) videos has increasingly served as a pathway for embodied agents to acquire skills from human demonstrations. To do this, video understanding models must be able to obtain structured understandings, such as the temporal segmentation of a demonstration into sequences of actions and skills, and to generalize the understandings to novel environments, tasks, and problem domains. In pursuit of this goal, we introduce Spacewalk-18, a benchmark containing two tasks: (1) step recognition and (2) video question answering, over a dataset of temporally segmented and labeled tasks in International Space Station spacewalk recordings. In tandem, the two tasks quantify a model's ability to: (1) generalize to novel domains; (2) utilize long temporal context and multimodal (e.g. visual and speech) information. Our extensive experimental analysis highlights the challenges of Spacewalk-18, but also suggests best practices for domain generalization and long-form understanding. Notably, we discover a promising adaptation via summarization technique that leads to significant performance improvement without model fine-tuning. The Spacewalk-18 benchmark is released at https://brown-palm.github.io/Spacewalk-18/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
