arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.15867cs.AI
收藏

基于伪标签的上下文集成学习提升视频-语言模型对低层级工作流理解

In-Context Ensemble Learning from Pseudo Labels Improves Video-Language Models for Low-Level Workflow Understanding

Agile Loop(敏捷循环公司)

Moucheng Xu, Evangelos Chatzaroulas, Luc McCutcheon, Abdul Ahad, Hamzah Azeem, Janusz Marecki, Ammar Anwar

AI总结:

本文针对视频-语言模型零样本生成标准操作程序(SOP)的困难,提出基于伪标签的上下文集成学习方法,通过聚合多条SOP路径并利用隐式一致性正则化突破上下文窗口限制,持续提升SOP生成的时间准确性。

摘要:

标准操作程序(SOP)定义了商业软件工作流的低层级、逐步书面指南。SOP生成是实现端到端软件工作流自动化的关键步骤。手动创建SOP可能非常耗时。大型视频-语言模型的最新进展为通过分析人类演示录像来自动化SOP生成提供了潜力。然而,当前的大型视频-语言模型在零样本SOP生成方面面临挑战。在这项工作中,我们首先探索了利用视频-语言模型进行SOP生成的上下文学习。然后,我们提出了一种以探索为重点的策略,称为上下文集成学习(In-Context Ensemble Learning),用于聚合SOP多条可能路径的伪标签。所提出的上下文集成学习还通过隐式一致性正则化使模型能够学习超出其上下文窗口限制的内容。我们报告称,上下文学习有助于视频-语言模型生成时间上更准确的SOP,并且所提出的上下文集成学习能够持续增强视频-语言模型在SOP生成方面的能力。

英文摘要:

A Standard Operating Procedure (SOP) defines a low-level, step-by-step written guide for a business software workflow. SOP generation is a crucial step towards automating end-to-end software workflows. Manually creating SOPs can be time-consuming. Recent advancements in large video-language models offer the potential for automating SOP generation by analyzing recordings of human demonstrations. However, current large video-language models face challenges with zero-shot SOP generation. In this work, we first explore in-context learning with video-language models for SOP generation. We then propose an exploration-focused strategy called In-Context Ensemble Learning, to aggregate pseudo labels of multiple possible paths of SOPs. The proposed in-context ensemble learning as well enables the models to learn beyond its context window limit with an implicit consistency regularisation. We report that in-context learning helps video-language models to generate more temporally accurate SOP, and the proposed in-context ensemble learning can consistently enhance the capabilities of the video-language models in SOP generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑