arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.08101cs.CV
收藏

说什么和何时说:Live Fitness Coaching作为情境交互的测试平台

What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction

TwentyBN GmbH(TwentyBN公司);Qualcomm AI Research(高通人工智能研究);Aignostics GmbH(Aignostics公司);UC San Diego(加州大学圣地亚哥分校)
Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Le… 展开作者

Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, Roland Memisevic

AI总结:

本文提出QEVD基准和数据集,研究人类与AI在健身指导中的情境交互,测试视觉语言模型在实时反馈中的能力,揭示现有模型的局限并提出异步流式基线方法。

摘要:

视觉语言模型近年来展现了显著的进步。然而,现有模型大多局限于回合制交互,其中每个回合必须由用户触发(即提示)。开放式的、异步的交互,其中AI模型可以根据实时展开的情况主动提供及时的回应或反馈,是一个开放性挑战。在本工作中,我们提出了QEVD基准和数据集,探索人类与AI在具有挑战性但受控的现实世界领域——健身指导中的交互——一项本质上需要监控实时用户活动并提供即时反馈的任务。该基准要求视觉语言模型识别复杂的人类动作,识别可能的错误,并在实时中提供适当的反馈。我们的实验揭示了现有最先进的视觉语言模型在这样的异步情境交互中的局限性。受此启发,我们提出了一种简单的端到端流式基线方法,能够异步响应人类动作并适时提供适当的反馈。

英文摘要:

Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching -- a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-language models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑