面向活动辅助的多模态大语言模型的用户在环评估
User-in-the-loop Evaluation of Multimodal LLMs for Activity Assistance
AI总结：
本研究针对多模态大语言模型的日常活动辅助能力，通过离线基准和首创的18人烹饪在线用户研究评估，发现Socratic方法优于VCLMs，且离线指标无法反映在线性能，长视觉历史锚定仍是难点。
摘要：
本研究探究了以大语言模型（LLM）为支撑的现代多模态推理模型，为多步骤日常活动提供视觉驱动辅助的能力。这类辅助系统必须具备三项能力：1）编码来自辅助设备传感器（如摄像头）的相关视觉历史信息；2）预测完成活动所需的未来动作；3）基于在环用户进行重新规划。为评估前两项能力，即视觉历史 grounding（锚定）与短期及长期动作预测，我们在基于视频的动作预判任务上，使用离线数据集对两类主流多模态LLM方法——Socratic Models（苏格拉底模型）和Vision Conditioned Language Models（VCLMs，视觉条件语言模型）开展了基准测试。但这类离线基准无法实现与用户的闭环交互，而闭环交互对于评估重新规划能力、衡量辅助场景下活动完成的成功率至关重要。为此，我们开展了一项首创性用户研究：18名参与者佩戴名为Aria的第一人称观测设备，在多模态LLM的辅助下完成3项不同的多步骤烹饪活动。研究发现，Socratic方法在离线和在线设置下的表现均优于VCLMs。我们进一步指出，对活动辅助中常见的长视觉历史进行锚定，对当前模型仍是挑战，对VCLMs而言尤其如此，同时证明离线指标无法反映在线性能。
英文摘要：
Our research investigates the capability of modern multimodal reasoning models, powered by Large Language Models (LLMs), to facilitate vision-powered assistants for multi-step daily activities. Such assistants must be able to 1) encode relevant visual history from the assistant's sensors, e.g., camera, 2) forecast future actions for accomplishing the activity, and 3) replan based on the user in the loop. To evaluate the first two capabilities, grounding visual history and forecasting in short and long horizons, we conduct benchmarking of two prominent classes of multimodal LLM approaches -- Socratic Models and Vision Conditioned Language Models (VCLMs) on video-based action anticipation tasks using offline datasets. These offline benchmarks, however, do not allow us to close the loop with the user, which is essential to evaluate the replanning capabilities and measure successful activity completion in assistive scenarios. To that end, we conduct a first-of-its-kind user study, with 18 participants performing 3 different multi-step cooking activities while wearing an egocentric observation device called Aria and following assistance from multimodal LLMs. We find that the Socratic approach outperforms VCLMs in both offline and online settings. We further highlight how grounding long visual history, common in activity assistance, remains challenging in current models, especially for VCLMs, and demonstrate that offline metrics do not indicate online performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
