DYAD：共处人类协助的多模态数据集
DYAD: A Multimodal Dataset of Co-Located Human Assistance
- Georgia Institute of Technology（佐治亚理工学院）
- Northeastern University（东北大学）
- Microsoft Research（微软研究院）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
DYAD是一个共处人类协助的多模态数据集，通过链接请求、干预和结果，支持因果理解、模式预测和响应生成等任务。
AI中文摘要：
一个在人身旁工作的具身助手必须跟踪任务状态、识别求助行为、选择干预方式并产生适当的响应。现有的程序性数据集丰富地描述了单个执行过程，而交互式数据集则捕获远程口头指令或无差别的协同工作。它们未能将共处助手的口头和物理干预与执行者的请求、任务状态、协助触发因素及结果联合起来。我们引入了DYAD（二元协助数据集），这是一个在齿轮箱装配过程中人类-人类协助的同步多模态记录。在20个会话中，一名受过训练的助手遵循指导优先策略，协助佩戴HoloLens 2的用户。DYAD将528个任务步骤区间和611个执行者请求与851条涵盖口头和物理帮助的有效协助记录相关联。DYAD的标注覆盖了整个协助过程；三个参考任务评估了选定的组件而非端到端系统：因果步骤理解、发作前模式预测和指导者响应生成。在829个符合条件的模式事件上，最强的四种子RGB平均宏F1为0.548±0.007；因果元数据达到0.624，特权触发器映射达到0.915，揭示了从发作前RGB中无法恢复的信息。DYAD的贡献不在于规模，而在于一种链接的交互结构，该结构在自我中心和工作区感知下跨越了求助、干预选择、执行和结果。
英文摘要：
An embodied assistant working beside a person must track task state, recognize help seeking, choose how to intervene, and produce an appropriate response. Existing procedural datasets richly describe individual execution, while interactive datasets capture remote verbal instruction or undifferentiated co-working. They do not jointly link a co-located helper's verbal and physical interventions to performer requests, task state, assistance triggers, and outcomes. We introduce DYAD (DYadic Assistance Dataset), a synchronized multimodal record of human-human assistance during gearbox assembly. Across 20 sessions, one trained helper follows a guidance-first policy while assisting HoloLens 2 wearers. DYAD links 528 task-step intervals and 611 performer requests with 851 valid assistance records spanning verbal and physical help. DYAD's annotations span the assistance process; three reference tasks evaluate selected components rather than an end-to-end system: causal step understanding, pre-onset mode anticipation, and instructor response generation. On 829 eligible mode events, the strongest four-seed RGB mean is 0.548 +/- 0.007 macro-F1; causal metadata reaches 0.624 and a privileged trigger mapping 0.915, revealing information not recovered from pre-onset RGB. DYAD's contribution is not scale, but a linked interaction structure spanning help seeking, intervention choice, execution, and outcome under egocentric and workspace sensing.
