发表机构
Columbia University; Emory University（哥伦比亚大学; 埃默里大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究人员提出OLIVE框架，通过融合EEG与XR游戏行为信号，适配基础模型生成实时辅助智能体，在目标切换时收敛更快，可提升用户目标检测能力且不依赖个人技能。
AI 中文摘要
我们提出了OLIVE框架，该框架用于适配基础模型，以在时间要求高、风险高且动态变化的任务中提供实时辅助。我们表明，被动式EEG（脑电图）与在线行为证据融合后，可显著扩展用户检测和参与目标的数量，超出其无辅助的动作带宽。OLIVE同时从显式行为信号（用户在XR第一人称射击游戏中击落的目标）和隐式生理信号（注视锁定的EEG）中学习，通过联合估计各源的可靠性，无需人工标签或离线训练，即可持续调整冻结的视觉语言模型对哪些项目与任务相关的推理，从而提供及时指导。通过三项用户研究，包括两次在XR中部署OLIVE驱动的辅助智能体的现场实验，我们表明OLIVE在帕累托最优上优于现有的测试时适配框架，在可比收敛速度下实现了最高的收敛率。结合隐式生理和显式行为信号，OLIVE智能体在用户检测和参与目标的能力方面产生了最大且最可靠的会话内提升，且在很大程度上独立于个人技能水平。当目标静默切换时，同时使用行为和生理信号的智能体重收敛速度比仅使用行为信号的智能体快得多（平均快1.27倍，p值为0.008），在任务变化的时刻恢复可靠的指导，而这正是可靠辅助最为关键的时刻。
英文摘要
We present OLIVE, a framework for adapting a foundation model to provide real-time assistance in temporally demanding, high-stakes, and dynamic tasks. We show that passive EEG, fused online with behavioral evidence, can meaningfully extend the number of targets users detect and engage beyond their unaided action bandwidth. OLIVE learns from both explicit behavioral signals (the targets the user shoots down in an XR first-person shooter game) and implicit physiological signals (fixation-locked EEG) to provide timely guidance, continuously adapting a frozen vision-language model's inference on which items are task-relevant by jointly estimating per-source reliability without manual labels or offline training. Through three user studies, including two live deployments of an assistive agent driven by OLIVE in XR, we show that OLIVE Pareto-dominates prior test-time adaptation frameworks, achieving the highest convergence rate at comparable convergence speed. Combining implicit physiological and explicit behavioral signals, the OLIVE agent produces the largest and most reliable within-session improvement to a user's ability to detect and engage targets, largely independent of the individual's skill. When the target switches silently, the agent that uses both behavioral and physiological signals reconverges significantly faster than the behavior-only agent (1.27 times faster on average, p = .008), restoring trustworthy guidance at the moment the task changes, precisely when reliable assistance matters most.
CommentsTo appear in ACM UIST 2026. 30 pages, 23 figures
