arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.05923cs.CLcs.HC
收藏

从人机交互中学习下一步动作预测器

Learning Next Action Predictors from Human-Computer Interaction

Stanford University(斯坦福大学);Hasso Plattner Institute(哈索普拉特纳研究所);New York University(纽约大学)

Omar Shaikh, Valentin Teutschbein, Kanishk Gandhi, Yikun Chi, Nick Haber, Thomas Robinson, Nilam Ram, Byron Reeves, Sherry Yang, Michael S. Bernstein, Diyi Yang

AI总结:

本文提出LongNAP模型,通过结合参数化和上下文学习,从用户行为的完整上下文中预测下一步动作,显著优于传统方法。

摘要:

真正具有前瞻性的AI系统必须预测我们下一步会做什么。这种预见性要求比我们输入提示中的稀疏信号要丰富得多的信息——它要求对所见所做整个上下文进行推理。我们将此正式定义为下一步动作预测(NAP):给定用户与计算机的多模态交互序列(截图、点击、传感器数据),预测用户下一步的动作。该任务的进步需要新的数据和建模方法。为了扩大数据集,我们使用视觉语言模型对纵向的自然计算机使用情况进行标注。我们发布了一个开源管道,用于在私有基础设施上执行这种标注,并对20名用户一个月的连续手机使用进行了标注,共计1800小时的屏幕时间。然后我们引入了LongNAP,一种用户模型,它结合了参数化和上下文学习来推理长交互历史。LongNAP通过策略梯度方法训练,以生成用户特定的推理轨迹,给定某些上下文;从过去的轨迹库中检索相关轨迹;然后将检索到的轨迹用于上下文以预测未来动作。使用LLM-as-judge评估度量(0-1相似度到地面真实值),LongNAP在测试数据上显著优于监督微调和提示基线(分别高出79%和39%)。此外,LongNAP在训练跨个体时能够泛化到测试用户。用户可能在任何时刻采取的下一步动作空间是无限的,涵盖数千种可能的结果。尽管如此,LongNAP预测轨迹中17.1%与用户下一步动作一致(LLM-judge得分≥0.5)。当我们将预测过滤到高度自信的预测时,这一比例上升到26%。总之,我们认为从用户行为的完整上下文中学习以预测用户需求现在是一项可行的任务,具有实质性的机会。

英文摘要:

Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts -- it demands reasoning over the entire context of what we see and do. We formalize this as next action prediction (NAP): given a sequence of a user's multimodal interactions with a computer (screenshots, clicks, sensor data), predict that user's next action. Progress on this task requires both new data and modeling approaches. To scale data, we annotate longitudinal, naturalistic computer use with vision-language models. We release an open-source pipeline for performing this labeling on private infrastructure, and label over 360K actions across one month of continuous phone usage from 20 users, amounting to 1,800 hours of screen time. We then introduce LongNAP, a user model that combines parametric and in-context learning to reason over long interaction histories. LongNAP is trained via policy gradient methods to generate user-specific reasoning traces given some context; retrieve relevant traces from a library of past traces; and then apply retrieved traces in-context to predict future actions. Using an LLM-as-judge evaluation metric (0-1 similarity to ground truth), LongNAP significantly outperforms supervised finetuning and prompted baselines on held-out data (by 79% and 39% respectively). Additionally, LongNAP generalizes to held out users when trained across individuals. The space of next actions a user might take at any moment is unbounded, spanning thousands of possible outcomes. Despite this, 17.1% of LongNAP's predicted trajectories are well-aligned with what a user does next (LLM-judge score $\geq$ 0.5). This rises to 26% when we filter to highly confident predictions. In sum, we argue that learning from the full context of user behavior to anticipate user needs is now a viable task with substantial opportunity.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑