arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.02458cs.AI
收藏

超越一次性:用于现场实验学习的AI智能体

Beyond One-shot: AI Agents for Learning in Field Experiments

Johns Hopkins School of Medicine(约翰斯·霍普金斯大学医学院);Carey Business School, Johns Hopkins University(约翰斯·霍普金斯大学凯瑞商学院)

Junjie Luo, Ritu Agarwal, Gordon Gao

AI总结:

研究通过工具增强的智能体AI自动从实验数据中学习并生成新干预措施,在医疗处方消息现场实验中证明其优于人类+聊天机器人方法。

摘要:

组织通常进行A/B测试实验,但一次实验产生的数据未被充分利用以指导后续干预设计。从先前实验数据中提取可操作知识以指导新干预存在重大障碍。我们研究工具增强的智能体AI能否自动从实验数据中学习,以在后续实验中生成新干预。通过医疗处方消息传递(693,139次患者就诊)的两阶段现场实验,我们比较了人类+聊天机器人方法(第一阶段:行为专家与对话式AI共同设计13种消息变体,444,691次患者就诊)与工具增强的智能体AI方法(第二阶段:AI自主从第一阶段数据中提取原则以生成17种新变体,248,448次患者就诊)。配备分析工具、结构化数据-信息-知识-智慧(DIKW)推理智能体和透明证据链的智能体AI方法产生了更优的干预:AI生成的最佳消息实现了69.8%的点击率(比基线高6.5个百分点)。关键的是,我们的结果表明价值来自特定领域的实验数据,而非通用推理能力:没有实验数据的前沿大语言模型无法预测哪些干预会成功。现场实验还揭示,用于干预设计的通用行为理论并不能统一适用于特定医疗环境,这激发了在实验规模上进行理论审计的智能体AI方法。我们的研究表明,工具增强的AI可以从实验数据中学习并生成改进的领域相关干预,将行为实验从一次性评估转变为可扩展的累积设计学习系统。

英文摘要:

Organizations routinely run experiments for A/B testing, yet the data generated from one experiment is underutilized to inform subsequent intervention design. Significant barriers exist to extracting actionable knowledge from prior experimental data to inform new interventions. We study whether tool-augmented agentic AI can automatically learn from experimental data to generate new interventions in subsequent experiments. Through two-stage field experiments in healthcare prescription messaging (693,139 patient visits), we compare a Human + Chatbot method (Stage 1: behavioral experts with conversational AI co-designing 13 message variants, 444,691 patient visits) against a Tool-Augmented Agentic AI method (Stage 2: AI autonomously extracting principles from Stage 1 data to generate 17 new variants, 248,448 patient visits). The Agentic AI method, equipped with analytical tools, structured Data-Information-Knowledge-Wisdom (DIKW) reasoning agents, and transparent evidence chains, produces superior interventions: the best AI-generated message achieved a 69.8% CTR (+6.5 percentage points over baseline). Critically, our results suggest that the value comes from domain-specific experimental data, not from general reasoning ability: frontier LLMs operating without experimental data failed to predict which interventions would succeed. The field experiments also revealed that general-purpose behavioral theories used for intervention design do not extend uniformly to specific healthcare contexts, motivating an agentic AI approach to theory audits at field-experiment scale. Our research shows that tool-augmented AI can learn from experimental data and generate improved domain-relevant interventions, transforming behavioral experimentation from one-shot evaluation into a scalable system for cumulative design learning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑