Plan-Then-Execute：将LLM智能体作为日常助手使用时用户信任与团队绩效的实证研究
Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
AI总结：
本研究实证探讨了在六项日常任务中采用Plan-then-execute方式的LLM智能体，分析了用户在规划与执行阶段的参与度对信任及团队绩效的影响，为日常助手设计与人机协作提供了启示。
摘要：
自ChatGPT爆火以来，大型语言模型（LLMs）持续影响着我们的日常生活。配备了为特定目的设计的外部工具（例如用于航班预订或闹钟），LLM智能体在协助人类完成日常工作方面展现出越来越强的能力。尽管LLM智能体作为日常助手展现出了极具前景的蓝图，但对于它们如何基于规划和顺序决策能力提供日常协助，人们的理解仍然有限。我们从近期的研究中汲取了灵感，这些研究强调了“LLM-modulo”设置结合人在回路（humans-in-the-loop）在规划任务中的价值。我们进行了一项实证研究（N = 248），将LLM智能体作为日常助手应用于六项通常伴随不同风险水平的常见任务中（例如机票预订和信用卡支付）。为了确保用户对LLM智能体的能动性和控制力，我们采用了Plan-then-execute（先规划后执行）方式的LLM智能体，其中智能体在模拟环境中进行逐步规划和逐步执行。我们分析了在每个阶段用户的参与度如何影响他们的信任和协作团队绩效。我们的研究结果表明，LLM智能体可能是一把双刃剑——（1）当有高质量的规划且执行过程中有必要的用户参与时，它们能很好地发挥作用；（2）对于看似合理的规划，用户很容易对LLM智能体产生不信任。我们综合了将LLM智能体作为日常助手使用的关键见解，以校准用户信任并实现更好的整体任务结果。我们的工作对未来日常助手的设计以及人类与LLM智能体的人机协作具有重要的启示意义。
英文摘要：
Since the explosion in popularity of ChatGPT, large language models (LLMs) have continued to impact our everyday lives. Equipped with external tools that are designed for a specific purpose (e.g., for flight booking or an alarm clock), LLM agents exercise an increasing capability to assist humans in their daily work. Although LLM agents have shown a promising blueprint as daily assistants, there is a limited understanding of how they can provide daily assistance based on planning and sequential decision making capabilities. We draw inspiration from recent work that has highlighted the value of 'LLM-modulo' setups in conjunction with humans-in-the-loop for planning tasks. We conducted an empirical study (N = 248) of LLM agents as daily assistants in six commonly occurring tasks with different levels of risk typically associated with them (e.g., flight ticket booking and credit card payments). To ensure user agency and control over the LLM agent, we adopted LLM agents in a plan-then-execute manner, wherein the agents conducted step-wise planning and step-by-step execution in a simulation environment. We analyzed how user involvement at each stage affects their trust and collaborative team performance. Our findings demonstrate that LLM agents can be a double-edged sword -- (1) they can work well when a high-quality plan and necessary user involvement in execution are available, and (2) users can easily mistrust the LLM agents with plans that seem plausible. We synthesized key insights for using LLM agents as daily assistants to calibrate user trust and achieve better overall task outcomes. Our work has important implications for the future design of daily assistants and human-AI collaboration with LLM agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
