主特质分析：面向人机协作中“技能”的推导

中文摘要 AI 辅助

基于大语言模型（LLM）的智能体正通过人机协作越来越多地应用于职场。在这个新工作时代，理解哪些提示特质有助于任务成功十分重要；此外，我们需要揭示现代专业人员所需的关键技能，并为教育工作者提供如何在学生中培养这些技能的指导。现有人机协作指南要么基于自上而下的理论，要么基于人机交互的特定情境观察。然而，由于LLM的能力正在快速提升，理论可能无法解释新兴的交互模式，而经验性指南可能很快就会过时。本研究探索一种自动化、数据驱动的方法，以揭示与任务结果一致的有效人机交互模式，我们将其称为“特质”。我们提出了主特质分析（Principal Trait Analysis，PTA），一种受主成分分析启发的算法，用于从LLM对话模式中推导共同特质。该算法使用基于LLM的处理阶段分析人机协作会话轨迹语料库，推导数据集中的共同特质，并按每个特质对每个人机协作者的使用风格进行评分。该方法还允许在特质发现过程中注入领域专业知识，并选择在协作者间表现出最高方差的特质作为最具区分度的特质。我们在两个人机协作编码数据集上评估PTA：一个是教育场景（学生与AI辅导教师协作），另一个是专业场景（开发人员与AI编码智能体协作）。我们发现，PTA推导的特质在两种场景下对解释协作者行为均具有显著性，且能够帮助预测任务结果。不过，由于关于可推广性以及用户特质随时间如何变化的结果尚无定论，这些特质是否可被视为技能仍有待观察。

英文摘要

Large Language Model-powered agents are increasingly used in the workplace via human-artificial intelligence (AI) collaboration. In this new era of work, it is important to understand the kinds of prompting traits that contribute to task success. Moreover, we need to uncover key skills required for modern professionals and inform educators on how to foster these skills among students. Existing guidelines for human-AI collaboration are built from either top-down theory or context-specific observations of human-AI interactions. However, since LLM capabilities are rapidly improving, theory may not be able to explain emerging interaction patterns, and empirical guidelines may become obsolete quickly. In this work, we explore an automated, data-driven approach to uncover patterns, which we term traits, of effective human-AI interaction that are aligned with task outcomes. We propose Principal Trait Analysis, a Principal Component Analysis-inspired algorithm for deriving common traits from patterns in LLM conversations. Our algorithm uses LLM-based processing stages to analyze corpora of human-AI collaborative session traces, deriving common traits across the dataset and scoring each human collaborator's usage style by each trait. The approach also allows domain expertise to be injected during trait discovery and selects the most distinguishing traits to be those that exhibit the highest variance across collaborators. We evaluate PTA on two human-AI collaborative coding datasets, an educational setting (students working with an AI tutor) and a professional setting (developers working with an AI coding agent). We find that PTA-derived traits are significant in explaining collaborator behavior across both settings and can help predict task outcomes. However, whether traits qualify as skills remains to be seen, due to inconclusive results on generalizability and how user traits change over time.

发表机构

• University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）

机构由 AI 辅助整理，请以论文原文为准。