人机协作的标志:DARPA与计算机通信计划的评估框架
Hallmarks of Human-Machine Collaboration: A framework for assessment in the DARPA Communicating with Computers Program
摘要:
人们越来越希望创建能够有效地通信以在复杂的、开放式的活动中与人类协作的计算机系统。评估这些系统提出了重大挑战。我们描述了一个框架,用于评估系统参与开放式复杂的情况下,评估人员没有比较性能的一个单一的正确答案的奢侈。该框架已被用于评估跨故事和音乐生成、交互式积木构建以及癌症分子机制探索的人机创意合作。这些活动与大多数当代个人助理执行的更受约束的任务有着根本的不同,因为它们通常是开放式的,没有单一的正确解决方案,并且通常没有明显的完成标准。我们确定了成功的系统必须表现出的关键特性。从那里,我们确定了成功的“标志”--评估人员可以观察到的能力和特征,这些能力和特征将指示实现关键属性的进展。除了作为评估框架外,关键属性和标志还旨在作为指导研究方向的目标。
英文摘要:
There is a growing desire to create computer systems that can communicate effectively to collaborate with humans on complex, open-ended activities. Assessing these systems presents significant challenges. We describe a framework for evaluating systems engaged in open-ended complex scenarios where evaluators do not have the luxury of comparing performance to a single right answer. This framework has been used to evaluate human-machine creative collaborations across story and music generation, interactive block building, and exploration of molecular mechanisms in cancer. These activities are fundamentally different from the more constrained tasks performed by most contemporary personal assistants as they are generally open-ended, with no single correct solution, and often no obvious completion criteria. We identified the Key Properties that must be exhibited by successful systems. From there we identified "Hallmarks" of success -- capabilities and features that evaluators can observe that would be indicative of progress toward achieving a Key Property. In addition to being a framework for assessment, the Key Properties and Hallmarks are intended to serve as goals in guiding research direction.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。