通过示例教人类何时遵从分类器
Teaching Humans When To Defer to a Classifier via Exemplars
摘要:
专家决策者开始依赖数据驱动的自动化代理来帮助他们完成各种任务。为了使这种合作能够正常进行,人类决策者必须有一个心理模型,知道何时以及何时不依赖代理。在这项工作中,我们的目标是确保人类决策者学习一个有效的心理模型的代理的优势和劣势。为了实现这一目标,我们提出了一个基于范例的教学策略,人类在代理的帮助下解决任务,并试图制定一套何时以及何时不推迟的指导方针。我们提出了一种新的参数化的人工智能的人的心理模型,适用于最近邻规则周围的教学实例的局部区域。使用这个模型,我们得出一个接近最优的策略选择一个有代表性的教学集。我们验证了我们的教学策略的好处多跳问答任务使用人群工作者,发现当工人从教学阶段吸取正确的教训,他们的任务绩效提高,我们进一步验证了我们的方法在一组合成实验。
英文摘要:
Expert decision makers are starting to rely on data-driven automated agents to assist them with various tasks. For this collaboration to perform properly, the human decision maker must have a mental model of when and when not to rely on the agent. In this work, we aim to ensure that human decision makers learn a valid mental model of the agent's strengths and weaknesses. To accomplish this goal, we propose an exemplar-based teaching strategy where humans solve the task with the help of the agent and try to formulate a set of guidelines of when and when not to defer. We present a novel parameterization of the human's mental model of the AI that applies a nearest neighbor rule in local regions surrounding the teaching examples. Using this model, we derive a near-optimal strategy for selecting a representative teaching set. We validate the benefits of our teaching strategy on a multi-hop question answering task using crowd workers and find that when workers draw the right lessons from the teaching stage, their task performance improves, we furthermore validate our method on a set of synthetic experiments.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。