为标注员提供可供性支持以高效标注对话数据
Supporting Annotators with Affordances for Efficiently Labeling Conversational Data
AI总结:
针对众包对话数据标注耗时昂贵且枯燥的问题,研究设计了具备防错选、引导标注等功能的新型标注界面CAL,用户研究显示其能降低认知负荷、更易用且受偏好,未增加任务时长。
摘要:
如果没有标注良好的基准真值数据,基于机器学习的系统不会像如今这样普及,但这类系统依赖大量正确标注的数据。遗憾的是,众包标注既耗时又昂贵。为解决标注工作量大且枯燥的问题,我们设计了CAL——一款辅助数据标注的新型界面。我们为CAL做出了多项关键设计决策,包括防止选择不当标签、在用户需要帮助时引导其选择合适标签、将标注文档整合进界面,以及提供查看历史标注的高效方式。我们实现了具备生产级质量的CAL版本,并开展了一项用户研究评估,将CAL与标准电子表格进行对比。研究的关键发现包括:使用CAL的用户报告认知负荷更低,任务耗时并未增加,用户认为CAL更易用,且相较于电子表格更偏好CAL。
英文摘要:
Without well-labeled ground truth data, machine learning-based systems would not be as ubiquitous as they are today, but these systems rely on substantial amounts of correctly labeled data. Unfortunately, crowdsourced labeling is time consuming and expensive. To address the concerns of effort and tedium, we designed CAL, a novel interface to aid in data labeling. We made several key design decisions for CAL, which include preventing inapt labels from being selected, guiding users in selecting an appropriate label when they need assistance, incorporating labeling documentation into the interface, and providing an efficient means to view previous labels. We implemented a production-quality implementation of CAL and report a user-study evaluation that compares CAL to a standard spreadsheet. Key findings of our study include users using CAL reported lower cognitive load, did not increase task time, users rated CAL to be easier to use, and users preferred CAL over the spreadsheet.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。