关于时间的许多事：时态数据的详尽注释
Much Ado About Time: Exhaustive Annotation of Temporal Data
摘要：
大规模的注释数据集允许AI系统从人群的知识中学习并建立在人群的知识基础上。已经开发了许多众包技术用于收集图像注释。这些技术通常隐含地依赖于这样一个事实，即一个新的输入图像需要一个可以忽略不计的时间来感知。相比之下，我们调查并确定为视频等时态数据获取高质量多标签注释的最具成本效益的方法。即使观看一个30秒的短视频剪辑也需要人群工作人员投入大量时间;因此，在单次观看后请求多个注释是一个重要的节省成本的策略。但是每个视频我们应该问多少个问题？我们得出结论，最佳策略是在HIT中提出尽可能多的问题（在我们的实验中，在观看30秒的视频剪辑后，最多提出52个二进制问题）。我们证明，虽然工人可能无法正确回答所有问题，但成本效益分析有利于从多个这样的廉价但不完美的迭代更复杂的替代品的共识。当与每个视频一个问题的基线相比时，我们的方法能够在大约一半的注释时间（我们的3.8分钟与7.1分钟基线相比）内以相当的精度（83.8%与83.0%基线相比）实现10%的召回率提高（76.7%与66.7%基线相比）。我们通过收集1，815个视频上157个人类活动的多标签注释来证明我们方法的有效性。
英文摘要：
Large-scale annotated datasets allow AI systems to learn from and build upon the knowledge of the crowd. Many crowdsourcing techniques have been developed for collecting image annotations. These techniques often implicitly rely on the fact that a new input image takes a negligible amount of time to perceive. In contrast, we investigate and determine the most cost-effective way of obtaining high-quality multi-label annotations for temporal data such as videos. Watching even a short 30-second video clip requires a significant time investment from a crowd worker; thus, requesting multiple annotations following a single viewing is an important cost-saving strategy. But how many questions should we ask per video? We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments). We demonstrate that while workers may not correctly answer all questions, the cost-benefit analysis nevertheless favors consensus from multiple such cheap-yet-imperfect iterations over more complex alternatives. When compared with a one-question-per-video baseline, our method is able to achieve a 10% improvement in recall 76.7% ours versus 66.7% baseline) at comparable precision (83.8% ours versus 83.0% baseline) in about half the annotation time (3.8 minutes ours compared to 7.1 minutes baseline). We demonstrate the effectiveness of our method by collecting multi-label annotations of 157 human activities on 1,815 videos.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
