Baby Bear：为标量标注寻找“刚刚好”的评分量表
Baby Bear: Seeking a Just Right Rating Scale for Scalar Annotations
AI总结：
针对大规模元素标量标注的效率与成本问题，提出迭代式IBWS方法生成鲁棒众包排序数据，再以其为基准评估高性价比的直接评估方法，并验证其可支撑对话、情感领域的鲁棒排序学习模型。
摘要：
我们的目标是构建一种机制，能高效地为大量元素中的每一个分配标量评分，例如“这条产品评论的正面或负面占比是多少？”。当样本量较小时，已有研究推崇最佳最差标度法（Best Worst Scaling, BWS）等方法，认为它们比直接序数标注（“李克特量表”）更鲁棒。本文首先提出IBWS（迭代最佳最差标度法），该方法通过Best-Worst Scaling迭代收集标注，生成排序鲁棒的众包数据。IBWS虽有效，但对大规模任务而言成本过高。我们以IBWS的结果作为理想最优结果，评估多种直接评估方法，以确定兼具成本效益、且与大规模BWS标注策略相关性最高的方法。最后，我们在对话和情感领域展示了这些标注如何支撑鲁棒的learning-to-rank模型。
英文摘要：
Our goal is a mechanism for efficiently assigning scalar ratings to each of a large set of elements. For example, "what percent positive or negative is this product review?" When sample sizes are small, prior work has advocated for methods such as Best Worst Scaling (BWS) as being more robust than direct ordinal annotation ("Likert scales"). Here we first introduce IBWS, which iteratively collects annotations through Best-Worst Scaling, resulting in robustly ranked crowd-sourced data. While effective, IBWS is too expensive for large-scale tasks. Using the results of IBWS as a best-desired outcome, we evaluate various direct assessment methods to determine what is both cost-efficient and best correlating to a large scale BWS annotation strategy. Finally, we illustrate in the domains of dialogue and sentiment how these annotations can support robust learning-to-rank models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
