A preference-based subjective evaluation is a key method for evaluating generative media reliably. However, its huge combinations of pairs prohibit it from being applied to large-scale evaluation using crowdsourcing. To address this issue, we propose an automatic optimization method for preference-based subjective evaluation in terms of pair combination selections and allocation of evaluation volumes with online learning in a crowdsourcing environment. We use a preference-based online learning method based on a sorting algorithm to identify the total order of evaluation targets with minimum sample volumes. Our online learning algorithm supports parallel and asynchronous execution under fixed-budget conditions required for crowdsourcing. Our experiment on preference-based subjective evaluation of synthetic speech shows that our method successfully optimizes the test by reducing pair combinations from 351 to 83 and allocating optimal evaluation volumes for each pair ranging from 30 to 663 without compromising evaluation accuracies and wasting budget allocations.
Towards personalized human AI interaction - adapting the behavior of AI agents using neural signatures of subjective interest
Department of Biomedical Engineering, Columbia University（生物医学工程系，哥伦比亚大学）；Data Science Institute, Columbia University（数据科学研究所，哥伦比亚大学）；National Institutes of Health（国家卫生研究院）
From Task Tuning to Task Assignment in Privacy-Preserving Crowdsourcing Platforms
众包平台的专业化工作者简档可以包含大量的识别和可能敏感的个人信息（例如，个人偏好、技能、可用插槽、可用设备）引起强烈的隐私问题。这导致了保护隐私的众包平台的设计，旨在实现高效的众包流程，同时即使在平台不完全可信的情况下也能提供强有力的隐私保证。在本文中，我们提出了两个贡献。首先，我们提出了PKD算法的目标是支持各种各样的聚合使用的工人配置文件在一个隐私保护的众包平台。PKD算法将同态加密和差分隐私组合在一起，用于计算实际工人群体的多维技能空间的（扰动的）分区和每个分区的工人的（扰动的）数目。其次，我们建议受益于私人信息检索技术的最新进展，以设计一个解决方案，任务分配，是私人和负担得起的。我们进行了深入研究的问题，使用PIR技术提出任务的工人，表明它是NP-难的，并提出了PKD PIR包装启发式分组任务一起根据分区输出的PKD算法。简而言之，我们设计了PKD算法和PKD PIR Packing启发式算法，我们正式证明了它们对诚实但好奇的工人和/或平台的安全性，我们分析了它们的复杂性，并通过在合成和现实数据集上进行广泛的实验评估，证明了它们在现实生活中的质量和可负担性。
