We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.
On Analyzing Annotation Consistency in Online Abusive Behavior Datasets
University of Saskatchewan（萨斯喀彻温大学）；University of Electronic Science and Technology of China（电子科技大学）；Istituto Dalle Molle di Studi sull’Intelligenza Artificiale（达莱莫勒人工智能研究所）
SciTweets -- A Dataset and Annotation Framework for Detecting Scientific Online Discourse
LIRMM；CNRS（法国国家科学研究中心）；University of Montpellier（蒙彼利埃大学）；GESIS - Leibniz Institute for the Social Sciences（GESIS-莱布尼茨社会科学研究所）；Heinrich-Heine-University（海因里希·海涅大学）
Dept. of Electrical and Computer Engineering（电气与计算机工程系）；Carnegie Mellon University（卡内基梅隆大学）；Dept. of Electrical Engineering（电气工程系）；Bilkent University（比尔肯特大学）
Codes, Patterns and Shapes of Contemporary Online Antisemitism and Conspiracy Narratives -- an Annotation Guide and Labeled German-Language Dataset in the Context of COVID-19
Hochschule für Technik und Wirtschaft Berlin（柏林工程与经济应用技术大学）；Alice Salomon Hochschule Berlin（柏林爱丽丝·萨洛蒙应用科学大学）
Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety
University of Southampton, UK（英国南安普顿大学）；Queen Mary University of London, UK（伦敦大学玛丽女王学院）；The Alan Turing Institute, UK（阿兰·图灵研究所）；Bar Ilan University, Israel（巴伊兰大学）
AQuA -- Combining Experts' and Non-Experts' Views To Assess Deliberation Quality in Online Discussions Using LLMs
Heinrich Heine University Düsseldorf（杜塞尔多夫海因里希·海涅大学）；Technical University Ilmenau（伊尔默瑙工业大学）；Johannes Gutenberg University Mainz（美因茨约翰内斯·古腾堡大学）；Technical University Dortmund（多特蒙德工业大学）
Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Unlimidata Ltd（Unlimidata有限公司）；The Chinese University of Hong Kong（香港中文大学）；University of Calabria（卡拉布里亚大学）
