When eliciting judgements from humans for an unknown quantity, one often has the choice of making direct-scoring (cardinal) or comparative (ordinal) measurements. In this paper we study the relative merits of either choice, providing empirical and theoretical guidelines for the selection of a measurement scheme. We provide empirical evidence based on experiments on Amazon Mechanical Turk that in a variety of tasks, (pairwise-comparative) ordinal measurements have lower per sample noise and are typically faster to elicit than cardinal ones. Ordinal measurements however typically provide less information. We then consider the popular Thurstone and Bradley-Terry-Luce (BTL) models for ordinal measurements and characterize the minimax error rates for estimating the unknown quantity. We compare these minimax error rates to those under cardinal measurement models and quantify for what noise levels ordinal measurements are better. Finally, we revisit the data collected from our experiments and show that fitting these models confirms this prediction: for tasks where the noise in ordinal measurements is sufficiently low, the ordinal approach results in smaller errors in the estimation.
Better Uncertainty Calibration via Proper Scores for Classification and Beyond
German Cancer Research Center (DKFZ)(德国癌症研究中心);German Cancer Consortium (DKTK)(德国癌症联盟);Goethe University Frankfurt(法兰克福大学);Frankfurt Cancer Institute(法兰克福癌症研究所)
Rectifying Conformity Scores for Better Conditional Coverage
Mohamed bin Zayed University of Artificial Intelligence(莫扎德·本·扎耶德人工智能大学);Skolkovo Institute of Science(斯克尔科沃科学研究所);University of Mons(蒙斯大学);Computing Research Center(计算研究中心)
An Explicit Sampling Dependent Spectral Error Bound for Column Subset Selection
the University of Iowa(爱荷华大学);Nanjing University(南京大学);Michigan State University(密歇根州立大学);Institute of Data Science and Technologies at Alibaba Group(阿里巴巴集团数据科学与技术研究院)