Utility-Theoretic Ranking for Semi-Automated Text Classification
Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche（意大利国家研究委员会信息科学与技术研究所）；Qatar Computing Research Institute（卡塔尔计算研究所）；Italian National Council of Research（意大利国家研究委员会）
Giacomo Berardi, Andrea Esuli, Fabrizio Sebastiani
备注：
Forthcoming on ACM Transactions on Knowledge Discovery from Data
发表信息：
Final version published in ACM Transactions on Knowledge Discovery from Data, 10(1):Article 6, 2015
\emph{Semi-Automated Text Classification}（SATC）可以被定义为以这样的方式对自动标记的文本文档的集合$\mathcal{D}$进行排序的任务，即，如果人类注释者验证（即，检查并在适当的情况下进行校正）在$\mathcal{D}$的排名靠前的部分中的文档，目的是增加$\mathcal{D}$的总体标记准确性，因此预期的增加被最大化。一个明显的SATC策略是对$\mathcal{D}$进行排名，以便分类器标记的置信度最低的文档排名靠前。在这项工作中，我们表明，这种策略是次优的。我们开发了新的效用理论的排名方法的基础上的概念\emph{validation gain}，定义为改进的分类效率，将通过验证一个给定的自动标记的文件。我们还提出了一个新的有效性措施，面向SATC的排名方法的基础上，预期减少分类错误所带来的部分验证由给定的排名方法产生的列表。我们报告的实验结果表明，相对于上述基线方法，并根据建议的措施，我们的效用理论排名方法可以实现更高的预期减少分类错误。
英文摘要：
\emph{Semi-Automated Text Classification} (SATC) may be defined as the task of ranking a set $\mathcal{D}$ of automatically labelled textual documents in such a way that, if a human annotator validates (i.e., inspects and corrects where appropriate) the documents in a top-ranked portion of $\mathcal{D}$ with the goal of increasing the overall labelling accuracy of $\mathcal{D}$, the expected increase is maximized. An obvious SATC strategy is to rank $\mathcal{D}$ so that the documents that the classifier has labelled with the lowest confidence are top-ranked. In this work we show that this strategy is suboptimal. We develop new utility-theoretic ranking methods based on the notion of \emph{validation gain}, defined as the improvement in classification effectiveness that would derive by validating a given automatically labelled document. We also propose a new effectiveness measure for SATC-oriented ranking methods, based on the expected reduction in classification error brought about by partially validating a list generated by a given ranking method. We report the results of experiments showing that, with respect to the baseline method above, and according to the proposed measure, our utility-theoretic ranking methods can achieve substantially higher expected reductions in classification error.
CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization
Columbia University（哥伦比亚大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；Carnegie Mellon University（卡内基梅隆大学）；University of Pennsylvania（宾夕法尼亚大学）；Trine University（特莱恩大学）；University of Michigan（密歇根大学）
Towards Efficient Pareto-optimal Utility-Fairness between Groups in Repeated Rankings
College of Engineering & Computer Science, VinUniversity（VinUniversity工程与计算机科学学院）；University of Engineering and Technology, Vietnam National University（越南国家大学工程技术大学）
Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Baidu Inc（百度公司）
Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification
IIE, Chinese Academy of Sciences（中国科学院自动化研究所）；SCS, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；IIP, ICT, Chinese Academy of Sciences（中国科学院计算技术研究所）；SCST, University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Alibaba Group（阿里巴巴集团）；SCST, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）；BDKM, University of Chinese Academy of Sciences（中国科学院大学大数据与知识管理实验室）
