In order to find experts, different approaches build rankings of people, assuming that they are ranked by level of expertise, and use typical Information Retrieval (IR) measures to evaluate their effectiveness. However, we figured out that expert rankings (i) tend to be partially ordered, (ii) incomplete, and (iii) consequently provide more an order rather than absolute ranks, which is not what usual IR measures exploit. To improve this state of the art, we propose to revise the formalism used in IR to design proper measures for comparing expert rankings. In this report, we investigate a first step by providing mitigation procedures for the three issues, and we analyse IR measures with the help of these procedures to identify interesting revisions and remaining limitations. From this analysis, we see that most of the measures can be exploited for this more generic context because of our mitigation procedures. Moreover, measures based on precision and recall, usually unable to consider the order of the ranked items, are of first interest if we represent a ranking as a set of ordered pairs. Cumulative measures, on the other hand, are specifically designed for considering the order but suffer from a higher complexity, motivating the use of precision/recall measures with the right representation.
MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval
Nanyang Technological University(南洋理工大学);Yale University(耶鲁大学);NYU Shanghai(纽约大学上海分校);Shanghai Jiao Tong University(上海交通大学);University of the Chinese Academy of Sciences(中国科学院大学);Center for Data Science, New York University(纽约大学数据科学中心);VinUniversity(越南 Vin 大学)
Language Bias in Information Retrieval: The Nature of the Beast and Mitigation Methods
School of Computing & Information Systems, The University of Melbourne(计算机与信息系统学院,墨尔本大学);Mohamed bin Zayed University of Artificial Intelligence, UAE(马尔代夫比兹亚德人工智能大学,阿联酋)
Retrieval of Scientific and Technological Resources for Experts and Scholars
School of Computer Science (National Pilot School of Software Engineering), Beijing University of Posts and Telecommunications(北京邮电大学计算机学院(国家软件工程试点学院));Beijing Key Laboratory of Intelligent Telecommunication Software and Multimedia(北京智能电信软件与多媒体重点实验室)
CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with Mixture-of-Textual-Experts for Passage Retrieval
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);Kuaishou Technology(快手科技)
IFIR: A Comprehensive Benchmark for Evaluating Instruction-Following in Expert-Domain Information Retrieval
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Chongqing Institute of Green and Intelligent Technology, Chinese Academy of Sciences(中国科学院重庆绿色智能技术研究院);Zhejiang University(浙江大学);Yale University(耶鲁大学)
MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
Department of Data Science & AI, Monash University, Australia(数据科学与人工智能系,墨尔本大学,澳大利亚);Department of Human Centred Computing, Monash University, Australia(以人为中心计算系,墨尔本大学,澳大利亚)
Cultural Analytics for Good: Building Inclusive Evaluation Frameworks for Historical IR
School of Computer Science, University College Dublin(都柏林大学学院计算机学院);Indian Institute of Science Education and Research, Kolkata(印度科学教育研究所加尔各答分校);School of English, Drama and Film, University College Dublin(都柏林大学学院英语、戏剧与电影学院);GESIS – Leibniz Institute for the Social Sciences, Cologne(科隆盖西-莱布尼茨社会科学研究所)
Diagnosing LLM-based Rerankers in Cold-Start Recommender Systems: Coverage, Exposure and Practical Mitigations
V. A. Trapeznikov ICS RAS Russian Biotechnological University (ROSBIOTECH)(Trapeznikov ICS 俄罗斯科学院俄罗斯生物技术大学);Russian Biotechnological University (ROSBIOTECH)(俄罗斯生物技术大学)
Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学耿丽人工智能学院);Beijing Normal University(北京师范大学);DataArc Tech Ltd.(数据弧科技有限公司);Huawei Cloud BU(华为云业务单元)
MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation
Istituto di Informatica e Telematica, Consiglio Nazionale Delle Ricerche(国家研究委员会信息与自动化研究所);TeCIP, Scuola Universitaria Superiore Sant’Anna(圣安娜高等学校TeCIP研究所);University of Padua(帕多瓦大学);Delft University of Technology(代尔夫特理工大学)