Practical machine learning systems often operate in multiple sequential stages, as seen in ranking and recommendation systems, which typically include a retrieval phase followed by a ranking phase. Effectively assessing prediction uncertainty and ensuring effective risk control in such systems pose significant challenges due to their inherent complexity. To address these challenges, we developed two-stage risk control methods based on the recently proposed learn-then-test (LTT) and conformal risk control (CRC) frameworks. Unlike the methods in prior work that address multiple risks, our approach leverages the sequential nature of the problem, resulting in reduced computational burden. We provide theoretical guarantees for our proposed methods and design novel loss functions tailored for ranked retrieval tasks. The effectiveness of our approach is validated through experiments on two large-scale, widely-used datasets: MSLR-Web and Yahoo LTRC.
Long Context Modeling with Ranked Memory-Augmented Retrieval
University of New South Wales(新南威尔士大学);Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);University of Southampton(南安普顿大学);Mohamed Bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
From Ranked Documents to Reliable Contexts: An Answer-Oriented Context Construct Framework for AI Search
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Baidu Inc.(百度公司);Wuhan University(武汉大学)
Exploiting Semantic Role Contextualized Video Features for Multi-Instance Text-Video Retrieval EPIC-KITCHENS-100 Multi-Instance Retrieval Challenge 2022
Institute for Infocomm Research, A*STAR(新加坡科技研究局资讯通信研究院);School of Computer Science and Engineering, NTU(南洋理工大学计算机科学与工程学院)
University of Science and Technology of China(中国科学技术大学);University of Electronic Science and Technology of China(电子科技大学);Lanzhou University(兰州大学);University of Glasgow(格拉斯哥大学);City University of Hong Kong(香港城市大学);The Ohio State University(俄亥俄州立大学)
AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval
Cyberagent, Inc.(赛博艾坚特公司);Waseda University(早稻田大学);Osaka University(大阪大学);Tampere University(坦佩雷大学);University of Oulu(奥卢大学)
评价措施对研究方向有着至关重要的影响。因此,对于常规方法不太适用的新应用,开发适当和可靠的评估方法至关重要。视频矩检索(VMR)就是这样一种应用,并且当前的实践是使用R@ $K,θ$来评估VMR系统。然而,这一措施有两个缺点。首先,它是排名不敏感的:它通过将列表视为集合而忽略排名前$K$的列表中成功定位的时刻的排名位置。其次,它使用阈值$θ$将每个检索到的视频时刻的交集(IoU)二值化,从而忽略排名时刻的细粒度定位质量。我们提出了一种替代措施,用于评估VMR,称为平均最大IoU(AxIoU),这是免费的上述两个问题。我们表明,AxIoU满足两个重要的公理VMR评估,即\textbf{Invariance against Redundant Moments}和\textbf{Monotonicity with respect to the Best Moment},也R@ $K,θ$满足第一个公理。我们还从经验上研究了AxIoU与R@ $K,θ$的一致性,以及它在测试数据和人类注释的时间边界变化方面的稳定性。
Deep Retrieval at CheckThat! 2025: Identifying Scientific Papers from Implicit Social Media Mentions via Hybrid Retrieval and Re-Ranking
University of Zurich(苏黎世大学);Centre for Artificial Intelligence(人工智能中心);Zurich University of Applied Sciences(应用科学大学);Institute of Neuroinformatics(神经信息学研究所);European Centre for Living Technology(欧洲生活技术中心)
C-RAG: Certified Generation Risks for Retrieval-Augmented Language Models
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Delft University of Technology(代尔夫特理工大学);Netflix Eyeline Studios;University of California, Berkeley(加州大学伯克利分校);University of Chicago(芝加哥大学)
Robustness Risk of Conversational Retrieval: Identifying and Mitigating Noise Sensitivity in Qwen3-Embedding Model
Beijing University of Posts and Telecommunications(北京邮电大学);SenseTime(商汤科技);Beijing Key Laboratory of Network System and Network Culture(网络系统与网络文化北京市重点实验室)
Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generation
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);City University of Hong Kong(香港城市大学);Tencent Technology(腾讯科技);Peking University(北京大学)
NPRF: A Neural Pseudo Relevance Feedback Framework for Ad-hoc Information Retrieval
University of Chinese Academy of Sciences(中国科学院大学);Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所);Computer Network Information Center, Chinese Academy of Sciences(中国科学院计算机网络信息中心);SAP SE(SAP公司);Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)
Evaluating Stochastic Rankings with Expected Exposure
我们引入了\emph{expected exposure}的概念,作为平均关注度排名项目收到来自用户的重复样本相同的查询。此外,我们主张采用平等的预期曝光原则:给定一个固定的信息需求,没有一个项目的预期曝光应该比相同相关性等级的任何其他项目更多或更少。我们认为,这一原则是可取的许多检索目标和场景,包括主题多样性和公平的排名。利用现有的检索指标的用户模型,我们提出了一个通用的评估方法,基于预期的曝光和绘制连接到相关的指标在信息检索评估。重要的是,这种方法放松了经典的信息检索假设,允许系统响应于查询产生一个\emph{distribution over rankings},而不是一个单一的固定排名。我们研究的行为,预期的曝光度量和随机排名在各种信息访问条件,包括\emph{ad hoc}检索和推荐。我们相信,使用随机化来测量和优化预期的暴露度量为检索算法的开发和进步开辟了一个新的领域。