Utilizing large language models (LLMs) to rank a set of items has become a common approach in recommendation and retrieval systems. Typically, these systems focus on ordering a substantial number of documents in a monotonic order based on a given query. However, real-world scenarios often present a different challenge: ranking a comparatively smaller set of items, but according to a variety of diverse and occasionally conflicting conditions. In this paper, we define and explore the task of multi-conditional ranking by introducing MCRank, a benchmark tailored for assessing multi-conditional ranking across various item types and conditions. Our analysis of LLMs using MCRank indicates a significant decrease in performance as the number and complexity of items and conditions grow. To overcome this limitation, we propose a novel decomposed reasoning method, consisting of EXtracting and Sorting the conditions, and then Iteratively Ranking the items (EXSIR). Our extensive experiments show that this decomposed reasoning method enhances LLMs' performance significantly, achieving up to a 14.4% improvement over existing LLMs. We also provide a detailed analysis of LLMs performance across various condition categories, and examine the effectiveness of decomposition step. Furthermore, we compare our method with existing approaches such as Chain-of-Thought and existing ranking models, demonstrating the superiority of our approach and complexity of MCR task. We released our dataset and code.
Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
Department of Statistics and Data Science, UCLA(统计与数据科学系,加州大学洛杉矶分校);Department of Statistics, University of Michigan(统计系,密歇根大学);Department of Statistics and Data Science, and of Biostatistics, UCLA(统计与数据科学系及生物统计学系,加州大学洛杉矶分校)
Empowering Language Model with Guided Knowledge Fusion for Biomedical Document Re-ranking
Lister Hill National Center for Biomedical Communications(利斯特·希尔国家生物医学交流中心);National Library of Medicine, National Institutes of Health(美国国立卫生研究院国家医学图书馆)
Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(中国科学技术大学认知智能国家重点实验室);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际 Graduate School);Zhejiang University(浙江大学)
TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy
Institute for Clarity in Documentation(文献清晰研究所);The Thørväld Group(索尔沃尔德集团);Inria(法国国家信息与自动化研究所);Rajiv Gandhi University(拉吉夫·甘地大学);Tsinghua University(清华大学);Palmer Research Laboratories(帕尔默研究实验室);The Kumquat Consortium(金桔联盟)
KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques
Texas A&M University(德克萨斯农工大学);The University of Tokyo(东京大学);Northwestern University(西北大学);Singapore General Hospital(新加坡中央医院);Zhejiang University(浙江大学);Zhejiang Lab(之江实验室);Yale University(耶鲁大学);National Institutes of Health(美国国立卫生研究院);Duke-NUS Medical School(杜克-新加坡国立大学医学院)
School of Mathematics, Northwest University(西北大学数学学院);Shanghai Institute for Mathematics and Interdisciplinary Sciences(上海数学与交叉科学研究院);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Cheriton School of Computer Science, University of Waterloo(滑铁卢大学切尔顿计算机科学学院)
Language Ranker: A Lightweight Ranking framework for LLM Decoding
State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University(一般人工智能国家重点实验室,智能科学与技术学院,北京大学);Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学);MIT CSAIL, MA, USA(MIT CSAIL,马萨诸塞州,美国);Microsoft Research Asia(微软亚洲研究院)
Unsupervised Ranking Model for Entity Coreference Resolution
Carnegie Mellon University(卡内基梅隆大学)
共指消解是深度语言理解的第一阶段,其重要性在自然语言处理领域得到了广泛的认可。本文通过引入消解模式变量,提出了一个生成式的、无监督的实体共指消解排序模型。我们的无监督系统在来自CoNLL-2012共享任务的英语数据上实现了CoNLL度量的58.44%F1得分(Pradhan等人,2012),优于斯坦福确定性系统(Lee et al.,2013年,3.01%。
ASU at TextGraphs 2019 Shared Task: Explanation ReGeneration using Language Models and Iterative Re-Ranking
Arizona State University(亚利桑那州立大学)
在这项工作中,我们描述了来自亚利桑那州立大学自然语言处理小组的TextGraphs 2019共享任务的系统。该任务的重点是解释再生,这是在大型图上进行一般多跳推理的中间步骤。我们的方法包括建模的解释再生任务作为一个\textit{learning to rank}问题,我们使用最先进的语言模型和探索数据集准备技术。我们利用一个迭代的重新排名为基础的方法,以进一步提高排名。我们的系统在测试集上以41.3%的平均精度(MAP)获得了任务的第二名。
PageRank without hyperlinks: Structural re-ranking using links induced by language models
受网页搜索的PageRank和HITS(hubs and authorities)算法的启发,我们提出了一种结构化的重新排序方法,用于ad hoc信息检索:我们通过利用它们之间的非对称关系对初始检索集中的文档进行重新排序。具体来说,我们考虑生成链接,这表明从一个文档诱导的语言模型分配给另一个文本的概率很高,在这样做的时候,我们要注意防止对长文档的偏见。我们研究了一些重新排名的标准的基础上的措施,在生成链接形成的图形的中心性,并表明,集成中心性到标准的基于语言模型的检索是非常有效的,在提高精度的顶级。
New York University(纽约大学);Facebook AI Research(Facebook人工智能研究院)
最近,神经模型在语言建模任务上进行了预训练,例如ELMo(Peters等人,2017)、OpenAI GPT(Radford等人,2018)和BERT(Devlin et al.,2018年),在各种自然语言处理任务上取得了令人印象深刻的成果,如问答和自然语言推理。在本文中,我们描述了一个简单的重新实现BERT的查询为基础的段落重新排名。我们的系统是TREC-CAR数据集的最新技术水平,也是MS MARCO通道检索任务排行榜的最佳条目,在MRR@10中比以前的最新技术水平高出27%(相对)。可在https://github.com/nyu-dl/dl4marco-bert上获得重现我们结果的代码