arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.00211cs.CLcs.LG
收藏

基于大型语言模型的多条件排序

Multi-Conditional Ranking with Large Language Models

Megagon Labs(美伽贡实验室)

Pouya Pezeshkpour, Estevam Hruschka

AI总结:

本文提出多条件排序任务及MCRank基准,并设计EXSIR分解推理方法,通过提取排序条件并迭代排序项目,显著提升LLMs在多条件排序中的性能,最高提升14.4%。

摘要:

利用大型语言模型(LLMs)对一组项目进行排序已成为推荐和检索系统中的常见方法。通常,这些系统侧重于根据给定查询以单调顺序对大量文档进行排序。然而,现实场景往往呈现不同的挑战:对相对较小的项目集合进行排序,但需依据多种多样且偶尔相互冲突的条件。在本文中,我们通过引入MCRank来定义和探索多条件排序任务,MCRank是一个专为评估跨不同项目类型和条件的多条件排序而设计的基准。我们使用MCRank对LLMs的分析表明,随着项目和条件的数量及复杂性的增长,性能显著下降。为克服这一局限,我们提出了一种新颖的分解推理方法,包括提取和排序条件,然后迭代地对项目进行排序(EXSIR)。我们的大量实验表明,这种分解推理方法显著提升了LLMs的性能,相较于现有LLMs最高实现了14.4%的提升。我们还提供了LLMs在不同条件类别上的性能详细分析,并检验了分解步骤的有效性。此外,我们将我们的方法与现有方法(如Chain-of-Thought和现有排序模型)进行了比较,证明了我们方法的优越性以及MCR任务的复杂性。我们已发布数据集和代码。

英文摘要:

Utilizing large language models (LLMs) to rank a set of items has become a common approach in recommendation and retrieval systems. Typically, these systems focus on ordering a substantial number of documents in a monotonic order based on a given query. However, real-world scenarios often present a different challenge: ranking a comparatively smaller set of items, but according to a variety of diverse and occasionally conflicting conditions. In this paper, we define and explore the task of multi-conditional ranking by introducing MCRank, a benchmark tailored for assessing multi-conditional ranking across various item types and conditions. Our analysis of LLMs using MCRank indicates a significant decrease in performance as the number and complexity of items and conditions grow. To overcome this limitation, we propose a novel decomposed reasoning method, consisting of EXtracting and Sorting the conditions, and then Iteratively Ranking the items (EXSIR). Our extensive experiments show that this decomposed reasoning method enhances LLMs' performance significantly, achieving up to a 14.4% improvement over existing LLMs. We also provide a detailed analysis of LLMs performance across various condition categories, and examine the effectiveness of decomposition step. Furthermore, we compare our method with existing approaches such as Chain-of-Thought and existing ranking models, demonstrating the superiority of our approach and complexity of MCR task. We released our dataset and code.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑