arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.38683cs.IRcs.CL
收藏

基于主动LLM反馈的量子启发式成对排名聚合

Quantum-Inspired Pairwise Ranking Aggregation via Active LLM Feedback

University of Arkansas(阿肯色大学);Walmart Inc.(沃尔玛公司)

Quan Mai, Susan Gauch, Gibson Nkhata

AI总结:

提出LLM-IRA,将量子启发式排名聚合框架扩展至文档重排序,仅用LLM对高分歧文档对做成对判断,以少量LLM调用实现优于或媲美静态融合基线的效果,且无需训练数据。

摘要:

来自多个检索器的排名聚合是现代检索和RAG系统中一项强大但未被充分探索的技术。我们推出LLM-IRA,这是量子启发式交互式排名聚合框架在文档重排序领域的首个扩展。我们的方法仅利用大语言模型(LLM)对高分歧度的文档对进行成对判断,以原则性的交互方式学习融合权重。最终排名是分数的简单、超快速的点态线性融合,无需完整的成对矩阵或昂贵的列表态提示。在BEIR基准和TREC深度学习轨道上,使用开放权重指令微调的评判模型Llama-3.1-8B-Instruct进行实验,结果显示LLM-IRA在6个评估基准中的5个上与强大的静态融合基线相当或在数值上超过它们。我们报告两种权重更新变体:动量融合和完整累积历史拟合,两者共同在6个基准中的4个上达到了统计显著性(Wilcoxon符号秩检验,p<0.05),其中动量融合在DL19基准、累积历史拟合在DL20、DBpedia和NFCorpus基准上表现更优,且累积历史拟合是两种配置中整体更强的。在我们分析的基准(DL19、DL20)中,每个查询仅需5次LLM调用就能超过最佳无监督基线,远未达到其典型的25-30次调用预算。LLM-IRA还实现了与最先进的零样本LLM重排器相当的成本效益,但LLM调用次数减少了一个数量级,且无需训练数据或模型微调。

英文摘要:

Ranking aggregation from multiple retrievers is a powerful yet under-explored technique in modern retrieval and RAG systems. We introduce LLM-IRA, the first extension of the quantum-inspired interactive ranking aggregation framework to document reranking. By leveraging large language models (LLMs) solely for pairwise judgments on high-disagreement document pairs, our method learns fusion weights in a principled, interactive way. The final ranking is a simple, ultra-fast pointwise linear fusion of scores, no full pairwise matrix or expensive listwise prompting is required. Experiments on the BEIR benchmark and TREC Deep Learning tracks with an open-weight instruction-tuned judge (Llama-3.1-8B-Instruct) show that LLM-IRA matches or numerically exceeds strong static fusion baselines on 5 of 6 evaluated benchmarks. We report two weight-update variants: momentum blending and a full accumulated-history refit, which together reach statistical significance (Wilcoxon signed-rank, p<0.05) over the best unsupervised aggregator on 4 of 6 benchmarks (DL19 under momentum; DL20, DBpedia, and NFCorpus under accumulated refit, which is the stronger of the two configurations overall). On the benchmarks we profiled (DL19, DL20), it already clears the best unsupervised baseline within as few as 5 LLM calls per query, well before exhausting its typical ~25--30 call budget. LLM-IRA also achieves cost-effectiveness comparable to state-of-the-art zero-shot LLM rerankers at an order of magnitude fewer LLM calls, and requires no training data or model fine-tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑