先排序,再提问:通过输入重排序最大化 LLM 在对称任务上的性能
Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks
AI总结:
针对 LLM 处理无序元素集合时易遗漏元素的问题,提出利用辅助 LLM 估计元素相关性与位置重要性的输入重排序方法,使对称任务准确率最高接近最优上界的 99%。
摘要:
大型语言模型(LLM)已迅速成为实用且通用的工具,为众多领域提供了新的解决方案。本文研究 LLM 在对称任务上的应用:在这类任务中,查询针对一个(无序)元素集合提出。此类任务的例子包括回答数据库表上的聚合查询。一般而言,当集合包含大量元素时,LLM 往往会忽略其中一些元素,从而给生成准确的查询响应带来挑战。LLM 以有序序列的形式接收输入;然而,在该问题中,我们利用了一个事实:对称输入本身没有顺序,因此重新排序不应影响 LLM 的响应。观察到 LLM 在输入的某些位置更不容易遗漏元素后,我们提出了 LLM 输入重排序问题:在不对查询作出显式假设的情况下,找到一种输入排序,使 LLM 对给定查询的准确率最大化。寻找最优排序需要确定两点:(i)每个输入元素对回答该查询的相关性,以及(ii)每个排序位置对 LLM 注意力的重要性。我们利用一个辅助 LLM,开发了用于高效估计这些数值的算法。我们在不同的合成数据集和真实数据集上进行了全面实验,以验证所提方案并评估所提算法的有效性。实验证实,我们的重排序方法可将 LLM 在对称任务上的准确率提升至与最优上界接近度最高达 99% 的水平。
英文摘要:
Large language models (LLMs) have quickly emerged as practical and versatile tools that provide new solutions for a wide range of domains. In this paper, we consider the application of LLMs on symmetric tasks where a query is asked on an (unordered) bag of elements. Examples of such tasks include answering aggregate queries on a database table. In general, when the bag contains a large number of elements, LLMs tend to overlook some elements, leading to challenges in generating accurate responses to the query. LLMs receive their inputs as ordered sequences. However, in this problem, we leverage the fact that the symmetric input is not ordered, and reordering should not affect the LLM's response. Observing that LLMs are less likely to miss elements at certain positions of the input, we introduce the problem of LLM input reranking: to find a ranking of the input that maximizes the LLM's accuracy for the given query without making explicit assumptions about the query. Finding the optimal ranking requires identifying (i) the relevance of each input element for answering the query and (ii) the importance of each rank position for the LLM's attention. We develop algorithms for estimating these values efficiently utilizing a helper LLM. We conduct comprehensive experiments on different synthetic and real datasets to validate our proposal and to evaluate the effectiveness of our proposed algorithms. Our experiments confirm that our reranking approach improves the accuracy of the LLMs on symmetric tasks by up to $99\%$ proximity to the optimum upper bound.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。