arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14860cs.CLcs.AIcs.LG
收藏

在无真实答案的情况下对大型语言模型进行排名

Ranking Large Language Models without Ground Truth

IBM Research, Ireland(IBM研究院爱尔兰分部);IBM Research, NY, USA(IBM研究院纽约分部)

Amit Dhurandhar, Rahul Nair, Moninder Singh, Elizabeth Daly, Karthikeyan Natesan Ramamurthy

AI总结:

本文提出在无真实答案的情况下,通过模型三元组互相评估并识别最差模型,反复应用以对LLMs进行排名,实验证明该方法能可靠恢复接近真实的排名。

摘要:

随着大型语言模型(LLMs)的激增及其影响,对它们的评估与排名已成为一个重要问题。评估方法要么需要获取成本高昂的人类回答,要么使用成对的LLMs互相评估,而后者可能不可靠。在本文中,我们提供了一个新颖的视角:给定一个提示数据集(即问题、指令等)和一组LLMs,我们在无法获得任何真实答案或参考回答的情况下对它们进行排名。受现实生活的启发——专家和知识渊博的人都能识别新手——我们的主要思想是考虑模型三元组,其中每个模型评估另外两个模型,并以高概率正确识别三元组中最差的模型。我们还分析了这一思想,并给出了其成功的充分条件。通过反复应用这一思想,我们提出了两种对LLMs进行排名的方法。在不同生成任务(摘要、多项选择和对话)的实验中,我们的方法在没有参考数据的情况下可靠地恢复出接近真实的排名。这为实际使用指明了一种可行的低资源机制。

英文摘要:

Evaluation and ranking of large language models (LLMs) has become an important problem with the proliferation of these models and their impact. Evaluation methods either require human responses which are expensive to acquire or use pairs of LLMs to evaluate each other which can be unreliable. In this paper, we provide a novel perspective where, given a dataset of prompts (viz. questions, instructions, etc.) and a set of LLMs, we rank them without access to any ground truth or reference responses. Inspired by real life where both an expert and a knowledgeable person can identify a novice our main idea is to consider triplets of models, where each one of them evaluates the other two, correctly identifying the worst model in the triplet with high probability. We also analyze our idea and provide sufficient conditions for it to succeed. Applying this idea repeatedly, we propose two methods to rank LLMs. In experiments on different generative tasks (summarization, multiple-choice, and dialog), our methods reliably recover close to true rankings without reference data. This points to a viable low-resource mechanism for practical use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑