arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.04671cs.CLcs.LGq-bio.PE
收藏

PhyloLM : 推断大语言模型的系统发育并预测其在基准测试中的性能

PhyloLM : Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

INSERM(法国国家健康与医学研究院);Flowers AI & CogSci Lab(Flowers人工智能与认知科学实验室);ENS, PSL(巴黎高等师范学院-PSL大学);Centre Inria de L’Université de Bordeaux(波尔多大学Inria中心)

Nicolas Yax, Pierre-Yves Oudeyer, Stefano Palminteri

AI总结:

PhyloLM通过系统发育算法推断大语言模型的系统发育关系并预测其在基准测试中的性能。

摘要:

本文介绍了PhyloLM,一种将系统发育算法适应于大语言模型(LLMs)的方法,以探讨它们之间以及与基准测试性能之间的关系。我们的方法基于LLM输出的相似性计算系统发育距离度量,该度量随后用于构建树状图,能够充分捕捉111个开源模型和45个闭源模型之间的已知关系。此外,我们的系统发育距离能够预测标准基准测试中的性能,从而证明了其功能有效性,并为高效估计LLM能力提供了途径。总而言之,通过将群体遗传概念转化为机器学习,我们提出并验证了一种评估LLM发展、关系和能力的工具,即使在缺乏透明训练信息的情况下。

英文摘要:

This paper introduces PhyloLM, a method adapting phylogenetic algorithms to Large Language Models (LLMs) to explore whether and how they relate to each other and to predict their performance characteristics. Our method calculates a phylogenetic distance metric based on the similarity of LLMs' output. The resulting metric is then used to construct dendrograms, which satisfactorily capture known relationships across a set of 111 open-source and 45 closed models. Furthermore, our phylogenetic distance predicts performance in standard benchmarks, thus demonstrating its functional validity and paving the way for a time and cost-effective estimation of LLM capabilities. To sum up, by translating population genetic concepts to machine learning, we propose and validate a tool to evaluate LLM development, relationships and capabilities, even in the absence of transparent training information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑