LEGOBench：科学排行榜生成基准
LEGOBench: Scientific Leaderboard Generation Benchmark
AI总结：
本文提出LEGOBench基准，利用arXiv和PapersWithCode数据评估科学排行榜自动生成系统，揭示现有模型在该任务上的性能差距。
摘要：
论文提交量的持续增长使得及时了解最新前沿研究变得困难。为应对这一挑战，我们提出了LEGOBench，一个用于评估生成科学排行榜系统的基准。LEGOBench基于arXiv上22年的预印本提交数据以及PapersWithCode门户上超过1.1万个机器学习排行榜构建而成。我们提出了四种基于图的方法和两种基于语言模型的排行榜生成任务配置。我们在这些任务配置上评估了流行的仅编码器科学语言模型以及仅解码器大语言模型。当前最先进的模型在LEGOBench上的自动排行榜生成任务中表现出显著的性能差距。代码可在GitHub上获取（https://github.com/lingo-iitgn/LEGOBench），数据集托管在OSF上（https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c）。
英文摘要：
The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate scientific leaderboards. LEGOBench is curated from 22 years of preprint submission data on arXiv and more than 11k machine learning leaderboards on the PapersWithCode portal. We present four graph-based and two language model-based leaderboard generation task configurations. We evaluate popular encoder-only scientific language models as well as decoder-only large language models across these task configurations. State-of-the-art models showcase significant performance gaps in automatic leaderboard generation on LEGOBench. The code is available on GitHub ( https://github.com/lingo-iitgn/LEGOBench ) and the dataset is hosted on OSF ( https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c ).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
