Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for AI (hessian.AI)（黑森州人工智能中心）；Amazon Alexa AI（亚马逊Alexa人工智能部门）；IBM Research Europe（IBM欧洲研究院）
Scientific leaderboards are standardized ranking systems that facilitate evaluating and comparing competitive methods. Typically, a leaderboard is defined by a task, dataset, and evaluation metric (TDM) triple, allowing objective performance assessment and fostering innovation through benchmarking. However, the exponential increase in publications has made it infeasible to construct and maintain these leaderboards manually. Automatic leaderboard construction has emerged as a solution to reduce manual labor. Existing datasets for this task are based on the community-contributed leaderboards without additional curation. Our analysis shows that a large portion of these leaderboards are incomplete, and some of them contain incorrect information. In this work, we present SciLead, a manually-curated Scientific Leaderboard dataset that overcomes the aforementioned problems. Building on this dataset, we propose three experimental settings that simulate real-world scenarios where TDM triples are fully defined, partially defined, or undefined during leaderboard construction. While previous research has only explored the first setting, the latter two are more representative of real-world applications. To address these diverse settings, we develop a comprehensive LLM-based framework for constructing leaderboards. Our experiments and analysis reveal that various LLMs often correctly identify TDM triples while struggling to extract result values from publications. We make our code and data publicly available.
ClimateCheck 2026: Scientific Fact-Checking and Disinformation Narrative Classification of Climate-related Claims
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)（德国人工智能研究中心）；Technische Universität Berlin（柏林工业大学）；Leuphana Universität Lüneburg（吕讷堡大学）；Humboldt-Universität zu Berlin（柏林洪堡大学）
The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks
Bundesministerium für wirtschaftliche Zusammenarbeit und Entwicklung (BMZ)（经济合作与发展部）；Gesellschaft für Internationale Zusammenarbeit (GIZ)（国际发展合作署）；Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI)（德国人工智能研究中心）
本文提出AI Language Proficiency Monitor，一个评估多达200种语言（含低资源语言）的多语言基准，整合翻译、问答等任务，提供开源自动更新的排行榜和仪表板，促进多语言AI的透明与包容。
Cohere Labs（Cohere 实验室）；Cohere（Cohere 公司）；Princeton University（普林斯顿大学）；Stanford University（斯坦福大学）；University of Waterloo（滑铁卢大学）；Massachusetts Institute of Technology（麻省理工学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Washington（华盛顿大学）
CABACE: Injecting Character Sequence Information and Domain Knowledge for Enhanced Acronym and Long-Form Extraction
Indian Institute of Technology Kharagpur（印度理工学院卡拉格普尔分校）
首字母缩略词和长形式在研究文件中很常见，在科学和法律领域的文件中更是如此。这些文档中使用的许多首字母缩略词都是特定于领域的，很少在普通文本语料库中找到。因此，基于transformer的NLP模型通常会检测首字母缩略词标记的OOV（Out of Vocabulary），特别是对于非英语语言，并且在提取过程中将首字母缩略词链接到其长形式时，其性能会受到影响。此外，像BERT这样的预训练的Transformer模型并不专门处理科学和法律文档。这些要点是这项工作背后的首要动机，我们提出了一个新的框架CABACE：字母缩写词提取的字符感知BERT，它考虑到文本中的字符序列，并适应于科学和法律领域的掩蔽语言建模。我们进一步使用具有增强的损失函数的目标，将最大损失和掩码损失项添加到用于训练CABACE的标准交叉熵损失。我们进一步利用伪标签和对抗性数据生成来提高框架的通用性。实验结果证明了所提出的框架相比，各种基线的优越性。此外，我们表明，所提出的框架比基线模型更适合zero-shot推广到非英语语言，从而加强了我们的方法的有效性。我们的团队BacKGProp在SDU AAAI-22的首字母缩略词提取（AE）共享任务的全球排行榜上获得了法语数据集的最高分，丹麦语和越南语的第二高分，以及英语法律数据集的第三高分。
Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy
Energy, Materials and Environment Research Centre, London South Bank University, London SE1 0AA, UK（能源、材料与环境研究中心，伦敦南银行大学）；School of Engineering and Design, London South Bank University, London SE1 0AA, UK（工程与设计学院，伦敦南银行大学）；Bioscience and Bioengineering Research Centre, London South Bank University, London SE1 0AA, UK（生物科学与生物工程研究中心，伦敦南银行大学）；Department of Physics, Kings College London, London WC2R 2LS, UK（物理系，伦敦国王学院）
