The recent rise in the popularity of large language models has spurred the development of extensive code datasets needed to train them. This has left limited code available for collection and use in the downstream investigation of specific behaviors, or evaluation of large language models without suffering from data contamination. To address this problem, we release The Heap, a large multilingual dataset covering 57 programming languages that has been deduplicated with respect to other open datasets of code, enabling researchers to conduct fair evaluations of large language models without significant data cleaning overhead.
Translation as a Scalable Proxy for Multilingual Evaluation
University of California, Los Angeles（加州大学洛杉矶分校）；African Languages Lab（非洲语言实验室）；Mila - Quebec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
Disentangling Language Roles in Multilingual LLM Task Execution
Marquette（马凯特大学）；Cornell（康奈尔大学）；UC San Diego（南加州大学圣地亚哥分校）；UPenn（普林斯顿大学）；UT Austin（德克萨斯大学奥斯汀分校）；Maryland（马里兰大学）；Michigan（密歇根大学）；UIUC（伊利诺伊大学香槟分校）；Melbourne（墨尔本大学）；Stanford（斯坦福大学）
Output Language Confusion under Multilingual Prompt Contamination
University of Tübingen（蒂宾根大学）；Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；Thapar Institute of Engineering & Technology（塔帕尔工程技术学院）；Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）；Indian AI Research Organization（印度人工智能研究组织）
CRUXEval-X: A Benchmark for Multilingual Code Reasoning, Understanding and Execution
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences（中国科学院大学）；The Hong Kong University of Science and Technology（香港科技大学）；Huazhong University of Science and Technology（华中科技大学）
When Flores Bloomz Wrong: Cross-Direction Contamination in Machine Translation Evaluation
Saarland University（萨尔兰大学）；Queen’s University Belfast（贝尔法斯特女王大学）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）；University of Edinburgh（爱丁堡大学）
BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism
Brno University of Technology（布拉格技术大学）；Masaryk University（马萨里大学）；University of Helsinki（赫尔辛基大学）；Czech Technical University in Prague（布拉格捷克技术大学）；University of Ostrava（奥斯特拉瓦大学）；CIIRC（捷克信息与通信研究中心）；Hugging Face
Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness
Queen’s University Belfast（贝尔法斯特女王大学）；University of Technology Nuremberg（纽伦堡工业大学）；Saarland University（萨尔大学）；University of Melbourne（墨尔本大学）；University of Helsinki（赫尔辛基大学）；IIIT Hyderabad（海得拉巴国际信息技术学院）；TCS Research（塔塔咨询服务公司研究院）；Johns Hopkins University（约翰斯·霍普金斯大学）；Imam Ja’afar Al-Sadiq University（伊玛目贾法尔·萨迪克大学）；Cohere（Cohere公司）；Last Token（Last Token公司）；Charles University（查理大学）；University of Edinburgh（爱丁堡大学）；ETH Zurich（苏黎世联邦理工学院）；University of Zurich（苏黎世大学）
GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek
LIX, Ecole Polytechnique（巴黎综合理工学院LIX实验室）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；National Technical University of Athens（雅典国家技术大学）；University of Ioannina（约阿尼纳大学）；University of Peloponnese（伯罗奔尼撒大学）
