Many state-of-the-art natural language understanding (NLU) models are based on pretrained neural language models. These models often make inferences using information from multiple sources. An important class of such inferences are those that require both background knowledge, presumably contained in a model's pretrained parameters, and instance-specific information that is supplied at inference time. However, the integration and reasoning abilities of NLU models in the presence of multiple knowledge sources have been largely understudied. In this work, we propose a test suite of coreference resolution subtasks that require reasoning over multiple facts. These subtasks differ in terms of which knowledge sources contain the relevant facts. We also introduce subtasks where knowledge is present only at inference time using fictional knowledge. We evaluate state-of-the-art coreference resolution models on our dataset. Our results indicate that several models struggle to reason on-the-fly over knowledge observed both at pretrain time and at inference time. However, with task-specific training, a subset of models demonstrates the ability to integrate certain knowledge types from multiple sources. Still, even the best performing models seem to have difficulties with reliably integrating knowledge presented only at inference time.
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models
University of Tübingen(蒂宾根大学);Tübingen AI Center(蒂宾根人工智能中心);Helmholtz Munich(亥姆霍兹慕尼黑中心);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);University of Virginia(弗吉尼亚大学);University of British Columbia(不列颠哥伦比亚大学);Harvard Medical School(哈佛医学院);Technical University of Munich(慕尼黑工业大学)
IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
Institut Teknologi Bandung(万隆理工学院);Gojek;Universitas Indonesia(印度尼西亚大学);The Hong Kong University of Science and Technology(香港科技大学);Universitas Multimedia Nusantara(多媒体 nusantara 大学)
Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning
Huawei Technologies, Munich Research Center(华为技术有限公司慕尼黑研究中心);Music Technology Group, Universitat Pompeu Fabra(庞培法布拉大学音乐技术研究组);Audio Research Group, Tampere University(坦佩雷大学音频研究组)
GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspective
Westlake University(西湖大学);Westlake Institute for Advanced Study(西湖高等研究院);Central South University(中南大学);University of Electronic Science and Technology of China(电子科技大学);Microsoft Research Asia(微软亚洲研究院)
BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);University of Science and Technology of China(中国科学技术大学);Huazhong University of Science and Technology(华中科技大学);Microsoft Research(微软研究院)
IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
The Hong Kong University of Science and Technology(香港科技大学);DeepMind;Gojek;Institut Teknologi Bandung(万隆理工学院);Universitas Multimedia Nusantara(多媒体 nusantara 大学)
自然语言生成(NLG)基准为衡量进展和开发更好的NLG系统提供了重要途径。不幸的是,缺乏低资源语言的公开NLG基准,这对构建适用于数据量有限的语言的NLG系统构成了一个具有挑战性的障碍。在这里,我们介绍IndoNLG,第一个基准来衡量自然语言生成(NLG)的进展,在三个低资源-但广泛使用-印度尼西亚的语言:印度尼西亚语,爪哇语和巽他语。总的来说,这些语言有超过1亿的母语使用者,因此构成了当今NLG系统的重要用例。具体地说,IndoNLG涵盖了六个任务:摘要、问答、聊天和三对不同的机器翻译(MT)任务。我们整理了一个干净的印度尼西亚语,巽他语和爪哇语数据集的预训练语料库Indo 4 B-Plus,用于预训练我们的模型:IndoBART和IndoGPT。我们证明了IndoBART和IndoGPT在所有任务上都取得了有竞争力的性能-尽管只使用了更大的多语言模型mBART-LARGE的五分之一的参数(Liu et al.,2020年)。这一发现强调了对密切相关的本地语言进行预训练的重要性,以实现更有效的学习和更快的推理,例如爪哇语和巽他语等资源非常低的语言。
MEG-MASC: a high-quality magneto-encephalography dataset for evaluating natural speech processing
University of California, San Francisco(加州大学旧金山分校);New York University(纽约大学);NYU Abu Dhabi Institute(纽约大学阿布扎比研究所);Ernst Struengmann Institute for Neuroscience(恩斯特·施特伦曼神经科学研究所);PSL University(巴黎文理研究大学)
AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages
University of Colorado Boulder(科罗拉多大学博尔德分校);University of Stuttgart(斯图加特大学);University of Edinburgh(爱丁堡大学);Microsoft Turing(微软图灵);Facebook AI Research(脸书人工智能研究院);New York University(纽约大学);Universidad de la República, Uruguay(乌拉圭共和国大学);Universidad Tecnológica de Tlaxcala(特拉斯卡拉理工大学);University of Zurich(苏黎世大学);Carnegie Mellon University(卡内基梅隆大学);Universidad Nacional Autónoma de México(墨西哥国立自治大学);Universidade Tecnológica Federal do Paraná(巴拉那联邦理工大学);Dartmouth College(达特茅斯学院)
ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models
Peking University(北京大学);National Key Laboratory of General Artificial Intelligence(通用人工智能国家重点实验室);School of Intelligence Science and Technology, Peking University(北京大学智能科学与技术学院);Department of Sociology, Peking University(北京大学社会学系);School of Psychological and Cognitive Sciences, Peking University(北京大学心理与认知科学学院);PKU-Wuhan Institute for Artificial Intelligence(北京大学武汉人工智能研究院)