Grasping the commonsense properties of everyday concepts is an important prerequisite to language understanding. While contextualised language models are reportedly capable of predicting such commonsense properties with human-level accuracy, we argue that such results have been inflated because of the high similarity between training and test concepts. This means that models which capture concept similarity can perform well, even if they do not capture any knowledge of the commonsense properties themselves. In settings where there is no overlap between the properties that are considered during training and testing, we find that the empirical performance of standard language models drops dramatically. To address this, we study the possibility of fine-tuning language models to explicitly model concepts and their properties. In particular, we train separate concept and property encoders on two types of readily available data: extracted hyponym-hypernym pairs and generic sentences. Our experimental results show that the resulting encoders allow us to predict commonsense properties with much higher accuracy than is possible by directly fine-tuning language models. We also present experimental results for the related task of unsupervised hypernym discovery.
Great Truths are Always Simple: A Rather Simple Knowledge Encoder for Enhancing the Commonsense Reasoning Capacity of Pre-Trained Models
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Information, Renmin University of China（中国人民大学信息学院）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）；Microsoft Research Asia（微软亚洲研究院）；Microsoft（微软）
预先训练的语言模型已经在广泛的自然语言处理（NLP）任务中取得了实质性的进展，但已经被证明对于具有高质量输出要求的自然语言生成任务具有局限性，例如常识生成和广告关键字生成。在这项工作中，我们提出了一种新的知识过滤和对比学习网络（KFCNet），参考外部知识，并取得更好的生成性能。具体来说，我们提出了一个基于BERT的过滤器模型，以消除低质量的候选人，并分别应用对比学习的编码器和解码器中的每一个，在一个通用的编码器-解码器架构。编码器对比模块有助于在编码过程中捕获全局目标语义，解码器对比模块在学习一般特征的同时增强检索到的原型的实用性。在CommonGen基准测试上进行的大量实验表明，我们的模型远远优于先前的最先进水平：BLEU-4为+6.6点（42.5 vs. 35.9），SPICE为+3.7点（33.3 vs. 29.6），CIDER为+1.3点（18.3 vs. 17.0）。我们进一步验证了所提出的对比模块在广告关键词生成上的有效性，并表明我们的模型具有潜在的商业价值。
Modelling Latent Translations for Cross-Lingual Transfer
Mila – Quebec Artificial Intelligence Institute（米拉-魁北克人工智能研究所）；McGill University（麦吉尔大学）；Google Research（谷歌研究院）；University of Cambridge（剑桥大学）；Facebook（脸书）
The Zero Resource Speech Challenge 2021: Spoken language modelling
University of Toronto（多伦多大学）；Cognitive Machine Learning (ENS--CNRS--EHESS--INRIA--PSL Research University)（认知机器学习实验室（巴黎高等师范学院--法国国家科学研究中心--社会科学高等研究院--法国国家信息与自动化研究所--巴黎文理研究大学））；Facebook AI Research（脸书人工智能研究院）
University of Copenhagen（哥本哈根大学）；ROCKWOOL Foundation Research Unit（洛克伍德基金会研究部）；Johns Hopkins University（约翰斯·霍普金斯大学）；KU Leuven（荷语鲁汶大学）；Pioneer Centre for AI（先锋人工智能中心）
M-Adapter: Modality Adaptation for End-to-End Speech-to-Text Translation
Monash University（莫纳什大学）
端到端的语音到文本翻译模型通常使用预训练的语音编码器和预训练的文本解码器进行初始化。这导致预训练和微调之间的显著训练间隙，这主要是由于来自编码器的语音输出和到解码器的文本输入之间的模态差异。在这项工作中，我们的目标是弥合语音和文本之间的模态差距，以提高翻译质量。我们提出了M-Adapter，一个新的基于transformer的模块，以适应语音表示文本。在缩小语音序列的同时，M-Adapter通过对语音序列的全局和局部依赖关系进行建模，产生语音到文本翻译所需的特征。我们的实验结果表明，我们的模型在Must-C En $\rightarrow$ DE数据集上的性能优于强基线，最高可达1个BLEU分数。\footnote{Our code is available at https://github.com/mingzi151/w2v2-st.}
