RICA：基于常识公理的鲁棒推理能力评估
RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms
摘要：
预训练的语言模型（PTLM）在常识推理基准测试中取得了令人印象深刻的性能，但它们利用常识进行鲁棒推理的能力存在争议，这对于与人类的有效沟通至关重要。为了推进流畅的人类-人工智能通信，我们提出了一个新的挑战，RICA：基于常识公理的鲁棒推理能力，它可以在文本扰动的情况下评估鲁棒的常识推理。为了生成这一挑战的数据，我们开发了一个系统的和可扩展的程序，使用常识知识库和探测PTLM在两个不同的评估设置。在我们生成的探针集上进行的大量实验表明，PTLM在zero-shot设置上的表现并不比随机猜测好，受到统计偏差的严重影响，并且对扰动攻击不鲁棒。我们还发现，微调类似的语句提供有限的收益，PTLM仍然无法推广到看不见的推论。我们新的大规模基准测试暴露了PTLM和人类水平的语言理解之间的巨大差距，并为PTLM展示常识提供了新的挑战。
英文摘要：
Pre-trained language models (PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA: Robust Inference capability based on Commonsense Axioms, that evaluates robust commonsense inference despite textual perturbations. To generate data for this challenge, we develop a systematic and scalable procedure using commonsense knowledge bases and probe PTLMs across two different evaluation settings. Extensive experiments on our generated probe sets with more than 10k statements show that PTLMs perform no better than random guessing on the zero-shot setting, are heavily impacted by statistical biases, and are not robust to perturbation attacks. We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
