The recent state-of-the-art natural language understanding (NLU) systems often behave unpredictably, failing on simpler reasoning examples. Despite this, there has been limited focus on quantifying progress towards systems with more predictable behavior. We think that reasoning capability-wise behavioral summary is a step towards bridging this gap. We create a CheckList test-suite (184K examples) for the Natural Language Inference (NLI) task, a representative NLU task. We benchmark state-of-the-art NLI systems on this test-suite, which reveals fine-grained insights into the reasoning abilities of BERT and RoBERTa. Our analysis further reveals inconsistencies of the models on examples derived from the same template or distinct templates but pertaining to same reasoning capability, indicating that generalizing the models' behavior through observations made on a CheckList is non-trivial. Through an user-study, we find that users were able to utilize behavioral information to generalize much better for examples predicted from RoBERTa, compared to that of BERT.
Implementing BERT and fine-tuned RobertA to detect AI generated news by ChatGPT
School of Software Technology, Zhejiang University(浙江大学软件学院);School of Computer Science and Engineering, University of New South Wales(新南威尔士大学计算机科学与工程学院);Computer and Information Engineering College, Inner Mongolia Normal University(内蒙古师范大学计算机与信息工程学院)
高质量的文本数据已经成为社会科学家的重要数据来源。我们已经见证了预训练的深度神经网络模型的成功,例如BERT和RoBERTa,在最近的社会科学研究中。在本文中,我们提出了一个紧凑的预训练深度神经网络,Transformer Encoder for Social Science(TESS),旨在解决社会科学研究中的文本处理任务。使用两个验证测试,我们证明了TESS优于BERT和RoBERTa平均16.7%的训练样本的数量是有限的(<1,000训练实例)。结果显示TESS在社会科学文本处理任务上优于BERT和RoBERTa。最后,我们讨论了我们的模型的局限性和未来的研究提出了建议。
Tourism Question Answer System in Indian Language using Domain-Adapted Foundation Models
Department of Linguistics Banaras Hindu University(语言学系班加罗尔 Hindu 大学);Department of Humanistic Studies Indian Institute of Technology (BHU)(人文科学系印度理工学院(BHU));Department of Computer Science and Engineering Indian Institute of Technology Bhilai(计算机科学与工程系印度理工学院 Bhilai)