There has been significant progress in recent years in the field of Natural Language Processing thanks to the introduction of the Transformer architecture. Current state-of-the-art models, via a large number of parameters and pre-training on massive text corpus, have shown impressive results on several downstream tasks. Many researchers have studied previous (non-Transformer) models to understand their actual behavior under different scenarios, showing that these models are taking advantage of clues or failures of datasets and that slight perturbations on the input data can severely reduce their performance. In contrast, recent models have not been systematically tested with adversarial-examples in order to show their robustness under severe stress conditions. For that reason, this work evaluates three Transformer-based models (RoBERTa, XLNet, and BERT) in Natural Language Inference (NLI) and Question Answering (QA) tasks to know if they are more robust or if they have the same flaws as their predecessors. As a result, our experiments reveal that RoBERTa, XLNet and BERT are more robust than recurrent neural network models to stress tests for both NLI and QA tasks. Nevertheless, they are still very fragile and demonstrate various unexpected behaviors, thus revealing that there is still room for future improvement in this field.
Analysing similarities between legal court documents using natural language processing approaches based on Transformers
TRT5 - Regional Labour Court of the 5th Region(第五地区地区劳工法院);SENAI CIMATEC University Center(SENAI CIMATEC大学中心);Surrey Institute for People-Centred AI(萨里以人为本人工智能研究所);University of Surrey(萨里大学)
From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation
United Arab Emirates University(阿联酋大学);Emirates Center for Mobility Research(阿联酋移动性研究中心)
An explainable vision transformer with transfer learning based efficient drought stress identification
Dept. of Computer Science & Engineering(计算机科学与工程系);North Eastern Regional Institute of Science and Technology(北东地区科学与技术研究所);Dept. of Bio-Science and Bio-Engineering(生物科学与生物工程系);Indian Institute of Technology Guwahati(印度理工学院古瓦哈蒂分校);Dept. of Physics(物理系)
Chart-to-Text: Generating Natural Language Descriptions for Charts by Adapting the Transformer Model
York University(约克大学)
信息可视化(如条形图和折线图)在探索数据和传达见解方面非常受欢迎。对某些人来说,解释和理解这种可视化可能具有挑战性,例如那些视力受损或视觉素养低的人。在这项工作中,我们引入了一个新的数据集,并提出了一个神经模型自动生成图表的自然语言摘要。生成的摘要提供了对图表的解释,并传达了在该图表中发现的关键见解。我们的神经模型是通过扩展数据到文本生成任务的最先进模型开发的,该模型利用了基于transformer的编码器-解码器架构。我们发现,我们的方法在内容选择指标上远远优于基础模型(55.42% vs. 8.49%),并生成了更多信息,简洁和连贯的摘要。
Boosting 3D Neuron Segmentation with 2D Vision Transformer Pre-trained on Natural Images
School of Computer Science, The University of Sydney(悉尼大学计算机科学学院);Data61, CSIRO(澳大利亚联邦科学与工业研究组织Data61);SEU-ALLEN Joint Center, Institute for Brain and Intelligence, Southeast University(东南大学SEU-ALLEN联合中心脑与智能研究院)
Modeling cognitive processes of natural reading with transformer-based Language Models
Universidad de Buenos Aires(布宜诺斯艾利斯大学);Laboratorio de Inteligencia Artificial Aplicada(应用人工智能实验室);Instituto de Ciencias de la Computación (ICC)(计算科学研究所);Maestría en Explotación de Datos y Descubrimiento del Conocimiento(数据开采与知识发现硕士项目)
Algebraic Language Models for Inverse Design of Metamaterials via Diffusion Transformers
Mechanics & Materials Laboratory, Department of Mechanical and Process Engineering, ETH Zürich(机械与材料实验室,机械与过程工程系,苏黎世联邦理工学院);Department of Materials Science and Engineering, Delft University of Technology(材料科学与工程系,代尔夫特理工大学)
The Chess Transformer: Mastering Play using Generative Language Models
PeopleTec;Auburn University(奥本大学)
这项工作表明,自然语言Transformers可以支持更通用的战略建模,特别是对文本存档的游戏。除了学习自然语言技能之外,抽象的Transformer架构还可以在棋盘上生成有意义的移动。通过进一步的微调,Transformer通过在Portable Game Notation中对280万盘国际象棋进行训练来学习复杂的游戏玩法。经过30,000个训练步骤后,OpenAI的生成式预训练Transformer(GPT-2)优化了7.74亿个参数的权重。这款经过微调的国际象棋Transformer可以生成合理的策略,并显示可识别为经典开局的游戏阵型,例如英语或斯拉夫交换。最后,在现场比赛中,新模型展示了一个人对Transformer的界面,正确过滤非法移动,并提供了一种新的方法来挑战Transformer的象棋策略。我们预计未来的工作将建立在这个Transformer的承诺,特别是在其他战略游戏中,功能可以捕获简单但富有表现力的玩家注释的基础复杂的规则语法。