We advance the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems. First, we report that using the recently proposed Transformer architecture even out-of-the-box outperforms previous neural and non-neural systems for code prediction. We then show that by making the Transformer architecture aware of the syntactic structure of code, we further increase the margin by which a Transformer-based system outperforms previous systems. With this, it outperforms the accuracy of an RNN-based system (similar to Hellendoorn et al. 2018) by 18.3%, the Deep3 system (Raychev et al 2016) by 14.1%, and an adaptation of Code2Seq (Alon et al., 2018) for code prediction by 14.4%.
We present in the paper several ways of communicating the code structure to the Transformer, which is fundamentally built for processing sequence data. We provide a comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on a Facebook internal Python corpus. Our code and data preparation pipeline will be available in open source.
Tree Meets Transformer: A Hybrid Architecture for Scalable Power Allocation in Cell-Free Networks
Smart Networks and Services Joint Undertaking (SNS JU)（智能网络与服务联合体）；Italian Ministry of Education and Research (MUR)（意大利教育与研究部）；HORIZON-JU-SNS-2022 EU project TIMES（HORIZON-JU-SNS-2022 EU项目TIMES）；European Union under the Italian National Recovery and Resilience Plan (NRRP) of NextGenerationEU（欧盟在意大利国家复苏与韧性计划（NRRP）下的NextGenerationEU）
Transformer是现代NLP模型的支柱。在本文中，我们提出了RealFormer，一个简单而通用的技术来创建剩余注意力层Transformer网络，显着优于规范的Transformer及其变体（BERT，ETC等）。在各种各样的任务上，包括Masked Language Modeling、GLUE、SQuAD、Neural机器翻译、WikiHop、HotpotQA、Natural Questions和OpenKP。我们还根据经验观察到，RealFormer稳定了训练，并导致模型具有更稀疏的注意力。RealFormer的源代码和预先训练的检查点可以在https://github.com/google-research/google-research/tree/master/realformer上找到。
SolarTformer: A Transformer Based Deep Learning Approach for Short Term Solar Power Forecasting
Positional Astronomy Centre, India Meteorological Department, Kolkata, India（印度气象局位置天文学中心）；Department of Computer Science and Engineering, Jadavpur University, Kolkata, India（贾瓦德普大学计算机科学与工程系）；Department of Computer Science and Engineering (Artificial Intelligence and Machine Learning), Heritage Institute of Technology, Kolkata, India（遗产理工学院计算机科学与工程系（人工智能和机器学习））；Department of Electronics and Communication Engineering, Heritage Institute of Technology, Kolkata, India（遗产理工学院电子与通信工程系）；Department of Mechanical Engineering, Heritage Institute of Technology, Kolkata, India（遗产理工学院机械工程系）
Pathformer: Recursive Path Query Encoding for Complex Logical Query Answering
School of Computer Science and Engineering, South China University of Technology（华南理工大学计算机科学与工程学院）；Guangdong University of Petrochemical Technology（广东石油化工学院）；Jiangmen Polytechnic（江门职业技术学院）
kTrans: Knowledge-Aware Transformer for Binary Code Embedding
Tsinghua University（清华大学）；Beijing University of Technology（北京工业大学）；Huazhong University of Science and Technology（华中科技大学）；Information Engineering University（信息工程大学）
