Ashish Vaswani

A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio

基于对数对齐比率的训练时泛化诊断

Ali Shehper, Ashish Vaswani

机构 * Essential AI 纠错

AI总结 提出对数对齐比率（LAR）作为参数-激活对齐的度量，通过捕捉训练中权重谱与激活谱的扩散来跟踪记忆与泛化的转换，并在grokking和语言模型预训练中预测泛化差距。

Comments 32 pages, 25 figures

Essential-Web v1.0: 24T tokens of organized web data

Essential-Web v1.0：24万亿token的有组织网络数据

Essential AI, :, Andrew Hojel, Michael Pust, Tim Romanski, Yash Vanjani, Ritvik Kapila, Mohit Parmar, Adarsh Chaluvaraju, Alok Tripathy, Anil Thomas, Ashish Tanwer, Darsh J Shah, Ishaan Shah, Karl Stratos, Khoi Nguyen, Kurt Smith, Michael Callahan, Peter Rushton, Philip Monk, Platon Mazarakis, Saad Jamal, Saurabh Srivastava, Somanshu Singla, Ashish Vaswani

AI总结 本文提出Essential-Web v1.0，一个24万亿token的带十二类分类标注的网络数据集，通过轻量模型标注和SQL过滤，在数学、代码、STEM和医学任务上取得竞争力提升。

Comments include MegaMath-Web-Pro

Practical Efficiency of Muon for Pretraining

Muon在预训练中的实际效率

Essential AI, :, Ishaan Shah, Anthony M. Polloreno, Karl Stratos, Philip Monk, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Ashish Tanwer, Darsh J Shah, Khoi Nguyen, Kurt Smith, Michael Callahan, Michael Pust, Mohit Parmar, Peter Rushton, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Somanshu Singla, Tim Romanski, Yash Vanjani, Ashish Vaswani

AI总结 该研究证明二阶优化器Muon在预训练的计算-时间权衡上拓展了优于AdamW的帕累托前沿，结合muP实现高效超参数迁移，提出低开销伸缩算法，经40亿参数规模实验验证可实现更经济的大批量训练。

Rethinking Reflection in Pre-Training

重新思考预训练中的反思

Essential AI, :, Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, Anil Thomas, Anthony Polloreno, Ashish Tanwer, Burhan Drak Sibai, Divya S Mansingka, Divya Shivaprasad, Ishaan Shah, Karl Stratos, Khoi Nguyen, Michael Callahan, Michael Pust, Mrinal Iyer, Philip Monk, Platon Mazarakis, Ritvik Kapila, Saurabh Srivastava, Tim Romanski

机构 * Essential AI 纠错

AI总结 该研究重新思考了预训练中的反思能力，通过在思维链中引入故意错误测试模型的自我纠正能力，发现这种能力在预训练早期即出现并稳步提升，如OLMo2-7B模型在六项自我反思任务上展现了自我纠正能力。

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin

机构 * Google Brain（谷歌大脑） ； Google Research（谷歌研究院） ； University of Toronto（多伦多大学） ； Google（谷歌） 纠错

Comments 15 pages, 5 figures

The Efficiency Misnomer

Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, Yi Tay

机构 * Google Research（谷歌研究院） 纠错

Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers

Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, Donald Metzler

机构 * Google Research & DeepMind（谷歌研究院与深度思维） 纠错

Comments ICLR 2022 + Updated Checkpoint Release

Bottleneck Transformers for Visual Recognition

Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani

机构 * UC Berkeley（加州大学伯克利分校） ； Google Research（谷歌研究院） 纠错

Comments Technical Report, 20 pages, 13 figures, 19 tables

Scaling Local Self-Attention for Parameter Efficient Visual Backbones

Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, Jonathon Shlens

机构 * Google Research（谷歌研究院） ； UC Berkeley（加州大学伯克利分校） 纠错

Comments CVPR 2021 Oral

Simple and Efficient ways to Improve REALM

Vidhisha Balachandran, Ashish Vaswani, Yulia Tsvetkov, Niki Parmar

机构 * Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所） ； Google Research（谷歌研究院） 纠错

Efficient Content-Based Sparse Attention with Routing Transformers

Aurko Roy, Mohammad Saffar, Ashish Vaswani, David Grangier

机构 * Google Research（谷歌研究院） 纠错

Comments TACL 2020; pre-MIT Press publication version; v5 has a random attention baseline

Attention Augmented Convolutional Networks

Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V. Le

机构 * Google Brain（谷歌大脑） 纠错

Comments ICCV 2019

Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation

Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, Jason Baldridge

机构 * Google Research（谷歌研究院） 纠错

Comments Accepted at ACL 2019 as long paper

Stand-Alone Self-Attention in Vision Models

Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, Jonathon Shlens

机构 * Google Research（谷歌研究院） 纠错

Music Transformer

Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck

机构 * Google Brain（谷歌大脑） 纠错

Comments Improved skewing section and accompanying figures. Previous titles are "An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation" and "Music Transformer"

Mesh-TensorFlow: Deep Learning for Supercomputers

Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, Blake Hechtman

机构 * Google Brain（谷歌大脑） 纠错

Relational inductive biases, deep learning, and graph networks

Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, Razvan Pascanu

机构 * DeepMind（深度思维） ； Google Brain（谷歌大脑） ； MIT（麻省理工学院） ； University of Edinburgh（爱丁堡大学） 纠错

Theory and Experiments on Vector Quantized Autoencoders

Aurko Roy, Ashish Vaswani, Arvind Neelakantan, Niki Parmar

Image Transformer

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran

机构 * Google Brain（谷歌大脑） ； University of California, Berkeley（加州大学伯克利分校） ； Google AI（谷歌AI） 纠错

Comments Appears in International Conference on Machine Learning, 2018. Code available at https://github.com/tensorflow/tensor2tensor

Fast Decoding in Sequence Models using Discrete Latent Variables

Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, Noam Shazeer

机构 * Google Brain（谷歌大脑） 纠错

Comments ICML 2018

Self-Attention with Relative Position Representations

Peter Shaw, Jakob Uszkoreit, Ashish Vaswani

机构 * Google（谷歌） ； Google Brain（谷歌大脑） 纠错

Comments NAACL 2018

Tensor2Tensor for Neural Machine Translation

Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N. Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, Jakob Uszkoreit

机构 * Google Brain（谷歌大脑） ； DeepMind（深度思维） 纠错

Comments arXiv admin note: text overlap with arXiv:1706.03762

One Model To Learn Them All

Lukasz Kaiser, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, Jakob Uszkoreit

机构 * Google Brain（谷歌大脑） ； University of Toronto（多伦多大学） ； Google Research（谷歌研究院） 纠错

Unsupervised Neural Hidden Markov Models

Ke Tran, Yonatan Bisk, Ashish Vaswani, Daniel Marcu, Kevin Knight

机构 * Information Sciences Institute, University of Southern California（南加州大学信息科学研究所） ； Informatics Institute, University of Amsterdam（阿姆斯特丹大学信息学研究所） ； Google Brain（谷歌大脑） 纠错

Comments accepted at EMNLP 2016, Workshop on Structured Prediction for NLP. Oral presentation