We introduce EdgeFormer -- a parameter-efficient Transformer for on-device seq2seq generation under the strict computation and memory constraints. Compared with the previous parameter-efficient Transformers, EdgeFormer applies two novel principles for cost-effective parameterization, allowing it to perform better given the same parameter budget; moreover, EdgeFormer is further enhanced by layer adaptation innovation that is proposed for improving the network with shared layers.
Extensive experiments show EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve competitive results under both the computation and memory constraints. Given the promising results, we release EdgeLM -- the pretrained version of EdgeFormer, which is the first publicly available pretrained on-device seq2seq model that can be easily fine-tuned for seq2seq tasks with strong results, facilitating on-device seq2seq generation in practice.
Toward Automated Regulatory Decision-Making: Trustworthy Medical Device Risk Classification with Multimodal Transformers and Self-Training
Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford（生物医学工程研究所，工程科学系，牛津大学）；Birmingham Law School, University of Birmingham（伯明翰法学院，伯明翰大学）
Re-Parameterization of Lightweight Transformer for On-Device Speech Emotion Recognition
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与电子工程学院）；Shenzhen Research Institute, Hunan University（湖南大学深圳研究院）；University of Cambridge（剑桥大学）
Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization
Department of Electrical and Computer Engineering, University of California, Santa Barbara（加州大学圣芭芭拉分校电子与计算机工程系）；Intel Corporation（英特尔公司）；Department of Computer Science, North Carolina State University（北卡罗来纳州立大学计算机科学系）；School of Electrical and Computer Engineering, Georgia Institute of Technology（佐治亚理工学院电气与计算机工程学院）
