Encoder-decoder foundation models have displayed state-of-the-art performance on a range of autoregressive sequence tasks. This paper proposes a simple and lightweight modification to such systems to control the behaviour according to a specific attribute of interest. This paper proposes a novel inference-efficient approach to modifying the behaviour of an encoder-decoder system according to a specific attribute of interest. Specifically, we show that a small proxy network can be used to find a sample-by-sample perturbation of the encoder output of a frozen foundation model to trigger the decoder to generate improved decodings. This work explores a specific realization of this framework focused on improving the COMET performance of Flan-T5 on Machine Translation and the WER of Whisper foundation models on Speech Recognition. Results display consistent improvements in performance evaluated through COMET and WER respectively. Furthermore, experiments also show that the proxies are robust to the exact nature of the data used to train them and can extend to other domains.
Rethinking Perturbations in Encoder-Decoders for Fast Training
Tokyo Institute of Technology(东京工业大学);RIKEN(理化学研究所);Tohoku University(东北大学)
我们经常使用扰动来正则化神经模型。对于神经编码器-解码器,先前的研究应用了预定采样(Bengio等人,2015)和对抗性扰动(Sato et al.,2019年)作为扰动,但这些方法需要相当长的计算时间。因此,本研究解决了这些方法是否足够有效的训练时间的问题。我们比较了几个扰动序列到序列的问题相对于计算时间。实验结果表明,简单的技术,如单词丢弃(Gal和Ghahramani,2016)和输入标记的随机替换,与最近提出的扰动相比,可以获得相当(或更好)的分数,尽管这些简单的方法更快。我们的代码可在https://github.com/takase/rethink_perturbations上公开获取。
Training Multilingual Machine Translation by Alternately Freezing Language-Specific Encoders-Decoders
TALP Research Center(TALP研究中心);Universitat Politècnica de Catalunya(加泰罗尼亚理工大学);IXA NLP Group(IXA自然语言处理组);University of the Basque Country (UPV/EHU)(巴斯克大学(UPV/EHU))
Improving Domain Adaptation Translation with Domain Invariant and Specific Information
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
Geo-Encoder: A Chunk-Argument Bi-Encoder Framework for Chinese Geographic Re-Ranking
Huazhong University of Science and Technology(华中科技大学);Alibaba Group(阿里巴巴集团);School of Computer Science and Engineering, South China University of Technology(华南理工大学计算机科学与工程学院);University of Copenhagen(哥本哈根大学)
Language-specific Characteristic Assistance for Code-switching Speech Recognition
Tianjin University(天津大学);National University of Singapore(新加坡国立大学);Kyoto University(京都大学);Japan Advanced Institute of Science and Technology(日本北陆先端科学技术大学院大学)