Conventional fine-tuning encounters increasing difficulties given the size of current Pre-trained Language Models, which makes parameter-efficient tuning become the focal point of frontier research. Previous methods in this field add tunable adapters into MHA or/and FFN of Transformer blocks to enable PLMs achieve transferability. However, as an important part of Transformer architecture, the power of layer normalization for parameter-efficent tuning is ignored. In this paper, we first propose LN-tuning, by tuning the gain and bias term of Layer Normalization module with only 0.03\% parameters, which is of high time-efficency and significantly superior to baselines which are less than 0.1\% tunable parameters. Further, we study the unified framework of combining LN-tuning with previous ones and we find that: (1) the unified framework of combining prefix-tuning, the adapter-based method working on MHA, and LN-tuning achieves SOTA performance. (2) unified framework which tunes MHA and LayerNorm simultaneously can get performance improvement but those which tune FFN and LayerNorm simultaneous will cause performance decrease. Ablation study validates LN-tuning is of no abundant parameters and gives a further understanding of it.
ÚFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5
Charles University（查理大学）
我们在W-NUT 2021上展示了多语言词汇规范化（MultiLexNorm）共享任务的获奖作品（van der Goot等人，2021 a），它评估了11种语言的12个社交媒体数据集的词汇规范化系统。我们的解决方案基于预先训练的字节级语言模型ByT 5（Xue et al.，2021 a），我们进一步对合成数据进行预训练，然后对真实的归一化数据进行微调。我们的系统实现了最好的性能，通过一个很大的利润率在内在的评价，也是最好的性能，通过依存句法分析的外在评价。源代码发布于https://github.com/ufal/multilexnorm2021，微调模型发布于https://huggingface.co/ufal。
Alibaba-Translate China's Submission for WMT 2022 Metrics Shared Task
DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）；University of Macau（澳门大学）；University of Science and Technology of China（中国科学技术大学）；National University of Singapore（新加坡国立大学）
PAST-TIDE: Prototype-Anchored Statement Tuning with Topic-Invariant Normalization for Stance Detection
Department of Computer Science and Engineering, Khulna University of Engineering & Technology（库尔纳工程技术大学计算机科学与工程系）；Department of Electrical and Computer Engineering, University of Waterloo（滑铁卢大学电气与计算机工程系）；Department of Machine Learning, Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学机器学习系）
On Layer Normalization in the Transformer Architecture
Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Beijing Institute of Big Data Research（北京大数据研究院）；Microsoft Research（微软研究院）；Nankai University（南开大学）
Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification
The University of Tokyo（东京大学）；University of Southern California（南加州大学）；The Chinese University of Hong Kong（香港中文大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）
