The University of Hong Kong(香港大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);The Simons Institute for the Theory of Computing(西蒙斯理论计算研究所);University of California, Berkeley(加州大学伯克利分校);Sun Yat-sen University(中山大学)
Prompting and context-based fine-tuning methods, which we call Prefix Learning, have been proposed to enhance the performance of language models on various downstream tasks. They are empirically efficient and effective, matching the performance of full parameter fine-tuning, but the theoretical understandings are limited. In this paper, we aim to address this limitation by studying their ability from the perspective of prefix length. In particular, we provide a convergence guarantee for training an ultra-long prefix in a stylized setting using the Neural Tangent Kernel (NTK) framework. Based on this strong theoretical guarantee, we design and implement an algorithm that only needs to introduce and fine-tune a few extra trainable parameters instead of an infinite-long prefix in each layer of a transformer, and can approximate the prefix attention to a guaranteed polynomial-small error. Preliminary experimental results on vision, natural language, and math data show that our method achieves superior or competitive performance compared to existing methods like full parameters fine-tuning, P-Tuning V2, and LoRA. This demonstrates our method is promising for parameter-efficient fine-tuning. Our code can be found at \url{https://github.com/ChristianYang37/chiwun/tree/main/src/NTK-Attention}.
Transformers on Multilingual Clause-Level Morphology
Koç University(科奇大学)
本文介绍了我们在KUIS AI NLP团队设计的MRL:第一次多语言子句级形态学共享任务(EMNLP 2022研讨会)中的获奖系统。我们提出了我们的工作的所有三个部分的共同任务:拐点,再反射和分析。我们主要通过两种方法探索Transformers:(i)从头开始训练模型,结合数据增强,以及(ii)在多语言形态任务中使用前缀调整的迁移学习。数据增强显著提高了大多数语言在变形和再变形任务中的性能。另一方面,对预训练的mGPT模型进行前缀调优可以帮助我们在低数据和多语言环境中适应分析任务。虽然具有数据增强的Transformer架构在拐点和再拐点任务中获得了最有希望的结果,但mGPT上的前缀调谐在分析任务中获得了最高的结果。我们的系统在MRL 2022的所有三项任务中均获得第一名。
Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers
Tsinghua University(清华大学);Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);Peking University(北京大学);Shenzhen University of Advanced Technology(深圳先进技术大学)
Infinite Limits of Multi-head Transformer Dynamics
John A. Paulson School of Engineering and Applied Sciences(约翰·A·保尔森工程与应用科学学院);Center for Brain Science(脑科学中心);Kempner Institute for the Study of Natural and Artificial Intelligence(坎普纳自然与人工智能研究所);Harvard University(哈佛大学)