Parameter efficient learning methods (PERMs) have recently gained significant attention as they provide an efficient way for pre-trained language models (PLMs) to adapt to a downstream task. However, these conclusions are mostly drawn from in-domain evaluations over the full training set. In this paper, we present comparisons between PERMs and finetuning from three new perspectives: (1) the effect of sample and model size to in-domain evaluations, (2) generalization to unseen domains and new datasets, and (3) the faithfulness of generations. Our results show that for in-domain settings (a) there is a cross point of sample size for which PERMs will perform better than finetuning when training with fewer samples, and (b) larger PLMs have larger cross points. For cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al., 2019) performs the best amongst all the PERMs studied here, and (b) it outperforms finetuning if the task dataset is below a certain size. We also compare the faithfulness of generations and show that PERMs can achieve better faithfulness score than finetuning, especially for small training set, by as much as 6%. Finally, we apply Adapter to MT-NLG 530b (Smith et al., 2022) and achieve new state-of-the-art results on Xsum (Narayan et al., 2018) for all ROUGE scores (ROUGE-1 49.17, ROUGE-2 27.20, ROUGE-L 40.98).
Evaluating, Understanding, and Improving Constrained Text Generation for Large Language Models
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Center for Data Science, Peking University(北京大学数据科学中心);The MOE Key Laboratory of Computational Linguistics, Peking University(北京大学教育部计算语言学重点实验室)
Evaluating Large Language Models on Controlled Generation Tasks
University of Southern California(南加州大学);University of California, Los Angeles(加州大学洛杉矶分校);Amazon(亚马逊);Google DeepMind(谷歌DeepMind);ETH Zurich(苏黎世联邦理工学院)
State Key Lab of AI Safety, Institute of Computing Technology, CAS(人工智能安全国家重点实验室,计算技术研究所,中国科学院);Key Lab of AI Safety, Chinese Academy of Sciences(人工智能安全重点实验室,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation
Communication University of China(中国传媒大学);Harbin Institute of Technology(哈尔滨工业大学);China Academy of Railway Sciences Corporation Limited(中国铁道科学研究院集团有限公司);School of Engineering, Santa Clara University(圣克拉拉大学工程学院)
MRScore: Evaluating Radiology Report Generation with LLM-based Reward System
The University of Sydney(悉尼大学);Guangzhou University of Chinese Medicine(广州中医药大学);The University of Adelaide(阿德莱德大学);The University of Wollongong(卧龙岗大学)
Fùxì: A Benchmark for Evaluating Language Models on Ancient Chinese Text Understanding and Generation
School of Computer Science and Technology, East China Normal University(华东师范大学计算机科学与技术学院);Lab of Artificial Intelligence for Education, East China Normal University(华东师范大学教育人工智能实验室);Shanghai Institute of Artificial Intelligence for Education, East China Normal University(华东师范大学上海教育人工智能研究院)
CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts
Department of Computer Science(计算机科学系);Chandigarh University Punjab(昌迪加尔大学旁遮普);Department of Technology Management and Innovation(技术管理与创新系);New York University(纽约大学)
Evaluating LLMs' Divergent Thinking Capabilities for Scientific Idea Generation with Minimal Context
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Bank of China(中国银行);School of Engineering Science, University of Chinese Academy of Sciences(中国科学院大学工程科学学院);State Key Laboratory of Nonlinear Mechanics, Institute of Mechanics, Chinese Academy of Sciences(中国科学院力学研究所非线性力学重点实验室)
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
University of California, Santa Barbara(加利福尼亚大学圣巴巴拉分校);School of Software Engineering, Xi’an Jiaotong University(西安交通大学软件学院);Zhejiang University(浙江大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Alibaba Group(阿里巴巴集团);TU Wien(维也纳技术大学);University of Chinese Academy of Sciences(中国科学院大学)