Revisiting Parameter-Efficient Tuning: Are We Really There Yet?
Sun Yat-sen University(中山大学);University of Cambridge(剑桥大学);University of Glasgow(格拉斯哥大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
Parameter-Efficient Tuning (PETuning) methods have been deemed by many as the new paradigm for using pretrained language models (PLMs). By tuning just a fraction amount of parameters comparing to full model finetuning, PETuning methods claim to have achieved performance on par with or even better than finetuning. In this work, we take a step back and re-examine these PETuning methods by conducting the first comprehensive investigation into the training and evaluation of them. We found the problematic validation and testing practice in current studies, when accompanied by the instability nature of PETuning methods, has led to unreliable conclusions. When being compared under a truly fair evaluation protocol, PETuning cannot yield consistently competitive performance while finetuning remains to be the best-performing method in medium- and high-resource settings. We delve deeper into the cause of the instability and observed that the number of trainable parameters and training iterations are two main factors: reducing trainable parameters and prolonging training iterations may lead to higher stability in PETuning methods.
SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Amazon(亚马逊);University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校);University of Texas at Austin(德克萨斯大学奥斯汀分校);University at Buffalo(布法罗大学);Northeastern University(东北大学)
Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark
Michigan State University(密歇根州立大学);The University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);UT Austin(德克萨斯大学奥斯汀分校);University of Minnesota Twin Cities(明尼苏达大学双城分校);IBM Research(IBM研究院);Princeton University(普林斯顿大学);DAMO Academy, Alibaba Group(阿里巴巴达摩院);MIT(麻省理工学院);Harvard University(哈佛大学)
Do Clinical Question Answering Systems Really Need Specialised Medical Fine Tuning?
University of Delhi(德里大学);Jamia Hamdard(贾迈亚哈马尔德大学);Cornell University(康奈尔大学);Expedia Group(Expedia集团);DSEU-Okhla;Center for SDGC(SDGC中心);Stanford University(斯坦福大学)
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Alibaba Group(阿里巴巴集团)
Is There a One-Model-Fits-All Approach to Information Extraction? Revisiting Task Definition Biases
Fudan University(复旦大学);School of Computer Science, Fudan University(复旦大学计算机科学学院);School of Data Science, Fudan University(复旦大学数据科学学院);Fudan-Aishu Cognitive Intelligence Joint Research Center(复旦-爱数认知智能联合研究中心)
Revisiting Self-Training for Few-Shot Learning of Language Model
National University of Singapore(新加坡国立大学);Southern University of Science and Technology(南方科技大学);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));Kriston AI Lab(Kriston人工智能实验室)