Large Language Models (LLMs) operating in 0-shot or few-shot settings achieve competitive results in Text Classification tasks. In-Context Learning (ICL) typically achieves better accuracy than the 0-shot setting, but it pays in terms of efficiency, due to the longer input prompt. In this paper, we propose a strategy to make LLMs as efficient as 0-shot text classifiers, while getting comparable or better accuracy than ICL. Our solution targets the low resource setting, i.e., when only 4 examples per class are available. Using a single LLM and few-shot real data we perform a sequence of generation, filtering and Parameter-Efficient Fine-Tuning steps to create a robust and efficient classifier. Experimental results show that our approach leads to competitive results on multiple text classification datasets.
Optimizing the Phi-2 Small Language Model for Real-time Chatbot Applications Using Parameter-Efficient Fine-Tuning (PEFT) with QLoRA Quantization
Beacom College of Computer and Cyber Sciences, Dakota State University（达科他州立大学比科姆计算机与网络科学学院）；Mathematics and Computer Science, Augustana College（奥古斯塔纳学院数学与计算机科学系）
Hybrid and Unitary PEFT for Resource-Efficient Large Language Models
Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong（信息工程系，香港中文大学）；Department of Electrical and Computer Engineering, University of California San Diego, USA（电气与计算机工程系，加州大学圣地亚哥分校）；Department of Computer Science, University of Copenhagen, Denmark（计算机科学系，哥本哈根大学）；Department of Electrical Engineering, Columbia University, USA（电气工程系，哥伦比亚大学）
KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Electronics and Telecommunications Research Institute（电子通信研究院）；The Hong Kong University of Science and Technology（香港科技大学）
RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
Tsinghua University（清华大学）；Beijing Information Science and Technology University（北京信息科技大学）；National University of Singapore（新加坡国立大学）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Beijing University of Posts and Telecommunications（北京邮电大学）
FISH-Tuning: Enhancing PEFT Methods with Fisher Information
Hubei Provincial Key Laboratory of Artificial Intelligence and Smart Learning（湖北省人工智能与智能学习重点实验室）；National Language Resources Monitoring and Research Center for Network Media（网络媒体语言资源监测与研究中心）；School of Computer, Central China Normal University（中央财经大学计算机学院）
DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation
University of Tübingen（图宾根大学）；Tübingen AI Center（图宾根人工智能中心）；Helmholtz Munich（海德堡-穆恩医疗中心）；Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）；MDSI（慕尼黑数据科学研究所）
Task-Specific Directions: Definition, Exploration, and Utilization in Parameter Efficient Fine-Tuning
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能基础理论实验室，人工智能研究院，上海交通大学）；School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University（电子信息与电气工程学院，上海交通大学）；School of Engineering and Applied Sciences, Harvard University（工程与应用科学学院，哈佛大学）
LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data
IDEA Research, International Digital Economy Academy（IDEA研究院，国际数字经济学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；DataArc Tech Ltd.（DataArc科技有限公司）；Hithink RoyalFlush Information Network Co., Ltd（Hithink皇家Flush信息网络有限公司）
Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures
Beijing International Center for Mathematical Research, Peking University（北京大学北京国际数学研究中心）；Center for Data Science, Peking University（北京大学数据科学中心）；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）
Breaking the Blocks: Continuous Low-Rank Decomposed Scaling for Unified LLM Quantization and Adaptation
Institute for Artificial Intelligence, Peking University, Beijing, China（北京大学人工智能研究院）；Yuanpei College, Peking University, Beijing, China（北京大学元培学院）；State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China（通用人工智能国家重点实验室）；Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学（深圳））
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS（中国科学院计算技术研究所智能信息处理重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；Peng Cheng Laboratory（鹏城实验室）
Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA（Chandra Family 电子与计算机工程系，德克萨斯大学奥斯汀分校）；Snap, Inc.（Snap公司）；Google Research（Google研究）
Enhancing Agentic Textual Graph Retrieval with Synthetic Stepwise Supervision
Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）；Peking University（北京大学）；School of Information, Renmin University of China（中国人民大学信息学院）；Harbin Institute of Technology（哈尔滨工业大学）；North China Electric Power University（华北电力大学）；GDS Holdings Limited（GDS控股有限公司）
Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuning
Faculty of Engineering, Computer Engineering Department, Cairo University（工程学院，计算机工程系，开罗大学）；Menzies School of Health Research, Charles Darwin University（健康研究学院，查尔斯达尔文大学）
