Large language models (LLMs) excel at general question-answering (Q&A) but often fall short in specialized domains due to a lack of domain-specific knowledge. Commercial companies face the dual challenges of privacy protection and resource constraints when involving LLMs for fine-tuning. This paper propose a novel framework, Self-Evolution, designed to address these issues by leveraging lightweight open-source LLMs through multiple iterative fine-tuning rounds. To enhance the efficiency of iterative fine-tuning, Self-Evolution employ a strategy that filters and reinforces the knowledge with higher value during the iterative process. We employed Self-Evolution on Qwen1.5-7B-Chat using 4,000 documents containing rich domain knowledge from China Mobile, achieving a performance score 174% higher on domain-specific question-answering evaluations than Qwen1.5-7B-Chat and even 22% higher than Qwen1.5-72B-Chat. Self-Evolution has been deployed in China Mobile's daily operation and maintenance for 117 days, and it improves the efficiency of locating alarms, fixing problems, and finding related reports, with an average efficiency improvement of over 18.6%. In addition, we release Self-Evolution framework code in https://github.com/Zero-Pointer/Self-Evolution.
Operationalising the Right to be Forgotten in LLMs: A Lightweight Sequential Unlearning Framework for Privacy-Aligned Deployment in Politically Sensitive Environments
Department of Mathematics(数学系);Munster Technological University(穆恩斯特技术大学);Department of Computer Science(计算机科学系)
Revisiting KRISP: A Lightweight Reproduction and Analysis of Knowledge-Enhanced Vision-Language Models
Centre For Digital Health(数字健康中心);Indian Institute of Technology Bombay(孟买印度理工学院);Department of Metallurgical engineering & Material science(冶金工程与材料科学系);Department of Bioscience & Bioengineering(生物科学与生物工程系)
Fine-tuning of lightweight large language models for sentiment classification on heterogeneous financial textual data
International Business School, Zhejiang University(浙江大学国际商务学院);Center for Data Science, Zhejiang University(浙江大学数据科学中心);Centre for Statistics, Georg-August-Universität Göttingen(哥廷根大学统计中心)
Analysing Lightweight Large Language Models for Biomedical Named Entity Recognition on Diverse Ouput Formats
Inria, Inserm, Université Paris Cité, HeKA, UMR 1346, Paris, France(法国国家科学研究中心(Inria)、法国国家医学研究院(Inserm)、巴黎城市大学(Université Paris Cité)、HeKA、法国国家科研机构(UMR 1346)、巴黎);Télécom Paris, Institut Polytechnique de Paris, France(巴黎电信学院(Télécom Paris)、巴黎理工学院(Institut Polytechnique de Paris)、法国)
Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models
School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院);Suzhou Institute for Advanced Research, University of Science and Technology of China(中国科学技术大学苏州高等研究院)
FedNano: Toward Lightweight Federated Tuning for Pretrained Multimodal Large Language Models
LMU Munich(慕尼黑莱布尼茨大学);Technical University of Munich(慕尼黑技术大学);Siemens Technology(西门子技术);Munich Center for Machine Learning(慕尼黑机器学习中心);University Heidelberg(海德堡大学)
MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awareness
Hong Kong University of Science and Technology(香港科技大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);UMass Amherst(马萨诸塞大学阿姆赫斯特分校);William & Mary(威廉与玛丽学院)
Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark
New York University(纽约大学);New York University Abu Dhabi(纽约大学阿布扎比分校);Indian Institute of Technology Kanpur(印度理工学院坎普尔分校);International Institute of Information Technology Hyderabad(国际信息技术学院海得拉巴分校)
Origin Quantum Computing Company Limited(起源量子计算有限公司);Institute of Artificial Intelligence(人工智能研究院);Anhui Engineering Research Center of Quantum Computing(安徽量子计算工程研究中心);Laboratory of Quantum Information(量子信息实验室);University of Science and Technology of China(中国科学技术大学);National University of Singapore(新加坡国立大学);Anhui University(安徽大学)
Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation
School of Engineering Science, Simon Fraser University(西蒙弗雷泽大学工程科学学院);School of Information Science and Technology, Eastern Institute of Technology(东方理工学院信息科学与技术学院)
A Lightweight LLM Framework for Disaster Humanitarian Information Classification
Department of Civil, Architectural & Environment Engineering, Sungkyunkwan University(苏州市立大学土木、建筑与环境工程系);School of Geography, University of Leeds(利兹大学地理学院)
PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Innovation Center of Tsinghua University(清华大学创新中心);National University of Singapore(新加坡国立大学)
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Columbia University(哥伦比亚大学);Data Science Institute, Columbia University(哥伦比亚大学数据科学研究所);Department of Computer Science, Columbia University(哥伦比亚大学计算机科学系)
Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Renmin University of China(中国人民大学);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);School of Information, Renmin University of China(中国人民大学信息学院)
LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting
Zhejiang University(浙江大学);AZFT Joint Lab for Knowledge Engine(AZFT知识引擎联合实验室);Hangzhou Innovation Center, Zhejiang University(浙江大学杭州国际科创中心);Alibaba Group(阿里巴巴集团);State Key Laboratory of Media Convergence Production Technology and Systems(媒体融合生产技术与系统国家重点实验室)
Streamlining Forest Wildfire Surveillance: AI-Enhanced UAVs Utilizing the FLAME Aerial Video Dataset for Lightweight and Efficient Monitoring
Renmin University of China(中国人民大学);Shenzhen Institute of Artificial Intelligence and Robotics for Society(深圳市人工智能与机器人研究院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);International Research Institute of Disaster Science (IRIDeS), Tohoku University(东北大学灾害科学国际研究所)
Knowledge-Enhanced Semi-Supervised Federated Learning for Aggregating Heterogeneous Lightweight Clients in IoT
Pennsylvania State University(宾夕法尼亚州立大学);University of Electronic Science and Technology of China(电子科技大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Purdue University(普渡大学);Georgia State University(佐治亚州立大学)
PromptRad: Knowledge-Enhanced Multi-Label Prompt-Tuning for Low-Resource Radiology Report Labeling
Department of Artificial Intelligence and AI Research Center, Chang Gung University(人工智能系及AI研究中心,长庚大学);Department of Radiology, Sijhih Cathay General Hospital(放射科,西吉医院);Department of Medical Imaging and Intervention, Chang Gung Memorial Hospital(医学影像与介入科,长庚纪念医院);Department of Trauma and Emergency Surgery, Chang Gung Memorial Hospital(创伤与急诊外科,长庚纪念医院);Department of Computer Science, National Tsing Hua University(计算机科学系,国立清华大学)
FL-TAC: Enhanced Fine-Tuning in Federated Learning via Low-Rank, Task-Specific Adapter Clustering
Tsinghua-Berkeley Shenzhen Institute(清华-伯克利深圳学院);Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院);Shanghai AI Lab(上海人工智能实验室);RISC-V International Open Source Laboratory(RISC-V国际开源实验室);Ryerson University(瑞尔森大学)