Supervised fine-tuning (SFT) relies critically on selecting training data that most benefits a model's downstream performance. Gradient-based data selection methods such as TracIn and Influence Functions leverage influence to identify useful samples, but their computational cost scales poorly, making them impractical for multi-billion-parameter large language models (LLMs). A common alternative is to use off-the-shelf smaller models as proxies, but they remain suboptimal since their learning dynamics are unclear, their sizes cannot be flexibly adjusted, and they cannot be further aligned with the target model in terms of gradient-based influence estimation. To address these challenges, we introduce Iprox, a two-stage framework that derives influence-preserving proxies directly from the target model. It first applies a low-rank compression stage to preserve influence information of the target model, and then an aligning stage to align both model gradients and logits, thereby constructing proxies that flexibly control computational cost while retaining the target model's influence. Experimental results across diverse LLM families and evaluation tasks show that Iprox consistently outperforms off-the-shelf proxies and baseline methods. On Qwen3-4B, a 1.5B proxy constructed with Iprox achieves stronger performance than the larger 1.7B off-the-shelf proxy. Notably, on Llama3.2, Iprox achieves better performance than baselines while reducing computational cost by more than half relative to the full 3B model. These results show that Iprox provides effective influence-preserving proxies, making gradient-based data selection more scalable for LLMs.
SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC
Pengcheng Laboratory（鹏城实验室）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Fudan University（复旦大学）；Shanghai Academy of AI for Science（上海人工智能科学研究院）；Institute of Statistical Interdisciplinary Research, Southwestern University of Finance and Economics（统计交叉学科研究所，西南财经大学）；National University of Singapore（新加坡国立大学）
Mila - Quebec AI Institute（魁北克AI研究所）；Polytechnique Montréal（蒙特利尔理工学院）；Université de Montréal（蒙特利尔大学）；McGill University（麦吉尔大学）；CIFAR AI Chair（CIFAR人工智能主席）；Google DeepMind（谷歌DeepMind）
Representation learning has been widely studied in the context of meta-learning, enabling rapid learning of new tasks through shared representations. Recent works such as MAML have explored using fine-tuning-based metrics, which measure the ease by which fine-tuning can achieve good performance, as proxies for obtaining representations. We present a theoretical framework for analyzing representations derived from a MAML-like algorithm, assuming the available tasks use approximately the same underlying representation. We then provide risk bounds on the best predictor found by fine-tuning via gr…
Federated learning (FL) has enabled global model training on decentralized data in a privacy-preserving way by aggregating model updates. However, for many natural language processing (NLP) tasks that utilize pre-trained language models (PLMs) with large numbers of parameters, there are considerable communication costs associated with FL. Recently, prompt tuning, which tunes some soft prompts without modifying PLMs, has achieved excellent performance as a new learning paradigm. Therefore we want to combine the two methods and explore the effect of prompt tuning under FL. In this paper, we prop…
FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
Shanghai Jiao Tong University（上海交通大学）；Queen's University Belfast（女王大学贝尔法斯特分校）；Stevens Institute of Technology（史蒂文斯理工学院）；Purdue University（普渡大学）；Rice University（里奇大学）
Fine-Tuning Foundation Models with Federated Learning for Privacy Preserving Medical Time Series Forecasting
University of Central Florida（中佛罗里达大学）；KnowledgeVis, LLC（诺维吉斯有限责任公司）；University of Michigan（密歇根大学）；University of California, Davis（加州大学戴维斯分校）；U.S. Army Institute of Surgical Research（美国陆军外科研究所）；University of Central Florida College of Medicine（中佛罗里达大学医学院）
The Energy Consumption of Transformer Fine-Tuning: A Roofline-Inspired Scaling Model
Université Lumière Lyon 2（里昂第二大学）；CNRS（法国国家科学研究中心）；Ecole Centrale de Lyon（里昂中央理工学院）；INSA Lyon（里昂国立应用科学学院）；Universite Claude Bernard Lyon 1（克洛德·贝尔纳里昂第一大学）；LIRIS, UMR5205（里昂信息图像与科学计算实验室，UMR5205）
Taming Noise-Induced Prototype Degradation for Privacy-Preserving Personalized Federated Fine-Tuning
School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；School of Statistics, Renmin University of China（中国人民大学统计学院）；School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）
Preserving Domain Generalization in Fine-Tuning via Joint Parameter Selection
School of Statistics and Data Science, KLMDASR, LEBPS, and LPMC, Nankai University（统计与数据科学学院，KLMDASR，LEBPS和LPMC，南开大学）；Image Processing Center, School of Astronautics, Beihang University（图像处理中心，航空学院，北航）；School of Computer Science and Technology, Tiangong University（计算机科学与技术学院，天工大学）
Preserving Diversity in Supervised Fine-Tuning of Large Language Models
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；Nanjing University（南京大学）；Hong Kong University of Science and Technology（香港科技大学）；University of Pennsylvania（宾夕法尼亚大学）
Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets
Guangdong Provincial Key Laboratory of Brain-Inspired Intelligent Computation, Department of CSE, SUSTech（脑启发智能计算广东省重点实验室，计算机科学与工程系，华南理工大学）；Department of CSE, HKUST（计算机科学与工程系，香港科技大学）；Department of CSE, PolyU（计算机科学与工程系，PolyU）；Department of CSE, SUSTech（计算机科学与工程系，华南理工大学）；Huawei Technologies CO.（华为技术有限公司）；College of Computing（计算学院）
Geometry Preserving Loss Functions Promote Improved Adaptation of Blackbox Generative Model
Geometric Media Lab（几何媒体实验室）；Arizona State University（亚利桑那州立大学）；Department of Statistics（统计学系）；Florida State University（佛罗里达州立大学）；Department of Computer Science and Engineering（计算机科学与工程系）；Seoul National University of Science and Technology（首尔科学技术大学）；Amazon（亚马逊）；Apple Inc.（苹果公司）
Transfer learning is important for foundation models to adapt to downstream tasks. However, many foundation models are proprietary, so users must share their data with model owners to fine-tune the models, which is costly and raise privacy concerns. Moreover, fine-tuning large foundation models is computation-intensive and impractical for most downstream users. In this paper, we propose Offsite-Tuning, a privacy-preserving and efficient transfer learning framework that can adapt billion-parameter foundation models to downstream data without access to the full model. In offsite-tuning, the mode…
FiberTune: Preserving Action-Fiber Visual Residuals in Vision-Language-Action Fine-Tuning
University of Chinese Academy of Sciences（中国科学院大学）；Hebei Key Laboratory of Cognitive Intelligence, Xiong’an Institute of Innovation（河北省认知智能重点实验室，雄安创新研究院）；Hebei University of Technology（河北工业大学）；Beijing Information Science and Technology University（北京信息科技大学）
