微调正被广泛用于在新的下游任务中发挥预训练基础模型的能力。尽管微调在各类任务上取得了诸多成功，但近期研究发现，微调模型在泛化到未见过的分布（即分布外，OOD）时存在挑战。为提升OOD泛化能力，以往部分研究指出了微调数据的局限性，并对微调过程加以约束，以保留从预训练数据中学到的通用表征。然而，预训练数据和模型中存在的潜在局限性往往被忽视。本文认为，过度依赖预训练表征可能会阻碍微调学习下游任务所需的关键表征，进而损害其OOD泛化能力。当新任务与预训练数据来自不同（子）领域时，这种影响可能尤为严重。为解决预训练数据与微调数据两方面的问题，我们提出了一种新型可泛化微调方法LEVI（不同视图的分层集成，Layer-wise Ensemble of different VIews），该方法在保持效率的同时，将预训练模型与一个小型任务特定模型进行分层自适应集成。通过结合两个互补的模型，LEVI有效抑制了微调数据和预训练模型中存在问题的特征，同时保留了对新任务有用的特征。针对大型语言模型和视觉模型的广泛实验表明，LEVI通过强调来自微调数据和预训练特征的不同视图，极大提升了微调的泛化能力。
英文摘要：
Fine-tuning is becoming widely used for leveraging the power of pre-trained foundation models in new downstream tasks. While there are many successes of fine-tuning on various tasks, recent studies have observed challenges in the generalization of fine-tuned models to unseen distributions (i.e., out-of-distribution; OOD). To improve OOD generalization, some previous studies identify the limitations of fine-tuning data and regulate fine-tuning to preserve the general representation learned from pre-training data. However, potential limitations in the pre-training data and models are often ignored. In this paper, we contend that overly relying on the pre-trained representation may hinder fine-tuning from learning essential representations for downstream tasks and thus hurt its OOD generalization. It can be especially catastrophic when new tasks are from different (sub)domains compared to pre-training data. To address the issues in both pre-training and fine-tuning data, we propose a novel generalizable fine-tuning method LEVI (Layer-wise Ensemble of different VIews), where the pre-trained model is adaptively ensembled layer-wise with a small task-specific model, while preserving its efficiencies. By combining two complementing models, LEVI effectively suppresses problematic features in both the fine-tuning data and pre-trained model and preserves useful features for new tasks. Broad experiments with large language and vision models show that LEVI greatly improves fine-tuning generalization via emphasizing different views from fine-tuning data and pre-trained features.
WISE-HAR: A Generalizable Ensemble Deep Learning Framework for WiFi-Based Human Activity Recognition
Department of Computing, School of Electrical Engineering and Computer Science（计算机系，电气工程与计算机科学学院）；National University of Sciences and Technology (NUST)（国家科学与技术大学（NUST））
Learning Spatio-Temporal Foundation Models from Pure Synthetic Data
Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；National University of Singapore（新加坡国立大学）；Hong Kong University of Science and Technology（香港科技大学）；Hong Kong Polytechnic University（香港理工大学）
Domain generalization (DG) serves as a promising solution to handle person Re-Identification (Re-ID), which trains the model using labels from the source domain alone, and then directly adopts the trained model to the target domain without model updating. However, existing DG approaches are usually disturbed by serious domain variations due to significant dataset variations. Subsequently, DG highly relies on designing domain-invariant features, which is however not well exploited, since most existing approaches directly mix multiple datasets to train DG based models without considering the loc…
ByteDance AI Lab（字节跳动人工智能实验室）；Harbin Institute of Technology（哈尔滨工业大学）
With the progress in AI-based facial forgery (i.e., deepfake), people are increasingly concerned about its abuse. Albeit effort has been made for training classification (also known as deepfake detection) models to recognize such forgeries, existing models suffer from poor generalization to unseen forgery technologies and high sensitivity to changes in image/video quality. In this paper, we advocate adversarial training for improving the generalization ability to both unseen facial forgeries and unseen image/video qualities. We believe training with samples that are adversarially crafted to at…
A level-wise training scheme for learning neural multigrid smoothers with application to integral equations
Hetao Institute of Mathematics and Interdisciplinary Sciences（河套数学与交叉科学研究所）；Hong Kong University of Science and Technology（香港科技大学）；Lingnan University（岭南大学）；University of Waterloo（多伦多大学）
Korea Advanced Institute of Science and Technology（韩国科学技术院）；University of California, Berkeley（加州大学伯克利分校）
Model-based reinforcement learning (RL) has shown great potential in various control tasks in terms of both sample-efficiency and final performance. However, learning a generalizable dynamics model robust to changes in dynamics remains a challenge since the target transition dynamics follow a multi-modal distribution. In this paper, we present a new model-based RL algorithm, coined trajectory-wise multiple choice learning, that learns a multi-headed dynamics model for dynamics generalization. The main idea is updating the most accurate prediction head to specialize each head in certain environ…
Learning More with Less: A Generalizable, Self-Supervised Framework for Privacy-Preserving Capacity Estimation with EV Charging Data
Engineering Product Development（工程产品开发）；Singapore University of Technology（新加坡科技设计大学）；School of Automation（自动化学院）；Chongqing University（重庆大学）；Design Singapore University of Technology（设计新加坡科技设计大学）；Computer Engineering（计算机工程）；Princeton University（普林斯顿大学）；Electronics Engineering（电子工程）；Nanyang Technological University（南洋理工大学）
From Time-series Generation, Model Selection to Transfer Learning: A Comparative Review of Pixel-wise Approaches for Large-scale Crop Mapping
College of Forest Resources and Environmental Science, Michigan Technological University（密歇根理工大学森林资源与环境科学学院）；Warnell School of Forestry and Natural Resources, University of Georgia（佐治亚大学沃内尔林业与自然资源学院）；Annis Water Resources Institute, Grand Valley State University（大峡谷州立大学安尼斯水资源研究所）；BioSense Institute（生物感知研究所）；Department of Agricultural, Food, and Resource Economics, Michigan State University（密歇根州立大学农业、食品与资源经济学系）
