解耦与蒸馏：面向少样本类增量学习的任务自适应LoRA教师与集成知识迁移
Decoupled and Distilled: Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer for Few-Shot Class-Incremental Learning
- Beihang University（北京航空航天大学）
- Beijing University of Posts and Telecommunications（北京邮电大学）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
TALON通过为每个增量任务分配独立LoRA教师并蒸馏至统一学生，实现推理高效的少样本类增量学习，在四个基准上取得领先精度且大幅降低部署参数与推理时间。
AI中文摘要：
少样本类增量学习（FSCIL）旨在解决从极少量样本中学习新类别同时保留已学类别知识的问题。尽管基于预训练模型的参数高效微调方法在类增量学习中展现出潜力，但在严重数据稀缺情况下，严格的基于梯度的约束可能不可靠，而多专家方法则会带来可观的推理时开销。我们提出了TALON（任务自适应LoRA教师与集成知识迁移），一个推理高效的FSCIL框架。TALON为每个增量任务动态分配独立的LoRA教师以进行任务特定表示学习，然后通过集成知识迁移将多个冻结的教师蒸馏到一个统一的LoRA学生中，从而消除了运行时的模块选择或生成。一种语义引导的蒸馏策略通过特征空间相似性对教师贡献进行加权，以减轻灾难性遗忘和过拟合。在三个类别顺序运行中，TALON在四个FSCIL基准上取得了相当或更好的平均准确率，在CUB200上达到86.68±1.22%，在CIFAR100上达到90.39±0.27%，在ImageNet-R上达到78.38±0.94%，在miniImageNet上达到96.34±0.33%。TALON使用的部署参数最多减少33倍，并将每个任务的平均推理时间降至26.7秒，相比ASP降低了41.70%。
英文摘要：
Few-Shot Class-Incremental Learning (FSCIL) addresses the challenge of learning new classes from very limited samples while retaining knowledge of previously learned ones. Although parameter-efficient fine-tuning methods with pre-trained models show promise for class-incremental learning, strict gradient-based constraints can be unreliable under severe data scarcity, while multi-expert approaches can impose substantial inference-time costs. We propose TALON (Task-Adaptive LoRA-Teachers with Ensemble Knowledge Transfer), an inference-efficient FSCIL framework. TALON dynamically allocates an independent LoRA-Teacher to each incremental task for task-specific representation learning, then distills multiple frozen teachers into a unified LoRA-Student through Ensemble Knowledge Transfer, eliminating runtime module selection or generation. A semantic-guided distillation strategy weights teacher contributions by feature-space similarity to mitigate catastrophic forgetting and overfitting. Across three class-order runs, TALON achieves comparable or better mean average accuracy across four FSCIL benchmarks, obtaining 86.68 +/- 1.22% on CUB200, 90.39 +/- 0.27% on CIFAR100, 78.38 +/- 0.94% on ImageNet-R, and 96.34 +/- 0.33% on miniImageNet. TALON uses up to 33x fewer deployment parameters and reduces average inference time per task to 26.7 s, a 41.70% reduction relative to ASP.
