Few-Medoids：一种用于少样本知识蒸馏的极其简单的核心集选择方法
Few-Medoids: An Embarrassingly Simple Coreset Selection Method for Few-Shot Knowledge Distillation
AI总结：
研究少样本知识蒸馏中核心集选择问题，提出简单的few-medoids策略，即选最接近类质心的样本，经多数据集和模型对实验，该方法能超越随机及其他策略，可替代常用基线。
摘要：
核心集选择旨在为高效模型训练识别大规模数据集的一个小而极具代表性的子集。即使在少样本知识蒸馏（KD）设置中，该问题仍具挑战性，其中全规模预训练教师为学生网络提供信息。典型的样本选择策略往往难以超越随机选择基线。本文展示了few-medoids，一种极其简单的核心集选择策略，它选择最接近每个类质心（平均图像）的样本。我们在四个涵盖广泛图像分类问题的数据集以及由卷积和变压器网络组成的三个师生模型对上进行了广泛的KD实验。尽管所提出的方法极其简单，但我们的实证结果表明，few-medoids能够持续超越随机选择基线以及其他核心集选择策略。因此，我们认为在未来核心集选择研究中，few-medoids可作为常用基线（如聚集或k中心贪心算法）的直接替代品。为重现报告结果，我们在这个https网址公开发布了我们的代码。
英文摘要：
Coreset selection aims to identify a small and highly representative subset of a massive dataset for efficient model training. The problem remains challenging even in the few-shot knowledge distillation (KD) setup, where a full-scale pre-trained teacher informs the student network. Typical sample selection strategies often struggle to surpass the random selection baseline. In this paper, we showcase few-medoids, an embarrassingly simple coreset selection strategy that chooses the samples closest to the centroid (average image) of each class. We present extensive KD experiments on four datasets, covering a wide range of image classification problems, and three teacher-student model pairs, comprising both convolutional and transformer networks. Although the proposed method is embarrassingly simple, our empirical results indicate that few-medoids is able to consistently surpass the random selection baseline, as well as the other coreset selection strategies. We therefore consider that few-medoids can be used as a drop-in replacement for commonly-used baselines (e.g. herding or k-center Greedy), in future research on coreset selection. To reproduce the reported results, we publicly release our code at https://github.com/CemilAndreiDilmac/Few-Shot-KD-Coreset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
