探究在虚假相关性下的子任务蒸馏鲁棒性
Investigating the Robustness of Subtask Distillation under Spurious Correlation
AI总结：
本文研究了在存在虚假相关性的数据下，子任务蒸馏方法的鲁棒性，发现先进方法如SubDistill仍保持稳健，而基础方法性能显著下降。
摘要：
子任务蒸馏是一种新兴范式，通过从大型通用'基础模型'中提取紧凑、专用的模型，用于资源受限环境或独立计算机系统中的部署。尽管蒸馏使用教师模型，但它仍依赖于通常规模有限且可能缺乏代表性的数据集，或表现出虚假相关性。在本文中，我们评估了现有蒸馏方法以及最近的SubDistill方法在使用具有虚假相关性的数据进行蒸馏时的表现。随着相关性的强度增加，我们观察到先进方法，如SubDistill，仍保持相对稳健，而一些基础方法则退化到接近随机性能。总体而言，我们的研究突显了在不完美、现实世界数据集上应用知识蒸馏的挑战，特别是那些存在虚假相关性的数据集。
英文摘要：
Subtask distillation is an emerging paradigm in which compact, specialized models are extracted from large, general-purpose 'foundation models' for deployment in environments with limited resources or in standalone computer systems. Although distillation uses a teacher model, it still relies on a dataset that is often limited in size and may lack representativeness or exhibit spurious correlations. In this paper, we evaluate established distillation methods, as well as the recent SubDistill method, when using data with spurious correlations for distillation. As the strength of the correlations increases, we observe a widening gap between advanced methods, such as SubDistill, which remain fairly robust, and some baseline methods, which degrade to near-random performance. Overall, our study underscores the challenges of knowledge distillation when applied to imperfect, real-world datasets, particularly those with spurious correlations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
