LCA-on-the-Line：基于类别分类法的分布外泛化基准测试
LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies
AI总结：
提出LCA-on-the-Line框架，利用类层次结构中的LCA距离基于分布内测量预测模型OOD性能，揭示了其与OOD准确率的强线性相关性，并证明对齐类别分类法可增强泛化。
摘要：
我们致力于解决在不依赖分布外（OOD）数据的情况下，利用分布内（ID）测量来预测模型分布外（OOD）性能的挑战。现有的使用“有效鲁棒性”（将ID准确率作为OOD准确率指标）的评估方法，在模型使用不同监督和分布（例如ImageNet上的类别标签视觉模型VMs，和LAION上的文本描述视觉-语言模型VLMs）进行训练时遇到了局限性。尽管具有相似或更低的ID性能，VLMs通常比VMs更好地泛化到OOD数据。为了改进从ID测量预测模型OOD性能，我们引入了Lowest Common Ancestor (LCA)-on-the-Line框架。该方法重新审视了LCA距离这一既定概念，即在预定义的类别层次结构（如WordNet）中测量标签与预测之间的层次距离。我们使用ImageNet作为ID数据集和五个显著偏移的OOD变体评估了75个模型，发现ID LCA距离与OOD top-1准确率之间存在强线性相关性。我们的方法为理解VLMs为何倾向于更好地泛化提供了令人信服的替代方案。此外，我们提出了一种使用K-means聚类在任何数据集上构建分类层次结构的技术，证明LCA距离对构建的分类层次结构具有鲁棒性。此外，我们证明通过软标签或提示工程使模型预测与类别分类法保持一致，可以增强模型的泛化能力。项目页面开源代码：https://elvishelvis.github.io/papers/lca/
英文摘要：
We tackle the challenge of predicting models' Out-of-Distribution (OOD) performance using in-distribution (ID) measurements without requiring OOD data. Existing evaluations with "Effective Robustness", which use ID accuracy as an indicator of OOD accuracy, encounter limitations when models are trained with diverse supervision and distributions, such as class labels (Vision Models, VMs, on ImageNet) and textual descriptions (Visual-Language Models, VLMs, on LAION). VLMs often generalize better to OOD data than VMs despite having similar or lower ID performance. To improve the prediction of models' OOD performance from ID measurements, we introduce the Lowest Common Ancestor (LCA)-on-the-Line framework. This approach revisits the established concept of LCA distance, which measures the hierarchical distance between labels and predictions within a predefined class hierarchy, such as WordNet. We assess 75 models using ImageNet as the ID dataset and five significantly shifted OOD variants, uncovering a strong linear correlation between ID LCA distance and OOD top-1 accuracy. Our method provides a compelling alternative for understanding why VLMs tend to generalize better. Additionally, we propose a technique to construct a taxonomic hierarchy on any dataset using K-means clustering, demonstrating that LCA distance is robust to the constructed taxonomic hierarchy. Moreover, we demonstrate that aligning model predictions with class taxonomies, through soft labels or prompt engineering, can enhance model generalization. Open source code in our Project Page: https://elvishelvis.github.io/papers/lca/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
