arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2511.18519cs.LG
收藏

CHIPS: 通过曲率感知的混合影响数据选择实现高效的CLIP适应

CHIPS: Efficient CLIP Adaptation via Curvature-aware Hybrid Influence-based Data Selection

Xinlin Zhuang, Yichen Li, Xiwei Liu, Haolin Yang, Yifan Lu, Ziyun Zou, Yulong Li, Huifa Li, Dongliang Chen, Qinglei Wang, Weiyang Liu, Ying Qian, Jiangming Shi, Imran Razzak

AI总结:

本文提出CHIPS方法,通过曲率感知和混合影响数据选择,实现高效的CLIP垂直领域适应,在医疗和通用领域基准中均表现出色。

摘要:

将CLIP适应到垂直领域通常通过新的微调策略或持续预训练(CPT)在大规模领域特定数据集上进行。然而,数据本身在这一过程中的作用仍被低估。我们从数据为中心的角度重新审视这一任务:有效数据选择能否替代大规模数据集在CPT中的作用?我们引入CHIPS(曲率感知混合影响投影子空间),为每个图像-文本对分配一个效用分数,整合了三个互补因素,以实现忠实、可扩展性和保留三个目标:通过曲率感知和牛顿式对齐在CLIP的终点子空间中实现忠实;通过信息NCE感知曲率估计器结合Johnson-Lindenstrauss(JL)降维;并通过选择感知的相关权重结合可学习性来平衡目标适应与通用领域保留。我们通过证明代理与全参数对齐的相关性下界保证,并通过曲率混合和JL降维引入的偏差-方差权衡进行理论证明。我们通过各种设置的实验证明:1)在17个医疗基准中,CHIPS在选择基线中达到最先进的性能,用30%的数据匹配全数据集CPT,仅用10%的数据就超越了半数据集CPT;2)在31个通用领域基准中,CHIPS在所有保留比例下都表现出最小的性能下降。

英文摘要:

Adapting CLIP to vertical domains is typically approached by novel fine-tuning strategies or by continual pre-training (CPT) on large domain-specific datasets. Yet, data itself remains an underexplored factor in this process. We revisit this task from a data-centric perspective: Can effective data selection substitute for large-scale datasets in CPT? We introduce CHIPS (Curvature-aware Hybrid Influence in Projection Subspace), which assigns each image-text pair a utility score that integrates three complementary factors aligned with three goals: faithfulness via a curvature-aware and Newton-style alignment computed in CLIP's end-point subspace; scalability via an InfoNCE-aware curvature estimator with Johnson-Lindenstrauss (JL) sketching; and retention via a selection-aware relevance weight combined with learnability to balance target adaptation against general-domain preservation. We justify this design theoretically by proving a lower-bound guarantee on the proxy's correlation with full-parameter alignment and by characterizing the bias-variance trade-offs introduced by curvature mixing and JL sketching. We evaluate CHIPS empirically across various settings: 1) CHIPS attains state-of-the-art performance among selection baselines on 17 medical benchmarks, matches full-dataset CPT with 30% of the data, and outperforms half-dataset CPT using only 10%; 2) on 31 general-domain benchmarks, CHIPS yields the least performance drop under all retention ratios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑