多模态机器去学习中的度量不可靠性：系统分析与原则统一分数
Validity-Gated Evaluation of Multimodal Machine Unlearning
AI总结：
本文系统分析了多模态去学习中度量可靠性问题，提出统一质量评分（UQS）以解决不同度量标准间的冲突，通过实验验证了UQS在稳定排序方面的有效性。
摘要：
在视觉-语言模型（VLMs）中，机器去学习对于遵守通用数据保护条例（GDPR）是必要的，但当前的评估实践却存在不一致。我们首次系统研究了多模态去学习中的度量可靠性。五个标准度量（Forget Accuracy, Retain Accuracy, Membership Inference Attack, Activation Distance, JS divergence）在三个VQA基准（MLLMU-Bench, UnLOK-VQA, MMUBench）上产生冲突的方法排名。对36个去学习的LLaVA-1.5-7B模型进行Kendall tau分析，揭示出两个对立的聚类，{FA, RA, MIA}和{AD, JS}，tau_FA_AD = -0.26，在BLIP-2 OPT-2.7B上复现。在多模态VQA中的一致性较低（平均tau = 0.086）比在单模态分类中（平均tau = 0.158；差异 = 0.072），表明双图像-文本路径放大了不一致性。我们引入了统一质量评分（UQS），一个综合度量，其权重来源于每个度量与Oracle距离d(M_hat, M_star)的Spearman相关性，其中M_star是仅在保留集上重新训练的Oracle模型。RA显示最强的可靠性（rho = 0.484, p = 0.003），而FA是负相关（rho = -0.418, p = 0.011）。UQS在100次随机权重扰动下产生稳定的排名（tau = 0.647 ± 0.262）。我们发布了基准、36个检查点和一个交互式排行榜。代码和预计算结果可在https://github.com/neurips26/UnifiedUnl获取。
英文摘要：
Machine unlearning aims to remove the influence of training data without retraining a model from scratch. Evaluation of vision-language models relies on forget accuracy (FA), retain accuracy (RA), membership inference, and similarity to a retain-only reference, but evaluation pipelines are rarely tested. We propose a validity-gated unlearning audit (VGUA) with three checks: the forget set must be memorized without duplicate leakage into the retain set, the scorer must measure correctness, and the training code must implement the intended procedure. Applying these checks reveals that random MMBench splits leak duplicate questions, a lenient scorer hides memorization, and two coding errors remove the signal while training completes. After correcting these problems, we evaluate 234 unlearned models using six methods, three learning rates, and three backbones. Each model is compared with a retain-only reference trained using low-rank adaptation (LoRA). The results depend on update strength. Among models with RA at or above 0.70, lower FA generally indicates closer agreement with the retrained FA-RA operating point. Aggressive updates overshoot this point, collapse RA, and reverse method rankings. Relative to the original model, 78% of healthy models move closer to the retrained model's scores, whereas 94% move farther from it in activation space. Membership-inference advantage does not reliably track retraining alignment. Under option-shuffled training, unlearned models answer 25% of reordered forgotten ScienceQA questions, compared with 6% for the retain-only reference. These results show that multimodal unlearning conclusions depend on pipeline validity, update strength, and the comparison criterion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
