RLSbench：松弛标签偏移下的域适应
RLSbench: Domain Adaptation Under Relaxed Label Shift
AI总结：
本文构建涵盖视觉、表格和语言的 RLSbench 松弛标签偏移基准，评估13种域适应方法，并提出可兼容多数启发式方法的伪平衡与分类器调整两步元算法。
摘要：
尽管标签偏移下的域适应已经出现了一些有原则的方法，但这些方法对类别条件分布变化的敏感性仍未得到充分研究，这种状况令人担忧。与此同时，流行的深度域适应启发式方法在面对标签比例变化时往往会失效。虽然已有若干论文对这些启发式方法进行修改，以尝试处理标签比例变化，但评估标准、数据集和基线之间的不一致，使得人们难以判断当前的最佳实践。本文提出 RLSbench，这是一个面向松弛标签偏移的大规模基准，包含跨越视觉、表格和语言模态的 500 多个分布偏移对，并具有不同的标签比例。不同于主要关注类别条件 p(x|y) 变化的现有基准，我们的基准还关注标签边缘分布变化。首先，我们评估了 13 种流行的域适应方法，证明标签比例变化下的失败比此前已知的更为普遍。接着，我们开发了一种有效的两步元算法，它可与大多数域适应启发式方法兼容：（i）在每个 epoch 对数据进行伪平衡；（ii）利用目标标签分布估计调整最终分类器。该元算法在较大标签比例变化下改进了现有域适应启发式方法，准确率通常提升 2—10 个百分点，而在标签比例不发生变化时影响极小（小于 0.5%）。我们希望这些发现以及 RLSbench 的公开可用能够鼓励研究人员在松弛标签偏移设置中严格评估所提出的方法。代码公开发布于 https://github.com/acmi-lab/RLSbench。
英文摘要：
Despite the emergence of principled methods for domain adaptation under label shift, their sensitivity to shifts in class conditional distributions is precariously under explored. Meanwhile, popular deep domain adaptation heuristics tend to falter when faced with label proportions shifts. While several papers modify these heuristics in attempts to handle label proportions shifts, inconsistencies in evaluation standards, datasets, and baselines make it difficult to gauge the current best practices. In this paper, we introduce RLSbench, a large-scale benchmark for relaxed label shift, consisting of $>$500 distribution shift pairs spanning vision, tabular, and language modalities, with varying label proportions. Unlike existing benchmarks, which primarily focus on shifts in class-conditional $p(x|y)$, our benchmark also focuses on label marginal shifts. First, we assess 13 popular domain adaptation methods, demonstrating more widespread failures under label proportion shifts than were previously known. Next, we develop an effective two-step meta-algorithm that is compatible with most domain adaptation heuristics: (i) pseudo-balance the data at each epoch; and (ii) adjust the final classifier with target label distribution estimate. The meta-algorithm improves existing domain adaptation heuristics under large label proportion shifts, often by 2--10\% accuracy points, while conferring minimal effect ($<$0.5\%) when label proportions do not shift. We hope that these findings and the availability of RLSbench will encourage researchers to rigorously evaluate proposed methods in relaxed label shift settings. Code is publicly available at https://github.com/acmi-lab/RLSbench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
