arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.07501cs.LG
收藏

通过记忆与细粒度长尾实现可信机器学习:关于交互、权衡及超越的综述

Trustworthy Machine Learning via Memorization and the Granular Long-Tail: A Survey on Interactions, Tradeoffs, and Beyond

Aalborg University(奥尔堡大学);Politecnico di Milano(米兰理工大学)

Qiongxiu Li, Xiaoyu Luo, Yiyi Chen, Johannes Bjerva

AI总结:

本文综述了记忆在可信机器学习中的作用,提出类别不平衡、非典型性与噪声的三层长尾细粒度框架,以厘清记忆在公平性、鲁棒性和隐私中的双重作用并规划未来研究方向。

摘要:

记忆在机器学习(ML)中的作用已引起广泛关注,尤其是现代模型被经验性地观察到会记忆训练数据的片段。先前的理论分析,如 Feldman 的开创性工作,将记忆归因于训练数据中长尾分布的普遍存在,并证明对于位于分布尾部的样本而言,记忆是不可避免的。然而,记忆与可信机器学习研究的交叉揭示了关键空白。虽然此前可信机器学习中的记忆研究仅关注类别不平衡,但近期工作开始区分类别级稀有性与非典型样本,后者是有效且稀有的类内实例。然而,一个关键的研究空白仍然存在:当前框架将非典型样本与噪声和错误数据混为一谈,忽视了它们对公平性、鲁棒性和隐私的不同影响。在本工作中,我们对现有研究及其关于可信机器学习与记忆作用的发现进行了全面综述。更进一步,我们识别并指出了尚未探索的空白,并为这一研究方向提出了新的途径。由于现有的理论和实证分析缺乏细致性,无法厘清记忆作为必要性和负担的双重性,我们形式化了三层长尾细粒度——类别不平衡、非典型性和噪声——以揭示当前框架如何误用这些层级,从而延续有缺陷的解决方案。通过系统化这种细粒度,我们为未来研究绘制了路线图。可信机器学习必须调和在为了公平保障而记忆非典型性与为了鲁棒性和隐私保障而抑制噪声之间的微妙权衡。通过这种细粒度重新定义记忆,重塑了可信机器学习的理论基础,并进一步为那些使性能与社会信任保持一致性的模型提供了经验性前提。

英文摘要:

The role of memorization in machine learning (ML) has garnered significant attention, particularly as modern models are empirically observed to memorize fragments of training data. Previous theoretical analyses, such as Feldman's seminal work, attribute memorization to the prevalence of long-tail distributions in training data, proving it unavoidable for samples that lie in the tail of the distribution. However, the intersection of memorization and trustworthy ML research reveals critical gaps. While prior research in memorization in trustworthy ML has solely focused on class imbalance, recent work starts to differentiate class-level rarity from atypical samples, which are valid and rare intra-class instances. However, a critical research gap remains: current frameworks conflate atypical samples with noisy and erroneous data, neglecting their divergent impacts on fairness, robustness, and privacy. In this work, we conduct a thorough survey of existing research and their findings on trustworthy ML and the role of memorization. More and beyond, we identify and highlight uncharted gaps and propose new revenues in this research direction. Since existing theoretical and empirical analyses lack the nuances to disentangle memorization's duality as both a necessity and a liability, we formalize three-level long-tail granularity - class imbalance, atypicality, and noise - to reveal how current frameworks misapply these levels, perpetuating flawed solutions. By systematizing this granularity, we draw a roadmap for future research. Trustworthy ML must reconcile the nuanced trade-offs between memorizing atypicality for fairness assurance and suppressing noise for robustness and privacy guarantee. Redefining memorization via this granularity reshapes the theoretical foundation for trustworthy ML, and further affords an empirical prerequisite for models that align performance with societal trust.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑