APAM: 面向噪声标签与长尾学习的语言模型自适应预训练与自适应元学习
APAM: Adaptive Pre-training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-tailed Learning
AI总结：
针对NLP任务中长尾分布与噪声标签问题，提出结合对比学习自适应预训练、元数据驱动重加权及多项式损失调整的APAM框架，实验证明其性能优于基线，可有效提升模型鲁棒性与泛化性。
摘要：
实际自然语言处理（NLP）任务普遍存在长尾分布且伴随噪声标签的问题，这些问题对深度神经网络（DNNs）等复杂模型的泛化性与鲁棒性构成挑战。过采样、欠采样等常用重采样技术极易导致过拟合，利用少量元数据学习数据权重的方法日益流行。此外，已有研究表明自监督预训练尤其是对样本量不足的数据具有优势。本文提出一个通用框架，同时解决长尾与噪声标签问题：模型通过对比学习方式适配问题领域；重加权模块为前馈网络，可学习显式加权函数并根据元数据调整权重；框架还结合交叉熵损失与焦点损失（focal loss）的多项式展开，进一步调整损失函数中各项的权重。大量实验表明，所提框架性能始终优于基线方法。敏感性分析进一步验证了该框架处理长尾问题、缓解噪声标签负面影响的能力。
英文摘要：
Practical natural language processing (NLP) tasks are commonly long-tailed with noisy labels. Those problems challenge the generalization and robustness of complex models such as Deep Neural Networks (DNNs). Some commonly used resampling techniques, such as oversampling or undersampling, could easily lead to overfitting. It is growing popular to learn the data weights leveraging a small amount of metadata. Besides, recent studies have shown the advantages of self-supervised pre-training, particularly to the under-represented data. In this work, we propose a general framework to handle the problem of both long-tail and noisy labels. The model is adapted to the domain of problems in a contrastive learning manner. The re-weighting module is a feed-forward network that learns explicit weighting functions and adapts weights according to metadata. The framework further adapts weights of terms in the loss function through a combination of the polynomial expansion of cross-entropy loss and focal loss. Our extensive experiments show that the proposed framework consistently outperforms baseline methods. Lastly, our sensitive analysis emphasizes the capability of the proposed framework to handle the long-tailed problem and mitigate the negative impact of noisy labels.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
