arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.13504cs.CL
收藏

IMO:用于预训练模型分布外文本分类的贪心逐层稀疏表示学习

IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models

Monash University(莫纳什大学)

Tao Feng, Lizhen Qu, Zhuang Li, Haolan Zhan, Yuncheng Hua, Gholamreza Haffari

AI总结:

针对跨未见领域的分布外文本分类问题,提出IMO方法,通过学习稀疏掩码层移除无关特征并保留不变特征,结合token级注意力模块,显著提升OOD泛化性能。

摘要:

机器学习模型取得了惊人的进展,但在应用于未见领域样本时仍面临挑战。本研究聚焦于领域泛化的特定问题,即模型在一个源领域上训练,并在训练期间未见的多个目标领域上进行测试。我们提出了IMO(Invariant features Masks for Out-of-Distribution text classification,用于分布外文本分类的不变特征掩码),通过学习不变特征来实现OOD泛化。在训练期间,IMO会学习稀疏掩码层以移除与预测无关的特征,其中保留的特征保持不变。此外,IMO在token级别具有注意力模块,以专注于对预测有用的token。我们的综合实验表明,IMO在各种评估指标和设置下均显著优于强基线。

英文摘要:

Machine learning models have made incredible progress, but they still struggle when applied to examples from unseen domains. This study focuses on a specific problem of domain generalization, where a model is trained on one source domain and tested on multiple target domains that are unseen during training. We propose IMO: Invariant features Masks for Out-of-Distribution text classification, to achieve OOD generalization by learning invariant features. During training, IMO would learn sparse mask layers to remove irrelevant features for prediction, where the remaining features keep invariant. Additionally, IMO has an attention module at the token level to focus on tokens that are useful for prediction. Our comprehensive experiments show that IMO substantially outperforms strong baselines in terms of various evaluation metrics and settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑