arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.13986cs.LG
收藏

序列模型中的符号脆性--符号数学中的系统推广

Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematics

University of Washington(华盛顿大学);Allen Institute for Artificial Intelligence(艾伦人工智能研究所)

Sean Welleck, Peter West, Jize Cao, Yejin Choi

摘要:

用最大似然估计训练的神经序列模型在许多任务中取得了突破,其中成功是由训练和测试性能之间的差距定义的。然而,它们实现更强形式的泛化的能力仍然不清楚。我们考虑的问题,符号数学积分,因为它需要系统地推广超出测试集。我们开发了一种方法来评估泛化,利用问题域的结构和访问验证。尽管有前途的序列到序列模型在这一领域的分布性能,我们表现出的挑战,实现鲁棒性,组合性,和分布外泛化,通过精心构造的手动测试套件和遗传算法,自动发现大量的失败在一个可控的方式。我们的调查强调了使用主流建模和学习方法进行泛化的困难,以及在泛化的不同方面进行测试集之外的评估的重要性。

英文摘要:

Neural sequence models trained with maximum likelihood estimation have led to breakthroughs in many tasks, where success is defined by the gap between training and test performance. However, their ability to achieve stronger forms of generalization remains unclear. We consider the problem of symbolic mathematical integration, as it requires generalizing systematically beyond the test set. We develop a methodology for evaluating generalization that takes advantage of the problem domain's structure and access to a verifier. Despite promising in-distribution performance of sequence-to-sequence models in this domain, we demonstrate challenges in achieving robustness, compositionality, and out-of-distribution generalization, through both carefully constructed manual test suites and a genetic algorithm that automatically finds large collections of failures in a controllable manner. Our investigation highlights the difficulty of generalizing well with the predominant modeling and learning approach, and the importance of evaluating beyond the test set, across different aspects of generalization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑