arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.06120cs.CL
收藏

探索大语言模型中的群与对称性原理

Exploring Group and Symmetry Principles in Large Language Models

Microsoft Research(微软研究院)

Shima Imani, Hamid Palangi

AI总结:

本文提出基于群与对称性原理的框架,通过封闭性、单位元、逆元和结合律四类算术测试评估大语言模型的推理能力,揭示其在保持群性质上的不足并发布合成数据集。

摘要:

大语言模型(LLMs)在广泛的应用中展现了令人印象深刻的表现;然而,评估其推理能力仍然是一个重大挑战。在本文中,我们引入了一个基于群与对称性原理的框架,这些原理在物理学和数学等领域中发挥了关键作用,并提供了另一种评估其能力的方式。虽然所提出的框架具有通用性,但为了展示利用这些性质的好处,我们聚焦于算术推理,并研究这些模型在四个群性质上的表现:封闭性、单位元、逆元和结合律。我们的研究结果表明,本文所研究的LLMs在不同的测试机制下难以保持群性质。在封闭性测试中,我们观察到对特定输出的偏置,以及其在特定序列长度之后性能从100%骤降至0%。它们在单位元测试中也表现不佳,该测试表示在上下文中添加不相关信息,并且在接受逆元测试时表现出敏感性,该测试考察模型对否定的鲁棒性。此外,我们证明了将问题分解为更小的步骤有助于LLMs通过我们所进行的结合律测试。为了支持这些测试,我们开发了一个合成数据集,并将予以发布。

英文摘要:

Large Language Models (LLMs) have demonstrated impressive performance across a wide range of applications; however, assessing their reasoning capabilities remains a significant challenge. In this paper, we introduce a framework grounded in group and symmetry principles, which have played a crucial role in fields such as physics and mathematics, and offer another way to evaluate their capabilities. While the proposed framework is general, to showcase the benefits of employing these properties, we focus on arithmetic reasoning and investigate the performance of these models on four group properties: closure, identity, inverse, and associativity. Our findings reveal that LLMs studied in this work struggle to preserve group properties across different test regimes. In the closure test, we observe biases towards specific outputs and an abrupt degradation in their performance from 100% to 0% after a specific sequence length. They also perform poorly in the identity test, which represents adding irrelevant information in the context, and show sensitivity when subjected to inverse test, which examines the robustness of the model with respect to negation. In addition, we demonstrate that breaking down problems into smaller steps helps LLMs in the associativity test that we have conducted. To support these tests we have developed a synthetic dataset which will be released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑