arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.18548cs.LOcs.AIcs.CCcs.LG
收藏

Transformer 编码器可满足性:复杂度及其对形式推理的影响

Transformer Encoder Satisfiability: Complexity and Impact on Formal Reasoning

University of Kaiserslautern-Landau, RPTU Kaiserslautern(凯撒斯劳滕-兰道大学,RPTU凯撒斯劳滕);University of Kassel(卡塞尔大学)

Marco Sälzer, Eric Alsmann, Martin Lange

AI总结:

本文分析 Transformer 编码器可满足性问题的复杂度,证明一般情形下不可判定,而量化 TE 可判定但具 NEXPTIME 难度,并探讨其对形式推理的影响。

摘要:

我们分析了 Transformer 编码器(TE)的可满足性问题(或类似的可行性问题,记为 trSAT)的复杂度,该问题在形式验证或解释中自然出现,统称为形式推理。我们发现,当考虑表达性社区中通常研究的 TE 时,trSAT 是不可判定的。此外,我们识别了 trSAT 可判定的实际场景,并建立了相应的复杂度界限。除平凡情况外,我们发现量化 TE(即受固定宽度算术限制的 TE)因其有限的注意力能力而导致 trSAT 的可判定性。然而,该问题仍然困难,因为我们建立了量化 TE 的 trSAT 为 NEXPTIME-难以及可在 NEXPTIME 内求解的场景。为补充我们的复杂度结果,我们将我们的发现及其含义置于形式推理的更广泛背景下。

英文摘要:

We analyse the complexity of the satisfiability problem, or similarly feasibility problem, (trSAT) for transformer encoders (TE), which naturally occurs in formal verification or interpretation, collectively referred to as formal reasoning. We find that trSAT is undecidable when considering TE as they are commonly studied in the expressiveness community. Furthermore, we identify practical scenarios where trSAT is decidable and establish corresponding complexity bounds. Beyond trivial cases, we find that quantized TE, those restricted by fixed-width arithmetic, lead to the decidability of trSAT due to their limited attention capabilities. However, the problem remains difficult, as we establish scenarios where trSAT is NEXPTIME-hard and others where it is solvable in NEXPTIME for quantized TE. To complement our complexity results, we place our findings and their implications in the broader context of formal reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑