arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.13629cs.CCcs.FLcs.LG
收藏

统一TC$^0$中的Transformer

Transformers in Uniform TC$^0$

University of Notre Dame(圣母大学)

David Chiang

AI总结:

本文证明无需近似的AHATs及具有多项式位精度或极小绝对误差的SMATs均属于DLOGTIME-uniform TC$^0$,改进了此前依赖有限精度假设的电路复杂度上界结果。

摘要:

先前工作已表明,平均硬注意力Transformer(AHATs)和softmax注意力Transformer(SMATs)识别的语言属于电路复杂度类TC$^0$。然而,这些结果假设有限精度算术:使用O(log n)位浮点数(其中n是输入字符串的长度),Strobl证明AHATs可在L-uniform TC$^0$中近似,Merrill和Sabharwal证明SMATs可在DLOGTIME-uniform TC$^0$中近似。本文改进了这些结果,证明无需近似的AHATs、具有O(poly(n))位浮点精度的SMATs,以及绝对误差至多$2^{-O(poly(n))}$的SMATs均属于DLOGTIME-uniform TC$^0$。

英文摘要:

Previous work has shown that the languages recognized by average-hard attention transformers (AHATs) and softmax-attention transformers (SMATs) are within the circuit complexity class TC$^0$. However, these results assume limited-precision arithmetic: using floating-point numbers with O(log n) bits (where n is the length of the input string), Strobl showed that AHATs can be approximated in L-uniform TC$^0$, and Merrill and Sabharwal showed that SMATs can be approximated in DLOGTIME-uniform TC$^0$. Here, we improve these results, showing that AHATs with no approximation, SMATs with O(poly(n)) bits of floating-point precision, and SMATs with at most $2^{-O(poly(n))}$ absolute error are all in DLOGTIME-uniform TC$^0$.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑