统一TC$^0$中的Transformer
Transformers in Uniform TC$^0$
AI总结:
本文证明无需近似的AHATs及具有多项式位精度或极小绝对误差的SMATs均属于DLOGTIME-uniform TC$^0$,改进了此前依赖有限精度假设的电路复杂度上界结果。
摘要:
先前工作已表明,平均硬注意力Transformer(AHATs)和softmax注意力Transformer(SMATs)识别的语言属于电路复杂度类TC$^0$。然而,这些结果假设有限精度算术:使用O(log n)位浮点数(其中n是输入字符串的长度),Strobl证明AHATs可在L-uniform TC$^0$中近似,Merrill和Sabharwal证明SMATs可在DLOGTIME-uniform TC$^0$中近似。本文改进了这些结果,证明无需近似的AHATs、具有O(poly(n))位浮点精度的SMATs,以及绝对误差至多$2^{-O(poly(n))}$的SMATs均属于DLOGTIME-uniform TC$^0$。
英文摘要:
Previous work has shown that the languages recognized by average-hard attention transformers (AHATs) and softmax-attention transformers (SMATs) are within the circuit complexity class TC$^0$. However, these results assume limited-precision arithmetic: using floating-point numbers with O(log n) bits (where n is the length of the input string), Strobl showed that AHATs can be approximated in L-uniform TC$^0$, and Merrill and Sabharwal showed that SMATs can be approximated in DLOGTIME-uniform TC$^0$. Here, we improve these results, showing that AHATs with no approximation, SMATs with O(poly(n)) bits of floating-point precision, and SMATs with at most $2^{-O(poly(n))}$ absolute error are all in DLOGTIME-uniform TC$^0$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。