平均硬注意力Transformer是常数深度均匀阈值电路
Average-Hard Attention Transformers are Constant-Depth Uniform Threshold Circuits
AI总结：
该研究拓展了平均硬注意力Transformer与常数深度阈值电路的关联结论，证明其对应的电路结果也可扩展为均匀电路，完善了Transformer的计算复杂度理论分析。
摘要：
Transformer已成为各类自然语言处理任务中广泛使用的神经网络模型。既往研究探索了其与常数深度阈值电路的关联，作出两项假设：平均硬注意力，以及内部计算相对于输入长度的对数精度。Merrill等人（2022）证明，平均硬注意力Transformer可识别的语言属于复杂度类TC0，即能被常数深度、多项式规模阈值电路识别的语言集合。同样，Merrill与Sabharwal（2023）表明，对数精度Transformer可识别的语言属于均匀TC0类。这说明两种Transformer模型都可被常数深度阈值电路模拟，后者因能生成均匀电路族而更具鲁棒性。本文证明，第一项结果也可扩展得到均匀电路。
英文摘要：
Transformers have emerged as a widely used neural network model for various natural language processing tasks. Previous research explored their relationship with constant-depth threshold circuits, making two assumptions: average-hard attention and logarithmic precision for internal computations relative to input length. Merrill et al. (2022) prove that average-hard attention transformers recognize languages that fall within the complexity class TC0, denoting the set of languages that can be recognized by constant-depth polynomial-size threshold circuits. Likewise, Merrill and Sabharwal (2023) show that log-precision transformers recognize languages within the class of uniform TC0. This shows that both transformer models can be simulated by constant-depth threshold circuits, with the latter being more robust due to generating a uniform circuit family. Our paper shows that the first result can be extended to yield uniform circuits as well.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
