Understanding what neural architectures can and cannot compute is a central challenge in the theory of AI. One of the fundamental problems in this context is the PARITY task, which asks whether the number of 1s in a binary input sequence is even or odd. PARITY is one of the central tasks studied in the theory of computation, yet it remains surprisingly unclear under which conditions transformers can or cannot solve it.
In this paper, we show that the minimal number of layers a transformer needs to compute PARITY is two. In particular, we solve the open problem asking whether a one-layer transformer can compute PARITY. We answer it negatively by showing that average sensitivity of a one-layer transformer grows slower than that of PARITY.
Furthermore, we show a new construction for transformer that computes PARITY, which improves on the existing constructions by removing a number of impractical assumptions. In particular, the existing transformers for PARITY rely on such impractical assumptions as length-dependent positional encoding, hardmax, layernorm without a regularisation parameter, or incompatibility with causal masking. We show that these assumptions can be removed, at the cost of increasing the number of layers from two to four. Specifically, we show that PARITY can be computed by a four-layer transformer using softmax attention, length-independent and polynomially bounded positional encoding, no layernorm, and compatible with both causal and non-causal masking.
A Transformer variant for multi-step forecasting of water level and hydrometeorological sensitivity analysis based on explainable artificial intelligence technology
School of Internet, Anhui University（安徽大学互联网学院）；National Engineering Research Center for Agro-Ecological Big Data Analysis & Application, Anhui University（安徽大学农业生态大数据分析与应用国家工程研究中心）；Forschungszentrum Jülich GmbH, Institute of Energy and Climate Research (IEK)（于利希研究中心能源与气候研究所）；Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）
Centro Nacional de Inteligencia Artificial（国家人工智能中心）；Instituto de Ingeniería Matemática y Computacional, Universidad Católica de Chile（智利天主教大学数学与计算工程研究所）；Institute of Fundamental Technological Research, Polish Academy of Sciences（波兰科学院基础技术研究所）
Structural Sensitivity in Compressed Transformers: Relative Error Propagation and Layer Removal
Indian Institute of Information Technology Allahabad, India（阿利哈巴德信息技术学院，印度）；National Institute of Electronics（电子研究所）；Indian Institute of Technology Patna, India（帕纳印度理工学院，印度）；Indian Institute of Information Technology Kalyani, India（卡里尼信息技术学院，印度）
