arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.12997cs.LGcs.AI
收藏

Ehrenfeucht-Haussler秩与思维链

Ehrenfeucht-Haussler Rank and Chain of Thought

National Center for Artificial Intelligence (CENIA Chile)(国家人工智能中心(智利CENIA));Millennium Institute for Foundational Research on Data (IMFD Chile)(数据基础研究千年研究所(智利IMFD));Institute for Mathematical and Computational Engineering, Pontifical Catholic University of Chile(数学与计算工程研究所,智利天主教大学);Institute of Fundamental Technological Research, Polish Academy of Sciences(基础技术研究所,波兰科学院)

Pablo Barceló, Alexander Kozachinskiy, Tomasz Steifer

AI总结:

本文基于Transformer架构提出布尔函数秩的新颖刻画,证明秩等于单层硬注意力Transformer计算该函数所需的最小思维链步数,并确立了特定问题的紧致界及多头秩函数类的PAC可学习性。

摘要:

布尔函数的秩概念一直是PAC学习理论的基石,为多项式规模的决策树启用了拟多项式时间学习算法。我们提出了一种基于著名Transformer架构的秩的新颖刻画。我们证明,函数$f$的秩对应于具有硬注意力的单层Transformer计算$f$所需的最小思维链(CoT)步数。基于此刻画,我们确立了特定问题所需CoT步数的紧致界,证明\\(\ell\\)重函数复合恰好需要\\(\ell\\)个CoT步数。此外,我们分析了识别布尔序列中第\\(k\\)个1出现位置的问题,证明其需要\\(k\\)个CoT步数。最后,我们引入了多头秩的概念以刻画多头单层Transformer,并对具有有界多头秩的函数类的PAC可学习性进行了分析。

英文摘要:

The notion of \emph{rank} of a Boolean function has been a cornerstone in PAC learning theory, enabling quasipolynomial-time learning algorithms for polynomial-size decision trees. We present a novel characterization of rank, grounded in the well-known Transformer architecture. We show that the rank of a function $f$ corresponds to the minimum number of \emph{Chain of Thought} (CoT) steps required by a single-layer Transformer with hard attention to compute $f$. Based on this characterization we establish tight bounds on the number of CoT steps required for specific problems, showing that \(\ell\)-fold function composition necessitates exactly \(\ell\) CoT steps. Furthermore, we analyze the problem of identifying the position of the \(k\)-th occurrence of 1 in a Boolean sequence, proving that it requires \(k\) CoT steps. Finally, we introduce the notion of the multi-head rank that captures multi-head single-layer transformers, and perform the analysis of PAC-learnability of the classes of functions with bounded multi-head rank.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑