arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.22964cs.LG
收藏

在最小过参数化下,从示例中认证对于电路和Transformer是困难的

Certification from Examples is Hard for Circuits and Transformers under Minimal Overparametrization

University of Waterloo(滑铁卢大学)

Artur Back de Luca, Kimon Fountoulakis

AI总结:

本文证明在最小过参数化下,对于深度≥2的阈值电路和对数精度Transformer,认证所需示例集大小可能呈指数增长,并探讨了近似认证的难度。

摘要:

随着最先进的神经网络被部署在推理和算法任务上,精确性保证变得越来越重要。然而,高平均准确率仍可能掩盖不一致的行为。这激发了精确认证的需求,即寻找最小的标记示例集,以证明学习到的假设与目标一致。我们表明,虽然某些假设易于认证,但即使是最小的过参数化,也可能使多个假设类别的认证变得指数级困难。对于深度≥2的阈值电路,添加一个额外的门就可能导致认证集大小在输入维度上呈指数增长。我们展示了对于仅具有恒定架构开销的对数精度Transformer,存在类似的困难结果。我们还刻画了近似认证,表明允许多项式数量的错误仍然需要指数级大小的证书,而常数相对误差保证可能隐藏指数级数量的错误。实验上,我们研究了用于识别二进制加法的构造电路和训练后的Transformer的认证。虽然构造电路实例化了认证的指数障碍,但训练后的Transformer分析表明,不完美的模型可以通过大的均匀采样候选证书来逃避检测。

英文摘要:

As state-of-the-art neural networks are deployed on reasoning and algorithmic tasks, exactness guarantees become increasingly important. However, high average-case accuracy can still mask incorrect behaviors. This motivates exact certification, which asks for the smallest set of labeled examples needed to certify that a learned hypothesis equals the target. We show that while some hypotheses are easy to certify, even minimal overparametrization can make certification exponentially hard across several hypothesis classes. For threshold circuits of depth $\ge 2$, adding a single extra gate can force certificate sizes exponential in the input dimension. We show an analogous hardness result for log-precision Transformers with only constant architectural overhead. We also characterize approximate certification, showing that allowing only polynomially many mistakes still requires exponentially large certificates, whereas constant relative-error guarantees can hide exponentially many failures. Empirically, we study binary-addition recognition using threshold circuits and Transformers trained with strong supervision. We find that incorrect models can survive extensive checks, with errors that often occur on different inputs and are difficult to detect through uniform sampling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑