正凹深度平衡模型
Positive concave deep equilibrium models
AI总结：
针对现有深度平衡模型不动点存在唯一性及收敛性缺乏理论保证、实际应用不稳定的问题，提出基于非线性Perron-Frobenius理论的正凹深度平衡模型，在简化训练的同时实现了与其他隐式模型相当的性能。
摘要：
深度平衡（DEQ）模型是公认的标准神经网络的内存高效替代方案，在语言建模和计算机视觉任务中达到了当前最优性能。这类模型通过求解不动点方程而非显式计算输出，这是它们与标准神经网络的区别所在。然而，现有DEQ模型通常缺乏不动点存在性与唯一性的正式保证，且用于计算不动点的数值方案的收敛性也未得到正式确立，因此DEQ模型在实际应用中存在潜在的不稳定性。为解决这些缺陷，我们提出了一类新型DEQ模型，称为正凹深度平衡（pcDEQ）模型。我们的方法基于非线性Perron-Frobenius理论，强制要求非负权重以及在正卦限上呈凹性的激活函数。通过施加这些约束，我们无需依赖DEQ文献中常见的额外复杂假设（如凸分析中基于单调算子理论的假设），即可轻松确保不动点的存在性与唯一性。此外，该不动点可通过标准不动点算法计算，我们还提供了其几何收敛的理论保证，这尤其简化了训练过程。实验表明，我们的pcDEQ模型与其他隐式模型相比具有竞争力。
英文摘要：
Deep equilibrium (DEQ) models are widely recognized as a memory efficient alternative to standard neural networks, achieving state-of-the-art performance in language modeling and computer vision tasks. These models solve a fixed point equation instead of explicitly computing the output, which sets them apart from standard neural networks. However, existing DEQ models often lack formal guarantees of the existence and uniqueness of the fixed point, and the convergence of the numerical scheme used for computing the fixed point is not formally established. As a result, DEQ models are potentially unstable in practice. To address these drawbacks, we introduce a novel class of DEQ models called positive concave deep equilibrium (pcDEQ) models. Our approach, which is based on nonlinear Perron-Frobenius theory, enforces nonnegative weights and activation functions that are concave on the positive orthant. By imposing these constraints, we can easily ensure the existence and uniqueness of the fixed point without relying on additional complex assumptions commonly found in the DEQ literature, such as those based on monotone operator theory in convex analysis. Furthermore, the fixed point can be computed with the standard fixed point algorithm, and we provide theoretical guarantees of its geometric convergence, which, in particular, simplifies the training process. Experiments demonstrate the competitiveness of our pcDEQ models against other implicit models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
