扩散策略学习中的表达能力与统计权衡
Expressivity and Statistical Trade-offs in Diffusion Policy Learning
AI总结：
研究扩散策略学习中表达能力与统计权衡，确定漂移Lipschitz预算K为核心量，通过近似量化表达能力，证明其与统计复杂性的关系，给出有限样本性能差距，数值实验验证，还提出根据样本大小选K及对应神经网络架构的实现原则。
摘要：
基于扩散的策略最近成为强化学习中强大的策略参数化方法，将状态条件动作分布表示为具有参数化漂移的扩散过程的终端定律。这种终端定律表示在实践中显示出很大的表达灵活性，使扩散策略能够对复杂、多模态和高度非高斯的动作分布进行建模。然而，尚不清楚是什么在数学上驱动了这种表达能力，以及当从有限数据中学习策略时如何充分利用它。在本文中，我们将漂移Lipschitz预算K确定为控制扩散策略的表达能力和统计行为的核心量。我们通过近似来量化表达能力：具有K-Lipschitz漂移的扩散策略可以集中在接近最优确定性策略附近，并实现1/K阶的值近似误差；此外，我们在非退化扩散噪声下证明了一个匹配的下界。这种增加的表达能力伴随着统计成本。当漂移由神经网络参数化时，增加K可以提高近似效果，但会增加统计复杂性。平衡这两个项会导致一般神经网络漂移的有限样本性能差距为\(\tilde{O}(n^{-2/(m+6)})\)阶，对于单侧耗散漂移类则有更尖锐的速率\(\tilde{O}(n^{-2/(m+4)})\)，其中n是样本大小，m是状态空间的维度。数值实验为K中依赖样本的权衡提供了经验证据，支持了两种理论情况。我们的框架还提出了一个实际的实现原则：根据可用样本大小选择扩散预算K，然后选择具有相应固定Lipschitz系数的神经网络架构。
英文摘要：
Diffusion-based policies have recently emerged as powerful policy parameterizations for reinforcement learning, representing state-conditioned action distributions as terminal laws of diffusion processes with parameterized drifts. This terminal-law representation has shown substantial expressive flexibility in practice, enabling diffusion policies to model complex, multimodal, and highly non-Gaussian action distributions; however, it remains unclear what mathematically drives this expressivity and how to fully exploit it when the policy is learned from finite data. In this paper, we identify the drift Lipschitz budget $K$ as a central quantity governing the expressivity and statistical behavior of diffusion policies. We quantify expressivity through approximation: diffusion policies with $K$-Lipschitz drifts can concentrate near optimal deterministic policies and achieve value approximation error of order $1/K$; moreover, we prove a matching lower bound under nondegenerate diffusion noise. This increased expressivity comes with a statistical cost. When the drift is parameterized by neural networks, increasing $K$ improves approximation but increases statistical complexity. Balancing these two terms yields a finite-sample performance gap of order $\tilde{O}(n^{-2/(m+6)})$ for generic neural-network drifts, and a sharper rate $\tilde{O}(n^{-2/(m+4)})$ for one-sided dissipative drift classes, where $n$ is the sample size and $m$ is the dimension of the state space. Numerical experiments provide empirical evidence for the sample-dependent trade-off in $K$, supporting both theoretical regimes. Our framework also suggests a practical implementation principle: choose the diffusion budget $K$ according to the available sample size, and then select a neural-network architecture with the corresponding fixed Lipschitz coefficient.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
