Almost Surely Stable Deep Dynamics

自适应 LQR 的几乎必然 $\sqrt{T}$ 遗憾

Almost Surely $\sqrt{T}$ Regret for Adaptive LQR

本文提出了一种具有几乎必然 $\tilde{\mathcal{O}}(\sqrt{T})$ 遗憾上界的自适应 LQR 控制器，其熔断机制可保障安全并保证参数估计收敛，且对渐近性能影响可忽略，经 TEP 仿真验证有效。

Densities of Almost Surely Terminating Probabilistic Programs are Differentiable Almost Everywhere

We study the differential properties of higher-order statistical probabilistic programs with recursion and conditioning. Our starting point is an open problem posed by Hongseok Yang: what class of statistical probabilistic programs have densities that are differentiable almost everywhere? To formalise the problem, we consider Statistical PCF (SPCF), an extension of call-by-value PCF with real numbers, and constructs for sampling and conditioning. We give SPCF a sampling-style operational semantics a la Borgstrom et al., and study the associated weight (commonly referred to as the density) func…

Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation

Satisfying safety constraints almost surely (or with probability one) can be critical for the deployment of Reinforcement Learning (RL) in real-life applications. For example, plane landing and take-off should ideally occur with probability one. We address the problem by introducing Safety Augmented (Saute) Markov Decision Processes (MDPs), where the safety constraints are eliminated by augmenting them into the state-space and reshaping the objective. We show that Saute MDP satisfies the Bellman equation and moves us closer to solving Safe RL with constraints satisfied almost surely. We argue …

论大型语言模型在推理时几乎必然安全的对齐

On Almost Surely Safe Alignment of Large Language Models at Inference-Time

本文提出InferenceGuard，一种基于约束马尔可夫决策过程的LLM推理时对齐方法，通过动态惩罚不安全生成实现几乎必然安全，无需修改权重，平衡安全与性能。

An Elementary Proof that Q-learning Converges Almost Surely

Watkins' and Dayan's Q-learning is a model-free reinforcement learning algorithm that iteratively refines an estimate for the optimal action-value function of an MDP by stochastically "visiting" many state-ation pairs [Watkins and Dayan, 1992]. Variants of the algorithm lie at the heart of numerous recent state-of-the-art achievements in reinforcement learning, including the superhuman Atari-playing deep Q-network [Mnih et al., 2015]. The goal of this paper is to reproduce a precise and (nearly) self-contained proof that Q-learning converges. Much of the available literature leverages powerful…

机器学习方法的“几乎必然”混沌性质

'Almost Sure' Chaotic Properties of Machine Learning Methods

先前已经证明，通用计算“几乎肯定”是混沌的。机器学习是计算不动点迭代的一种形式，在可计算函数空间上迭代。我们展示了这种迭代的一些性质，并建立一般的迭代是“几乎肯定”的混沌性质。该理论解释了深度学习方法的反直觉性质的观察。本文证明了这些性质将是任何学习方法的普遍性。