Semantically meaningful information content in perceptual signals is usually unevenly distributed. In speech signals for example, there are often many silences, and the speed of pronunciation can vary considerably. In this work, we propose slow autoencoders (SlowAEs) for unsupervised learning of high-level variable-rate discrete representations of sequences, and apply them to speech. We show that the resulting event-based representations automatically grow or shrink depending on the density of salient information in the input signals, while still allowing for faithful signal reconstruction. We develop run-length Transformers (RLTs) for event-based representation modelling and use them to construct language models in the speech domain, which are able to generate grammatical and semantically coherent utterances and continuations.
Learning Discrete Latent Variable Structures with Tensor Rank Conditions
School of Computer Science, Guangdong University of Technology（广东工业大学计算机学院）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Beijing Technology and Business University（北京工商大学）；Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
Google Brain（谷歌大脑）；DeepMind；University of Oxford（牛津大学）
由于高方差梯度估计量，具有离散潜变量的模型中的学习具有挑战性。通常，方法依赖于控制变量来减少REINFORCE估计量的方差。最近的工作（Jang et al. 2016，Maddison et al. 2016）采取了不同的方法，引入了离散变量的连续松弛来产生低方差但有偏的梯度估计。在这项工作中，我们结合了这两种方法，通过一种新的控制变量，产生低方差，\emph{unbiased}梯度估计。然后，我们介绍了一个修改的连续松弛，并表明放松的紧密度可以在线适应，删除它作为一个超参数。我们在几个基准生成建模任务上展示了最先进的方差减少，通常会更快地收敛到更好的最终对数似然。
