潜意识时钟：扩散语言模型中的潜在时间建模
Subliminal Clocks: Latent Time Modelling in Diffusion Language Models
浏览论文内容
中文总结 AI 辅助
本文发现扩散语言模型在残差流中编码了与扩散时间步相关的潜在表示，可通过探针提取，并通过沿推断时间步的低维子空间引导来调节去噪进度，影响模型置信度和熵。
中文摘要 AI 辅助
扩散语言模型（DLM）最近已成为自回归模型的有前途的替代方案。与标准的基于扩散的方法不同，DLM 并不显式地以时间步为条件，这引发了一个自然的问题：这些模型是否在内部表示去噪进度，以及此类信息如何在下游使用？在这项工作中，我们表明 DLM 实际上在其残差流中编码了与扩散时间步相关的潜在表示。我们发现，可以使用跨层的探针可靠地提取该信号，表明去噪进度可以从内部激活中解码。我们进一步证明，沿着与推断时间步相关的低维子空间引导模型，可以系统地调节其对去噪进度的概念，导致模型置信度和熵的可预测变化。最后，我们分析了所识别表示的几何结构，表明它在激活空间中表现出结构化和可解释的特性，并揭示了这些模型如何处理此类信号。
英文摘要
Diffusion Language Models (DLMs) have recently emerged as a promising alternative to autoregressive models. Unlike standard diffusion-based approaches, DLMs are not explicitly conditioned on a timestep, raising a natural question: do these models internally represent denoising progress, and how is such information used downstream? In this work, we show that DLMs do in fact encode a latent representation related to the diffusion timestep within their residual streams. We find that this signal can be reliably extracted using probes across layers, indicating that denoising progress is decodable from internal activations. We further demonstrate that steering the model along a low-dimensional subspace associated with the inferred timestep allows us to systematically modulate its notion of denoising progress, leading to predictable changes in model confidence and entropy. Finally, we analyse the geometry of the identified representation, showing that it exhibits structured and interpretable properties in activation space, and shedding light on how such a signal is processed by these models.
发表机构
- Sapienza University of Rome（罗马大学）
- EPFL（洛桑联邦理工学院）
- Independent researcher（独立研究者）
机构由 AI 辅助整理，请以论文原文为准。
