arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17691cs.LGcs.CL
收藏

tethered Reasoning: 通过流形引导解耦熵与幻觉在量化大语言模型中

Tethered Reasoning: Decoupling Entropy from Hallucination in Quantized LLMs via Manifold Steering

Craig Atkinson

AI总结:

HELIX 通过几何引导解耦量化大语言模型中的熵与幻觉,通过引导隐藏状态轨迹到预计算的诚实性流形,提高高温度下的输出质量与多样性。

摘要:

量化语言模型面临一个根本性的困境:低采样温度会导致重复、模式坍缩的输出,而高温度(T > 2.0)会导致轨迹发散和语义不连贯。我们提出HELIX,一个几何框架,通过将隐藏状态轨迹 tether 到预计算的诚实性流形来解耦输出熵与幻觉。HELIX 计算一个统一的真实性分数(UTS),结合 token 级语义熵与流形的马哈拉诺斯距离。当 UTS 表示轨迹发散时,渐进引导向量将激活方向引导到结构上一致的区域,同时只影响 0.2-2.5% 的 token。在 4 位量化 Granite 4.0 H Small(32B/9B 活跃,混合 Mamba-Transformer)上:GSM8K 在 T = 3.0 时保持 88.84% 的准确率(在 T = 0.5 时下降 2.81pp);MMLU 在 14,042 个问题上保持 72.49%(下降 1.24pp)。这表明高温度幻觉主要是轨迹发散而不是语义坍缩。值得注意的是,引导稀疏的 Transformer 注意力层(约 10% 的层)足以纠正 Mamba-2 状态空间公式中的漂移。几何 tethering 揭示了一个 previously-masked 的高熵创造性水库。在 T > 2.0 时,引导输出表现出 5-20% 的想法重复,而保守设置下为 70-80%。跨架构验证(Qwen3-30B-A3B MOE)证实这种现象是架构无关的,具有 46.7% 更高的独特概念生成。HELIX 作为语法 tether,使在不违反逻辑骨架的前提下探索语义多样性成为可能。这使 Multi-Temperature Synthesis 成为可能,生成比单温度推理多 200% 的独特概念。

英文摘要:

Quantized language models face a fundamental dilemma: low sampling temperatures yield repetitive, mode-collapsed outputs, while high temperatures (T > 2.0) cause trajectory divergence and semantic incoherence. We present HELIX, a geometric framework that decouples output entropy from hallucination by tethering hidden-state trajectories to a pre-computed truthfulness manifold. HELIX computes a Unified Truth Score (UTS) combining token-level semantic entropy with Mahalanobis distance from the manifold. When UTS indicates trajectory divergence, graduated steering vectors redirect activations toward structurally coherent regions while affecting only 0.2-2.5% of tokens. On 4-bit quantized Granite 4.0 H Small (32B/9B active, hybrid Mamba-Transformer): GSM8K maintains 88.84% accuracy at T = 3.0 (2.81pp degradation from T = 0.5); MMLU maintains 72.49% across 14,042 questions (1.24pp degradation). This demonstrates that high-temperature hallucination is primarily trajectory divergence rather than semantic collapse. Notably, steering the sparse Transformer attention layers (~10% of layers) is sufficient to correct drift in the Mamba-2 state-space formulation. Geometric tethering reveals a previously-masked High-Entropy Creative Reservoir. At T > 2.0, steered outputs exhibit 5-20% idea duplication versus 70-80% at conservative settings. Cross-architecture validation (Qwen3-30B-A3B MOE) confirms this phenomenon is architecture-independent, with 46.7% higher unique concept generation. HELIX acts as a syntax tether, enabling exploration of semantic diversity without violating the logical backbone required for valid output. This enables Multi-Temperature Synthesis, generating 200% more unique concepts than single-temperature inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑