AI 中文总结
研究探讨令牌压缩极限，提出渐进式填充方法，通过逐个增长目标前缀来研究压缩，发现完美重构不足以进行有意义压缩，该方法可定位为研究压缩极限的工具。
AI 中文摘要
令牌填充可将序列压缩为具有近乎完美重构的学习嵌入，但固定的令牌预算和99%的准确率阈值使得尚不清楚残余误差是反映优化失败还是基本限制。我们引入了渐进式填充，它逐个令牌地增长目标前缀，仅在固定优化预算内无法实现重构时才停止。渐进轨迹在嵌入空间中占据低维结构。即使上下文中有原始前缀，在多项选择基准测试中，前置填充嵌入也会导致适度但一致的准确率下降，并且在生成评估下几乎完全丧失能力。因果注意力剔除干预将这种退化追溯到嵌入在模型早期层中的交互作用。这些结果将渐进式填充定位为研究压缩极限的工具，并表明通过脆弱的引导而非可转移语义实现的完美重构不足以进行有意义的压缩。
英文摘要
Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99\% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding's interactions in the model's early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.
