截断盲区：解码策略如何系统性地排除人类样式的令牌选择
The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices
AI总结：
研究探讨了文本生成中解码策略与人类语言生成的差异，发现基于概率的解码方法排除了人类可选但统计上稀有的令牌，导致生成文本可检测性增强。
摘要：
标准文本生成解码策略，包括top-k、nucleus采样和对比搜索，基于概率选择令牌，限制选择范围在高概率区域。人类语言生成则基于交际适当性而非统计频率。这种不匹配导致截断盲区：上下文合适但统计上稀有的令牌对人类可及但对基于概率的解码不可达。我们假设这增强了机器生成文本的可检测性。分析超过180万文本，涵盖八种语言模型、五种解码策略和53种超参数配置，发现8-18%的人类选择令牌超出典型截断边界。简单分类器在可预测性和词汇多样性上表现优异。关键发现是模型规模和架构与可检测性无强相关性；截断参数解释了大部分变异。实现低可检测性的配置往往产生不连贯的文本，表明逃避检测和生成自然文本是不同目标。这些发现表明可检测性由基于概率的令牌选择增强，而非仅仅是模型能力问题。
英文摘要：
Why does machine-generated text remain detectable? We investigate a mechanistic explanation at the decoding stage: standard strategies such as top-$k$ and nucleus sampling restrict generation to high-probability tokens, while human writers routinely choose contextually appropriate words from deeper in the model's probability distribution. Truncation makes a measurable share of these choices unreachable; we call this the \emph{truncation blind spot}. Across five open models and three domains, 8--18\% of human-selected tokens fall outside common truncation boundaries. Linguistic analysis further reveals disproportionate exclusion of content-word tokens. In a benchmark comprising 1.8 million machine generations, classifiers using only predictability and lexical diversity achieve mean AUC-ROC near 0.97, with substantial variation across decoding settings and strong transfer across generators. Probability-floor samplers substantially narrow the blind spot, demonstrating that the choice of truncation criterion matters for retaining human-used tokens. Together, these findings characterize a source of human--machine distributional mismatch and motivate decoding methods that preserve contextually appropriate low-probability choices while maintaining generation quality. Code and data are available at https://github.com/EstebanGarces/human_vs_machine.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
