说任何事但并非如此：当分词器背叛大语言模型的推理
Say Anything but This: When Tokenizer Betrays Reasoning in LLMs
AI总结：
本研究发现分词器的不一致导致LLM推理错误，通过一致性探针揭示了分词引起的表征缺陷，并提出分词层面的修复方法。
摘要：
大型语言模型（LLMs）在离散的token ID序列上进行推理，然而现代的子词分词器经常会产生非唯一的编码：多个token ID序列可以解码为相同的表面字符串。这种表征不匹配会创造出一种未被测量的脆弱性，其中推理过程可能会失败。LLMs可能会将两个内部表示视为不同的
英文摘要：
Large language models (LLMs) reason over discrete token ID sequences, yet modern subword tokenizers routinely produce non-unique encodings: multiple token ID sequences can detokenize to identical surface strings. This representational mismatch creates an unmeasured fragility wherein reasoning processes can fail. LLMs may treat two internal representations as distinct "words" even when they are semantically identical at the text level. In this work, we show that tokenization can betray LLM reasoning through one-to-many token ID mappings. We introduce a tokenization-consistency probe that requires models to replace designated target words in context while leaving all other content unchanged. The task is intentionally simple at the surface level, enabling us to attribute failures to tokenizer-detokenizer artifacts rather than to knowledge gaps or parameter limitations. Through analysis of over 11000 replacement trials across state-of-the-art open-source LLMs, we find a non-trivial rate of outputs exhibit phantom edits: cases where models operate under the illusion of correct reasoning, a phenomenon arising from tokenizer-induced representational defects. We further analyze these cases and provide a taxonomy of eight systematic tokenizer artifacts, including whitespace-boundary shifts and intra-word resegmentation. These findings indicate that part of apparent reasoning deficiency originates in the tokenizer layer, motivating tokenizer-level remedies before incurring the cost of training ever-larger models on ever-larger corpora.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
