arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.09089cs.CLcs.AI
收藏

SubTokenTest: 一个用于现实世界子token理解的实用基准

SubTokenTest: A Practical Benchmark for Real-World Sub-token Understanding

Institute for Artificial Intelligence, Peking University(人工智能研究院,北京大学)

Shuyang Hou, Yi Hu, Muhan Zhang

AI总结:

SubTokenTest通过实用任务评估大语言模型在子token理解上的能力,揭示分词过程对性能的影响,并分析字符级信息的编码方式。

摘要:

近年来,大语言模型(LLMs)的进展显著增强了其推理能力。然而,它们仍然在基本的字符级任务上存在困难,例如计算单词中的字母数,这一问题根植于它们的分词过程。虽然现有基准通过基本的字符操作突显了这一弱点,但这些失败往往被忽视,因为缺乏实际相关性。然而,许多现实应用,如导航基于文本的地图或解读结构化表格,都严重依赖于精确的子token理解。在这方面,我们引入了SubTokenTest,一个全面的基准,通过实用、以用途为导向的任务来评估子token理解。我们的基准包括四个领域中的十个任务,并通过将性能与复杂推理解耦来隔离与分词相关的失败。我们对九个先进的LLMs进行了全面评估。此外,我们研究了测试时缩放对子token推理的影响,并探讨了字符级信息如何在隐藏状态中编码。

英文摘要:

Recent advancements in large language models (LLMs) have significantly enhanced their reasoning capabilities. However, they continue to struggle with basic character-level tasks, such as counting letters in words, a problem rooted in their tokenization process. While existing benchmarks have highlighted this weakness through basic character operations, such failures are often dismissed due to lacking practical relevance. Yet, many real-world applications, such as navigating text-based maps or interpreting structured tables, rely heavily on precise sub-token understanding. In this regard, we introduce SubTokenTest, a comprehensive benchmark that assesses sub-token understanding through practical, utility-driven tasks. Our benchmark includes ten tasks across four domains and isolates tokenization-related failures by decoupling performance from complex reasoning. We provide a comprehensive evaluation of nine advanced LLMs. Additionally, we investigate the impact of test-time scaling on sub-token reasoning and explore how character-level information is encoded within the hidden states.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑