TASE：面向多语言模型的令牌感知与结构化评估
TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
AI总结：
TASE基准测试评估多语言模型在令牌级感知与结构推理上的能力，涵盖10项任务和35,927个实例，发现人类表现远超现有LLMs，为低级语言理解提供诊断工具。
摘要：
尽管大型语言模型（LLMs）在高层次语义任务上表现出色，但它们往往在细粒度的令牌级理解和结构推理方面存在困难——而这些能力对于需要精确性和控制的应用至关重要。我们引入了TASE，一个全面的基准测试，旨在评估LLMs跨语言感知和推理令牌级信息的能力。TASE涵盖了两个核心类别下的10项任务：令牌感知和结构理解，覆盖中文、英语和韩语，包含35,927个实例的评估集，以及一个可扩展的合成数据生成管道用于训练。任务包括字符计数、令牌对齐、句法结构解析和长度约束满足。我们评估了30多个领先的商业和开源LLMs，包括O3、Claude 4、Gemini 2.5 Pro和DeepSeek-R1，并使用GRPO训练方法训练了一个定制的Qwen2.5-14B模型。结果表明，人类表现显著超过当前LLMs，揭示了令牌级推理中的持续弱点。TASE揭示了这些局限性，并为未来在低级语言理解和跨语言泛化方面的改进提供了新的诊断视角。我们的代码和数据集公开于https://github.com/cyzcz/Tase。
英文摘要：
While large language models (LLMs) have demonstrated remarkable performance on high-level semantic tasks, they often struggle with fine-grained, token-level understanding and structural reasoning--capabilities that are essential for applications requiring precision and control. We introduce TASE, a comprehensive benchmark designed to evaluate LLMs' ability to perceive and reason about token-level information across languages. TASE covers 10 tasks under two core categories: token awareness and structural understanding, spanning Chinese, English, and Korean, with a 35,927-instance evaluation set and a scalable synthetic data generation pipeline for training. Tasks include character counting, token alignment, syntactic structure parsing, and length constraint satisfaction. We evaluate over 30 leading commercial and open-source LLMs, including O3, Claude 4, Gemini 2.5 Pro, and DeepSeek-R1, and train a custom Qwen2.5-14B model using the GRPO training method. Results show that human performance significantly outpaces current LLMs, revealing persistent weaknesses in token-level reasoning. TASE sheds light on these limitations and provides a new diagnostic lens for future improvements in low-level language understanding and cross-lingual generalization. Our code and dataset are publicly available at https://github.com/cyzcz/Tase .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
