Large Language Models (LLMs) have shown remarkable capabilities in language understanding and generation. Nonetheless, it was also witnessed that LLMs tend to produce inaccurate responses to specific queries. This deficiency can be traced to the tokenization step LLMs must undergo, which is an inevitable limitation inherent to all LLMs. In fact, incorrect tokenization is the critical point that hinders LLMs in understanding the input precisely, thus leading to unsatisfactory output. This defect is more obvious in Chinese scenarios. To demonstrate this flaw of LLMs, we construct an adversarial dataset, named as $\textbf{ADT (Adversarial Dataset for Tokenizer)}$, which draws upon the vocabularies of various open-source LLMs to challenge LLMs' tokenization. ADT consists of two subsets: the manually constructed ADT-Human and the automatically generated ADT-Auto. Our empirical results reveal that our ADT is highly effective on challenging the tokenization of leading LLMs, including GPT-4o, Llama-3, Deepseek-R1 and so on, thus degrading these LLMs' capabilities. Moreover, our method of automatic data generation has been proven efficient and robust, which can be applied to any open-source LLMs. In this paper, we substantially investigate LLMs' vulnerability in terms of challenging their token segmentation, which will shed light on the subsequent research of improving LLMs' capabilities through optimizing their tokenization process and algorithms.
Beyond Hearing: Learning Task-Agnostic ExG Representations from Earphones via Physiology-Informed Tokenization
KAIST(韩国科学技术院);Carnegie Mellon University(卡内基梅隆大学);University of Cambridge(剑桥大学);Shanghai Jiao Tong University(上海交通大学);Pennsylvania State University(宾夕法尼亚州立大学);UCLA(加州大学洛杉矶分校);Northwest University(北华大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);Microsoft Research(微软研究院)
QuickGrasp: Responsive Video-Language Querying Service via Accelerated Tokenization and Edge-Augmented Inference
School of Computing Science, Simon Fraser University(计算科学学院,西蒙弗雷泽大学);Computer Science Department, University of Illinois Urbana-Champaign(计算机科学系,伊利诺伊大学厄巴纳-香槟分校);College of Cryptology and Cyber Science, Nankai University(密码学与网络科学学院,南开大学);College of Computer Science and Software Engineering, Shenzhen University(计算机科学与软件工程学院,深圳大学)