arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.17067cs.CLcs.AI
收藏

分词至关重要!通过挑战其分词来削弱大型语言模型

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization

School of Data Science, Fudan University(复旦大学数据科学学院);School of Management, Fudan University(复旦大学管理学院)

Dixuan Wang, Yanda Li, Junyuan Jiang, Zepeng Ding, Ziqin Luo, Guochao Jiang, Jiaqing Liang, Deqing Yang

AI总结:

针对 LLM 分词错误影响中文理解的问题,本文构建包含人工与自动子集的 ADT 对抗数据集,验证其可削弱 GPT-4o 等主流模型,并提出可迁移的自动生成方法。

摘要:

大型语言模型(LLM)在语言理解和生成方面展现出卓越能力。然而,人们也观察到 LLM 倾向于对特定查询产生不准确的回答。这一缺陷可追溯至 LLM 必须经历的分词步骤,这是所有 LLM 固有的不可避免限制。事实上,错误分词是阻碍 LLM 精确理解输入、进而导致输出不理想的关键问题。该缺陷在中文场景中更为明显。为证明 LLM 的这一缺陷,我们构建了一个名为 ADT(Adversarial Dataset for Tokenizer,面向分词器的对抗数据集)的对抗数据集,该数据集利用各种开源 LLM 的词表来挑战 LLM 的分词。ADT 包含两个子集:人工构建的 ADT-Human 和自动生成的 ADT-Auto。我们的实证结果表明,ADT 在挑战主流 LLM(包括 GPT-4o、Llama-3、Deepseek-R1 等)的分词方面非常有效,从而削弱这些 LLM 的能力。此外,我们的自动数据生成方法被证明高效且稳健,可应用于任何开源 LLM。本文从挑战 token 切分的角度深入研究了 LLM 的脆弱性,这将为后续通过优化分词过程和算法来提升 LLM 能力的研究提供启示。

英文摘要:

Large Language Models (LLMs) have shown remarkable capabilities in language understanding and generation. Nonetheless, it was also witnessed that LLMs tend to produce inaccurate responses to specific queries. This deficiency can be traced to the tokenization step LLMs must undergo, which is an inevitable limitation inherent to all LLMs. In fact, incorrect tokenization is the critical point that hinders LLMs in understanding the input precisely, thus leading to unsatisfactory output. This defect is more obvious in Chinese scenarios. To demonstrate this flaw of LLMs, we construct an adversarial dataset, named as $\textbf{ADT (Adversarial Dataset for Tokenizer)}$, which draws upon the vocabularies of various open-source LLMs to challenge LLMs' tokenization. ADT consists of two subsets: the manually constructed ADT-Human and the automatically generated ADT-Auto. Our empirical results reveal that our ADT is highly effective on challenging the tokenization of leading LLMs, including GPT-4o, Llama-3, Deepseek-R1 and so on, thus degrading these LLMs' capabilities. Moreover, our method of automatic data generation has been proven efficient and robust, which can be applied to any open-source LLMs. In this paper, we substantially investigate LLMs' vulnerability in terms of challenging their token segmentation, which will shed light on the subsequent research of improving LLMs' capabilities through optimizing their tokenization process and algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑