尽管多模态大语言模型（MM-LLMs）近期取得了令人振奋的进展，但它们在高效建模多模态输入间的交互以及非文本模态的生成方面仍面临挑战。在本研究中，我们提出了TEAL（Tokenize and Embed ALl）方法，将任意模态的输入视为token序列，并为所有模态学习一个联合嵌入空间。具体而言，对于任意模态的输入，TEAL首先利用现成的分词器将其离散化为token序列，并通过可学习的嵌入矩阵将该token序列嵌入到联合嵌入空间中。MM-LLMs只需像文本LLMs那样，以自回归方式预测多模态token。最后，应用相应的解分词器，基于预测的token序列生成各模态的输出。借助联合嵌入空间，TEAL使冻结的LLMs能够执行涉及图像和音频等非文本模态的理解与生成任务。因此，文本LLM可作为接口工作，并保持其在文本理解与生成方面的高性能。实验表明，TEAL在多模态理解上取得了显著提升，并实现了一种简单的多模态生成方案。
英文摘要：
Despite Multi-modal Large Language Models (MM-LLMs) have made exciting strides recently, they are still struggling to efficiently model the interactions among multi-modal inputs and the generation in non-textual modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an approach to treat the input from any modality as a token sequence and learn a joint embedding space for all modalities. Specifically, for the input from any modality, TEAL first discretizes it into a token sequence with the off-the-shelf tokenizer and embeds the token sequence into a joint embedding space with a learnable embedding matrix. MM-LLMs just need to predict the multi-modal tokens autoregressively as the textual LLMs do. Finally, the corresponding de-tokenizer is applied to generate the output in each modality based on the predicted token sequence. With the joint embedding space, TEAL enables the frozen LLMs to perform both understanding and generation tasks involving non-textual modalities, such as image and audio. Thus, the textual LLM can just work as an interface and maintain its high performance in textual understanding and generation. Experiments show that TEAL achieves substantial improvements in multi-modal understanding, and implements a simple scheme for multi-modal generations.
T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
Aleph Alpha；Technical University Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMs
Kyutai；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Institute of Intelligent Systems and Robotics（智能系统与机器人研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Grenoble INP（格勒诺布尔国立理工学院）；LIG（格勒诺布尔信息学实验室）
From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages
OpenBabylon；ARG-Tech, University of Dundee（邓迪大学ARG-Tech公司）；Georg-August Universität Göttingen（哥廷根大学）；Doha Institute for Graduate Studies（多哈研究生学院）；Arab Center for Research and Policy Studies（阿拉伯研究与政策研究中心）；Google（谷歌）；PolyAgent
A Novel Semantic Manifold Alignment Attack against Embedding-to-Embedding Obfuscation in Privacy-Preserving LLMs
University of Houston（休斯顿大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Waseda University（早稻田大学）；Stevens Institute of Technology（史蒂文斯理工学院）；National Taiwan University（国立台湾大学）
Beyond Atomic Tokens: Factorizing Syllables for Language Model Pretraining
University of Information Technology, Vietnam National University, Ho Chi Minh city（胡志明市越南国立大学信息技术大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
AtomDisc: An Atom-level Tokenizer that Boosts Molecular LLMs and Reveals Structure--Property Associations
The Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（人工智能动力部，香港科学与技术大学（广州））；The College of Computer Science and Technology, The Nanjing University of Aeronautics and Astronautics（计算机科学与技术学院，南京航空航天大学）
Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing
Hasso Plattner Institute（霍普夫纳研究所）；University of Potsdam（波茨坦大学）；University of Cambridge（剑桥大学）；University of Oxford（牛津大学）；German University of Digital Science（德国数字科学大学）
Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection
Huazhong University of Science and Technology（华中科技大学）；Nanyang Technological University（南洋理工大学）；Virginia Tech（弗吉尼亚理工大学）；The University of New South Wales（新南威尔士大学）
When marine radar target detection meets pretrained large language models
Department of Electronic Engineering, Tsinghua University（电子工程系，清华大学）；State Key Laboratory of Space Network and Communications, Tsinghua University（空间网络与通信国家重点实验室，清华大学）；Shenzhen Key Laboratory of Ubiquitous Data Enabling, Tsinghua Shenzhen International Graduate School, Tsinghua University（深圳 ubiquitous 数据赋能重点实验室，清华大学深圳国际研究生院）
