Pipelined NLP systems have largely been superseded by end-to-end neural modeling, yet nearly all commonly-used models still require an explicit tokenization step. While recent tokenization approaches based on data-derived subword lexicons are less brittle than manually engineered tokenizers, these techniques are not equally suited to all languages, and the use of any fixed vocabulary may limit a model's ability to adapt. In this paper, we present CANINE, a neural encoder that operates directly on character sequences, without explicit tokenization or vocabulary, and a pre-training strategy that operates either directly on characters or optionally uses subwords as a soft inductive bias. To use its finer-grained input effectively and efficiently, CANINE combines downsampling, which reduces the input sequence length, with a deep transformer stack, which encodes context. CANINE outperforms a comparable mBERT model by 2.8 F1 on TyDi QA, a challenging multilingual benchmark, despite having 28% fewer model parameters.
Contextual morphologically-guided tokenization for Latin encoder models
Manning College of Information & Computer Sciences, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校信息与计算机科学学院）；Institute for the Study of the Ancient World, New York University（纽约大学古代世界研究所）
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Yuanpei College, Peking University（北京大学元培学院）；Beijing Institute for General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国）；Amazon GenAI（亚马逊生成人工智能）；Qatar Computing Research Institute（卡塔尔计算研究所）；University of Virginia（弗吉尼亚大学）
ActionPiece: Rethinking Action Tokenization for Autoregressive Vision-Language-Action Models
Huazhong University of Science and Technology（华中科技大学）；Zhongguancun Academy（中关村学院）；DeepCybo；Harbin Institute of Technology（哈尔滨工业大学）；Beihang University（北京航空航天大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhengzhou University（郑州大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）
FuseCodec: Semantic-Contextual Fusion and Supervision for Neural Codecs
Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国）；Amazon GenAI（亚马逊生成人工智能）；Qatar Computing Research Institute（卡塔尔计算研究所）；University of Virginia（弗吉尼亚大学）
