由于分词器碎片化和较弱的跨语言耦合，经过指令微调的大型语言模型（LLMs）在低资源、非拉丁字母语种上表现不佳。我们提出了 LLINK（Latent Language Injection for Non-English Knowledge，面向非英语知识的潜在语言注入），这是一种计算高效的“语言即模态”方法，它在不改变分词器或重新训练解码器的情况下对指令微调解码器进行条件化。首先，我们通过轻量级对比投影器，将来自冻结多语言编码器的句子嵌入在预留位置对齐到解码器的潜在嵌入空间。其次，该向量被扩展为 K 个软槽，并使用最少的适配器进行训练，以便冻结的解码器能够消耗该信号。LLINK 显著提升了双语检索能力，并在 LLM 评判的问答评估中，相比基础模型获得了 81.3% 的偏好度，相比直接微调获得了 63.6% 的偏好度。我们进一步发现，尽管模型在数值保真度上存在残余弱点，但这些改进可归因于分词膨胀的减少和更强的跨语言对齐。将低资源语言视为一种模态，为轻量级 LLM 实现更强的跨语言对齐提供了一条实用路径。
英文摘要：
Instruction-tuned Large Language Models (LLMs) underperform on low resource, non-Latin scripts due to tokenizer fragmentation and weak cross-lingual coupling. We present LLINK (Latent Language Injection for Non-English Knowledge), a compute efficient language-as-modality method that conditions an instruction-tuned decoder without changing the tokenizer or retraining the decoder. First, we align sentence embeddings from a frozen multilingual encoder to the decoder's latent embedding space at a reserved position via a lightweight contrastive projector. Second, the vector is expanded into K soft slots and trained with minimal adapters so the frozen decoder consumes the signal. LLINK substantially improves bilingual retrieval and achieves 81.3% preference over the base model and 63.6% over direct fine-tuning in LLM-judged Q&A evaluations. We further find that improvements can be attributed to reduced tokenization inflation and a stronger cross lingual alignment, despite the model having residual weaknesses in numeric fidelity. Treating low resource languages as a modality offers a practical path to stronger cross-lingual alignment in lightweight LLMs.
Efficient Training for Cross-lingual Speech Language Models
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（中国科学院计算技术研究所智能信息处理重点实验室）；State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
IISER Kolkata（印度科学教育与研究学院加尔各答分院）；Indian Institute of Technology Patna（印度理工学院巴特那分校）
Significant developments in techniques such as encoder-decoder models have enabled us to represent information comprising multiple modalities. This information can further enhance many downstream tasks in the field of information retrieval and natural language processing; however, improvements in multi-modal techniques and their performance evaluation require large-scale multi-modal data which offers sufficient diversity. Multi-lingual modeling for a variety of tasks like multi-modal summarization, text generation, and translation leverages information derived from high-quality multi-lingual a…
Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机科学学院，北京大学）；WeChat AI, Tencent Inc., China（腾讯公司，中国）；The University of Hong Kong（香港大学）
Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
M$^3$-Med: A Benchmark for Multi-lingual, Multi-modal, and Multi-hop Reasoning in Medical Instructional Video Understanding
Beijing Institute of Technology（北京理工大学）；The Hong Kong Polytechnic University（香港理工大学）；Chinese Academy of Sciences（中国科学院）；Air Force Hospital of Southern Theater Command of PLA（中国人民解放军南部战区空军医院）；Shenzhen Traditional Chinese Medicine Hospital（深圳中医药医院）
Knowledge-enhanced language representation learning has shown promising results across various knowledge-intensive NLP tasks. However, prior methods are limited in efficient utilization of multilingual knowledge graph (KG) data for language model (LM) pretraining. They often train LMs with KGs in indirect ways, relying on extra entity/relation embeddings to facilitate knowledge injection. In this work, we explore methods to make better use of the multilingual annotation and language agnostic property of KG triples, and present novel knowledge based multilingual language models (KMLMs) trained …
Microsoft Research Asia（微软亚洲研究院）；Microsoft Azure AI（微软Azure人工智能）
Multimodal pre-training with text, layout, and image has achieved SOTA performance for visually-rich document understanding tasks recently, which demonstrates the great potential for joint learning across different modalities. In this paper, we present LayoutXLM, a multimodal pre-trained model for multilingual document understanding, which aims to bridge the language barriers for visually-rich document understanding. To accurately evaluate LayoutXLM, we also introduce a multilingual form understanding benchmark dataset named XFUND, which includes form understanding samples in 7 languages (Chin…
Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs
School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）；Kexin Technology（可心科技）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent Hunyuan（腾讯混元）；College of Computer Science, Chongqing University（重庆大学计算机学院）
A Zero-shot Learning Method Based on Large Language Models for Multi-modal Knowledge Graph Embedding
School of Software, Shandong University（山东大学软件学院）；School of Computer and Artificial Intelligence, Shandong University of Finance and Economics（山东财经大学计算机与人工智能学院）
MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation
Singapore University of Technology and Design（新加坡科技设计大学）；Agency for Science, Technology and Research, Singapore（新加坡科技研究局）；Indian Institute of Technology Delhi（印度理工学院德里分校）；Alibaba DAMO Academy（阿里巴巴达摩院）；Microsoft Research Asia（微软亚洲研究院）；Shanghai University of Finance and Economics（上海财经大学）；Inner Mongolia University（内蒙古大学）；Kyoto University（京都大学）；Jiangxi Normal University（江西师范大学）；Korea University（韩国大学）；Nanyang Technological University（南洋理工大学）
A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings
Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Chinese Academy of Sciences（中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Hong Kong Baptist University（香港 Baptist大学）
University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）；ISTI - CNR（意大利国家研究委员会信息科学与技术研究所）
Vision-and-Language Navigation (VLN) is a challenging task in which an agent needs to follow a language-specified path to reach a target destination. The goal gets even harder as the actions available to the agent get simpler and move towards low-level, atomic interactions with the environment. This setting takes the name of low-level VLN. In this paper, we strive for the creation of an agent able to tackle three key issues: multi-modality, long-term dependencies, and adaptability towards different locomotive settings. To that end, we devise "Perceive, Transform, and Act" (PTA): a fully-attent…
Testing the Depth of ChatGPT's Comprehension via Cross-Modal Tasks Based on ASCII-Art: GPT3.5's Abilities in Regard to Recognizing and Generating ASCII-Art Are Not Totally Lacking
Domain-Conditioned Safety in Frontier Computer-Using Agents: A 793-Episode Browser Benchmark, a Coding-Domain Cross-Reference, and a Reproducibility Audit of Recent Red-Teaming
Patronus AI University of California, Berkeley（Patronus AI 伯克利大学）
More than half of the 7,000 languages in the world are in imminent danger of going extinct. Traditional methods of documenting language proceed by collecting audio data followed by manual annotation by trained linguists at different levels of granularity. This time consuming and painstaking process could benefit from machine learning. Many endangered languages do not have any orthographic form but usually have speakers that are bi-lingual and trained in a high resource language. It is relatively easy to obtain textual translations corresponding to speech. In this work, we provide a multimodal …
Generation-Augmented Generation: A Plug-and-Play Framework for Private Knowledge Injection in Large Language Models
MAIS, Institute of Automation, Chinese Academy of Sciences（自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy（中关村学院）；School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学先进交叉学科学院）；AI Research（360人工智能研究院）
The Straight and Narrow: Do LLMs Possess an Internal Moral Path?
School of Future Technology, Dalian University of Technology, China（大连理工大学未来技术学院）；School of Computer Science and Technology, Dalian University of Technology, China（大连理工大学计算机科学与技术学院）；Key Laboratory of Social Computing and Cognitive Intelligence, Ministry of Education, China（教育部社会计算与认知智能重点实验室）
Computer Science and Artificial Intelligence Laboratory（计算机科学与人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embeddings, in this paper we target learning a cross-modal alignment between the embedding spaces of speech and text learned from corpora of their respective modalities in an unsupervised fashion. The proposed framework learns the individual speech and text embedding spaces, and attempts to align the two spaces via adversarial training, followed by a refinement procedure. We show ho…
Transferable Modeling Strategies for Low-Resource LLM Tasks: A Prompt and Alignment-Based Approach
Carnegie Mellon University（卡内基梅隆大学）；Georgia Institute of Technology（佐治亚理工学院）；San Francisco State University（旧金山州立大学）；Northeastern University（东北大学）；Rutgers University（罗格斯大学）
One Mind, Many Tongues: A Deep Dive into Language-Agnostic Knowledge Neurons in Large Language Models
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
