arXiv 2506.008752025-06-01cs.CL

CC-Tuning：一种用于改进联合多语言监督微调的跨语言连接机制

CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning

Harbin Institute of Technology（哈尔滨工业大学）；Peng Cheng Laboratory（鹏城实验室）；The University of Hong Kong（香港大学）；Central South University（中南大学）

Yangfan Ye, Xiaocheng Feng, Zekun Yuan, Xiachong Feng, Libo Qin, Lei Huang, Weitao Ma, Yichong Huang, Zhirui Zhang, Yunfei Lu, Xiaohui Yan, Duyu Tang, Dandan Tu, Bing Qin

备注：: ACL2025 main conference, long paper

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出CC-Tuning，通过在潜在层面显式融合英与非英输入的前馈激活并利用决策器和变换矩阵，改进多语言监督微调，在22种语言的六个基准上超越普通SFT。

摘要：

当前的大语言模型（LLMs）由于其以英语为中心的训练语料，往往表现出不平衡的多语言能力。为了解决这一问题，现有的在数据层面运作的微调方法（例如通过数据增强或蒸馏）通常引入隐式的跨语言对齐，忽视了更深层次的、潜在层面的跨语言交互的潜力。在这项工作中，我们提出了CC-Tuning，这是一种新颖的多语言微调范式，它在潜在层面显式地建立了跨语言连接机制。在训练过程中，CC-Tuning融合来自英语和非英语输入的前馈激活，使模型能够同时受益于两种语言资源。这一过程借助一个可训练的决策器（Decision Maker）来实现，该决策器识别有益的激活。此外，在推理过程中，利用变换矩阵（Transform Matrix）通过表示变换在单语设置下模拟跨语言连接。我们在涵盖22种语言的六个基准上的实验表明，CC-Tuning优于普通的监督微调（vanilla SFT），并为数据层面增强方法提供了一种强有力的潜在层面替代方案。进一步的分析还突显了CC-Tuning的实用性以及潜在层面跨语言交互在提升大语言模型多语言性能方面的潜力。

英文摘要：

Current large language models (LLMs) often exhibit imbalanced multilingual capabilities due to their English-centric training corpora. To address this, existing fine-tuning approaches operating at the data-level (e.g., through data augmentation or distillation) typically introduce implicit cross-lingual alignment, overlooking the potential for more profound, latent-level cross-lingual interactions. In this work, we propose CC-Tuning, a novel multilingual fine-tuning paradigm that explicitly establishes a cross-lingual connection mechanism at the latent level. During training, CC-Tuning fuses the feed forward activations from both English and non-English inputs, enabling the model to benefit from both linguistic resources. This process is facilitated with a trainable Decision Maker that identifies beneficial activations. Furthermore, during inference, a Transform Matrix is utilized to simulate the cross-lingual connection under monolingual setting through representation transformation. Our experiments on six benchmarks covering 22 languages show that CC-Tuning outperforms vanilla SFT and offers a strong latent-level alternative to data-level augmentation methods. Further analysis also highlights the practicality of CC-Tuning and the potential of latent-level cross-lingual interactions in advancing the multilingual performance of LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。