Facilitating large language model Russian adaptation with Learned Embedding Propagation
Lomonosov Moscow State University(莫斯科罗蒙诺索夫国立大学)
Mikhail Tikhomirov, Daniil Chernyshev
备注:
Preprint version of an article published in the Journal of Language and Education. Copyright held by the owner/author(s). Publication rights licensed to the Journal of Language and Education
Rapid advancements of large language model (LLM) technologies led to the introduction of powerful open-source instruction-tuned LLMs that have the same text generation quality as the state-of-the-art counterparts such as GPT-4. While the emergence of such models accelerates the adoption of LLM technologies in sensitive-information environments the authors of such models don not disclose the training data necessary for replication of the results thus making the achievements model-exclusive. Since those open-source models are also multilingual this in turn reduces the benefits of training a language specific LLMs as improved inference computation efficiency becomes the only guaranteed advantage of such costly procedure. More cost-efficient options such as vocabulary extension and subsequent continued pre-training are also inhibited by the lack of access to high-quality instruction-tuning data since it is the major factor behind the resulting LLM task-solving capabilities. To address the limitations and cut the costs of the language adaptation pipeline we propose Learned Embedding Propagation (LEP). Unlike existing approaches our method has lower training data size requirements due to minimal impact on existing LLM knowledge which we reinforce using novel ad-hoc embedding propagation procedure that allows to skip the instruction-tuning step and instead implant the new language knowledge directly into any existing instruct-tuned variant. We evaluated four Russian vocabulary adaptations for LLaMa-3-8B and Mistral-7B, showing that LEP is competitive with traditional instruction-tuning methods, achieving performance comparable to OpenChat 3.5 and LLaMa-3-8B-Instruct, with further improvements via self-calibration and continued tuning enhancing task-solving capabilities.
Friend or Foe? Language as an ideological switch in open-weight LLMs under Russian disinformation stress
Institute of Culture Studies, University of Silesia in Katowice(文化研究学院,卡托维察大学);University of Texas at Austin(德克萨斯大学奥斯汀分校);National Aviation University(国家航空大学)
Contextual Attention Modulation: Towards Efficient Multi-Task Adaptation in Large Language Models
SCSE, Beihang University(北京航空航天大学系统工程学院);ERC of ACAT, MOE(教育部人工智能教育联合体);City University of Hong Kong(香港城市大学);Huawei Technologies Ltd.(华为技术有限公司);Key Lab of DIM, MIIT(信息通信技术重点实验室);SEM, Beihang University(北京航空航天大学电子机械学院);Zhejiang University of Technology(浙江工业大学)
CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation
Sber AI;Moscow Institute of Physics and Technology (MIPT)(莫斯科物理技术学院);Information Technologies, Mechanics and Optics University (ITMO University)(圣彼得堡国立信息技术、机械与光学大学);Artificial Intelligence Research Institute (AIRI)(人工智能研究所)
Domain adaptation of Russian ModernBERT for long legal documents
Independent Researcher(独立研究者);National Research Nuclear University MEPhI(国立核研究大学 MEPhI);Moscow Center for Advanced Studies(莫斯科高级研究中心);University of Waterloo(滑铁卢大学)
Learning Evolution via Optimization Knowledge Adaptation
Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education(教育部智能感知与图像理解重点实验室);International Research Center for Intelligent Perception and Computation(智能感知与计算国际研究中心);Xidian University(西安电子科技大学)
Dynamic Personality Adaptation in Large Language Models via State Machines
Institute for Geoinformatics, University of Münster(地理信息研究所,穆尔斯特大学);Faculty of Mathematics and Computer Science, University of Münster(数学与计算机科学学院,穆尔斯特大学);Department of Psychology, University of Münster(心理学系,穆尔斯特大学);Institute of Medical Education and Student Affairs, University of Münster(医学教育与学生事务研究所,穆尔斯特大学)
QuaLLM-Health: An Adaptation of an LLM-Based Framework for Quantitative Data Extraction from Online Health Discussions
The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Texas Tech Health Science Center El Paso(德克萨斯理工大学埃尔帕索健康科学中心);University of Texas at Austin(德克萨斯大学奥斯汀分校)