Domain Adaptation of Foundation LLMs for e-Commerce
eBay Inc.（eBay公司）
Christian Herold, Michael Kozielski, Tala Bazazo, Pavel Petrushkov, Patrycja Cieplicka, Dominika Basaj, Yannick Versley, Seyyed Hadi Hashemi, Shahram Khadivi
We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data.
We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks.
We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains.
Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data
Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）；Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）；Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）
ASR Adaptation for E-commerce Chatbots using Cross-Utterance Context and Multi-Task Language Modeling
Amazon AWS AI（亚马逊AWS人工智能部门）
对插槽实体的自动语音识别（ASR）鲁棒性在涉及货币交易和购买的电子商务语音助理中至关重要。随着有效的领域适应，它是直观的，跨话语上下文线索在消歧领域特定的内容词从语音中起着重要的作用。在本文中，我们研究了各种技术，以提高上下文，内容词的鲁棒性和域自适应的transformer-XL神经语言模型（NLM），以rescore ASR N-best假设。为了提高语境化，我们利用轮级对话行为以及跨话语语境结转。此外，为了使我们的域通用NLM适应电子商务的运行，我们在域内数据上使用来自微调掩码LM的嵌入。最后，为了提高对域内内容词的鲁棒性，我们提出了一个多任务模型，可以联合执行内容词检测和语言建模任务。与非上下文LSTM LM基线相比，我们表现最好的NLM重新评分器在电子商务音频测试集上的内容WER减少了19.2%，插槽标签F1提高了6.4%。
Bayesian Low-Rank Factorization for Robust Model Adaptation
Interactive Systems Lab, Karlsruhe Institut of Technology (KIT), Germany（交互系统实验室，卡尔斯鲁厄理工学院（KIT），德国）；InterACT, Carnegie Mellon University (CMU), USA（InterACT，卡内基梅隆大学（CMU），美国）
