Taking a Deep Breath: Enhancing Language Modeling of Large Language Models with Sentinel Tokens
深呼吸:通过哨兵令牌增强大语言模型的语言建模
Weiyao Luo, Suncong Zheng, Heming Xia, Weikang Wang, Yan Lei, Tianyu Liu, Shuang Chen, Zhifang Sui
机构
*
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,北京大学计算机学院)
;
School of Software & Microelectronics, Peking University(北京大学软件与微电子学院)
;
Tencent Hunyuan(腾讯文言)
;
Department of Computing, The Hong Kong Polytechnic University(香港理工大学计算机系)
;
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
;
Alibaba Group(阿里巴巴集团)
机构
*
The State Key Lab for Novel Software Technology, Nanjing University(新型软件技术国家重点实验室,南京大学)
;
Australian National University(澳大利亚国立大学)
;
Imperial College London(伦敦帝国理工学院)
;
Tencent(腾讯)
机构
*
City University of Hong Kong
;
University of Illinois Urbana-Champaign
;
Tencent
;
Institute of Artificial Intelligence, China Telecom (TeleAI)
;
Tsinghua University. Corresponding Author