Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenization
通过质量感知的分词解锁噪声真实世界语料用于基础模型预训练
Arvid E. Gollwitzer, Paridhi Latawa, David de Gruijl, Deepak A. Subramanian, Adrián Noriega de la Colina
机构
*
Broad Institute of MIT and Harvard, Cambridge, MA, USA(麻省理工学院与哈佛大学Broad研究所)
;
Massachusetts Institute of Technology, Cambridge, MA, USA(麻省理工学院)
;
Koch Institute for Integrative Cancer Research, MIT, Cambridge, MA, USA(麻省理工学院Koch整合癌症研究 institute)
;
Department of Neurology and Neurosurgery, McGill University, Montreal, Canada(麦吉尔大学神经学与神经外科系)
;
The Montreal Neurological Hospital-Institute, Montreal, Canada(蒙特利尔神经科学医院-研究所)
Encoding syntactic objects and Merge operations in function spaces
在函数空间中编码语法对象和Merge操作
Matilde Marcolli, Robert C. Berwick
机构
*
Department of Mathematics(数学系)
;
Department of Computing(计算系)
;
Mathematical Sciences, California Institute of Technology, CA 91125, USA(数学科学,加州理工学院)
;
Institute for Data Systems(数据系统研究所)
;
Society, Massachusetts Institute of Technology, Cambridge, MA 02139, USA(社会研究所,麻省理工学院)