The latest advancements in unsupervised learning of sentence embeddings predominantly involve employing contrastive learning-based (CL-based) fine-tuning over pre-trained language models. In this study, we analyze the latest sentence embedding methods by adopting representation rank as the primary tool of analysis. We first define Phase 1 and Phase 2 of fine-tuning based on when representation rank peaks. Utilizing these phases, we conduct a thorough analysis and obtain essential findings across key aspects, including alignment and uniformity, linguistic abilities, and correlation between performance and rank. For instance, we find that the dynamics of the key aspects can undergo significant changes as fine-tuning transitions from Phase 1 to Phase 2. Based on these findings, we experiment with a rank reduction (RR) strategy that facilitates rapid and stable fine-tuning of the latest CL-based methods. Through empirical investigations, we showcase the efficacy of RR in enhancing the performance and stability of five state-of-the-art sentence embedding methods.
Refining Sentence Embedding Model through Ranking Sentences Generation with Large Language Models
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China(中国科学技术大学认知智能国家重点实验室);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际 Graduate School);Zhejiang University(浙江大学)
Unveiling Large Language Models Generated Texts: A Multi-Level Fine-Grained Detection Framework
School of Information, Renmin University of China(中国人民大学信息学院);Institute for Advanced Algorithms Research, Shanghai(上海高级算法研究院);School of Information Technology and Management, University of International Business and Economics(对外经济贸易大学信息技术与管理学院);Shanghai University of Finance and Economics(上海财经大学)
Unveiling In-Context Learning: A Coordinate System to Understand Its Working Mechanism
Southwest Jiaotong University(西南交通大学);University College London(伦敦大学学院);National University of Singapore(新加坡国立大学);Digital Twin Institute(数字孪生研究院);Eastern Institute of Technology(宁波工程学院)
From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models
University of Michigan(密歇根大学);School of Information(信息学院);Allen Institute for AI(艾伦人工智能研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院);Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
Comparing Human and Language Models Sentence Processing Difficulties on Complex Structures
Blavatnik School of Computer Science, Tel Aviv University, Israel(巴尔-艾塔夫大学计算机科学学院);Department of Linguistics, Tel Aviv University, Israel(巴尔-艾塔夫大学语言学系);Sagol School of Neuroscience, Tel Aviv University, Israel(巴尔-艾塔夫大学神经科学学院)
Why Stop at Words? Unveiling the Bigger Picture through Line-Level OCR
Typeface, India(印度Typeface公司);University of Maryland, College Park, USA(美国马里兰大学 College Park分校);Tata 1mg, India(印度Tata 1mg公司);Vellore Institute of Technology, Vellore, India(印度维洛雷技术学院);Indian Institute of Technology Delhi, India(印度德里理工学院)
RankCSE: Unsupervised Sentence Representations Learning via Learning to Rank
Peking University(北京大学);Meituan(美团);Meta AI;National Key Laboratory of General Artificial Intelligence(通用人工智能全国重点实验室);BIGAI(北京通用人工智能研究院);Renmin University of China(中国人民大学)
BERT4SO: Neural Sentence Ordering by Fine-tuning BERT
Université de Montréal(蒙特利尔大学);School of Information, Renmin University of China(中国人民大学信息学院);Mila - Québec Artificial Intelligence Institute(米拉-魁北克人工智能研究所)
Just Rank: Rethinking Evaluation with Word and Sentence Similarities
National University of Singapore(新加坡国立大学);University of Southern California(南加州大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Kriston AI(科通智能)
DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learning
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University(航天信息安全部门与可信计算重点实验室,教育部,网络安全科学与工程学院,武汉大学)
Exploring Embeddings for Measuring Text Relatedness: Unveiling Sentiments and Relationships in Online Comments
Bellarmine College Preparatory(贝拉明预备学院);Barrington High School(巴林顿高中);Los Osos High School(洛斯奥索斯高中);California High School(加利福尼亚高中);William Fremd High School(威廉·弗伦德高中);Basis Independent Silicon Valley(贝西斯独立硅谷学校);University of Maryland(马里兰大学)
虽然BERT是学习单语句子嵌入以获得语义相似性和基于嵌入的迁移学习的有效方法(Reimers和Gurevych,2019),但基于BERT的跨语言句子嵌入还有待探索。我们通过结合学习单语和跨语言表示的最佳方法,系统地研究了学习多语言句子嵌入的方法,包括:掩蔽语言建模(MLM),翻译语言建模(TLM)(Conneau和Lample,2019),双编码器翻译排名(Guo等人,2018),以及附加边际softmax(Yang et al.,2019年a)。我们表明,引入一个预先训练的多语言语言模型显着减少了80%的并行训练数据,以实现良好的性能。组合这些方法中最好的一种方法产生了一个模型,在Tatoeba上的112种语言中实现了83.7%的双文本检索准确率,远高于Artetxe和Schwenk(2019 b)实现的65.5%,同时在单语迁移学习基准测试中仍然具有竞争力(Conneau和Kiela,2018)。使用我们最好的模型从CommonCrawl挖掘的并行数据可以训练en-zh和en-de的竞争性NMT模型。我们在https://tfhub.dev/google/LaBSE上公开发布了适用于109多种语言的最佳多语言句子嵌入模型。