GigaEmbeddings：高效俄语嵌入模型
GigaEmbeddings: Efficient Russian Language Embedding Model
AI总结：
GigaEmbeddings通过三阶段层次化指令微调GigaChat-3B，结合双向注意力、潜在注意力池化及剪除25%层，在ruMTEB基准上以69.1平均分实现俄语嵌入最先进性能。
摘要：
我们推出了GigaEmbeddings，一个通过层次化指令微调专为俄语设计的仅解码器大语言模型（GigaChat-3B）来训练高性能俄语文本嵌入的新框架。我们的三阶段流程包括：在大规模网络语料上进行大规模对比预训练、使用难负样本进行微调，以及在检索、分类和聚类任务中进行多任务泛化，通过统一多样目标并利用合成数据生成，解决了现有方法的关键局限。架构创新包括用于上下文建模的双向注意力、用于稳健序列聚合的潜在注意力池化，以及战略性剪除25%的Transformer层以在不牺牲性能的情况下提升效率。在涵盖23个多语言任务的ruMTEB基准上评估，GigaEmbeddings取得了最先进的结果（平均得分69.1），超越了参数数量更多的强基线模型。
英文摘要：
We introduce GigaEmbeddings, a novel framework for training high-performance Russian-focused text embeddings through hierarchical instruction tuning of the decoder-only LLM designed specifically for Russian language (GigaChat-3B). Our three-stage pipeline, comprising large-scale contrastive pre-training in web-scale corpora, fine-tuning with hard negatives, and multitask generalization across retrieval, classification, and clustering tasks, addresses key limitations of existing methods by unifying diverse objectives and leveraging synthetic data generation. Architectural innovations include bidirectional attention for contextual modeling, latent attention pooling for robust sequence aggregation, and strategic pruning of 25% of transformer layers to enhance efficiency without compromising performance. Evaluated on the ruMTEB benchmark spanning 23 multilingual tasks, GigaEmbeddings achieves state-of-the-art results (69.1 avg. score), outperforming strong baselines with a larger number of parameters.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
