arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.10205cs.CLcs.AI

一条指令并不适用于所有情况:嵌入模型如何在低资源印度语言中对人设和指令进行对齐?

One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?

Indian Institute of Technology Gandhinagar(印度古吉拉特邦理工学院加尔各答)

Arya Shah, Himanshu beniwal, Mayank Singh

AI总结:

本文提出一个涵盖12种印度语言的统一基准测试,评估多语言嵌入模型在人设与指令对齐任务中的表现,发现E5-Large-Instruct在单语言检索中表现最佳,BGE-M3在反向检索中表现突出。

摘要:

将多语言助手与基于文化背景的用户偏好对齐,对于服务超过十亿使用者的印度多语言人口至关重要。然而,现有基准测试要么专注于单一语言,要么将检索与生成混淆,留下了一个问题:当前的嵌入模型是否能够在不依赖响应合成的情况下编码人设-指令兼容性?我们提出一个涵盖12种印度语言和四个评估任务的统一基准测试:单语言和跨语言的人设到指令检索、从指令到人设的反向检索,以及二元兼容性分类。在冻结编码器设置下,使用薄的逻辑回归头进行分类,对八种多语言嵌入模型进行了评估。E5-Large-Instruct在单语言检索中实现了27.4%的Recall@1,在跨语言转移中实现了20.7%的Recall@1,而BGE-M3在反向检索中达到了32.1%的Recall@1。在分类任务中,LaBSE实现了75.3%的AUROC,具有较强的校准能力。这些发现为印度多语言检索中的模型选择提供了实际指导,并为未来工作建立了可重复的基准线。

英文摘要:

Aligning multilingual assistants with culturally grounded user preferences is essential for serving India's linguistically diverse population of over one billion speakers across multiple scripts. However, existing benchmarks either focus on a single language or conflate retrieval with generation, leaving open the question of whether current embedding models can encode persona-instruction compatibility without relying on response synthesis. We present a unified benchmark spanning 12 Indian languages and four evaluation tasks: monolingual and cross-lingual persona-to-instruction retrieval, reverse retrieval from instruction to persona, and binary compatibility classification. Eight multilingual embedding models are evaluated in a frozen-encoder setting with a thin logistic regression head for classification. E5-Large-Instruct achieves the highest Recall@1 of 27.4\% on monolingual retrieval and 20.7\% on cross-lingual transfer, while BGE-M3 leads reverse retrieval at 32.1\% Recall@1. For classification, LaBSE attains 75.3\% AUROC with strong calibration. These findings offer practical guidance for model selection in Indic multilingual retrieval and establish reproducible baselines for future work\footnote{Code, datasets, and models are publicly available at https://github.com/aryashah2k/PI-Indic-Align.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑