在文本排序的模型选择中利用估计的迁移性而非人类直觉
Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking
AI总结:
针对预训练语言模型在文本排序任务中模型选择困难的问题,提出自适应排序迁移性方法AiRTran,通过计算期望排名并自适应缩放各向同性句子嵌入来准确估计迁移性,在多个数据集上显著优于现有TE方法、人类直觉和ChatGPT。
摘要:
得益于由预训练语言模型(PLMs)增强的双编码器的应用,文本排序取得了显著进展。鉴于可用PLMs的激增,为给定数据集选择最有效的PLM已成为一项重要挑战。作为人类直觉和暴力微调的有前景的替代方案,迁移性估计(TE)已成为一种有效的模型选择方法。然而,当前的TE方法主要为分类任务设计,其估计的迁移性可能与文本排序的目标不完全一致。为应对这一挑战,我们提出计算期望排名作为迁移性,以明确反映模型的排序能力。此外,为缓解各向异性并融入训练动态,我们自适应地缩放各向同性句子嵌入,以产生准确的期望排名分数。我们由此提出的方法——自适应排序迁移性(AiRTran),能够有效捕捉模型间的细微差异。在跨越多个文本排序数据集的极具挑战性的模型选择场景中,与以往面向分类的TE方法、人类直觉以及ChatGPT相比,AiRTran展现出了显著的提升,且时间消耗极少。
英文摘要:
Text ranking has witnessed significant advancements, attributed to the utilization of dual-encoder enhanced by Pre-trained Language Models (PLMs). Given the proliferation of available PLMs, selecting the most effective one for a given dataset has become a non-trivial challenge. As a promising alternative to human intuition and brute-force fine-tuning, Transferability Estimation (TE) has emerged as an effective approach to model selection. However, current TE methods are primarily designed for classification tasks, and their estimated transferability may not align well with the objectives of text ranking. To address this challenge, we propose to compute the expected rank as transferability, explicitly reflecting the model's ranking capability. Furthermore, to mitigate anisotropy and incorporate training dynamics, we adaptively scale isotropic sentence embeddings to yield an accurate expected rank score. Our resulting method, Adaptive Ranking Transferability (AiRTran), can effectively capture subtle differences between models. On challenging model selection scenarios across various text ranking datasets, it demonstrates significant improvements over previous classification-oriented TE methods, human intuition, and ChatGPT with minor time consumption.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。