OrdRankBen：面向NLP中序数相关性的新型排序基准
OrdRankBen: A Novel Ranking Benchmark for Ordinal Relevance in NLP
AI总结：
针对现有NLP排序基准忽略序数相关性的问题，提出新型基准OrdRankBen及两个数据集，通过引入结构化序数标签实现更精确的多粒度排序评估。
摘要：
在自然语言处理（NLP）中，排序任务的评估仍然是一个重大挑战，这主要是由于现实场景中缺乏直接的结果标签。基准数据集在提供标准化测试平台方面发挥着关键作用，这些平台可确保公平比较、增强可复现性并实现进展跟踪，从而促进对排序模型的严格评估和持续改进。现有的NLP排序基准通常使用二元相关性标签或连续相关性分数，忽略了序数相关性分数。然而，二元标签过度简化了相关性区分，而连续分数缺乏清晰的序数结构，使得有效捕捉细微的排序差异极具挑战。为了应对这些挑战，我们引入了OrdRankBen，这是一种旨在捕捉多粒度相关性区分的新型基准。与传统的基准不同，OrdRankBen纳入了结构化的序数标签，从而实现更精确的排序评估。鉴于NLP中缺乏适用于序数相关性排序的数据集，我们构建了两个具有不同序数标签分布的数据集。我们进一步在这些数据集上评估了三种模型类型的多种模型：基于排序的语言模型、通用大语言模型和专注于排序的大语言模型。实验结果表明，序数相关性建模提供了对排序模型更精确的评估，提升了其区分排序项之间多粒度差异的能力，这对于需要细粒度相关性区分的任务至关重要。
英文摘要：
The evaluation of ranking tasks remains a significant challenge in natural language processing (NLP), particularly due to the lack of direct labels for results in real-world scenarios. Benchmark datasets play a crucial role in providing standardized testbeds that ensure fair comparisons, enhance reproducibility, and enable progress tracking, facilitating rigorous assessment and continuous improvement of ranking models. Existing NLP ranking benchmarks typically use binary relevance labels or continuous relevance scores, neglecting ordinal relevance scores. However, binary labels oversimplify relevance distinctions, while continuous scores lack a clear ordinal structure, making it challenging to capture nuanced ranking differences effectively. To address these challenges, we introduce OrdRankBen, a novel benchmark designed to capture multi-granularity relevance distinctions. Unlike conventional benchmarks, OrdRankBen incorporates structured ordinal labels, enabling more precise ranking evaluations. Given the absence of suitable datasets for ordinal relevance ranking in NLP, we constructed two datasets with distinct ordinal label distributions. We further evaluate various models for three model types, ranking-based language models, general large language models, and ranking-focused large language models on these datasets. Experimental results show that ordinal relevance modeling provides a more precise evaluation of ranking models, improving their ability to distinguish multi-granularity differences among ranked items-crucial for tasks that demand fine-grained relevance differentiation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
