arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.34875cs.CL
收藏

RubriX:一种快速、可微调的非自回归评分标准评判器

RubriX: A Fast, Fine-Tunable Non-Autoregressive Rubric Judge

CobuildX(科构思公司)

Balamurugan Paramasivam, Sudalai Rajkumar

AI总结:

提出基于对比语言模型的非自回归评分标准评判器RubriX,在准确率接近现有模型的同时,推理速度大幅提升,优于多个基准模型。

摘要:

基于评分标准的评估旨在判断响应符合评分标准的哪一等级,通常通过将大型语言模型(LLM)作为评判器实现,每次评判需要一次生成式调用,且需读取所有等级描述,因此其成本随评分标准的规模增长而增加。我们提出RubriX,一种评分标准评判器,其架构受对比语言模型(CLM)启发:采用LoRA适配的编码器对输入和每个等级描述进行嵌入,通过缩放余弦相似度的softmax计算得到每个等级的概率,无需文本生成。在公开评分标准数据和专有合成数据上训练后,RubriX在保留的评分标准上达到75.4%的精确准确率,而TypeSafe的Jev为57.1%,Qwen3-8B评判器为49.8%。在8个人类标注的基准数据集上,RubriX在大多数数据集上的准确率与Jev相差不超过6个百分点,而Jev在两个数据集上保持更大优势且平均排名最佳;与G-Eval论文报告的结果相比,RubriX在所有SummEval维度和两个QAGS子集上均优于基于GPT-3.5的G-Eval。在等级描述从10个token增长到3000个token时,若缓存等级嵌入,RubriX的每次评判延迟保持在约28毫秒,而Jev(减去近似网络时间)从44毫秒上升到211毫秒,Qwen3-8B评判器从32毫秒上升到516毫秒,在每个等级3000个token时,RubriX比Jev快7.7倍,比Qwen3-8B评判器快18.8倍。

英文摘要:

Rubric-based evaluation asks which level of a rubric a response meets. It is usually done by prompting a large language model (LLM) as a judge, which needs a generative call per judgment and reads every level description, so its cost grows with the rubric. We present RubriX, a rubric judge whose architecture is inspired by contrastive language models (CLM): a LoRA-adapted encoder embeds the input and each level description, and a softmax over scaled cosine similarities gives a probability for every level without text generation. Trained on public rubric data and proprietary synthetic data, RubriX reaches 75.4% exact accuracy on held-out rubrics, against 57.1% for TypeSafe's Jev and 49.8% for a Qwen3-8B judge. On eight human-labelled benchmark sets it is within 6 percentage points of Jev's accuracy on most of them, while Jev keeps a larger lead on two and ranks responses best on average; against the results reported in the G-Eval paper, RubriX outperforms G-Eval with GPT-3.5 on every SummEval dimension and both QAGS subsets. With level embeddings cached, RubriX's latency stays at about 28 ms per judgment as level descriptions grow from 10 to 3,000 tokens, while Jev's (approximate network time subtracted) rises from 44 ms to 211 ms and the Qwen3-8B judge's from 32 ms to 516 ms, making RubriX 7.7x faster than Jev and 18.8x faster than the Qwen3-8B judge at 3,000 tokens per level.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑