A-VERT：基于嵌入排序目标的无偏验证
A-VERT: Agnostic Verification with Embedding Ranking Targets
AI总结：
针对现有语言模型响应自动评估方法成本高或脱离真实场景的问题，提出无结构评估方法A-VERT，通过语义嵌入距离匹配文本与候选项，低成本实现鲁棒分类，在多数据集和架构上接近人工标注水平。
摘要：
语言模型（LM）响应的自动评估是基准与指标开发中的关键环节，既服务于模型训练，也用于生产模型端点的质量评估。当前的响应分类方法要么成本过高（如LLM-as-a-Judge（大模型作为评判者）），要么与真实场景条件相差甚远（如字符串匹配、对数概率法）。本文提出一种无结构的评估方法，该方法利用语义嵌入距离将目标候选项与任意LM生成的文本进行匹配，能以较低的计算成本（参数量小于10B的嵌入模型）实现鲁棒的响应分类。在3个数据集和3种不同LM架构上的测试结果显示，相较于人工标注，该方法的回归得分约为0.97，准确率约为96%。
英文摘要：
The automatic evaluation of Language Model (LM) responses is a critical piece in the development of benchmarks and metrics, both for model training and quality assessment of production model endpoints. The current approaches to response classification relies on methods that are too expensive (i.e. LLM-as-a-Judge) or that are far from real-world conditions (string-matching, logprob). In this paper, a structure-free evaluation method is presented. The method makes use of semantic embedding distances to match target candidates with arbitrary LM-generated text, resulting in a robust classification of the response at a relatively low compute cost (embedding models of less than $10B$ parameters). The results show a regression score of ~0.97 and an accuracy of ~96% against human annotators, tested over 3 data sets and 3 different LM architectures.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
