arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.00437cs.CL
收藏

BatchEval:迈向类人的文本评估

BatchEval: Towards Human-like Text Evaluation

Beijing Institute of Technology(北京理工大学);Xiaohongshu Inc(小红书公司)

Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Boyuan Pan, Heda Wang, Kan Li

AI总结:

针对当前LLM逐样本文本评估范式对提示敏感、抗噪差且集成弱的问题,提出迭代批量评估范式BatchEval,采用两阶段异构批次与十进制评分,在降低API成本的同时显著提升了评估相关性。

摘要:

随着大型语言模型(LLMs)作为评估器的引入,自动文本评估取得了显著进展。然而,当前的逐样本评估范式存在以下问题:(1)对提示设计敏感;(2)抗噪能力差;(3)在静态参考下集成性能较差。受人类将标准定义和样本间比较都作为评估参考这一事实的启发,我们提出了BatchEval,这是一种迭代进行批量评估的范式,旨在缓解上述问题。我们探索了该范式下的变体,并确认最佳设置为两阶段流程,采用异构批次构成策略和十进制评分格式。在4个文本评估任务上跨3个LLM的全面实验表明,BatchEval在Pearson相关性上优于现有最先进方法10.5%,且平均API成本仅为64%。进一步的分析验证了BatchEval的鲁棒性、泛化性和工作机制。

英文摘要:

Significant progress has been made in automatic text evaluation with the introduction of large language models (LLMs) as evaluators. However, current sample-wise evaluation paradigm suffers from the following issues: (1) Sensitive to prompt design; (2) Poor resistance to noise; (3) Inferior ensemble performance with static reference. Inspired by the fact that humans treat both criterion definition and inter sample comparison as references for evaluation, we propose BatchEval, a paradigm that conducts batch-wise evaluation iteratively to alleviate the above problems. We explore variants under this paradigm and confirm the optimal settings are two stage procedure with heterogeneous batch composition strategy and decimal scoring format. Comprehensive experiments across 3 LLMs on 4 text evaluation tasks demonstrate that BatchEval outperforms state-of-the-art methods by 10.5% on Pearson correlations with only 64% API cost on average. Further analyses have been conducted to verify the robustness, generalization, and working mechanism of BatchEval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑