arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.09775cs.AIcs.CL
收藏

EasyJudge:一个用于LLM全面响应评估的易用工具

EasyJudge: an Easy-to-use Tool for Comprehensive Response Evaluation of LLMs

Minzu University of China(中央民族大学);National Language Resource Monitoring & Research Center Minority Languages Branch(国家语言资源监测与研究中心少数民族语言分中心)

Yijie Li, Yuan Sun

AI总结:

EasyJudge是一个轻量、高效的开源LLM评估工具,通过精细提示和数据集优化,在消费级硬件上实现与人类及专有模型高度一致的评估结果,并提供直观的可视化界面。

摘要:

近年来,使用大型语言模型(LLMs)评判其他LLM质量的趋势日益增长。许多研究采用闭源模型,主要使用GPT-4作为评估器。然而,由于GPT-4模型的闭源性质,将其作为评估器导致了透明度、可控性和成本效益等问题。一些研究者转向使用微调的开源LLMs作为评估器。然而,现有的开源评估LLMs普遍缺乏用户友好的可视化工具,且未针对加速模型推理进行优化,这给资源有限和跨领域工作的研究者带来了不便。本文提出EasyJudge,一个用于评估重要语言模型响应的模型。它轻量、精确、高效且用户友好,具有直观的可视化界面,便于部署和使用。EasyJudge使用详细的数据集和精细的提示进行模型优化,实现了与人类和专有模型评估的强一致性。通过定量方法优化的模型使EasyJudge能在消费级GPU甚至CPU上高效运行。我们还提供了详细的分析和案例研究,以进一步揭示该方法的潜力。

英文摘要:

Recently, there has been a growing trend of employing large language models (LLMs) to judge the quality of other LLMs. Many studies have adopted closed-source models, mainly using GPT-4 as the evaluator. However, due to the closed-source nature of the GPT-4 model, employing it as an evaluator has resulted in issues including transparency, controllability, and cost-effectiveness. Some researchers have turned to using fine-tuned open-source LLMs as evaluators. However, existing open-source evaluation LLMs generally lack a user-friendly visualization tool, and they have not been optimized for accelerated model inference, which causes inconvenience for researchers with limited resources and those working across different fields. This paper presents EasyJudge, a model developed to evaluate significant language model responses. It is lightweight, precise, efficient, and user-friendly, featuring an intuitive visualization interface for ease of deployment and use. EasyJudge uses detailed datasets and refined prompts for model optimization, achieving strong consistency with human and proprietary model evaluations. The model optimized with quantitative methods enables EasyJudge to run efficiently on consumer-grade GPUs or even CPUs. We also provide detailed analysis and case studies to further reveal the potential of our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑