评估 LLM 应用的挑战：对自动化、人工及基于 LLM 方法的分析
The Challenges of Evaluating LLM Applications: An Analysis of Automated, Human, and LLM-Based Approaches
AI总结：
本文针对 LLM 聊天机器人评估难题，比较自动化、人工与基于 LLM 的评估，提出可结合人工和 LLM 的因子化评估机制，并通过教育报告聊天机器人实验验证其诊断价值。
摘要：
自自然语言生成诞生以来，聊天机器人一直是一项引人关注的应用。借助新型基于 transformer 的生成式 AI 方法，构建聊天机器人已变得轻而易举。面向医学和心理学等特定领域的聊天机器人正被迅速实现。然而，这不应分散人们对评估聊天机器人回复这一需求的关注。尤其是自然语言生成领域尚未就如何有效评估此类应用完全达成一致。在本研究中，我们结合日益流行的基于 LLM 的评估进一步讨论该问题，并分析它们与人工评估的相关性。此外，我们提出一种综合的因子化评估机制，可与人工评估和基于 LLM 的评估结合使用。我们展示了使用该方案在我们一个消费教育报告的聊天机器人实现中开展的实验评估结果，随后比较了自动化评估、传统人工评估、因子化人工评估和因子化 LLM 评估。结果表明，基于因子的评估能够更深入地揭示 LLM 应用中哪些方面需要改进，并进一步支持在主要功能并非直接检索的关键场景中使用人工评估的论点。
英文摘要：
Chatbots have been an interesting application of natural language generation since its inception. With novel transformer based Generative AI methods, building chatbots have become trivial. Chatbots which are targeted at specific domains for example medicine and psychology are implemented rapidly. This however, should not distract from the need to evaluate the chatbot responses. Especially because the natural language generation community does not entirely agree upon how to effectively evaluate such applications. With this work we discuss the issue further with the increasingly popular LLM based evaluations and how they correlate with human evaluations. Additionally, we introduce a comprehensive factored evaluation mechanism that can be utilized in conjunction with both human and LLM-based evaluations. We present the results of an experimental evaluation conducted using this scheme in one of our chatbot implementations which consumed educational reports, and subsequently compare automated, traditional human evaluation, factored human evaluation, and factored LLM evaluation. Results show that factor based evaluation produces better insights on which aspects need to be improved in LLM applications and further strengthens the argument to use human evaluation in critical spaces where main functionality is not direct retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
