arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2502.13396cs.CL
收藏

两步将加权机制提示引入 LLM-as-a-Judge:一项案例研究

Prompting a Weighting Mechanism into LLM-as-a-Judge in Two-Step: A Case Study

Databricks;Gustavus Adolphus College(古斯塔夫斯·阿道弗斯学院)

Wenwen Xie, Gray Gwizdz, Dongji Feng

AI总结:

针对 LLM-as-a-Judge 难以权衡信息重要性的问题,本文提出含显式加权机制的两步提示设计,使 HAR 平均提升 6%。

摘要:

尽管大语言模型(LLM)已成为评估自然语言生成(NLG)任务的有力工具,但其有效性受到限制:它们无法恰当地权衡不同主题的重要性,往往过度强调次要细节而低估关键信息,从而产生误导性评估。我们的工作提出一种高效的提示设计机制来解决这一具体局限,并提供案例研究。通过结合显式重要性加权机制的策略性提示工程,我们增强了使用 LLM-as-a-Judge 有效优先处理相关信息的能力;Human Alignment Rate(HAR,人类一致率)指标平均提升 6% 证明了这一点。

英文摘要:

While Large Language Models (LLMs) have emerged as promising tools for evaluating Natural Language Generation (NLG) tasks, their effectiveness is limited by their inability to appropriately weigh the importance of different topics, often overemphasizing minor details while undervaluing critical information, leading to misleading assessments. Our work proposes an efficient prompt design mechanism to address this specific limitation and provide a case study. Through strategic prompt engineering that incorporates explicit importance weighting mechanisms, we enhance using LLM-as-a-Judge ability to prioritize relevant information effectively, as demonstrated by an average improvement of 6% in the Human Alignment Rate (HAR) metric.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑