arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.12043cs.AI
收藏

Grade Score:量化大语言模型在选项选择中的性能

Grade Score: Quantifying LLM Performance in Option Selection

Dmitri Iourovitski

AI总结:

本研究提出新型指标Grade Score,结合熵与众数频率评估LLM作为选择题评判者的顺序偏差与选择一致性,验证了提示工程等优化方法的效果,发现指令遵循模型的偏差适配涌现行为,为提升LLM决策可靠性与公平性提供支撑。

摘要:

本研究提出了“Grade Score(等级分数)”这一新型指标,用于评估大语言模型(LLM)作为选择题评判者时,在顺序偏差和选择一致性方面的表现稳定性与公平性。该指标结合了衡量顺序偏差的熵(Entropy)和评估选择稳定性的众数频率(Mode Frequency),可反映LLM的可靠性与公正性。研究探索了提示工程、选项采样策略等优化Grade Score的技术,证明了这些方法能有效提升LLM的表现。结果显示,不同LLM对提示的响应表现存在差异,同时凸显了加入无关选项的积极作用。研究还发现指令遵循模型中存在一种涌现行为:它们能适应针对特定偏差的指令,展现出适配能力。Grade Score可用于不同LLM间的比较,推动优化其决策过程的后续研究,对提升各类应用中LLM的可靠性与公平性具有潜在意义。所有代码已开源至GitHub:https://github.com/IoDmitri/GradeLab

英文摘要:

This study introduces the "Grade Score", a novel metric designed to evaluate the consistency and fairness of Large Language Models (LLMs) when used as multiple-choice judges with respect to order bias and choice consistency. The Grade Score combines Entropy, which measures order bias, and Mode Frequency, which assesses choice stability, offering insights into LLMs' reliability and impartiality. The study explores techniques such as prompt engineering and option sampling strategies to optimize the Grade Score, demonstrating their effectiveness in enhancing LLMs' performance. Results showcase varying performances among LLMs with respect to prompts and highlight the positive impact of including irrelevant options. The study also identifies an emergent behavior in instruction-following models, where they adapt to instructions targeting specific biases, demonstrating their adaptability. The Grade Score facilitates comparisons between LLMs and encourages ongoing research towards optimizing their decision-making processes, with potential implications for improving their reliability and fairness in various applications. All code is available on GitHub https://github.com/IoDmitri/GradeLab

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑