arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2505.18545cs.LGcs.CL
收藏

B-score:利用响应历史检测大型语言模型中的偏见

B-score: Detecting biases in large language models using response history

KAIST, South Korea(韩国科学技术院);Auburn University, USA(奥本大学)

An Vo, Mohammad Reza Taesiri, Daeyoung Kim, Anh Totti Nguyen

AI总结:

提出B-score指标,利用多轮对话中的响应历史检测并减少LLM偏见,在MMLU、HLE和CSQA上提升答案验证准确性。

摘要:

大型语言模型(LLMs)经常表现出强烈的偏见,例如,针对女性或偏爱数字7。我们研究当LLMs在多轮对话中被允许观察其对同一问题的先前答案时,是否能够输出较少偏见的答案。为了理解哪些类型的问题会引发更多偏见的答案,我们在提出的问题集上测试LLMs,该问题集涵盖9个主题,并属于三种类型:(1)主观性;(2)随机性;(3)客观性。有趣的是,LLMs能够在多轮对话中针对寻求随机、无偏答案的问题进行“去偏见”自我修正。此外,我们提出了B-score,一种新颖的度量标准,能有效检测主观性、随机性、简单和困难问题中的偏见。在MMLU、HLE和CSQA上,利用B-score相比单独使用口头化置信度分数或单轮答案频率,显著提高了LLM答案的验证准确性(即接受LLM正确答案并拒绝错误答案)。代码和数据可在:https://b-score.github.io 获取。

英文摘要:

Large language models (LLMs) often exhibit strong biases, e.g, against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves in a multi-turn conversation in response to questions that seek an Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases to Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e, accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: https://b-score.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑