arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2404.08382cs.CLcs.AI
收藏

看文本:指令微调语言模型是比你想象的更稳健的多项选择题选择器

Look at the Text: Instruction-Tuned Language Models are More Robust Multiple Choice Selectors than You Think

Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);LMU Munich(慕尼黑大学);Bocconi University(博科尼大学)

Xinpeng Wang, Chengzhi Hu, Bolei Ma, Paul Röttger, Barbara Plank

AI总结:

本文研究指令微调语言模型在多项选择题中的稳健性,发现文本答案比首个词元概率对问题扰动更稳健,尤其在不匹配率超过50%时,优于去偏方法,支持文本答案评估。

摘要:

多项选择题(MCQs)常用于评估大型语言模型(LLMs)的能力。评估模型响应的一种常见方式是基于首个词元预测的对数概率对候选答案进行排序。另一种方式是检查文本输出。先前的工作表明,首个词元概率对MCQ措辞变化的稳健性不足,且对于指令微调模型,首个词元概率与文本答案不一致。因此,本文研究了文本答案的稳健性。我们表明,当首个词元答案与文本答案不匹配时,文本答案对问题扰动的稳健性高于首个词元概率。随着不匹配率的增加,稳健性差异增大。当不匹配率超过50%时,与使用最先进的去偏方法(如PriDe)去偏后的首个词元概率相比,文本答案对选项顺序变化更为稳健。我们的发现进一步证明了文本答案评估相对于首个词元概率评估的优势。

英文摘要:

Multiple choice questions (MCQs) are commonly used to evaluate the capabilities of large language models (LLMs). One common way to evaluate the model response is to rank the candidate answers based on the log probability of the first token prediction. An alternative way is to examine the text output. Prior work has shown that first token probabilities lack robustness to changes in MCQ phrasing, and that first token probabilities do not match text answers for instruction-tuned models. Therefore, in this paper, we investigate the robustness of text answers. We show that the text answers are more robust to question perturbations than the first token probabilities, when the first token answers mismatch the text answers. The difference in robustness increases as the mismatch rate becomes greater. As the mismatch reaches over 50\%, the text answer is more robust to option order changes than the debiased first token probabilities using state-of-the-art debiasing methods such as PriDe. Our findings provide further evidence for the benefits of text answer evaluation over first token probability evaluation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑