arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2306.01694cs.LGcs.HC
收藏

通过交互评估面向数学领域的语言模型

Evaluating Language Models for Mathematics through Interactions

University of Cambridge(剑桥大学);University of Oxford(牛津大学);MIT(麻省理工学院);The Alan Turing Institute(艾伦·图灵研究所);NYU(纽约大学);Vienna University of Technology(维也纳工业大学);Collège de France(法兰西公学院)
Katherine M. Collins, Albert Q. Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B. Tenenbaum, William Hart, Ti… 展开作者

Katherine M. Collins, Albert Q. Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B. Tenenbaum, William Hart, Timothy Gowers, Wenda Li, Adrian Weller, Mateja Jamnik

AI总结:

该研究推出交互评估平台CheckMate,评估三款LLM的本科数学证明辅助能力,发布数据集MathConverse,分析得出人类行为分类等发现,为相关从业者提供了模型优化与使用的可行建议。

摘要:

当前人们对利用大语言模型(large language models, LLMs)构建解题助手的机遇抱有极大热情。然而,评估LLM的标准方法依赖静态的输入-输出对,不足以帮助人们明确判断哪些LLM可被合理使用,以及在哪些辅助场景下可使用。静态评估未考虑LLM部署中必不可少的交互要素,因此限制了我们对语言模型能力的理解。\n我们推出了CheckMate——一款可供人类与LLM交互并对其进行评估的适配性原型平台。我们借助CheckMate开展了一项研究,评估InstructGPT、ChatGPT和GPT-4三款语言模型作为本科数学证明助手的表现,参与研究的混合群体涵盖本科生到数学教授等不同身份的人员。我们发布了由此产生的交互与评分数据集MathConverse。\n通过分析MathConverse,我们归纳出人类行为的分类体系,并发现尽管正确率与感知有用性总体呈正相关,但仍存在显著的二者背离的情况,此外还有其他多项发现。进一步地,我们通过数学专家贡献的一系列案例研究,对GPT-4的数学解题能力有了更细致的理解。最后,我们为机器学习从业者和数学家提出了可落地的启示:能够传达不确定性、对用户修正反馈良好、更具可解释性且更简洁的模型,可能是更优质的助手。交互评估是探索这类模型能力的一种很有前景的方式;人类应当意识到语言模型在代数运算上的易错性,并辨别其适用场景。

英文摘要:

There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs, and is insufficient for making an informed decision about which LLMs and under which assistive settings can they be sensibly used. Static assessment fails to account for the essential interactive element in LLM deployment, and therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analysing MathConverse, we derive a taxonomy of human behaviours and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, amongst other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by expert mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty respond well to user corrections, and are more interpretable and concise may constitute better assistants. Interactive evaluation is a promising way to navigate the capability of these models; humans should be aware of language models' algebraic fallibility and discern where they are appropriate to use.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑