TurtleBench:通过真实世界是/否谜题评估顶尖语言模型
TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles
AI总结:
本研究提出基于真实海龟汤谜题用户猜测的TurtleBench评估基准,动态评估大语言模型推理能力,评估9种先进LLM发现o1系列未领先,并提出相关研究假设。
摘要:
随着大语言模型(LLM)的应用不断拓展,对可靠评估的需求日益增长。现有LLM评估基准主要依赖静态数据集,难以评估模型在与用户动态交互中的表现。此外,这些基准往往依赖特定背景知识,使得衡量模型的逻辑推理能力变得复杂。其他基于强模型或人工工作的动态评估方法可能引入偏差,且成本高、耗时长,阻碍了大规模应用。为解决这些问题,我们提出了TurtleBench。TurtleBench从我们开发的在线“海龟汤”谜题平台收集真实用户的猜测内容。这种方法能够相对动态地生成评估数据集,降低模型作弊的风险,同时使评估更贴合用户对推理能力的真实需求,从而提升评估的可靠性。TurtleBench包含1532条用户猜测以及标注后的猜测正确性结果。利用该数据集,我们全面评估了当前九种最先进的大语言模型。值得注意的是,OpenAI o1系列模型在这些评估中并未取得领先结果。我们提出了若干供后续研究的假设,例如“o1的潜在推理使用了普通的思维链(CoT)技术”以及“增加CoT长度不仅能带来推理收益,还会产生噪声成本”。
英文摘要:
As the application of Large Language Models (LLMs) expands, the demand for reliable evaluations increases. Existing LLM evaluation benchmarks primarily rely on static datasets, making it challenging to assess model performance in dynamic interactions with users. Moreover, these benchmarks often depend on specific background knowledge, complicating the measurement of a model's logical reasoning capabilities. Other dynamic evaluation methods based on strong models or manual efforts may introduce biases and incur high costs and time demands, hindering large-scale application. To address these issues, we propose TurtleBench. TurtleBench collects real user guesses from our online Turtle Soup Puzzle platform that we developed. This approach allows for the relatively dynamic generation of evaluation datasets, mitigating the risk of model cheating while aligning assessments more closely with genuine user needs for reasoning capabilities, thus enhancing the reliability of evaluations. TurtleBench includes 1,532 user guesses along with the correctness of guesses after annotation. Using this dataset, we thoroughly evaluated nine of the most advanced LLMs available today. Notably, the OpenAI o1 series models did not achieve leading results in these evaluations. We propose several hypotheses for further research, such as "the latent reasoning of o1 utilizes trivial Chain-of-Thought (CoT) techniques" and "increasing CoT length not only provides reasoning benefits but also incurs noise costs."
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。