思维链中心：持续衡量大语言模型推理性能的努力
Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
AI总结：
本文提出开源评估套件 Chain-of-Thought Hub，用于衡量大语言模型的多步推理能力，发现模型规模与推理能力相关，并指出开源模型与 GPT-4 等闭源模型的差距及 RLHF 的追赶潜力。
摘要：
随着大语言模型（LLMs）的持续发展，其评估变得日益重要但也更具挑战性。本工作提出了 Chain-of-Thought Hub，一个针对大语言模型多步推理能力的开源评估套件。我们关注这一设定有两个原因：（1）从 GPT 和 PaLM 模型家族的行为中，我们观察到复杂推理很可能是较弱与较强 LLMs 之间的关键区分因素；（2）我们设想大语言模型将成为下一代计算平台，并培育基于 LLM 的新应用生态，这自然要求基础模型执行往往涉及语言与逻辑运算组合的复杂任务。我们的方法是汇编一套具有挑战性的推理基准，以跟踪 LLMs 的进展。我们当前的结果表明：（1）模型规模与推理能力明显相关；（2）截至 2023 年 5 月，Claude-v1.3 和 PaLM-2 是仅有的两个可与 GPT-4 相媲美的模型，而开源模型仍然落后；（3）LLaMA-65B 的表现接近 code-davinci-002，表明通过进一步成功开发，例如基于人类反馈的强化学习（RLHF），它有巨大潜力接近 GPT-3.5-Turbo。我们的结果还表明，为了让开源努力迎头赶上，社区可能应更加专注于构建更好的基础模型并探索 RLHF。
英文摘要：
As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging. This work proposes Chain-of-Thought Hub, an open-source evaluation suite on the multi-step reasoning capabilities of large language models. We are interested in this setting for two reasons: (1) from the behavior of GPT and PaLM model family, we observe that complex reasoning is likely to be a key differentiator between weaker and stronger LLMs; (2) we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications, this naturally requires the foundation models to perform complex tasks that often involve the composition of linguistic and logical operations. Our approach is to compile a suite of challenging reasoning benchmarks to track the progress of LLMs. Our current results show that: (1) model scale clearly correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and PaLM-2 are the only two models that are comparable with GPT-4, while open-sourced models still lag behind; (3) LLaMA-65B performs closely to code-davinci-002, indicating that with successful further development such as reinforcement learning from human feedback (RLHF), it has great potential to be close to GPT-3.5-Turbo. Our results also suggest that for the open-source efforts to catch up, the community may focus more on building better base models and exploring RLHF.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
