ACCORD：弥合常识可度量性差距
ACCORD: Closing the Commonsense Measurability Gap
AI总结：
ACCORD通过受控多跳反事实和形式化要素解耦并量化LLM的常识推理能力，可自动生成任意复杂度基准，实验显示现有顶尖模型在适度扩展下性能降至随机水平，为未来改进提供空间。
摘要：
我们提出ACCORD，一个通过受控多跳反事实来解耦大型语言模型（LLMs）常识基础与推理能力的框架和基准套件。ACCORD为常识推理引入形式化要素，以显式控制和量化超出典型1或2跳的推理复杂度。独特的是，ACCORD能够自动生成任意推理复杂度的基准，因此可随未来LLM的改进而扩展。对最先进LLMs——包括GPT-4o（2024-05-13）、Llama-3-70B-Instruct和Mixtral-8x22B-Instruct-v0.1——进行基准测试表明，性能在仅适度扩展时即降至随机水平，留下了巨大的改进空间。我们发布了本工作中测试的基准套件排行榜，以及用于自动生成更复杂基准的代码。
英文摘要：
We present ACCORD, a framework and benchmark suite for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs) through controlled, multi-hop counterfactuals. ACCORD introduces formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops. Uniquely, ACCORD can automatically generate benchmarks of arbitrary reasoning complexity, and so it scales with future LLM improvements. Benchmarking state-of-the-art LLMs -- including GPT-4o (2024-05-13), Llama-3-70B-Instruct, and Mixtral-8x22B-Instruct-v0.1 -- shows performance degrading to random chance with only moderate scaling, leaving substantial headroom for improvement. We release a leaderboard of the benchmark suite tested in this work, as well as code for automatically generating more complex benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
