HiBench：基准测试大型语言模型在层次结构推理上的能力
HiBench: Benchmarking LLMs Capability on Hierarchical Structure Reasoning
AI总结：
本文提出首个层次结构推理基准框架HiBench，包含6个场景、30个任务及39,519个查询，通过5个能力维度评估20个LLM，发现其在复杂和隐式层次结构上的不足，并构建指令数据集使Llama-3.1-8B和Qwen2.5-7B性能平均提升88.84%和31.38%。
摘要：
结构推理是大型语言模型（LLMs）的一项基本能力，使其能够对结构化常识进行推理并回答多跳问题。然而，现有的结构推理基准主要关注横向和并列结构（例如图），忽略了其中的层次关系。层次结构推理对人类认知至关重要，特别是在记忆组织和问题解决方面。它还在各种现实任务中发挥关键作用，如信息抽取和决策。为了填补这一空白，我们提出了HiBench，这是第一个从初始结构生成到最终能力评估的框架，旨在系统地基准测试LLMs的层次推理能力。HiBench包含六个代表性场景，覆盖基础和实用方面，由30个具有不同层次复杂度的任务组成，总计39,519个查询。为了全面评估LLMs，我们开发了五个能力维度，描绘层次结构理解的不同侧面。通过对10个模型家族的20个LLM进行广泛评估，我们揭示了其能力与局限的关键洞察：1）现有LLMs在基本层次推理任务上表现出熟练度；2）它们在更复杂结构和隐式层次表示上仍存在困难，尤其是在结构修改和文本推理方面。基于这些发现，我们创建了一个小而精的指令数据集，使LLMs在HiBench上的性能平均提升了88.84%（Llama-3.1-8B）和31.38%（Qwen2.5-7B）。HiBench数据集和工具包可在此获取，https://github.com/jzzh/HiBench，以鼓励评估。
英文摘要：
Structure reasoning is a fundamental capability of large language models (LLMs), enabling them to reason about structured commonsense and answer multi-hop questions. However, existing benchmarks for structure reasoning mainly focus on horizontal and coordinate structures (\emph{e.g.} graphs), overlooking the hierarchical relationships within them. Hierarchical structure reasoning is crucial for human cognition, particularly in memory organization and problem-solving. It also plays a key role in various real-world tasks, such as information extraction and decision-making. To address this gap, we propose HiBench, the first framework spanning from initial structure generation to final proficiency assessment, designed to benchmark the hierarchical reasoning capabilities of LLMs systematically. HiBench encompasses six representative scenarios, covering both fundamental and practical aspects, and consists of 30 tasks with varying hierarchical complexity, totaling 39,519 queries. To evaluate LLMs comprehensively, we develop five capability dimensions that depict different facets of hierarchical structure understanding. Through extensive evaluation of 20 LLMs from 10 model families, we reveal key insights into their capabilities and limitations: 1) existing LLMs show proficiency in basic hierarchical reasoning tasks; 2) they still struggle with more complex structures and implicit hierarchical representations, especially in structural modification and textual reasoning. Based on these findings, we create a small yet well-designed instruction dataset, which enhances LLMs' performance on HiBench by an average of 88.84\% (Llama-3.1-8B) and 31.38\% (Qwen2.5-7B) across all tasks. The HiBench dataset and toolkit are available here, https://github.com/jzzzzh/HiBench, to encourage evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
