arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.09997cs.SIcs.SE
收藏

GraphInstruct: 一个用于诊断LLM图生成能力差距的渐进性基准

GraphInstruct: A Progressive Benchmark for Diagnosing Capability Gaps in LLM Graph Generation

Tongji University(同济大学);Zhejiang Gongshang University(浙江工商大学)

Zihe Wei, Sheng Xiang, Ying Zhang, Changjun Jiang

AI总结:

本文提出GraphInstruct基准,通过六个复杂性层级和五个评估维度,结合800个人工指令和1582个算法生成的参考解决方案,评估LLM图生成能力,发现多约束组合而非推理深度具有更强区分能力,且无单一提示策略能跨层级或模型家族主导,域语义约束在所有测试方法中保持迭代不变,表明检索而非额外计算是下一步研究方向。

摘要:

图结构数据支撑着从引用分析和社会网络建模到分子设计和知识图谱构建的应用,大型语言模型(LLMs)正越来越多地用作提示驱动的图生成器。经典图生成审查列举了深度生成模型及其评估原语,但早于LLM时代,无法为评估遵循指示的图生成提供基础。最近的LLM时代基准评估模型沿图类型或任务领域轴进行;然而,这些组织在结构复杂性上平均,无法定位在复杂性谱中LLM崩溃的位置。为填补这一诊断差距,我们引入GraphInstruct,一个渐进复杂度基准,将LLM图生成分为六个复杂性层级和五个评估维度,配以800个手工编写指令、1582个算法合成的参考解决方案以及12个LLM的45(模型,策略)配置能力评估。我们发现区分能力峰值在多约束组合而非推理深度,没有单一提示策略在各层级或模型家族中占主导,且域语义约束在所有测试方法下保持迭代不变——指向检索而非额外计算是下一步研究前沿。在该基准之上,一个基于验证引导的迭代框架,配备约束意识的自适应提示,持续超越在测试目标模型上的提示工程上限,证明该基准的细粒度信号驱动方法开发。数据、代码和可重复性工具包在论文发布时同步发布于https://github.com/AI4DataSynth/GraphInstruct_formal

英文摘要:

Graph-structured data underpins applications from citation analysis and social-network modeling to molecular design and knowledge-graph construction, and Large Language Models (LLMs) are increasingly used as prompt-driven graph synthesizers. Classical graph-generation reviews catalog deep generative models and their evaluation primitives, but predate the LLM era and provide no foundation for evaluating instruction-following graph synthesis. Recent LLM-era benchmarks evaluate models along graph-type or task-domain axes; such organizations, however, average over structural complexity and cannot localize where in the complexity spectrum an LLM breaks down. To close this diagnostic gap, we introduce GraphInstruct, a progressive-complexity benchmark that stratifies LLM graph generation into six complexity levels and five evaluation dimensions, paired with 800 hand-authored instructions, 1,582 algorithmically synthesized reference solutions, and a 12-LLM capability evaluation across 45 (model, strategy) configurations. We find that discriminative power peaks at multi-constraint composition rather than reasoning depth, that no single prompting strategy dominates across levels or model families, and that domain-semantic constraints remain iteration-invariant under all tested methods -- pointing to retrieval rather than additional compute as the next research frontier. Atop the benchmark, a verification-guided iterative framework with constraint-aware adaptive prompting consistently surpasses the prompt-engineering ceiling on tested target models, demonstrating that the benchmark's fine-grained signals drive method development. Data, code, and reproducibility artifacts are released alongside the paper at https://github.com/AI4DataSynth/GraphInstruct_formal

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑