评估大型语言模型中的多跳推理：以化学为中心的案例研究
Evaluating Multi-Hop Reasoning in Large Language Models: A Chemistry-Centric Case Study
AI总结：
本研究提出一个化学领域多跳推理基准，结合NER与知识图谱生成问题，发现即便先进LLM在组合推理上仍有困难，文档检索可提升性能但无法完全消除错误，并展示了可跨领域应用的数据生成流程。
摘要：
在本研究中，我们引入了一个新基准，该基准包含一个精选数据集和一个明确的评估流程，用于评估大型语言模型在化学领域内的组合推理能力。我们设计并验证了一个全自动流程，该流程经领域专家确认，以促进此任务。我们的方法将OpenAI推理模型与命名实体识别（NER）系统相结合，从近期文献中提取化学实体，然后通过外部知识库对这些实体进行增强，以构建一个全面的知识图谱。通过在这些图谱上生成多跳问题，我们在上下文增强和非上下文增强两种设置下评估LLM的性能。我们的实验表明，即使是最先进的模型在多跳组合推理中也面临重大挑战。结果反映了用文档检索增强LLM的重要性，这能对提升其性能产生显著影响。然而，即使检索准确率完美且提供完整上下文，也不能消除推理错误，这凸显了组合推理的复杂性。这项工作不仅对当前LLM的局限性进行了基准测试和强调，还提出了一种新颖的数据生成流程，该流程能够在多个领域生成具有挑战性的推理数据集。总体而言，这项研究增进了我们对计算语言学中推理的理解。
英文摘要：
In this study, we introduced a new benchmark consisting of a curated dataset and a defined evaluation process to assess the compositional reasoning capabilities of large language models within the chemistry domain. We designed and validated a fully automated pipeline, verified by subject matter experts, to facilitate this task. Our approach integrates OpenAI reasoning models with named entity recognition (NER) systems to extract chemical entities from recent literature, which are then augmented with external knowledge bases to form a comprehensive knowledge graph. By generating multi-hop questions across these graphs, we assess LLM performance in both context-augmented and non-context augmented settings. Our experiments reveal that even state-of-the-art models face significant challenges in multi-hop compositional reasoning. The results reflect the importance of augmenting LLMs with document retrieval, which can have a substantial impact on improving their performance. However, even perfect retrieval accuracy with full context does not eliminate reasoning errors, underscoring the complexity of compositional reasoning. This work not only benchmarks and highlights the limitations of current LLMs but also presents a novel data generation pipeline capable of producing challenging reasoning datasets across various domains. Overall, this research advances our understanding of reasoning in computational linguistics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
