用于过程视频问答的神经符号知识推理
Neuro Symbolic Knowledge Reasoning for Procedural Video Question Answering
AI总结：
本文提出KML框架，通过神经符号方法提升过程视频问答的推理能力，结合知识图谱实现可解释的多步骤推理。
摘要：
在本文中，我们提出了知识模块学习（KML）以理解和推理由过程任务所需的模型学习结构化和组合性过程知识。KML是一种神经符号框架，它在知识图中学习关系类别作为神经知识模块，并将它们组合成由大型语言模型（LLMs）生成的可执行推理程序。每个模块编码特定的过程关系，捕捉如何每个实体类型如工具与步骤、每个工具的目的以及每个任务的步骤相关。给定一个基于视频中任务的提问，KML则执行多步骤推理，具有透明且可追溯的中间状态。我们的理论分析证明了KML的两个期望属性。KML满足对建模KG关系作为神经映射的强最优条件，为可推广的过程推理提供了坚实的基础。它还展示了在执行多步骤推理时预期误差的上限。为了评估此模型，我们构建了一个由整合COIN教学视频数据集、COIN本体、ConceptNet常识关系和LLMs的结构化提取组成的大型过程知识图（PKG），并通过专家验证。然后，我们通过在PKG上应用图遍历模板生成问题和答案对，构建了过程知识推理基准（PKR-QA）。实验表明，KML在提高结构化推理性能的同时提供了可解释的逐步跟踪，优于仅LLM和黑盒神经基线。代码可在https://github.com/LUNAProject22/KML上公开获取。
英文摘要：
In this work we present Knowledge Module Learning (KML) to understand and reason over procedural tasks that requires models to learn structured and compositional procedural knowledge. KML is a neurosymbolic framework that learns relation categories within a knowledge graph as neural knowledge modules and composes them into executable reasoning programs generated by large language models (LLMs). Each module encodes a specific procedural relation capturing how each entity type such as tools are related to steps, purpose of each tool, and steps of each task. Given a question conditioned on a task shown in a video, then KML performs multistep reasoning with transparent, traceable intermediate states. Our theoretical analysis demonstrated two desired properties of KML. KML satisfy strong optimal conditions for modelling KG relations as neural mappings, providing strong foundations for generalizable procedural reasoning. It also shows a bound on the expected error when it performs multistep reasoning. To evaluate this model, we construct a large procedural knowledge graph (PKG) consisting of diverse instructional domains by integrating the COIN instructional video dataset, and COIN ontology, commonsense relations from ConceptNet, and structured extractions from LLMs, followed by expert verification. We then generate question and answer pairs by applying graph traversal templates over the PKG, constructing the PKR-QA benchmark for procedural knowledge reasoning. Experiments show that KML improves structured reasoning performance while providing interpretable step-by-step traces, outperforming LLM-only and black-box neural baselines. Code is publicly available at https://github.com/LUNAProject22/KML.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
