构建一个思维宫殿：为有效长视频分析构建基于环境的语义图谱
Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs
AI总结：
VideoMindPalace通过构建环境基础的语义图谱，提升长视频分析中空间-时间连贯性和类人推理能力。
摘要：
利用大视觉语言模型进行长视频理解面临挑战，即需要在有限的上下文窗口内分析时间分散但空间集中的关键时刻。在本工作中，我们引入了VideoMindPalace，一种受“思维宫殿”启发的新框架，将关键视频时刻组织成拓扑结构化的语义图。VideoMindPalace通过（i）手-物体跟踪和交互、（ii）表示特定重复活动区域的聚类活动区以及（iii）环境布局映射来组织关键信息，使LLMs能够通过自然语言解析提供基于空间-时间及3D上下文的深入洞察。此外，我们提出了Video MindPalace基准（VMB），以评估类人推理能力，包括空间定位、时间推理和布局感知的顺序理解。在VMB和已建立的视频问答数据集上进行评估，包括EgoSchema、NExT-QA、IntentQA以及Active Memories Benchmark，VideoMindPalace在空间-时间连贯性和类人推理方面表现出显著提升，推动了VLMs在长视频分析能力的发展。
英文摘要：
Long-form video understanding with Large Vision Language Models is challenged by the need to analyze temporally dispersed yet spatially concentrated key moments within limited context windows. In this work, we introduce VideoMindPalace, a new framework inspired by the "Mind Palace", which organizes critical video moments into a topologically structured semantic graph. VideoMindPalace organizes key information through (i) hand-object tracking and interaction, (ii) clustered activity zones representing specific areas of recurring activities, and (iii) environment layout mapping, allowing natural language parsing by LLMs to provide grounded insights on spatio-temporal and 3D context. In addition, we propose the Video MindPalace Benchmark (VMB), to assess human-like reasoning, including spatial localization, temporal reasoning, and layout-aware sequential understanding. Evaluated on VMB and established video QA datasets, including EgoSchema, NExT-QA, IntentQA, and the Active Memories Benchmark, VideoMindPalace demonstrates notable gains in spatio-temporal coherence and human-aligned reasoning, advancing long-form video analysis capabilities in VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
