LABRAD-OR：用于动态手术室精确双峰推理的轻量级记忆场景图
LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Rooms
摘要：
现代手术是在复杂和动态的环境中进行的，包括医务人员、患者和设备之间不断变化的互动。因此，手术室（OR）的整体建模是一项具有挑战性但必不可少的任务，有可能优化手术团队的表现，并有助于开发新的手术技术以改善患者的预后。手术场景的整体表示为语义场景图（SGG），其中实体表示为节点，它们之间的关系表示为边，是细粒度语义OR理解的一个有前途的方向。我们建议，第一次，使用时间信息更准确和一致的整体或建模。具体来说，我们引入记忆场景图，其中先前时间步的场景图充当引导当前预测的时间表示。我们设计了一个端到端的架构，智能地融合了我们的轻量级内存场景图的时间信息与点云和图像的视觉信息。我们在4D-OR数据集上评估了我们的方法，并证明了整合时间性会导致更准确和一致的结果，在宏F1中实现了+5%的增加和0.88的新SOTA。这项工作开辟了用记忆场景图表示整个手术历史的道路，并提高了对手术室的整体理解。引入场景图作为记忆表示可以为许多时间理解任务提供有价值的工具。
英文摘要：
Modern surgeries are performed in complex and dynamic settings, including ever-changing interactions between medical staff, patients, and equipment. The holistic modeling of the operating room (OR) is, therefore, a challenging but essential task, with the potential to optimize the performance of surgical teams and aid in developing new surgical technologies to improve patient outcomes. The holistic representation of surgical scenes as semantic scene graphs (SGG), where entities are represented as nodes and relations between them as edges, is a promising direction for fine-grained semantic OR understanding. We propose, for the first time, the use of temporal information for more accurate and consistent holistic OR modeling. Specifically, we introduce memory scene graphs, where the scene graphs of previous time steps act as the temporal representation guiding the current prediction. We design an end-to-end architecture that intelligently fuses the temporal information of our lightweight memory scene graphs with the visual information from point clouds and images. We evaluate our method on the 4D-OR dataset and demonstrate that integrating temporality leads to more accurate and consistent results achieving an +5% increase and a new SOTA of 0.88 in macro F1. This work opens the path for representing the entire surgery history with memory scene graphs and improves the holistic understanding in the OR. Introducing scene graphs as memory representations can offer a valuable tool for many temporal understanding tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
