arXiv 2605.112082026-05-16cs.CV

Hi-GaTA：用于外科视频报告生成的分层门控时间聚合适配器

Hi-GaTA: Hierarchical Gated Temporal Aggregation Adapter for Surgical Video Report Generation

School of Engineering, College of Engineering and Physical Sciences, University of Birmingham, Birmingham, UK（英国伯明翰大学工程学院）；School of Computer Science, University of Birmingham, Birmingham, UK（英国伯明翰大学计算机科学学院）；Department of Applied Health Sciences, University of Birmingham, Birmingham, UK（英国伯明翰大学应用健康科学系）

Kedi Sun, Chaohui Dang, Yue Feng, James Glasbey, Theodoros N. Arvanitis, Le Zhang

备注：: 11 pages, 2 figures

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Hi-GaTA框架，通过时间聚合压缩长视频序列生成LLM兼容的视觉前缀令牌，结合预训练的外科专用视频编码器和LoRA微调，实现高质量外科报告生成。

摘要：

自动化、临床级的外科手术评估报告可减少文档负担并提供客观反馈，但面临视频时空表示与语言推理对齐困难及高质量隐私数据稀缺的挑战。为此，我们建立包含214个高质量模拟外科视频及外科医生撰写的评估报告的基准。基于此资源，我们提出包含Hi-GaTA的感知-对齐-推理框架，其中Hi-GaTA是一种新型轻量级时间适配器，通过短到长范围时间聚合高效压缩长视频序列为紧凑的LLM兼容视觉前缀令牌。为实现稳健的视觉感知，我们预训练了Sur40k，一种针对外科专用的ViViT风格视频编码器，在40,000分钟的公开外科视频上进行预训练以捕捉细粒度的时空手术先验。Hi-GaTA采用带有文本条件双交叉注意力的时间金字塔，并通过跨层门控融合和递增深度策略提高多尺度一致性。最后，我们使用LoRA微调LLM主干以在有限监督下实现连贯且风格一致的外科报告生成。实验表明，我们的方法在整体性能上最佳，且在强大的多模态大语言模型（MLLM）基线中表现出一致的优势。消融研究进一步验证了每个提出组件的有效性。

英文摘要：

Automated, clinician-grade assessment reports for surgical procedures could reduce documentation burden and provide objective feedback, yet remain challenging due to the difficulty of aligning dense spatio-temporal video representations with language-based reasoning and the scarcity of high-quality, privacy-preserving datasets. To address this gap, we establish a benchmark comprising 214 high-quality simulated surgical videos paired with surgeon-authored evaluation reports. Building on this resource, we propose a Perception-Alignment-Reasoning framework for surgical video report generation, featuring Hi-GaTA, a novel lightweight temporal adapter that efficiently compresses long video sequences into compact, LLM-compatible visual prefix tokens through short-to-long-range temporal aggregation. For robust visual perception, we pretrain Sur40k, a surgical-specific ViViT-style video encoder on 40,000 minutes of public surgical videos to capture fine-grained spatio-temporal procedural priors. Hi-GaTA employs a temporal pyramid with text-conditioned dual cross-attention, and improves multi-scale consistency through cross-level gated fusion and an increasing-depth strategy. Finally, we fine-tune the LLM backbone using LoRA to enable coherent and stylistically consistent surgical report generation under limited supervision. Experiments show our approach achieves the best overall performance, with consistent gains over strong Multimodal Large Language Model (MLLM) baselines. Ablation studies further validate the effectiveness of each proposed component.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。