arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.11050cs.CV
收藏

Endora:视频生成模型作为内窥镜模拟器

Endora: Video Generation Models as Endoscopy Simulators

The Chinese University of Hong Kong(香港中文大学);Massachusetts Institute of Technology(麻省理工学院);Centre for Artificial Intelligence and Robotics(人工智能与机器人中心);Shanghai AI Lab(上海人工智能实验室)

Chenxin Li, Hengyu Liu, Yifan Liu, Brandon Y. Feng, Wuyang Li, Xinyu Liu, Zhen Chen, Jing Shao, Yixuan Yuan

AI总结:

Endora提出一种结合时空视频变换器与二维视觉先验的生成模型,用于模拟临床内窥镜视频,并首个建立内窥镜模拟基准,在视觉质量和下游任务上超越现有方法。

摘要:

生成模型有望彻底改变医学教育、机器人辅助手术和机器学习的数据增强。尽管在生成二维医学图像方面取得了进展,但临床视频生成这一复杂领域在很大程度上仍未得到开发。本文介绍了Endora,一种生成模拟临床内窥镜场景的医学视频的创新方法。我们提出了一种新颖的生成模型设计,该设计将精心构建的时空视频变换器与先进的二维视觉基础模型先验相结合,在视频生成过程中显式建模时空动态。我们还率先推出了首个用于内窥镜模拟的视频生成模型公共基准,并为此改编了现有的最先进方法。Endora在生成内窥镜视频方面展现了卓越的视觉质量,在广泛测试中超越了最先进的方法。此外,我们探索了这种内窥镜模拟器如何赋能下游视频分析任务,甚至生成具有多视图一致性的三维医学场景。总之,Endora标志着生成式人工智能在临床内窥镜研究部署中的显著突破,为医学内容生成的进一步进展奠定了重要基础。更多详情,请访问我们的项目页面:https://endora-medvidgen.github.io/。

英文摘要:

Generative models hold promise for revolutionizing medical education, robot-assisted surgery, and data augmentation for machine learning. Despite progress in generating 2D medical images, the complex domain of clinical video generation has largely remained untapped.This paper introduces \model, an innovative approach to generate medical videos that simulate clinical endoscopy scenes. We present a novel generative model design that integrates a meticulously crafted spatial-temporal video transformer with advanced 2D vision foundation model priors, explicitly modeling spatial-temporal dynamics during video generation. We also pioneer the first public benchmark for endoscopy simulation with video generation models, adapting existing state-of-the-art methods for this endeavor.Endora demonstrates exceptional visual quality in generating endoscopy videos, surpassing state-of-the-art methods in extensive testing. Moreover, we explore how this endoscopy simulator can empower downstream video analysis tasks and even generate 3D medical scenes with multi-view consistency. In a nutshell, Endora marks a notable breakthrough in the deployment of generative AI for clinical endoscopy research, setting a substantial stage for further advances in medical content generation. For more details, please visit our project page: https://endora-medvidgen.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑