Research in unpaired video translation has mainly focused on short-term temporal consistency by conditioning on neighboring frames. However for transfer from simulated to photorealistic sequences, available information on the underlying geometry offers potential for achieving global consistency across views. We propose a novel approach which combines unpaired image translation with neural rendering to transfer simulated to photorealistic surgical abdominal scenes. By introducing global learnable textures and a lighting-invariant view-consistency loss, our method produces consistent translations of arbitrary views and thus enables long-term consistent video synthesis. We design and test our model to generate video sequences from minimally-invasive surgical abdominal scenes. Because labeled data is often limited in this domain, photorealistic data where ground truth information from the simulated domain is preserved is especially relevant. By extending existing image-based methods to view-consistent videos, we aim to impact the applicability of simulated training and evaluation environments for surgical applications. Code and data: http://opencas.dkfz.de/video-sim2real.
SurgicaL-CD: Generating Surgical Images via Unpaired Image Translation with Latent Consistency Diffusion Models
NCT/UCC Dresden（德累斯顿癌症中心/癌症联盟）；DKFZ Heidelberg（海德堡德国癌症研究中心）；Faculty of Medicine and University Hospital Carl Gustav Carus, TUD Dresden University of Technology（德累斯顿工业大学卡尔·古斯塔夫·卡鲁斯医学院及大学医院）；HZDR Dresden（德累斯顿亥姆霍兹中心）；The Centre for Tactile Internet with Human-in-the-Loop (CeTI), TUD Dresden University of Technology（德累斯顿工业大学人机交互触觉互联网中心）
Exploring Semantic Consistency in Unpaired Image Translation to Generate Data for Surgical Applications
NCT/UCC Dresden（德累斯顿国家肿瘤疾病中心/德累斯顿肿瘤中心）；SECAI, TU Dresden（德累斯顿工业大学SECAI中心）；Faculty of medicine & University Hospital TU Dresden（德累斯顿工业大学医学院及大学医院）；CeTI, TU Dresden（德累斯顿工业大学CeTI中心）
The CycleGAN framework allows for unsupervised image-to-image translation of unpaired data. In a scenario of surgical training on a physical surgical simulator, this method can be used to transform endoscopic images of phantoms into images which more closely resemble the intra-operative appearance of the same surgical target structure. This can be viewed as a novel augmented reality approach, which we coined Hyperrealism in previous work. In this use case, it is of paramount importance to display objects like needles, sutures or instruments consistent in both domains while altering the style t…
Indian Institute of Technology Madras（印度理工学院马德拉斯分校）
Magnetic Resonance (MR) Imaging and Computed Tomography (CT) are the primary diagnostic imaging modalities quite frequently used for surgical planning and analysis. A general problem with medical imaging is that the acquisition process is quite expensive and time-consuming. Deep learning techniques like generative adversarial networks (GANs) can help us to leverage the possibility of an image to image translation between multiple imaging modalities, which in turn helps in saving time and cost. These techniques will help to conduct surgical planning under CT with the feedback of MRI information…
EndoLive: Real-Time Style Transfer for Endoscopic Endonasal Skull Base Surgical Video
University of Pittsburgh（匹兹堡大学）；University of Pittsburgh School of Computing and Information（匹兹堡大学计算与信息学院）；School of Computing and Information（计算与信息学院）
Our work explores temporal self-supervision for GAN-based video generation tasks. While adversarial training successfully yields generative models for a variety of areas, temporal relationships in the generated data are much less explored. Natural temporal changes are crucial for sequential generation tasks, e.g. video super-resolution and unpaired video translation. For the former, state-of-the-art methods often favor simpler norm losses such as $L^2$ over adversarial training. However, their averaging nature easily leads to temporally smooth results with an undesirable lack of spatial detail…
Surgical tool segmentation in endoscopic images is an important problem: it is a crucial step towards full instrument pose estimation and it is used for integration of pre- and intra-operative images into the endoscopic view. While many recent approaches based on convolutional neural networks have shown great results, a key barrier to progress lies in the acquisition of a large number of manually-annotated images which is necessary for an algorithm to generalize and work well in diverse surgical scenarios. Unlike the surgical image data itself, annotations are difficult to acquire and may be o…
Unpaired video-to-video translation aims to translate videos between a source and a target domain without the need of paired training data, making it more feasible for real applications. Unfortunately, the translated videos generally suffer from temporal and semantic inconsistency. To address this, many existing works adopt spatiotemporal consistency constraints incorporating temporal information based on motion estimation. However, the inaccuracies in the estimation of motion deteriorate the quality of the guidance towards spatiotemporal consistency, which leads to unstable translation. In th…
Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions
Center for Humanoid Research, Korea Institute of Science and Technology（韩国科学技术研究院仿人机器人研究中心）；School of Electrical and Electronic Engineering, Yonsei University（延世大学电气与电子工程学院）
SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB)（电子、信息与生物工程系）；Politecnico di Milano（米兰理工大学）；IRCCS Humanitas Research Hospital（IRCCS人类itas研究医院）；UCL Hawkes Institute and Department of Computer Science（UCL Hawkes研究所和计算机科学系）；University College London（伦敦大学学院）；University of Manchester（曼彻斯特大学）
Beijing Film Academy（北京电影学院）；Tel-Aviv University（特拉维夫大学）；Peking University（北京大学）；The Hebrew University of Jerusalem（耶路撒冷希伯来大学）
Transferring the motion style from one animation clip to another, while preserving the motion content of the latter, has been a long-standing problem in character animation. Most existing data-driven approaches are supervised and rely on paired data, where motions with the same content are performed in different styles. In addition, these approaches are limited to transfer of styles that were seen during training. In this paper, we present a novel data-driven framework for motion style transfer, which learns from an unpaired collection of motions with style labels, and enables transferring mot…
ACT-Net: Anchor-context Action Detection in Surgery Videos
University of Birmingham（伯明翰大学）；Southern University of Science and Technology（南方科技大学）；Research Institute of Trustworthy Autonomous Systems（可信自主系统研究院）；National University of Singapore（新加坡国立大学）；Third medical center of Chinese PLAGH（中国人民解放军总医院第三医学中心）；Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局高性能计算研究所）
F2FLDM: Latent Diffusion Models with Histopathology Pre-Trained Embeddings for Unpaired Frozen Section to FFPE Translation
University of Utah（犹他大学）；The Catholic University of Korea College of Medicine（韩国天主教大学医学院）；Huntsman Cancer Institute（亨茨曼癌症研究所）；Scientific Computing and Imaging Institute（科学计算与成像研究所）
We propose a general purpose approach to detect landmarks with improved temporal consistency, and personalization. Most sparse landmark detection methods rely on laborious, manually labelled landmarks, where inconsistency in annotations over a temporal volume leads to sub-optimal landmark learning. Further, high-quality landmarks with personalization is often hard to achieve. We pose landmark detection as an image translation problem. We capture two sets of unpaired marked (with paint) and unmarked videos. We then use a generative adversarial network and cyclic consistency to predict deformati…
HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition
School of Information Science and Technology, University of Science and Technology of China（中国科学技术大学信息科学技术学院）；Department of General Surgery, Beijing Friendship Hospital, Capital Medical University（首都医科大学附属北京友谊医院普通外科）；Chinese PLA General Hospital（中国人民解放军总医院）
A spatio-temporal network for video semantic segmentation in surgical videos
Medtronic Digital Surgery（美敦力数字外科）；Wellcome/EPSRC Centre for Internventional and Surgical Sciences, University College London（伦敦大学学院惠康/工程与物理科学研究委员会介入与外科学中心）
University of Washington（华盛顿大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Deep learning-based methods have achieved promising results on surgical instrument segmentation. However, the high computation cost may limit the application of deep models to time-sensitive tasks such as online surgical video analysis for robotic-assisted surgery. Moreover, current methods may still suffer from challenging conditions in surgical images such as various lighting conditions and the presence of blood. We propose a novel Multi-frame Feature Aggregation (MFFA) module to aggregate video frame features temporally and spatially in a recurrent mode. By distributing the computation load…
TemporalDoRA: Temporal PEFT for Robust Surgical Video Question Answering
Dipartimento di Elettronica, Informazione e Bioingegneria (DEIB), Politecnico di Milano（电子信息与生物工程系（DEIB），米兰理工学院）；IRCCS Humanitas Research Hospital（人类学研究所医院）；UCL Hawkes Institute and Department of Computer Science, University College London（伦敦大学学院 Hawkes 研究所和计算机科学系）；Division of Informatics, Imaging and Data Science, University of Manchester（信息学、影像学与数据科学系，曼彻斯特大学）
SurgNarrator: A Generative Retrieval Framework for Surgical Video Understanding
University of Liverpool（利物浦大学）；City University of Hong Kong（香港城市大学）；University of Oxford（牛津大学）；University of Strasbourg（斯特拉斯堡大学）；IHU Strasbourg（斯特拉斯堡大学医院研究所）；Imperial College London（帝国理工学院）
Sound Source Localization for Spatial Mapping of Surgical Actions in Dynamic Scenes
ROCS（罗克辛研究所）；Balgrist University Hospital（巴尔格里斯大学医院）；University of Zurich（苏黎世大学）；Dept. of Orthopedics（骨科部门）；Computer Vision and Geometry Group（计算机视觉与几何组）；ETH Zurich（苏黎世联邦理工学院）
LoViT: Long Video Transformer for Surgical Phase Recognition
Department of Surgical & Interventional Engineering, King’s College London（外科与介入工程系，伦敦国王学院）；Department of Peter Gorer（彼得·戈尔德系）；Department of Immunobiology, King’s College London（免疫生物学系，伦敦国王学院）
Amazon Lab126（亚马逊Lab126）；School of Computing, University of Utah（犹他大学计算学院）
Recovering 3D human pose from 2D joints is a highly unconstrained problem. We propose a novel neural network framework, PoseNet3D, that takes 2D joints as input and outputs 3D skeletons and SMPL body model parameters. By casting our learning approach in a student-teacher framework, we avoid using any 3D data such as paired/unpaired 3D data, motion capture sequences, depth images or multi-view images during training. We first train a teacher network that outputs 3D skeletons, using only 2D poses for training. The teacher network distills its knowledge to a student network that predicts 3D pose …
Revisiting the Evaluation Bias Introduced by Frame Sampling Strategies in Surgical Video Segmentation Using SAM2
1 Center for Biosystems；Biotech Data Science, Ghent University Global Campus, Incheon, Republic of Korea 2 IDLab, ELIS, Ghent University, Ghent, Belgium 3 Department of GI Surgery, Ghent University Hospital, Ghent, Belgium 4 Department of Human Structure；Repair, Ghent University, Ghent, Belgium 5 Department of HPB Surgery \& Liver Transplantation, Ghent University Hospital, Ghent, Belgium 6 Department of Mathematics, Computer Science；Statistics,\ University, Ghent, Belgium ( )
National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心（NCT））；TU Dresden（德累斯顿工业大学）；Technische Universität Dresden（德累斯顿工业大学）
Estimating the remaining surgery duration (RSD) during surgical procedures can be useful for OR planning and anesthesia dose estimation. With the recent success of deep learning-based methods in computer vision, several neural network approaches have been proposed for fully automatic RSD prediction based solely on visual data from the endoscopic camera. We investigate whether RSD prediction can be improved using unsupervised temporal video segmentation as an auxiliary learning task. As opposed to previous work, which presented supervised surgical phase recognition as auxiliary task, we avoid t…
ICube, University of Strasbourg, CNRS（斯特拉斯堡大学 ICube 实验室、法国国家科学研究中心）；Fondazione Policlinico Universitario A. Gemelli IRCCS（A. Gemelli 大学综合医院基金会 IRCCS）；IHU-Strasbourg, Institute of Image-Guided Surgery（斯特拉斯堡 IHU 影像引导外科研究所）；Institute for Research against Digestive Cancer (IRCAD)（消化道癌症研究院（IRCAD））
A major obstacle to building models for effective semantic segmentation, and particularly video semantic segmentation, is a lack of large and well annotated datasets. This bottleneck is particularly prohibitive in highly specialized and regulated fields such as medicine and surgery, where video semantic segmentation could have important applications but data and expert annotations are scarce. In these settings, temporal clues and anatomical constraints could be leveraged during training to improve performance. Here, we present Temporally Constrained Neural Networks (TCNN), a semi-supervised fr…
SegCol Challenge: Semantic Segmentation for Tools and Fold Edges in Colonoscopy data
UCL Hawkes Institute, University College London（伦敦大学学院霍克斯学院）；Department of Computer Science, University College London（伦敦大学学院计算机科学系）；Medtronic（美敦力）；Department of Computer Science, University Liverpool（利物浦大学计算机科学系）
SEDMamba: Enhancing Selective State Space Modelling with Bottleneck Mechanism and Fine-to-Coarse Temporal Fusion for Efficient Error Detection in Robot-Assisted Surgery
UCL Hawkes Institute（伦敦大学学院霍克斯研究所）；University College London（伦敦大学学院）；Griffin Institute（格里芬研究所）；Northwick Park and St Mark’s Hospital（诺斯威克公园与圣马克医院）
AICFVE Beijing Film Academy（北京电影学院AICFVE）；Shandong University（山东大学）；Microsoft Research Asia（微软亚洲研究院）；Hebrew University of Jerusalem（耶路撒冷希伯来大学）；Peking University（北京大学）；Tel-Aviv University（特拉维夫大学）
We present a new video-based performance cloning technique. After training a deep generative network using a reference video capturing the appearance and dynamics of a target actor, we are able to generate videos where this actor reenacts other performances. All of the training data and the driving performances are provided as ordinary video segments, without motion capture or depth information. Our generative model is realized as a deep neural network with two branches, both of which train the same space-time conditional generator, using shared weights. One branch, responsible for learning to…
In this paper, we tackle the problem of human motion transfer, where we synthesize novel motion video for a target person that imitates the movement from a reference video. It is a video-to-video translation task in which the estimated poses are used to bridge two domains. Despite substantial progress on the topic, there exist several problems with the previous methods. First, there is a domain gap between training and testing pose sequences--the model is tested on poses it has not seen during training, such as difficult dancing moves. Furthermore, pose detection errors are inevitable, making …
Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and \emph{neural neural textures}. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic-looking learnable neural textures. By s…
MedScope: Incentivizing "Think with Videos" for Clinical Reasoning via Coarse-to-Fine Tool Calling
College of Health Science and Technology, Shanghai Jiao Tong University School of Medicine, Shanghai, China（上海交通大学医学院健康科学与技术学院）；Fudan University, Shanghai, China（复旦大学）；Shanghai Innovation Institute, Shanghai, China（上海创新研究院）；Tsinghua University, Beijing, China（清华大学）；Shanghai Artificial Intelligence Laboratory, Shanghai, China（上海人工智能实验室）；The Chinese University of Hong Kong, Hong Kong, China（香港中文大学）；Ruijin Hospital, Shanghai Jiaotong University, Shanghai, China（上海交通大学瑞金医院）
Temporal activity localization in long videos is an important problem. The cost of obtaining frame level label for long Wireless Capsule Endoscopy (WCE) videos is prohibitive. In this paper, we propose an end-to-end temporal abnormality localization for long WCE videos using only weak video level labels. Physicians use Capsule Endoscopy (CE) as a non-surgical and non-invasive method to examine the entire digestive tract in order to diagnose diseases or abnormalities. While CE has revolutionized traditional endoscopy procedures, a single CE examination could last up to 8 hours generating as muc…
HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding
German Cancer Research Center (DKFZ) Heidelberg（德国癌症研究中心（DKFZ）海德堡）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心（NCT））；Helmholtz Imaging（亥姆霍兹影像）；Stanford University（斯坦福大学）；University of Pennsylvania（宾夕法尼亚大学）
