Abstract

TL;DR:Endora enables the high-fidelity medical video generation on endoscopy scenes and demonstrates the versatile ability through successful applications in video-based disease diagnosis and 3D surgical scene reconstruction.

Generative models hold promise for revolutionizing medical education, robot-assisted surgery, and data augmentation for machine learning. Despite progress in generating 2D medical images, the complex domain of clinical video generation has largely remained untapped.

This paper introduces Endora, an innovative approach to generate medical videos to simulate clinical endoscopy scenes. We present a novel generative model design that integrates a meticulously crafted spatial-temporal video transformer with advanced 2D vision foundation model priors, explicitly modeling spatial-temporal dynamics during video generation.

We also pioneer the first public benchmark for endoscopy simulation with video generation models, adapting existing state-of-the-art methods for this endeavor. Endora demonstrates exceptional visual quality in generating endoscopy videos, surpassing state-of-the-art methods in extensive testing.

Moreover, we explore how this endoscopy simulator can empower downstream video analysis tasks and even effectively create 3D scenes with multi-view consistency.

In a nutshell, Endora marks a notable breakthrough in the deployment of generative AI for clinical endoscopy research, setting a substantial stage for further advances in medical content generation.

Results on Colonoscopic Dataset

Results on Kvasir-Capsule Dataset

Results on CholecTriplet Dataset

Endora Generating Video as Efficient 3D Creators (Cocurrent with SV3D)

We train a Gaussian Splatting representationon on the sampled videos by Endora and observe the multi-view consistent geometry (shown by rendered RGB and depth maps) as if in the real 3D world.