Universal style transfer (UST) infuses styles from arbitrary reference images into content images. Existing methods, while enjoying many practical successes, are unable of explaining experimental observations, including different performances of UST algorithms in preserving the spatial structure of content images. In addition, methods are limited to cumbersome global controls on stylization, so that they require additional spatial masks for desired stylization. In this work, we provide a systematic Fourier analysis on a general framework for UST. We present an equivalent form of the framework in the frequency domain. The form implies that existing algorithms treat all frequency components and pixels of feature maps equally, except for the zero-frequency component. We connect Fourier amplitude and phase with Gram matrices and a content reconstruction loss in style transfer, respectively. Based on such equivalence and connections, we can thus interpret different structure preservation behaviors between algorithms with Fourier phase. Given the interpretations we have, we propose two manipulations in practice for structure preservation and desired stylization. Both qualitative and quantitative experiments demonstrate the competitive performance of our method against the state-of-the-art methods. We also conduct experiments to demonstrate (1) the abovementioned equivalence, (2) the interpretability based on Fourier amplitude and phase and (3) the controllability associated with frequency components.
DepthPilot: From Controllability to Interpretability in Colonoscopy Video Generation
College of Biomedical Engineering, Fudan University, Shanghai 200433, China(复旦大学生物医学工程学院);Key Laboratory of Medical Imaging Computing and Computer Assisted Intervention of Shanghai, Shanghai 200032, China(上海医学影像计算与计算机辅助干预重点实验室);Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai 200032, China(复旦大学中山医院内窥镜研究所);Shanghai Collaborative Innovation Center of Endoscopy, Shanghai 200032, China(上海内窥镜协同创新中心);Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH 44106, USA(凯斯西储大学生物医学工程系);Department of Computer and Data Science, Case Western Reserve University, Cleveland, OH 44106, USA(凯斯西储大学计算机与数据科学系)
3D-StyleGAN: A Style-Based Generative Adversarial Network for Generative Modeling of Three-Dimensional Medical Images
Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院);Massachusetts Institute of Technology(麻省理工学院);Univ. Lille(里尔大学);Inserm(法国国家健康与医学研究院);CHU Lille(里尔大学医院)
Versatile Framework with Semantic and Structural guidance for Image Reconstruction from Brain Activity
State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology(脑认知与脑启发智能技术国家重点实验室);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Future Technology, University of Chinese Academy of Sciences(中国科学院大学未来技术学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
University of Virginia(弗吉尼亚大学);Columbia University(哥伦比亚大学);Vanderbilt University(范德比大学);Adobe Research(Adobe研究);Dolby Laboratories(杜比实验室);Cisco Research(思科研究);University of Southern California(南加州大学);University of Wisconsin-Madison(威斯康星大学麦迪逊分校);University of Oregon(俄勒冈大学);Texas A&M University(德克萨斯大学)