Semantic segmentation takes pivotal roles in various applications such as autonomous driving and medical image analysis. When deploying segmentation models in practice, it is critical to test their behaviors in varied and complex scenes in advance. In this paper, we construct an automatic data generation pipeline Gen4Seg to stress-test semantic segmentation models by generating various challenging samples with different attribute changes. Beyond previous evaluation paradigms focusing solely on global weather and style transfer, we investigate variations in both appearance and geometry attributes at the object and image level. These include object color, material, size, position, as well as image-level variations such as weather and style. To achieve this, we propose to edit visual attributes of existing real images with precise control of structural information, empowered by diffusion models. In this way, the existing segmentation labels can be reused for the edited images, which greatly reduces the labor costs. Using our pipeline, we construct two new benchmarks, Pascal-EA and COCO-EA. We benchmark a wide variety of semantic segmentation models, spanning from closed-set models to open-vocabulary large models. We have several key findings: 1) advanced open-vocabulary models do not exhibit greater robustness compared to closed-set methods under geometric variations; 2) data augmentation techniques, such as CutOut and CutMix, are limited in enhancing robustness against appearance variations; 3) our pipeline can also be employed as a data augmentation tool and improve both in-distribution and out-of-distribution performances. Our work suggests the potential of generative models as effective tools for automatically analyzing segmentation models, and we hope our findings will assist practitioners and researchers in developing more robust and reliable segmentation models.
FD-DB: Frequency-Decoupled Dual-Branch Network for Unpaired Synthetic-to-Real Domain Translation
State Key Laboratory of Fluid Power and Mechatronic System, School of Mechanical Engineering, Zhejiang University（流体动力与机械系统国家重点实验室，机械工程学院，浙江大学）；Zhejiang Key Laboratory of Advanced Manufacturing Technology, School of Mechanical Engineering, Zhejiang University（浙江省先进制造技术重点实验室，机械工程学院，浙江大学）
We present a neural rendering framework for simultaneous view synthesis and appearance editing of a scene from multi-view images captured under known environment illumination. Existing approaches either achieve view synthesis alone or view synthesis along with relighting, without direct control over the scene's appearance. Our approach explicitly disentangles the appearance and learns a lighting representation that is independent of it. Specifically, we independently estimate the BRDF and use it to learn a lighting-only representation of the scene. Such disentanglement allows our approach to g…
East China Normal University（华东师范大学）；Xiaohongshu Inc.（小红书）；KLATASDS, MOE, China（中国教育部统计与数据科学重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；University of Oxford（牛津大学）；Shanghai Jiao Tong University（上海交通大学）；University of Science and Technology of China（中国科学技术大学）；Nanjing University（南京大学）
Relative radiometric normalization (RRN) mosaicking among multiple remote sensing images is crucial for the downstream tasks, including map-making, image recognition, semantic segmentation, and change detection. However, there are often seam lines on the mosaic boundary and radiometric contrast left, especially in complex scenarios, making the appearance of mosaic images unsightly and reducing the accuracy of the latter classification/recognition algorithms. This paper renders a novel automatical approach to eliminate seam lines in complex RRN mosaicking scenarios. It utilizes the histogram ma…
Southeast University（东南大学）；Max Planck Institute for Informatics（马克斯·普朗克信息研究所）；Shanghai Jiao Tong University（上海交通大学）；StepFun；University of California, Berkeley（加州大学伯克利分校）；University of California, Merced（加州大学默塞德分校）
The University of Hong Kong（香港大学）；JD Explore Academy（京东探索研究院）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）
Polyp-E: Benchmarking the Robustness of Deep Segmentation Models via Polyp Editing
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；The Fifth Medical Center, Chinese PLA General Hospital（中国人民解放军总医院第五医学中心）；Children’s Hospital of Zhejiang University School of Medicine（浙江大学医学院附属儿童医院）
ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motions
ByteDance Seed（字节跳动种子）；University of Southern California（南加州大学）；University of Tokyo（东京大学）；University of California Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；University of California Los Angeles（加州大学洛杉矶分校）
Recent advances in neural radiance fields have enabled the high-fidelity 3D reconstruction of complex scenes for novel view synthesis. However, it remains underexplored how the appearance of such representations can be efficiently edited while maintaining photorealism. In this work, we present PaletteNeRF, a novel method for photorealistic appearance editing of neural radiance fields (NeRF) based on 3D color decomposition. Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases (i.e., 3D segmentations defined by a group of NeRF-type functions) tha…
Alibaba Group（阿里巴巴集团）；Zhejiang University（浙江大学）；Inst. of Comput. Tech., CAS（中国科学院计算技术研究所）
Recent studies have shown that higher accuracy on ImageNet usually leads to better robustness against different corruptions. Therefore, in this paper, instead of following the traditional research paradigm that investigates new out-of-distribution corruptions or perturbations deep models may encounter, we conduct model debugging in in-distribution data to explore which object attributes a model may be sensitive to. To achieve this goal, we create a toolkit for object editing with controls of backgrounds, sizes, positions, and directions, and create a rigorous benchmark named ImageNet-E(diting)…
DriveEditor: A Unified 3D Information-Guided Framework for Controllable Object Editing in Driving Scenes
Huazhong University of Science and Technology（华中科技大学）；National Key Laboratory of Multispectral Information Intelligent Processing Technology（多光谱信息智能处理技术国家重点实验室）；Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Beijing Jiaotong University（北京交通大学）；University of Chinese Academy of Sciences（中国科学院大学）；City University of Hong Kong（香港城市大学）
Recent methods for synthesizing 3D-aware face images have achieved rapid development thanks to neural radiance fields, allowing for high quality and fast inference speed. However, existing solutions for editing facial geometry and appearance independently usually require retraining and are not optimized for the recent work of generation, thus tending to lag behind the generation process. To address these issues, we introduce NeRFFaceEditing, which enables editing and decoupling geometry and appearance in the pretrained tri-plane-based neural radiance field while retaining its high quality and …
MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion
Fudan University（复旦大学）；Shanghai Collaborative Innovation Center of Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Microsoft Research Asia（微软亚洲研究院）；Carnegie Mellon University（卡内基梅隆大学）；Huya Inc.（虎牙公司）
WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；University of Maryland, College Park（马里兰大学学院公园分校）；Zhejiang University（浙江大学）
Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing
Xidian University（西安电子科技大学）；School of Artificial Intelligence（人工智能学院）；Key Laboratory of Intelligent Perception and Image Understanding of the Ministry of Education of China（智能感知与图像理解教育部重点实验室）；International Research Center of Intelligent Perception and Computation（智能感知与计算国际研究中心）
LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs
Institute of Image Communication and Network Engineering, Shanghai JiaoTong University（上海交通大学图像通信与网络工程研究所）；University of Electronic and Science Technology of China（电子科技大学）；Nanyang Technological University（南洋理工大学）
Recently, a surge of advanced facial editing techniques have been proposed that leverage the generative power of a pre-trained StyleGAN. To successfully edit an image this way, one must first project (or invert) the image into the pre-trained generator's domain. As it turns out, however, StyleGAN's latent space induces an inherent tradeoff between distortion and editability, i.e. between maintaining the original appearance and convincingly altering some of its attributes. Practically, this means it is still challenging to apply ID-preserving facial latent-space editing to faces which are out o…
