We introduce OLATverse, a large-scale dataset comprising around 9M images of 765 real-world objects, captured from multiple viewpoints under a diverse set of precisely controlled lighting conditions. While recent advances in object-centric inverse rendering, novel view synthesis and relighting have shown promising results, most techniques still heavily rely on the synthetic datasets for training and small-scale real-world datasets for benchmarking, which limits their realism and generalization. To address this gap, OLATverse offers two key advantages over existing datasets: large-scale coverage of real objects and high-fidelity appearance under precisely controlled illuminations. Specifically, OLATverse contains 765 common and uncommon real-world objects, spanning a wide range of material categories. Each object is captured using 35 DSLR cameras and 331 individually controlled light sources, enabling the simulation of diverse illumination conditions. In addition, for each object, we provide well-calibrated camera parameters, accurate object masks, photometric surface normals, and diffuse albedo as auxiliary resources. We also construct an extensive evaluation set, establishing the first comprehensive real-world object-centric benchmark for inverse rendering and normal estimation. We believe that OLATverse represents a pivotal step toward integrating the next generation of inverse rendering and relighting methods with real-world data. The full dataset, along with all post-processing workflows, will be publicly released at https://vcai.mpi-inf.mpg.de/projects/OLATverse/.
RelightMaster: Precise Video Relighting with Multi-plane Light Images
Multimedia Laboratory, The Chinese University of Hong Kong（香港中文大学多媒体实验室）；Kling Team, Kuaishou Technology（快手技术 Kling 团队）；CPII under InnoHK（创新香港 CPII）；Zhejiang University（浙江大学）；Dalian University of Technology（大连理工大学）
Data seems cheap to get, and in many ways it is, but the process of creating a high quality labeled dataset from a mass of data is time-consuming and expensive. With the advent of rich 3D repositories, photo-realistic rendering systems offer the opportunity to provide nearly limitless data. Yet, their primary value for visual learning may be the quality of the data they can provide rather than the quantity. Rendering engines offer the promise of perfect labels in addition to the data: what the precise camera pose is; what the precise lighting location, temperature, and distribution is; what th…
University of the Witwatersrand（威特沃特斯兰德大学）；Southwest Jiaotong University（西南交通大学）
To perform contingent teaching and be responsive to students' needs during class, lecturers must be able to quickly assess the state of their audience. While effective teachers are able to gauge easily the affective state of the students, as class sizes grow this becomes increasingly difficult and less precise. The Wits Intelligent Teaching System (WITS) aims to assist lecturers with real-time feedback regarding student affect. The focus is primarily on recognising engagement or lack thereof. Student engagement is labelled based on behaviour and postures that are common to classroom settings. …
Xiamen University（厦门大学）；The Chinese University of Hong Kong（香港中文大学）；Zhongshan Hospital affiliated to Xiamen University（厦门大学附属中山医院）；The Hong Kong Polytechnic University（香港理工大学）
We propose a novel shape-aware relation network for accurate and real-time landmark detection in endoscopic submucosal dissection (ESD) surgery. This task is of great clinical significance but extremely challenging due to bleeding, lighting reflection, and motion blur in the complicated surgical environment. Compared with existing solutions, which either neglect geometric relationships among targeting objects or capture the relationships by using complicated aggregation schemes, the proposed network is capable of achieving satisfactory accuracy while maintaining real-time performance by taking…
Geometric Image Editing via Effects-Sensitive In-Context Inpainting with Diffusion Transformers
PRIS, Beijing University of Posts and Telecommunications（北京邮电大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院）；Beijing Institute of Technology（北京理工大学）
VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation
School of Data Science, Fudan University（复旦大学数据科学学院）；Shanghai Innovation Institute（上海创新研究院）；Zhejiang University（浙江大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Westlake University（西湖大学）；School of Data Science and MOE Frontiers Center for Brain Science, Fudan University（复旦大学数据科学学院和脑科学前沿中心）；Fudan ISTBI–ZJNU Algorithm Centre for Brain-inspired Intelligence, Zhejiang Normal University（复旦大学-浙江师范大学脑启发智能算法中心）
Lighting-grounded Video Generation with Renderer-based Agent Reasoning
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University（北京大学计算机学院多媒体信息处理国家重点实验室）；National Engineering Research Center of Visual Technology, School of Computer Science, Peking University（北京大学计算机学院国家视觉技术工程研究中心）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）；OpenBayes Information Technology Co., Ltd.（北京开贝信息技术有限公司）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Cross-Camera Cow Identification via Disentangled Representation Learning
College of Information Science and Engineering, Shandong Agricultural University（信息科学与工程学院，山东农业大学）；Centre for Vision Speech and Signal Processing (CVSSP), University of Surrey（视觉语音与信号处理中心（CVSSP）， Surrey 大学）；Australian Institute for Machine Learning, University of Adelaide（机器学习研究院，阿德莱德大学）
LumiCtrl : Learning Illuminant Prompts for Lighting Control in Personalized Text-to-Image Models
Computer Vision Center, Spain（西班牙计算机视觉中心）；City University of Hong Kong（香港城市大学）；Computer Sciences Department, Universitat Autònoma de Barcelona, Spain（西班牙巴塞罗那自治大学计算机科学系）；Program of Computer Science, City University of Hong Kong (Dongguan)（香港城市大学（东莞）计算机科学项目）
Generative Adversarial Networks (GANs) are able to learn mappings between simple, relatively low-dimensional, random distributions and points on the manifold of realistic images in image-space. The semantics of this mapping, however, are typically entangled such that meaningful image properties cannot be controlled independently of one another. Conditional GANs (cGANs) provide a potential solution to this problem, allowing specific semantics to be enforced during training. This solution, however, depends on the availability of precise labels, which are sometimes difficult or near impossible to…
Tianjin University（天津大学）；Nanyang Technological University（南洋理工大学）；Alibaba Group（阿里巴巴集团）；Cleveland State University（克利夫兰州立大学）
Deep face recognition (FR) has achieved significantly high accuracy on several challenging datasets and fosters successful real-world applications, even showing high robustness to the illumination variation that is usually regarded as a main threat to the FR system. However, in the real world, illumination variation caused by diverse lighting conditions cannot be fully covered by the limited face dataset. In this paper, we study the threat of lighting against FR from a new angle, i.e., adversarial attack, and identify a new task, i.e., adversarial relighting. Given a face image, adversarial re…
GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Rendering
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；State Key Laboratory of Multimedia Information Processing, Peking University（北京大学多媒体信息处理国家重点实验室）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Capturing and labeling camera images in the real world is an expensive task, whereas synthesizing labeled images in a simulation environment is easy for collecting large-scale image data. However, learning from only synthetic images may not achieve the desired performance in the real world due to a gap between synthetic and real images. We propose a method that transfers learned detection of an object position from a simulation environment to the real world. This method uses only a significantly limited dataset of real images while leveraging a large dataset of synthetic images using variation…
Current Generative Adversarial Networks (GANs) produce photorealistic renderings of portrait images. Embedding real images into the latent space of such models enables high-level image editing. While recent methods provide considerable semantic control over the (re-)generated images, they can only generate a limited set of viewpoints and cannot explicitly control the camera. Such 3D camera control is required for 3D virtual and mixed reality applications. In our solution, we use a few images of a face to perform 3D reconstruction, and we introduce the notion of the GAN camera manifold, the key…
An Intelligent Water-Saving Irrigation System Based on Multi-Sensor Fusion and Visual Servoing Control
Xi'an Jiaotong University Faculty of Microelectronics Science（西安交通大学微电子科学与工程学院）；Xi'an Jiaotong University School of Electrical Engineering（西安交通大学电气工程学院）
Transparent objects are a very challenging problem in computer vision. They are hard to segment or classify due to their lack of precise boundaries, and there is limited data available for training deep neural networks. As such, current solutions for this problem employ rigid synthetic datasets, which lack flexibility and lead to severe performance degradation when deployed on real-world scenarios. In particular, these synthetic datasets omit features such as refraction, dispersion and caustics due to limitations in the rendering pipeline. To address this issue, we present SuperCaustics, a rea…
More Clear, More Flexible, More Precise: A Comprehensive Oriented Object Detection benchmark for UAV
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China（多媒体可信感知与高效计算重点实验室，中华人民共和国教育部）；Institute of Artificial Intelligence, Xiamen University（人工智能学院，厦门大学）；School of Informatics, Xiamen University（信息学院，厦门大学）；Fujian Engineering Research Center of Trusted Artificial Intelligence Analysis and Application, Institute of Artificial Intelligence, Xiamen University（福建工程研究中心可信人工智能分析与应用，人工智能学院，厦门大学）；Peng Cheng Laboratory, Shenzhen（鹏城实验室，深圳）
MVTec Software GmbH（MVTec软件有限公司）；Technical University of Munich（慕尼黑工业大学）
We introduce the Densely Segmented Supermarket (D2S) dataset, a novel benchmark for instance-aware semantic segmentation in an industrial domain. It contains 21,000 high-resolution images with pixel-wise labels of all object instances. The objects comprise groceries and everyday products from 60 categories. The benchmark is designed such that it resembles the real-world setting of an automatic checkout, inventory, or warehouse system. The training images only contain objects of a single class on a homogeneous background, while the validation and test sets are much more complex and diverse. To …
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；MIT（麻省理工学院）
2D images are observations of the 3D physical world depicted with the geometry, material, and illumination components. Recovering these underlying intrinsic components from 2D images, also known as inverse rendering, usually requires a supervised setting with paired images collected from multiple viewpoints and lighting conditions, which is resource-demanding. In this work, we present GAN2X, a new method for unsupervised inverse rendering that only uses unpaired images for training. Unlike previous Shape-from-GAN approaches that mainly focus on 3D shapes, we take the first attempt to also reco…
IBM Research - Tokyo, IBM Japan（IBM日本东京研究院）；LeapMind Inc.（LeapMind公司）
Robotic learning in simulation environments provides a faster, more scalable, and safer training methodology than learning directly with physical robots. Also, synthesizing images in a simulation environment for collecting large-scale image data is easy, whereas capturing camera images in the real world is time consuming and expensive. However, learning from only synthetic images may not achieve the desired performance in real environments due to the gap between synthetic and real images. We thus propose a method that transfers learned capability of detecting object position from a simulation …
Developing robot perception systems for recognizing objects in the real-world requires computer vision algorithms to be carefully scrutinized with respect to the expected operating domain. This demands large quantities of ground truth data to rigorously evaluate the performance of algorithms. This paper presents the EasyLabel tool for easily acquiring high quality ground truth annotation of objects at the pixel-level in densely cluttered scenes. In a semi-automatic process, complex scenes are incrementally built and EasyLabel exploits depth change to extract precise object masks at each step. …
Make-it-Real: Unleashing Large Multimodal Model for Painting 3D Objects with Realistic Materials
Fudan University（复旦大学）；Shanghai AI Laboratory（上海人工智能实验室）；Shanghai Jiao Tong University（上海交通大学）；The Chinese University of Hong Kong（香港中文大学）；Stanford University（斯坦福大学）；Nanyang Technological University（南洋理工大学）
