Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main cause is that recent approaches learn human interactive relations via shallow graphical models, which is inadequate to model complicated human interactions. In this paper, we propose a consistency-aware graph network, which combines the representative ability of graph network and the consistency-aware reasoning to facilitate the HIU task. Our network consists of three components, a backbone CNN to extract image features, a factor graph network to learn third-order interactive relations among participants, and a consistency-aware reasoning module to enforce labeling and grouping consistencies. Our key observation is that the consistency-aware-reasoning bias for HIU can be embedded into an energy function, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained jointly in an end-to-end manner. Experimental results show that our approach achieves leading performance on three benchmarks.
Hong Kong University of Science and Technology（香港科技大学）；Everest Innovation Technology（珠峰创新科技）
Recent learning-based approaches, in which models are trained by single-view images have shown promising results for monocular 3D face reconstruction, but they suffer from the ill-posed face pose and depth ambiguity issue. In contrast to previous works that only enforce 2D feature constraints, we propose a self-supervised training architecture by leveraging the multi-view geometry consistency, which provides reliable constraints on face pose and depth estimation. We first propose an occlusion-aware view synthesis method to apply multi-view geometry consistency to self-supervised learning. Then…
School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用重点实验室（东南大学），教育部，中国）；State Key Laboratory of Media Convergence and Communication, Communication University of China（媒体融合与传播国家重点实验室，中国传媒大学）
Unsupervised domain adaptation (UDA) aims to adapt existing models of the source domain to a new target domain with only unlabeled data. Most existing methods suffer from noticeable negative transfer resulting from either the error-prone discriminator network or the unreasonable teacher model. Besides, the local regional consistency in UDA has been largely neglected, and only extracting the global-level pattern information is not powerful enough for feature alignment due to the abuse use of contexts. To this end, we propose an uncertainty-aware consistency regularization method for cross-domai…
Structure-aware Semantic Discrepancy and Consistency for 3D Medical Image Self-supervised Learning
Artificial Intelligence Innovation and Incubation Institute, Fudan University（复旦大学人工智能创新与孵化院）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）；The University of Queensland（昆士兰大学）；Zhongshan Hospital, Fudan University（复旦大学中山医院）
The Chinese University of Hong Kong（香港中文大学）；SmartMore（思谋科技）；University of Oxford（牛津大学）
Semantic segmentation has made tremendous progress in recent years. However, satisfying performance highly depends on a large number of pixel-level annotations. Therefore, in this paper, we focus on the semi-supervised segmentation problem where only a small set of labeled data is provided with a much larger collection of totally unlabeled images. Nevertheless, due to the limited annotations, models may overly rely on the contexts available in the training data, which causes poor generalization to the scenes unseen before. A preferred high-level representation should capture the contextual inf…
University of Southern California（南加州大学）；Baidu Research（百度研究院）
Learning to reconstruct depths in a single image by watching unlabeled videos via deep convolutional network (DCN) is attracting significant attention in recent years. In this paper, we introduce a surface normal representation for unsupervised depth estimation framework. Our estimated depths are constrained to be compatible with predicted normals, yielding more robust geometry results. Specifically, we formulate an edge-aware depth-normal consistency term, and solve it by constructing a depth-to-normal layer and a normal-to-depth layer inside of the DCN. The depth-to-normal layer takes estima…
Academia Sinica（中央研究院）；NEC Laboratories America（美国NEC实验室）；University of California, Merced（加州大学默塞德分校）；Google Cloud（谷歌云）
Referring expressions are natural language descriptions that identify a particular object within a scene and are widely used in our daily conversations. In this work, we focus on segmenting the object in an image specified by a referring expression. To this end, we propose an end-to-end trainable comprehension network that consists of the language and visual encoders to extract feature representations from both domains. We introduce the spatial-aware dynamic filters to transfer knowledge from text to image, and effectively capture the spatial information of the specified object. To better comm…
Style-Aware Blending and Prototype-Based Cross-Contrast Consistency for Semi-Supervised Medical Image Segmentation
School of Information Science and Engineering（信息科学与工程学院）；Yunnan University（云南大学）；Yunnan Key Laboratory of Intelligent Systems and Computing（智能系统与计算云南省重点实验室）
ISPDiffuser: Learning RAW-to-sRGB Mappings with Texture-Aware Diffusion Models and Histogram-Guided Color Consistency
School of Aeronautics and Astronautics, Sichuan University（四川大学航空航天学院）；University of Electronic Science and Technology of China（电子科技大学）；Megvii Technology（旷视科技）
Ground4D: Consistency-Aware 4D Reconstruction from Monocular Video
School of Computer Science and Engineering, Sun Yat-sen University, China（中山大学计算机科学与工程学院）；Tsinghua Shenzhen International Graduate School, Tsinghua University, China（清华大学深圳国际研究生院）
Video-based Generalized Category Discovery via Memory-Guided Consistency-Aware Contrastive Learning
Laboratory for Big Data and Decision（大数据与决策实验室）；National University of Defense Technology（国防科技大学）；Department of Information Engineering and Computer Science（信息工程与计算机科学系）
Nankai University（南开大学）；Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Mohamed bin Zayed University of AI（穆罕默德·本·扎耶德人工智能大学）；Linkoping University（林雪平大学）；Beijing Jiaotong University（北京交通大学）
Recent advances in 3D-aware generative models (3D-aware GANs) combined with Neural Radiance Fields (NeRF) have achieved impressive results. However no prior works investigate 3D-aware GANs for 3D consistent multi-class image-to-image (3D-aware I2I) translation. Naively using 2D-I2I translation methods suffers from unrealistic shape/identity change. To perform 3D-aware multi-class I2I translation, we decouple this learning process into a multi-class 3D-aware GAN step and a 3D-aware I2I translation step. In the first step, we propose two novel techniques: a new conditional architecture and an ef…
Topology-aware Pathological Consistency Matching for Weakly-Paired IHC Virtual Staining
Guangdong Research Center of Polarization Imaging and Measurement Engineering Technology（广东极化成像与测量工程研究中心）；Shenzhen Key Laboratory for Minimal Invasive Medical Technologies（深圳微创医疗技术重点实验室）；Institute of Biopharmaceutical and Health Engineering（生物制药与健康工程研究院）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际 Graduate School）；Tsinghua University（清华大学）
Instance-Aware Robust Consistency Regularization for Semi-Supervised Nuclei Instance Segmentation
School of Automation Science and Engineering, South China University of Technology, Guangzhou 510641, China（自动化科学与工程学院，华南理工大学，广州510641，中国）；Department of Breast and Thyroid Surgery, the Second Affiliated Hospital, University of South China, Hengyang 421001, China（乳腺甲状腺外科，南方大学第二附属医院，湖南衡阳421001，中国）；School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China（人工智能与自动化学院，华中科技大学，武汉430074，中国）；School of Informatics, Xiamen University, Xiamen 361005, China（信息学院，厦门大学，厦门361005，中国）；Pazhou Laboratory, Guangzhou 510335, China（琶洲实验室，广州510335，中国）；School of Future Technology, South China University of Technology, Guangzhou 510641, China（未来技术学院，华南理工大学，广州510641，中国）
University of Chinese Academy of Science（中国科学院大学）；Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Peng Cheng Laboratory（鹏城实验室）；University of California, Merced（加州大学默塞德分校）；University of Trento（特伦托大学）
Crowd localization aims to predict the spatial position of humans in a crowd scenario. We observe that the performance of existing methods is challenged from two aspects: (i) ranking inconsistency between test and training phases; and (ii) fixed anchor resolution may underfit or overfit crowd densities of local regions. To address these problems, we design a supervision target reassignment strategy for training to reduce ranking inconsistency and propose an anchor pyramid scheme to adaptively determine the anchor density in each image region. Extensive experimental results on three widely adop…
The rapid development of multi-view 3D human pose estimation (HPE) is attributed to the maturation of monocular 2D HPE and the geometry of 3D reconstruction. However, 2D detection outliers in occluded views due to neglect of view consistency, and 3D implausible poses due to lack of pose coherence, remain challenges. To solve this, we introduce a Multi-View Fusion module to refine 2D results by establishing view correlations. Then, Holistic Triangulation is proposed to infer the whole pose as an entirety, and anatomy prior is injected to maintain the pose coherence and improve the plausibility.…
Structure-Aware Consistency Priors for Shape from Polarization in Complex Media
The School of Mechanical Engineering and Automation, Fuzhou University, Fuzhou, China（福州大学机械工程与自动化学院）；Research Institute of Highway, Ministry of Transport, Beijing, China（交通部公路科学研究院）
VGGT-Motion: Motion-Aware Calibration-Free Monocular SLAM for Long-Range Consistency
National Key Laboratory of Science and Technology on Multi-spectral Information Processing（科学与技术多光谱信息处理国家重点实验室）；Huazhong University of Science and Technology（华中科技大学）；College of Computing and Data Science（计算与数据科学学院）；Nanyang Technological University（南洋理工大学）
University of Science and Technology of China（中国科学技术大学）
Intra-frame inconsistency has been proved to be effective for the generalization of face forgery detection. However, learning to focus on these inconsistency requires extra pixel-level forged location annotations. Acquiring such annotations is non-trivial. Some existing methods generate large-scale synthesized data with location annotations, which is only composed of real images and cannot capture the properties of forgery regions. Others generate forgery location labels by subtracting paired real and fake images, yet such paired data is difficult to collected and the generated label is usuall…
Avon Old Farms School（埃文老农场中学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
Recognition of floor plans has been a challenging and popular task. Despite that many recent approaches have been proposed for this task, they typically fail to make the room-level unified prediction. Specifically, multiple semantic categories can be assigned in a single room, which seriously limits their visual quality and applicability. In this paper, we propose a novel approach to recognize the floor plan layouts with a newly proposed Offset-Guided Attention mechanism to improve the semantic consistency within a room. In addition, we present a Feature Fusion Attention module that leverages …
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, School of Informatics, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学信息学院）；Fujian Key Laboratory of Pattern Recognition and Image Understanding, Xiamen University of Technology（厦门理工学院福建省模式识别与图像理解重点实验室）；School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院）
Multiple object tracking (MOT) is a crucial task in computer vision society. However, most tracking-by-detection MOT methods, with available detected bounding boxes, cannot effectively handle static, slow-moving and fast-moving camera scenarios simultaneously due to ego-motion and frequent occlusion. In this work, we propose a novel tracking framework, called "instance-aware MOT" (IA-MOT), that can track multiple objects in either static or moving cameras by jointly considering the instance-level features and object motions. First, robust appearance features are extracted from a variant of Mas…
InfinityStory: Unlimited Video Generation with World Consistency and Character-Aware Shot Transitions
Adobe Research（Adobe研究院）；KAUST（卡塔尔科技大学）；University of Oregon（俄勒冈大学）；University of Memphis（密苏里大学孟菲斯分校）；Johns Hopkins University（约翰霍普金斯大学）；Meta AI；Texas A&M University（德克萨斯农工大学）；Dolby Labs（杜比实验室）；Virginia Tech（弗吉尼亚理工大学）；Cisco（思科）；University of Maryland, College Park（马里兰大学学院市分校）
CAD-SLAM: Consistency-Aware Dynamic SLAM with Dynamic-Static Decoupled Mapping
Department of Automation, Shanghai Jiao Tong University（自动化系，上海交通大学）；Department of Computer Science, University College London（计算机科学系，伦敦大学学院）；Department of Engineering, University of Cambridge（工程系，剑桥大学）；Department of Automation, Key Laboratory of System Control and Information Processing of Ministry of Education, State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services, Shanghai Jiao Tong University（自动化系，教育部系统控制与信息处理重点实验室，航空系统集成与航空系统-of-系统综合国家重点实验室，导航与基于位置的服务上海实验室，上海交通大学）
Dual Cross-image Semantic Consistency with Self-aware Pseudo Labeling for Semi-supervised Medical Image Segmentation
School of Biomedical Engineering & State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University（生物医学工程学院及先进医疗材料与器件国家重点实验室，上海科技大学）；Lingang Laboratory（临港实验室）；Department of Radiology, Stanford University（放射科，斯坦福大学）
Self-aware and Cross-sample Prototypical Learning for Semi-supervised Medical Image Segmentation
Xidian University（西安电子科技大学）；The First Affiliated Hospital of Xi’an Jiaotong University（西安交通大学第一附属医院）；AIQ；Warren Alpert Medical School of Brown University（布朗大学沃伦·阿尔珀特医学院）
UCAD: Uncertainty-guided Contour-aware Displacement for semi-supervised medical image segmentation
University of Science and Technology of China（科学技术大学）；Suzhou Institute for Advanced Research（苏州先进研究院）；State Key Laboratory of Precision and Intelligent Chemistry（精密与智能化学国家重点实验室）
Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（西安交通大学人工智能与机器人研究所）
We address the problem of decomposing a single image into reflectance and shading. The difficulty comes from the fact that the components of image---the surface albedo, the direct illumination, and the ambient illumination---are coupled heavily in observed image. We propose to infer the shading by ordering pixels by their relative brightness, without knowing the absolute values of the image components beforehand. The pairwise shading orders are estimated in two ways: brightness order and low-order fittings of local shading field. The brightness order is a non-local measure, which can be applie…
CAP-UDF: Learning Unsigned Distance Functions Progressively from Raw Point Clouds with Consistency-Aware Field Optimization
School of Software, Tsinghua University（清华大学软件学院）；Center for Artificial Intelligence and Robotics, New York University Abu Dhabi（纽约大学阿布扎比分校人工智能与机器人中心）；Wayne State University（韦恩州立大学）
Adversarial attacks aim to perturb images such that a predictor outputs incorrect results. Due to the limited research in structured attacks, imposing consistency checks on natural multi-object scenes is a promising yet practical defense against conventional adversarial attacks. More desired attacks, to this end, should be able to fool defenses with such consistency checks. Therefore, we present the first approach GLOW that copes with various attack requests by generating global layout-aware adversarial attacks, in which both categorical and geometric layout constraints are explicitly establis…
