Cross-modal Deep Face Normals with Deactivable Skip Connections
University of Oxford（牛津大学）；Inria（法国国家信息与自动化研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；CNRS（法国国家科学研究中心）；Grenoble INP（格勒诺布尔国立理工学院）；Univ. Rennes（雷恩大学）；IRISA（雷恩第一大学计算机科学研究所）
Victoria Fernandez Abrevaya, Adnane Boukhayma, Philip H. S. Torr, Edmond Boyer
We present an approach for estimating surface normals from in-the-wild color images of faces. While data-driven strategies have been proposed for single face images, limited available ground truth data makes this problem difficult. To alleviate this issue, we propose a method that can leverage all available image and normal data, whether paired or not, thanks to a novel cross-modal learning architecture. In particular, we enable additional training with single modality data, either color or normal, by using two encoder-decoder networks with a shared latent space. The proposed architecture also enables face details to be transferred between the image and normal domains, given paired data, through skip connections between the image encoder and normal decoder. Core to our approach is a novel module that we call deactivable skip connections, which allows integrating both the auto-encoded and image-to-normal branches within the same architecture that can be trained end-to-end. This allows learning of a rich latent space that can accurately capture the normal information. We compare against state-of-the-art methods and show that our approach can achieve significant improvements, both quantitative and qualitative, with natural face images.
Deep generative models have led to significant advances in cross-modal generation such as text-to-image synthesis. Training these models typically requires paired data with direct correspondence between modalities. We introduce the novel problem of translating instances from one modality to another without paired data by leveraging an intermediate modality shared by the two other modalities. To demonstrate this, we take the problem of translating images to speech. In this case, one could leverage disjoint datasets with one shared modality, e.g., image-text pairs and text-speech pairs, with tex…
This work studies the representational mapping across multimodal data such that given a piece of the raw data in one modality the corresponding semantic description in terms of the raw data in another modality is immediately obtained. Such a representational mapping can be found in a wide spectrum of real-world applications including image/video retrieval, object recognition, action/behavior recognition, and event understanding and prediction. To that end, we introduce a simplified training objective for learning multimodal embeddings using the skip-gram architecture by introducing convolution…
As an essential prerequisite for developing a medical intelligent assistant system, medical image segmentation has received extensive research and concentration from the neural network community. A series of UNet-like networks with encoder-decoder architecture has achieved extraordinary success, in which UNet2+ and UNet3+ redesign skip connections, respectively proposing dense skip connection and full-scale skip connection and dramatically improving compared with UNet in medical image segmentation. However, UNet2+ lacks sufficient information explored from the full scale, which will affect the…
The state-of-the-art models for medical image segmentation are variants of U-Net and fully convolutional networks (FCN). Despite their success, these models have two limitations: (1) their optimal depth is apriori unknown, requiring extensive architecture search or inefficient ensemble of models of varying depths; and (2) their skip connections impose an unnecessarily restrictive fusion scheme, forcing aggregation only at the same-scale feature maps of the encoder and decoder sub-networks. To overcome these two limitations, we propose UNet++, a new neural architecture for semantic and instance…
COMSATS University Islamabad（伊斯兰堡COMSATS大学）；Umeå University（于默奥大学）
U-Net is a widely adopted neural network in the domain of medical image segmentation. Despite its quick embracement by the medical imaging community, its performance suffers on complicated datasets. The problem can be ascribed to its simple feature extracting blocks: encoder/decoder, and the semantic gap between encoder and decoder. Variants of U-Net (such as R2U-Net) have been proposed to address the problem of simple feature extracting blocks by making the network deeper, but it does not deal with the semantic gap problem. On the other hand, another variant UNET++ deals with the semantic gap…
Indian Institute of Techonology, Bombay（印度理工学院孟买分校）
In this paper, we present a novel approach to the audio-visual video parsing (AVVP) task that demarcates events from a video separately for audio and visual modalities. The proposed parsing approach simultaneously detects the temporal boundaries in terms of start and end times of such events. We show how AVVP can benefit from the following techniques geared towards effective cross-modal learning: (i) adversarial training and skip connections (ii) global context aware attention and, (iii) self-supervised pretraining using an audio-video grounding objective to obtain cross-modal audio-video repr…
Trier University（特里尔大学）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；National Institute for Science Communication（国家科学传播研究所）
Representation learning is a key element of state-of-the-art deep learning approaches. It enables to transform raw data into structured vector space embeddings. Such embeddings are able to capture the distributional semantics of their context, e.g. by word windows on natural language sentences, graph walks on knowledge graphs or convolutions on images. So far, this context is manually defined, resulting in heuristics which are solely optimized for computational performance on certain tasks like link-prediction. However, such heuristic models of context are fundamentally different to how humans…
DSU-Net: An Attention-Enhanced Dense Skip U-Net for Breast Lesion Segmentation in Mammographic Images
Department of Biomedical Engineering, University of Wisconsin-Milwaukee（威斯康星大学密尔沃基分校生物医学工程系）；Department of Mechanical Engineering, Northern Arizona University（北亚利桑那大学机械工程系）
When Fusion Fails: Corruption-Aware Rebalanced Fusion for Multi-Modal Medical Image Segmentation
Central China Normal University（华中师范大学）；Huazhong University of Science and Technology（华中科技大学）；Guangzhou University of Chinese Medicine（广州中医药大学）；The First Affiliated Hospital of Guangzhou University of Chinese Medicine（广州中医药大学第一附属医院）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models
Shanghai AI Laboratory（上海人工智能实验室）；CUHK（香港中文大学）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Rutgers University（罗格斯大学）；Centre for Perceptual and Interactive Intelligence (CPII)（感知与交互智能中心（CPII））
Is the medical image segmentation problem solved? A survey of current developments and future directions
University of Texas Southwestern Medical Center（德克萨斯大学西南医学中心）；University of Pennsylvania（宾夕法尼亚大学）；Cleveland Clinic（克利夫兰诊所）；Mayo Clinic（梅奥诊所）；Brigham and Women’s Hospital, Harvard Medical School（哈佛医学院布莱根妇女医院）
TBConvL-Net: A Hybrid Deep Learning Architecture for Robust Medical Image Segmentation
COMSATS University Islamabad (CUI)（COMSATS大学伊斯兰堡校区）；University of New South Wales（新南威尔士大学）；University of Engineering and Technology (UET) Lahore（拉合尔工程技术大学）
本文提出混合架构 TBConvL-Net，融合 CNN、双卷积 LSTM 与 ViT 建模局部、长程及时序上下文，并采用复合损失提升边界与鲁棒性，在十种公开数据集上优于现有方法。
Understanding and Improving Transfer Learning of Deep Models via Neural Collapse
University of Michigan（密歇根大学）；Stanford University（斯坦福大学）；New York University（纽约大学）；Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）；Ohio State University（俄亥俄州立大学）；Carnegie Mellon University（卡内基梅隆大学）
Simulating Sinogram-Domain Motion and Correcting Image-Domain Artifacts Using Deep Learning in HR-pQCT Bone Imaging
Weldon School of Biomedical Engineering, Purdue University（沃伦生物医学工程学院，普渡大学）；Department of Radiology and Imaging Sciences, Indiana University School of Medicine（放射学与影像科学系，印第安纳大学医学院）；Department of Physical Therapy, School of Health & Human Sciences, Indiana University Indianapolis（物理治疗系，健康与人类科学学院，印第安纳大学印第安纳波利斯分校）
