In autonomous driving tasks, scene understanding is the first step towards predicting the future behavior of the surrounding traffic participants. Yet, how to represent a given scene and extract its features are still open research questions. In this study, we propose a novel text-based representation of traffic scenes and process it with a pre-trained language encoder.
First, we show that text-based representations, combined with classical rasterized image representations, lead to descriptive scene embeddings. Second, we benchmark our predictions on the nuScenes dataset and show significant improvements compared to baselines. Third, we show in an ablation study that a joint encoder of text and rasterized images outperforms the individual encoders confirming that both representations have their complementary strengths.
WiTUnet: A U-Shaped Architecture Integrating CNN and Transformer for Improved Feature Alignment and Local Information Fusion
College of Computer Science and Cyber Security, Chengdu University of Technology（成都理工大学计算机与网络安全学院）；College of Geophysics, Chengdu University of Technology（成都理工大学地球物理学院）；College of Mechanical and Vehicle Engineering, Changchun University（长春大学机械与车辆工程学院）
Vision-language alignment learning for video-text retrieval arouses a lot of attention in recent years. Most of the existing methods either transfer the knowledge of image-text pretraining model to video-text retrieval task without fully exploring the multi-modal information of videos, or simply fuse multi-modal features in a brute force manner without explicit guidance. In this paper, we integrate multi-modal information in an explicit manner by tagging, and use the tags as the anchors for better video-text alignment. Various pretrained experts are utilized for extracting the information of m…
CDPDNet: Integrating Text Guidance with Hybrid Vision Encoders for Medical Image Segmentation
J. Crayton Pruitt Family Department of Biomedical Engineering, University of Florida（J. Crayton Pruitt家庭生物医学工程系，佛罗里达大学）；Department of Medicine, University of Florida（医学系，佛罗里达大学）
Towards Unified Molecule-Enhanced Pathology Image Representation Learning via Integrating Spatial Transcriptomics
Academy for Engineering and Technology, Fudan University（复旦大学工程技术研究院）；Cognition and Intelligent Technology Laboratory, Fudan University（复旦大学认知与智能技术实验室）；Shanghai Jiaotong University（上海交通大学）；Fudan University（复旦大学）
Absolute pose regressor (APR) networks are trained to estimate the pose of the camera given a captured image. They compute latent image representations from which the camera position and orientation are regressed. APRs provide a different tradeoff between localization accuracy, runtime, and memory, compared to structure-based localization schemes that provide state-of-the-art accuracy. In this work, we introduce Camera Pose Auto-Encoders (PAEs), multilayer perceptrons that are trained via a Teacher-Student approach to encode camera poses using APRs as their teachers. We show that the resulting…
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion
ZIP Lab, Zhejiang University（浙江大学 ZIP 实验室）；South China University of Technology（华南理工大学）；Pazhou Lab（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（大数据与智能机器人教育部重点实验室）
Picsart AI Research (PAIR)（Picsart人工智能研究院（PAIR））；University of Oregon（俄勒冈大学）；UIUC（伊利诺伊大学厄巴纳-香槟分校）；IIT Roorkee（印度理工学院鲁尔基分校）
Finetuning a pretrained backbone in the encoder part of an image transformer network has been the traditional approach for the semantic segmentation task. However, such an approach leaves out the semantic context that an image provides during the encoding stage. This paper argues that incorporating semantic information of the image into pretrained hierarchical transformer-based backbones while finetuning improves the performance considerably. To achieve this, we propose SeMask, a simple and effective framework that incorporates semantic information into the encoder with the help of a semantic …
Generating Multimodal Images with GAN: Integrating Text, Image, and Style
Northeastern University（东北大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；Tsinghua University（清华大学）；Montclair State University（蒙特克莱尔州立大学）；Northern Arizona University（北亚利桑那大学）
提出一种基于 GAN 的多模态图像生成方法，通过文本、图像与风格编码模块及多重一致性损失，在多个公开数据集上显著提升生成图像的清晰度与风格一致性。
Integrating Pathology and CT Imaging for Personalized Recurrence Risk Prediction in Renal Cancer
Department of Radiology, Antoni van Leeuwenhoek-Netherlands Cancer Institute, Amsterdam, The Netherlands（放射科，Antoni van Leeuwenhoek荷兰癌症研究所，阿姆斯特丹，荷兰）；University of Amsterdam（阿姆斯特丹大学）；AI Technology for Life, Department of Information and Computing Sciences, Department of Biology, Utrecht University（AI技术生命，信息与计算科学系，生物学系，乌得勒支大学）；GROW Oncology, Maastricht University（GROW肿瘤学，马斯特里赫特大学）；Department of Oncology, University of Cambridge（肿瘤科，剑桥大学）；Cancer Research UK Cambridge Centre, University of Cambridge（英国癌症研究会剑桥中心，剑桥大学）；Early Cancer Institute, University of Cambridge（早期癌症研究所，剑桥大学）；Cambridge University Hospitals NHS Foundation Trust（剑桥大学医院 NHS 基础信托）
Department of Electrical Engineering, Indian Institute of Technology Madras (IITM)（电子工程系，印度理工学院Madras（IITM））；Sudha Gopalakrishnan Brain Centre (SGBC), IITM（Sudha Gopalakrishnan脑中心（SGBC），IITM）
Taking full advantage of the excellent performance of StyleGAN, style transfer-based face swapping methods have been extensively investigated recently. However, these studies require separate face segmentation and blending modules for successful face swapping, and the fixed selection of the manipulated latent code in these works is reckless, thus degrading face swapping quality, generalizability, and practicability. This paper proposes a novel and end-to-end integrated framework for high resolution and attribute preservation face swapping via Adaptive Latent Representation Learning. Specifical…
In recent years, multimodal AI has seen an upward trend as researchers are integrating data of different types such as text, images, speech into modelling to get the best results. This project leverages multimodal AI and matrix factorization techniques for representation learning, on text and image data simultaneously, thereby employing the widely used techniques of Natural Language Processing (NLP) and Computer Vision. The learnt representations are evaluated using downstream classification and regression tasks. The methodology adopted can be extended beyond the scope of this project as it us…
Convolutional neural networks (CNNs) have been applied to learn spatial features for high-resolution (HR) synthetic aperture radar (SAR) image classification. However, there has been little work on integrating the unique statistical distributions of SAR images which can reveal physical properties of terrain objects, into CNNs in a supervised feature learning framework. To address this problem, a novel end-to-end supervised classification method is proposed for HR SAR images by considering both spatial context and statistical features. First, to extract more effective spatial features from SAR …
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC)（布宜诺斯艾利斯大学。精确与自然科学学院。计算机科学系）；CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC)（布宜诺斯艾利斯大学CONICET联合体。计算机科学研究所）
