本文提出TwinMixing模型，通过共享编码器和任务专用解码器实现特征共享与任务专精，结合高效金字塔混合模块和双分支上采样块，在BDD100K数据集上验证了其在 drivable-area 和 lane 分割中的高效准确性能。
摘要：
准确且高效的感知对于自动驾驶至关重要，其中如 drivable-area 和 lane 分割等分割任务为运动规划和控制提供关键线索。然而，实现高分割精度同时保持在低成本硬件上的实时性能仍是一个挑战。为此，我们引入了TwinMixing，一种轻量级的多任务分割模型，专门针对 drivable-area 和 lane 分割设计。该网络具有共享编码器和任务专用解码器，实现特征共享和任务专精。在编码器中，我们提出了一种高效的金字塔混合（EPM）模块，通过组卷积、深度可分离扩张卷积和通道洗牌操作的组合，增强多尺度特征提取，有效扩展感受野同时最小化计算成本。每个解码器采用双分支上采样（DBU）块，由可学习的转置卷积基的细节分支和参数无关的双线性插值基的粗粒度分支组成，实现细节且空间一致的特征重建。在BDD100K数据集上的大量实验验证了TwinMixing在三种配置（tiny、base、large）中的有效性。其中，base配置在精度和计算效率之间达到了最佳平衡，达到92.0%的mIoU drivable-area分割和32.3%的IoU lane分割，仅使用0.43M参数和3.95 GFLOPs。此外，TwinMixing在相同任务上优于现有分割模型，如图1所示。得益于其紧凑且模块化的设计，TwinMixing在自动驾驶和嵌入式感知系统中的实时部署具有强大潜力。源代码：https://github.com/Jun0se7en/TwinMixing。
英文摘要：
Accurate and efficient perception is essential for autonomous driving, where segmentation tasks such as drivable-area and lane segmentation provide critical cues for motion planning and control. However, achieving high segmentation accuracy while maintaining real-time performance on low-cost hardware remains a challenging problem. To address this issue, we introduce TwinMixing, a lightweight multi-task segmentation model designed explicitly for drivable-area and lane segmentation. The proposed network features a shared encoder and task-specific decoders, enabling both feature sharing and task specialization. Within the encoder, we propose an Efficient Pyramid Mixing (EPM) module that enhances multi-scale feature extraction through a combination of grouped convolutions, depthwise dilated convolutions and channel shuffle operations, effectively expanding the receptive field while minimizing computational cost. Each decoder adopts a Dual-Branch Upsampling (DBU) Block composed of a learnable transposed convolution-based Fine detailed branch and a parameter-free bilinear interpolation-based Coarse grained branch, achieving detailed yet spatially consistent feature reconstruction. Extensive experiments on the BDD100K dataset validate the effectiveness of TwinMixing across three configurations - tiny, base, and large. Among them, the base configuration achieves the best trade-off between accuracy and computational efficiency, reaching 92.0% mIoU for drivable-area segmentation and 32.3% IoU for lane segmentation with only 0.43M parameters and 3.95 GFLOPs. Moreover, TwinMixing consistently outperforms existing segmentation models on the same tasks, as illustrated in Fig. 1. Thanks to its compact and modular design, TwinMixing demonstrates strong potential for real-time deployment in autonomous driving and embedded perception systems. The source code: https://github.com/Jun0se7en/TwinMixing.
SegMix:Shuffle-based Feedback Learning for Semantic Segmentation of Pathology Images
JD AI Research（京东人工智能研究院）；School of Microelectronics, Xi’an Jiaotong University（西安交通大学微电子学院）；Beijing Advanced Innovation Center for Biomedical Engineering, School of Biological Science and Medical Engineering, Beihang University（北京生物医学工程先进创新中心，北京航空航天大学生物科学与医学工程学院）
Assigning a label to each pixel in an image, namely semantic segmentation, has been an important task in computer vision, and has applications in autonomous driving, robotic navigation, localization, and scene understanding. Fully convolutional neural networks have proved to be a successful solution for the task over the years but most of the work being done focuses primarily on accuracy. In this paper, we present a computationally efficient approach to semantic segmentation, while achieving a high mean intersection over union (mIOU), 70.33% on Cityscapes challenge. The network proposed is cap…
University of Georgia（佐治亚大学）；Florida International University（佛罗里达国际大学）
Understanding 3D point cloud models for learning purposes has become an imperative challenge for real-world identification such as autonomous driving systems. A wide variety of solutions using deep learning have been proposed for point cloud segmentation, object detection, and classification. These methods, however, often require a considerable number of model parameters and are computationally expensive. We study a semantic dimension of given 3D data points and propose an efficient method called Meta-Semantic Learning (Meta-SeL). Meta-SeL is an integrated framework that leverages two input 3D…
This paper addresses unsupervised action segmentation. Prior work captures the frame-level temporal structure of videos by a feature embedding that encodes time locations of frames in the video. We advance prior work with a new self-supervised learning (SSL) of a feature embedding that accounts for both frame- and action-level structure of videos. Our SSL trains an RNN to recognize positive and negative action sequences, and the RNN's hidden layer is taken as our new action-level feature embedding. The positive and negative sequences consist of action segments sampled from videos, where in the…
Attention mechanisms, which enable a neural network to accurately focus on all the relevant elements of the input, have become an essential component to improve the performance of deep neural networks. There are mainly two attention mechanisms widely used in computer vision studies, \textit{spatial attention} and \textit{channel attention}, which aim to capture the pixel-level pairwise relationship and channel dependency, respectively. Although fusing them together may achieve better performance than their individual implementations, it will inevitably increase the computational overhead. In t…
This is a short technical report introducing the solution of the Team TCParser for Short-video Face Parsing Track of The 3rd Person in Context (PIC) Workshop and Challenge at CVPR 2021. In this paper, we introduce a strong backbone which is cross-window based Shuffle Transformer for presenting accurate face parsing representation. To further obtain the finer segmentation results, especially on the edges, we introduce a Feature Alignment Aggregation (FAA) module. It can effectively relieve the feature misalignment issue caused by multi-resolution feature aggregation. Benefiting from the stronge…
University of Alberta（阿尔伯塔大学）；Valeo Vision Systems（法雷奥视觉系统公司）；Cairo University（开罗大学）
Semantic segmentation benefits robotics related applications especially autonomous driving. Most of the research on semantic segmentation is only on increasing the accuracy of segmentation models with little attention to computationally efficient solutions. The few work conducted in this direction does not provide principled methods to evaluate the different design choices for segmentation. In this paper, we address this gap by presenting a real-time semantic segmentation benchmarking framework with a decoupled design for feature extraction and decoding methods. The framework is comprised of d…
University of Washington（华盛顿大学）；Allen Institute for AI（艾伦人工智能研究所）
We introduce a fast and efficient convolutional neural network, ESPNet, for semantic segmentation of high resolution images under resource constraints. ESPNet is based on a new convolutional module, efficient spatial pyramid (ESP), which is efficient in terms of computation, memory, and power. ESPNet is 22 times faster (on a standard GPU) and 180 times smaller than the state-of-the-art semantic segmentation network PSPNet, while its category-wise accuracy is only 8% less. We evaluated ESPNet on a variety of semantic segmentation datasets including Cityscapes, PASCAL VOC, and a breast biopsy wh…
University of Electronic Science and Technology of China（电子科技大学）
Most of existing synthetic aperture radar (SAR) ship in-stance segmentation models do not achieve mask interac-tion or offer limited interaction performance. Besides, their multi-scale ship instance segmentation performance is moderate especially for small ships. To solve these problems, we propose a mask attention interaction and scale enhancement network (MAI-SE-Net) for SAR ship instance segmentation. MAI uses an atrous spatial pyra-mid pooling (ASPP) to gain multi-resolution feature re-sponses, a non-local block (NLB) to model long-range spa-tial dependencies, and a concatenation shuffle a…
Deep CNNs for semantic segmentation have high memory and run time requirements. Various approaches have been proposed to make CNNs efficient like grouped, shuffled, depth-wise separable convolutions. We study the effectiveness of these techniques on a real-time semantic segmentation architecture like ERFNet for improving run time by over 5X. We apply these techniques to CNN layers partially or fully and evaluate the testing accuracies on Cityscapes dataset. We obtain accuracy vs parameters/FLOPs trade offs, giving accuracy scores for models that can run under specified runtime budgets. We furt…
Johns Hopkins University（约翰斯·霍普金斯大学）；Johns Hopkins University School of Medicine（约翰斯·霍普金斯大学医学院）
In this paper, we present a novel unsupervised domain adaptation (UDA) method, named Domain Adaptive Relational Reasoning (DARR), to generalize 3D multi-organ segmentation models to medical data collected from different scanners and/or protocols (domains). Our method is inspired by the fact that the spatial relationship between internal structures in medical images is relatively fixed, e.g., a spleen is always located at the tail of a pancreas, which serves as a latent variable to transfer the knowledge shared across multiple domains. We formulate the spatial relationship by solving a jigsaw p…
Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
Beijing Institute of Technology（北京理工大学）；University of Birmingham（伯明翰大学）；Beijing Jiaotong University（北京交通大学）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
We propose a new convolution called Dynamic Region-Aware Convolution (DRConv), which can automatically assign multiple filters to corresponding spatial regions where features have similar representation. In this way, DRConv outperforms standard convolution in modeling semantic variations. Standard convolutional layer can increase the number of filers to extract more visual elements but results in high computational cost. More gracefully, our DRConv transfers the increasing channel-wise filters to spatial dimension with learnable instructor, which not only improve representation ability of conv…
In order to achieve better performance for point cloud analysis, many researchers apply deeper neural networks using stacked Multi-Layer-Perceptron (MLP) convolutions over irregular point cloud. However, applying dense MLP convolutions over large amount of points (e.g. autonomous driving application) leads to inefficiency in memory and computation. To achieve high performance but less complexity, we propose a deep-wide neural network, called ShufflePointNet, to exploit fine-grained local features and reduce redundancy in parallel using group convolution and channel shuffle operation. Unlike co…
DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopy
CyPhi AI Lab, Monash University, Malaysia Campus, Malaysia Department of Electronic \& Computer Engineering, Hong Kong University of Science \& Technology, Hong Kong, P.R. China Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, P.R. China Harbin Institute of Technology, Harbin, P.R. China
We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal f…
