参数少于30亿的视觉语言模型的出现加速了设备上多模态智能的实现。然而，对组件级量化的详细理解仍然是优化部署的瓶颈。本文提出了一个系统评估框架，在Jetson Orin NX和AGX上针对六种量化配置实证验证五个假设。通过分离视觉编码器、投影仪和大语言模型主干得出以下结果：（1）量化敏感性由结构范式（混合专家模型与密集模型）而非规模单独决定；（2）SigLIP编码器在Jetson Ampere上有不成比例的INT8延迟；（3）大语言模型的INT4量化虽大幅降低VRAM消耗，但因反量化开销导致令牌生成变慢；（4）复合量化误差大多是可加的；（5）由于内存带宽限制，每焦耳智能在不同平台上差异显著。
英文摘要：
The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and large language model backbone yields the following results: (1) Quantization sensitivity is governed by the structural paradigm (MoE vs. dense) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; (2) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere--a deployment-specific encoder-kernel-hardware interaction, not a SigLIP flaw; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) Composite quantization errors are largely additive, except along the modality-alignment path, which is architecture-dependent; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.
Horizon Robotics（地平线机器人）；University at Buffalo（布法罗大学）；Tsinghua University（清华大学）
Knowledge distillation is an effective approach to leverage a well-trained network or an ensemble of them, named as the teacher, to guide the training of a student network. The outputs from the teacher network are used as soft labels for supervising the training of a new network. Recent studies \citep{muller2019does,yuan2020revisiting} revealed an intriguing property of the soft labels that making labels soft serves as a good regularization to the student network. From the perspective of statistical learning, regularization aims to reduce the variance, however how bias and variance change is n…
Goethe University（歌德大学）；Frankfurt Institute for Advanced Studies（法兰克福高等研究院）
We characterize convolutional neural networks with respect to the relative amount of features per layer. Using a skew normal distribution as a parametrized framework, we investigate the common assumption of monotonously increasing feature-counts with higher layers of architecture designs. Our evaluation on models with VGG-type layers on the MNIST, Fashion-MNIST and CIFAR-10 image classification benchmarks provides evidence that motivates rethinking of our common assumption: architectures that favor larger early layers seem to yield better accuracy.
In document classification for, e.g., legal and biomedical text, we often deal with hundreds of classes, including very infrequent ones, as well as temporal concept drift caused by the influence of real world events, e.g., policy changes, conflicts, or pandemics. Class imbalance and drift can sometimes be mitigated by resampling the training data to simulate (or compensate for) a known target distribution, but what if the target distribution is determined by unknown future events? Instead of simply resampling uniformly to hedge our bets, we focus on the underlying optimization algorithms used …
CMoS: Rethinking Time Series Prediction Through the Lens of Chunk-wise Spatial Correlations
Computer Network Information Center, CAS（中国科学院计算机网络信息中心）；University of Chinese Academy of Sciences（中国科学院大学）；Nanjing University（南京大学）；Tsinghua University（清华大学）
The real human attention is an interactive activity between our visual system and our brain, using both low-level visual stimulus and high-level semantic information. Previous image salient object detection (SOD) works conduct their saliency predictions in a multi-task manner, i.e., performing pixel-wise saliency regression and segmentation-like saliency refinement at the same time, which degenerates their feature backbones in revealing semantic information. However, given an image, we tend to pay more attention to those regions which are semantically salient even in the case that these region…
University of Oxford（牛津大学）；Peking University（北京大学）；Yale University（耶鲁大学）；Tencent AI Lab（腾讯人工智能实验室）
In sequence-to-sequence learning, e.g., natural language generation, the decoder relies on the attention mechanism to efficiently extract information from the encoder. While it is common practice to draw information from only the last encoder layer, recent work has proposed to use representations from different encoder layers for diversified levels of information. Nonetheless, the decoder still obtains only a single view of the source sequences, which might lead to insufficient training of the encoder layer stack due to the hierarchy bypassing problem. In this work, we propose layer-wise multi…
Rethinking Expressivity and Efficiency in Test-Time Training
Fraunhofer IOSB（弗劳恩霍夫IOSB研究所）；National University of Singapore（新加坡国立大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；University of Bonn（波恩大学）；Karlsruhe Institute of Technology (KIT)（卡尔斯鲁厄理工学院）
This paper discusses the deep learning architectures currently used for pixel-wise segmentation of primary and secondary glioblastomas and low-grade gliomas. We implement various models such as the popular UNet architecture and compare the performance of these implementations on the BRATS dataset. This paper will explore the different approaches and combinations, offering an in depth discussion of how they perform and how we may improve upon them using more recent advancements in deep learning architectures.
Model compression and acceleration are attracting increasing attentions due to the demand for embedded devices and mobile applications. Research on efficient convolutional neural networks (CNNs) aims at removing feature redundancy by decomposing or optimizing the convolutional calculation. In this work, feature redundancy is assumed to exist among channels in CNN architectures, which provides some leeway to boost calculation efficiency. Aiming at channel compression, a novel convolutional construction named compact convolution is proposed to embrace the progress in spatial convolution, channel…
Northeastern University（东北大学）；Key Laboratory of Intelligent Computing in Medical Image of Ministry of Education, Northeastern University（东北大学医学影像智能计算教育部重点实验室）；Amii, University of Alberta（阿尔伯塔大学阿尔伯塔机器智能研究所）
Most recent semantic segmentation methods adopt a U-Net framework with an encoder-decoder architecture. It is still challenging for U-Net with a simple skip connection scheme to model the global multi-scale context: 1) Not each skip connection setting is effective due to the issue of incompatible feature sets of encoder and decoder stage, even some skip connection negatively influence the segmentation performance; 2) The original U-Net is worse than the one without any skip connection on some datasets. Based on our findings, we propose a new segmentation framework, named UCTransNet (with a pro…
Shanghai Jiao Tong University（上海交通大学）；South China University of Technology（南方科技大学）；Beihang University（北航）；Huawei Technologies Ltd（华为技术有限公司）；Shanghai AI Lab（上海人工智能实验室）；Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Knowledge Boosting: Rethinking Medical Contrastive Vision-Language Pre-Training
Southeast University（东南大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Case Western Reserve University（凯斯西储大学）；Centre de Recherche en Information Biomédicale Sino-Français（中法生物医学信息研究中心）
A very recent trend in generative modeling is building 3D-aware generators from 2D image collections. To induce the 3D bias, such models typically rely on volumetric rendering, which is expensive to employ at high resolutions. During the past months, there appeared more than 10 works that address this scaling issue by training a separate 2D decoder to upsample a low-resolution image (or a feature tensor) produced from a pure 3D generator. But this solution comes at a cost: not only does it break multi-view consistency (i.e. shape and texture change when the camera moves), but it also learns th…
