Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens. In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM.
HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院）；CUHK（香港中文大学）
Estimating the preferences of consumers is of utmost importance for the fashion industry as appropriately leveraging this information can be beneficial in terms of profit. Trend detection in fashion is a challenging task due to the fast pace of change in the fashion industry. Moreover, forecasting the visual popularity of new garment designs is even more demanding due to lack of historical data. To this end, we propose MuQAR, a Multimodal Quasi-AutoRegressive deep learning architecture that combines two modules: (1) a multi-modal multi-layer perceptron processing categorical, visual and textua…
Story Visualization is an advanced task of computed vision that targets sequential image synthesis, where the generated samples need to be realistic, faithful to their conditioning and sequentially consistent. Our work proposes a novel architectural and training approach: the Impartial Transformer achieves both text-relevant plausible scenes and sequential consistency utilizing as few trainable parameters as possible. This enhancement is even able to handle synthesis of 'hard' samples with occluded objects, achieving improved evaluation metrics comparing to past approaches.
Language-Guided Token Compression with Reinforcement Learning in Large Vision-Language Models
School of Computer Science and Engineering（计算机科学与工程学院）；University of Electronic Science and Technology of China（电子科学与技术大学）；School of Robotics and Advanced Manufacture（机器人与先进制造学院）；Harbin Institute of Technology（哈尔滨工业大学）；College of Information and Control Engineering（信息与控制工程学院）
Reshetnev Siberian State University of Science and Technology（列舍特涅夫西伯利亚国立科技大学）；Siberian Federal University（西伯利亚联邦大学）
The article describes the application of the Hough transform to a honeycomb block image. The problem of cutting a mold from a honeycomb block is described. A number of image transformations are considered to increase the efficiency of the Hough algorithm. A method for obtaining a binary image using a simple threshold, a method for obtaining a binary image using Otsu binarization, and the Canny Edge Detection algorithm are considered. The method of binary skeleton (skeletonization) is considered, in which the skeleton is obtained using 2 main morphological operations: Dilation and Erosion. As a…
The Australian National University（澳大利亚国立大学）；University of California, Merced（加州大学默塞德分校）；Google Research（谷歌研究院）；Yonsei University（延世大学）；Australian Institute for Machine Learning, The University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；Monash University（莫纳什大学）
Vision-and-Language Navigation (VLN) requires an agent to find a path to a remote location on the basis of natural-language instructions and a set of photo-realistic panoramas. Most existing methods take the words in the instructions and the discrete views of each panorama as the minimal unit of encoding. However, this requires a model to match different nouns (e.g., TV, table) against the same input view feature. In this work, we propose an object-informed sequential BERT to encode visual perceptions and linguistic instructions at the same fine-grained level, namely objects and words. Our seq…
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation
The Chinese University of Hong Kong（香港中文大学）；Fudan University（复旦大学）；Sensetime Research Institute（商汤研究院）；Tsinghua University（清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
The Perceptual Bandwidth Bottleneck in Vision-Language Models: Active Visual Reasoning via Sequential Experimental Design
The Hong Kong University of Science（香港科学与技术大学）；University College London, London, United Kingdom（伦敦大学学院, 英国伦敦）；ShanghaiTech University, Shanghai, China（上海科技大学, 中国上海）；AI Lab, The Yangtze River Delta, China（人工智能实验室, 长江三角洲, 中国）
Reinforcement Learning-based Token Pruning in Vision Transformers: A Markov Game Approach
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Data Intelligence Institute of Paris（巴黎数据智能研究院）；Université Paris Cité（巴黎西岱大学）；Shijiazhuang Tiedao University（石家庄铁道大学）
Carnegie Mellon University（卡内基梅隆大学）；Amazon Web Services（亚马逊云科技）；Amazon（亚马逊）
Recent methods in self-supervised learning have demonstrated that masking-based pretext tasks extend beyond NLP, serving as useful pretraining objectives in computer vision. However, existing approaches apply random or ad hoc masking strategies that limit the difficulty of the reconstruction task and, consequently, the strength of the learnt representations. We improve upon current state-of-the-art work in learning adversarial masks by proposing a new framework that generates masks in a sequential fashion with different constraints on the adversary. This leads to improvements in performance on…
Technical University of Munich（慕尼黑工业大学）；Hochschule Kempten（肯普滕应用技术大学）
Several autonomous driving strategies have been applied to autonomous vehicles, especially in the collision avoidance area. The purpose of collision avoidance is achieved by adjusting the trajectory of autonomous vehicles (AV) to avoid intersection or overlap with the trajectory of surrounding vehicles. A large number of sophisticated vision algorithms have been designed for target inspection, classification, and other tasks, such as ResNet, YOLO, etc., which have achieved excellent performance in vision tasks because of their ability to accurately and quickly capture regional features. Howeve…
Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers
Brandenburg University of Technology Cottbus-Senftenberg（科特布斯-森夫滕贝格勃兰登堡工业大学）；University of Zanjan（赞詹大学）；Humboldt University（洪堡大学）；Tallinn University of Technology（塔林理工大学）
University of Wisconsin（威斯康星大学）；William S. Middleton VA Hospital（威廉·S·米德尔顿退伍军人医院）
The adoption of "human-in-the-loop" paradigms in computer vision and machine learning is leading to various applications where the actual data acquisition (e.g., human supervision) and the underlying inference algorithms are closely interwined. While classical work in active learning provides effective solutions when the learning module involves classification and regression tasks, many practical issues such as partially observed measurements, financial constraints and even additional distributional or structural aspects of the data typically fall outside the scope of this treatment. For insta…
University of California, Davis（加州大学戴维斯分校）；Visa Research（Visa研究院）
Vision transformer (ViT) expands the success of transformer models from sequential data to images. The model decomposes an image into many smaller patches and arranges them into a sequence. Multi-head self-attentions are then applied to the sequence to learn the attention between patches. Despite many successful interpretations of transformers on sequential data, little effort has been devoted to the interpretation of ViTs, and many questions remain unanswered. For example, among the numerous attention heads, which one is more important? How strong are individual patches attending to their spa…
ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts
Institute of Biopharmaceutical and Health Engineering, Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学生物医药与健康工程研究院，清华大学深圳国际研究生院）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）；Department of Pathology, The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院病理科）；Medical Optical Technology R&D Center, Research Institute of Tsinghua, Pearl River Delta（清华珠三角研究院医学光学技术研发中心）
Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs
School of Computing, University of Georgia（佐治亚大学计算机学院）；Department of Computer Science and Engineering, University of Texas, Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Massachusetts General Hospital, Harvard Medical School（麻省总医院哈佛医学院）；Department of Biomedical Engineering, New Jersey Institute of Technology（新泽西理工学院生物医学工程系）
Online continual learning (online CL) studies the problem of learning sequential tasks from an online data stream without task boundaries, aiming to adapt to new data while alleviating catastrophic forgetting on the past tasks. This paper proposes a framework Contrastive Vision Transformer (CVT), which designs a focal contrastive learning strategy based on a transformer architecture, to achieve a better stability-plasticity trade-off for online CL. Specifically, we design a new external attention mechanism for online CL that implicitly captures previous tasks' information. Besides, CVT contain…
We characterise some of the quirks and shortcomings in the exploration of Visual Dialogue - a sequential question-answering task where the questions and corresponding answers are related through given visual stimuli. To do so, we develop an embarrassingly simple method based on Canonical Correlation Analysis (CCA) that, on the standard dataset, achieves near state-of-the-art performance on mean rank (MR). In direct contrast to current complex and over-parametrised architectures that are both compute and time intensive, our method ignores the visual stimuli, ignores the sequencing of dialogue, …
