Amidst the advancements in image-based Large Vision-Language Models (image-LVLM), the transition to video-based models (video-LVLM) is hindered by the limited availability of quality video data. This paper addresses the challenge by leveraging the visual commonalities between images and videos to efficiently evolve image-LVLMs into video-LVLMs. We present a cost-effective video-LVLM that enhances model architecture, introduces innovative training strategies, and identifies the most effective types of video instruction data. Our innovative weighted token sampler significantly compresses the visual token numbers of each video frame, effectively cutting computational expenses. We also find that judiciously using just 10% of the video data, compared to prior video-LVLMs, yields impressive results during various training phases. Moreover, we delve into the influence of video instruction data in limited-resource settings, highlighting the significance of incorporating video training data that emphasizes temporal understanding to enhance model performance. The resulting Fewer Tokens and Fewer Videos LVLM (FTFV-LVLM) exhibits exceptional performance across video and image benchmarks, validating our model's design and training approaches.
RADAR: Revealing Asymmetric Development of Abilities in MLLM Pre-training
Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Peng Cheng Laboratory（鹏城实验室）；Guangdong Key Laboratory of Big Data Analysis and Processing（广东大数据分析与处理重点实验室）；Tsinghua Shenzhen International Graduate School（清华大学深圳国际 Graduate School）；Tsinghua University（清华大学）；Shanghai Jiao Tong University（上海交通大学）；Yinwang Intelligent Technology Co., Ltd.（亿纬智能科技有限公司）；Huawei’s 2012 Lab（华为2012实验室）
NUMINA: A Natural Understanding Benchmark for Multi-dimensional Intelligence and Numerical Reasoning Abilities
School of Advanced Technology, Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学先进技术学院）；Department of Computer Science, University of Liverpool（利物浦大学计算机科学系）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of Deep Perception Technology, JITRI（深度感知技术研究所）
LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
Ludwig Maximilian University of Munich（慕尼黑大学）；Munich Research Center, Huawei Technologies（华为技术有限公司慕尼黑研发中心）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
The Hong Kong University of Science & Technology (Guangzhou)（香港科技大学（广州））；The Hong Kong University of Science & Technology（香港科技大学）；Zhejiang University（浙江大学）；Lappeenranta-Lahti University of Technology（拉佩兰塔-拉赫蒂技术大学）
The Impact of Reasoning Step Length on Large Language Models
Rutgers University（罗格斯大学）；University of Liverpool（利物浦大学）；Northwestern University（西北大学）；New Jersey Institute of Technology（新泽西理工学院）；University of Exeter（埃克塞特大学）
RLFR: Extending Reinforcement Learning for LLMs with Flow Environment
University of Science and Technology of China（中国科学技术大学）；Shanghai Innovation Institute（上海创新研究院）；ByteDance（字节跳动）；Wuhan University（武汉大学）；Southeast University（东南大学）
ChatGPT is a groundbreaking ``chatbot"--an AI interface built on a large language model that was trained on an enormous corpus of human text to emulate human conversation. Beyond its ability to converse in a plausible way, it has attracted attention for its ability to competently answer questions from the bar exam and from MBA coursework, and to provide useful assistance in writing computer code. These apparent abilities have prompted discussion of ChatGPT as both a threat to the integrity of higher education and conversely as a powerful teaching tool. In this work we present a preliminary ana…
Paul G. Allen School of Computer Science, University of Washington（华盛顿大学保罗·G·艾伦计算机科学学院）；Nvidia Allen Institute for Artificial Intelligence（英伟达艾伦人工智能研究所）
Long-term planning poses a major difficulty to many reinforcement learning algorithms. This problem becomes even more pronounced in dynamic visual environments. In this work we propose Hierarchical Planning and Reinforcement Learning (HIP-RL), a method for merging the benefits and capabilities of Symbolic Planning with the learning abilities of Deep Reinforcement Learning. We apply HIPRL to the complex visual tasks of interactive question answering and visual semantic planning and achieve state-of-the-art results on three challenging datasets all while taking fewer steps at test time and train…
Translation as a Scalable Proxy for Multilingual Evaluation
University of California, Los Angeles（加州大学洛杉矶分校）；African Languages Lab（非洲语言实验室）；Mila - Quebec AI Institute（魁北克人工智能研究所）；McGill University（麦吉尔大学）；Canada CIFAR AI Chair（加拿大CIFAR人工智能主席）
VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuning
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；National University of Singapore（新加坡国立大学）；MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China（中国科学技术大学脑启发智能感知与认知教育部重点实验室）
Artificial Intelligence Institute, University of South Carolina（南卡罗来纳大学人工智能研究所）；IBM Research（IBM研究院）；University of Parma（帕尔马大学）；University of Brescia（布雷西亚大学）
Large Language Models (LLMs) have been the subject of active research, significantly advancing the field of Natural Language Processing (NLP). From BERT to BLOOM, LLMs have surpassed state-of-the-art results in various natural language tasks such as question answering, summarization, and text generation. Many ongoing efforts focus on understanding LLMs' capabilities, including their knowledge of the world, syntax, and semantics. However, extending the textual prowess of LLMs to symbolic reasoning has been slow and predominantly focused on tackling problems related to the mathematical field. In…
Cross-Modal Obfuscation for Jailbreak Attacks on Large Vision-Language Models
University of Science and Technology of China（中国科学技术大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Institute of High Performance Computing, A*STAR, Singapore（新加坡A*STAR高性能计算研究所）
Carnegie Mellon University（卡内基梅隆大学）；University of Southern California（南加州大学）；Bosch Research（博世研究中心）
Recent developments in pre-trained neural language modeling have led to leaps in accuracy on commonsense question-answering benchmarks. However, there is increasing concern that models overfit to specific tasks, without learning to utilize external knowledge or perform general semantic reasoning. In contrast, zero-shot evaluations have shown promise as a more robust measure of a model's general reasoning abilities. In this paper, we propose a novel neuro-symbolic framework for zero-shot question answering across commonsense tasks. Guided by a set of hypotheses, the framework studies how to tra…
SenseTime Research（商汤科技研究院）；Tsinghua University（清华大学）；University of Oxford（牛津大学）；Qing Yuan Research Institute, Shanghai Jiao Tong University（上海交通大学清源研究院）；Centre for Perceptual and Interactive Intelligence（感知与交互智能研究中心）；The Chinese University of Hong Kong（香港中文大学）
Transformers with powerful global relation modeling abilities have been introduced to fundamental computer vision tasks recently. As a typical example, the Vision Transformer (ViT) directly applies a pure transformer architecture on image classification, by simply splitting images into tokens with a fixed length, and employing transformers to learn relations between these tokens. However, such naive tokenization could destruct object structures, assign grids to uninterested regions such as background, and introduce interference signals. To mitigate the above issues, in this paper, we propose a…
Zhejiang University（浙江大学）；Renmin University of China（中国人民大学）；University of Amsterdam（阿姆斯特丹大学）
This paper strives to find the sentence best describing the content of an image or video. Different from existing works, which rely on a joint subspace for image / video to sentence matching, we propose to do so in a visual space only. We contribute Word2VisualVec, a deep neural network architecture that learns to predict a deep visual encoding of textual input based on sentence vectorization and a multi-layer perceptron. We thoroughly analyze its architectural design, by varying the sentence vectorization strategy, network depth and the deep feature to predict for image to sentence matching. …
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；College of Information Science and Technology and Artificial Intelligence, Nanjing Forestry University（南京林业大学信息科学技术与人工智能学院）
