Recent studies have shown that Large Vision-Language Models (VLMs) tend to neglect image content and over-rely on language-model priors, resulting in errors in visually grounded tasks and hallucinations. We hypothesize that this issue arises because existing VLMs are not explicitly trained to generate texts that are accurately grounded in fine-grained image details. To enhance visual feedback during VLM training, we propose S-VCO (Symmetrical Visual Contrastive Optimization), a novel finetuning objective that steers the model toward capturing important visual details and aligning them with corresponding text tokens. To further facilitate this detailed alignment, we introduce MVC, a paired image-text dataset built by automatically filtering and augmenting visual counterfactual data to challenge the model with hard contrastive cases involving Minimal Visual Contrasts. Experiments show that our method consistently improves VLM performance across diverse benchmarks covering various abilities and domains, achieving up to a 22% reduction in hallucinations, and significant gains in vision-centric and general tasks. Notably, these improvements become increasingly pronounced in benchmarks with higher visual dependency. In short, S-VCO offers a significant enhancement of VLM's visually-dependent task performance while retaining or even improving the model's general abilities. We opensource our code at https://s-vco.github.io/
HSCR: Hierarchical Self-Contrastive Rewarding for Aligning Medical Vision Language Models
Zhejiang University（浙江大学）；Byte Dance（字节跳动）；National University of Singapore（新加坡国立大学）；Angelalign Inc China（Angelalign中国公司）；Zhejiang Key Laboratory of Medical Imaging Artificial Intelligence（浙江省医学影像人工智能重点实验室）
SECOND: Mitigating Perceptual Hallucination in Vision-Language Models via Selective and Contrastive Decoding
Department of Electrical and Computer Engineering, Seoul National University（首尔大学电气与计算机工程系）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能跨学科项目）
DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Renmin University of China（中国人民大学）；Beijing Institute of AI Safety and Governance（北京人工智能安全与治理研究院）；University of Chinese Academy of Sciences（中国科学院大学）
SpecCLIP: Aligning and Translating Spectroscopic Measurements for Stars
School of Astronomy；Space Science, University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China；National Astronomical Observatories, Chinese Academy of Sciences, Beijing 100012, People's Republic of China；Department of Physics \& Astronomy, The Johns Hopkins University, Baltimore, MD 21218, USA；Zhejiang Laboratory, Hangzhou 311121, People's Republic of China；Research Center for Astronomical Computing, Zhejiang Laboratory, Hangzhou 311121, People's Republic of China；Department of Physics；Astronomy, University of Notre Dame, Notre Dame, IN 46556, USA；Joint Institute for Nuclear Astrophysics -- Center for the Evolution of the Elements (JINA-CEE), USA；Department of Astronomy, The Ohio State University, 140 West 18th Avenue, Columbus, OH 43210, USA；Center for Cosmology；AstroParticle Physics (CCAPP), The Ohio State University, Columbus, OH 43210, USA
Visual Perturbation and Adaptive Hard Negative Contrastive Learning for Compositional Reasoning in Vision-Language Models
School of Artificial Intelligence and Software Engineering, Nanyang Normal University, Henan, China（人工智能与软件工程学院，南阳师范学院，河南）；Institute for Artificial Intelligence, Peking University, Beijing, China（人工智能研究院，北京大学，北京）；Collaborative Innovation Center of Intelligent Explosion-proof Equipment, Henan, China（智能防爆设备协同创新中心，河南）
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data, augmented with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level alignment, and progressively samples minibatches where all imag…
CLIM: Contrastive Language-Image Mosaic for Region Representation
Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）
Penn State University（宾夕法尼亚州立大学）；Carnegie Mellon University（卡内基梅隆大学）
We present a simple unsupervised method for learning an encoder mapping short 3D pose sequences into embedding vectors suitable for sequence-to-sequence alignment by dynamic time warping. Training samples consist of temporal windows of frames containing 3D body points such as mocap markers or skeleton joints. A light-weight, 3-layer encoder is trained using a contrastive loss function that encourages embedding vectors of augmented sample pairs to have cosine similarity 1, and similarity 0 with all other samples in a minibatch. When multiple scripted training sequences are available, temporal a…
Image-level contrastive representation learning has proven to be highly effective as a generic model for transfer learning. Such generality for transfer learning, however, sacrifices specificity if we are interested in a certain downstream task. We argue that this could be sub-optimal and thus advocate a design principle which encourages alignment between the self-supervised pretext task and the downstream task. In this paper, we follow this principle with a pretraining method specifically designed for the task of object detection. We attain alignment in the following three aspects: 1) object-…
Various state-of-the-art self-supervised visual representation learning approaches take advantage of data from multiple sensors by aligning the feature representations across views and/or modalities. In this work, we investigate how aligning representations affects the visual features obtained from cross-view and cross-modal contrastive learning on images and point clouds. On five real-world datasets and on five tasks, we train and evaluate 108 models based on four pretraining variations. We find that cross-modal representation alignment discards complementary visual information, such as color…
University of Tübingen（蒂宾根大学）；MPI for Informatics（马克斯·普朗克信息学研究所）；Tencent PCG（腾讯平台与内容事业群）；MPI for Intelligent Systems（马克斯·普朗克智能系统研究所）
Having access to multi-modal cues (e.g. vision and audio) empowers some cognitive tasks to be done faster compared to learning from a single modality. In this work, we propose to transfer knowledge across heterogeneous modalities, even though these data modalities may not be semantically correlated. Rather than directly aligning the representations of different modalities, we compose audio, image, and video representations across modalities to uncover richer multi-modal knowledge. Our main idea is to learn a compositional embedding that closes the cross-modal semantic gap and captures the task…
Annotated medical images are typically rarer than labeled natural images since they are limited by domain knowledge and privacy constraints. Recent advances in transfer and contrastive learning have provided effective solutions to tackle such issues from different perspectives. The state-of-the-art transfer learning (e.g., Big Transfer (BiT)) and contrastive learning (e.g., Simple Siamese Contrastive Learning (SimSiam)) approaches have been investigated independently, without considering the complementary nature of such techniques. It would be appealing to accelerate contrastive learning with …
Pioneering dual-encoder pre-training works (e.g., CLIP and ALIGN) have revealed the potential of aligning multi-modal representations with contrastive learning. However, these works require a tremendous amount of data and computational resources (e.g., billion-level web data and hundreds of GPUs), which prevent researchers with limited resources from reproduction and further exploration. To this end, we propose a stack of novel methods, which significantly cut down the heavy resource dependency and allow us to conduct dual-encoder multi-modal representation alignment with limited resources. Be…
Enhancing Contrastive Learning for Geolocalization by Discovering Hard Negatives on Semivariograms
Professorship Big Geospatial Data Management, Technical University of Munich Germany（技术大学慕尼黑大数据地理空间管理教授职位）；University of Maine United States（缅因大学）；University of the Bundeswehr Munich Germany（联邦国防军大学慕尼黑）
Exploring the Potential of Contrastive Language-Image Pre-training for Multi-Source Remote Sensing Data
School of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Space-based Computing System Research Center, Zhejiang Lab（之江实验室天基计算系统研究中心）
Contrastive Knowledge Amalgamation for Unsupervised Image Classification
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；ZJU-Hangzhou Global Scientific and Technological Innovation Center（浙江大学杭州国际科创中心）
Multimodal Medical Endoscopic Image Analysis via Progressive Disentangle-aware Contrastive Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；First Affiliated Hospital, Sun Yat-sen University（中山大学第一附属医院）；College of Big Data and Internet, Shenzhen Technology University（深圳技术大学大数据与互联网学院）
Aligning Diffusion Models with Noise-Conditioned Perception
Artificial Intelligence Research Institute (AIRI)（人工智能研究 institute（AIRI））；Skolkovo Institute of Science and Technology（斯克尔科沃科学与技术研究所）；HSE University（俄罗斯高等经济学院）；Research Center for Trusted Artificial Intelligence, ISP RAS（可信人工智能研究中心，信息与通信技术研究院）；ISP RAS（信息与通信技术研究院）
Contrastive learning-based vision-language pre-training approaches, such as CLIP, have demonstrated great success in many vision-language tasks. These methods achieve cross-modal alignment by encoding a matched image-text pair with similar feature embeddings, which are generated by aggregating information from visual patches and language tokens. However, direct aligning cross-modal information using such representations is challenging, as visual patches and text tokens differ in semantic levels and granularities. To alleviate this issue, we propose a Finite Discrete Tokens (FDT) based multimod…
The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Shenzhen Research Institute of Big Data（深圳大数据研究院）；The University of Hong Kong（香港大学）；Tencent（腾讯）
To further reduce the cost of semi-supervised domain adaptation (SSDA) labeling, a more effective way is to use active learning (AL) to annotate a selected subset with specific properties. However, domain adaptation tasks are always addressed in two interactive aspects: domain transfer and the enhancement of discrimination, which requires the selected data to be both uncertain under the model and diverse in feature space. Contrary to active learning in classification tasks, it is usually challenging to select pixels that contain both the above properties in segmentation tasks, leading to the c…
