How to evaluate large language models (LLMs) cleanly has been established as an important research era to genuinely report the performance of possibly contaminated LLMs. Yet, how to cleanly evaluate the visual language models (VLMs) is an under-studied problem. We propose a novel approach to achieve such goals through data augmentation methods on the visual input information. We then craft a new visual clean evaluation benchmark with thousands of data instances. Through extensive experiments, we found that the traditional visual data augmentation methods are useful, but they are at risk of being used as a part of the training data as a workaround. We further propose using BGR augmentation to switch the colour channel of the visual information. We found that it is a simple yet effective method for reducing the effect of data contamination and fortunately, it is also harmful to be used as a data augmentation method during training. It means that it is hard to integrate such data augmentation into training by malicious trainers and it could be a promising technique to cleanly evaluate visual LLMs. Our code, data, and model weights will be released upon publication.
Reinforcing Multimodal Reasoning Against Visual Degradation
Tencent Hunyuan(腾讯文言);University of Maryland, College Park(马里兰大学 College Park 分校);University of Virginia(弗吉尼亚大学);University of North Carolina, Chapel Hill(北卡罗来纳大学 Chapel Hill 分校)
Latent Noise Mask for Reducing Visual Redundancy in Multimodal Large Language Models
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(人工智能学院、光学和电子学(iOPEN)、西北工业大学);Institute of Artificial Intelligence, China Telecom (TeleAI)(人工智能研究院、中国电信(TeleAI));Fudan University(复旦大学);The University of Hong Kong(香港大学)
No-Clean-Reference Image Super-Resolution: Application to Electron Microscopy
A. I. Virtanen Institute for Molecular Sciences, Faculty of Health Sciences, University of Eastern Finland(东芬兰大学健康科学学院A. I. Virtanen分子科学研究所);Technical University of Munich(慕尼黑工业大学)