As computer vision and NLP make progress, Vision-Language(VL) is becoming an important area of research. Despite the importance, evaluation metrics of the research domain is still at a preliminary stage of development. In this paper, we propose a quantitative metric "Equivariance Score" and evaluation dataset "Human Puzzle" to assess whether a VL model is understanding an image like a human. We observed that the VL model does not interpret the overall context of an input image but instead shows biases toward a specific object or shape that forms the local context. We aim to quantitatively measure a model's performance in understanding context. To verify the current existing VL model's capability, we sliced the original input image into pieces and randomly placed them, distorting the global context of the image. Our paper discusses each VL model's level of interpretation on global context and addresses how the structural characteristics influenced the results.
VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision
University of Information Technology, VNU-HCM(越南国立胡志明市大学信息技术大学);University College Cork(科克大学);Hanoi University of Science and Technology(河内科技大学);Trinity College Dublin(都柏林圣三一大学)
Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Information, Renmin University of China(中国人民大学信息学院);Alibaba Group(阿里巴巴集团)
Can Vision-Language Models Judge Olympic Diving? From Reasoning to Scores in Zero-Shot Action Quality Assessment
ESPOL Polytechnic University(ESPOL理工大学);University of Granada(格拉纳达大学);Universidad de Las Palmas de Gran Canaria(拉斯帕尔马斯·德·大加那利大学);Michigan Technological University(密歇根理工大学)
Automated Evaluation of Large Vision-Language Models on Self-driving Corner Cases
Hong Kong University of Science and Technology(香港科技大学);Dalian University of Technology(大连理工大学);The Chinese University of Hong Kong(香港中文大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement
Business School, Shandong University of Technology, Shandong, China(山东科技大学商学院);Institute of International Language Services Studies, Macau Millennium College, Macau SAR, China(国际语言服务研究所,澳门 Millennium 学院)
Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
From 3D Pose to Prose: Biomechanics-Grounded Vision--Language Coaching
Department of Computer Science, Drexel University(德雷塞尔大学计算机科学系);Department of Computer Science and Engineering, Michigan State University(密歇根州立大学计算机科学与工程系)
Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style
Columbia University, Department of Computer Science(哥伦比亚大学计算机科学系);Columbia University, Department of Art History & Archaeology(哥伦比亚大学艺术史与考古系);University of Texas at Austin(德克萨斯大学奥斯汀分校);UNC Chapel Hill(北卡罗来纳大学教堂山分校)
Does Your 3D Encoder Really Work? When Pretrain-SFT from 2D VLMs Meets 3D VLMs
Shenzhen campus of Sun Yat-sen University(中山大学深圳校区);Huawei Noah’s Ark Lab(华为诺亚实验室);MBZUAI;Peng Cheng Laboratory(鹏城实验室);Guangdong Key Laboratory of Big Data Analysis and Processing(广东省大数据分析与处理重点实验室)
本文分析以 3D 场景为中心的 VLM 性能受限原因,发现其弱用 3D 编码器、依赖语言捷径,并提出 3D Relevance Discrimination QA 数据集以促进真实 3D 理解。
3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
Roblox Corporation(罗布乐思公司);Berkeley AI Research Lab & Department of Industrial Engineering and Operations Research, University of California, Berkeley(加州大学伯克利分校伯克利人工智能研究实验室及工业工程与运筹学系);Computer Science Department, Stanford University(斯坦福大学计算机科学系);Division of Biostatistics, University of California, Berkeley(加州大学伯克利分校生物统计学系)
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?
University of Chicago(芝加哥大学);NUS(新加坡国立大学);Stanford University(斯坦福大学);UCSC(加州大学圣克鲁兹分校);Duke University(杜克大学);ESSEC(埃塞克高等商学院);Illinois Tech(伊利诺伊理工学院);UCSD(加州大学圣地亚哥分校);Peking University(北京大学);University of Queensland(昆士兰大学);Stony Brook University(石溪大学);UNC-Chapel Hill(北卡罗来纳大学教堂山分校);USTC(中国科学技术大学)
GeoArena: Evaluating Open-World Geographic Reasoning in Large Vision-Language Models
Department of Data Science, City University of Hong Kong(香港城市大学数据科学系);Department of Computer Sciences, University of Wisconsin-Madison(威斯康星大学麦迪逊分校计算机科学系)
Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping
School of Artificial Intelligence, Shanghai Jiao Tong University(上海交通大学人工智能学院);Shanghai AI Laboratory(上海人工智能实验室);Tongji University(同济大学);The University of Hong Kong(香港大学)
What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Models
Institute of Digital Twin, Eastern Institute of Technology(数字孪生研究所,东部技术研究所);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative(宁波空间智能与数字衍生关键实验室);Shanghai Jiao Tong University(上海交通大学);The Hong Kong Polytechnic University(香港理工大学)