Visual Enumeration Remains Challenging for Multimodal Generative AI
Department of General Psychology and Department of Mathematics University of Padova（帕多瓦大学心理学系和数学系）；Department of General Psychology University of Padova（帕多瓦大学心理学系）；Department of General Psychology and Padova Neuroscience Center University of Padova（帕多瓦大学心理学系和帕多瓦神经科学中心）；IRCSS San Camillo Hospital, Venice-Lido（威尼斯利多医院IRCSS桑卡莫医院）
Many animal species can approximately judge the number of objects in a visual scene at a single glance, and humans can further determine the exact cardinality of a set by deploying systematic counting procedures. In contrast, it has been observed that even state-of-the-art AI systems have very limited enumeration skills. In this work, we propose two benchmark tasks inspired by cognitive science that allow to precisely evaluate the visual enumeration capabilities of multimodal foundation models, thereby providing an objective measure of their number sense and counting level. We consider popular visual question answering models (BLIP, LLaVA and ViLT) as well as advanced image-to-text (Gemini, GPT and Qwen) and text-to-image (DALL-E, FLUX and Stable Diffusion) AI systems. Our analyses show that even the most advanced models cannot reliably name the number of objects in simple visual stimuli or generate images containing a target number of items, as indexed by their low accuracy in both types of tasks. Especially for numbers outside the subitizing range, their responses are often far from the target numerosity, and, in stark contrast with human behavior, in many cases the distribution of errors depends on the object category. We also observe some striking mistakes with small numbers. Our findings demonstrate that developing an intuitive visual understanding of number remains challenging for AI models and that merely increasing model size might not be a viable strategy to promote the emergence of systematic counting skills. We release the full code of our benchmark to facilitate the evaluation of enumeration skills in future AI systems.
Recent video and language pretraining frameworks lack the ability to generate sentences. We present Multimodal Video Generative Pretraining (MV-GPT), a new pretraining framework for learning from unlabelled videos which can be effectively used for generative tasks such as multimodal video captioning. Unlike recent video-language pretraining frameworks, our framework trains both a multimodal video encoder and a sentence decoder jointly. To overcome the lack of captions in unlabelled videos, we leverage the future utterance as an additional text source and propose a bidirectional generation obje…
Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning
The Hong Kong Polytechnic University（香港理工大学）；Singapore Management University（新加坡国立大学）；Shandong University（山东大学）；University of Science and Technology of China（中国科学技术大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Evaluating Compositional Scene Understanding in Multimodal Generative Models
Department of Psychology, University of California, Los Angeles（加州大学洛杉矶分校心理学系）；Microsoft Research, NYC（微软研究院纽约分部）；Air Force Research Laboratory（美国空军研究实验室）；Department of Statistics, University of California, Los Angeles（加州大学洛杉矶分校统计学系）
School of Informatics, Xiamen University, China（厦门大学信息学院）；WeChat AI, Tencent Inc, China（腾讯公司微信AI部门）；Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China（福建省和台湾非物质文化遗产数字化保护与智能处理重点实验室）；Shanghai Artificial Intelligence Laboratory, China（上海人工智能实验室）
TerraMind: Large-Scale Generative Multimodality for Earth Observation
IBM Research – Europe（IBM欧洲研究院）；ETH Zurich（苏黎世联邦理工学院）；Forschungszentrum Jülich（尤利希研究中心）；European Space Agency（欧洲航天局）；Φ \Phi -Lab（Φ实验室）；NASA IMPACT；University of Iceland（爱沙尼亚大学）
GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection
State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS)（人工智能安全国家重点实验室，计算技术研究所（ICT），中国科学院（CAS））；University of Chinese Academy of Sciences, CAS（中国科学院大学）；Pengcheng Laboratory（鹏城实验室）；Shenzhen Stock Exchange（深圳证券交易所）
LLM4Rec: Large Language Models for Multimodal Generative Recommendation with Causal Debiasing
Department of Software \& Microelectronics Peking University Beijing, China；Department of Software \& Microelectronics Peking University Beijing, China hangli\；Department of Software \& Microelectronics Peking University Beijing, China zehua\；Department of Software \& Microelectronics Peking University Beijing, China xiaofan\；Economic Law School China University of Political Science；Peking University Changsha, China
InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
University of California, Davis（加利福尼亚大学戴维斯分校）；Bosch Research North America（博世北美研究院）；Bosch Center for Artificial Intelligence (BCAI)（博世人工智能中心）；Robert Bosch GmbH（罗伯特·博世有限公司）
Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers
Georgetown University（乔治城大学）；University of Southern California（南加州大学）；University of Maryland, College Park（马里兰大学学院公园分校）；Arizona State University（亚利桑那州立大学）
Test-Time Scaling Strategies for Generative Retrieval in Multimodal Conversational Recommendations
Research Center for Information Technology Innovation, Academia Sinica（资讯科技创新研究所以）；NVIDIA（NVIDIA公司）；Department of Computer Science, National Chengchi University（国立政治大学计算机科学系）
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
School of Computer Science, South China Normal University（华南师范大学计算机科学学院）；School of Information Technology in Education, South China Normal University（华南师范大学教育信息技术学院）
Recent years have witnessed astonishing advances in the field of multimodal representation learning, with contrastive learning being the cornerstone for major breakthroughs. Latest works delivered further improvements by incorporating different objectives such as masked modeling and captioning into the frameworks, but our understanding on how these objectives facilitate learning remains vastly incomplete. In this paper, we leverage the fact that classifier-guided diffusion models generate images that reflect the semantic signals provided by the classifier to study the characteristics of multim…
Many applications of generative models rely on the marginalization of their high-dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginalization more computationally tractable. However, sparse normalization functions usually require alternative loss functions for training since the log-likelihood is undefined for sparse probability distributions. Furthermore, many sparse normalization functions often collapse the multimodality of distributions. In this work, we present $\textit{ev-softmax}$, a sparse normalization…
Virginia Tech（弗吉尼亚理工大学）；U. S. Army Corps of Engineers（美国陆军工程兵团）；Google Inc.（谷歌公司）
Deriving event storylines is an effective summarization method to succinctly organize extensive information, which can significantly alleviate the pain of information overload. The critical challenge is the lack of widely recognized definition of storyline metric. Prior studies have developed various approaches based on different assumptions about users' interests. These works can extract interesting patterns, but their assumptions do not guarantee that the derived patterns will match users' preference. On the other hand, their exclusiveness of single modality source misses cross-modality info…
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey
School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；School of Computer Science, University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校计算机科学学院）；State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；New Laboratory of Pattern Recognition, CASIA（中国科学院自动化研究所新模式识别实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
IBM Research（IBM研究院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
Storytelling is an open-ended task that entails creative thinking and requires a constant flow of ideas. Natural language generation (NLG) for storytelling is especially challenging because it requires the generated text to follow an overall theme while remaining creative and diverse to engage the reader. In this work, we introduce a system and a web-based demo, FairyTailor, for human-in-the-loop visual story co-creation. Users can create a cohesive children's fairytale by weaving generated texts and retrieved images with their input. FairyTailor adds another modality and modifies the text gen…
