Multimodal systems have highly complex processing pipelines and are pretrained over large datasets before being fine-tuned for specific tasks such as visual captioning. However, it becomes hard to disentangle what the model learns during the fine-tuning process from what it already knows due to its pretraining. In this work, we learn a probabilistic model using Hybrid Markov Logic Networks (HMLNs) over the training examples by relating symbolic knowledge (extracted from the caption) with visual features (extracted from the image). For a generated caption, we quantify the influence of training examples based on the HMLN distribution using probabilistic inference. We evaluate two types of inference procedures on the MSCOCO dataset for different types of captioning models. Our results show that for BLIP2 (a model that uses a LLM), the fine-tuning may have smaller influence on the knowledge the model has acquired since it may have more general knowledge to perform visual captioning as compared to models that do not use a LLM
Disentangling Reasoning Logic to Resolve Explicit Knowledge Conflicts
School of Computer Science, University of Auckland（奥克兰大学计算机科学学院）；Department of Electronics and Electrical Engineering, National Yang Ming Chiao Tung University（国立阳明交通大学电子与电机工程学系）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）；Shenzhen Key Laboratory of Internet Information Collaboration, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）深圳市互联网信息协同重点实验室）
HiSem: Hierarchical Semantic Disentangling for Remote Sensing Image Change Captioning
College of Computer Science, Inner Mongolia University（内蒙古大学计算机学院）；Research Center for Spatiotemporal Intelligence, Inner Mongolia University（内蒙古大学时空智能研究中心）；Department of Aerospace Intelligent Science and Technology, School of Astronautics, Beihang University（北航宇航学院航空航天智能科学与技术系）；State Key Laboratory of Virtual Reality Technology and Systems, Beihang University（北航虚拟现实技术与系统国家重点实验室）；China Mobile Communications Group Inner Mongolia Co., Ltd.（中国移动通信集团内蒙古有限公司）
Tongji University（同济大学）；Hong Kong Polytechnic University（香港理工大学）
Translating visual data into natural language is essential for machines to understand the world and interact with humans. In this work, a comprehensive study is conducted on video paragraph captioning, with the goal to generate paragraph-level descriptions for a given video. However, current researches mainly focus on detecting objective facts, ignoring the needs to establish the logical associations between sentences and to discover more accurate emotions related to video contents. Such a problem impairs fluent and abundant expressions of predicted captions, which are far below human language…
Microsoft Applied Sciences Group (ASG)（微软应用科学集团）；Microsoft Research AI（微软人工智能研究院）
Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements (e.g. scene graph generation) rather than reasoning. Neuro-symbolic models such as Neural Module Networks bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, and thus neural reasoning is hard to improve and assess on its own. To address this, we propose (1) a framewo…
