Vision-extended LLMs have made significant strides in Visual Question Answering (VQA). Despite these advancements, VLLMs still encounter substantial difficulties in handling queries involving long-tail entities, with a tendency to produce erroneous or hallucinated responses. In this work, we introduce a novel evaluative benchmark named \textbf{SnapNTell}, specifically tailored for entity-centric VQA. This task aims to test the models' capabilities in identifying entities and providing detailed, entity-specific knowledge. We have developed the \textbf{SnapNTell Dataset}, distinct from traditional VQA datasets: (1) It encompasses a wide range of categorized entities, each represented by images and explicitly named in the answers; (2) It features QA pairs that require extensive knowledge for accurate responses. The dataset is organized into 22 major categories, containing 7,568 unique entities in total. For each entity, we curated 10 illustrative images and crafted 10 knowledge-intensive QA pairs. To address this novel task, we devised a scalable, efficient, and transparent retrieval-augmented multimodal LLM. Our approach markedly outperforms existing methods on the SnapNTell dataset, achieving a 66.5\% improvement in the BELURT score. We will soon make the dataset and the source code publicly accessible.
Knowledge-Based Visual Question Answering (KBVQA) is a bi-modal task requiring external world knowledge in order to correctly answer a text question and associated image. Recent single modality text work has shown knowledge injection into pre-trained language models, specifically entity enhanced knowledge graph embeddings, can improve performance on downstream entity-centric tasks. In this work, we empirically study how and whether such methods, applied in a bi-modal setting, can improve an existing VQA system's performance on the KBVQA task. We experiment with two large publicly available VQA…
GRACE: Boosting Video MLLMs with Grounded Action-Centric Evidence for Viewer Sentiment Prediction
Shanghai Jiao Tong University（上海交通大学）；Hangzhou Dianzi University（杭州电子科技大学）；The 52nd Research Institute of China Electronics Technology Group Corporation（中国电子科技集团公司第五十二研究所）；Hangzhou Bywin Technology Co., Ltd.（杭州百威科技有限公司）；Zhejiang Dahua Technology Co., Ltd.（浙江大华技术股份有限公司）；School of Information Science and Engineering, Shandong University（山东大学信息科学与工程学院）；Haihe Laboratory of Information Technology Application Innovation（海河信息技术应用创新实验室）
HARE: an entity and relation centric evaluation framework for histopathology reports
University College London（伦敦大学学院）；University of Glasgow（格拉斯哥大学）；Human Technopole（人类技术极地）；University College London Hospitals NHS Foundation Trust（伦敦大学学院医院国家健康服务信托）
本文提出以实体和关系为中心的组织病理学报告评估框架 HARE，通过构建标注基准、微调 NER 与 RE 模型并设计新指标，在专家评估相关性上优于传统及放射学指标。
University of California Berkeley（加州大学伯克利分校）；Stanford University（斯坦福大学）；Massachusetts Institute of Technology（麻省理工学院）
This paper tests the hypothesis that modeling a scene in terms of entities and their local interactions, as opposed to modeling the scene globally, provides a significant benefit in generalizing to physical tasks in a combinatorial space the learner has not encountered before. We present object-centric perception, prediction, and planning (OP3), which to the best of our knowledge is the first fully probabilistic entity-centric dynamic latent variable framework for model-based reinforcement learning that acquires entity representations from raw visual observations without supervision and uses t…
With massive explosion of social media such as Twitter and Instagram, people daily share billions of multimedia posts, containing images and text. Typically, text in these posts is short, informal and noisy, leading to ambiguities which can be resolved using images. In this paper we explore text-centric Named Entity Recognition task on these multimedia posts. We propose an end to end model which learns a joint representation of a text and an image. Our model extends multi-dimensional self attention technique, where now image helps to enhance relationship between words. Experiments show that ou…
Beihang University（北京航空航天大学）；Sea AI Lab（Sea AI实验室）
Vision and language understanding techniques have achieved remarkable progress, but currently it is still difficult to well handle problems involving very fine-grained details. For example, when the robot is told to "bring me the book in the girl's left hand", most existing methods would fail if the girl holds one book respectively in her left and right hand. In this work, we introduce a new task named human-centric relation segmentation (HRS), as a fine-grained case of HOI-det. HRS aims to predict the relations between the human and surrounding entities and identify the relation-correlated hu…
