Generating natural, diverse, and meaningful questions from images is an essential task for multimodal assistants as it confirms whether they have understood the object and scene in the images properly. The research in visual question answering (VQA) and visual question generation (VQG) is a great step. However, this research does not capture questions that a visually-abled person would ask multimodal assistants. Recently published datasets such as KB-VQA, FVQA, and OK-VQA try to collect questions that look for external knowledge which makes them appropriate for multimodal assistants. However, they still contain many obvious and common-sense questions that humans would not usually ask a digital assistant. In this paper, we provide a new benchmark dataset that contains questions generated by human annotators keeping in mind what they would ask multimodal digital assistants. Large scale annotations for several hundred thousand images are expensive and time-consuming, so we also present an effective way of automatically generating questions from unseen images. In this paper, we present an approach for generating diverse and meaningful questions that consider image content and metadata of image (e.g., location, associated keyword). We evaluate our approach using standard evaluation metrics such as BLEU, METEOR, ROUGE, and CIDEr to show the relevance of generated questions with human-provided questions. We also measure the diversity of generated questions using generative strength and inventiveness metrics. We report new state-of-the-art results on the public and our datasets.
Agri-LLaVA: Knowledge-Infused Large Multimodal Assistant on Agricultural Pests and Diseases
China Three Gorges University（三峡大学）；College of Computer and Information Technology, China Three Gorges University（三峡大学计算机与信息学院）；University of Birmingham（伯明翰大学）；Southern University of Science and Technology（南方科技大学）
AgriDoctor: A Multimodal Intelligent Assistant for Agriculture
State Key Laboratory of Multimodal Artificial Intelligence Systems（多模态人工智能系统国家重点实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly
Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；National Institute of Advanced Industrial Science and Technology (AIST)（国立研究开发法人产业技术综合研究所（AIST））
IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants
Fraunhofer IPK（弗劳恩霍夫研究所）；Technical University of Berlin（柏林技术大学）；University of Tübingen（图宾根大学）；RWTH Aachen University（亚琛工业大学）；Leibniz University Hannover（汉诺威莱布尼茨大学）
ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；School of Engineering, Westlake University（西湖大学工程学院）；University of Minnesota – Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）
Probing the limitations of multimodal language models for chemistry and materials research
Friedrich Schiller University Jena（弗里德里希·席勒大学耶拿分校）；Indian Institute of Technology Delhi（印度理工学院德里分校）；Center for Energy and Environmental Chemistry Jena (CEEC Jena)（耶拿能源与环境化学中心）；Helmholtz Institute for Polymers in Energy Applications Jena (HIPOLE Jena)（耶拿亥姆霍兹能源应用聚合物研究所）；Jena Center for Soft Matter (JCSM)（耶拿软物质中心）
EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
Tsinghua University（清华大学）；Tongji University（同济大学）；Renmin University of China（中国人民大学）；The University of Tokyo（东京大学）；Lenovo Group（联想集团）；Peking University（北京大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Shanghai Qi Zhi Institute（上海启智研究院）
A Foundational Multimodal Vision Language AI Assistant for Human Pathology
Brigham and Women’s Hospital（布莱根妇女医院）；Harvard Medical School（哈佛医学院）；Massachusetts General Hospital（麻省总医院）；Broad Institute of Harvard and MIT（哈佛-麻省理工博德研究所）；Dana-Farber Cancer Institute（达纳-法伯癌症研究所）；Massachusetts Institute of Technology (MIT)（麻省理工学院）；Harvard John A. Paulson School of Engineering And Applied Sciences（哈佛大学约翰·A·保尔森工程与应用科学学院）；Harvard University（哈佛大学）；Wexner Medical Center, Ohio State University（俄亥俄州立大学韦克斯纳医学中心）；Ohio State University（俄亥俄州立大学）；Harvard Data Science Initiative（哈佛大学数据科学计划）
Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic；University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic；Centrum Wiskunde \& Informatic Leiden University University College London；Vrije University of Amsterdam；Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University；Vrije University of Amsterdam Centrum Wiskunde \& Informatic
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China（中国科学院人工智能安全重点实验室，计算技术研究所，中国科学院，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Tsinghua University, Beijing, China（清华大学，北京，中国）
TalkFashion: Intelligent Virtual Try-On Assistant Based on Multimodal Large Language Model
School of Electronic and Computer Engineering, Peking University, China（北京大学电子与计算机工程学院）；Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University（北京大学深圳研究生院）
RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models
School of Computing and Augmented Intelligence, Arizona State University（亚利桑那州立大学计算与增强智能学院）；School of Computing, Clemson University（克莱姆森大学计算机学院）；Washington University School of Medicine in St. Louis（圣路易斯华盛顿大学医学院）；The Keck School of Medicine of the University of Southern California（南加州大学凯克医学院）；Mayo Clinic（梅奥诊所）；Banner Alzheimer’s Institute（班纳阿尔茨海默病研究所）
OphGLM: Training an Ophthalmology Large Language-and-Vision Assistant based on Instructions and Dialogue
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；National Health Commission Capacity Building and Continuing Education Center（国家卫生健康委能力建设和继续教育中心）；Beijing Tongren Hospital（北京同仁医院）
GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinement
Institute of Material Systems Modeling（材料系统建模研究所）；Helmholtz Zentrum Hereon（海德堡研究中心Hereon）；Institute of Material and Process Design（材料与加工设计研究所）；German Research Center for Artificial Intelligence（德国人工智能研究中心）
LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound
ZJU-UIUC Institute, Zhejiang University（浙江大学ZJU-UIUC学院）；University of Washington（华盛顿大学）；Zhejiang University School of Medicine Sir Run Run Shaw Hospital（浙江大学医学院附属邵逸夫医院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）
UFM: Unified Feature Matching Pre-training with Multi-Modal Image Assistants
Information Science and Technology College, Dalian Maritime University（大连海事大学信息科学技术学院）；National Pilot School of Software, Yunnan University（云南大学国家示范性软件学院）；Yunnan Lanyi Network Technology Co（云南揽翼网络科技有限公司）
VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding
The Chinese University of Hong Kong（香港中文大学）；Ant Group（蚂蚁集团）；Tsinghua University（清华大学）；Tongji University（同济大学）；The Hong Kong University of Science and Technology（香港科技大学）
