FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models
Beijing Institute of Technology(北京理工大学);BIGAI(北京智源人工智能研究院);Peking University(北京大学);Monash University(莫纳什大学);Shenzhen MSU-BIT University(深圳北理莫斯科大学);Tsinghua University(清华大学)
Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun Zhu, Qing Li
Vision language models (VLMs) have achieved impressive progress in diverse applications, becoming a prevalent research direction. In this paper, we build FIRE, a feedback-refinement dataset, consisting of 1.1M multi-turn conversations that are derived from 27 source datasets, empowering VLMs to spontaneously refine their responses based on user feedback across diverse tasks. To scale up the data collection, FIRE is collected in two components: FIRE-100K and FIRE-1M, where FIRE-100K is generated by GPT-4V, and FIRE-1M is freely generated via models trained on FIRE-100K. Then, we build FIRE-Bench, a benchmark to comprehensively evaluate the feedback-refining capability of VLMs, which contains 11K feedback-refinement conversations as the test data, two evaluation settings, and a model to provide feedback for VLMs. We develop the FIRE-LLaVA model by fine-tuning LLaVA on FIRE-100K and FIRE-1M, which shows remarkable feedback-refining capability on FIRE-Bench and outperforms untrained VLMs by 50%, making more efficient user-agent interactions and underscoring the significance of the FIRE dataset.
FUELVISION: A Multimodal Data Fusion and Multimodel Ensemble Algorithm for Wildfire Fuels Mapping
University of California Los Angeles(加利福尼亚大学洛杉矶分校);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Desert Research Institute(沙漠研究所);Amazon(亚马逊公司);United States Forest Service(美国林务局)
本文提出 FUELVISION 算法,融合多源遥感与地形数据,结合生成式 AI 合成数据增强训练集,采用多模型集成方法实现近实时野火可燃物制图,准确率近 80%。
MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment
Qiushi Academy for Advanced Studies (QAAS), Zhejiang University(浙江大学启斯特先进研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Software Technology, Zhejiang University(浙江大学软件学院);State Key Lab of Brain-Machine Intelligence(脑机智能国家重点实验室);Collaborative Innovation Center for Artificial Intelligence by MOE and Zhejiang Provincial Government (ZJU)(教育部-浙江省人工智能协同创新中心);Zhejiang Lab(浙江实验室)
LDRFusion: A LiDAR-Dominant multimodal refinement framework for 3D object detection
School of Computer Science and Technology, Tongji University, Shanghai 201804, China(计算机科学与技术学院,同济大学,上海);School of Electronics and Information Engineering, Tongji University, Shanghai 201804, China(电子信息工程学院,同济大学,上海)
Hierarchical Refinement of Universal Multimodal Attacks on Vision-Language Models
School of Electrical Engineering and Computer Science, the University of Queensland(电气工程与计算机科学学院,昆士兰大学);Department of Computer Science, City University of Hong Kong(计算机科学系,香港城市大学)
Motion Transformer with Global Intention Localization and Local Movement Refinement
Max Planck Institute for Informatics(马克斯·普朗克信息学研究所)
预测交通参与者的多模态未来行为对于机器人车辆做出安全决策至关重要。现有的工作探索直接预测未来的轨迹的基础上潜在的功能或利用密集的目标候选人,以确定代理的目的地,其中前一种策略收敛缓慢,因为所有的运动模式都来自相同的功能,而后者的策略有效率问题,因为它的性能高度依赖于目标候选人的密度。在本文中,我们提出了运动Transformer(MTR)框架,模型运动预测的全局意图定位和局部运动细化的联合优化。而不是使用目标候选人,MTR采用了一个小的可学习的运动查询对集的空间意图先验。每个运动查询对负责特定运动模式的轨迹预测和细化,这稳定了训练过程并有助于更好地进行多模态预测。实验表明,MTR在边缘和联合运动预测挑战上都达到了最先进的性能,在Waymo Open Motion Dataset的排行榜上排名第一。源代码可在https://github.com/sshaoshuai/MTR上获得。
FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval
Shandong University(山东大学);City University of Hong Kong(香港城市大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Shandong Jianzhu University(山东建筑大学)
MERLIN: Multimodal Embedding Refinement via LLM-based Iterative Navigation for Text-Video Retrieval-Rerank Pipeline
Buzzni AI Lab(Buzzni人工智能实验室);KAIST(韩国科学技术院);Mathpresso Inc.(Mathpresso公司);UC Berkeley(加州大学伯克利分校);Fainders AI(Fainders人工智能公司);Seoul National University(首尔大学)
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
School of Engineering, Westlake University(西湖大学工程学院);City University of Hong Kong(香港城市大学);Research Center for Industries of the Future, Westlake University(西湖大学未来产业研究中心);Johns Hopkins University(约翰斯·霍普金斯大学)
GReFEM: Multimodal LLMs as Zero-Shot Semantic Assistants for Physics-Guided 3D Mesh Refinement
Institute of Material Systems Modeling(材料系统建模研究所);Helmholtz Zentrum Hereon(海德堡研究中心Hereon);Institute of Material and Process Design(材料与加工设计研究所);German Research Center for Artificial Intelligence(德国人工智能研究中心)
Seeing Symbols, Missing Cultures: Probing Vision-Language Models' Reasoning on Fire Imagery and Cultural Meaning
Duncan of Jordanstone College of Art & Design (DJCAD), University of Dundee(邓迪大学邓迪艺术与设计学院(DJCAD));Guangzhou Institute of Science and Technology (GZIST)(广州科学技术研究院);Faculty of Arts, Xiamen University(厦门大学艺术学院);University of Birmingham(伯明翰大学)
VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
National Engineering Research Center for Software Engineering, Peking University(软件工程国家工程研究中心,北京大学);Alibaba Group(阿里巴巴集团);ZEEKR Intelligent Technology Holding Limited(ZEKR智能科技控股有限公司)
Transformer-Driven Multimodal Fusion for Explainable Suspiciousness Estimation in Visual Surveillance
Big Data Research and Supercomputing Division, CSIR Fourth Paradigm Institute(CSIR第四范式研究所大数据研究与超级计算部门);Department of Electrical Engineering, Bharti School of Telecommunication, Yardi School of Artificial Intelligence, IIT Delhi(电信学院电子工程系、Yardi人工智能学院、德里理工学院)