多模态大模型

KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detection

Yuexin Li, Chengyu Huang, Shumin Deng, Mei Lin Lock, Tri Cao, Nay Oo, Hoon Wei Lim, Bryan Hooi

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by USENIX Security 2024

Journal ref 33rd USENIX Security Symposium (USENIX Security 2024), 793--810

MAN TruckScenes: A multimodal dataset for autonomous trucking in diverse conditions

Felix Fent, Fabian Kuttenreich, Florian Ruch, Farija Rizwin, Stefan Juergens, Lorenz Lechermann, Christian Nissler, Andrea Perl, Ulrich Voll, Min Yan, Markus Lienkamp

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted to NeurIPS 2024 Datasets and Benchmarks Track

An Empirical Analysis on Spatial Reasoning Capabilities of Large Multimodal Models

Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, Yuan-Fang Li

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

M3SciQA: A Multi-Modal Multi-Document Scientific QA Benchmark for Evaluating Foundation Models

Chuhan Li, Ziyao Shangguan, Yilun Zhao, Deyuan Li, Yixin Liu, Arman Cohan

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Models

Enming Zhang, Ruobing Yao, Huanyong Liu, Junhui Yu, Jiale Wang

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Multimodal Commonsense Knowledge Distillation for Visual Question Answering

Shuo Yang, Siwen Luo, Soyeon Caren Han

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments AAAI 2025 (Accepted, Oral)

Cross-Fundus Transformer for Multi-modal Diabetic Retinopathy Grading with Cataract

Fan Xiao, Junlin Hou, Ruiwei Zhao, Rui Feng, Haidong Zou, Lina Lu, Yi Xu, Juzhao Zhang

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 4 figures

M3LEO: A Multi-Modal, Multi-Label Earth Observation Dataset Integrating Interferometric SAR and Multispectral Data

Matthew J Allen, Francisco Dorr, Joseph Alejandro Gallego Mejia, Laura Martínez-Ferrer, Anna Jungbluth, Freddie Kalaitzis, Raúl Ramos-Pollán

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures

MangaUB: A Manga Understanding Benchmark for Large Multimodal Models

Hikaru Ikuta, Leslie Wöhler, Kiyoharu Aizawa

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Comments This work has been submitted to the IEEE for possible publication

LiMTR: Time Series Motion Prediction for Diverse Road Users through Multimodal Feature Integration

Camiel Oerlemans, Bram Grooten, Michiel Braat, Alaa Alassi, Emilia Silvas, Decebal Constantin Mocanu

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted at the NeurIPS 2024 workshop Time Series in the Age of Large Models. Code available at https://github.com/Cing2/LiMTR

SemiHVision: Enhancing Medical Multimodal Models with a Semi-Human Annotated Dataset and Fine-Tuned Instruction Generation

Junda Wang, Yujan Ting, Eric Z. Chen, Hieu Tran, Hong Yu, Weijing Huang, Terrence Chen

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs

Sihang Zhao, Youliang Yuan, Xiaoying Tang, Pinjia He

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Findings

Introducing MeMo: A Multimodal Dataset for Memory Modelling in Multiparty Conversations

Maria Tsfasman, Bernd Dudzik, Kristian Fenech, Andras Lorincz, Catholijn M. Jonker, Catharine Oertel

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

MiRAGeNews: Multimodal Realistic AI-Generated News Detection

Runsheng Huang, Liam Dugan, Yue Yang, Chris Callison-Burch

专题命中 多模态评测 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2024 Findings

GSR-BENCH: A Benchmark for Grounded Spatial Reasoning Evaluation via Multimodal LLMs

Navid Rajabi, Jana Kosecka

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to NeurIPS 2024 Workshop on Compositional Learning

Needle In A Multimodal Haystack

Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, Wenqi Shao, Wenhai Wang

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted to NeurIPS 2024 Track Datasets and Benchmarks

DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM

Xuchen Li, Shiyu Hu, Xiaokun Feng, Dailing Zhang, Meiqi Wu, Jing Zhang, Kaiqi Huang

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Preprint, Under Review

R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?

Chunyi Li, Jianbo Zhang, Zicheng Zhang, Haoning Wu, Yuan Tian, Wei Sun, Guo Lu, Xiaohong Liu, Xiongkuo Min, Weisi Lin, Guangtao Zhai

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Exploring Attention Mechanisms in Integration of Multi-Modal Information for Sign Language Recognition and Translation

Zaber Ibn Abdul Hakim, Rasman Mubtasim Swargo, Muhammad Abdullah Adnan

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription

Reza Basiri, Ali Abedi, Chau Nguyen, Milos R. Popovic, Shehroz S. Khan

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 13 pages, 3 figures, ICPR 2024 Conference (PRHA workshop)

BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data

Xuwu Wang, Qiwen Cui, Yunzhe Tao, Yiran Wang, Ziwei Chai, Xiaotian Han, Boyi Liu, Jianbo Yuan, Jing Su, Guoyin Wang, Tingkai Liu, Liyu Chen, Tianyi Liu, Tao Sun, Yufeng Zhang, Sirui Zheng, Quanzeng You, Yang Yang, Hongxia Yang

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Insight: A Multi-Modal Diagnostic Pipeline using LLMs for Ocular Surface Disease Diagnosis

Chun-Hsiao Yeh, Jiayun Wang, Andrew D. Graham, Andrea J. Liu, Bo Tan, Yubei Chen, Yi Ma, Meng C. Lin

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to MICCAI 2024. Project Webpage: https://danielchyeh.github.io/MDPipe/

MM-Conv: A Multi-modal Conversational Dataset for Virtual Humans

Anna Deichler, Jim O'Regan, Jonas Beskow

专题命中 多模态评测 ：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL

MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmark

Elliot L. Epstein, Kaisheng Yao, Jing Li, Xinyi Bai, Hamid Palangi

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 24 pages, 16 figures

MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features

Katharina Anderer, Andreas Reich, Matthias Wölfel

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Journal ref Proceedings of Interspeech 2024

Model-in-the-Loop (MILO): Accelerating Multimodal AI Data Annotation with LLMs

Yifan Wang, David Stevens, Pranay Shah, Wenwen Jiang, Miao Liu, Xu Chen, Robert Kuo, Na Li, Boying Gong, Daniel Lee, Jiabo Hu, Ning Zhang, Bob Kamma

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Multi-modal Auto-regressive Modeling via Visual Words

Tianshuo Peng, Zuchao Li, Lefei Zhang, Hai Zhao, Ping Wang, Bo Du

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments ACM MM 2024

m&m's: A Benchmark to Evaluate Tool-Use for multi-step multi-modal Tasks

Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, Ranjay Krishna

专题命中 多模态评测 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking

Ting-Chih Chen, Chia-Wei Tang, Chris Thomas

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 16 pages, 7 figures, The 62nd Annual Meeting of the Association for Computational Linguistics

Mutual Information-based Representations Disentanglement for Unaligned Multimodal Language Sequences

Fan Qian, Jiqing Han, Jianchen Li, Yongjun He, Tieran Zheng, Guibin Zheng

专题命中 多模态评测 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments 31 pages, 8 figures