多模态大模型

Deep histological synthesis from mass spectrometry imaging for multimodal registration

Kimberley M. Bird, Xujiong Ye, Alan M. Race, James M. Brown

机构 * University of Lincoln（林肯大学） ； University of Exeter（埃克塞特大学） ； AstraZeneca Computational Pathology GmbH（阿斯利康计算病理学 GmbH） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments Medical Image Understanding and Analysis (MIUA) 2025 Extended Abstract Submission

On VLMs for Diverse Tasks in Multimodal Meme Classification

Deepesh Gavit, Debajyoti Mazumder, Samiran Das, Jasabanta Patro

机构 * Peng Wang and Shuai Bai and Sinan Tan and Shijie Wang and Zhihao Fan and Jinze Bai and Keqin Chen and Xuejing Liu and Jialin Wang and Wenbin Ge and Yang Fan and Kai Dang and Mengfei Du and Xuancheng Ren and Rui Men and Dayiheng Liu and Chang Zhou and Jingren Zhou and Junyang Lin（研究人员） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CL

Comments 16 pages

SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion

Ahmed Nassar, Andres Marafioti, Matteo Omenetti, Maksym Lysak, Nikolaos Livathinos, Christoph Auer, Lucas Morin, Rafael Teixeira de Lima, Yusik Kim, A. Said Gurbuz, Michele Dolfi, Miquel Farré, Peter W. J. Staar

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments 24 pages, 10 figures

GCS-M3VLT: Guided Context Self-Attention based Multi-modal Medical Vision Language Transformer for Retinal Image Captioning

Teja Krishna Cherukuri, Nagur Shareef Shaik, Jyostna Devi Bodapati, Dong Hye Ye

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments This paper has been accepted for presentation at the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2025)

Attacks on multimodal models

Viacheslav Iablochnikov, Alexander Rogachev

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments 19 pages, 13 figures, 3 tables

Multi-Modal CLIP-Informed Protein Editing

Mingze Yin, Hanjing Zhou, Yiheng Zhu, Miao Lin, Yixuan Wu, Jialu Wu, Hongxia Xu, Chang-Yu Hsieh, Tingjun Hou, Jintai Chen, Jian Wu

专题命中 图文多模态 ：multi-modal(title)；分类 cs.AI

Comments 13 pages, 7 figures, 5 tables

Journal ref Health Data Science, 2024

Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations

Chenyu You, Yifei Min, Weicheng Dai, Jasjeet S. Sekhon, Lawrence Staib, James S. Duncan

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments Accepted by CVPR 2024

Towards Multimodal In-Context Learning for Vision & Language Models

Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Wei Lin, Amit Alfassy, Assaf Arbelle, Shimon Ullman, Leonid Karlinsky

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Enhancing Weakly Supervised Semantic Segmentation with Multi-modal Foundation Models: An End-to-End Approach

Elham Ravanbakhsh, Cheng Niu, Yongqing Liang, J. Ramanujam, Xin Li

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Fewshot learning on global multimodal embeddings for earth observation tasks

Matt Allen, Francisco Dorr, Joseph A. Gallego-Mejia, Laura Martínez-Ferrer, Anna Jungbluth, Freddie Kalaitzis, Raúl Ramos-Pollán

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments 9 pages, 6 figures, presented on NeurIPS workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models

Multimodal Search on Iconclass using Vision-Language Pre-Trained Models

Cristian Santini, Etienne Posthumus, Mary Ann Tan, Oleksandra Bruns, Tabea Tietz, Harald Sack

专题命中 图文多模态 ：multimodal(title)；分类 cs.AI

Uncurated Image-Text Datasets: Shedding Light on Demographic Bias

Noa Garcia, Yusuke Hirota, Yankun Wu, Yuta Nakashima

专题命中 图文多模态 ：image-text(title)；分类 cs.CV

Comments CVPR 2023

Task-Oriented Multi-Modal Mutual Leaning for Vision-Language Models

Sifan Long, Zhen Zhao, Junkun Yuan, Zichang Tan, Jiangjiang Liu, Luping Zhou, Shengsheng Wang, Jingdong Wang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, Tiberiu Popa

专题命中 图文多模态 ：image-text(title)；分类 cs.CV

Comments 8 pages, 8 figures, Accepted at SIGGRAPH ASIA 2022, Project Page at https://www.nasir.lol/clipmesh

How to find a good image-text embedding for remote sensing visual question answering?

Christel Chappuis, Sylvain Lobry, Benjamin Kellenberger, Bertrand Le Saux, Devis Tuia

专题命中 图文多模态 ：image-text(title)；分类 cs.CV

Comments 10 pages, 4 figures, presented in the MACLEAN workshop during ECML PKDD 2021

Multi-Modal Image Captioning for the Visually Impaired

Hiba Ahsan, Nikita Bhalla, Daivat Bhatt, Kaivankumar Shah

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CL

Comments 8 pages, 2 figures, 2 tables, accepted to NAACL-HLT SRW 2021

Multi-modal Feature Fusion with Feature Attention for VATEX Captioning Challenge 2020

Ke Lin, Zhuoxin Gan, Liwei Wang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching

Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzhu Zhang, Bin Wang, Yongdong Zhang

专题命中 图文多模态 ：image-text(title)；分类 cs.MM

Comments Accepted by ACMMM2019

Towards Unsupervised Image Captioning with Shared Multimodal Embeddings

Iro Laina, Christian Rupprecht, Nassir Navab

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments ICCV 2019

Image Captioning with Clause-Focused Metrics in a Multi-Modal Setting for Marketing

Philipp Harzig, Dan Zecha, Rainer Lienhart, Carolin Kaiser, René Schallner

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments 6 pages, accepted at MIPR 2019

Multimodal Image Captioning for Marketing Analysis

Philipp Harzig, Stephan Brehm, Rainer Lienhart, Carolin Kaiser, René Schallner

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments 4 pages, 1 figure, accepted at MIPR2018

Joint Image-Text News Topic Detection and Tracking with And-Or Graph Representation

Weixin Li, Jungseock Joo, Hang Qi, Song-Chun Zhu

专题命中 图文多模态 ：image-text(title)；分类 cs.CL

StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models

StateSight：评测视觉语言模型中的潜在空间状态重建能力

Michelle Lin

机构 * Thomas Jefferson High School for Science and Technology（托马斯·杰斐逊科学技术高中） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本研究推出StateSight基准及配套数据集StateSight-Steps，评估视觉语言模型的潜在空间状态重建能力，发现GPT-5.5、Claude Sonnet 5的表现均逊于人类基线，格式正确的响应可能掩盖空间结构恢复失败的问题。

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment

回答前看清楚：通过显著性驱动的感知重新对齐减轻LVLMs中的幻觉

Pengxu Chen, Yao Zhu, Guangming Zhu, Jun Sheng, Jincai Huang, Xiangyang Ji, Liang Zhang

机构 * Xidian University（西安电子科技大学） ； Tsinghua University（清华大学） ； Shanghai Road Transport Development Center（上海市道路运输发展中心） ； Hunan Institute of Advanced Technology（湖南先进技术研究院） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

AI总结 研究针对LVLMs易产生幻觉问题，提出无需训练的SDPR框架，通过显著性驱动注意力重新分配、缓存对齐及先验约束对比解码，整体对齐视觉意识，在多基准测试中优于现有方法，无需额外训练且开销小。

Comments Accepted by ACM Multimedia 2026

Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges

分割一切模型中的提示工程：方法、应用与新兴挑战

Yidong Jiang, Jiangtong Li, Daiwei Cheng

机构 * School of Computer Science and Technology, Tongji University（计算机科学与技术学院，同济大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 本综述系统梳理了SAM的提示工程技术，将其分为三类并分析方法演变，探讨其跨领域应用，指出三大挑战并提出有前景的研究方向，为相关领域提供参考。

Intern-S2-Preview: Scientific Agentic Foundation Model

Intern-S2-Preview：科学智能体基础模型

Lei Bai, Jiaqi Cao, Chiyu Chen, Guanzhou Chen, Kai Chen, Guangran Cheng, Erfei Cui, Xuanlang Dai, Shengyuan Ding, Shangheng Du, Yanhui Duan, Yue Fan, Youqing Fang, Quan Gan, Yuanyuan Gao, Jiaye Ge, Lixin Gu, Yuzhe Gu, Qipeng Guo, Junjun He, Xin Hong, Ming Hu, Zhouqi Hua, Haian Huang, Junhao Huang, Zixian Huang, Minxi Jin, Lingkai Kong, Alexander Lam, Zehao Li, Zonglin Li, Tianhao Liang, Dahua Lin, Junyao Lin, Tianyang Lin, Zhouhan Lin, Jiangning Liu, Jin Liu, Kuikun Liu, Wenran Liu, Yifei Liu, Yuhong Liu, Yuhong Liu, Zhoumianze Liu, Ziyan Liu, Ziyu Liu, Haijun Lv, Han Lv, Chengqi Lyu, Le Ma, Ningsheng Ma, Zerun Ma, Haoyang Peng, Runyu Peng, Jifei Shan, Zixin Shang, Kou Shi, Xiang Shi, Qisheng Su, Xuerui Su, Hao Sun, Xiao Sun, Yanan Sun, Yu Sun, Huanze Tang, Yinghao Tang, Wenhui Tian, Zhongbo Tian, Bingli Wang, Haomin Wang, Jiarui Wang, Jingzhi Wang, Rui Wang, Xiquan Wang, Yi Wang, Zhecan Wang, Ziyi Wang, Zun Wang, Rubin Wei, Lianyi Wu, Wen Wu, Yue Wu, Yuhan Wu, Zhenyu Wu, Zijian Wu, Shuhao Xing, Jun Xu, Xingle Xu, Xuenan Xu, Xiangchao Yan, Ziang Yan, Bowen Yang, Danni Yang, Lin Yang, Zhiqi Yang, Qian Yao, Haochen Ye, Peng Ye, Jinhui Yin, Jiashuo Yu, Dingbo Yuan, Fei Yuan, Yuhang Zang, Bo Zhang, Chao Zhang, Chen Zhang, Hongjie Zhang, Junming Zhang, Wenlong Zhang, Wenwei Zhang, Yiming Zhang, Zhuo Zhang, Ziyang Zhang, Haiteng Zhao, Penghao Zhao, Yibo Zhao, Zhonghan Zhao, Zhihang Zhong, Bowen Zhou, Peiheng Zhou, Xin Zhou, Xinyu Zhou, Yunhua Zhou, Dongsheng Zhu, Yicheng Zou

机构 * Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

AI总结 本研究提出Intern-S2-Preview系列科学智能体基础模型，通过多阶段训练与架构优化，在多类基准上取得领先结果，相关模块可提升科学任务表现且无需修改主干模型。

Comments 35 pages, 12 figures

SpikeWorld: Fast-State Adaptation for Frozen Spiking World Models

SpikeWorld：用于冻结脉冲世界模型的快速状态适应

Ziqiao Yu

机构 * DiDi International Business Group（滴滴国际业务集团） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 SpikeWorld是145万参数的稀疏脉冲模型，冻结训练参数后通过外部路径实现快速状态适应，提升多模态预测等性能，在Meta-World任务中提高冻结策略奖励与成功率。

Comments 14 pages, 2 figures, 4 tables. Code: https://github.com/Oooorca/SpikeWorld

Gaze Behavior in Visual World Experiments Can be Modeled With Off-the-shelf Language-Vision Encoders

视觉世界实验中的注视行为可采用现成的语言-视觉编码器建模

Rahul Murali Shankar, Titus von der Malsburg, Sebastian Padó

机构 * University of Stuttgart（斯图加特大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

AI总结 该研究提出结合CLIP双编码器与双模态归因方法的新方法，无需微调或生成式架构，即可复现经典视觉世界研究的人类注视行为预测结果。

Intersectional Fairness in Vision-Language Models for Medical Image Disease Classification

视觉-语言模型在医学影像疾病分类中的交叉公平性

Yupeng Zhang, Adam G. Dunn, Usman Naseem, Jinman Kim

机构 * Biomedical Data Analysis and Visualisation (BDAV) Lab, School of Computer Science（生物医学数据分析与可视化实验室，计算机科学学院） ； Sydney School of Public Health（悉尼公共卫生学院） ； School of Computing（计算学院） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出CMAC-MMD框架，通过标准化交叉亚组的诊断确定性，减少医学影像疾病分类中的交叉偏见，提升诊断准确性和公平性。

QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision-Language Models

用于红外视觉语言模型定向语义攻击的QR结构化热触发装置

Xiang Chen, Yingying Zhao, Chao Li, Jiaju Han, Ben Zhang, Ang Li, Jiahuan Long, Yiwei Wei, Jiujiang Guo, Chengyin Hu

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本研究提出QR-STT框架，作为隐蔽黑盒攻击手段，可定向操控IR-VLMs的语义对齐，其扰动还能跨任务迁移，凸显了对该类攻击进行鲁棒性评估的必要性。