多模态大模型

Disentangling Fine-Tuning from Pre-Training in Visual Captioning with Hybrid Markov Logic

Monika Shah, Somdeb Sarkhel, Deepak Venugopal

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 2024 IEEE International Conference on Big Data (BigData), 10 pages

3D-MolT5: Leveraging Discrete Structural Information for Molecule-Text Modeling

Qizhi Pei, Rui Yan, Kaiyuan Gao, Jinhua Zhu, Lijun Wu

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ICLR 2025

Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation

Henglyu Liu, Andong Chen, Kehai Chen, Xuefeng Bai, Meizhi Zhong, Yuan Qiu, Min Zhang

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments 12 pages, 7 figures

LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RL

Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, Xu Yang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Small but Mighty: Enhancing Time Series Forecasting with Lightweight LLMs

Haoran Fan, Bin Li, Yixuan Weng, Shoujun Zhou

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CL、cs.AI

Comments 20 pages, 10 figures

Can We Talk Models Into Seeing the World Differently?

Paul Gavrikov, Jovita Lukasik, Steffen Jung, Robert Geirhos, M. Jehanzeb Mirza, Margret Keuper, Janis Keuper

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at ICLR 2025

High-Resolution Image Synthesis via Next-Token Prediction

Dengsheng Chen, Jie Hu, Tiezhu Yue, Xiaoming Wei, Enhua Wu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 31 pages

Atomas: Hierarchical Alignment on Molecule-Text for Unified Molecule Understanding and Generation

Yikun Zhang, Geyan Ye, Chaohao Yuan, Bo Han, Long-Kai Huang, Jianhua Yao, Wei Liu, Yu Rong

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CL、cs.AI

Frequency-Adaptive Low-Latency Object Detection Using Events and Frames

Haitian Zhang, Xiangyuan Wang, Chang Xu, Xinya Wang, Fang Xu, Huai Yu, Lei Yu, Wen Yang

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Talking to the brain: Using Large Language Models as Proxies to Model Brain Semantic Representation

Xin Liu, Ziyue Zhang, Jingxin Nie

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 20 pages, 6 figures

ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignment

Yangyifei Luo, Zhuo Chen, Lingbing Guo, Qian Li, Wenxuan Zeng, Zhixin Cai, Jianxin Li

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Ongoing work; 16 pages, 9 Tables, 8 Figures; Code: https://github.com/lyyf2002/ASGEA

TLDR: Token-Level Detective Reward Model for Large Vision Language Models

Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, Lawrence Chen

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Published as a conference paper at ICLR 2025

VisCon-100K: Leveraging Contextual Web Data for Fine-tuning Vision Language Models

Gokul Karthik Kumar, Iheb Chaabane, Kebin Wu

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted at PAKDD 2025

SentiFormer: Metadata Enhanced Transformer for Image Sentiment Analysis

Bin Feng, Shulan Ruan, Mingzheng Yang, Dongxuan Han, Huijie Liu, Kai Zhang, Qi Liu

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Intuitive physics understanding emerges from self-supervised pretraining on natural videos

Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 24 pages,14 figures, 5 tables

Segment Any Anomaly without Training via Hybrid Prompt Regularization

Yunkang Cao, Xiaohao Xu, Chen Sun, Yuqi Cheng, Zongwei Du, Liang Gao, Weiming Shen

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments The first two authors contribute equally. The code will be available on https://github.com/caoyunkang/Segment-Any-Anomaly

Journal ref IEEE Transactions on Cybernetics, 2025

AI-Driven HSI: Multimodality, Fusion, Challenges, and the Deep Learning Revolution

David S. Bhatti, Yougin Choi, Rahman S M Wahidur, Maleeka Bakhtawar, Sumin Kim, Surin Lee, Yongtae Lee, Heung-No Lee

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 39 Pages, 22 figures, 20 tables

Deep Learning-Based Facial Expression Recognition for the Elderly: A Systematic Review

F. Xavier Gaya-Morey, Jose M. Buades-Rubio, Philippe Palanque, Raquel Lacuesta, Cristina Manresa-Yee

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Registration-Enhanced Segmentation Method for Prostate Cancer in Ultrasound Images

Shengtian Sang, Hassan Jahanandish, Cynthia Xinran Li, Indrani Bhattachary, Jeong Hoon Lee, Lichun Zhang, Sulaiman Vesal, Pejman Ghanouni, Richard Fan, Geoffrey A. Sonn, Mirabela Rusu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Arbitrary Data as Images: Fusion of Patient Data Across Modalities and Irregular Intervals with Vision Transformers

Malte Tölle, Mohamad Scharaf, Samantha Fischer, Christoph Reich, Silav Zeid, Christoph Dieterich, Benjamin Meder, Norbert Frey, Philipp Wild, Sandy Engelhardt

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscape

Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, Malka N. Halgamuge

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 30 pages

GaussianToken: An Effective Image Tokenizer with 2D Gaussian Splatting

Jiajun Dong, Chengkun Wang, Wenzhao Zheng, Lei Chen, Jiwen Lu, Yansong Tang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Parameter-Efficient Fine-Tuning for Foundation Models

Dan Zhang, Tao Feng, Lilong Xue, Yuandong Wang, Yuxiao Dong, Jie Tang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 25 pages, 6 figures, 7 tables

Evaluating alignment between humans and neural network representations in image-based learning tasks

Can Demircan, Tankred Saanum, Leonardo Pettini, Marcel Binz, Blazej M Baczkowski, Christian F Doeller, Mona M Garvert, Eric Schulz

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Multi-task Visual Grounding with Coarse-to-Fine Consistency Constraints

Ming Dai, Jian Li, Jiedong Zhuang, Xian Zhang, Wankou Yang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments AAAI2025

LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, Yunpu Ma

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

From Generalist to Specialist: A Survey of Large Language Models for Chemistry

Yang Han, Ziping Wan, Lu Chen, Kai Yu, Xin Chen

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments COLING2025,We maintain an up-to-date Github repository at: https://github.com/OpenDFM/LLM4Chemistry

SwitchCIT: Switching for Continual Instruction Tuning

Xinbo Wu, Max Hartman, Vidhata Arjun Jayaraman, Lav R. Varshney

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Black-box Model Ensembling for Textual and Visual Question Answering via Information Fusion

Yuxi Xia, Kilm Zaporojets, Benjamin Roth

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 6 figures, 9 tables

Human-Centric NLP or AI-Centric Illusion?: A Critical Investigation

Piyapath T Spencer

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Preprint to be published in Proceedings of PACLIC38