多模态大模型

Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling

Eric Egli, Matteo Manica, Jannis Born

专题命中 多模态评测 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CL、cs.AI

Comments Under Review

FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehension

Junzhuo Liu, Xuzheng Yang, Weiwei Li, Peng Wang

机构 * University of Electronic Science and Technology of China（电子科技大学） 纠错

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 18 pages, EMNLP 2024 main

Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature

Lingdong Shen, Qigqi, Kun Ding, Gaofeng Meng, Shiming Xiang

专题命中 多模态评测 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning

Yuti Liu, Shice Liu, Junyuan Gao, Pengtao Jiang, Hao Zhang, Jinwei Chen, Bo Li

专题命中 多模态评测 ：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by AAAI 2025

Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, Neel Joshi

机构 * University of Wisconsin–Madison（威斯康星大学麦迪逊分校） ； Salesforce AI Research（Salesforce AI研究部） ； Microsoft Research（微软研究院） 纠错

专题命中 多模态评测 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to NeurIPS 2024

LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound

Xuechen Guo, Wenhao Chai, Shi-Yan Li, Gaoang Wang

专题命中 多模态评测 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

AutoGeo: Automating Geometric Image Dataset Creation for Enhanced Geometry Understanding

Zihan Huang, Tao Wu, Wang Lin, Shengyu Zhang, Jingyuan Chen, Fei Wu

专题命中 多模态评测 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Quantum Inverse Contextual Vision Transformers (Q-ICVT): A New Frontier in 3D Object Detection for AVs

Sanjay Bhargav Dharavath, Tanmoy Dam, Supriyo Chakraborty, Prithwiraj Roy, Aniruddha Maiti

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments The paper has been accepted as a short paper at CIKM '24

ARPA: A Novel Hybrid Model for Advancing Visual Word Disambiguation Using Large Language Models and Transformers

Aristi Papastavrou, Maria Lymperaiou, Giorgos Stamou

专题命中 多模态评测 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

STimage-1K4M: A histopathology image-gene expression dataset for spatial transcriptomics

Jiawen Chen, Muqing Zhou, Wenrong Wu, Jinwei Zhang, Yun Li, Didong Li

专题命中 多模态评测 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainability

Faseela Abdullakutty, Younes Akbari, Somaya Al-Maadeed, Ahmed Bouridane, Rifat Hamoudi

专题命中 多模态评测 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 31 pages including references

MMMNA-Net for Overall Survival Time Prediction of Brain Tumor Patients

Wen Tang, Haoyue Zhang, Pengxin Yu, Han Kang, Rongguo Zhang

专题命中 多模态评测 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted EMBC 2022

AKGNet: Attribute Knowledge-Guided Unsupervised Lung-Infected Area Segmentation

Qing En, Yuhong Guo

专题命中 多模态评测 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

VoxBlink: A Large Scale Speaker Verification Dataset on Camera

Yuke Lin, Xiaoyi Qin, Guoqing Zhao, Ming Cheng, Ning Jiang, Haiyang Wu, Ming Li

专题命中 多模态评测 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Comments Accepted By ICASSP2024

Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning

Hao Wang, Xiahua Chen, Rui Wang, Chenhui Chu

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 14 pages, 6 figures, Accepted by EMNLP2023

ViFiT: Reconstructing Vision Trajectories from IMU and Wi-Fi Fine Time Measurements

Bryan Bo Cao, Abrar Alali, Hansi Liu, Nicholas Meegan, Marco Gruteser, Kristin Dana, Ashwin Ashok, Shubham Jain

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 22 pages, 12 figures, 9 tables. MobiCom 2023 ISACom

Visual-Kinematics Graph Learning for Procedure-agnostic Instrument Tip Segmentation in Robotic Surgeries

Jiaqi Liu, Yonghao Long, Kai Chen, Cheuk Hei Leung, Zerui Wang, Qi Dou

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to IROS 2023

Breaking Down the Task: A Unit-Grained Hybrid Training Framework for Vision and Language Decision Making

Ruipu Luo, Jiwen Zhang, Zhongyu Wei

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Open-TransMind: A New Baseline and Benchmark for 1st Foundation Model Challenge of Intelligent Transportation

Yifeng Shi, Feng Lv, Xinliang Wang, Chunlong Xia, Shaojie Li, Shujie Yang, Teng Xi, Gang Zhang

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

SI-LSTM: Speaker Hybrid Long-short Term Memory and Cross Modal Attention for Emotion Recognition in Conversation

Xingwei Liang, You Zou, Ruifeng Xu

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments modification needed

MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization

Shivam Sharma, Ramaneswaran S, Udit Arora, Md. Shad Akhtar, Tanmoy Chakraborty

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

Comments 9 pages main + 1 ethics + 3 pages ref. + 4 pages app (Total: 17 pages)

Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild

Jose Vargas-Quiros, Laura Cabrera-Quiros, Catharine Oertel, Hayley Hung

专题命中 多模态评测 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows

Keisuke Shirai, Atsushi Hashimoto, Taichi Nishimura, Hirotaka Kameko, Shuhei Kurita, Yoshitaka Ushiku, Shinsuke Mori

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments COLING 2022

IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages

Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, Ivan Vulić

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments ICML 2022

UMSNet: An Universal Multi-sensor Network for Human Activity Recognition

Jialiang Wang, Haotian Wei, Yi Wang, Shu Yang, Chi Li

专题命中 多模态评测 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Learning to Answer Visual Questions from Web Videos

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted at the TPAMI Special Issue on the Best Papers of ICCV 2021. Journal extension of the conference paper arXiv:2012.00451. 16 pages, 13 figures

Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling

Tengpeng Li, Hanli Wang, Bin He, Chang Wen Chen

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Just Ask: Learning to Answer Questions from Millions of Narrated Videos

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted at ICCV 2021 (Oral); 20 pages; 14 figures

MultiModalQA: Complex Question Answering over Text, Tables and Images

Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, Jonathan Berant

专题命中 多模态评测 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2021

Predicting the Leading Political Ideology of YouTube Channels Using Acoustic, Textual, and Metadata Information

Yoan Dinkov, Ahmed Ali, Ivan Koychev, Preslav Nakov

专题命中 多模态评测 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CL、eess.AS

Comments media bias, political ideology, Youtube channels, propaganda, disinformation, fake news

Journal ref INTERSPEECH-2019