多模态大模型

Text or Image? What is More Important in Cross-Domain Generalization Capabilities of Hate Meme Detection Models?

Piush Aggarwal, Jawar Mehrabanian, Weigang Huang, Özge Alacam, Torsten Zesch

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at EACL'2024 Findings

MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts

Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, Jianfeng Gao

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 116 pages, 120 figures. Accepted to ICLR 2024

DeepArt: A Benchmark to Advance Fidelity Research in AI-Generated Content

Wentao Wang, Xuanyao Huang, Tianyang Wang, Swalpa Kumar Roy

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments This is the second version of this work, and new contributors join and the modification content is greatly increased

VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use

Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, Ludwig Schmidt

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to NeurIPS 2023, Datasets and Benchmarks. Website: https://visit-bench.github.io/

IRFL: Image Recognition of Figurative Language

Ron Yosef, Yonatan Bitton, Dafna Shahaf

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

GROOViST: A Metric for Grounding Objects in Visual Storytelling

Aditya K Surikuchi, Sandro Pezzelle, Raquel Fernández

专题命中 多模态评测 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments In EMNLP 2023 main conference proceedings (to appear)

Image Manipulation via Multi-Hop Instructions -- A New Dataset and Weakly-Supervised Neuro-Symbolic Approach

Harman Singh, Poorva Garg, Mohit Gupta, Kevin Shah, Ashish Goswami, Satyam Modi, Arnab Kumar Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2023 (long paper, main conference)

ECHo: A Visio-Linguistic Dataset for Event Causality Inference via Human-Centric Reasoning

Yuxi Xie, Guanzhen Li, Min-Yen Kan

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Findings of EMNLP 2023. 10 pages, 6 figures, 5 tables (22 pages, 8 figures, 15 tables including references and appendices)

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, Ming-Wei Chang

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2023 (main conference); Our dataset and evaluation is available at https://open-vision-language.github.io/infoseek/

Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving

Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, Jamie Shotton

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Yue Jiang, Eldon Schoop, Amanda Swearngin, Jeffrey Nichols

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Co-Located Human-Human Interaction Analysis using Nonverbal Cues: A Survey

Cigdem Beyan, Alessandro Vinciarelli, Alessio Del Bue

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive version was published in ACM Computing Surveys, https://doi.org/10.1145/3626516

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz

专题命中 多模态评测 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ICCV 2023. Website: whoops-benchmark.github.io

SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs

Shengzhi Li, Nima Tajbakhsh

专题命中 多模态评测 ：MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

LOIS: Looking Out of Instance Semantics for Visual Question Answering

Siyu Zhang, Yeming Chen, Yaoru Sun, Fang Wang, Haibo Shi, Haoran Wang

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

I2I: Initializing Adapters with Improvised Knowledge

Tejas Srinivasan, Furong Jia, Mohammad Rostami, Jesse Thomason

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at 2nd Conference on Lifelong Learning Agents (CoLLAs), 2023

It is not Sexually Suggestive, It is Educative. Separating Sex Education from Suggestive Content on TikTok Videos

Enfa George, Mihai Surdeanu

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ACL Findings 2023. 10 pages, 3 figures, 5 tables . Please refer to https://github.com/enfageorge/SexTok for dataset and related details

Path to Medical AGI: Unify Domain-specific Medical LLMs with the Lowest Cost

Juexiao Zhou, Xiuying Chen, Xin Gao

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Large language models improve Alzheimer's disease diagnosis using multi-modality data

Yingjie Feng, Jun Wang, Xianfeng Gu, Xiaoyin Xu, Min Zhang

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

TotalDefMeme: A Multi-Attribute Meme dataset on Total Defence in Singapore

Nirmalendu Prakash, Ming Shan Hee, Roy Ka-Wei Lee

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 6 pages. Accepted at ACM MMSys 2023

PuMer: Pruning and Merging Tokens for Efficient Vision Language Models

Qingqing Cao, Bhargavi Paranjape, Hannaneh Hajishirzi

专题命中 多模态评测 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ACL 2023 Main Conference

SQA3D: Situated Question Answering in 3D Scenes

Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, Siyuan Huang

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2023. First two authors contributed equally. Project website: https://sqa3d.github.io

Reassessing Evaluation Practices in Visual Question Answering: A Case Study on Out-of-Distribution Generalization

Aishwarya Agrawal, Ivana Kajić, Emanuele Bugliarello, Elnaz Davoodi, Anita Gergely, Phil Blunsom, Aida Nematzadeh

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Findings of EACL 2023. Aishwarya, Ivana, Emanuele and Aida had equal first author contributions. Elnaz and Anita had equal contributions. Aida and Aishwarya had equal senior contributions

ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation

Wanrong Zhu, Xin Eric Wang, An Yan, Miguel Eckstein, William Yang Wang

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EACL 2023

Abstract Visual Reasoning with Tangram Shapes

Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D. Hawkins, Yoav Artzi

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2022 long paper

TPA-Net: Generate A Dataset for Text to Physics-based Animation

Yuxing Qiu, Feng Gao, Minchen Li, Govind Thattai, Yin Yang, Chenfanfu Jiang

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks

Tejas Srinivasan, Ting-Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, Jesse Thomason

专题命中 多模态评测 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to NeurIPS 2022 Datasets and Benchmarks track

Visuo-Tactile Transformers for Manipulation

Yizhou Chen, Andrea Sipos, Mark Van der Merwe, Nima Fazeli

专题命中 多模态评测 ：multimodal(abstract)；cross-modal(abstract)

Comments Accepted to CoRL 2022

Uncertainty-based Visual Question Answering: Estimating Semantic Inconsistency between Image and Knowledge Base

Jinyeong Chae, Jihie Kim

专题命中 多模态评测 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by the 2022 International Joint Conference on Neural Networks (IJCNN 2022)

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, Song-Chun Zhu

专题命中 多模态评测 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Corrected typos. Accepted to NeurIPS 2021, 27 pages, 18 figures. Data and code are available at https://iconqa.github.io