视觉大模型 / VLM

Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models

Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, Ranjay Krishna

专题命中 视觉推理 ：visual reasoning(abstract)；分类 cs.CV

Comments Accepted to NeurIPS 2024. Project and codes url: https://visualsketchpad.github.io/

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, Alexander Hauptmann

专题命中 视觉推理 ：multimodal large language model(abstract)；分类 cs.AI

Comments Accepted at NeurIPS 2024. 49 pages, 13 figures, Project: https://github.com/ZebangCheng/Emotion-LLaMA, Demo: https://huggingface.co/spaces/ZebangCheng/Emotion-LLaMA

PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, Tsung-Yen Yang

专题命中 视觉推理 ：grounding(abstract)；分类 cs.AI

Comments Alphabetical author order

MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding

Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, Kai Chen

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted in NeurIPS 2024 Datasets and Benchmarks Track

Code Repair with LLMs gives an Exploration-Exploitation Tradeoff

Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, Kevin Ellis

专题命中 视觉推理 ：visual reasoning(abstract)；分类 cs.AI

ImageInWords: Unlocking Hyper-Detailed Image Descriptions

Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, Radu Soricut

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments Webpage (https://google.github.io/imageinwords), GitHub (https://github.com/google/imageinwords), HuggingFace (https://huggingface.co/datasets/google/imageinwords)

Beyond Color and Lines: Zero-Shot Style-Specific Image Variations with Coordinated Semantics

Jinghao Hu, Yuhe Zhang, GuoHua Geng, Liuyuxin Yang, JiaRui Yan, Jingtao Cheng, YaDong Zhang, Kang Li

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments 13 pages,6 figures

Scene Graph Generation with Role-Playing Large Language Models

Guikun Chen, Jin Li, Wenguan Wang

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments NeurIPS 2024. Code: https://github.com/guikunchen/SDSGG

Differentiable Robot Rendering

Ruoshi Liu, Alper Canberk, Shuran Song, Carl Vondrick

专题命中 视觉推理 ：vision language model(abstract)；分类 cs.CV

Comments Project Page: https://drrobot.cs.columbia.edu/

VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Tasks

Shailaja Keyur Sampat, Mutsumi Nakamura, Shankar Kailas, Kartik Aggarwal, Mandy Zhou, Yezhou Yang, Chitta Baral

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments 18 pages, 7 figures

Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions

Zhenyu Jiang, Yuqi Xie, Jinhan Li, Ye Yuan, Yifeng Zhu, Yuke Zhu

专题命中 视觉推理 ：vision language model(abstract)；分类 cs.AI

Comments Accepted for oral presentation at 8th Annual Conference on Robot Learning. Project website: https://ut-austin-rpl.github.io/Harmon/

MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark

Bin Shan, Xiang Fei, Wei Shi, An-Lan Wang, Guozhi Tang, Lei Liao, Jingqun Tang, Xiang Bai, Can Huang

专题命中 视觉推理 ：visual reasoning(abstract)；分类 cs.CV

Comments 12 pages, 5 figures, project page: https://github.com/xfey/MCTBench?tab=readme-ov-file

Structured Spatial Reasoning with Open Vocabulary Object Detectors

Negar Nejatishahidin, Madhukar Reddy Vongala, Jana Kosecka

专题命中 视觉推理 ：grounding(abstract)；分类 cs.CV

The Cognitive Capabilities of Generative AI: A Comparative Analysis with Human Benchmarks

Isaac R. Galatzer-Levy, David Munday, Jed McGiffin, Xin Liu, Danny Karmon, Ilia Labzovsky, Rivka Moroshko, Amir Zait, Daniel McDuff

专题命中 视觉推理 ：vision language model(abstract)；分类 cs.AI

ReFeR: Improving Evaluation and Reasoning through Hierarchy of Models

Yaswanth Narsupalli, Abhranil Chandra, Sreevatsa Muppirala, Manish Gupta, Pawan Goyal

专题命中 视觉推理 ：vision language model(abstract)；分类 cs.AI

Comments Paper Under Review

Temporal Reasoning Transfer from Text to Video

Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, Qi Liu

专题命中 视觉推理 ：InternVL(abstract)；分类 cs.CV

Comments Project page: https://video-t3.github.io

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning

Geewook Kim, Minjoon Seo

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments EMNLP 2024 Main

Reasoning with Natural Language Explanations

Marco Valentino, André Freitas

专题命中 视觉推理 ：grounding(abstract)；分类 cs.AI

Comments Tutorial to be presented at EMNLP 2024. Website: https://sites.google.com/view/reasoning-with-explanations

VDebugger: Harnessing Execution Feedback for Debugging Visual Programs

Xueqing Wu, Zongyu Lin, Songyan Zhao, Te-Lin Wu, Pan Lu, Nanyun Peng, Kai-Wei Chang

专题命中 视觉推理 ：visual reasoning(abstract)；分类 cs.CV

Comments EMNLP 2024 Findings

ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation

Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, Yan Chang

专题命中 视觉推理 ：VLM(abstract)；分类 cs.AI

Impact of ML Optimization Tactics on Greener Pre-Trained ML Models

Alexandra González Álvarez, Joel Castaño, Xavier Franch, Silverio Martínez-Fernández

专题命中 视觉推理 ：visual reasoning(abstract)；分类 cs.LG

MAGDA: Multi-agent guideline-driven diagnostic assistance

David Bani-Harouni, Nassir Navab, Matthias Keicher

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.AI

PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis

Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong-Li Lee, Wynne Hsu

专题命中 视觉推理 ：multimodal large language model(abstract)；分类 cs.AI

Comments Accepted by ACM MM 2024 (Oral)

X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning

Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, Juan Carlos Niebles

专题命中 视觉推理 ：visual reasoning(abstract)；分类 cs.CV

MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models

Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, Zhi Tang

专题命中 视觉推理 ：multimodal large language model(abstract)；分类 cs.AI

UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models

Liu Qi, He Yongyi, Lian Defu, Zheng Zhi, Xu Tong, Liu Che, Chen Enhong

专题命中 视觉推理 ：multimodal large language model(abstract)；分类 cs.AI

Comments CIKM 2024. The first two authors contributed equally to this work

MMBench: Is Your Multi-modal Model an All-around Player?

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, Dahua Lin

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted in ECCV2024 as Oral Presentation

Multi-Modal Dialogue State Tracking for Playing GuessWhich Game

Wei Pang, Ruixue Duan, Jinfu Yang, Ning Li

专题命中 视觉推理 ：visual reasoning(abstract)；分类 cs.AI

Comments Published at CICAI 2023 (CAAI-A), codes at https://github.com/xubuvd/GuessWhich

Do Vision-Language Foundational models show Robust Visual Perception?

Shivam Chandhok, Pranav Tandon

专题命中 视觉推理 ：vision-language model(abstract)；分类 cs.CV

Comments UBC Report

Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationales

Minghe Gao, Shuang Chen, Liang Pang, Yuan Yao, Jisheng Dang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Yueting Zhuang, Tat-Seng Chua

专题命中 视觉推理 ：multimodal large language model(abstract)；分类 cs.CV