多模态大模型

Transcrib3D: 3D Referring Expression Resolution through Large Language Models

Jiading Fang, Xiangshan Tan, Shengjie Lin, Igor Vasiljevic, Vitor Guizilini, Hongyuan Mei, Rares Ambrus, Gregory Shakhnarovich, Matthew R Walter

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments CORLW 2023

Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering

Cuong Nhat Ha, Shima Asaadi, Sanjeev Kumar Karn, Oladimeji Farri, Tobias Heimann, Thomas Runkler

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Clinical NLP @ NAACL 2024

Supervised Fine-tuning in turn Improves Visual Foundation Models

Xiaohu Jiang, Yixiao Ge, Yuying Ge, Dachuan Shi, Chun Yuan, Ying Shan

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments 23 pages, 3 figures, Project page: https://github.com/TencentARC/ViSFT/tree/main

Segment Any 3D Object with Language

Seungjun Lee, Yuyang Zhao, Gim Hee Lee

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Project Page: https://cvrp-sole.github.io

BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning

Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by AAAI 2023, Oral

Visual Grounding Helps Learn Word Meanings in Low-Data Regimes

Chengxu Zhuang, Evelina Fedorenko, Jacob Andreas

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by NAACL 2024

LCV2: An Efficient Pretraining-Free Framework for Grounded Visual Question Answering

Yuhan Chen, Lumei Su, Lihua Chen, Zhiwei Lin

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 21 pages,9 figures

SurgicalPart-SAM: Part-to-Whole Collaborative Prompting for Surgical Instrument Segmentation

Wenxi Yue, Jing Zhang, Kun Hu, Qiuxia Wu, Zongyuan Ge, Yong Xia, Jiebo Luo, Zhiyong Wang

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Technical Report. The source code will be released at https://github.com/wenxi-yue/SurgicalPart-SAM

HalluciDet: Hallucinating RGB Modality for Person Detection Through Privileged Information

Heitor Rapela Medeiros, Fidel A. Guerrero Pena, Masih Aminbeidokhti, Thomas Dubail, Eric Granger, Marco Pedersoli

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024

Journal ref Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2024

From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series Forecasting

Zhen Zeng, Rachneet Kaur, Suchetha Siddagangappa, Tucker Balch, Manuela Veloso

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Published at ACM ICAIF 2023

Adversarial Training with OCR Modality Perturbation for Scene-Text Visual Question Answering

Zhixuan Shen, Haonan Luo, Sijia Li, Tianrui Li

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 6 pages, 3 figures, accepted by 2024 IEEE International Conference on Multimedia and Expo

GPT-4 Technical Report

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, Barret Zoph

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 100 pages; updated authors list; fixed author names and added citation

On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models

Xinpeng Wang, Shitong Duan, Xiaoyuan Yi, Jing Yao, Shanlin Zhou, Zhihua Wei, Peng Zhang, Dongkuan Xu, Maosong Sun, Xing Xie

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 23 pages, 7 figures

adaptMLLM: Fine-Tuning Multilingual Language Models on Low-Resource Languages with Integrated LLM Playgrounds

Séamus Lankford, Haithem Afli, Andy Way

专题命中 多模态训练与对齐 ：MLLM(abstract)；分类 cs.CL、cs.AI

Journal ref Information 2023, 14(12), 638

Complementary Random Masking for RGB-Thermal Semantic Segmentation

Ukcheol Shin, Kyunghyun Lee, In So Kweon, Jean Oh

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments ICRA 2024, Our source code is available at https://github.com/UkcheolShin/CRM_RGBTSeg

Few-Shot Relation Extraction with Hybrid Visual Evidence

Jiaying Gong, Hoda Eldardiry

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments 16 pages, 5 figures

Journal ref LREC-COLING 2024

LAPDoc: Layout-Aware Prompting for Documents

Marcel Lamott, Yves-Noel Weweler, Adrian Ulges, Faisal Shafait, Dirk Krechel, Darko Obradovic

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Under review at ICDAR2024

One-to-many Reconstruction of 3D Geometry of cultural Artifacts using a synthetically trained Generative Model

Thomas Pöllabauer, Julius Kühn, Jiayi Li, Arjan Kuijper

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Journal ref 21st Eurographics Workshop on Graphics and Cultural Heritage (GCH 2023)

Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications

Francisco Mena, Diego Arenas, Marlon Nuske, Andreas Dengel

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments appendix with additional tables. Preprint submitted to journal

Journal ref IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pp. 1-21 (2024)

2D-3D Interlaced Transformer for Point Cloud Segmentation with Scene-Level Supervision

Cheng-Kun Yang, Min-Hung Chen, Yung-Yu Chuang, Yen-Yu Lin

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments ICCV 2023 (main + supp). Website: https://jimmy15923.github.io/mit_web/

Object Attribute Matters in Visual Question Answering

Peize Li, Qingyi Si, Peng Fu, Zheng Lin, Yan Wang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments AAAI 2024

MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment

Yequan Bie, Luyang Luo, Hao Chen

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

GATS: Gather-Attend-Scatter

Konrad Zolna, Serkan Cabi, Yutian Chen, Eric Lau, Claudio Fantacci, Jurgis Pasukonis, Jost Tobias Springenberg, Sergio Gomez Colmenarejo

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

BusReF: Infrared-Visible images registration and fusion focus on reconstructible area using one set of features

Zeyang Zhang, Hui Li, Tianyang Xu, Xiaojun Wu, Josef Kittler

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Modality-Collaborative Transformer with Hybrid Feature Reconstruction for Robust Emotion Recognition

Chengxin Chen, Pengyuan Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 23 pages, 9 figures, under review

CRN: Camera Radar Net for Accurate, Robust, Efficient 3D Perception

Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, Dongsuk Kum

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments IEEE/CVF International Conference on Computer Vision (ICCV'23). Code is available at https://github.com/youngskkim/CRN

Visual Instruction Tuning with Polite Flamingo

Delong Chen, Jianfeng Liu, Wenliang Dai, Baoyuan Wang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments In AAAI-24

AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making

Shusen Liu, Haichao Miao, Zhimin Li, Matthew Olson, Valerio Pascucci, Peer-Timo Bremer

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers

Giulia Rizzoli, Donald Shenaj, Pietro Zanuttigh

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.MM

Comments WACV 2024, 2nd Workshop on Pretraining (WACVW)

Fine-tuning pre-trained extractive QA models for clinical document parsing

Ashwyn Sharma, David I. Feldman, Aneesh Jain

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI