多模态大模型

Initial Study On Improving Segmentation By Combining Preoperative CT And Intraoperative CBCT Using Synthetic Data

Maximilian E. Tschuchnig, Philipp Steininger, Michael Gadermayr

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at BVM 2025. arXiv admin note: text overlap with arXiv:2406.11650

Fine-Grained Alignment in Vision-and-Language Navigation through Bayesian Optimization

Yuhang Song, Mario Gianni, Chenguang Yang, Kunyang Lin, Te-Chuan Chiu, Anh Nguyen, Chun-Yi Lee

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge

Yaqi Zhao, Yuanyang Yin, Lin Li, Mingan Lin, Victor Shea-Jay Huang, Siwei Chen, Weipeng Chen, Baoqun Yin, Zenan Zhou, Wentao Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

Zeren Chen, Ziqin Wang, Zhen Wang, Huayang Liu, Zhenfei Yin, Si Liu, Lu Sheng, Wanli Ouyang, Yu Qiao, Jing Shao

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments 22 pages, 12 figures. Accepted in ICLR 2024

Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases

Zian Su, Xiangzhe Xu, Ziyang Huang, Kaiyuan Zhang, Xiangyu Zhang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

SparrowVQE: Visual Question Explanation for Course Content Understanding

Jialu Li, Manish Kumar Thota, Ruslan Gokhman, Radek Holik, Youshan Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

Anand Syamkumar, Nora Tseng, Kaycie Barron, Shanglin Yang, Shamya Karumbaiah, Rheeya Uppal, Junjie Hu

专题命中 多模态训练与对齐 ：MLLM(abstract)；分类 cs.CL、cs.AI

Towards Calibrated Robust Fine-Tuning of Vision-Language Models

Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, Kyungwoo Song

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 (a short version was presented at the NeurIPS 2023 Workshop on Distribution Shifts); Major modification of (v7): Fixing the x-axis of Figure 3 and Pearson correlation, accordingly

Survey of Cultural Awareness in Language Models: Text and Beyond

Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, Isabelle Augenstein

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Multi Modal Information Fusion of Acoustic and Linguistic Data for Decoding Dairy Cow Vocalizations in Animal Welfare Assessment

Bubacarr Jobarteh, Madalina Mincu, Gavojdian Dinu, Suresh Neethirajan

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments 31 pages, 22 figures, 2 tables

NeuroPath: A Neural Pathway Transformer for Joining the Dots of Human Connectomes

Ziquan Wei, Tingting Dan, Jiaqi Ding, Guorong Wu

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by NeurIPS 2024

Proof of Thought : Neurosymbolic Program Synthesis allows Robust and Interpretable Reasoning

Debargha Ganguly, Srinivasan Iyengar, Vipin Chaudhary, Shivkumar Kalyanaraman

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 38th Conference on Neural Information Processing Systems (NeurIPS 2024) System 2 Reasoning At Scale Workshop

Mitigating Object Hallucination via Concentric Causal Attention

Yun Xing, Yiheng Li, Ivan Laptev, Shijian Lu

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments To appear at NeurIPS 2024. Code is available at https://github.com/xing0047/cca-llava

Representation Learning of Structured Data for Medical Foundation Models

Vijay Prakash Dwivedi, Viktor Schlegel, Andy T. Liu, Thanh-Tung Nguyen, Abhinav Ramesh Kashyap, Jeng Wei, Wei-Hsian Yin, Stefan Winkler, Robby T. Tan

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2024 Workshop on Unifying Representations in Neural Models (UniReps 2024)

Unveiling the Mystery of Visual Attributes of Concrete and Abstract Concepts: Variability, Nearest Neighbors, and Challenging Categories

Tarun Tater, Sabine Schulte im Walde, Diego Frassinelli

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.CL

Learning Shared RGB-D Fields: Unified Self-supervised Pre-training for Label-efficient LiDAR-Camera 3D Perception

Xiaohao Xu, Ye Li, Tianyi Zhang, Jinrong Yang, Matthew Johnson-Roberson, Xiaonan Huang

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 8 pages

Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models

Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Preprint

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

Hugo Malard, Michel Olvera, Stéphane Lathuiliere, Slim Essid

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、eess.AS

SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detection

Zishuo Wang, Wenhao Zhou, Jinglin Xu, Yuxin Peng

专题命中 多模态训练与对齐 ：image-text(abstract)；分类 cs.CV、cs.MM

Comments 9 pages, 7 figures

Gamified crowd-sourcing of high-quality data for visual fine-tuning

Shashank Yadav, Rohan Tomar, Garvit Jain, Chirag Ahooja, Shubham Chaudhary, Charles Elkan

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Intriguing Properties of Large Language and Vision Models

Young-Jun Lee, Byungsoo Ko, Han-Gyu Kim, Yechan Hwang, Ho-Jin Choi

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Code is available in https://github.com/passing2961/IP-LLVM

CoLLAP: Contrastive Long-form Language-Audio Pretraining with Musical Temporal Structure Augmentation

Junda Wu, Warren Li, Zachary Novack, Amit Namburi, Carol Chen, Julian McAuley

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.AI、eess.AS

Comments 4 pages

One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang, Mike Zheng Shou

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by NeurlPS 2024

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization

Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 Findings, Codes: https://github.com/HuangOwen/RoLoRA, Models: https://huggingface.co/collections/ScarletAce/rolora-66f5f228a90681c7c4512b28

Fixed-length Dense Descriptor for Efficient Fingerprint Matching

Zhiyu Pan, Yongjie Duan, Jianjiang Feng, Jie Zhou

专题命中 多模态训练与对齐 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by WIFS 2024

Revolutionizing Disease Diagnosis with simultaneous functional PET/MR and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks

Luoyu Wang, Yitian Tao, Qing Yang, Yan Liang, Siwei Liu, Hongcheng Shi, Dinggang Shen, Han Zhang

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 11 pages

Optical Flow Matters: an Empirical Comparative Study on Fusing Monocular Extracted Modalities for Better Steering

Fouad Makiyeh, Mark Bastourous, Anass Bairouk, Wei Xiao, Mirjana Maras, Tsun-Hsuan Wangb, Marc Blanchon, Ramin Hasani, Patrick Chareyre, Daniela Rus

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI

Multi-OCT-SelfNet: Integrating Self-Supervised Learning with Multi-Source Data Fusion for Enhanced Multi-Class Retinal Disease Classification

Fatema-E- Jannat, Sina Gholami, Jennifer I. Lim, Theodore Leng, Minhaj Nur Alam, Hamed Tabkhi

专题命中 多模态训练与对齐 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 25 pages, 9 tables, 10 figures

Quantile Regression for Distributional Reward Models in RLHF

Nicolai Dorka

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CL、cs.AI

NeuralOOD: Improving Out-of-Distribution Generalization Performance with Brain-machine Fusion Learning Framework

Shuangchen Zhao, Changde Du, Hui Li, Huiguang He

专题命中 多模态训练与对齐 ：multimodal(abstract)；分类 cs.CV、cs.AI