Although instruction-tuned large language models (LLMs) have exhibited remarkable capabilities across various NLP tasks, their effectiveness on other data modalities beyond text has not been fully studied. In this work, we propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual, audio, and textual information. Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations. Our novel alignment module seamlessly bridges multi-modal features to textual features, simplifying the adaptation process from the modality modules to the cognitive module. In addition, we construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances. We have made our data, code and model publicly available, which we hope can pave the way for future research in multi-modal LLMs and expand the capabilities of LLMs to handle diverse data modalities and address complex real-world scenarios.
Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
Tianjin Key Laboratory of Cognitive Computing(天津认知计算实验室);Tianjin University(天津大学);Huiyan Technology Company, Ltd.(慧颜科技有限公司);Chinese Academy of Sciences(中国科学院);Tencent(腾讯)
Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation
Columbia University(哥伦比亚大学);University of Washington(华盛顿大学)
OmniFocus: Query-Guided Modality-Balanced Token Compression for Omni-Modal Large Language Models
School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences(中国科学院大学先进交叉科学学院);Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences(中国科学院软件研究所中文信息处理实验室);University of Limerick(利默里克大学);CUHK, Shenzhen(香港中文大学(深圳))
Probing Cross-modal Information Hubs in Audio-Visual LLMs
Department of Electrical Engineering, Korea Advanced Institute of Science;The Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University, Seoul, Republic of Korea
LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos
Southern University of Science and Technology(南方科技大学);University of Birmingham(伯明翰大学);University of Electronic Science and Technology of China(电子科技大学);The University of Hong Kong(香港大学);University of Manchester(曼彻斯特大学)
Multi-Modulation Network for Audio-Visual Event Localization
University of Science and Technology of China(中国科学技术大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Rochester(罗切斯特大学)
Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval
Zhejiang University(浙江大学);Peking University(北京大学);Zhejiang Gongshang University(浙江工商大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);University of Science and Technology of China(中国科学技术大学)
Learning to Unify Audio, Visual and Text for Audio-Enhanced Multilingual Visual Answer Localization
Systems Engineering Institute(系统工程研究所);Xi’an Jiaotong University(西安交通大学);Shenzhen Institute of Advanced Technology(深圳先进技术研究院);Chinese Academy of Sciences(中国科学院)
COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings
School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院);Centre for Vision, Speech, and Signal Processing (CVSSP), University of Surrey(Surrey 大学视觉、语音和信号处理中心)
Multi-Modality in Music: Predicting Emotion in Music from High-Level Audio Features and Lyrics
University of Copenhagen(哥本哈根大学)
本文旨在测试多模态方法的音乐情感识别(MER)是否比单模态的高层次的歌曲特征和歌词。我们使用从Spotify API检索的11个歌曲特征,组合歌词特征,包括情感,TF-IDF和Anew,以预测Deezer情绪检测数据集(DMDD)(Delbouys et al.,2018),具有4种不同的回归模型。我们发现,在11个高级歌曲特征中,主要有5个对性能有贡献,多模态特征在预测效价时比单独的音频做得更好。我们使我们的代码可供使用。
Pay More Attention To Audio: Mitigating Imbalance of Cross-Modal Attention in Large Audio Language Models
1Laboratory of Cognitive Computing;Application, College of Intelligence;Computing, Tianjin University, Tianjin, China 2School of Computer Science, Shanghai Jiao Tong University, Shanghai, China 3School of Electrical \& Electronic Engineering, Nanyang Technological University, Singapore 4Huiyan Technology (Tianjin) Co., Ltd, Tianjin, China