AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
Fudan University(复旦大学);Multimodal Art Projection Research Community(多模态艺术投影研究社区);Shanghai AI Laboratory(上海人工智能实验室)
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, … 展开作者
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yu-Gang Jiang, Xipeng Qiu
备注:
28 pages, 16 figures, under review, work in progress
We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages. We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model. Demos are shown in https://junzhan2000.github.io/AnyGPT.github.io/
Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding
Tsinghua University(清华大学);Sun Yat-sen University(中山大学);Independent Researcher(独立研究员);The Hong Kong Polytechnic University(香港理工大学);Guangdong Laboratory of AI and Digital Economy (SZ)(广东省人工智能与数字经济实验室(深圳));University of Illinois Chicago(伊利诺伊大学芝加哥分校)
Discrete Prompt Tuning via Recursive Utilization of Black-box Multimodal Large Language Model for Personalized Visual Emotion Recognition
Graduate School of Information Science(信息科学研究生学校);Technology, Hokkaido University, Sapporo 060-0814, Japan(技术,北海道大学,札幌060-0814,日本);Office of Institutional Research, Hokkaido University, Sapporo 060-0808, Japan(机构研究办公室,北海道大学,札幌060-0808,日本);Faculty of Information Science(信息科学学院)
NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);University of Chinese Academy of Sciences(中国科学院大学);NExT++ Research Center(NExT++研究中心);National University of Singapore(新加坡国立大学)
VISTANet: VIsual Spoken Textual Additive Net for Interpretable Multimodal Emotion Recognition
Center for Machine Vision and Signal Analysis, University of Oulu(机器视觉与信号分析中心,奥卢大学);Indian Institute of Technology Roorkee(印度理工学院罗尔基分校);State Key Laboratory of Blockchain and Data Security, Zhejiang University(区块链与数据安全国家重点实验室,浙江大学)
HybridCodec: Modeling Discrete and Continuous Representations for Efficient Speech Language Models
Mila, Quebec AI Institute(Mila-魁北克人工智能研究所);Concordia University(康考迪亚大学);Sapienza University of Rome(罗马大学);Inria, Université Grenoble Alpes CNRS, LJK(法国国家信息与自动化研究所,格勒诺布尔阿尔卑斯大学国家科学研究中心,让·库尔曼实验室)
VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence
School of Automation Science and Electrical Engineering, Beihang University(北京航空航天大学自动化科学与电气工程学院);Hangzhou International Innovation Institute, Beihang University(北京航空航天大学杭州国际创新研究院);State Key Laboratory of Intelligent Manufacturing System Technology(智能制造系统技术国家重点实验室);School of Computer Science and Artificial Intelligence, Zhengzhou University(郑州大学计算机科学与人工智能学院)
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China(中国科学院人工智能安全重点实验室,计算技术研究所,中国科学院,北京,中国);University of Chinese Academy of Sciences, Beijing, China(中国科学院大学,北京,中国);Tsinghua University, Beijing, China(清华大学,北京,中国)
Quantized-Tinyllava: a new multimodal foundation model enables efficient split learning
Department of Statistics University of Michigan(统计学系密歇根大学);Department of Computational Medicine & Bioinformatics University of Michigan(计算医学与生物信息学系密歇根大学);Department of Biostatistics University of Michigan(生物统计学系密歇根大学);Department of Computer Science University of California, Los Angeles(计算机科学系加州大学洛杉矶分校)
GAViD: A Large-Scale Multimodal Dataset for Context-Aware Group Affect Recognition from Videos
Department of Computer Science and Engineering, Indian Institute of Technology Roorkee(计算机科学与工程系,印度理工学院Roorkee分校);Indian Institute of Technology Ropar(印度理工学院Ropar分校);Zhejiang University(浙江大学);University of Oulu(奥卢大学)
Task-Oriented Visual Feature Compression via Residual Vector Quantization for Device-Edge Multimodal Inference
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);China Telecom(中国电信);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI))