多模态大模型 - arXivDaily 专题

2310.02050 2023-10-04 cs.CL cs.CV 73%

Tuning Large language model for End-to-end Speech Translation

Hao Zhang, Nianwen Si, Yaqi Chen, Wenlin Zhang, Xukui Yang, Dan Qu, Xiaolin Jiao

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

2310.00455 2023-10-03 cs.MM cs.GR cs.LG cs.SD eess.AS 73%

Music- and Lyrics-driven Dance Synthesis

Wenjie Yin, Qingyuan Yao, Yi Yu, Hang Yin, Danica Kragic, Mårten Björkman

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

2308.15930 2023-09-19 cs.CL cs.LG cs.SD eess.AS 73%

LLaSM: Large Language and Speech Model

Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, Yemin Shi

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

2303.04585 2023-08-08 cs.SD cs.AI eess.AS 73%

Exploring Efficient-Tuned Learning Audio Representation Method from BriVL

Sen Fang, Yangjian Wu, Bowen Gao, Jingwen Cai, Teik Toe Teoh

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.AI、eess.AS

Comments 13 pages, 2023.3 Finished

2306.12559 2023-06-23 cs.CV cs.SD eess.AS 73%

Exploring the Role of Audio in Video Captioning

Yuhan Shen, Linjie Yang, Longyin Wen, Haichao Yu, Ehsan Elhamifar, Heng Wang

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

2305.14042 2023-05-24 cs.CL cs.SD eess.AS 73%

Improving speech translation by fusing speech and text

Wenbiao Yin, Zhicheng Liu, Chengqi Zhao, Tao Wang, Jian Tong, Rong Ye

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

2303.04439 2023-03-09 cs.CV cs.SD eess.AS 73%

A Light Weight Model for Active Speaker Detection

Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, Liangyin Chen

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted by CVPR 2023

2301.10295 2023-01-26 cs.CV cs.SD eess.AS 73%

Object Segmentation with Audio Context

Kaihui Zheng, Yuqing Ren, Zixin Shen, Tianxu Qin

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Research project for Introduction to Deep Learning (11785) at Carnegie Mellon University

2204.01235 2023-01-20 cs.CL cs.LG eess.AS 73%

An Analysis of Semantically-Aligned Speech-Text Embeddings

Muhammad Huzaifah, Ivan Kukanov

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments This is the accepted version of the paper published at IEEE Spoken Language Technology (SLT) Workshop 2022

2208.12208 2022-08-26 cs.SD cs.CL cs.LG eess.AS 73%

Contrastive Audio-Language Learning for Music

Ilaria Manco, Emmanouil Benetos, Elio Quinton, György Fazekas

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to ISMIR 2022

2205.14295 2022-06-03 cs.CV eess.AS 73%

Is Lip Region-of-Interest Sufficient for Lipreading?

Jing-Xuan Zhang, Gen-Shun Wan, Jia Pan

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments preprint

2203.12122 2022-04-22 cs.SD cs.MM eess.AS 73%

On Adversarial Robustness of Large-scale Audio Visual Learning

Juncheng B Li, Shuhui Qu, Xinjian Li, Po-Yao Huang, Florian Metze

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Journal ref 2022 International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2022)

2112.05410 2022-04-05 cs.CV cs.MM 73%

Multimedia Datasets for Anomaly Detection: A Review

Pratibha Kumari, Anterpreet Kaur Bedi, Mukesh Saini

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

Comments 17 pages, 11 figures, 8 tables

2203.13031 2022-03-31 cs.MM cs.CV 73%

Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3

Su Zhang, Ruyi An, Yi Ding, Cuntai Guan

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 5 pages, 1 figure. arXiv admin note: substantial text overlap with arXiv:2107.01175

2203.13472 2022-03-28 cs.CV cs.AI 73%

Facial Expression Recognition with Swin Transformer

Jun-Hwa Kim, Namho Kim, Chee Sun Won

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

2203.10426 2022-03-22 cs.CL cs.AI 73%

STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

Qingkai Fang, Rong Ye, Lei Li, Yang Feng, Mingxuan Wang

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments ACL 2022 main conference

2109.02763 2022-03-01 cs.SD cs.CV eess.AS 73%

Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds

Dengxin Dai, Arun Balajee Vasudevan, Jiri Matas, Luc Van Gool

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted by TPAMI. arXiv admin note: substantial text overlap with arXiv:2003.04210

2202.09979 2022-02-22 cs.CL cs.CV 73%

Audio Visual Scene-Aware Dialog Generation with Transformer-based Video Representations

Yoshihiro Yamazaki, Shota Orihashi, Ryo Masumura, Mihiro Uchida, Akihiko Takashima

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments Accepted at DSTC10 Workshop at AAAI 2022

2112.04214 2022-02-18 cs.SD cs.CL cs.IR cs.LG eess.AS 73%

Learning music audio representations via weak language supervision

Ilaria Manco, Emmanouil Benetos, Elio Quinton, Gyorgy Fazekas

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to ICASSP 2022

2112.00007 2021-12-02 cs.GR cs.CV cs.LG cs.SD eess.AS 73%

Sound-Guided Semantic Image Manipulation

Seung Hyun Lee, Wonseok Roh, Wonmin Byeon, Sang Ho Yoon, Chan Young Kim, Jinkyu Kim, Sangpil Kim

专题命中音频语音多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、eess.AS

2109.05522 2021-09-14 cs.CL cs.AI cs.LG 73%

TEASEL: A Transformer-Based Speech-Prefixed Language Model

Mehdi Arjmand, Mohammad Javad Dousti, Hadi Moradi

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

2101.08779 2021-08-03 cs.CV cs.GR cs.MM 73%

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

Ruilong Li, Shan Yang, David A. Ross, Angjoo Kanazawa

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Project page: https://google.github.io/aichoreographer/; Dataset page: https://google.github.io/aistplusplus_dataset/

2012.00290 2021-05-10 cs.SD cs.DB cs.IR cs.MM eess.AS 73%

MusicTM-Dataset for Joint Representation Learning among Sheet Music, Lyrics, and Musical Audio

Donghuo Zeng, Yi Yu, Keizo Oyama

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments 12 pages, 5 figures, 2 tables

Journal ref CSMT2020

2007.12130 2020-07-24 cs.CV cs.LG cs.SD eess.AS 73%

Sound2Sight: Generating Visual Dynamics from Sound and Context

Anoop Cherian, Moitreya Chatterjee, Narendra Ahuja

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted at ECCV 2020

2004.05830 2020-04-14 eess.AS cs.CV cs.LG cs.SD eess.IV 73%

From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech

Hyeong-Seok Choi, Changdae Park, Kyogu Lee

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments 18 pages, 12 figures, Published as a conference paper at International Conference on Learning Representations (ICLR) 2020. (camera-ready version)

1910.11760 2019-10-28 cs.CV cs.LG cs.SD eess.AS 73%

Self-supervised Moving Vehicle Tracking with Stereo Sound

Chuang Gan, Hang Zhao, Peihao Chen, David Cox, Antonio Torralba

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments To appear at ICCV 2019. Project page: http://sound-track.csail.mit.edu

1809.00496 2018-10-30 cs.CV 72%

LRS3-TED: a large-scale dataset for visual speech recognition

Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman

专题命中音频语音多模态：audio-visual(abstract,comments)；multi-modal(abstract)；分类 cs.CV

Comments The audio-visual dataset can be downloaded from http://www.robots.ox.ac.uk/~vgg/data/lip_reading/

2608.21387 2026-08-25 cs.RO cs.SY eess.SY 新提交 71%

Multimodal-Language-Model-Driven Interaction and Companionship for Service Robots in Elderly-Care Facilities

面向养老机构服务机器人的多模态语言模型驱动的交互与陪伴

Ching-Chieh Liu, Cong-Thanh Vu, Yen-Chen Liu

机构 * National Cheng Kung University (NCKU)（成功大学）

专题命中音频语音多模态：multimodal(title)

AI总结该研究针对养老机构服务机器人缺乏综合陪伴与安全能力的问题，提出整合主动视觉跟人、LLM语音交互及VLM安全监测的智能陪伴机器人系统，实验验证其跟随交互与跌倒检测的有效性。

Comments Accepted to the 2026 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)

2608.05543 2026-08-14 cs.IR 版本更新 71%

omni-macos: On-Device Omni-Modal Search on Apple Silicon

omni-macos：在Apple Silicon上运行的端侧全模态搜索系统

Han Xiao

专题命中音频语音多模态：omni-modal(title)

AI总结 omni-macos是一款在Apple Silicon端侧运行的全模态搜索系统，可在用户Mac设备上处理多模态数据搜索，数据不离开设备，适配不同硬件规格的Mac并优化内存使用。

Comments 18 pages, 5 figures, 10 tables

2110.07559 2026-07-28 astro-ph.EP physics.soc-ph physics.space-ph 71%

PI -- Multimodal Planetary Defense

PI -- 多模态行星防御

Philip Lubin

专题命中音频语音多模态：multimodal(title)

AI总结本文提出了一种高效的行星防御方法，通过高速动能穿甲弹粉碎小行星，可在不同预警时间内有效防御直径20-1000米的天体。

Comments 195 pages, 148 figures. Published in Advances in Space Research (ASR) 10-22; https://www.sciencedirect.com/science/article/pii/S0273117722009395

1. 音频语音多模态 4597 篇

Tuning Large language model for End-to-end Speech Translation

Music- and Lyrics-driven Dance Synthesis

LLaSM: Large Language and Speech Model

Exploring Efficient-Tuned Learning Audio Representation Method from BriVL

Exploring the Role of Audio in Video Captioning

Improving speech translation by fusing speech and text

A Light Weight Model for Active Speaker Detection

Object Segmentation with Audio Context

An Analysis of Semantically-Aligned Speech-Text Embeddings

Contrastive Audio-Language Learning for Music

Is Lip Region-of-Interest Sufficient for Lipreading?

On Adversarial Robustness of Large-scale Audio Visual Learning

Multimedia Datasets for Anomaly Detection: A Review

Continuous Emotion Recognition using Visual-audio-linguistic information: A Technical Report for ABAW3

Facial Expression Recognition with Swin Transformer

STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

Binaural SoundNet: Predicting Semantics, Depth and Motion with Binaural Sounds

Audio Visual Scene-Aware Dialog Generation with Transformer-based Video Representations

Learning music audio representations via weak language supervision

Sound-Guided Semantic Image Manipulation

TEASEL: A Transformer-Based Speech-Prefixed Language Model

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

MusicTM-Dataset for Joint Representation Learning among Sheet Music, Lyrics, and Musical Audio

Sound2Sight: Generating Visual Dynamics from Sound and Context

From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech

Self-supervised Moving Vehicle Tracking with Stereo Sound

LRS3-TED: a large-scale dataset for visual speech recognition

Multimodal-Language-Model-Driven Interaction and Companionship for Service Robots in Elderly-Care Facilities

omni-macos: On-Device Omni-Modal Search on Apple Silicon

PI -- Multimodal Planetary Defense