多模态大模型 - arXivDaily 专题

2310.05181 2024-01-11 eess.AS cs.GR cs.HC cs.LG cs.SD 70%

Unified speech and gesture synthesis using flow matching

Shivam Mehta, Ruibo Tu, Simon Alexanderson, Jonas Beskow, Éva Székely, Gustav Eje Henter

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments 5 pages, 1 figure. Final version, accepted to IEEE ICASSP 2024

2312.09034 2023-12-15 eess.AS cs.SD eess.IV 70%

Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

Davide Berghi, Peipei Wu, Jinzheng Zhao, Wenwu Wang, Philip J. B. Jackson

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments ICASSP 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

2308.09300 2023-12-15 cs.CV cs.AI cs.MM cs.SD eess.AS 70%

V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models

Heng Wang, Jianbo Ma, Santiago Pascual, Richard Cartwright, Weidong Cai

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments AAAI 2024. Demo page: https://v2a-mapper.github.io/

2305.18500 2023-10-10 cs.CV cs.AI cs.CL cs.LG eess.AS 70%

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, Jing Liu

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by NeurIPS 2023

2309.12158 2023-09-22 cs.SD cs.IR cs.LG eess.AS 70%

Towards Robust and Truly Large-Scale Audio-Sheet Music Retrieval

Luis Carvalho, Gerhard Widmer

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments Proceedings of the IEEE 6th International Conference on Multimedia Information Processing and Retrieval (MIPR)

2309.09501 2023-09-19 cs.CV 70%

Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation

Shaofei Huang, Han Li, Yuqing Wang, Hongji Zhu, Jiao Dai, Jizhong Han, Wenge Rong, Si Liu

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted by IJCAI 2023

2308.07221 2023-08-28 cs.SD cs.LG eess.AS 70%

AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes

Zhaohui Li, Haitao Wang, Xinghua Jiang

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments Need to supplement more detailed experiments

2305.16103 2023-05-26 cs.CV cs.AI cs.CL cs.MM 70%

ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, Jing Liu

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2305.14359 2023-05-25 cs.MM cs.AI cs.CV cs.SD eess.AS 70%

Zero-shot personalized lip-to-speech synthesis with face image based voice control

Zheng-Yan Sheng, Yang Ai, Zhen-Hua Ling

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICASSP 2023

2305.04160 2023-05-23 cs.CL cs.AI cs.CV eess.AS 70%

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, Bo Xu

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2305.12311 2023-05-23 cs.CL cs.AI cs.CV cs.LG eess.AS 70%

i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data

Ziyi Yang, Mahmoud Khademi, Yichong Xu, Reid Pryzant, Yuwei Fang, Chenguang Zhu, Dongdong Chen, Yao Qian, Mei Gao, Yi-Ling Chen, Robert Gmyr, Naoyuki Kanda, Noel Codella, Bin Xiao, Yu Shi, Lu Yuan, Takuya Yoshioka, Michael Zeng, Xuedong Huang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2305.10763 2023-05-19 cs.SD eess.AS 70%

CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training

Zhenhui Ye, Rongjie Huang, Yi Ren, Ziyue Jiang, Jinglin Liu, Jinzheng He, Xiang Yin, Zhou Zhao

专题命中音频语音多模态：multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments Accepted by ACL 2023 (Main Conference)

2208.14114 2023-04-26 cs.CV 70%

Robust Sound-Guided Image Manipulation

Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Sang Ho Yoon, Jinkyu Kim, Sangpil Kim

专题命中音频语音多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments arXiv admin note: text overlap with arXiv:2112.00007

2301.02379 2023-04-04 cs.CV 70%

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, Tien-Tsin Wong

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments CVPR2023 Camera-Ready. Project Page: https://doubiiu.github.io/projects/codetalker/, Code: https://github.com/Doubiiu/CodeTalker

2303.00091 2023-03-02 eess.AS cs.AI cs.CL cs.CV cs.SD eess.IV 70%

Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model

Jaeyoung Huh, Sangjoon Park, Jeong Eun Lee, Jong Chul Ye

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

2302.02845 2023-02-07 cs.SD cs.LG eess.AS 70%

Audio Representation Learning by Distilling Video as Privileged Information

Amirhossein Hajavi, Ali Etemad

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 eess.AS

2211.01950 2022-12-23 cs.NE cs.CV cs.LG 70%

Unlocking the potential of two-point cells for energy-efficient and resilient training of deep nets

Ahsan Adeel, Adewale Adetomi, Khubaib Ahmed, Amir Hussain, Tughrul Arslan, W. A. Phillips

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

2211.14419 2022-11-29 cs.CV 70%

Panoramic Video Salient Object Detection with Ambisonic Audio Guidance

Xiang Li, Haoyuan Cao, Shijie Zhao, Junlin Li, Li Zhang, Bhiksha Raj

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

2107.03465 2022-11-04 cs.CV 70%

An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild

Panagiotis Antoniadis, Ioannis Pikoulis, Panagiotis P. Filntisis, Petros Maragos

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments 7 pages, 1 figure, 3 tables, accepted to the 2nd Workshop and Competition on Affective Behavior Analysis In-the-Wild (ABAW2)

Journal ref 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

2210.05076 2022-10-12 cs.SD cs.IR eess.AS 70%

ConchShell: A Generative Adversarial Networks that Turns Pictures into Piano Music

Wanpeng Fan, Yuanzhi Su, Yuxin Huang

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)；分类 eess.AS

Comments 5 pages

2208.10238 2022-08-23 cs.CV 70%

Learning Branched Fusion and Orthogonal Projection for Face-Voice Association

Muhammad Saad Saeed, Shah Nawaz, Muhammad Haris Khan, Sajid Javed, Muhammad Haroon Yousaf, Alessio Del Bue

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Submitted: IEEE Transactions on Multimedia. arXiv admin note: substantial text overlap with arXiv:2112.10483

2207.03048 2022-08-15 cs.CV 70%

AV-Gaze: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-Profilic Faces

Shreya Ghosh, Abhinav Dhall, Munawar Hayat, Jarrod Knibbe

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

2204.04645 2022-04-12 cs.SD cs.LG eess.AS 70%

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data

Yu Kang, Tianqiao Liu, Hang Li, Yang Hao, Wenbiao Ding

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments AAAI 2022

2203.14240 2022-03-30 cs.CV 70%

Audio-Adaptive Activity Recognition Across Video Domains

Yunhua Zhang, Hazel Doughty, Ling Shao, Cees G. M. Snoek

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted at CVPR 2022

2112.05329 2022-03-18 cs.CV cs.GR 70%

FaceFormer: Speech-Driven 3D Facial Animation with Transformers

Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, Taku Komura

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to CVPR 2022

2203.03291 2022-03-08 eess.AS cs.SD eess.IV 70%

Visually Supervised Speaker Detection and Localization via Microphone Array

Davide Berghi, Adrian Hilton, Philip J. B. Jackson

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments Erratum: Due to a bug in the evaluation script, the correct average distance (aD) metric is here reported in yellow. The analysis remains unchanged from the original paper as the trend between the old and new measures are perfectly monotonic. The bug was caused by an incorrect normalization factor

Journal ref IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP), 2021

2112.13350 2021-12-28 cs.SD cs.LG eess.AS 70%

Novel Dual-Channel Long Short-Term Memory Compressed Capsule Networks for Emotion Recognition

Ismail Shahin, Noor Hindawi, Ali Bou Nassif, Adi Alhudhaif, Kemal Polat

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 19 pages, 11 figures

Journal ref Published in Expert Systems With Applications, 2021

2111.11859 2021-11-24 cs.SD eess.AS q-bio.QM 70%

Longitudinal Speech Biomarkers for Automated Alzheimer's Detection

Jordi Laguarta Soler, Brian Subirana

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)；分类 eess.AS

Journal ref Frontiers in Computer Science, 08 April 2021

2111.05526 2021-11-11 cs.CV 70%

Space-Time Memory Network for Sounding Object Localization in Videos

Sizhe Li, Yapeng Tian, Chenliang Xu

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to BMVC2021. Project page: https://sites.google.com/view/bmvc2021stm

2110.13384 2021-10-27 cs.CV 70%

ViDA-MAN: Visual Dialog with Digital Humans

Tong Shen, Jiawei Zuo, Fan Shi, Jin Zhang, Liqin Jiang, Meng Chen, Zhengchen Zhang, Wei Zhang, Xiaodong He, Tao Mei

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV

1. 音频语音多模态 4597 篇

Unified speech and gesture synthesis using flow matching

Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection

V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

Towards Robust and Truly Large-Scale Audio-Sheet Music Retrieval

Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation

AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes

ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst

Zero-shot personalized lip-to-speech synthesis with face image based voice control

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data

CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training

Robust Sound-Guided Image Manipulation

CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior

Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model

Audio Representation Learning by Distilling Video as Privileged Information

Unlocking the potential of two-point cells for energy-efficient and resilient training of deep nets

Panoramic Video Salient Object Detection with Ambisonic Audio Guidance

An audiovisual and contextual approach for categorical and continuous emotion recognition in-the-wild

ConchShell: A Generative Adversarial Networks that Turns Pictures into Piano Music

Learning Branched Fusion and Orthogonal Projection for Face-Voice Association

AV-Gaze: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-Profilic Faces

Self-Supervised Audio-and-Text Pre-training with Extremely Low-Resource Parallel Data

Audio-Adaptive Activity Recognition Across Video Domains

FaceFormer: Speech-Driven 3D Facial Animation with Transformers

Visually Supervised Speaker Detection and Localization via Microphone Array

Novel Dual-Channel Long Short-Term Memory Compressed Capsule Networks for Emotion Recognition

Longitudinal Speech Biomarkers for Automated Alzheimer's Detection

Space-Time Memory Network for Sounding Object Localization in Videos

ViDA-MAN: Visual Dialog with Digital Humans