This survey presents a review of state-of-the-art deep neural network architectures, algorithms, and systems in vision and speech applications. Recent advances in deep artificial neural network algorithms and architectures have spurred rapid innovation and development of intelligent vision and speech systems. With availability of vast amounts of sensor data and cloud computing for processing and training of deep neural networks, and with increased sophistication in mobile and embedded technology, the next-generation intelligent systems are poised to revolutionize personal and commercial computing. This survey begins by providing background and evolution of some of the most successful deep learning models for intelligent vision and speech systems to date. An overview of large-scale industrial research and development efforts is provided to emphasize future trends and prospects of intelligent vision and speech systems. Robust and efficient intelligent systems demand low-latency and high fidelity in resource-constrained hardware platforms such as mobile devices, robots, and automobiles. Therefore, this survey also provides a summary of key challenges and recent successes in running deep neural networks on hardware-restricted platforms, i.e. within limited memory, battery life, and processing capabilities. Finally, emerging applications of vision and speech across disciplines such as affective computing, intelligent transportation, and precision medicine are discussed. To our knowledge, this paper provides one of the most comprehensive surveys on the latest developments in intelligent vision and speech applications from the perspectives of both software and hardware systems. Many of these emerging technologies using deep neural networks show tremendous promise to revolutionize research and development for future vision and speech systems.
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）；Tencent Youtu Lab（腾讯优图实验室）；XMU（厦门大学）；CASIA（中国科学院自动化研究所）
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（中国科学院智能信息处理重点实验室）；Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）
Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；Chungnam National University Hospital（忠南大学医院）；Chungnam National University College of Medicine（忠南大学医学院）
Parts of Speech-Grounded Subspaces in Vision-Language Models
Queen Mary University of London（伦敦玛丽女王大学）；National and Kapodistrian University of Athens（雅典国立卡波季斯特里安大学）；Archimedes/Athena RC（阿基米德/雅典研究与创新中心）；The Cyprus Institute（塞浦路斯研究所）
SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
School of Informatics, Xiamen University, China（厦门大学信息学院）；MiLM Plus, Xiaomi Inc., China（小米公司）；School of Electronic Science and Engineering, Xiamen University, China（厦门大学电子科学与技术学院）
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions
Hong Kong University of Science and Technology（香港科技大学）；The University of Hong Kong（香港大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；The Chinese University of Hong Kong（香港中文大学）；Sun Yat-sen University（中山大学）；Southern University of Science and Technology（南方科技大学）
Nexus: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision
Imperial College London（伦敦帝国学院）；University of Manchester（曼彻斯特大学）；HiThink Research（HiThink研究院）；Soochow University（苏州大学）；Hong Kong Baptist University（香港 Baptist大学）；Idiap Research Institute（Idiap研究 institute）；Meta AI；Fudan University（复旦大学）
From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models
School of Computer Science, Sichuan University（四川大学计算机学院）；School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院电子与计算机工程学院）；Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University（中国电信与西北工业大学人工智能研究院（TeleAI））
cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所 (AIST)）；University of Tsukuba（筑波大学）；Keio University（庆应义塾大学）；Tokyo Denki University（东京电机大学）
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；Tianjin University（天津大学）；Linköping University（林雪平大学）；Australian National University（澳大利亚国立大学）；University of California at Merced（加州大学默塞德分校）；Yonsei University（延世大学）；Google（谷歌）
