arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.00430cs.AI
收藏

脑条件多模态合成:综述与分类

Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy

South China University of Technology(华南理工大学);Southeast University(东南大学)

Weijian Mai, Jian Zhang, Pengfei Fang, Zhijun Zhang

AI总结:

本综述全面考察了基于AIGC的脑条件多模态合成(AIGC-Brain)领域,介绍了相关数据集、脑区和生成模型基础,提出了解码模型的分类法,并探讨了当前挑战与未来方向,为该新兴领域提供了基础概述。

摘要:

在人工智能生成内容(AIGC)时代,条件多模态合成技术(如文本到图像、文本到视频、文本到音频等)正逐渐重塑现实世界中的自然内容。多模态合成技术的关键在于建立不同模态之间的映射关系。脑信号作为大脑如何解释外部信息的潜在反映,与各种外部模态展现出独特的“一对多”对应关系。这种对应关系使得脑信号成为多模态内容合成的一种极具前景的指导条件。脑条件多模态合成是指将脑信号解码回感知体验,这对于开发实用的脑机接口系统以及揭示大脑感知和理解外部刺激的复杂机制至关重要。本综述全面考察了基于AIGC的脑条件多模态合成这一新兴领域(称为AIGC-Brain),以描绘其当前现状和未来方向。首先,介绍了相关的脑神经成像数据集、功能脑区和主流生成模型,作为AIGC-Brain解码和分析的基础。接下来,我们为AIGC-Brain解码模型提供了全面的分类法,并展示了特定任务的代表性工作和详细的实现策略,以便于比较和深入分析。随后,引入了质量评估方法,用于定性和定量评估。最后,本综述探讨了所获得的见解,提出了当前面临的挑战,并概述了AIGC-Brain的前景。作为该领域的首篇综述,本文为AIGC-Brain研究的进展铺平了道路,提供了指导未来工作的基础概述。

英文摘要:

In the era of Artificial Intelligence Generated Content (AIGC), conditional multimodal synthesis technologies (e.g., text-to-image, text-to-video, text-to-audio, etc) are gradually reshaping the natural content in the real world. The key to multimodal synthesis technology is to establish the mapping relationship between different modalities. Brain signals, serving as potential reflections of how the brain interprets external information, exhibit a distinctive One-to-Many correspondence with various external modalities. This correspondence makes brain signals emerge as a promising guiding condition for multimodal content synthesis. Brian-conditional multimodal synthesis refers to decoding brain signals back to perceptual experience, which is crucial for developing practical brain-computer interface systems and unraveling complex mechanisms underlying how the brain perceives and comprehends external stimuli. This survey comprehensively examines the emerging field of AIGC-based Brain-conditional Multimodal Synthesis, termed AIGC-Brain, to delineate the current landscape and future directions. To begin, related brain neuroimaging datasets, functional brain regions, and mainstream generative models are introduced as the foundation of AIGC-Brain decoding and analysis. Next, we provide a comprehensive taxonomy for AIGC-Brain decoding models and present task-specific representative work and detailed implementation strategies to facilitate comparison and in-depth analysis. Quality assessments are then introduced for both qualitative and quantitative evaluation. Finally, this survey explores insights gained, providing current challenges and outlining prospects of AIGC-Brain. Being the inaugural survey in this domain, this paper paves the way for the progress of AIGC-Brain research, offering a foundational overview to guide future work.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑