神经驱动的图像编辑
Neural-Driven Image Editing
AI总结：
LoongX通过多模态神经信号驱动图像编辑，结合扩散模型和对比学习实现高性能图像编辑任务。
摘要：
传统图像编辑通常依赖于手动提示，使其变得劳动密集且对那些有有限运动控制能力或语言能力的人不可行。利用最近在脑机接口（BCIs）和生成模型方面的进展，我们提出了LoongX，一种无需手部操作的图像编辑方法，该方法由多模态神经生理信号驱动。LoongX利用最先进的扩散模型，这些模型在包含23928对图像编辑数据的全面数据集中进行训练，每对数据都配以同步的脑电图（EEG）、功能近红外光谱（fNIRS）、光电容积图（PPG）和头部运动信号，这些信号捕捉了用户意图。为了有效解决这些信号的异质性，LoongX集成了两个关键模块。交叉尺度状态空间（CS3）模块编码信息丰富的模态特定特征。动态门控融合（DGF）模块进一步将这些特征聚合到统一的潜在空间中，然后通过在扩散变压器（DiT）上微调来对齐编辑语义。此外，我们使用对比学习预训练编码器，以对齐认知状态与嵌入的自然语言中的语义意图。广泛的实验表明，LoongX在性能上与文本驱动方法（CLIP-I：0.6605 vs. 0.6558；DINO：0.4812 vs. 0.4636）相当，并在神经信号与语音结合时表现更优（CLIP-T：0.2588 vs. 0.2549）。这些结果突显了神经驱动生成模型在实现可访问、直观的图像编辑方面的潜力，并为认知驱动的创意技术开辟了新方向。代码和数据集已发布在项目网站上：https://loongx1.github.io。
英文摘要：
Traditional image editing typically relies on manual prompting, making it labor-intensive and inaccessible to individuals with limited motor control or language abilities. Leveraging recent advances in brain-computer interfaces (BCIs) and generative models, we propose LoongX, a hands-free image editing approach driven by multimodal neurophysiological signals. LoongX utilizes state-of-the-art diffusion models trained on a comprehensive dataset of 23,928 image editing pairs, each paired with synchronized electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS), photoplethysmography (PPG), and head motion signals that capture user intent. To effectively address the heterogeneity of these signals, LoongX integrates two key modules. The cross-scale state space (CS3) module encodes informative modality-specific features. The dynamic gated fusion (DGF) module further aggregates these features into a unified latent space, which is then aligned with edit semantics via fine-tuning on a diffusion transformer (DiT). Additionally, we pre-train the encoders using contrastive learning to align cognitive states with semantic intentions from embedded natural language. Extensive experiments demonstrate that LoongX achieves performance comparable to text-driven methods (CLIP-I: 0.6605 vs. 0.6558; DINO: 0.4812 vs. 0.4636) and outperforms them when neural signals are combined with speech (CLIP-T: 0.2588 vs. 0.2549). These results highlight the promise of neural-driven generative models in enabling accessible, intuitive image editing and open new directions for cognitive-driven creative technologies. The code and dataset are released on the project website: https://loongx1.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
