面向创意领域多模态问题的带横向评估的MAP-Elites算法
MAP-Elites with Transverse Assessment for Multimodal Problems in Creative Domains
AI总结:
针对多模态创意任务,提出结合跨模态一致性评估的MEliTA算法,通过解耦模态、促进精英交叉,改善文本-图像映射,推动多模态自下而上编排发展。
摘要:
基于语言的生成模型的最新进展,为将多种不同人工制品类型(文本、图像、音频等)的生成器整合到一个系统中铺平了道路。目前,许多开源预训练模型将文本与其他模态相结合,从而实现了可在不同生成器之间比较的共享向量嵌入。在此背景下,我们提出了一种使用质量多样性进化处理多模态创意任务的新方法。我们的贡献是MAP-Elites算法的一个变体——带横向评估的MAP-Elites(MEliTA),它专为多模态创意任务量身定制,利用深度学习模型评估跨模态的一致性。MEliTA解耦了人工制品的模态,并促进精英个体之间的交叉融合。作为该算法的测试平台,我们为一款假设的电子游戏生成文本描述和封面图像,并为每件人工制品分配独特的模态特定行为特征。结果表明,与严格将每个图像-文本对视为一个解决方案的基线MAP-Elites算法相比,MEliTA能够改善解空间内的文本到图像映射。我们的方法代表了多模态自下而上编排的重大进步,并为未来协调多模态创意智能体的更复杂系统奠定了基础。
英文摘要:
The recent advances in language-based generative models have paved the way for the orchestration of multiple generators of different artefact types (text, image, audio, etc.) into one system. Presently, many open-source pre-trained models combine text with other modalities, thus enabling shared vector embeddings to be compared across different generators. Within this context we propose a novel approach to handle multimodal creative tasks using Quality Diversity evolution. Our contribution is a variation of the MAP-Elites algorithm, MAP-Elites with Transverse Assessment (MEliTA), which is tailored for multimodal creative tasks and leverages deep learned models that assess coherence across modalities. MEliTA decouples the artefacts' modalities and promotes cross-pollination between elites. As a test bed for this algorithm, we generate text descriptions and cover images for a hypothetical video game and assign each artefact a unique modality-specific behavioural characteristic. Results indicate that MEliTA can improve text-to-image mappings within the solution space, compared to a baseline MAP-Elites algorithm that strictly treats each image-text pair as one solution. Our approach represents a significant step forward in multimodal bottom-up orchestration and lays the groundwork for more complex systems coordinating multimodal creative agents in the future.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。