机器人联觉：一种声音与情绪引导的AI绘画机器人
Robot Synesthesia: A Sound and Emotion Guided AI Painter
AI总结：
该研究提出机器人联觉方法，将声音、语音（含文本内容与语调情绪）作为机器人绘画的引导输入，集成至FRIDA框架，实验验证了其声音-画作对应性的有效性，拓展了绘画机器人的输入模态。
摘要：
如果一幅画能描绘千言万语，那声音或许能传递万种情思。尽管近期的机器人绘画与图像合成方法在从文本输入生成视觉内容方面已取得进展，但声音到图像的转换仍远未得到充分探索。总体而言，基于声音的交互界面和声波交互有望提升用户的可及性与操控性，并为传递复杂情绪及现实世界的动态特性提供途径。\n本文提出了一种利用声音与语音引导机器人绘画过程的方法，我们将其命名为机器人联觉（robot synesthesia）。对于通用声音，我们将模拟生成的画作与输入声音编码至同一潜在空间中。对于语音，我们将其拆分为转录文本与语音语调两部分：利用文本控制绘画内容，同时从语调中估算情绪，以此引导绘画的整体氛围。\n我们的方法已与机器人绘画框架FRIDA完全集成，为FRIDA现有的输入模态（如文本、风格）新增了声音与语音两种输入形式。在两项用户调研中，参与者正确猜出生成对应画作所用的情绪或自然声音的概率，是随机猜测概率的两倍以上。我们还对声音引导的图像编辑与音乐引导的绘画结果进行了定性讨论。
英文摘要：
If a picture paints a thousand words, sound may voice a million. While recent robotic painting and image synthesis methods have achieved progress in generating visuals from text inputs, the translation of sound into images is vastly unexplored. Generally, sound-based interfaces and sonic interactions have the potential to expand accessibility and control for the user and provide a means to convey complex emotions and the dynamic aspects of the real world. In this paper, we propose an approach for using sound and speech to guide a robotic painting process, known here as robot synesthesia. For general sound, we encode the simulated paintings and input sounds into the same latent space. For speech, we decouple speech into its transcribed text and the tone of the speech. Whereas we use the text to control the content, we estimate the emotions from the tone to guide the mood of the painting. Our approach has been fully integrated with FRIDA, a robotic painting framework, adding sound and speech to FRIDA's existing input modalities, such as text and style. In two surveys, participants were able to correctly guess the emotion or natural sound used to generate a given painting more than twice as likely as random chance. On our sound-guided image manipulation and music-guided paintings, we discuss the results qualitatively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
