We consider the cross-modal task of producing color representations for text phrases. Motivated by the fact that a significant fraction of user queries on an image search engine follow an (attribute, object) structure, we propose a generative adversarial network that generates color profiles for such bigrams. We design our pipeline to learn composition - the ability to combine seen attributes and objects to unseen pairs. We propose a novel dataset curation pipeline from existing public sources. We describe how a set of phrases of interest can be compiled using a graph propagation technique, and then mapped to images. While this dataset is specialized for our investigations on color, the method can be extended to other visual dimensions where composition is of interest. We provide detailed ablation studies that test the behavior of our GAN architecture with loss functions from the contrastive learning literature. We show that the generative model achieves lower Frechet Inception Distance than discriminative ones, and therefore predicts color profiles that better match those from real images. Finally, we demonstrate improved performance in image retrieval and classification, indicating the crucial role that color plays in these downstream tasks.
Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task
Center for Brain Science, Harvard University(哈佛大学脑科学中心);Physics & Informatics Laboratories, NTT Research, Inc.(NTT研究所物理与信息实验室);EECS Department, University of Michigan(密歇根大学电子工程与计算机科学系)
Compositional Text-to-Image Generation Via Region-aware Bimodal Direct Preference Optimization
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University(上海智能信息处理关键实验室,复旦大学计算机学院);Shanghai Collaborative Innovation Center of Intelligent Visual Computing(上海智能视觉计算协同创新中心)
ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance
S-Lab, Nanyang Technological University(南洋理工大学S-Lab);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Learning Compositional Visual Concepts with Mutual Consistency
School of Electrical and Computer Engineering, Cornell University(康奈尔大学电气与计算机工程学院);Siemens Corporate Technology(西门子公司技术部);Nancy E. and Peter C. Meinig School of Biomedical Engineering, Cornell University(康奈尔大学南希·E·和彼得·C·迈尼格生物医学工程学院)
CardioComposer: Leveraging Differentiable Geometry for Compositional Control of Anatomical Diffusion Models
Massachusetts Institute of Technology(麻省理工学院);Brigham and Women’s Hospital(布里奇沃特医院);American University in Cairo(开罗美国大学);École Polytechnique Fédérale de Lausanne(洛桑联邦理工学院)
Training-free Subject-Enhanced Attention Guidance for Compositional Text-to-image Generation
University of Chinese Academy of Sciences, Beijing, China(中国科学院大学);Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所);Kuaishou Technology, Beijing, China(快手科技)
Compositional Inversion for Stable Diffusion Models
the Hong Kong Polytechnic University(香港理工大学);Center for Artificial Intelligence and Robotics, HKISI, CAS(中国科学院香港创新研究院人工智能与机器人创新中心);Sichuan University(四川大学);UCAS(中国科学院大学);State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室)
Exploring the AI Obedience: Why is Generating a Pure Color Image Harder than CyberPunk?
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));City University of Hong Kong(城市大学);Chinese University of Hong Kong(香港中文大学);McGill University(麦吉尔大学)
Improving Compositional Text-to-image Generation with Large Vision-Language Models
Rutgers University(罗格斯大学);Sun Yat-sen University(中山大学);Shanghai AI Laboratory(上海人工智能实验室);The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学)
Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Models
Zhejiang University(浙江大学);Shanghai Institute for Advanced Study of Zhejiang University(浙江大学上海高等研究院);Endoscopy Center, Zhongshan Hospital, Fudan University(复旦大学中山医院内窥镜中心);Shanghai Collaborative Innovation Center of Endoscopy(上海内窥镜协同创新中心);Digital Medical Research Center, School of Basic Medical Sciences, Fudan University(复旦大学基础医学院数字医疗研究中心);Shanghai Key Laboratory of MICCAI(上海MICCAI重点实验室);Alliance Manchester Business School, The University of Manchester(曼彻斯特大学联盟商学院);Data Science Institute, Imperial College London(伦敦帝国学院数据科学研究所)
COMBO: Compositional World Models for Embodied Multi-Agent Cooperation
University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);IIIS, Tsinghua University(清华大学信息科学技术学院);Peking University(北京大学);Johns Hopkins University(约翰斯·霍普金斯大学);Honda Research Institute USA(美国本田研究所);MIT(麻省理工学院)