A word-as-image is a semantic typography technique where a word illustration presents a visualization of the meaning of the word, while also preserving its readability. We present a method to create word-as-image illustrations automatically. This task is highly challenging as it requires semantic understanding of the word and a creative idea of where and how to depict these semantics in a visually pleasing and legible manner. We rely on the remarkable ability of recent large pretrained language-vision models to distill textual concepts visually. We target simple, concise, black-and-white designs that convey the semantics clearly. We deliberately do not change the color or texture of the letters and do not use embellishments. Our method optimizes the outline of each letter to convey the desired concept, guided by a pretrained Stable Diffusion model. We incorporate additional loss terms to ensure the legibility of the text and the preservation of the style of the font. We show high quality and engaging results on numerous examples and compare to alternative techniques.
WordCraft: Interactive Artistic Typography with Attention Awareness and Noise Blending
Jiangxi University of Finance and Economics(江西财经大学);Tencent Robotics X Lab(腾讯机器人X实验室);Nanyang Technological University(南洋理工大学);Monash University(墨尔本大学);Hong Kong University(香港大学)
LEWIS: Latent Embeddings for Word Images and their Semantics
Xerox Research Centre Europe(施乐欧洲研究中心);Facebook AI Research(脸书人工智能研究院)
这项工作的目标是将语义纳入自然图像中的文本识别和检索任务。虽然文本识别和检索近年来受到了很多关注,但以前的工作都集中在识别或检索用作查询的完全相同的词,而不考虑语义。在本文中,我们提出了以下问题:\emph{can we predict semantic concepts directly from a word image, without explicitly trying to transcribe the word image or its characters at any point?}为了这个目标,我们提出了一个卷积神经网络(CNN)与加权排名损失的目标,确保相关的查询图像的概念排名领先于那些不相关的。这也可以解释为学习一个欧几里德空间,其中单词图像和概念联合嵌入。该模型是以端到端的方式学习的,从图像像素到语义概念,使用从词汇数据库(WordNet)中挖掘的综合生成的单词图像和概念的数据集。我们的研究结果表明,尽管任务的复杂性,单词图像和概念确实可以与高准确度的关联
Aligning Linguistic Words and Visual Semantic Units for Image Captioning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Nanjing University of Science and Technology(南京理工大学);Huawei Devices(华为终端);University of Chinese Academy of Sciences(中国科学院大学)