Language is ambiguous; many terms and expressions can convey the same idea. This is especially true in creative practice, where ideas and design intents are highly subjective. We present a dataset, Ambiguous Descriptions of Art Images (ADARI), of contemporary workpieces, which aims to provide a foundational resource for subjective image description and multimodal word disambiguation in the context of creative practice. The dataset contains a total of 240k images labeled with 260k descriptive sentences. It is additionally organized into sub-domains of architecture, art, design, fashion, furniture, product design and technology. In subjective image description, labels are not deterministic: for example, the ambiguous label dynamic might correspond to hundreds of different images. To understand this complexity, we analyze the ambiguity and relevance of text with respect to images using the state-of-the-art pre-trained BERT model for sentence classification. We provide a baseline for multi-label classification tasks and demonstrate the potential of multimodal approaches for understanding ambiguity in design intentions. We hope that ADARI dataset and baselines constitute a first step towards subjective label classification.
Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations
2 The Bredesen Center for Interdisciplinary Research；3 Department of Electrical Engineering \& Computer Science, University of Tennessee, Knoxville, TN 37916；4 Oak Ridge National Laboratory (ORNL), Oak Ridge, TN 37830
Named Entity Recognition and Disambiguation (NERD) systems are foundational for information retrieval, question answering, event detection, and other natural language processing (NLP) applications. We introduce TweetNERD, a dataset of 340K+ Tweets across 2010-2021, for benchmarking NERD systems on Tweets. This is the largest and most temporally diverse open sourced dataset benchmark for NERD on Tweets and can be used to facilitate research in this area. We describe evaluation setup with TweetNERD for three NERD tasks: Named Entity Recognition (NER), Entity Linking with True Spans (EL), and End…
Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
University of British Columbia（不列颠哥伦比亚大学）；Zuse School（Zuse学校）；Qatar Computing Research Institute (QCRI)（卡塔尔计算研究所）；Hamad Bin Khalifa University（哈马德·本·哈利法大学）
We consider and propose a new problem of retrieving audio files relevant to multimodal design document inputs comprising both textual elements and visual imagery, e.g., birthday/greeting cards. In addition to enhancing user experience, integrating audio that matches the theme/style of these inputs also helps improve the accessibility of these documents (e.g., visually impaired people can listen to the audio instead). While recent work in audio retrieval exists, these methods and datasets are targeted explicitly towards natural images. However, our problem considers multimodal design documents …
TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models
National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；University of Science and Technology of China（中国科学技术大学）；Hong Kong Polytechnic University（香港理工大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Verb Sense Disambiguation is a well-known task in NLP, the aim is to find the correct sense of a verb in a sentence. Recently, this problem has been extended in a multimodal scenario, by exploiting both textual and visual features of ambiguous verbs leading to a new problem, the Visual Verb Sense Disambiguation (VVSD). Here, the sense of a verb is assigned considering the content of an image paired with it rather than a sentence in which the verb appears. Annotating a dataset for this task is more complex than textual disambiguation, because assigning the correct sense to a pair of $<$image, v…
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
CUHK MMLab（香港中文大学多媒体实验室）；Waseda University（早稻田大学）；Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）；CPII under InnoHK（创新香港旗下的CPII机构）
Singing Timbre Popularity Assessment Based on Multimodal Large Foundation Model
Zhejiang University（浙江大学）；Carnegie Mellon University（卡内基梅隆大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；University of California, Berkeley（加州大学伯克利分校）；University of Manchester（曼彻斯特大学）；Innovation Center of Yangtze River Delta, Zhejiang University（长江三角洲创新中心，浙江大学）
IBM Research（IBM研究院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
Storytelling is an open-ended task that entails creative thinking and requires a constant flow of ideas. Natural language generation (NLG) for storytelling is especially challenging because it requires the generated text to follow an overall theme while remaining creative and diverse to engage the reader. In this work, we introduce a system and a web-based demo, FairyTailor, for human-in-the-loop visual story co-creation. Users can create a cohesive children's fairytale by weaving generated texts and retrieved images with their input. FairyTailor adds another modality and modifies the text gen…
Shanghai Jiao Tong University（上海交通大学）；National Institute of Information（国立信息研究所）
Word representation is a fundamental component in neural language understanding models. Recently, pre-trained language models (PrLMs) offer a new performant method of contextualized word representations by leveraging the sequence-level context for modeling. Although the PrLMs generally give more accurate contextualized word representations than non-contextualized models do, they are still subject to a sequence of text contexts without diverse hints for word representation from multimodality. This paper thus proposes a visual representation method to explicitly enhance conventional word embeddi…
University of Dayton Research Institute（代顿大学研究院）；SIL International（国际暑期语言学院）；TU Darmstadt（达姆施塔特工业大学）；Masakhane
We present Bloom Library, a linguistically diverse set of multimodal and multilingual datasets for language modeling, image captioning, visual storytelling, and speech synthesis/recognition. These datasets represent either the most, or among the most, multilingual datasets for each of the included downstream tasks. In total, the initial release of the Bloom Library datasets covers 363 languages across 32 language families. We train downstream task models for various languages represented in the data, showing the viability of the data for future work in low-resource, multimodal NLP and establis…
We introduce CM3, a family of causally masked generative models trained over a large corpus of structured multi-modal documents that can contain both text and image tokens. Our new causally masked approach generates tokens left to right while also masking out a small number of long token spans that are generated at the end of the string, instead of their original positions. The casual masking object provides a type of hybrid of the more common causal and masked language models, by enabling full generative modeling while also providing bidirectional context when generating the masked spans. We …
Shanghai Jiao Tong University（上海交通大学）；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；National Institute of Information and Communications Technology（日本国立信息通信研究机构）
Representation learning is the foundation of natural language processing (NLP). This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs or a shared cross-modal embedding space that is pre-trained on out-of-shelf text-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively. The two sequences of representations are further fuse…
Imperial College London（帝国理工学院）；Federal University of Mato Grosso（马托格罗索联邦大学）
This paper introduces a large-scale multimodal and multilingual dataset that aims to facilitate research on grounding words to images in their contextual usage in language. The dataset consists of images selected to unambiguously illustrate concepts expressed in sentences from movie subtitles. The dataset is a valuable resource as (i) the images are aligned to text fragments rather than whole sentences; (ii) multiple images are possible for a text fragment and a sentence; (iii) the sentences are free-form and real-world like; (iv) the parallel texts are multilingual. We set up a fill-in-the-bl…
$\left|\,\circlearrowright\,\boxed{\text{BUS}}\,\right|$: A Large and Diverse Multimodal Benchmark for evaluating the ability of Vision-Language Models to understand Rebus Puzzles
Tredence Inc.（特伦德公司）；Indian Institute of Technology Kharagpur（印度理工学院克拉格浦尔分校）；Inria Paris-Rocquencourt（巴黎-罗克琴库特研究所）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒姆研究实验室）
