Deriving event storylines is an effective summarization method to succinctly organize extensive information, which can significantly alleviate the pain of information overload. The critical challenge is the lack of widely recognized definition of storyline metric. Prior studies have developed various approaches based on different assumptions about users' interests. These works can extract interesting patterns, but their assumptions do not guarantee that the derived patterns will match users' preference. On the other hand, their exclusiveness of single modality source misses cross-modality information. This paper proposes a method, multimodal imitation learning via generative adversarial networks(MIL-GAN), to directly model users' interests as reflected by various data. In particular, the proposed model addresses the critical challenge by imitating users' demonstrated storylines. Our proposed model is designed to learn the reward patterns given user-provided storylines and then applies the learned policy to unseen data. The proposed approach is demonstrated to be capable of acquiring the user's implicit intent and outperforming competing methods by a substantial margin with a user study.
IBM Research（IBM研究院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）
Storytelling is an open-ended task that entails creative thinking and requires a constant flow of ideas. Natural language generation (NLG) for storytelling is especially challenging because it requires the generated text to follow an overall theme while remaining creative and diverse to engage the reader. In this work, we introduce a system and a web-based demo, FairyTailor, for human-in-the-loop visual story co-creation. Users can create a cohesive children's fairytale by weaving generated texts and retrieved images with their input. FairyTailor adds another modality and modifies the text gen…
TaleForge: Interactive Multimodal System for Personalized Story Creation
1 University of Science, VNU-HCM, Ho Chi Minh City, Vietnam；2 Vietnam National University, Ho Chi Minh City, Vietnam；3 Department of Computer Science \ of Dayton Ohio, United States
Illuminating Unified Multimodal Model for Free-form Interleaved Text-Image Generation
Harbin Institute of Technology（哈尔滨工业大学）；Huawei Noah’s Ark Lab（华为诺亚实验室）；Zhengzhou Advanced Research Institute of Harbin Institute of Technology（哈尔滨工业大学郑州先进研究院）；Nankai University（南开大学）
University of Maryland, College Park（马里兰大学帕克分校）；Microsoft Research（微软研究院）
Mobile agents that can leverage help from humans can potentially accomplish more complex tasks than they could entirely on their own. We develop "Help, Anna!" (HANNA), an interactive photo-realistic simulator in which an agent fulfills object-finding tasks by requesting and interpreting natural language-and-vision assistance. An agent solving tasks in a HANNA environment can leverage simulated human assistants, called ANNA (Automatic Natural Navigation Assistants), which, upon request, provide natural language and visual instructions to direct the agent towards the goals. To address the HANNA …
Imitation Game for Adversarial Disillusion with Chain-of-Thought Reasoning in Generative AI
Information and Society Research Division, National Institute of Informatics（信息与社会研究部，国立信息研究所）；Department of Information Engineering and Computer Science, Feng Chia University（信息工程与计算机科学系，逢甲大学）
Imitation learning from human-provided demonstrations is a strong approach for learning policies for robot manipulation. While the ideal dataset for imitation learning is homogenous and low-variance -- reflecting a single, optimal method for performing a task -- natural human behavior has a great deal of heterogeneity, with several optimal ways to demonstrate a task. This multimodality is inconsequential to human users, with task variations manifesting as subconscious choices; for example, reaching down, then across to grasp an object, versus reaching across, then down. Yet, this mismatch pres…
A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
JTB Technology Corp.（JTB技术公司）；Stockton University（斯托顿大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）；AppCubic USA（AppCubic美国公司）；Nomad Sustaintech LTD（Nomad可持续科技有限公司）；Georgia Institute of Technology（佐治亚理工学院）；Emory University（埃默里大学）；University of Liverpool（利物浦大学）；Indiana University（印第安纳大学）；Purdue University（普渡大学）
Liaohe-CobotMagic-PnP: an Imitation Learning Dataset of Intelligent Robot for Industrial Applications
School of Information Science and Engineering, Shandong Normal University（信息科学与工程学院，山东师范大学）；School of Advanced Technology, Xi’an Jiaotong-Liverpool University（先进技术学院，西安交通大学利物浦大学）；Liaohe Laboratory, Innovation Institute of Intelligent Robotics Shenyang Co., Ltd.（辽河实验室，沈阳智能机器人创新研究院有限公司）；School of Information Engineering, Shenyang University of Chemical Technology（信息工程学院，沈阳化工大学）；Shenyang Institute of Automation, Chinese Academy of Sciences（沈阳自动化研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）
