It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM Research（IBM研究院）；University of Washington（华盛顿大学）；University of Notre Dame（圣母大学）；WeChat AI, Tencent（腾讯微信人工智能）；University of California Irvine（加州大学欧文分校）
Existing question answering (QA) techniques are created mainly to answer questions asked by humans. But in educational applications, teachers often need to decide what questions they should ask, in order to help students to improve their narrative understanding capabilities. We design an automated question-answer generation (QAG) system for this education scenario: given a story book at the kindergarten to eighth-grade level as input, our system can automatically generate QA pairs that are capable of testing a variety of dimensions of a student's comprehension skills. Our proposed QAG model architecture is demonstrated using a new expert-annotated FairytaleQA dataset, which has 278 child-friendly storybooks with 10,580 QA pairs. Automatic and human evaluations show that our model outperforms state-of-the-art QAG baseline systems. On top of our QAG system, we also start to build an interactive story-telling application for the future real-world deployment in this educational scenario.
StorySparkQA: Expert-Annotated QA Pairs with Real-World Knowledge for Children's Story-Based Learning
East China Normal University（华东师范大学）；Northeastern University（东北大学）；Shanghai Jiao Tong University（上海交通大学）；Stanford University（斯坦福大学）；Harvard University（哈佛大学）；Adobe（奥多比公司）；University of Minnesota（明尼苏达大学）；Lab of Artificial Intelligence for Education（教育人工智能实验室）
Educational Question Generation of Children Storybooks via Question Type Distribution Learning and Event-Centric Summarization
IBM Research Europe（IBM欧洲研究院）；IBM Research（IBM研究院）；WeChat AI（微信AI）；The Hong Kong University of Science and Technology（香港科技大学）；Nanjing University of Information Science and Technology（南京信息工程大学）；Nankai University（南开大学）
School of Computer Science and Engineering, Seoul National University（首尔大学计算机科学与工程学院）；Surromind Robotics
Question-answering (QA) on video contents is a significant challenge for achieving human-level intelligence as it involves both vision and language in real-world settings. Here we demonstrate the possibility of an AI agent performing video story QA by learning from a large amount of cartoon videos. We develop a video-story learning model, i.e. Deep Embedded Memory Networks (DEMN), to reconstruct stories from a joint scene-dialogue video stream using a latent embedding space of observed data. The video stories are stored in a long-term memory component. For a given question, an LSTM-based atten…
University of Toronto（多伦多大学）；Massachusetts Institute of Technology（麻省理工学院）
Books are a rich source of both fine-grained information, how a character, an object or a scene looks like, as well as high-level semantics, what someone is thinking, feeling and how these states evolve through a story. This paper aims to align books to their movie releases in order to provide rich descriptive explanations for visual content that go semantically far beyond the captions available in current datasets. To align movies and books we exploit a neural sentence embedding that is trained in an unsupervised way from a large corpus of books, as well as a video-text neural embedding for c…
RoboBuddy in the Classroom: Exploring LLM-Powered Social Robots for Storytelling in Learning and Integration Activities
Computer-Human Interaction in Learning and Instruction (CHILI) Lab, EPFL, Lausanne, Switzerland（计算机-人类交互在学习与教学（CHILI）实验室，EPFL，拉沃斯，瑞士）；Depart of Computer Science of the University College London, London, UK（大学学院伦敦计算机科学系，伦敦，英国）
We present a novel deep learning architecture to address the cloze-style question answering task. Existing approaches employ reading mechanisms that do not fully exploit the interdependency between the document and the query. In this paper, we propose a novel \emph{dependent gated reading} bidirectional GRU network (DGR) to efficiently model the relationship between the document and the query during encoding and decision making. Our evaluation shows that DGR obtains highly competitive performance on well-known machine comprehension benchmarks such as the Children's Book Test (CBT-NE and CBT-CN…
Facebook AI Research（脸书人工智能研究院）；University of Cambridge（剑桥大学）
We introduce a new test of how well language models capture meaning in children's books. Unlike standard language modelling benchmarks, it distinguishes the task of predicting syntactic function words from that of predicting lower-frequency words, which carry greater semantic content. We compare a range of state-of-the-art models, each with a different way of encoding what has been previously read. We show that models which store explicit representations of long-term contexts outperform state-of-the-art neural language models at predicting semantic content words, although this advantage is not…
Personality Understanding of Fictional Characters during Book Reading
Princeton University（普林斯顿大学）；Syracuse University（雪城大学）；Pattern Recognition Center, WeChat AI（微信人工智能模式识别中心）；Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
Several large cloze-style context-question-answer datasets have been introduced recently: the CNN and Daily Mail news data and the Children's Book Test. Thanks to the size of these datasets, the associated text comprehension task is well suited for deep-learning techniques that currently seem to outperform all alternative approaches. We present a new, simple model that uses attention to directly pick the answer from the context as opposed to computing the answer using a blended representation of words in the document as is usual in similar models. This makes the model particularly suitable for…
Reading comprehension (RC)---in contrast to information retrieval---requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper c…
University of Notre Dame（圣母大学）；University of California, Irvine（加州大学欧文分校）；Georgia Institute of Technology（佐治亚理工学院）；Rensselaer Polytechnic Institute（伦斯勒理工学院）；University of Washington（华盛顿大学）；IBM Research（IBM研究院）
Despite its benefits for children's skill development and parent-child bonding, many parents do not often engage in interactive storytelling by having story-related dialogues with their child due to limited availability or challenges in coming up with appropriate questions. While recent advances made AI generation of questions from stories possible, the fully-automated approach excludes parent involvement, disregards educational goals, and underoptimizes for child engagement. Informed by need-finding interviews and participatory design (PD) results, we developed StoryBuddy, an AI-enabled syste…
"Once Upon a Time..." Literary Narrative Connectedness Progresses with Grade Level: Potential Impact on Reading Fluency and Literacy Skills
Mobile Brain（移动脑机构）；Bioinformatics Multidisciplinary Environment (BioME), Federal University of Rio Grande do Norte (UFRN)（巴西北里奥格兰德联邦大学生物信息学多学科环境）；Árvore（阿沃雷机构）；Institute of Psychiatry (IPUB), Federal University of Rio de Janeiro (UFRJ)（里约热内卢联邦大学精神病学研究所）
University of Maryland, College Park（马里兰大学学院市分校）；University of Colorado, Boulder（科罗拉多大学博尔德分校）
Visual narrative is often a combination of explicit information and judicious omissions, relying on the viewer to supply missing details. In comics, most movements in time and space are hidden in the "gutters" between panels. To follow the story, readers logically connect panels together by inferring unseen actions through a process called "closure". While computers can now describe what is explicitly depicted in natural images, in this paper we examine whether they can understand the closure-driven narratives conveyed by stylized artwork and dialogue in comic book panels. We construct a datas…
Previously on the Stories: Recap Snippet Identification for Story Reading
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；Xi’an Jiaotong University（西安交通大学）；Pattern Recognition Center, WeChat AI, Tencent Inc.（腾讯微信AI模式识别中心）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
The ability to automatically determine the age audience of a novel provides many opportunities for the development of information retrieval tools. Firstly, developers of book recommendation systems and electronic libraries may be interested in filtering texts by the age of the most likely readers. Further, parents may want to select literature for children. Finally, it will be useful for writers and publishers to determine which features influence whether the texts are suitable for children. In this article, we compare the empirical effectiveness of various types of linguistic features for the…
The increasing adoption of technology to augment or even replace traditional face-to-face learning has led to the development of a myriad of tools and platforms aimed at engaging the students and facilitating the teacher's ability to present new information. The IMapBook project aims at improving the literacy and reading comprehension skills of elementary school-aged children by presenting them with interactive e-books and letting them take part in moderated book discussions. This study aims to develop and illustrate a machine learning-based approach to message classification that could be use…
