Understanding and reasoning about cooking recipes is a fruitful research direction towards enabling machines to interpret procedural text. In this work, we introduce RecipeQA, a dataset for multimodal comprehension of cooking recipes. It comprises of approximately 20K instructional recipes with multiple modalities such as titles, descriptions and aligned set of images. With over 36K automatically generated question-answer pairs, we design a set of comprehension and reasoning tasks that require joint understanding of images and text, capturing the temporal flow of events and making sense of procedural knowledge. Our preliminary results indicate that RecipeQA will serve as a challenging test bed and an ideal benchmark for evaluating machine comprehension systems. The data and leaderboard are available at http://hucvl.github.io/recipeqa.
Zhejiang University（浙江大学）；Singapore Management University（新加坡管理大学）；Alibaba-Zhejiang University Joint Institute of Frontier Technologies（阿里巴巴-浙江大学前沿技术联合研究院）
Massachusetts Institute of Technology（麻省理工学院）；Qatar Computing Research Institute, HBKU（哈马德·本·哈利法大学卡塔尔计算研究所）；Universitat Politècnica de Catalunya（加泰罗尼亚理工大学）
Sorbonne Université（索邦大学）；ETIS, UMR 8051（ETIS联合研究单位8051）；Université Paris Seine（巴黎塞纳大学）；Université Cergy-Pontoise（塞吉-蓬图瓦兹大学）；ENSEA（法国国立高等电子与应用学院）；CNRS（法国国家科学研究中心）
Graduate School of Informatics, Kyoto University（京都大学情报学研究科）；OMRON SINIC X Corporation（欧姆龙 SINIC X 公司）；Academic Center for Computing and Media Studies, Kyoto University（京都大学信息与媒体学术中心）
EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models
École Polytechnique Fédérale de Lausanne (EPFL)（瑞士联邦理工学院洛桑校区）；Microsoft（微软）；Scuola Superiore Sant’Anna（圣安娜高等学院）；Swiss Federal Institute of Technology Valais (EPFL Valais)（瑞士联邦技术学院瓦莱分校）；University of Geneva Medical School（日内瓦大学医学院）；Eidgenössische Technische Hochschule (ETH)（瑞士联邦理工学院）
