Building a question-answering agent currently requires large annotated datasets, which are prohibitively expensive. This paper proposes Schema2QA, an open-source toolkit that can generate a Q&A system from a database schema augmented with a few annotations for each field. The key concept is to cover the space of possible compound queries on the database with a large number of in-domain questions synthesized with the help of a corpus of generic query templates. The synthesized data and a small paraphrase set are used to train a novel neural network based on the BERT pretrained model. We use Schema2QA to generate Q&A systems for five Schema.org domains, restaurants, people, movies, books and music, and obtain an overall accuracy between 64% and 75% on crowdsourced questions for these domains. Once annotations and paraphrases are obtained for a Schema.org schema, no additional manual effort is needed to create a Q&A agent for any website that uses the same schema. Furthermore, we demonstrate that learning can be transferred from the restaurant to the hotel domain, obtaining a 64% accuracy on crowdsourced questions with no manual effort. Schema2QA achieves an accuracy of 60% on popular restaurant questions that can be answered using Schema.org. Its performance is comparable to Google Assistant, 7% lower than Siri, and 15% higher than Alexa. It outperforms all these assistants by at least 18% on more complex, long-tail questions.
University of Washington（华盛顿大学）；Amazon（亚马逊）；Allen Institute for AI（艾伦人工智能研究所）
In many documents, such as semi-structured webpages, textual semantics are augmented with additional information conveyed using visual elements including layout, font size, and color. Prior work on information extraction from semi-structured websites has required learning an extraction model specific to a given template via either manually labeled or distantly supervised data from that template. In this work, we propose a solution for "zero-shot" open-domain relation extraction from webpages with a previously unseen template, including from websites with little overlap with existing sources of…
The abundant semi-structured data on the Web, such as HTML-based tables and lists, provide commercial search engines a rich information source for question answering (QA). Different from plain text passages in Web documents, Web tables and lists have inherent structures, which carry semantic correlations among various elements in tables and lists. Many existing studies treat tables and lists as flat documents with pieces of text and do not make good use of semantic information hidden in structures. In this paper, we propose a novel graph representation of Web tables and lists based on a system…
OpenAI's ChatGPT Atlas introduces new capabilities for web interaction, enabling the model to analyze webpages, process user intents, and execute cursor and keyboard inputs directly within the browser. While its capacity for information retrieval tasks has been demonstrated, its performance in dynamic, interactive environments remains less explored. In this study, we conduct an early evaluation of Atlas's web interaction capabilities using browser-based games as test scenarios, including Google's T-Rex Runner, Sudoku, Flappy Bird, and Stein.world. We employ in-game performance scores as quanti…
In this paper, we propose a method to extract descriptions of technical terms from Web pages in order to utilize the World Wide Web as an encyclopedia. We use linguistic patterns and HTML text structures to extract text fragments containing term descriptions. We also use a language model to discard extraneous descriptions, and a clustering method to summarize resultant descriptions. We show the effectiveness of our method by way of experiments.
WebRouter: Query-specific Router via Variational Information Bottleneck for Cost-sensitive Web Agent
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（人工智能学院，南京航空航天大学）；Hong Kong Baptist University（香港 Baptist 大学）；SKLCCSE Lab, Beihang University（SKLCCSE 实验室，北京航空航天大学）；Pengcheng Laboratory, Shenzhen, China（鹏城实验室，深圳，中国）
In this paper, we investigate the use of linguistically motivated and computationally efficient structured language models for reranking N-best hypotheses in a statistical machine translation system. These language models, developed from Constraint Dependency Grammar parses, tightly integrate knowledge of words, morphological and lexical features, and syntactic dependency constraints. Two structured language models are applied for N-best rescoring, one is an almost-parsing language model, and the other utilizes more syntactic features by explicitly modeling syntactic dependencies between words…
DynaWeb: Model-Based Reinforcement Learning of Web Agents
Shanghai Jiao Tong University（上海交通大学）；Sichuan University（四川大学）；Hong Kong University of Science and Technology（香港科学与技术大学）；McGill University（麦吉尔大学）；Shanghai AI Lab（上海人工智能实验室）；Gradient；University of Toronto（多伦多大学）；Mila - Quebec AI Institute（魁北克AI研究所）
Large Language Models Empowered Personalized Web Agents
National University of Singapore（新加坡国立大学）；The Hong Kong Polytechnic University（香港理工大学）；University of Science and Technology of China（中国科学技术大学）；Eastern Institute of Technology（宁波东方理工大学）
Web-CogReasoner: Towards Multimodal Knowledge-Induced Cognitive Reasoning for Web Agents
Southwestern University of Finance and Economics（西南财经大学）；Shanghai Jiao Tong University（上海交通大学）；Central South University（中南大学）；Hithink Research（Hithink研究）；Westlake University（西湖大学）；Harbin Institute of Technology（哈尔滨工业大学）；University of Manchester（曼彻斯特大学）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Adelaide（阿德莱德大学）；Fudan University（复旦大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；Chengdu Everimaging Science and Technology Co., Ltd（成都亿联科技有限公司）
Existing benchmarks for grounding language in interactive environments either lack real-world linguistic elements, or prove difficult to scale up due to substantial human involvement in the collection of data or feedback signals. To bridge this gap, we develop WebShop -- a simulated e-commerce website environment with $1.18$ million real-world products and $12,087$ crowd-sourced text instructions. Given a text instruction specifying a product requirement, an agent needs to navigate multiple types of webpages and issue diverse actions to find, customize, and purchase an item. WebShop provides s…
