In this paper, we introduce SciGen, a new challenge dataset for the task of reasoning-aware data-to-text generation consisting of tables from scientific articles and their corresponding descriptions. Describing scientific tables goes beyond the surface realization of the table content and requires reasoning over table values. The unique properties of SciGen are that (1) tables mostly contain numerical values, and (2) the corresponding descriptions require arithmetic reasoning. SciGen is therefore the first dataset that assesses the arithmetic reasoning capabilities of generation models on complex input structures, i.e., tables from scientific articles. We study the effectiveness of state-of-the-art data-to-text generation models on SciGen and evaluate the results using common metrics as well as human evaluation. Our results and analyses show that (a) while humans like to reason for describing scientific tables, the ability of state-of-the-art models is severely limited on this task, (b) while adding more training data improves the results, it is not the solution for reasoning-aware text generation, and (c) one of the main bottlenecks for this task is the lack of proper automatic evaluation metrics. The data, code, and annotations for human evaluation will be available at https://github.com/UKPLab/SciGen. SciGen opens new avenues for future research in reasoning-aware text generation and evaluation.
Genentech Inc（基因泰克公司）；Pangaea Data Limited（泛大陆数据有限公司）
Structured (tabular) data in the preclinical and clinical domains contains valuable information about individuals and an efficient table-to-text summarization system can drastically reduce manual efforts to condense this data into reports. However, in practice, the problem is heavily impeded by the data paucity, data sparsity and inability of the state-of-the-art natural language generation models (including T5, PEGASUS and GPT-Neo) to produce accurate and reliable outputs. In this paper, we propose a novel table-to-text approach and tackle these problems with a novel two-step architecture whi…
HeLM: Highlighted Evidence augmented Language Model for Enhanced Table-to-Text Generation
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Shanghai Key Lab of Intelligent Information Processing, Fudan University（复旦大学智能信息处理上海市重点实验室）；Netease Games AI Lab（网易游戏AI实验室）；School of Data Science, Fudan University（复旦大学数据科学学院）；ShanghaiTech University（上海科技大学）
Shanghai Jiao Tong University（上海交通大学）；Microsoft Research Asia（微软亚洲研究院）；Carnegie Mellon University（卡内基梅隆大学）；Xi’an Jiaotong University（西安交通大学）
Tables are often created with hierarchies, but existing works on table reasoning mainly focus on flat tables and neglect hierarchical tables. Hierarchical tables challenge existing methods by hierarchical indexing, as well as implicit relationships of calculation and semantics. This work presents HiTab, a free and open dataset to study question answering (QA) and natural language generation (NLG) over hierarchical tables. HiTab is a cross-domain dataset constructed from a wealth of statistical reports (analyses) and Wikipedia pages, and has unique characteristics: (1) nearly all tables are hie…
Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data
Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI)（德国人工智能研究中心有限公司）；Technische Universität Berlin（柏林技术大学）；BIFOLD；Deutsche Telekom（德国电信）；Common Crawl Foundation（Common Crawl基金会）；Humboldt-Universität zu Berlin（柏林洪堡大学）
Tokyo Institute of Technology（东京工业大学）；National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所（AIST））
Numerical tables are widely used to present experimental results in scientific papers. For table understanding, a metric-type is essential to discriminate numbers in the tables. We introduce a new information extraction task, metric-type identification from multi-level header numerical tables, and provide a dataset extracted from scientific papers consisting of header tables, captions, and metric-types. We then propose two joint-learning neural classification and generation schemes featuring pointer-generator-based and BERT-based models. Our results show that the joint models can handle both i…
Table-Text Alignment: Explaining Claim Verification Against Tables in Scientific Papers
National Institute of Informatics, Japan（日本信息机构国家研究所）；The University of Tokyo, Japan（日本东京大学）；National Taiwan University（台湾国立大学）；JFLI, CNRS, Nantes Université, France（法国CNRS-纳antes大学联合研究所）
Georgia Institute of Technology（佐治亚理工学院）；Adobe Research（奥多比研究院）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；BioMap（百图生科）
We study the problem of extracting N-ary relation tuples from scientific articles. This task is challenging because the target knowledge tuples can reside in multiple parts and modalities of the document. Our proposed method ReSel decomposes this task into a two-stage procedure that first retrieves the most relevant paragraph/table and then selects the target entity from the retrieved component. For the high-level retrieval stage, ReSel designs a simple and effective feature set, which captures multi-level lexical and semantic similarities between the query and components. For the low-level se…
Understanding tables is an important and relevant task that involves understanding table structure as well as being able to compare and contrast information within cells. In this paper, we address this challenge by presenting a new dataset and tasks that addresses this goal in a shared task in SemEval 2020 Task 9: Fact Verification and Evidence Finding for Tabular Data in Scientific Documents (SEM-TAB-FACTS). Our dataset contains 981 manually-generated tables and an auto-generated dataset of 1980 tables providing over 180K statement and over 16M evidence annotations. SEM-TAB-FACTS featured two…
All Data on the Table: Novel Dataset and Benchmark for Cross-Modality Scientific Information Extraction
Nankai University（南开大学）；Tokyo Institute of Technology（东京工业大学）；Microsoft Research Asia（微软亚洲研究院）；Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院（BAAI））
Encoded but Not Routed: Explaining the Table-Chart Gap in Scientific Claim Verification
The University of Tokyo（东京大学）；NII LLMC（日本信息处理学会LLMC）；National Institute of Informatics（日本信息处理研究所）；University of Göttingen（哥廷根大学）；Inria, LS2N, Nantes Université（法国国家信息与自动化技术研究院（Inria）、LS2N实验室、南特大学）
Yale University（耶鲁大学）；Zhejiang University（浙江大学）；Penn State University（宾夕法尼亚州立大学）
Reasoning over tabular data requires both table structure understanding and a broad set of table reasoning skills. Current models with table-specific architectures and pre-training methods perform well on understanding table structures, but they still struggle with tasks that require various table reasoning skills. In this work, we develop ReasTAP to show that high-level table reasoning skills can be injected into models during pre-training without a complex table-specific architecture design. We define 7 table reasoning skills, such as numerical operation, temporal comparison, and conjunction…
Rethinking Stepwise Model Routing: A Cost-Efficient Table Reasoning Perspective
University of Science and Technology of China（中国科学技术大学）；The University of Melbourne（墨尔本大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合国家科学中心人工智能研究院）
University of Waterloo（滑铁卢大学）；Vector Institute（向量研究所）
Recent literature has shown that large language models (LLMs) are generally excellent few-shot reasoners to solve text reasoning tasks. However, the capability of LLMs on table reasoning tasks is yet to be explored. In this paper, we aim at understanding how well LLMs can perform table-related tasks with few-shot in-context learning. Specifically, we evaluated LLMs on popular table QA and fact verification datasets like WikiTableQuestion, FetaQA, TabFact, and FEVEROUS and found that LLMs are competent at complex reasoning over table structures, though these models are not pre-trained on any ta…
Tabular and textual question answering requires systems to perform reasoning over heterogeneous information, considering table structure, and the connections among table and text. In this paper, we propose a ChAin-centric Reasoning and Pre-training framework (CARP). CARP utilizes hybrid chain to model the explicit intermediate reasoning process across table and text for question answering. We also propose a novel chain-centric pre-training method, to enhance the pre-trained model in identifying the cross-modality reasoning process and alleviating the data sparsity problem. This method construc…
Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning
University of Science and Technology of China（中国科学技术大学）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；DAMO Academy, Alibaba Group（阿里巴巴集团达摩院）
TACR: A Table-alignment-based Cell-selection and Reasoning Model for Hybrid Question-Answering
Tokyo Institute of Technology（东京工业大学）；Nanyang Technological University（南洋理工大学）；Microsoft Research Asia（微软亚洲研究院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
Document-Level Numerical Reasoning across Single and Multiple Tables in Financial Reports
Department of Computer Science and Information Engineering, National Taiwan University（国立台湾大学资讯工程学系）；FinTech Center, National Taiwan University（国立台湾大学金融科技中心）
Reasoning by Commented Code for Table Question Answering
Department of Data Science, Seoul National University, Seoul, Republic of Korea（数据科学系，首尔国立大学，首尔，大韩民国）；Department of Computer Science（计算机科学系）；Engineering, Seoul National University, Seoul, Republic of Korea（工程系，首尔国立大学，首尔，大韩民国）
Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding
University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；Singapore University of Technology and Design（新加坡科技设计大学）；Shanghai Jiao Tong University（上海交通大学）
When TableQA Meets Noise: A Dual Denoising Framework for Complex Questions and Large-scale Tables
University of Science and Technology of China（中国科学技术大学）；The University of Melbourne（墨尔本大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Institute of Automation, CAS（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Beijing Academy of Artificial Intelligence（北京智源人工智能研究院）
In the real-world question answering scenarios, hybrid form combining both tabular and textual contents has attracted more and more attention, among which numerical reasoning problem is one of the most typical and challenging problems. Existing methods usually adopt encoder-decoder framework to represent hybrid contents and generate answers. However, it can not capture the rich relationship among numerical value, table schema, and text information on the encoder side. The decoder uses a simple predefined operator classifier which is not flexible enough to handle numerical reasoning processes w…
Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tables
State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；University of Science and Technology of China（中国科学技术大学）；Artificial Intelligence Research Institute（人工智能研究院）；iFLYTEK Co., Ltd（iFLYTEK公司）
Harbin Institute of Technology（哈尔滨工业大学）；JD AI Research（京东人工智能研究院）
Question answering requiring discrete reasoning, e.g., arithmetic computing, comparison, and counting, over knowledge is a challenging task. In this paper, we propose UniRPG, a semantic-parsing-based approach advanced in interpretability and scalability, to perform unified discrete reasoning over heterogeneous knowledge resources, i.e., table and text, as program generation. Concretely, UniRPG consists of a neural programmer and a symbolic program executor, where a program is the composition of a set of pre-defined general atomic and higher-order operations and arguments extracted from table a…
