Text summarization has been a crucial problem in natural language processing (NLP) for several decades. It aims to condense lengthy documents into shorter versions while retaining the most critical information. Various methods have been proposed for text summarization, including extractive and abstractive summarization. The emergence of large language models (LLMs) like GPT3 and ChatGPT has recently created significant interest in using these models for text summarization tasks. Recent studies \cite{goyal2022news, zhang2023benchmarking} have shown that LLMs-generated news summaries are already on par with humans. However, the performance of LLMs for more practical applications like aspect or query-based summaries is underexplored. To fill this gap, we conducted an evaluation of ChatGPT's performance on four widely used benchmark datasets, encompassing diverse summaries from Reddit posts, news articles, dialogue meetings, and stories. Our experiments reveal that ChatGPT's performance is comparable to traditional fine-tuning methods in terms of Rouge scores. Moreover, we highlight some unique differences between ChatGPT-generated summaries and human references, providing valuable insights into the superpower of ChatGPT for diverse text summarization tasks. Our findings call for new directions in this area, and we plan to conduct further research to systematically examine the characteristics of ChatGPT-generated summaries through extensive human evaluation.
Information-Theoretic Distillation for Reference-less Summarization
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗 G. 艾伦计算机科学与工程学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT
School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机学院）；School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）；Shanghai Clinical Research and Trial Center（上海临床研究与试验中心）；School of Computing, University of Georgia（佐治亚大学计算学院）；Department of Computer Science and Engineering, The University of Texas at Arlington（德克萨斯大学阿灵顿分校计算机科学与工程系）；Department of Radiology, Massachusetts General Hospital（麻省总医院放射科）
Columbia University（哥伦比亚大学）；University of Pennsylvania（宾夕法尼亚大学）；Amazon AI（亚马逊人工智能实验室）
We present a new summarization task, generating summaries of novel chapters using summary/chapter pairs from online study guides. This is a harder task than the news summarization task, given the chapter length as well as the extreme paraphrasing and generalization found in the summaries. We focus on extractive summarization, which requires the creation of a gold-standard set of extractive summaries. We present a new metric for aligning reference summary sentences with chapter sentences to create gold extracts and also experiment with different alignment methods. Our experiments demonstrate si…
This paper explores the effect of using multitask learning for abstractive summarization in the context of small training corpora. In particular, we incorporate four different tasks (extractive summarization, language modeling, concept detection, and paraphrase detection) both individually and in combination, with the goal of enhancing the target task of abstractive summarization via multitask learning. We show that for many task combinations, a model trained in a multitask setting outperforms a model trained only for abstractive summarization, with no additional summarization data introduced.…
Although domain shift has been well explored in many NLP applications, it still has received little attention in the domain of extractive text summarization. As a result, the model is under-utilizing the nature of the training data due to ignoring the difference in the distribution of training sets and shows poor generalization on the unseen domain. With the above limitation in mind, in this paper, we first extend the conventional definition of the domain from categories into data sources for the text summarization task. Then we re-purpose a multi-domain summarization dataset and verify how th…
HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
Tokyo Metropolitan University（东京都立大学）；RIKEN（理化学研究所）；National Hospital Organization（国立医院机构）；Kitami Institute of Technology（北见工业大学）
Automated summarization of clinical texts can reduce the burden of medical professionals. "Discharge summaries" are one promising application of the summarization, because they can be generated from daily inpatient records. Our preliminary experiment suggests that 20-31% of the descriptions in discharge summaries overlap with the content of the inpatient records. However, it remains unclear how the summaries should be generated from the unstructured source. To decompose the physician's summarization process, this study aimed to identify the optimal granularity in summarization. We first define…
Exploring the Limits of ChatGPT in Software Security Applications
University of Michigan, Ann Arbor（密歇根大学安娜堡分校）；ASU（亚利桑那州立大学）；Washington University in St. Louis（圣路易斯华盛顿大学）；University of Wisconsin, Madison（威斯康星大学麦迪逊分校）
AI Ethics and Social Norms: Exploring ChatGPT's Capabilities From What to How
University of Toronto（多伦多大学）；Drexel University（德雷塞尔大学）；University of Siegen（锡根大学）；Westphal College of Media Arts and Design（媒体艺术与设计西弗尔学院）；Digital Media Department（数字媒体系）；Information Systems（信息系统）
Institute for Language, Cognition and Computation, School of Informatics, University of Edinburgh（爱丁堡大学信息学院语言、认知与计算研究所）
Like humans, document summarization models can interpret a document's contents in a number of ways. Unfortunately, the neural models of today are largely black boxes that provide little explanation of how or why they generated a summary in the way they did. Therefore, to begin prying open the black box and to inject a level of control into the substance of the final summary, we developed a novel select-and-generate framework that focuses on explainability. By revealing the latent centrality and interactions between sentences, along with scores for sentence novelty and relevance, users are give…
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
York University（约克大学）；Nanyang Technological University（南洋理工大学）；Dialpad Canada Inc.（Dialpad加拿大公司）；Royal Bank of Canada（加拿大皇家银行）；Salesforce Research（赛富时研究院）
Leipzig University（莱比锡大学）；University of Southern Denmark（南丹麦大学）
This paper introduces Summary Explorer, a new tool to support the manual inspection of text summarization systems by compiling the outputs of 55~state-of-the-art single document summarization approaches on three benchmark datasets, and visually exploring them during a qualitative assessment. The underlying design of the tool considers three well-known summary quality criteria (coverage, faithfulness, and position bias), encapsulated in a guided assessment based on tailored visualizations. The tool complements existing approaches for locally debugging summarization models and improves upon them…
MALSIGHT: Exploring Malicious Source Code and Benign Pseudocode for Iterative Binary Malware Summarization
National Engineering Research Center for Mobile Network Technologies（移动网络技术国家工程研究中心）；Beijing University of Posts and Telecommunications（北京邮电大学）；Zhongguancun Laboratory（中关村实验室）
