Dialogues are a predominant mode of communication for humans, and it is immensely helpful to have automatically generated summaries of them (e.g., to revise key points discussed in a meeting, to review conversations between customer agents and product users). Prior works on dialogue summary evaluation largely ignore the complexities specific to this task: (i) shift in structure, from multiple speakers discussing information in a scattered fashion across several turns, to a summary's sentences, and (ii) shift in narration viewpoint, from speakers' first/second-person narration, standardized third-person narration in the summary. In this work, we introduce our framework DIALSUMMER to address the above. We propose DIAL-SUMMER's taxonomy of errors to comprehensively evaluate dialogue summaries at two hierarchical levels: DIALOGUE-LEVEL that focuses on the broader speakers/turns, and WITHIN-TURN-LEVEL that focuses on the information talked about inside a turn. We then present DIAL-SUMMER's dataset composed of dialogue summaries manually annotated with our taxonomy's fine-grained errors. We conduct empirical analyses of these annotated errors, and observe interesting trends (e.g., turns occurring in middle of the dialogue are the most frequently missed in the summary, extrinsic hallucinations largely occur at the end of the summary). We also conduct experiments on LLM-Judges' capability at detecting these errors, through which we demonstrate the challenging nature of our dataset, the robustness of our taxonomy, and the need for future work in this field to enhance LLMs' performance in the same. Code and inference dataset coming soon.
Baichuan2-Sum: Instruction Finetune Baichuan2-7B Model for Dialogue Summarization
Viterbi School of Engineering University of Southern California Los Angeles, United States;School of Computing National University of Singapore Singapore, Singapore;College of Computing;Information Science Cornell University New York, United States.;Computer Science College Taiyuan Normal University Guangzhou, China
Reasoning or Not? A Comprehensive Evaluation of Reasoning LLMs for Dialogue Summarization
Faculty of Applied Sciences, Macao Polytechnic University(澳门理工大学应用科学学院);Computer Science and Communication Research Center, School of Technology and Management, Polytechnic of Leiria(莱里亚理工学院技术与管理学院计算机科学与通信研究中心);Institute of International Language Services Studies, Macau Millennium College(澳门 millennium 学院国际语言服务研究学院);CISUC, Department of Informatics Engineering, University of Coimbra(科英布拉大学信息工程系 CISUC)
Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models
the Department of Computer Science and Engineering, Hong Kong University of Science and Technology(香港科技大学计算机科学与工程系);the institute of information engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);the College of Computer, National University of Defense Technology(国防科技大学计算机学院);the University of Sydney(悉尼大学)
PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generation
Anhui Robot Technology Standard Innovation Base, School of Computer Science and Technology, University of Science and Technology of China(中国科学技术大学计算机科学与技术学院安徽机器人技术标准创新基地)
CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization
National Laboratory of Pattern Recognition, Institute of Automation, CAS(中国科学院自动化研究所模式识别国家重点实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Fanyu AI Laboratory, Zhongke Fanyu Technology Co., Ltd(中科凡语科技有限公司凡语AI实验室)
A Video-grounded Dialogue Dataset and Metric for Event-driven Activities
National Institute of Advanced Industrial Science and Technology (AIST)(日本国立先进工业科学技术研究所(AIST));Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所)
What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization
State Key Laboratory of Complex & Critical Software Environment(复杂与关键软件环境国家重点实验室);Beihang University(北京航空航天大学);State Key Laboratory of Multimodal Artificial Intelligence Systems(多模态人工智能系统国家重点实验室);Institute of Automation, CAS(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);ByteDance Inc.(字节跳动公司)
QUARTZ : QA-based Unsupervised Abstractive Refinement for Task-oriented Dialogue Summarization
Université de Lorraine, CNRS, Inria, LORIA(法国洛林大学、法国国家科学研究中心、法国国家信息与自动化研究所、LORIA研究所);Université Sorbonne Paris Nord, CNRS, Laboratoire d’Informatique de Paris Nord, LIPN(法国巴黎北大学、法国国家科学研究中心、巴黎北信息学实验室、LIPN研究所);Inserm, Centre de Recherche des Cordeliers, Université Paris Cité, Sorbonne Université(法国国家医学研究院、cordeliers研究中心、巴黎城市大学、索邦大学);HeKA, Inria Paris(HeKA、法国国家信息与自动化研究所巴黎分所);Department of Medical Informatics, Assistance Publique Hôpitaux de Paris, Georges Pompidou European Hospital(医学信息学系、巴黎公共医院、乔治·蓬皮杜欧洲医院)
本文提出无监督框架 QUARTZ,利用 LLM 生成任务问答并据此筛选高质量对话摘要,再微调模型,在零样本设置下达到接近完全监督 SotA 的效果。
Reading Between the Lines: The One-Sided Conversation Problem
Paul G. Allen School of Computer Science & Engineering, University of Washington(保罗·G·阿伦计算机科学与工程学院,华盛顿大学);Allen Institute for Artificial Intelligence(阿伦人工智能研究所);Hearvana AI
Systematic Exploration of Dialogue Summarization Approaches for Reproducibility, Comparative Assessment, and Methodological Innovations for Advancing Natural Language Processing in Abstractive Summarization
SRM Institute of Science and Technology(SRM科学技术学院);SRM University, AP(SRM大学(安得拉邦))