首页/论文/2002.09836

arXiv 2002.098362020-02cs.CL

Fill in the BLANC: Human-free quality estimation of document summaries

Primer Technologies Inc.（Primer科技公司）

Oleg Vasilyev, Vedant Dharnidharka, John Bohannon

备注：: 10 pages, 9 figures, 3 tables. In: Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP, Nov. 2020) p.11-20, ACL
发表信息：: Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems (Nov.2020) 11-20

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

英文摘要相关论文

英文摘要：

We present BLANC, a new approach to the automatic estimation of document summary quality. Our goal is to measure the functional performance of a summary with an objective, reproducible, and fully automated method. Our approach achieves this by measuring the performance boost gained by a pre-trained language model with access to a document summary while carrying out its language understanding task on the document's text. We present evidence that BLANC scores have as good correlation with human evaluations as do the ROUGE family of summary quality measurements. And unlike ROUGE, the BLANC method does not require human-written reference summaries, allowing for fully human-free summary quality estimation.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

阅读原文

阅读 PDF ↗

arXiv 页面 ↗HTML 原文 ↗

本文内容

英文摘要相关论文

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

继续探索

相关论文

Sensitivity of BLANC to human-scored qualities of text summaries

Primer Technologies Inc.

We explore the sensitivity of a document summary quality estimator, BLANC, to human assessment of qualities for the same summaries. In our human evaluations, we distinguish five summary qualities, defined by how fluent, understandable, informative, compact, and factually correct the summary is. We make the case for optimal BLANC parameters, at which the BLANC sensitivity to almost all of summary qualities is about as good as the sensitivity of a human annotator.

2020-10-13cs.CL

Towards Human-Free Automatic Quality Evaluation of German Summarization

Technische Universität Berlin（柏林工业大学）；Quality and Usability Lab（质量与可用性实验室）；Primer Technologies Inc.（普里默技术公司）

Evaluating large summarization corpora using humans has proven to be expensive from both the organizational and the financial perspective. Therefore, many automatic evaluation metrics have been developed to measure the summarization quality in a fast and reproducible way. However, most of the metrics still rely on humans and need gold standard summaries generated by linguistic experts. Since BLANC does not require golden summaries and supposedly can use any underlying language model, we consider its application to the evaluation of summarization in German. This work demonstrates how to adjust …

2021-05-13cs.CL

公平摘要：弥合抽取式摘要中的质量与多样性

Fair Summarization: Bridging Quality and Diversity in Extractive Summaries

Portland State University（波特兰州立大学）

针对多文档摘要中社会群体代表性不公的问题，提出 FairExtract 与 FairGPT，并结合质量和公平性复合指标验证其能在保持摘要质量的同时提升公平性。

2025-03-18cs.AIcs.CL

Is human scoring the best criteria for summary evaluation?

Primer Technologies Inc.（Primer科技公司）

Normally, summary quality measures are compared with quality scores produced by human annotators. A higher correlation with human scores is considered to be a fair indicator of a better measure. We discuss observations that cast doubt on this view. We attempt to show a possibility of an alternative indicator. Given a family of measures, we explore a criterion of selecting the best measure not relying on correlations with human scores. Our observations for the BLANC family of measures suggest that the criterion is universal across very different styles of summaries.

2020-12-29cs.CL

MaLei在MultiClinSUM：使用视角感知迭代自提示与大型语言模型的临床文档摘要

MaLei at MultiClinSUM: Summarisation of Clinical Documents using Perspective-Aware Iterative Self-Prompting with LLMs

University of Manchester, UK（曼彻斯特大学）；Modul University Vienna, Austria（维也纳Modul大学）；Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands（莱顿先进计算机科学研究所（LIACS）、莱顿大学）；Leiden University Medical Center, The Netherlands（莱顿大学医学中心）

本文提出视角感知迭代自提示方法，利用GPT-4和GPT-4o总结临床报告，在3,396份病例上取得高BERT分数，改善医患沟通。

2025-09-09cs.CL

MemoryForge：为类人LLM智能体合成终身记忆

MemoryForge: Synthesize Lifelong Memory for Human-Like LLM Agents

Tencent（腾讯）

MemoryForge是从简短目标画像合成终身记忆的框架，通过记忆条件让冻结LLM动态检索相关记忆，在两类任务实验中使其表现更类人。

2026-08-04cs.AIcs.CLcs.LG

更多相关论文收起更多论文

EDU-level Extractive Summarization with Varying Summary Lengths

University of Manchester（曼彻斯特大学）

Extractive models usually formulate text summarization as extracting fixed top-$k$ salient sentences from the document as a summary. Few works exploited extracting finer-grained Elementary Discourse Unit (EDU) with little analysis and justification for the extractive unit selection. Further, the selection strategy of the fixed top-$k$ salient sentences fits the summarization need poorly, as the number of salient sentences in different documents varies and therefore a common or best $k$ does not exist in reality. To fill these gaps, this paper first conducts the comparison analysis of oracle su…

2023-03-13cs.CL

用于临床摘要事实对齐的合成模仿编辑反馈

Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization

University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；Fudan University（复旦大学）；University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校）

针对临床摘要任务中大语言模型易出现事实幻觉、人工标注反馈成本高的问题，提出用ChatGPT生成专家级编辑反馈的新流水线，验证了其在事实对齐方面的应用潜力。

2023-11-03cs.AIcs.CL

The Summary Loop: Learning to Write Abstractive Summaries Without Examples

UC Berkeley（加州大学伯克利分校）；Bloomberg（彭博社）

This work presents a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. It introduces a novel method that encourages the inclusion of key terms from the original document into the summary: key terms are masked out of the original document and must be filled in by a coverage model using the current generated summary. A novel unsupervised training procedure leverages this coverage model along with a fluency model to generate and score summaries. When tested on popular news summarization datasets, the met…

2021-05-11cs.CLcs.LG

Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization

University of California, Santa Barbara（加州大学圣巴巴拉分校）；Microsoft（微软）；NEC Laboratories America（NEC美国实验室）

Text summarization has been a crucial problem in natural language processing (NLP) for several decades. It aims to condense lengthy documents into shorter versions while retaining the most critical information. Various methods have been proposed for text summarization, including extractive and abstractive summarization. The emergence of large language models (LLMs) like GPT3 and ChatGPT has recently created significant interest in using these models for text summarization tasks. Recent studies \cite{goyal2022news, zhang2023benchmarking} have shown that LLMs-generated news summaries are already…

2023-02-16cs.AIcs.CL

面向文本摘要的交互式编辑

Interactive Editing for Text Summarization

Microsoft（微软）

本文提出 REVISE 框架，通过 encoder-decoder fill-in-the-middle 模型和定制评估指标，支持人类对摘要草稿进行局部、迭代式交互编辑。

2023-06-05cs.AIcs.CL

CodeExp: Explanatory Code Document Generation

University of Toronto（多伦多大学）；Microsoft Research（微软研究院）；Beihang University（北京航空航天大学）

Developing models that can automatically generate detailed code explanation can greatly benefit software maintenance and programming education. However, existing code-to-text generation models often produce only high-level summaries of code that do not capture implementation-level choices essential for these scenarios. To fill in this gap, we propose the code explanation generation task. We first conducted a human study to identify the criteria for high-quality explanatory docstring for code. Based on that, we collected and refined a large-scale code docstring corpus and formulated automatic e…

2022-11-25cs.CLcs.LG