arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2311.07884cs.CL
收藏

面向多元视角的公平抽象摘要

Fair Abstractive Summarization of Diverse Perspectives

Penn State University(宾夕法尼亚州立大学);Yale University(耶鲁大学);Salesforce Research(Salesforce研究院);Texas A&M University(德克萨斯农工大学);University of Southern California(南加利福尼亚大学);Columbia University(哥伦比亚大学)

Yusen Zhang, Nan Zhang, Yixin Liu, Alexander Fabbri, Junru Liu, Ryo Kamoi, Xiaoxin Lu, Caiming Xiong, Jieyu Zhao, Dragomir Radev, Kathleen McKeown, Rui Zhang

AI总结:

本文针对用户生成数据中的公平抽象摘要问题,定义公平性并提出四个无参考自动指标,评估九个LLMs在六个数据集上的表现,分析不公平因素并提出三种缓解方法。

摘要:

来自不同社会和人口群体的人们在产品评论、医疗保健、法律和政治等广泛话题上表达着多元视角和相互冲突的意见。公平的摘要应当全面覆盖多元视角,而不应低估某些群体。然而,当前摘要指标和大语言模型(LLMs)评估方面的工作尚未探索公平抽象摘要。在本文中,我们系统地研究了面向用户生成数据的公平抽象摘要。我们首先将抽象摘要中的公平性正式定义为不低估任何人群的视角,并通过测量目标摘要与源文本视角之间的差异,提出了四个无参考自动指标。我们在从社交媒体、在线评论和录音转录中收集的六个数据集上评估了九个LLMs,包括三个GPT模型、四个LLaMA模型、PaLM 2和Claude。实验表明,无论是模型生成的摘要还是人工撰写的参考摘要,都存在公平性较低的问题。我们对影响公平性的常见因素进行了全面分析,并提出了三种简单但有效的方法来缓解不公平摘要问题。我们的数据集和代码可在 https://github.com/psunlpgroup/FairSumm 获取。

英文摘要:

People from different social and demographic groups express diverse perspectives and conflicting opinions on a broad set of topics such as product reviews, healthcare, law, and politics. A fair summary should provide a comprehensive coverage of diverse perspectives without underrepresenting certain groups. However, current work in summarization metrics and Large Language Models (LLMs) evaluation has not explored fair abstractive summarization. In this paper, we systematically investigate fair abstractive summarization for user-generated data. We first formally define fairness in abstractive summarization as not underrepresenting perspectives of any groups of people, and we propose four reference-free automatic metrics by measuring the differences between target and source perspectives. We evaluate nine LLMs, including three GPT models, four LLaMA models, PaLM 2, and Claude, on six datasets collected from social media, online reviews, and recorded transcripts. Experiments show that both the model-generated and the human-written reference summaries suffer from low fairness. We conduct a comprehensive analysis of the common factors influencing fairness and propose three simple but effective methods to alleviate unfair summarization. Our dataset and code are available at https://github.com/psunlpgroup/FairSumm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑