基于大语言模型的多对多摘要生成实证研究
An Empirical Study of Many-to-Many Summarization with Large Language Models
AI总结：
本研究针对大语言模型的多对多摘要生成（M2MS）能力开展系统实证，构建跨5领域6语言的4.78万样本数据集，测试18个大语言模型的零样本与指令微调效果，发现指令微调可显著提升开源模型性能但可能加剧事实性错误。
摘要：
多对多摘要生成（Many-to-many summarization，M2MS）旨在处理任意语言的文档，并生成同样为任意语言的对应摘要。近年来，大语言模型（Large Language Models，LLMs）展现出强大的多语言能力，使其具备在实际应用中执行M2MS任务的潜力。本研究针对大语言模型的M2MS能力开展了系统的实证研究。具体而言，我们首先基于8个已有的领域特定数据集重新整理了M2MS数据，整理后的数据包含4.78万个样本，覆盖5个领域、6种语言，可用于训练和评估大语言模型。随后，我们以零样本（zero-shot）方式和指令微调（instruction-tuning）方式对18个大语言模型进行了基准测试，同时也对经过微调的传统模型（如mBART）进行了实验以作对比。实验结果显示，零样本大语言模型取得了与微调后传统模型相当的效果；经过指令微调后，开源大语言模型的M2MS能力可得到显著提升，在自动评估指标上优于零样本大语言模型（包括GPT-4）。此外，我们证实这种任务特定的性能提升并不会牺牲大语言模型的通用任务解决能力。然而，人工评估结果显示，大语言模型仍面临事实性问题，且指令微调可能会加剧这一问题。因此，在实际应用中构建大语言模型摘要生成器时，如何控制事实错误成为关键问题，也值得未来研究重点关注。
英文摘要：
Many-to-many summarization (M2MS) aims to process documents in any language and generate the corresponding summaries also in any language. Recently, large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform M2MS in real applications. This work presents a systematic empirical study on LLMs' M2MS ability. Specifically, we first reorganize M2MS data based on eight previous domain-specific datasets. The reorganized data contains 47.8K samples spanning five domains and six languages, which could be used to train and evaluate LLMs. Then, we benchmark 18 LLMs in a zero-shot manner and an instruction-tuning manner. Fine-tuned traditional models (e.g., mBART) are also conducted for comparisons. Our experiments reveal that, zero-shot LLMs achieve competitive results with fine-tuned traditional models. After instruct-tuning, open-source LLMs can significantly improve their M2MS ability, and outperform zero-shot LLMs (including GPT-4) in terms of automatic evaluations. In addition, we demonstrate that this task-specific improvement does not sacrifice the LLMs' general task-solving abilities. However, as revealed by our human evaluation, LLMs still face the factuality issue, and the instruction tuning might intensify the issue. Thus, how to control factual errors becomes the key when building LLM summarizers in real applications, and is worth noting in future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
