文本摘要的实证比较：大语言模型的多维评估
An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Models
AI总结：
本研究通过多维框架在七个数据集上评估17个大语言模型的摘要性能，揭示了不同模型在事实准确性、类人质量和效率方面的差异及事实一致性与感知质量之间的权衡，并为不同应用场景提供模型选择建议。
摘要：
文本摘要对于缓解新闻、医学和商业等领域的信息过载至关重要。本研究使用一种新颖的多维框架，评估了17个大语言模型（OpenAI、Google、Anthropic、开源模型）的摘要性能。我们在七个不同的数据集（BigPatent、BillSum、CNN/DailyMail、PubMed、SAMSum、WikiHow、XSum）上，以三种输出长度（50、100、150个token）对模型进行了评估，使用的指标涵盖事实一致性、语义相似度、词汇重叠度和类人质量，同时考虑了效率因素。我们的发现揭示了显著的性能差异，特定模型在事实准确性（deepseek-v3）、类人质量（claude-3-5-sonnet）以及处理效率/成本效益（gemini-1.5-flash、gemini-2.0-flash）方面表现突出。性能因数据集不同而差异巨大，模型在技术领域表现吃力，但在对话内容上表现良好。我们发现事实一致性（在50个token时最佳）与感知质量（在150个token时最佳）之间存在关键张力。我们的分析为不同使用场景提供了基于证据的建议，从需要事实准确性的高风险应用到需要高效处理的资源受限环境。这种综合方法通过将质量指标与操作考量相结合，纳入准确性、效率和成本效益之间的权衡，增强了评估方法论，以指导针对特定应用的模型选择。
英文摘要：
Text summarization is crucial for mitigating information overload across domains like journalism, medicine, and business. This research evaluates summarization performance across 17 large language models (OpenAI, Google, Anthropic, open-source) using a novel multi-dimensional framework. We assessed models on seven diverse datasets (BigPatent, BillSum, CNN/DailyMail, PubMed, SAMSum, WikiHow, XSum) at three output lengths (50, 100, 150 tokens) using metrics for factual consistency, semantic similarity, lexical overlap, and human-like quality, while also considering efficiency factors. Our findings reveal significant performance differences, with specific models excelling in factual accuracy (deepseek-v3), human-like quality (claude-3-5-sonnet), and processing efficiency/cost-effectiveness (gemini-1.5-flash, gemini-2.0-flash). Performance varies dramatically by dataset, with models struggling on technical domains but performing well on conversational content. We identified a critical tension between factual consistency (best at 50 tokens) and perceived quality (best at 150 tokens). Our analysis provides evidence-based recommendations for different use cases, from high-stakes applications requiring factual accuracy to resource-constrained environments needing efficient processing. This comprehensive approach enhances evaluation methodology by integrating quality metrics with operational considerations, incorporating trade-offs between accuracy, efficiency, and cost-effectiveness to guide model selection for specific applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
