arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.11591cs.CL
收藏

AdaptEval:评估大型语言模型在文本摘要领域自适应上的表现

AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization

Technical University of Munich(慕尼黑工业大学);ETH Zürich(苏黎世联邦理工学院)

Anum Afzal, Ribin Chalumattu, Florian Matthes, Laura Mascarell

AI总结:

本文评估了多种大型语言模型在微调和上下文学习设置下跨领域文本摘要的自适应能力,并提出首个领域自适应评估套件AdaptEval,发现LLM在上下文学习中的表现与参数规模无关。

摘要:

尽管使用大型语言模型(LLM)在抽象摘要任务上取得了进展,但目前缺乏评估其轻松适应不同领域能力的研究。我们在微调和上下文学习设置下,评估了多种LLM在跨领域摘要任务上的领域自适应能力。我们还提出了首个领域自适应评估套件AdaptEval。AdaptEval包含一个领域基准和一组指标,以促进对领域自适应的分析。结果表明,在上下文学习设置中,无论参数规模如何,LLM均表现出相当的性能。

英文摘要:

Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑