arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.18446cs.CL
收藏

探索低资源抽取式问答场景下语言模型的泛化能力

Exploring Language Model Generalization in Low-Resource Extractive QA

Pennsylvania State University(宾夕法尼亚州立大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Indian Institute of Technology (IIT) Bhubaneswar(印度理工学院布巴内斯瓦尔分校)

Saptarshi Sengupta, Wenpeng Yin, Preslav Nakov, Shreya Ghosh, Suhang Wang

AI总结:

该研究针对零样本跨领域抽取式问答的LLM泛化问题开展实验,揭示了封闭领域任务难点、模型预处理缺陷、参数缩放局限等结论,为LLM优化提供方向。

摘要:

本文研究领域漂移场景下大语言模型(LLM)的抽取式问答(EQA)问题,即LLM能否在无额外领域内训练的零样本设置下,泛化到医学、法律等需要特定知识的领域?为此,我们设计了一系列实验,从实证角度解释性能差距。研究发现:(a) LLM难以满足封闭领域的数据集要求,例如检索长答案片段;(b) 部分LLM尽管整体表现强劲,但在满足区分单词领域特定含义这类基本要求上存在缺陷,我们将该问题与预处理决策相关联;(c) 扩大模型参数规模对跨领域泛化并非始终有效;(d) 封闭领域数据集与开放领域EQA数据集在量化特征上差异显著,现有LLM难以应对。这些发现为改进现有LLM指出了重要方向。

英文摘要:

In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize to domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to explain the performance gap empirically. Our findings suggest that: (a) LLMs struggle with dataset demands of closed domains such as retrieving long answer spans; (b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; (c) Scaling model parameters is not always effective for cross domain generalization; and (d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑