arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.04040cs.CL
收藏

ProQA:基于结构化推理的统一问答预训练

ProQA: Structural Prompt-based Pre-training for Unified Question Answering

Sun Yat-sen University(中山大学);Tsinghua University(清华大学);Langboat Technology(澜舟科技);Microsoft Research Asia(微软亚洲研究院);Chinese University of Hong Kong(香港中文大学)

Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, Nan Duan

摘要:

问答(QA)是自然语言处理中的一个长期挑战。现有的问答工作大多集中在特定的问题类型,知识领域,或推理技能。QA研究的专业性阻碍了系统对任务之间的共性建模和更广泛应用的泛化。为了解决这个问题,我们提出了ProQA,一个统一的QA范式,通过一个单一的模型来解决各种任务。ProQA以统一的结构化提示为桥梁,通过基于结构化提示的预训练提高以QA为中心的能力。通过一个结构化设计的基于知识的输入模式,ProQA同时模型的知识概括的所有QA任务,同时保持知识定制的每一个特定的QA任务。此外,ProQA是预先训练的结构化的格式化的大规模合成语料库,这赋予了通常所需的QA能力的模型。在11个QA基准测试上的实验结果表明,ProQA在全数据微调、少样本学习和zero-shot测试场景下都能持续提高性能。此外,ProQA利用结构提示的优势,在持续学习和迁移学习方面表现出较强的能力。

英文摘要:

Question Answering (QA) is a longstanding challenge in natural language processing. Existing QA works mostly focus on specific question types, knowledge domains, or reasoning skills. The specialty in QA research hinders systems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt-based pre-training. Through a structurally designed prompt-based input schema, ProQA concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. Furthermore, ProQA is pre-trained with structural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimental results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑