arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.10129cs.CLcs.IR
收藏

高效检索优化的多任务学习

Efficient Retrieval Optimized Multi-task Learning

Amazon Alexa(亚马逊Alexa)

Hengxin Fun, Sunil Gandhi, Sujith Ravi

摘要:

最近,在处理知识密集型任务(如开放领域问答(QA))的神经方法方面取得了重大进展。这些进步是通过将大型预训练语言模型与可学习的文档检索相结合来推动的。这些模型中的大多数使用单独的编码器来学习查询表示,检索器的通道表示和下游任务的附加编码器。为每个阶段/任务使用单独的编码器会占用大量内存,并且难以扩展到大量任务。在本文中,我们提出了一种新的检索优化多任务(ROM)框架,用于联合训练自监督任务,知识检索和提取问答。我们的ROM方法提出了一个统一的和可推广的框架,可以有效地扩展到多个任务,不同级别的监督和优化选择,如不同的学习时间表,而不改变模型架构。它还提供了在不改变系统架构的情况下改变编码器的灵活性。使用我们的框架,我们实现了可比或更好的性能比最近的QA方法,同时大大减少了参数的数量。

英文摘要:

Recently, there have been significant advances in neural methods for tackling knowledge-intensive tasks such as open domain question answering (QA). These advances are fueled by combining large pre-trained language models with learnable retrieval of documents. Majority of these models use separate encoders for learning query representation, passage representation for the retriever and an additional encoder for the downstream task. Using separate encoders for each stage/task occupies a lot of memory and makes it difficult to scale to a large number of tasks. In this paper, we propose a novel Retrieval Optimized Multi-task (ROM) framework for jointly training self-supervised tasks, knowledge retrieval, and extractive question answering. Our ROM approach presents a unified and generalizable framework that enables scaling efficiently to multiple tasks, varying levels of supervision, and optimization choices such as different learning schedules without changing the model architecture. It also provides the flexibility of changing the encoders without changing the architecture of the system. Using our framework, we achieve comparable or better performance than recent methods on QA, while drastically reducing the number of parameters.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑