arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.12356cs.IR
收藏

内存约束下稠密检索器的一种梯度累积方法

A Gradient Accumulation Method for Dense Retriever under Memory Constraint

Jaehee Kim, Yukyung Lee, Pilsung Kang

AI总结:

本文提出ContAccum方法,通过双记忆库结构实现内存受限条件下稠密检索器的稳定高效训练,在五个检索数据集上超越现有内存减少方法及高资源场景。

摘要:

InfoNCE损失常用于训练信息检索任务中的稠密检索器。众所周知,大批量对于使用InfoNCE损失进行稳定且有效的训练至关重要,而这需要大量的硬件资源。由于对大批量的依赖,稠密检索器在应用和研究方面存在瓶颈。近年来,通过分解前向与后向过程或使用记忆库来减少内存的方法已被广泛采用,以解决硬件瓶颈。然而,现有方法仍然存在训练缓慢且不稳定的问题。为了解决这些问题,我们提出了Contrastive Accumulation(ContAccum),这是一种稳定且高效的内存减少方法,用于稠密检索器训练,它采用双记忆库结构来利用先前生成的查询表示和段落表示。在五个广泛使用的信息检索数据集上的实验表明,ContAccum不仅能超越现有的内存减少方法,而且还能超越高资源场景。此外,理论分析和实验结果证实,与当前的记忆库利用方法相比,ContAccum能提供更稳定的双编码器训练。

英文摘要:

InfoNCE loss is commonly used to train dense retriever in information retrieval tasks. It is well known that a large batch is essential to stable and effective training with InfoNCE loss, which requires significant hardware resources. Due to the dependency of large batch, dense retriever has bottleneck of application and research. Recently, memory reduction methods have been broadly adopted to resolve the hardware bottleneck by decomposing forward and backward or using a memory bank. However, current methods still suffer from slow and unstable training. To address these issues, we propose Contrastive Accumulation (ContAccum), a stable and efficient memory reduction method for dense retriever trains that uses a dual memory bank structure to leverage previously generated query and passage representations. Experiments on widely used five information retrieval datasets indicate that ContAccum can surpass not only existing memory reduction methods but also high-resource scenario. Moreover, theoretical analysis and experimental results confirm that ContAccum provides more stable dual-encoder training than current memory bank utilization methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑