探索生成式检索中的训练与推理缩放定律
Exploring Training and Inference Scaling Laws in Generative Retrieval
AI总结：
本文系统研究生成式检索中模型规模、训练数据量与推理计算的缩放定律，并提出连续评估指标，验证三者互补提升性能及 LLaMA 相对 T5 的优势。
摘要：
生成式检索将检索重新表述为一项自回归生成任务，其中大型语言模型（LLM）直接根据查询生成目标文档。作为一种新颖范式，支撑其性能和可扩展性的机制在很大程度上仍未得到探索。我们系统地研究生成式检索中的训练和推理缩放定律，考察模型规模、训练数据规模以及推理时计算如何共同影响性能。我们提出一种受对比熵和生成损失启发的新型评估指标，提供连续的性能信号，从而能够在多种生成式检索方法之间进行稳健比较。实验表明，基于 n-gram 的方法与训练和推理缩放定律高度吻合。我们发现，增大模型规模、扩大训练数据规模以及增加推理时计算都有助于提升性能，凸显了这些因素在增强生成式检索方面的互补作用。在这些设置中，LLaMA 模型始终优于 T5 模型，表明较大的仅解码器模型在生成式检索中具有特定优势。我们的研究结果强调，模型规模、数据可用性和推理计算相互作用，可释放生成式检索的全部潜力，并为未来系统的设计和优化提供新的见解。
英文摘要：
Generative retrieval reformulates retrieval as an autoregressive generation task, where large language models (LLMs) generate target documents directly from a query. As a novel paradigm, the mechanisms that underpin its performance and scalability remain largely unexplored. We systematically investigate training and inference scaling laws in generative retrieval, exploring how model size, training data scale, and inference-time compute jointly influence performance. We propose a novel evaluation metric inspired by contrastive entropy and generation loss, providing a continuous performance signal that enables robust comparisons across diverse generative retrieval methods. Our experiments show that n-gram-based methods align strongly with training and inference scaling laws. We find that increasing model size, training data scale, and inference-time compute all contribute to improved performance, highlighting the complementary roles of these factors in enhancing generative retrieval. Across these settings, LLaMA models consistently outperform T5 models, suggesting a particular advantage for larger decoder-only models in generative retrieval. Our findings underscore that model sizes, data availability, and inference computation interact to unlock the full potential of generative retrieval, offering new insights for designing and optimizing future systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
