大型语言模型在商品搜索相关性判断中的应用
Large Language Models for Relevance Judgment in Product Search
AI总结：
本文利用大型语言模型自动化商品搜索中查询-商品对的相关性判断，通过微调十亿参数级模型并优化超参数，实现了与人工评估相当的相关性标注效果。
摘要：
检索和重排序后的商品与搜索查询之间的高相关性是成功商品搜索的基石，然而衡量商品与查询之间的相关性是商品信息检索中最具挑战性的任务之一，商品搜索的质量高度依赖于可用相关性标注数据的精度和规模。在本文中，我们提出了一系列利用大型语言模型（LLMs）自动化大规模查询-商品对（QIPs）相关性判断的技术。利用一个包含数百万查询-商品对的独特数据集，该数据集由人工评估者标注，我们测试并优化了微调十亿参数级LLMs的超参数，包括使用和不使用低秩适应（LoRA）的情况，以及LLM微调中商品属性拼接和提示的各种模式，并考虑了商品属性包含在相关性预测质量中的权衡。我们展示了相对于前几代LLMs的基线和现成模型的显著改进，达到了与人工相关性评估者相当的相关性标注水平。我们的研究结果对商品搜索中相关性判断自动化这一日益增长的领域具有直接意义。
英文摘要：
High relevance of retrieved and re-ranked items to the search query is the cornerstone of successful product search, yet measuring relevance of items to queries is one of the most challenging tasks in product information retrieval, and quality of product search is highly influenced by the precision and scale of available relevance-labelled data. In this paper, we present an array of techniques for leveraging Large Language Models (LLMs) for automating the relevance judgment of query-item pairs (QIPs) at scale. Using a unique dataset of multi-million QIPs, annotated by human evaluators, we test and optimize hyper parameters for finetuning billion-parameter LLMs with and without Low Rank Adaption (LoRA), as well as various modes of item attribute concatenation and prompting in LLM finetuning, and consider trade offs in item attribute inclusion for quality of relevance predictions. We demonstrate considerable improvement over baselines of prior generations of LLMs, as well as off-the-shelf models, towards relevance annotations on par with the human relevance evaluators. Our findings have immediate implications for the growing field of relevance judgment automation in product search.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
