arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.05048cs.CLcs.AIcs.DBcs.IRcs.LG
收藏

利用大语言模型在上市平台实现自然语言搜索

Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms

University of California, Berkeley(加州大学伯克利分校);Zoominfo Technologies Inc.(Zoominfo科技公司)

Jesse Yao, Saurav Acharya, Priyaranjan Parida, Srinivas Attipalli, Ali Dasdan

AI总结:

本文针对GTM平台搜索繁琐问题,提出利用LLM将自然语言转为搜索查询的方法,通过实体提取和提示工程,微调Llama-3-8B,实验显示闭源模型平均准确率达97%,验证了方案有效性。

摘要:

企业搜索要求用户具备关于查询、配置和元数据的复杂知识,这使得他们难以按需访问信息。大多数上市(GTM)平台采用高级搜索,这是一种允许用户通过类别或关键字按各种字段筛选查询的界面,然而,历史上这已被证明极为繁琐,因为用户面对的是看似数百个选项、字段和按钮。因此,长期以来,使用自然语言进行查询一直是理想之选,这一概念因大语言模型(LLMs)而进一步得到增强。\n在本文中,我们为Zoominfo面向销售人员的产品实现并评估了一种解决方案,该方案以自然语言提示LLM,通过实体提取生成搜索字段,进而转换为搜索查询。中间搜索字段为每次查询提供了诸多优势,包括消除语法错误、简化真实标签,以及为LLM提供直观的解读格式。\n我们将此流程与许多先进的提示工程策略相结合,包括复杂的系统消息、少样本提示、思维链(CoT)推理和执行优化。此外,我们手动为500多个自然语言查询创建了真实标签,从而能够对Llama-3-8B-Instruct进行监督微调,并引入复杂的数值指标。\n通过精确、Jaccard、余弦和语义相似度对单个搜索实体进行了全面实验,涉及闭源、开源和微调LLM模型,以证明我们方法的有效性。总体而言,最准确的闭源模型每次查询的平均准确率为97%,仅有一个字段低于90%,微调模型也观察到了类似的结果。

英文摘要:

Enterprise searches require users to have complex knowledge of queries, configurations, and metadata, rendering it difficult for them to access information as needed. Most go-to-market (GTM) platforms utilize advanced search, an interface that enables users to filter queries by various fields using categories or keywords, which, historically, however, has proven to be exceedingly cumbersome, as users are faced with seemingly hundreds of options, fields, and buttons. Consequently, querying with natural language has long been ideal, a notion further empowered by Large Language Models (LLMs). In this paper, we implement and evaluate a solution for the Zoominfo product for sellers, which prompts the LLM with natural language, producing search fields through entity extraction that are then converted into a search query. The intermediary search fields offer numerous advantages for each query, including the elimination of syntax errors, simpler ground truths, and an intuitive format for the LLM to interpret. We paired this pipeline with many advanced prompt engineering strategies, featuring an intricate system message, few-shot prompting, chain-of-thought (CoT) reasoning, and execution refinement. Furthermore, we manually created the ground truth for 500+ natural language queries, enabling the supervised fine-tuning of Llama-3-8B-Instruct and the introduction of sophisticated numerical metrics. Comprehensive experiments with closed, open source, and fine-tuned LLM models were conducted through exact, Jaccard, cosine, and semantic similarity on individual search entities to demonstrate the efficacy of our approach. Overall, the most accurate closed model had an average accuracy of 97% per query, with only one field performing under 90%, with comparable results observed from the fine-tuned models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑