arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2502.20196cs.CL
收藏

ChineseEcomQA:面向大型语言模型的可扩展电子商务概念评估基准

ChineseEcomQA: A Scalable E-commerce Concept Evaluation Benchmark for Large Language Models

Taobao & Tmall Group of Alibaba(阿里巴巴淘宝天猫集团)

Haibin Chen, Kangtao Lv, Chengwei Hu, Yanshi Li, Yujin Yuan, Yancheng He, Xingyao Zhang, Langming Liu, Shilei Liu, Wenbo Su, Bo Zheng

AI总结:

提出可扩展的电子商务概念问答基准ChineseEcomQA,通过聚焦基础概念、平衡通用性与专业性,结合LLM、RAG和人工标注构建,以评估LLM领域能力。

摘要:

随着大型语言模型(LLM)在电子商务等领域的应用日益广泛,针对特定领域的概念评估基准对于评估其领域能力至关重要。现有的LLM在复杂的电子商务应用中可能会生成事实性错误的信息。因此,有必要构建一个电子商务概念基准。现有基准面临两个主要挑战:(1)处理任务的异构性和多样性;(2)区分电子商务领域内的通用性与特殊性。为解决这些问题,我们提出了ChineseEcomQA,一个聚焦于基础电子商务概念的可扩展问答基准。ChineseEcomQA建立在三个核心特征之上:聚焦基础概念、电子商务通用性和电子商务专业性。基础概念被设计为适用于各种电子商务任务,从而解决异构性和多样性的挑战。此外,通过仔细平衡通用性与特殊性,ChineseEcomQA有效区分了广泛的电子商务概念,从而能够精确验证领域能力。我们通过一个可扩展的基准构建流程实现这一目标,该流程结合了LLM验证、检索增强生成(RAG)验证和严格的人工标注。基于ChineseEcomQA,我们对主流LLM进行了广泛评估,并提供了一些有价值的见解。我们希望ChineseEcomQA能够指导未来的领域特定评估,并促进LLM在电子商务应用中的更广泛采用。

英文摘要:

With the increasing use of Large Language Models (LLMs) in fields such as e-commerce, domain-specific concept evaluation benchmarks are crucial for assessing their domain capabilities. Existing LLMs may generate factually incorrect information within the complex e-commerce applications. Therefore, it is necessary to build an e-commerce concept benchmark. Existing benchmarks encounter two primary challenges: (1) handle the heterogeneous and diverse nature of tasks, (2) distinguish between generality and specificity within the e-commerce field. To address these problems, we propose \textbf{ChineseEcomQA}, a scalable question-answering benchmark focused on fundamental e-commerce concepts. ChineseEcomQA is built on three core characteristics: \textbf{Focus on Fundamental Concept}, \textbf{E-commerce Generality} and \textbf{E-commerce Expertise}. Fundamental concepts are designed to be applicable across a diverse array of e-commerce tasks, thus addressing the challenge of heterogeneity and diversity. Additionally, by carefully balancing generality and specificity, ChineseEcomQA effectively differentiates between broad e-commerce concepts, allowing for precise validation of domain capabilities. We achieve this through a scalable benchmark construction process that combines LLM validation, Retrieval-Augmented Generation (RAG) validation, and rigorous manual annotation. Based on ChineseEcomQA, we conduct extensive evaluations on mainstream LLMs and provide some valuable insights. We hope that ChineseEcomQA could guide future domain-specific evaluations, and facilitate broader LLM adoption in e-commerce applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑