EComStage:面向电商大语言模型的分阶段和方向特定基准测试
EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commerce
AI总结:
EComStage提出一个分阶段和方向特定的电商大语言模型基准测试,评估LLM在感知、规划和行动阶段的表现,揭示不同规模和类型的模型在电商场景中的优势与不足。
摘要:
基于大语言模型(LLM)的智能体正越来越多地应用于电商领域,以协助客户服务完成产品咨询、推荐和订单管理等任务。现有基准测试主要评估这些智能体是否能完成最终任务,忽略了对有效决策至关重要的中间推理阶段。为解决这一差距,我们提出了EComStage,一个统一的基准测试,用于评估具有智能体能力的LLM在全面的分阶段推理过程中的表现:感知(理解用户意图)、规划(制定行动计划)和行动(执行决策)。EComStage通过七个代表性的任务对LLM进行评估,涵盖多样化的电商场景,所有样本均有人工标注并经过质量检查。不同于以往仅关注客户导向互动的基准测试,EComStage还评估了商家导向的场景,包括促销管理、内容审核和运营支持等与实际应用相关的内容。我们评估了超过30种LLM,从10亿到超过2000亿参数,包括开源模型和闭源API,揭示了阶段/方向特定的优势和劣势。我们的结果为设计和优化实际电商环境中的LLM智能体提供了精细的、可操作的见解。
英文摘要:
Large Language Model (LLM)-based agents are increasingly deployed in e-commerce applications to assist customer services in tasks such as product inquiries, recommendations, and order management. Existing benchmarks primarily evaluate whether these agents successfully complete the final task, overlooking the intermediate reasoning stages that are crucial for effective decision-making. To address this gap, we propose EComStage, a unified benchmark for evaluating agent-capable LLMs across the comprehensive stage-wise reasoning process: Perception (understanding user intent), Planning (formulating an action plan), and Action (executing the decision). EComStage evaluates LLMs through seven separate representative tasks spanning diverse e-commerce scenarios, with all samples human-annotated and quality-checked. Unlike prior benchmarks that focus only on customer-oriented interactions, EComStage also evaluates merchant-oriented scenarios, including promotion management, content review, and operational support relevant to real-world applications. We evaluate a wide range of over 30 LLMs, spanning from 1B to over 200B parameters, including open-source models and closed-source APIs, revealing stage/orientation-specific strengths and weaknesses. Our results provide fine-grained, actionable insights for designing and optimizing LLM-based agents in real-world e-commerce settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。