arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.20745cs.LGcs.AI
收藏

Shopping MMLU:面向大语言模型的大规模多任务在线购物基准

Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models

HKUST(香港科技大学);University of Notre Dame(圣母大学)
Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju X… 展开作者

Yilun Jin, Zheng Li, Chenwei Zhang, Tianyu Cao, Yifan Gao, Pratik Jayarao, Mao Li, Xin Liu, Ritesh Sarkhel, Xianfeng Tang, Haodong Wang, Zhengyang Wang, Wenju Xu, Jingfeng Yang, Qingyu Yin, Xian Li, Priyanka Nigam, Yi Xu, Kai Chen, Qiang Yang, Meng Jiang, Bing Yin

AI总结:

针对现有在线购物基准难以覆盖场景全复杂性的问题,研究提出基于真实亚马逊数据的多任务基准Shopping MMLU,含57项任务覆盖4类购物能力,评测20余个LLM并据此举办KDD Cup 2024竞赛,为构建通用LLM购物助手提供参考。

摘要:

在线购物是一个复杂的多任务少样本学习问题,涉及广泛且不断演变的实体、关系和任务。然而,现有模型和基准通常针对特定任务定制,不足以捕捉在线购物的全部复杂性。大语言模型(LLMs)凭借其多任务和少样本学习能力,有望通过减少特定任务的工程工作量、为用户提供交互式对话,深刻变革在线购物领域。尽管具备上述潜力,LLMs在在线购物场景中仍面临独特挑战,例如领域特定概念、隐性知识以及异构用户行为。受上述潜力与挑战的启发,我们提出了Shopping MMLU——一个基于真实亚马逊数据构建的多样化多任务在线购物基准。Shopping MMLU包含57项任务,覆盖4大核心购物能力:概念理解、知识推理、用户行为对齐以及多语言能力,因此能够全面评估LLMs作为通用购物助手的能力。借助Shopping MMLU,我们对20余个现有LLM进行了基准测试,揭示了关于构建多功能LLM购物助手的实践方法与未来前景的宝贵洞见。Shopping MMLU可通过https://github.com/KL4805/ShoppingMMLU公开访问。此外,我们基于Shopping MMLU在KDD Cup 2024中举办了竞赛,吸引了500余支队伍参赛。获胜方案及相关研讨会信息可在我们的网站https://amazon-kddcup24.github.io/获取。

英文摘要:

Online shopping is a complex multi-task, few-shot learning problem with a wide and evolving range of entities, relations, and tasks. However, existing models and benchmarks are commonly tailored to specific tasks, falling short of capturing the full complexity of online shopping. Large Language Models (LLMs), with their multi-task and few-shot learning abilities, have the potential to profoundly transform online shopping by alleviating task-specific engineering efforts and by providing users with interactive conversations. Despite the potential, LLMs face unique challenges in online shopping, such as domain-specific concepts, implicit knowledge, and heterogeneous user behaviors. Motivated by the potential and challenges, we propose Shopping MMLU, a diverse multi-task online shopping benchmark derived from real-world Amazon data. Shopping MMLU consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality, and can thus comprehensively evaluate the abilities of LLMs as general shop assistants. With Shopping MMLU, we benchmark over 20 existing LLMs and uncover valuable insights about practices and prospects of building versatile LLM-based shop assistants. Shopping MMLU can be publicly accessed at https://github.com/KL4805/ShoppingMMLU. In addition, with Shopping MMLU, we host a competition in KDD Cup 2024 with over 500 participating teams. The winning solutions and the associated workshop can be accessed at our website https://amazon-kddcup24.github.io/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑