赢得 Amazon KDD Cup'24
Winning Amazon KDD Cup'24
AI总结:
本文提出 Amazon KDD Cup 2024 在线购物 LLM 挑战赛的总冠军方案,通过自建数据微调 Qwen2-72B、wise-ft、多 LoRA 集成、Logits Processors 及 AWQ/vLLM 推理,在四个赛道均获第一。
摘要:
本文介绍了 Amazon KDD Cup 2024 面向 LLM 的多任务在线购物挑战赛全部 5 项任务的获胜方案。该挑战赛旨在构建一个有用的助手,回答在线购物领域的问题。比赛包含 57 个不同任务,覆盖 5 种不同任务类型(例如多项选择),并横跨 4 个不同赛道(例如多语言)。我们的方案是每个赛道使用一个单一模型。我们在自建训练数据集上微调 Qwen2-72B-Instruct。由于比赛仅发布了 96 个示例问题,我们通过处理多个公开数据集,或使用大型语言模型进行数据增强和合成数据生成,开发了自己的训练数据集。我们应用 wise-ft 来应对分布偏移,并在一个模型中集成多个 LoRA 适配器。我们采用 Logits Processors,将模型在任务上的输出约束在相关 token 上。推理期间使用 AWQ 4-bit Quantization 和 vLLM,以便根据赛道不同,在 20 至 140 分钟的时间限制内完成测试数据集预测。我们的方案在每个独立赛道均获得第一名,并获得 Amazon KDD Cup 2024 总冠军。
英文摘要:
This paper describes the winning solution of all 5 tasks for the Amazon KDD Cup 2024 Multi Task Online Shopping Challenge for LLMs. The challenge was to build a useful assistant, answering questions in the domain of online shopping. The competition contained 57 diverse tasks, covering 5 different task types (e.g. multiple choice) and across 4 different tracks (e.g. multi-lingual). Our solution is a single model per track. We fine-tune Qwen2-72B-Instruct on our own training dataset. As the competition released only 96 example questions, we developed our own training dataset by processing multiple public datasets or using Large Language Models for data augmentation and synthetic data generation. We apply wise-ft to account for distribution shifts and ensemble multiple LoRA adapters in one model. We employed Logits Processors to constrain the model output on relevant tokens for the tasks. AWQ 4-bit Quantization and vLLM are used during inference to predict the test dataset in the time constraints of 20 to 140 minutes depending on the track. Our solution achieved the first place in each individual track and is the first place overall of Amazons KDD Cup 2024.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。