从合成到原生：物流客户服务中多语言意图分类的基准测试
From Synthetic to Native: Benchmarking Multilingual Intent Classification in Logistics Customer Service
AI总结：
本文提出一个基于真实物流客户服务日志的多语言意图分类基准，包含3万条去标识用户查询，涵盖英语、西班牙语、阿拉伯语及零样本语言，通过对比合成与真实数据测试，揭示现有基准对真实场景的过度乐观估计问题。
摘要：
多语言意图分类是全球物流平台客户服务平台的核心，其中模型必须处理跨语言和层级标签空间的嘈杂用户查询。然而，大多数现有多语言基准依赖机器翻译文本，通常比原生客户请求更干净、标准化，因此可能高估现实世界中的鲁棒性。我们提出一个公开的基准，由真实物流客户服务日志构建，包含约3万条去标识、独立用户查询，通过过滤、LLM辅助质量控制和人工验证从60万条历史记录中筛选而来，组织成两级分类法，包含13个父意图和17个叶意图。英语、西班牙语和阿拉伯语作为可见语言，而印尼语、中文和额外的测试-only语言支持零样本评估。为了直接衡量合成与真实评估之间的差距，我们提供配对的原生和机器翻译测试集，并在平铺和分层协议下基准测试多语言编码器、嵌入模型和小型语言模型。结果表明，翻译测试集在嘈杂的原生查询上显著高估性能，尤其是长尾意图和跨语言迁移，凸显了对更现实多语言意图基准的需求。
英文摘要：
Multilingual intent classification is central to customer-service systems on global logistics platforms, where models must process noisy user queries across languages and hierarchical label spaces. Yet most existing multilingual benchmarks rely on machine-translated text, which is typically cleaner and more standardized than native customer requests and can therefore overestimate real-world robustness. We present a public benchmark for hierarchical multilingual intent classification constructed from real logistics customer-service logs. The dataset contains approximately 30K de-identified, stand-alone user queries curated from 600K historical records through filtering, LLM-assisted quality control, and human verification, and is organized into a two-level taxonomy with 13 parent and 17 leaf intents. English, Spanish, and Arabic are included as seen languages, while Indonesian, Chinese, and additional test-only languages support zero-shot evaluation. To directly measure the gap between synthetic and real evaluation, we provide paired native and machine-translated test sets and benchmark multilingual encoders, embedding models, and small language models under flat and hierarchical protocols. Results show that translated test sets substantially overestimate performance on noisy native queries, especially for long-tail intents and cross-lingual transfer, underscoring the need for more realistic multilingual intent benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
