面向规模化定制营销内容生成与评估的大语言模型
LLMs for Customized Marketing Content Generation and Evaluation at Scale
AI总结：
本文提出MarketingFM检索增强系统生成关键词特定广告文案，及AutoEval-Main和AutoEval-Update自动化评估框架，通过离线与在线实验验证了点击率提升和评估一致性提高。
摘要：
站外营销在电子商务中至关重要，它使企业能够通过外部平台触达客户并引导流量至零售网站。然而，目前大多数站外营销内容过于通用、基于模板，且与落地页匹配度差，限制了其有效性。为解决这些局限，我们提出了MarketingFM，一个检索增强系统，整合多种数据源，以最少的人工干预生成关键词特定的广告文案。我们通过离线人工与自动化评估以及大规模在线A/B测试验证了MarketingFM。在一项实验中，关键词聚焦的广告文案优于模板，实现了最高9%的点击率提升、12%的展示量增加和0.38%的每次点击成本降低，展示了广告排名和成本效率方面的提升。尽管有这些收益，对生成广告的人工审核仍然成本高昂。为此，我们提出了AutoEval-Main，一个自动化评估系统，结合基于规则的指标与LLM-as-a-Judge技术，确保与营销原则的一致性。在大规模人工标注实验中，AutoEval-Main与人工审核者的一致性达到89.57%。在此基础上，我们提出了AutoEval-Update，一个成本高效的LLM-人工协作框架，以动态优化评估提示并适应变化的标准，同时最小化人工输入。通过选择性采样代表性广告供人工审核，并使用评论LLM生成一致性报告，AutoEval-Update在减少人工工作量的同时提高了评估一致性。实验表明，评论LLM提出了有意义的改进建议，提高了LLM与人工的一致性。尽管如此，在部署前，人工监督对于设定阈值和验证改进仍然至关重要。
英文摘要：
Offsite marketing is essential in e-commerce, enabling businesses to reach customers through external platforms and drive traffic to retail websites. However, most current offsite marketing content is overly generic, template-based, and poorly aligned with landing pages, limiting its effectiveness. To address these limitations, we propose MarketingFM, a retrieval-augmented system that integrates multiple data sources to generate keyword-specific ad copy with minimal human intervention. We validate MarketingFM via offline human and automated evaluations and large-scale online A/B tests. In one experiment, keyword-focused ad copy outperformed templates, achieving up to 9% higher CTR, 12% more impressions, and 0.38% lower CPC, demonstrating gains in ad ranking and cost efficiency. Despite these gains, human review of generated ads remains costly. To address this, we propose AutoEval-Main, an automated evaluation system that combines rule-based metrics with LLM-as-a-Judge techniques to ensure alignment with marketing principles. In experiments with large-scale human annotations, AutoEval-Main achieved 89.57% agreement with human reviewers. Building on this, we propose AutoEval-Update, a cost-efficient LLM-human collaborative framework to dynamically refine evaluation prompts and adapt to shifting criteria with minimal human input. By selectively sampling representative ads for human review and using a critic LLM to generate alignment reports, AutoEval-Update improves evaluation consistency while reducing manual effort. Experiments show the critic LLM suggests meaningful refinements, improving LLM-human agreement. Nonetheless, human oversight remains essential for setting thresholds and validating refinements before deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
