TextGNN:基于图神经网络的文本编码器改进
TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Search
摘要:
基于C-DSSM或transformers的文本编码器在许多自然语言处理(NLP)任务中表现出强大的性能。近年来,这些模型的低延迟变体也被开发出来,以便将它们应用于具有严格计算约束的赞助搜索领域。然而,这些模型并不是解决所有自然语言理解(NLU)挑战的灵丹妙药,因为数据中的纯语义信息不足以完全识别用户意图。我们提出了TextGNN模型,该模型自然地扩展了强大的双塔结构编码器,并从用户历史行为中获得了互补的图形信息,这可以作为一个自然的指南,帮助我们更好地理解意图,从而生成更好的语言表示。该模型继承了双塔模型(如C-DSSM和TwinBERT)的所有优点,因此它仍然可以在低延迟环境中使用,同时在离线评估和在线生产系统中实现比仅强编码器对应基线模型显着的性能增益。在离线实验中,该模型实现了0.14%的ROC-AUC整体提高,长尾低频广告的准确率提高了1%,在在线A/B测试中,该模型显示出2.03%的每千收入增加,广告缺陷率降低了2.32%。
英文摘要:
Text encoders based on C-DSSM or transformers have demonstrated strong performance in many Natural Language Processing (NLP) tasks. Low latency variants of these models have also been developed in recent years in order to apply them in the field of sponsored search which has strict computational constraints. However these models are not the panacea to solve all the Natural Language Understanding (NLU) challenges as the pure semantic information in the data is not sufficient to fully identify the user intents. We propose the TextGNN model that naturally extends the strong twin tower structured encoders with the complementary graph information from user historical behaviors, which serves as a natural guide to help us better understand the intents and hence generate better language representations. The model inherits all the benefits of twin tower models such as C-DSSM and TwinBERT so that it can still be used in the low latency environment while achieving a significant performance gain than the strong encoder-only counterpart baseline models in both offline evaluations and online production system. In offline experiments, the model achieves a 0.14% overall increase in ROC-AUC with a 1% increased accuracy for long-tail low-frequency Ads, and in the online A/B testing, the model shows a 2.03% increase in Revenue Per Mille with a 2.32% decrease in Ad defect rate.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。