使用数据科学方法识别语义重复问题：Quora案例研究
Identifying Semantically Duplicate Questions Using Data Science Approach: A Quora Case Study
摘要：
在Quora等社交媒体平台上识别语义相同的问题并进行提问对于确保根据问题的意图向用户呈现内容的质量和数量并从而丰富整体用户体验非常重要。检测重复问题是一个具有挑战性的问题，因为自然语言是非常有表现力的，并且可以使用不同的单词、短语和句子结构来传达独特的意图。众所周知，机器学习和深度学习方法在识别相似文本方面取得了优于传统自然语言处理技术的结果。在本文中，以Quora为案例研究，我们探索并应用了不同的机器学习和深度学习技术来识别Quora数据集上的重复问题。通过使用特征工程，特征重要性技术，并与七个选定的机器学习分类器进行实验，我们证明了我们的模型优于以前的研究在这项任务上。具有字符级词频和逆词频的Xgboost模型是我们最好的机器学习模型，它的性能也超过了一些深度学习基线模型。我们应用深度学习技术对四种不同的多层深度神经网络进行建模，包括Glove嵌入、长短期记忆、卷积、最大池化、Dense、批量归一化、激活函数和模型合并。我们的深度学习模型实现了比机器学习模型更好的准确性。四分之三的架构优于之前的机器学习和深度学习研究工作的准确性，四分之二的模型优于之前对Quora问题对数据集的深度学习研究的准确性，我们最好的模型达到了85.82%的准确性，接近Quora最先进的准确性。
英文摘要：
Identifying semantically identical questions on, Question and Answering social media platforms like Quora is exceptionally significant to ensure that the quality and the quantity of content are presented to users, based on the intent of the question and thus enriching overall user experience. Detecting duplicate questions is a challenging problem because natural language is very expressive, and a unique intent can be conveyed using different words, phrases, and sentence structuring. Machine learning and deep learning methods are known to have accomplished superior results over traditional natural language processing techniques in identifying similar texts. In this paper, taking Quora for our case study, we explored and applied different machine learning and deep learning techniques on the task of identifying duplicate questions on Quora's dataset. By using feature engineering, feature importance techniques, and experimenting with seven selected machine learning classifiers, we demonstrated that our models outperformed previous studies on this task. Xgboost model with character level term frequency and inverse term frequency is our best machine learning model that has also outperformed a few of the Deep learning baseline models. We applied deep learning techniques to model four different deep neural networks of multiple layers consisting of Glove embeddings, Long Short Term Memory, Convolution, Max pooling, Dense, Batch Normalization, Activation functions, and model merge. Our deep learning models achieved better accuracy than machine learning models. Three out of four proposed architectures outperformed the accuracy from previous machine learning and deep learning research work, two out of four models outperformed accuracy from previous deep learning study on Quora's question pair dataset, and our best model achieved accuracy of 85.82% which is close to Quora state of the art accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
