arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1410.2686cs.LGcs.CL
收藏

基于Mapreduce的支持向量机高维社交媒体消息极化度量

Polarization Measurement of High Dimensional Social Media Messages With Support Vector Machine Algorithm Using Mapreduce

TÜBİTAK – BİLGEM – SİBER GÜVENLİK ENSTİTÜSÜ(土耳其科学技术研究委员会信息与信息安全研究所)

Ferhat Özgür Çatak

摘要:

提出了一种基于分布式MapReduce技术的支持向量机(SVM)训练算法。大量的研究表明,在机器学习领域的分类算法中,支持向量机具有最好的泛化性能。而且,SVM分类器模型不受特征相关性的影响。但SVM在训练阶段使用二次优化技术。将SVM算法转化为二次优化问题。二次优化问题具有$O(m^3)$时间复杂度和$O(m^2)$空间复杂度,其中m是训练集的大小。支持向量机训练的计算时间是训练实例数的二次方。因此,SVM不是一种适合大规模数据集分类的分类算法。为了解决这个训练问题,我们开发了一种新的分布式MapReduce方法。相应地,(i)在分布式数据集中单独训练SVM算法;(ii)然后合并每个训练节点中的分类器模型的所有支持向量;以及(iii)重复这两个步骤,直到分类器模型收敛到最优分类器函数。在实现阶段,大规模的社会媒体数据集在TFxIDF矩阵。该矩阵用于情感分析,得到极化值。分类方法建立了两类和三类模型。每个分类模型的混淆矩阵在表中呈现。社交媒体消息语料库包括土耳其108所公立大学和66所私立大学的消息。Twitter被用作语料库的来源。Twitter用户消息使用Twitter Streaming API收集。结果以图形和表格显示。

英文摘要:

In this article, we propose a new Support Vector Machine (SVM) training algorithm based on distributed MapReduce technique. In literature, there are a lots of research that shows us SVM has highest generalization property among classification algorithms used in machine learning area. Also, SVM classifier model is not affected by correlations of the features. But SVM uses quadratic optimization techniques in its training phase. The SVM algorithm is formulated as quadratic optimization problem. Quadratic optimization problem has $O(m^3)$ time and $O(m^2)$ space complexity, where m is the training set size. The computation time of SVM training is quadratic in the number of training instances. In this reason, SVM is not a suitable classification algorithm for large scale dataset classification. To solve this training problem we developed a new distributed MapReduce method developed. Accordingly, (i) SVM algorithm is trained in distributed dataset individually; (ii) then merge all support vectors of classifier model in every trained node; and (iii) iterate these two steps until the classifier model converges to the optimal classifier function. In the implementation phase, large scale social media dataset is presented in TFxIDF matrix. The matrix is used for sentiment analysis to get polarization value. Two and three class models are created for classification method. Confusion matrices of each classification model are presented in tables. Social media messages corpus consists of 108 public and 66 private universities messages in Turkey. Twitter is used for source of corpus. Twitter user messages are collected using Twitter Streaming API. Results are shown in graphics and tables.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑