arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2001.11899cs.CL
收藏

大规模计算比较语言学的一种有效的自动数据分析方法

An efficient automated data analytics approach to large scale computational comparative linguistics

Department of Computer Science Brunel University London(伦敦布鲁内尔大学计算机科学系)

Gabija Mikulyte, David Gilbert

摘要:

该研究项目旨在克服分析人类语言关系的挑战,通过开发自动比较技术促进语言分组和它们之间的谱系关系的形成。技术是基于某些关键词和概念的语音表示。示例词集包括数字1-10(策划),数字1-10和绵羊计数数字1-10(其他来源)的大型数据库,颜色(策划),基本词(策划)。为了能够在组内进行比较,基于Levenshtein距离度量计算编辑距离的度量。两个字符串之间的度量是单个字符编辑的最小数量,操作包括:插入,删除或替换。为了探索哪些单词表现出或多或少的变化,哪些单词更容易保存,并研究如何根据集合内的语言距离对语言进行分组,涉及了几种数据分析技术。其中包括密度评估、等级聚类、轮廓、平均值、标准差和Bhattacharya系数计算。这些技术导致了后来通过结合Unix shell脚本,开发的R包和SWI Prolog实现的工作流的开发。这被证明是计算效率高,并允许快速探索大型语言集及其分析。

英文摘要:

This research project aimed to overcome the challenge of analysing human language relationships, facilitate the grouping of languages and formation of genealogical relationship between them by developing automated comparison techniques. Techniques were based on the phonetic representation of certain key words and concept. Example word sets included numbers 1-10 (curated), large database of numbers 1-10 and sheep counting numbers 1-10 (other sources), colours (curated), basic words (curated). To enable comparison within the sets the measure of Edit distance was calculated based on Levenshtein distance metric. This metric between two strings is the minimum number of single-character edits, operations including: insertions, deletions or substitutions. To explore which words exhibit more or less variation, which words are more preserved and examine how languages could be grouped based on linguistic distances within sets, several data analytics techniques were involved. Those included density evaluation, hierarchical clustering, silhouette, mean, standard deviation and Bhattacharya coefficient calculations. These techniques lead to the development of a workflow which was later implemented by combining Unix shell scripts, a developed R package and SWI Prolog. This proved to be computationally efficient and permitted the fast exploration of large language sets and their analysis.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑