基于内容的大规模文本文件类型检测
Content-Based Textual File Type Detection at Scale
摘要：
程序设计语言检测是大型源代码库分析中的一个常见需求。许多现有的工具都支持它，这些工具依赖于几个功能，尤其是文件扩展名来确定文件类型。我们认为，准确地检测软件代码库中常见的文件类型的问题，仅仅基于文本文件内容。这样做有助于对缺少文件扩展名的源代码进行分类（例如，发布在Web上的代码片段或可执行脚本），以避免对使用错误或不常见的文件扩展名记录的源代码进行错误分类，并对源代码文件的内在可识别性提供一些帮助。我们提出了一个简单的模型，（a）使用一个语言无关的文字tokenizer文本文件，（b）在1-/2-gram组令牌，（c）建立基于N-gram频率的特征向量，（d）使用一个简单的全连接神经网络作为分类器。作为训练集，我们使用从GitHub存储库中提取的文本文件，至少有1000颗星，使用现有的文件扩展名作为基础事实。尽管它的简单性，建议的模型达到85%，在我们的实验中识别类的数量相对较高（超过130个文件类型）。
英文摘要：
Programming language detection is a common need in the analysis of large source code bases. It is supported by a number of existing tools that rely on several features, and most notably file extensions, to determine file types. We consider the problem of accurately detecting the type of files commonly found in software code bases, based solely on textual file content. Doing so is helpful to classify source code that lack file extensions (e.g., code snippets posted on the Web or executable scripts), to avoid misclassifying source code that has been recorded with wrong or uncommon file extensions, and also shed some light on the intrinsic recognizability of source code files. We propose a simple model that (a) use a language-agnostic word tokenizer for textual files, (b) group tokens in 1-/2-grams, (c) build feature vectors based on N-gram frequencies, and (d) use a simple fully connected neural network as classifier. As training set we use textual files extracted from GitHub repositories with at least 1000 stars, using existing file extensions as ground truth. Despite its simplicity the proposed model reaches 85% in our experiments for a relatively high number of recognized classes (more than 130 file types).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
