对“通过学习Source Code的通用标记重复来改进代码完成的语言建模（具有优化内存）”的更正和补充
Corrigendum and Supplement to "Improve Language Modelling for Code Completion through Learning General Token Repetition of Source Code (with Optimized Memory)"
摘要：
写这篇论文是因为我收到了几封询问邮件，说在应用令牌重复学习技术时很难取得好的效果。如果直接将REP（我提出的）或Pointer-Mixture（李健提出的）应用于源代码来决定所有令牌重复，模型性能将急剧下降。由于我们使用前序遍历来遍历抽象树（AST）以生成标记序列，因此在学习标记重复时忽略了AST语法对应的标记。对于非语法标记，有很多种：字符串、字符、数字和标识符。对于每一种标记，我们试图学习它的重复模式，发现只有标识符具有标记重复的属性。对于标识符，也有许多种类，如变量，包名称，方法名称，简单类型，限定类型或限定名称。实际上，某些类型的标识符（如包名、方法名、限定名或限定类型）不太可能重复。因此，我们在学习标记重复时忽略了这些不太可能重复的标识符。这一步是至关重要的，这一重要的实现技巧在本文中没有明确提出，因为我们认为这是微不足道的，太多的细节可能会困扰读者。我们在会议论文中提供了模型的GitHub地址，读者可以在该存储库中查看描述和实现。因此，在本文中，我们补充了已经发表的论文的重要实现优化的细节。
英文摘要：
This paper is written because I receive several inquiry emails saying it is hard to achieve good results when applying token repetition learning techniques. If REP (proposed by me) or Pointer-Mixture (proposed by Jian Li) is directly applied to source code to decide all token repetitions, the model performance will decrease sharply. As we use pre-order traversal to traverse the Abstract Syntax Tree (AST) to generate token sequence, tokens corresponding to AST grammar are ignored when learning token repetition. For non-grammar tokens, there are many kinds: strings, chars, numbers and identifiers. For each kind of tokens, we try to learn its repetition pattern and find that only identifiers have the property of token repetition. For identifiers, there are also many kinds such as variables, package names, method names, simple types, qualified types or qualified names. Actually, some kinds of identifiers such as package names, method names, qualified names or qualified types are unlikely to be repeated. Thus, we ignore these kinds of identifiers that are unlikely to be repeated when learning token repetition. This step is crucial and this important implementation trick is not clearly presented in the paper because we think it is trivial and too many details may bother readers. We offer the GitHub address of our model in our conference paper and readers can check the description and implementation in that repository. Thus, in this paper, we supplement the important implementation optimization details for the already published papers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
