arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2109.07452cs.CLcs.AI
收藏

机器能阅读编码手册吗?--为代码理解构建更好的语言模型的基准

Can Machines Read Coding Manuals Yet? -- A Benchmark for Building Better Language Models for Code Understanding

IBM Research, T.J. Watson Research Center(IBM研究院托马斯·J·沃森研究中心);Rensselaer Polytechnic Institute (RPI)(伦斯勒理工学院)

Ibrahim Abdelaziz, Julian Dolby, Jamie McCusker, Kavitha Srinivas

摘要:

代码理解是人工智能日益重要的应用。理解代码的一个基本方面是理解关于代码的文本,例如,文件和论坛讨论。预先训练的语言模型(例如,BERT)是各种NLP任务的流行方法,现在有各种基准测试,如GLUE,以帮助改进此类模型的开发,以实现自然语言理解。然而,人们对此类模型在有关代码的文本工件上的效果知之甚少,而且我们不知道此类评估的任何系统性下游任务集。在本文中,我们得出了一组基准(BLANCA -基准语言模型编码工件),评估代码理解的基础上的任务,如预测的最佳答案的问题在一个论坛帖子,找到相关的论坛帖子,或预测类相关的层次结构从类文档。我们评估了当前最先进的语言模型在这些任务上的性能,并表明微调对每个任务都有显着的改善。我们还表明,多任务训练BLANCA任务有助于建立更好的语言模型的代码理解。

英文摘要:

Code understanding is an increasingly important application of Artificial Intelligence. A fundamental aspect of understanding code is understanding text about code, e.g., documentation and forum discussions. Pre-trained language models (e.g., BERT) are a popular approach for various NLP tasks, and there are now a variety of benchmarks, such as GLUE, to help improve the development of such models for natural language understanding. However, little is known about how well such models work on textual artifacts about code, and we are unaware of any systematic set of downstream tasks for such an evaluation. In this paper, we derive a set of benchmarks (BLANCA - Benchmarks for LANguage models on Coding Artifacts) that assess code understanding based on tasks such as predicting the best answer to a question in a forum post, finding related forum posts, or predicting classes related in a hierarchy from class documentation. We evaluate the performance of current state-of-the-art language models on these tasks and show that there is a significant improvement on each task from fine tuning. We also show that multi-task training over BLANCA tasks helps build better language models for code understanding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑