arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.04166cs.SEcs.AI
收藏

用于实时语法高亮的多语言模型

Multi Language Models for On-the-Fly Syntax Highlighting

University of Zurich(苏黎世大学)

Marco Edoardo Palma, Pooja Rani, Harald C. Gall

AI总结:

针对单语言语法高亮模型部署复杂、依赖大规模蛮力生成数据集的问题,提出支持6种主流语言的统一实时语法高亮模型,结合新型归一化技术与少样本学习,降低部署成本并提升泛化性。

摘要:

语法高亮是现代软件开发环境中的一项核心功能,可提升代码可读性与开发者生产力。然而,由于后端服务存在严格的时间与内存限制,在线及基于网页的开发工具要实现实时准确高亮仍具挑战。这类系统必须快速且频繁地返回高亮结果,即便代码处于部分有效或完全无效的状态。由此催生了实时(on-the-fly)语法高亮技术,即在内容即将被提供前生成视觉标注,通常面临高请求速率与输入不完整的场景。为高效满足这些需求,当前最优模型采用深度学习来学习蛮力语法高亮解析器的行为——这类工具易于实现,但速度过慢无法用于生产环境。通过深度抽象(Deep Abstraction)流程,蛮力策略被编码为快速统计模型,可同时实现高精度与低延迟推理。尽管这类模型已取得成功,但仍存在关键挑战:单个模型仅支持一种编程语言,需要从速度缓慢的蛮力生成器获取大规模数据集,且训练过程资源消耗大。在多语言环境中,这意味着需要维护多个独立模型,增加了系统复杂度与运营成本。本研究针对这些问题,提出了一种统一模型,最多可支持6种主流编程语言的高亮,将部署复杂度降低至原来的1/6,并提升了在未见过的语言上的表现。一种新颖的归一化技术显著增强了模型的泛化能力,同时少样本学习实验表明,少量 oracle 样本即可替代大规模数据集,最大限度降低了对蛮力生成器的依赖。这些创新相结合,实现了跨多种编程语言的高效、可扩展且具成本效益的语法高亮。

英文摘要:

Syntax highlighting is a critical feature in modern software development environments, enhancing code readability and developer productivity. However, delivering accurate highlighting in real time remains challenging for online and web-based development tools due to strict time and memory constraints on backend services. These systems must serve highlights rapidly and frequently, even when code is partially valid or invalid. This has led to on-the-fly syntax highlighting, where visual annotations are generated just before content is served, often at high request rates and under incomplete input conditions. To meet these demands efficiently, state-of-the-art models use deep learning to learn the behavior of brute-force syntax highlighting resolvers, tools that are easy to implement but too slow for production. Through the Deep Abstraction process, brute-force strategies are encoded into fast statistical models that achieve both high accuracy and low-latency inference. Despite their success, such models face key challenges: they support only one programming language per model, require large datasets from slow brute-force generators, and involve resource-intensive training. In multi-language environments, this means maintaining multiple independent models, increasing system complexity and operational cost. This work addresses these issues by introducing a unified model capable of highlighting up to six mainstream programming languages, reducing deployment complexity by a factor of six and improving performance on unseen languages. A novel normalization technique significantly enhances model generalization, while few-shot learning experiments show that a small number of oracle samples can replace large datasets, minimizing dependence on brute-force generators. Combined, these innovations enable efficient, scalable, and cost-effective syntax highlighting across diverse programming languages.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑