仓颉基准：在低资源通用编程语言上评估大语言模型
CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
AI总结：
本文提出CangjieBench，用于评估大语言模型在低资源通用编程语言上的表现，通过248个高质量样本覆盖文本到代码和代码到代码任务，发现语法受限生成在准确性和计算成本之间取得最佳平衡。
摘要：
大型语言模型在高资源编程语言上表现出色，但在低资源语言上表现不佳。现有研究主要集中在领域特定语言上，而通用编程语言因数据稀缺未被充分探索。为此，我们引入CangjieBench，一个无污染的基准，用于Cangjie，一种代表性的低资源通用语言。该基准包含248个高质量样本，手动翻译自HumanEval和ClassEval，涵盖文本到代码和代码到代码任务。我们对多种LLM进行了系统评估，四种设置：直接生成、语法受限生成、检索增强生成（RAG）和代理。实验表明，直接生成表现不佳，而语法受限生成在准确性和计算成本之间取得最佳平衡。代理实现最先进准确性但消耗大量token。此外，我们发现代码到代码翻译常表现不佳，表明负迁移现象，即模型过度适应源语言模式。我们希望本工作能为LLM在未见和低资源编程语言上的泛化提供有价值的见解。代码和数据可在https://github.com/cjhCoder7/CangjieBench获取。
英文摘要：
Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resource programming languages primarily focuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suffer from data scarcity underexplored. To address this gap, we introduce CangjieBench, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval, covering both Text-to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs under four settings: Direct Generation, Syntax-Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost. Agent achieve state-of-the-art accuracy but incur high token consumption. Furthermore, we observe that Code-to-Code translation often underperforms Text-to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low-resource programming languages. Our code and data are available at https://github.com/cjhCoder7/CangjieBench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
