超越BLEU：代码翻译的语义评估方法
Beyond BLEU: A Semantic Evaluation Method for Code Translation
AI总结：
本文提出一种基于语义等价的代码翻译评估方法，通过编译器测试方法评估LLM的二进制提升任务性能，发现语义正确性得分与BLEU得分无显著相关性。
摘要：
代码翻译是LLM的核心能力之一，但评估翻译正确性困难，因为BLEU等指标仅衡量语法相似性而忽略程序语义。本文提出一种新的代码翻译评估方法，强调语义等价而非表面字符串相似性。我们的方法将已有的编译器测试方法应用于新领域，允许评估经过二进制提升任务（即反编译二进制到更高层次表示）微调的LLM。我们引入语义正确性得分，定义为产生正确执行结果的翻译比例，并通过评估基于LLM和启发式反编译器来展示其应用。我们的发现表明，基于LLM的方法显著优于启发式方法，而BLEU得分与语义正确性无显著相关性（r = -0.127至0.354），证明语法指标无法预测功能准确性。
英文摘要：
Code translation is one of the core capabilities of LLMs. However, evaluating the correctness of translations remains difficult, as commonly used metrics such as BLEU measure only syntactic similarity, disregarding program semantics. We propose a novel evaluation methodology for code translation tasks, emphasizing semantic equivalence over surface-level string similarity. Our approach applies established compiler testing methodology to a new domain, allowing the assessment of an LLM fine-tuned for binary lifting tasks (i.e. decompiling binaries to higher-level representations). We introduce a semantic correctness score, defined as the proportion of translations that produce correct execution outcomes, and demonstrate its application by evaluating LLM-based and heuristic decompilers. Our findings show that LLM-based approaches significantly outperform heuristic ones, while BLEU scores show negligible correlation with semantic correctness (r = -0.127 to 0.354), demonstrating that syntactic metrics fail to predict functional accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
