arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.02430cs.SEcs.AI
收藏

WebCoderBench: 用全面且可解释的评估指标对Web应用生成进行基准测试

WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang, Tao Xie

AI总结:

本文提出WebCoderBench,首个基于真实用户需求的Web应用生成基准,包含1572个真实用户需求及24个细粒度评估指标,结合规则和LLM评估方法,提供可解释的评估结果,实验显示无单一最优模型,为模型优化提供方向。

摘要:

Web应用已成为大型语言模型展示代码生成能力的关键领域。然而,构建LLM生成Web应用的基准仍具挑战性,需真实用户需求、通用评估指标且不依赖地面真相实现或测试用例。为解决这些挑战,我们引入WebCoderBench,首个真实收集、通用且可解释的Web应用生成基准。WebCoderBench包含1,572个真实用户需求,涵盖多样化的模态和表达风格,反映真实用户意图。WebCoderBench提供24个细粒度评估指标,结合规则基础和LLM作为评判者的方法,实现全自动、客观和通用的评估。此外,WebCoderBench采用与人类偏好对齐的指标权重,以获得可解释的总体评分。实验显示,12个代表性LLM和2个LLM基于代理在所有评估指标中均无主导模型,为LLM开发者提供针对性优化的机会,以获得更强大的版本。

英文摘要:

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results. To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation. WebCoderBench comprises 1,572 real user requirements, covering diverse modalities and expression styles that reflect realistic user intentions. WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation. Moreover, WebCoderBench adopts human-preference-aligned weights over metrics to yield interpretable overall scores. Experiments across 12 representative LLMs and 2 LLM-based agents show that there exists no dominant model across all evaluation metrics, offering an opportunity for LLM developers to optimize their models in a targeted manner for a more powerful version.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑