Hints-In-Browser：面向编程反馈生成的语言模型基准评测
Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation
AI总结：
本文提出Hints-In-Browser基准，从质量、成本、时间和隐私多维度评测编程反馈生成模型，并通过GPT-4合成数据微调小模型，在浏览器内实现高效推理。
摘要：
生成式人工智能和大语言模型在为学习者生成个性化反馈与提示方面展现出巨大潜力，有望提升编程教育效果。近期研究主要聚焦于提升生成反馈的质量，以达到人类导师的水平。虽然质量是一项重要的性能标准，但在真实教育部署中，它并不是唯一需要优化的标准。本文从多个性能标准出发，对用于编程反馈生成的语言模型进行基准评测，包括质量、成本、时间和数据隐私。核心思路是利用浏览器内推理这一新范式的最新进展，使这些模型能够直接在浏览器中运行，从而在成本和数据隐私方面带来直接收益。为了提升与浏览器内推理引擎兼容的小模型反馈质量，我们开发了一套基于GPT-4生成合成数据的微调流程。我们使用WebLLM的浏览器内推理引擎，在三个不同的Python编程数据集上展示了微调后的Llama3-8B和Phi3-3.8B 4位量化模型的有效性。我们将发布完整实现以及一个Web应用和数据集，以促进浏览器内语言模型的进一步研究。
英文摘要：
Generative AI and large language models hold great promise in enhancing programming education by generating individualized feedback and hints for learners. Recent works have primarily focused on improving the quality of generated feedback to achieve human tutors' quality. While quality is an important performance criterion, it is not the only criterion to optimize for real-world educational deployments. In this paper, we benchmark language models for programming feedback generation across several performance criteria, including quality, cost, time, and data privacy. The key idea is to leverage recent advances in the new paradigm of in-browser inference that allow running these models directly in the browser, thereby providing direct benefits across cost and data privacy. To boost the feedback quality of small models compatible with in-browser inference engines, we develop a fine-tuning pipeline based on GPT-4 generated synthetic data. We showcase the efficacy of fine-tuned Llama3-8B and Phi3-3.8B 4-bit quantized models using WebLLM's in-browser inference engine on three different Python programming datasets. We will release the full implementation along with a web app and datasets to facilitate further research on in-browser language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
