InfoSynth: 信息引导的大语言模型基准合成
InfoSynth: Information-Guided Benchmark Synthesis for LLMs
AI总结：
提出基于信息论（KL散度和熵）的InfoSynth框架，自动生成高难度、多样化的Python编程基准，97%的测试用例和解决方案准确。
摘要：
大型语言模型（LLM）在推理和代码生成方面取得了显著进展，但高效创建新基准来评估这些能力仍然是一个挑战。传统的基准创建依赖人工，成本高且耗时。此外，现有基准常常污染LLM训练数据，因此需要新颖多样的基准来准确评估其真实能力。本文介绍了InfoSynth，一个基于信息论原理自动生成和评估推理基准的新框架。我们提出了基于KL散度和熵的度量标准，无需昂贵的模型评估即可量化基准的新颖性和多样性。在此框架基础上，我们开发了一个端到端的流水线，使用遗传算法和迭代代码反馈从种子数据集中合成稳健的Python编程问题。我们的方法在97%的情况下生成准确的新问题测试用例和解决方案，并且合成的基准在难度上始终高于先前的工作。此外，我们的算法提供了控制生成问题的新颖性/多样性和难度的方法。InfoSynth为LLM构建高质量、具有挑战性的编程基准提供了一个可扩展、自验证的流水线。项目页面：https://ishirgarg.github.io/infosynth_web/
英文摘要：
Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation, but efficiently creating new benchmarks to evaluate these capabilities remains a challenge. Traditional benchmark creation relies on manual human effort, which is expensive and time-consuming. Furthermore, existing benchmarks often contaminate LLM training data, necessitating novel and diverse benchmarks to accurately assess their genuine capabilities. This work introduces InfoSynth, a novel framework for automatically generating and evaluating reasoning benchmarks guided by information-theoretic principles. We propose metrics based on KL-divergence and entropy to quantify benchmark novelty and diversity without relying on costly model evaluations. Building on this framework, we develop an end-to-end pipeline that synthesizes robust Python coding problems from seed datasets using genetic algorithms and iterative code feedback. Our method generates accurate test cases and solutions to new problems 97% of the time, and the synthesized benchmarks consistently exhibit higher difficulty compared to prior works. Moreover, our algorithm provides a method for controlling the novelty/diversity and difficulty of generated problems. InfoSynth offers a scalable, self-verifying pipeline for constructing high-quality, challenging coding benchmarks for LLMs. Project Page: https://ishirgarg.github.io/infosynth_web/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
