StructTest：通过组合式结构化输出来基准测试大语言模型的推理能力
StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs
AI总结：
针对现有大语言模型评估方法的成本高、有偏差、易受污染作弊等问题，提出基于规则评估的StructTest基准，通过多领域组合式结构化输出任务测试推理能力，验证了其对顶尖模型仍具挑战性，可作为客观评估的补充方案。
摘要：
大语言模型（LLMs）的快速发展需要稳健、无偏且可扩展的评估方法。然而，人工标注的规模化成本高昂，基于模型的评估易受风格偏差影响，基于目标答案的基准测试则容易受到数据污染和作弊的影响。为解决这些局限，我们提出StructTest——一种新型基准测试，用于评估大语言模型遵循组合式指令并生成结构化输出的能力，提供了一套无偏、经济高效且难以作弊的评估框架。评估工作通过基于规则的评估器以确定性方式执行，可轻松扩展至新任务和数据集。我们在摘要生成、代码、HTML、数学等多个领域测试结构化输出能力，并评估了17款主流大语言模型，结果表明即使对于Deepseek-V3/R1和GPT-4o这类性能顶尖的模型，StructTest仍具挑战性，可作为衡量推理能力的稳健代理指标。我们认为StructTest为实现客观全面的模型评估提供了一种重要的补充方法。
英文摘要：
The rapid advancement of large language models (LLMs) demands robust, unbiased, and scalable evaluation methods. However, human annotations are costly to scale, model-based evaluations are susceptible to stylistic biases, and target-answer-based benchmarks are vulnerable to data contamination and cheating. To address these limitations, we propose StructTest, a novel benchmark that evaluates LLMs on their ability to follow compositional instructions and generate structured outputs, providing an unbiased, cost-effective, and difficult-to-cheat evaluation framework. Assessments are conducted deterministically using a rule-based evaluator, which can be easily extended to new tasks and datasets. By testing structured outputs across diverse domains including Summarization, Code, HTML, and Math, and evaluating 17 popular LLMs, we demonstrate that StructTest remains challenging even for top-performing models like Deepseek-V3/R1 and GPT-4o, establishing it as a robust proxy for measuring reasoning capabilities. We believe StructTest offers a critical and complementary approach to achieving objective and comprehensive model evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
