FireBench: 评估企业及 API 驱动的 LLM 应用中的指令遵循
FireBench: Evaluating Instruction Following in Enterprise and API-Driven LLM Applications
AI总结：
FireBench 是一个评估企业及 API 驱动 LLM 应用中指令遵循能力的基准测试，通过 2400 个样本评估六个核心能力维度，评估 11 个 LLM 的表现。
摘要：
指令遵循对于部署在企业及 API 驱动环境中的 LLM 至关重要，其中严格遵守输出格式、内容限制和程序要求对于启用可靠的 LLM 辅助工作流至关重要。然而，现有指令遵循基准测试主要评估自然语言生成约束，这些约束反映了聊天助手的需求，而非企业用户的需求。为弥合这一差距，我们引入 FireBench，一个基于真实企业及 API 使用模式的 LLM 指令遵循基准测试。FireBench 在包括信息提取、客户支持和编码代理在内的多样化应用中评估六个核心能力维度，包含超过 2,400 个样本。我们评估了 11 个 LLM，并展示了它们在企业场景中的指令遵循行为。我们开源 FireBench 至 fire-bench.com，以帮助用户评估模型适用性，支持模型开发者诊断性能，并邀请社区贡献。
英文摘要：
Instruction following is critical for LLMs deployed in enterprise and API-driven settings, where strict adherence to output formats, content constraints, and procedural requirements is essential for enabling reliable LLM-assisted workflows. However, existing instruction following benchmarks predominantly evaluate natural language generation constraints that reflect the needs of chat assistants rather than enterprise users. To bridge this gap, we introduce FireBench, an LLM instruction following benchmark grounded in real-world enterprise and API usage patterns. FireBench evaluates six core capability dimensions across diverse applications including information extraction, customer support, and coding agents, comprising over 2,400 samples. We evaluate 11 LLMs and present key findings on their instruction following behavior in enterprise scenarios. We open-source FireBench at fire-bench.com to help users assess model suitability, support model developers in diagnosing performance, and invite community contributions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
