Vibe Patenting：评估用于专业专利撰写智能体的LLM裁判
Vibe Patenting: Evaluating LLM Judges for Professional Patent-Drafting Agents
- Mitsubishi Electric Research Laboratories (MERL)（三菱电机研究实验室）
机构由 AI 辅助整理，请以论文原文为准。
AI总结：
本研究通过Vibe Patenting测试平台，评估LLM裁判在专利撰写智能体中的可靠性，发现裁判引导的迭代修订能提升草稿质量，使低推理智能体接近高推理性能，但裁判与专家评估存在指标依赖的差异。
AI中文摘要：
LLM裁判越来越多地被用于评估和改进AI生成的输出，但其在复杂专业工作中的可靠性仍不明确。我们通过Vibe Patenting研究这一问题，这是一个用于AI智能体评估的端到端专利撰写测试平台。一个单独调用的LLM裁判评估生成的专利草稿，并提供结构化反馈以进行迭代修订。在多个发明和撰写智能体配置中，裁判引导的修订持续提高了裁判评估的质量，而无引导的修订则趋于饱和。值得注意的是，迭代的裁判反馈使低推理能力的智能体能够接近性能显著更高、成本更高的高推理能力智能体。更强的模型和增加的推理通常能提高裁判评估的撰写质量，而特定领域的智能体工作流则带来进一步的提升。我们通过与专业专利律师的独立评估对比来验证裁判，发现存在有意义但强烈依赖于指标的一致性以及系统性的校准差异。这些结果突显了LLM裁判作为评估器和优化信号在复杂专业工作流中的效用与局限性。
英文摘要：
LLM judges are increasingly used to evaluate and improve AI-generated outputs, yet their reliability for complex professional work remains unclear. We study this problem through Vibe Patenting, an end-to-end patent-drafting testbed for AI-agent evaluation. A separately-invoked LLM judge evaluates generated patent drafts and provides structured feedback for iterative revision. Across multiple inventions and drafting-agent configurations, judge-guided revision consistently improves judge-assessed quality, while unguided revision tends to saturate. Notably, iterative judge feedback enables a low-reasoning agent to approach the performance of a substantially more expensive high-reasoning agent. Stronger models and increased reasoning generally improve judge-assessed drafting quality, while domain-specific agentic workflows provide further gains. We validate the judge against independent evaluation by a professional patent attorney and find meaningful but strongly metric-dependent agreement and systematic calibration differences. These results highlight both the utility and limitations of LLM judges as evaluators and optimization signals for complex professional workflows.
