AutoChecklist: 用于检查表生成和评分的可组合流水线
AutoChecklist: Composable Pipelines for Checklist Generation and Scoring with LLM-as-a-Judge
AI总结:
AutoChecklist通过可组合的流水线实现检查表生成与评分,支持多种LLM提供者,并展示了在领域适应中的灵活性。
摘要:
检查表已成为一种流行的可解释和细粒度评估方法,尤其是在LLM-as-a-Judge的应用中。除了评估之外,这些结构化标准还可以作为对齐模型、强化学习和自我纠正的信号。为了支持这些用例,我们提出了AutoChecklist,一个开源库,将基于检查表的评估统一到可组合的流水线中。其核心是一种五种检查表生成抽象的分类,每种编码一种不同的策略来推导评估标准。一个模块化的生成器→细化器→评分器流水线将任何生成器连接到统一的评分器,并且新的配置可以通过提示模板单独注册。该库附带了十个内置流水线,实现了已发布的方案,并支持多个LLM提供者(OpenAI、OpenRouter、vLLM)。除了Python API外,该库还包含一个CLI用于即开即用的评估和一个Web界面用于交互式探索。验证实验证实这些检查表方法显著与人类偏好和质量评分一致,并且在ICLR同行评审反驳案例研究中展示了灵活的领域适应。AutoChecklist在https://github.com/ChicagoHAI/AutoChecklist上公开可用。
英文摘要:
Checklists have emerged as a popular approach for interpretable and fine-grained evaluation, particularly with LLM-as-a-Judge. Beyond evaluation, these structured criteria can serve as signals for model alignment, reinforcement learning, and self-correction. To support these use cases, we present AutoChecklist, an open-source library that unifies checklist-based evaluation into composable pipelines. At its core is a taxonomy of five checklist generation abstractions, each encoding a distinct strategy for deriving evaluation criteria. A modular Generator $\rightarrow$ Refiner $\rightarrow$ Scorer pipeline connects any generator with a unified scorer, and new configurations can be registered via prompt templates alone. The library ships with ten built-in pipelines implementing published approaches and supports multiple LLM providers (OpenAI, OpenRouter, vLLM). Beyond the Python API, the library includes a CLI for off-the-shelf evaluation and a web interface for interactive exploration. Validation experiments confirm that these checklist methods significantly align with human preferences and quality ratings, and a case study on ICLR peer review rebuttals demonstrates flexible domain adaptation. AutoChecklist is publicly available at https://github.com/ChicagoHAI/AutoChecklist.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。