发表机构
UC San Diego; University of Southern California（加州大学圣地亚哥分校; 南加州大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
CALICO提出以人为中心、与编码手册对齐的标注工作流，将提示作为可编辑优化工件，集成多种优化器，在AI伴侣对话标注中平均性能提升13.0和7.4个绝对点。
AI 中文摘要
大型语言模型越来越多地被用于扩展科学研究中基于编码手册的标注，但现有工作流在将领域专家的编码手册转化为可靠、可修订且可审计的提示方面支持有限。提示通常被视为固定指令，并对标注者隐藏，这使得非技术领域的专家在输出违反编码手册指南时难以诊断和纠正模型行为。在本文中，我们提出了CALICO，一种以人为中心、与编码手册对齐的标注工作流，它将提示视为可编辑、可版本化和可优化的工件。CALICO整合了编码手册解析、提示生成、结果检查、提示版本管理、自然语言人类反馈以及通过现有优化器（如GEPA、MIPROv2和OPRO）以及我们基于反思的优化器ReflectAgent进行的标签监督提示优化。在实证中，我们在特定领域的AI伴侣聊天机器人对话编码手册上评估了CALICO。在评估的维度上，对于两位编码员，CALICO分别将平均保留性能提高了+13.0和+7.4个绝对点。编码员特异性分析进一步表明，优化后的提示捕获了编码员特定的解释，而不仅仅是通用的编码手册澄清。CALICO作为Web应用程序运行，将用户从原始编码手册材料带到可检查、可导出的标签；网站、代码库和实时演示已在Apache 2.0许可证下于该HTTPS URL发布。
英文摘要
Large language models are increasingly used to scale codebook-based annotation in scientific research, but existing workflows provide limited support for translating domain experts' codebooks into reliable, revisable, and auditable prompts. Prompts are often treated as fixed instructions and hidden from annotators, making it difficult for non-technical domain experts to diagnose and correct model behavior when outputs violate codebook guidelines. In this paper, we present CALICO, a human-centered, codebook-aligned annotation workflow that treats prompts as editable, versioned, and optimizable artifacts. CALICO integrates codebook parsing, prompt generation, result inspection, prompt versioning, natural language human feedback, and label-supervised prompt optimization through existing optimizers such as GEPA, MIPROv2, and OPRO, together with our reflection-based optimizer, ReflectAgent. Empirically, we evaluate CALICO on domain-specific AI-companion chatbot conversation codebooks. Across evaluated dimensions, CALICO improves mean held-out performance by +13.0 and +7.4 absolute points for two coders, respectively. A coder-specificity analysis further suggests that optimized prompts capture coder-specific interpretations rather than only generic codebook clarification. CALICO runs as a web application that takes users from raw codebook materials to inspectable, exportable labels; the website, codebase, and live demo are released at https://calico-annotation.github.io/ under the Apache 2.0 License.
Comments14 pages, 5 figures, 5 tables
