HyPAC: 低成本的LLM-人类混合标注与PAC误差保证
HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees
AI总结：
HyPAC通过自适应路由和PAC误差保证，实现低成本高效标注，降低78.51%成本并严格控制误差。
摘要：
数据标注通常涉及多个来源，具有不同的成本-质量权衡，例如快速的大语言模型（LLMs）、缓慢的推理模型和人类专家。在本文中，我们研究如何将输入路由到最低成本的标注源，同时控制测试实例的标注误差。我们提出了HyPAC方法，该方法能够自适应地将输入路由到最低成本的标注源，同时提供无分布的标注误差保证。HyPAC通过重要性采样和上置信界校准两个决策阈值，根据不确定性将输入划分为三个区域，并将每个区域路由到合适的标注源。我们证明HyPAC在无数据分布和预训练模型的前提下，实现了最小的预期成本，并具有近似正确（PAC）的标注误差保证。在常见基准测试中的实验表明，我们的方法有效，将标注成本降低了78.51%，同时紧密控制标注误差。
英文摘要：
Data annotation often involves multiple sources with different cost-quality trade-offs, such as fast large language models (LLMs), slow reasoning models, and human experts. In this work, we study the problem of routing inputs to the most cost-efficient annotation source while controlling the labeling error on test instances. We propose \textbf{HyPAC}, a method that adaptively labels inputs to the most cost-efficient annotation source while providing distribution-free guarantees on annotation error. HyPAC calibrates two decision thresholds using importance sampling and upper confidence bounds, partitioning inputs into three regions based on uncertainty and routing each to the appropriate annotation source. We prove that HyPAC achieves the minimum expected cost with a probably approximately correct (PAC) guarantee on the annotation error, free of data distribution and pre-trained models. Experiments on common benchmarks demonstrate the effectiveness of our method, reducing the annotation cost by 78.51\% while tightly controlling the annotation error.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
