以更少精力实现可靠标注:评估搜索澄清中的LLM-人类协作
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
AI总结:
本研究评估LLMs在搜索澄清多维标注任务中的局限性,并提出一种基于置信度阈值和模型间分歧的人在环工作流,在减少45%人工量的同时显著提升标注可靠性。
摘要:
尽管使用大型语言模型(LLMs)自动化标注的兴趣日益浓厚,但它们在复杂、细微且多维度的标注任务中的有效性仍相对未被充分探索。本研究聚焦于搜索澄清任务的标注,利用一个包含五个不同细粒度标注子任务的高质量、多维度数据集。尽管LLMs在通用环境中展现了令人印象深刻的能力,但我们的研究表明,即使是最先进的模型也难以在主观或细粒度评估任务中复制人类水平的性能。通过系统评估,我们证明LLM的预测通常不一致、校准不良,且对提示变化高度敏感。为了解决这些局限性,我们提出了一种简单而有效的人在环(HITL)工作流,该工作流使用置信度阈值和模型间分歧来选择性地引入人工审查。我们的研究结果表明,这种轻量级干预显著提高了标注可靠性,同时将人工工作量减少了高达45%,为在现实世界评估环境中部署LLMs提供了一条相对可扩展、具有成本效益且准确的路径。
英文摘要:
Despite growing interest in using large language models (LLMs) to automate annotation, their effectiveness in complex, nuanced, and multi-dimensional labelling tasks remains relatively underexplored. This study focuses on annotation for the search clarification task, leveraging a high-quality, multi-dimensional dataset that includes five distinct fine-grained annotation subtasks. Although LLMs have shown impressive capabilities in general settings, our study reveals that even state-of-the-art models struggle to replicate human-level performance in subjective or fine-grained evaluation tasks. Through a systematic assessment, we demonstrate that LLM predictions are often inconsistent, poorly calibrated, and highly sensitive to prompt variations. To address these limitations, we propose a simple yet effective human-in-the-loop (HITL) workflow that uses confidence thresholds and inter-model disagreement to selectively involve human review. Our findings show that this lightweight intervention significantly improves annotation reliability while reducing human effort by up to 45%, offering a relatively scalable and cost-effective yet accurate path forward for deploying LLMs in real-world evaluation settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。