探索基于技能的行为特征标注：在模式引导执行下的人类可操作性和LLM可行性
Exploring and Testing Skill-Based Behavioral Profile Annotation: Human Operability and LLM Feasibility under Schema-Guided Execution
AI总结：
本文探讨了基于技能的行为特征标注，通过人类和LLM在模式引导执行下的表现，发现标注任务在技能层面高度异质，LLM在特定技能上表现可靠，但整体可行性有限。
摘要：
行为特征标注（BP）难以自动化，因为它需要同时在多个语言维度上编码。我们将BP标注视为一系列标注技能的集合，而非单一任务，并从这一视角评估LLM辅助的BP标注。使用3,134条30个中文隐喻颜色术语衍生词的对齐行和一个14特征的BP模式，我们实现了一个技能文件驱动的流程，其中每个特征通过模式文件、决策规则和示例外部定义。两名人类标注员在300个实例验证子集上完成了两次仅模式的协议，使BP技能被分类为直接可操作、在聚焦重新标注下可恢复或结构上未指定。随后在相同设置下评估了GPT-5.4和三个本地可部署的开源模型。结果表明，BP标注在技能层面高度异质：5个技能直接可操作，4个在聚焦重新标注后可恢复，5个仍结构上未指定。GPT-5.4在保留的技能上表现出显著的可靠性（准确率=0.678，κ=0.665，加权F1=0.695），但这种可行性是选择性的而非全局的。人类和GPT的难度剖面在技能层面高度一致（r=0.881），但在实例层面（r=0.016）和词汇项层面（r=-0.142）不一致，我们描述为共享的分类法，独立的执行。成对同意进一步表明，GPT应被视为独立的第三技能声音，而非直接的人类替代品。开源模型的失败集中在模式到技能执行问题上。这些发现表明，自动标注应根据技能可行性而非任务层面自动化进行评估。
英文摘要：
Behavioral Profile (BP) annotation is difficult to automate because it requires simultaneous coding across multiple linguistic dimensions. We treat BP annotation as a bundle of annotation skills rather than a single task and evaluate LLM-assisted BP annotation from this perspective. Using 3,134 concordance lines of 30 Chinese metaphorical color-term derivatives and a 14-feature BP schema, we implement a skill-file-driven pipeline in which each feature is externally defined through schema files, decision rules, and examples. Two human annotators completed a two-round schema-only protocol on a 300-instance validation subset, enabling BP skills to be classified as directly operable, recoverable under focused re-annotation, or structurally underspecified. GPT-5.4 and three locally deployable open-source models were then evaluated under the same setup. Results show that BP annotation is highly heterogeneous at the skill level: 5 skills are directly operable, 4 are recoverable after focused re-annotation, and 5 remain structurally underspecified. GPT-5.4 executes the retained skills with substantial reliability (accuracy = 0.678, \k{appa} = 0.665, weighted F1 = 0.695), but this feasibility is selective rather than global. Human and GPT difficulty profiles are strongly aligned at the skill level (r = 0.881), but not at the instance level (r = 0.016) or lexical-item level (r = -0.142), a pattern we describe as shared taxonomy, independent execution. Pairwise agreement further suggests that GPT is better understood as an independent third skill voice than as a direct human substitute. Open-source failures are concentrated in schema-to-skill execution problems. These findings suggest that automatic annotation should be evaluated in terms of skill feasibility rather than task-level automation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
