基于大型语言模型的汉语叙述转录自动标注
LLMs for automatic annotation of Mandarin narrative transcripts
AI总结：
本研究探讨了大型语言模型在自动标注汉语叙述转录中的有效性，通过比较四种LLM与训练好的人类标注者，发现最佳模型在标注叙事宏观结构时能与人类标注者达成高一致性，同时显著减少标注时间，但轻量级本地部署模型表现较差，且标注难度因宏观结构元素类型而异，尤其在需要微妙语义区分的类别中存在持续挑战。
摘要：
对转录语音进行语言标注对于语言习得、语言障碍和社会语言学研究至关重要，但这一过程仍然耗费大量人力和时间。尽管大型语言模型（LLMs）在自动化标注任务中显示出潜力，但其在非英语语言中处理复杂语篇层面标注的能力仍缺乏研究。本研究评估了LLMs在标注汉语口语中的叙事宏观结构（即故事语法元素的层级组织）的可靠性，使用多语言叙述评估仪器（MAIN）作为测试平台。我们比较了四种LLM与训练好的人类标注者在儿童、年轻人和老年人生成的叙述上的表现。最佳模型在与人类评分者达成一致（k=.794）方面接近人类-人类可靠性水平（k=.872），同时将标注时间减少了65%，而本地可部署的轻量级模型表现则明显较差。标注难度系统性地因宏观结构元素类型而异，需要微妙语义区分的类别提出了持续挑战。此外，模型可靠性在年轻人叙述中下降，因为年轻人叙述表现出更大的词汇变化、语义模糊性和单个语句中的多元素整合。这些发现表明，LLMs可以有效支持非英语口语语料库的语篇层面标注，同时强调在语义复杂任务中仍需持续的人类监督。我们的提示模板已开源供未来使用。
英文摘要：
Linguistic annotation of transcribed speech is essential for research in language acquisition, language disorders, and sociolinguistics, yet remains labor-intensive and time-consuming. While Large Language Models (LLMs) have shown promise in automating annotation tasks, their ability to handle complex discourse-level annotation in non-English languages remains understudied. This study evaluates whether LLMs can reliably annotate narrative macrostructure-the hierarchical organization of story grammar elements-in spoken Mandarin, using the Multilingual Assessment Instrument for Narratives (MAIN) as a testbed. We compared four LLMs against trained human annotators on narratives produced by children, young adults, and older adults. The best-performing model achieved agreement with human raters (k=.794) approaching human-human reliability levels (k=.872) while reducing annotation time by 65%, whereas the locally deployable lightweight model performed substantially worse. Annotation difficulty varied systematically by macrostructure element type, with categories requiring subtle semantic differentiation posing persistent challenges. Furthermore, model reliability decreased on young adult narratives, which exhibited greater lexical variation, semantic ambiguity, and multi-element integration within single utterances. These findings suggest that LLMs can effectively support discourse-level annotation in non-English spoken corpora, while highlighting the continued need for human oversight in semantically complex tasks. Our prompt templates are open sourced for future use.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
