arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.05435cs.CLcs.AIcs.LG
收藏

GPT-4 基于结构化叙事提示生成生活事件叙事:一项验证研究

GPT-4 Generated Narratives of Life Events using a Structured Narrative Prompt: A Validation Study

Virginia Modeling, Analysis, and Simulation Center(弗吉尼亚建模、分析与仿真中心);Old Dominion University(奥多明尼昂大学);College of Engineering, Montana State University(蒙大拿州立大学工程学院)

Christopher J. Lynch, Erik Jensen, Madison H. Munro, Virginia Zamponi, Joseph Martinez, Kevin O'Brien, Brandon Feldhaus, Katherine Smith, Ann Marie Reinhold, Ross Gore

AI总结:

本研究用零样本结构化提示让GPT-4生成24000条生活事件叙事,验证发现87.43%有效,训练9个ML模型自动分类时模型识别有效叙事表现好但识别无效叙事有困难,为LLM研究和NLP应用提供参考。

摘要:

大语言模型(LLM)在生成海量叙事方面发挥着关键作用,为系统探究其以叙事形式传达生活事件的有效性提供了便利。本研究中,我们采用零样本结构化叙事提示,利用OpenAI的GPT-4生成了24000条叙事。我们从该数据集中手动分类了2880条叙事,评估了它们在传达出生、死亡、雇佣和解雇事件方面的有效性。值得注意的是,87.43%的叙事充分传达了结构化提示的意图。为自动识别有效和无效叙事,我们在已分类的数据集上训练并验证了9个机器学习模型。借助这些模型,我们扩展了分析范围,对剩余21120条叙事进行分类预测。所有机器学习模型都能出色地将有效叙事判定为有效,但在同时将无效叙事判定为无效方面存在困难。我们的研究结果不仅推进了对LLM能力、局限性和有效性的研究,还为叙事生成和自然语言处理应用提供了实践见解。

英文摘要:

Large Language Models (LLMs) play a pivotal role in generating vast arrays of narratives, facilitating a systematic exploration of their effectiveness for communicating life events in narrative form. In this study, we employ a zero-shot structured narrative prompt to generate 24,000 narratives using OpenAI's GPT-4. From this dataset, we manually classify 2,880 narratives and evaluate their validity in conveying birth, death, hiring, and firing events. Remarkably, 87.43% of the narratives sufficiently convey the intention of the structured prompt. To automate the identification of valid and invalid narratives, we train and validate nine Machine Learning models on the classified datasets. Leveraging these models, we extend our analysis to predict the classifications of the remaining 21,120 narratives. All the ML models excelled at classifying valid narratives as valid, but experienced challenges at simultaneously classifying invalid narratives as invalid. Our findings not only advance the study of LLM capabilities, limitations, and validity but also offer practical insights for narrative generation and natural language processing applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑