大语言模型在事故叙述分析中的应用——ChatGPT、BARD与GPT-4的对比研究
Large Language Models in Analyzing Crash Narratives -- A Comparative Study of ChatGPT, BARD and GPT-4
AI总结：
本研究对比ChatGPT、BARD、GPT-4三款大语言模型在100份事故叙述的信息提取任务中的表现，发现简单问题相似度高、复杂问题差异大，建议采用多模型提取信息并谨慎获取关键安全数据。
摘要：
在交通安全研究中，利用文本分析从事故叙述中提取信息是一种常见做法。随着大语言模型（LLM）的最新发展，了解主流大语言模型接口在事故叙述的信息分类或提取任务中的表现具有实用价值。为探究这一问题，本研究采用了三款最热门的公开可用大语言模型接口——ChatGPT、BARD和GPT-4。本研究基于来自艾奥瓦州和堪萨斯州的100份事故叙述，考察了上述模型在提取事故相关信息、回答事故相关查询方面的效用与边界。研究过程中评估了它们的能力与局限性，并对比了它们对查询的响应结果。研究围绕事故叙述提出了五个问题：1）谁负有责任？2）碰撞方式是什么？3）事故是否发生在施工区？4）事故是否涉及行人？5）事故中的有害事件序列是什么？对于第1至第4个问题，大语言模型之间的整体相似度分别为70%、35%、96%和89%。在回答需要二元响应的直接问题时，模型间的相似度更高；而对于复杂问题，相似度则显著降低。为对比第5个问题的回答结果，研究分析了网络图和中心性度量指标。三款大语言模型生成的网络图并不总是相似，尽管它们有时会出现相同的高影响事件，这些事件具有较高的入度、出度和介数中心性。本研究建议采用多个模型从事故叙述中提取可靠信息，同时在使用这些接口获取关键安全相关信息时必须保持谨慎。
英文摘要：
In traffic safety research, extracting information from crash narratives using text analysis is a common practice. With recent advancements of large language models (LLM), it would be useful to know how the popular LLM interfaces perform in classifying or extracting information from crash narratives. To explore this, our study has used the three most popular publicly available LLM interfaces- ChatGPT, BARD and GPT4. This study investigated their usefulness and boundaries in extracting information and answering queries related to accidents from 100 crash narratives from Iowa and Kansas. During the investigation, their capabilities and limitations were assessed and their responses to the queries were compared. Five questions were asked related to the narratives: 1) Who is at-fault? 2) What is the manner of collision? 3) Has the crash occurred in a work-zone? 4) Did the crash involve pedestrians? and 5) What are the sequence of harmful events in the crash? For questions 1 through 4, the overall similarity among the LLMs were 70%, 35%, 96% and 89%, respectively. The similarities were higher while answering direct questions requiring binary responses and significantly lower for complex questions. To compare the responses to question 5, network diagram and centrality measures were analyzed. The network diagram from the three LLMs were not always similar although they sometimes have the same influencing events with high in-degree, out-degree and betweenness centrality. This study suggests using multiple models to extract viable information from narratives. Also, caution must be practiced while using these interfaces to obtain crucial safety related information.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
