arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.12862cs.CLcs.AI
收藏

分析大型语言模型聊天机器人:一种使用概率测试的实验方法

Analyzing Large language models chatbots: An experimental approach using a probability test

Federal University of Santa Catarina(圣卡塔琳娜联邦大学)

Melise Peruchini, Julio Monteiro Teixeira

AI总结:

本研究通过概率测试(琳达问题和玛丽问题)实验,分析ChatGPT和Gemini两个大型语言模型聊天机器人的逻辑推理能力,发现它们在已知概率问题上表现良好,但在新问题上性能显著下降。

摘要:

本研究为定性实证研究,通过对两个不同的大型语言模型(LLM)聊天机器人——ChatGPT和Gemini——进行探索性测试来开展。方法论程序涉及基于设计有概率问题的提示词的探索性测试。以认知心理学中广泛认可的“琳达问题”(Linda Problem)为基础来创建测试,同时为本实验开发了一个新问题——“玛丽问题”(Mary Problem)。分析对象是每次聊天机器人交互所提供的输出所构成的数据集。分析的目的在于验证聊天机器人主要是采用与概率论一致的逻辑推理,还是更频繁地受到提示词中刻板文本描述的影响。研究结果提供了关于每个聊天机器人在处理逻辑和文本结构时所采用方法的见解,表明:虽然所分析的聊天机器人在一个广为人知的概率问题上表现令人满意,但在需要直接应用概率逻辑的新测试上,它们表现出显著较低的性能。

英文摘要:

This study consists of qualitative empirical research, conducted through exploratory tests with two different Large Language Models (LLMs) chatbots: ChatGPT and Gemini. The methodological procedure involved exploratory tests based on prompts designed with a probability question. The "Linda Problem", widely recognized in cognitive psychology, was used as a basis to create the tests, along with the development of a new problem specifically for this experiment, the "Mary Problem". The object of analysis is the dataset with the outputs provided by each chatbot interaction. The purpose of the analysis is to verify whether the chatbots mainly employ logical reasoning that aligns with probability theory or if they are more frequently affected by the stereotypical textual descriptions in the prompts. The findings provide insights about the approach each chatbot employs in handling logic and textual constructions, suggesting that, while the analyzed chatbots perform satisfactorily on a well-known probabilistic problem, they exhibit significantly lower performance on new tests that require direct application of probabilistic logic.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑