作为研究科学家的ChatGPT：探究GPT作为研究馆员、研究伦理学家、数据生成器和数据预测器的能力
ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictor
AI总结：
本研究以心理学为测试领域，系统探究GPT-3.5和GPT-4在科研四大环节的能力，发现其有幻觉问题但伦理检测、数据生成有一定能力，无法预测训练外新数据模式。
摘要：
ChatGPT作为研究科学家的水平如何？我们以心理学为测试领域，系统探究了GPT-3.5和GPT-4在科学研究四大核心环节中的能力：作为研究馆员、研究伦理学家、数据生成器和新型数据预测器的表现。\n在研究1（研究馆员）中，与人类研究者不同，GPT-3.5和GPT-4会产生幻觉，分别有36.0%和5.4%的概率权威式地生成虚构参考文献，不过GPT-4展现出了逐渐承认自身虚构内容的能力。\n在研究2（研究伦理学家）中，GPT-4（但GPT-3.5不行）能够检测虚构研究方案中的p值操纵等违规行为，对明显呈现的问题纠正率达88.6%，对隐蔽呈现的问题纠正率为72.6%。\n在研究3（数据生成器）中，两个模型都一致复现了此前在大型语言语料库中发现的文化偏见模式，表明ChatGPT能够模拟已知结果，这是其在数据生成及假设生成等技能方面具备实用性的前提。\n相比之下，在研究4（新型数据预测器）中，两个模型都无法成功预测训练数据中不存在的新结果，且在预测新颖性更高与更低的结果时，都未表现出对实质性新信息的利用。\n综上，这些结果表明，GPT是一位有缺陷但进步迅速的馆员，目前已是一名合格的研究伦理学家，能够在具有已知特征的简单领域中生成数据，但不擅长预测实证数据的新模式以辅助未来实验。
英文摘要：
How good a research scientist is ChatGPT? We systematically probed the capabilities of GPT-3.5 and GPT-4 across four central components of the scientific process: as a Research Librarian, Research Ethicist, Data Generator, and Novel Data Predictor, using psychological science as a testing field. In Study 1 (Research Librarian), unlike human researchers, GPT-3.5 and GPT-4 hallucinated, authoritatively generating fictional references 36.0% and 5.4% of the time, respectively, although GPT-4 exhibited an evolving capacity to acknowledge its fictions. In Study 2 (Research Ethicist), GPT-4 (though not GPT-3.5) proved capable of detecting violations like p-hacking in fictional research protocols, correcting 88.6% of blatantly presented issues, and 72.6% of subtly presented issues. In Study 3 (Data Generator), both models consistently replicated patterns of cultural bias previously discovered in large language corpora, indicating that ChatGPT can simulate known results, an antecedent to usefulness for both data generation and skills like hypothesis generation. Contrastingly, in Study 4 (Novel Data Predictor), neither model was successful at predicting new results absent in their training data, and neither appeared to leverage substantially new information when predicting more versus less novel outcomes. Together, these results suggest that GPT is a flawed but rapidly improving librarian, a decent research ethicist already, capable of data generation in simple domains with known characteristics but poor at predicting novel patterns of empirical data to aid future experimentation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
