人类输入，人类输出：论 GPT 在成功与失败中向常识的趋同
Humans in Humans Out: On GPT Converging Toward Common Sense in both Success and Failure
AI总结：
本研究基于 Erotetic Theory of Reason (ETR) 提出 ETR61 基准测试 GPT-3、GPT-3.5 和 GPT-4，发现更大规模的 LLMs 在提升正确推理能力的同时，也展现出更多类似人类的谬误判断倾向，且受 ETR 启发的提示工程可减少此类错误。
摘要：
计算规模和微调的增加使得像 GPT 这样的大型语言模型（LLMs）的输出质量有了显著提高。鉴于 GPT-3 和 GPT-4 都是在大量人类生成的文本上训练的，我们可以问，它们的输出在多大程度上反映了人类的思维模式，包括正确和错误的情况。 Erotetic Theory of Reason (ETR) 提供了一个符号生成模型，涵盖了命题、量化和概率推理以及决策制定中人类成功和失败的思维。我们向 GPT-3、GPT-3.5 和 GPT-4 提出了来自最近一本关于 ETR 的书中的 61 个核心推理和判断问题，包括实验验证的人类判断数据点和 ETR 预测的推断数据点，以及正确的推理模式和谬误和框架效应（ETR61 基准）。ETR61 包括 Wason 的卡片任务、幻觉推理、诱饵效应和机会成本忽视等经典问题。GPT-3 显示出 59% 的例子有 ETR 预测的输出，在 GPT-3.5 中上升到 77%，在 GPT-4 中为 75%。值得注意的是，类似人类的谬误判断的产生从 GPT-3 的 18% 增加到 GPT-3.5 的 33% 和 GPT-4 的 34%。这表明，更大、更先进的 LLMs 可能会发展出更类似人类错误的倾向，因为相关的思维模式是人类生成的训练数据中固有的。根据 ETR，成功和不成功的普通推理涉及相同的基本模式，因此“坏”案例可以矛盾地从“好”案例中学习。我们进一步提供了初步证据，表明受 ETR 启发的提示工程可以减少这些错误的发生。
英文摘要：
Increase in computational scale and fine-tuning has seen a dramatic improvement in the quality of outputs of large language models (LLMs) like GPT. Given that both GPT-3 and GPT-4 were trained on large quantities of human-generated text, we might ask to what extent their outputs reflect patterns of human thinking, both for correct and incorrect cases. The Erotetic Theory of Reason (ETR) provides a symbolic generative model of both human success and failure in thinking, across propositional, quantified, and probabilistic reasoning, as well as decision-making. We presented GPT-3, GPT-3.5, and GPT-4 with 61 central inference and judgment problems from a recent book-length presentation of ETR, consisting of experimentally verified data-points on human judgment and extrapolated data-points predicted by ETR, with correct inference patterns as well as fallacies and framing effects (the ETR61 benchmark). ETR61 includes classics like Wason's card task, illusory inferences, the decoy effect, and opportunity-cost neglect, among others. GPT-3 showed evidence of ETR-predicted outputs for 59% of these examples, rising to 77% in GPT-3.5 and 75% in GPT-4. Remarkably, the production of human-like fallacious judgments increased from 18% in GPT-3 to 33% in GPT-3.5 and 34% in GPT-4. This suggests that larger and more advanced LLMs may develop a tendency toward more human-like mistakes, as relevant thought patterns are inherent in human-produced training data. According to ETR, the same fundamental patterns are involved both in successful and unsuccessful ordinary reasoning, so that the "bad" cases could paradoxically be learned from the "good" cases. We further present preliminary evidence that ETR-inspired prompt engineering could reduce instances of these mistakes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
