作为元认知智能体的生成式AI：一项在模拟ICF考试表现上与人类参与者进行的比较混合方法研究
Generative AI as a metacognitive agent: A comparative mixed-method study with human participants on ICF-mimicking exam performance
AI总结：
研究通过混合方法比较人类与五种先进LLM在模拟ICF情境判断考试中的元认知表现，发现LLM整体更优、过度自信更低，但二者在模糊情境中均缺乏适应性。
摘要：
本研究考察了大语言模型相对于人类元认知的元认知能力，研究情境为国际教练联盟（International Coaching Federation，ICF）模拟考试，这是一项与教练胜任力相关的情境判断测试。采用混合方法，我们评估了人类参与者和五个先进LLM（GPT-4、Claude-3-Opus 3、Mistral Large、Llama 3和Gemini 1.5 Pro）的元认知表现，包括敏感性、概率预测准确性和偏差。结果表明，LLM在所有元认知指标上均优于人类，尤其是与人类相比，其过度自信程度更低。然而，LLM和人类在模糊情境中都表现出较弱的适应性，均紧密遵循预定义的决策框架。研究表明，生成式AI能够在没有意识觉知的情况下有效参与类人元认知过程。本文讨论了研究对开发AI模拟器的启示，这类模拟器可为掌握教练胜任力中的认知与元认知方面提供支架。更广泛地，结果的启示还涉及元认知模块的开发，这些模块将推动更自主、更具直觉的AI系统。
英文摘要：
This study investigates the metacognitive capabilities of Large Language Models relative to human metacognition in the context of the International Coaching Federation ICF mimicking exam, a situational judgment test related to coaching competencies. Using a mixed method approach, we assessed the metacognitive performance, including sensitivity, accuracy in probabilistic predictions, and bias, of human participants and five advanced LLMs (GPT-4, Claude-3-Opus 3, Mistral Large, Llama 3, and Gemini 1.5 Pro). The results indicate that LLMs outperformed humans across all metacognitive metrics, particularly in terms of reduced overconfidence, compared to humans. However, both LLMs and humans showed less adaptability in ambiguous scenarios, adhering closely to predefined decision frameworks. The study suggests that Generative AI can effectively engage in human-like metacognitive processing without conscious awareness. Implications of the study are discussed in relation to development of AI simulators that scaffold cognitive and metacognitive aspects of mastering coaching competencies. More broadly, implications of these results are discussed in relation to development of metacognitive modules that lead towards more autonomous and intuitive AI systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
