Forma mentis 网络通过可解释人工智能预测人类和 GPT 模拟评分者对短文本的创造力评分
Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters
AI总结：
本研究利用 TFMN 特征、XGBoost 与 SHAP 比较人类和 GPT-3.5 对短故事的创造力评分，揭示二者依赖的特征模式差异及 GPT-3.5 对齐人类判断的局限。
摘要：
创造力是人类认知的一项基本技能。我们使用文本 forma mentis 网络（TFMN）从约一千篇由人类和 GPT3.5 生成的故事中提取网络（语义/句法关联）和情感特征。利用可解释人工智能（XAI），我们检验与 Mednick 创造力联想理论相关的特征是否能够解释人类和 GPT-3.5 给出的创造力评分。我们使用 XGBoost 考察了三种场景：（i）人类对人类故事的评分，（ii）GPT-3.5 对人类故事的评分，以及（iii）GPT-3.5 对 GPT 生成故事的评分。研究结果表明，GPT-3.5 的评分与人类评分存在显著差异，这不仅体现在相关性方面，也体现在通过 XAI 方法识别出的特征模式方面。GPT-3.5 偏爱“它自己的”故事，并且对人类故事的评分方式与人类不同。基于 SHAP 分数的特征重要性分析显示：（i）网络特征对人类创造力评分具有更强的预测力，同时对 GPT-3.5 评价人类故事的评分也更具预测性；（ii）在 GPT-3.5 评价其自身故事时，情感特征比语义/句法网络结构发挥了更大作用。这些定量结果凸显了 GPT-3.5 在与人类创造力评估保持一致方面的关键局限。我们强调，在使用 GPT-3.5 评估和生成创意内容时需要谨慎，因为它尚未捕捉到人类创造力所特有的微妙复杂性。
英文摘要：
Creativity is a fundamental skill of human cognition. We use textual forma mentis networks (TFMN) to extract network (semantic/syntactic associations) and emotional features from approximately one thousand human- and GPT3.5-generated stories. Using Explainable Artificial Intelligence (XAI), we test whether features relative to Mednick's associative theory of creativity can explain creativity ratings assigned by humans and GPT-3.5. Using XGBoost, we examine three scenarios: (i) human ratings of human stories, (ii) GPT-3.5 ratings of human stories, and (iii) GPT-3.5 ratings of GPT-generated stories. Our findings reveal that GPT-3.5 ratings differ significantly from human ratings not only in terms of correlations but also because of feature patterns identified with XAI methods. GPT-3.5 favours 'its own' stories and rates human stories differently from humans. Feature importance analysis with SHAP scores shows that: (i) network features are more predictive for human creativity ratings but also for GPT-3.5's ratings of human stories; (ii) emotional features played a greater role than semantic/syntactic network structure in GPT-3.5 rating its own stories. These quantitative results underscore key limitations in GPT-3.5's ability to align with human assessments of creativity. We emphasise the need for caution when using GPT-3.5 to assess and generate creative content, as it does not yet capture the nuanced complexity that characterises human creativity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
