目的:本研究考察领先的中国和西方大型语言模型在理解和应用中国社会工作原则方面的表现,重点关注其在非西方专业环境下的基础知识。我们测试了发展中国家的文化背景是否影响模型推理和准确性。方法:使用已出版的中国社会工作考试自学版本(160题),涵盖法理学和应用知识,我们对8个基于云的大型语言模型(4个中国模型和4个西方模型)实施了三种测试条件。我们根据官方指南审查其回答,并评估其解释的推理质量。结果:7个模型在两个部分均超过60分的及格线。中国模型在法理学上表现更好(中位数=77.0 vs. 70.3),但在应用知识上略低(中位数=65.5 vs. 67.0)。两组均表现出文化偏见,特别是在性别平等和家庭动态方面。模型展现出较强的专业术语知识,但在特定文化的干预上存在困难。错误答案中的有效推理比例从16.4%到45.0%不等。结论:尽管中国和西方模型均具备中国社会工作原则的基础知识,但技术语言熟练度并不能确保文化胜任力。中国模型在法规内容上表现出优势,但两组在具有文化细微差别的实践场景中均面临挑战。这些发现有助于指导将AI负责任地融入跨文化社会工作实践。
英文摘要:
Objective: This study examines how well leading Chinese and Western large language models understand and apply Chinese social work principles, focusing on their foundational knowledge within a non-Western professional setting. We test whether the cultural context in the developing country influences model reasoning and accuracy.
Method: Using a published self-study version of the Chinese National Social Work Examination (160 questions) covering jurisprudence and applied knowledge, we administered three testing conditions to eight cloud-based large language models - four Chinese and four Western. We examined their responses following official guidelines and evaluated their explanations' reasoning quality.
Results: Seven models exceeded the 60-point passing threshold in both sections. Chinese models performed better in jurisprudence (median = 77.0 vs. 70.3) but slightly lower in applied knowledge (median = 65.5 vs. 67.0). Both groups showed cultural biases, particularly regarding gender equality and family dynamics. Models demonstrated strong professional terminology knowledge but struggled with culturally specific interventions. Valid reasoning in incorrect answers ranged from 16.4% to 45.0%.
Conclusions: While both Chinese and Western models show foundational knowledge of Chinese social work principles, technical language proficiency does not ensure cultural competence. Chinese models demonstrate advantages in regulatory content, yet both Chinese and Western models struggle with culturally nuanced practice scenarios. These findings contribute to informing responsible AI integration into cross-cultural social work practice.
Are LLMs Ready for Computer Science Education? A Cross-Domain, Cross-Lingual and Cognitive-Level Evaluation Using Professional Certification Exams
Zhongshan Ophthalmic Center, Sun Yat-sen University(中山大学中山眼科中心);Faculty of Data Science, City University of Macau(澳门城市大学数据科学学院);Department of Accounting, Faculty of Business and Economics, The University of Melbourne(墨尔本大学商学院会计系);School of Information Technology, Faculty of Business and Hospitality, Torrens University Australia(托伦斯大学商业与酒店管理学院信息技术学院)
Beyond Instrumental and Substitutive Paradigms: Introducing Machine Culture as an Emergent Phenomenon in Large Language Models
Institute of Neuroscience, Chinese Academy of Sciences(中国科学院神经科学研究所);University of Cambridge(剑桥大学);Chinese University of Hong Kong(香港中文大学);Institute of Public Opinion Survey, Hong Kong SAR(香港特别行政区民意调查研究所);Tsinghua University(清华大学)
Effectiveness of Large Language Models in Simulating Regional Psychological Structures: An Empirical Examination of Personality and Subjective Well-being
Tsinghua University(清华大学);Neoma Business School(诺欧商学院);Beijing Technology and Business University(北京工商大学);Beijing Normal University(北京师范大学)