Assessment of proficiency of the learner is an essential part of Intelligent Tutoring Systems (ITS). We use Item Response Theory (IRT) in computer-aided language learning for assessment of student ability in two contexts: in test sessions, and in exercises during practice sessions. Exhaustive testing across a wide range of skills can provide a detailed picture of proficiency, but may be undesirable for a number of reasons. Therefore, we first aim to replace exhaustive tests with efficient but accurate adaptive tests. We use learner data collected from exhaustive tests under imperfect conditions, to train an IRT model to guide adaptive tests. Simulations and experiments with real learner data confirm that this approach is efficient and accurate. Second, we explore whether we can accurately estimate learner ability directly from the context of practice with exercises, without testing. We transform learner data collected from exercise sessions into a form that can be used for IRT modeling. This is done by linking the exercises to {\em linguistic constructs}; the constructs are then treated as "items" within IRT. We present results from large-scale studies with thousands of learners. Using teacher assessments of student ability as "ground truth," we compare the estimates obtained from tests vs. those from exercises. The experiments confirm that the IRT models can produce accurate ability estimation based on exercises.
An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020)
Inserm(法国国家健康与医学研究院);Université Paul Sabatier Toulouse 3(图卢兹第三大学);GIS Genotoul(Genotoul科学研究集团);Université Fédérale de Toulouse(图卢兹联邦大学);Groupement d’Intérêt Public e.santé Occitanie(奥克西塔尼电子健康公共利益集团);IRIT-CNRS(图卢兹信息研究所-法国国家科学研究中心)
Landmark-free Assessment of Lower-limb Alignment with Implicit Neural Shape Functions from Knee Radiographs
Division of Informatics, Imaging and Data Sciences, The University of Manchester(曼彻斯特大学信息学、影像与数据科学部);Research Unit of Health Sciences and Technology, University of Oulu(奥卢大学健康科学与技术研究部);Medical Research Center Oulu, University of Oulu and Oulu University Hospital(奥卢大学与奥卢大学医院医学研究中心);Department of Trauma and Orthopaedics, Stockport NHS Foundation Trust, Stepping Hill Hospital(斯泰平希尔医院斯托克波特NHS基金会创伤与骨科);School of Health and Society, University of Salford(索尔福德大学健康与社会学院);School of Biological Sciences, The University of Manchester(曼彻斯特大学生物科学学院);Weill Cornell Medicine, Cornell University(康奈尔大学威尔康奈尔医学院)
Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practice
Euler Institute, Università della Svizzera italiana(欧拉研究所,瑞士大学);National Institute of Informatics(国家信息研究所);Brown University(布朗大学);University of Tübingen(图宾根大学);Insubria University(因斯布里亚大学)
Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice
IIHR – Hydroscience & Engineering, University of Iowa(爱荷华大学IIHR水文科学与工程研究所);Civil and Environmental Engineering, University of Iowa(爱荷华大学土木与环境工程学院);Electrical and Computer Engineering, University of Iowa(爱荷华大学电气与计算机工程学院)
本文评估基于 GPT-4 的 ChatGPT 在美国 FE 环境考试中的表现,发现非侵入式提示词修改可显著提升准确率,并探讨其教育应用、局限及考试诚信意义。