This paper follows previous research we have already performed in the area of Bayesian networks models for CAT. We present models using Item Response Theory (IRT - standard CAT method), Bayesian networks, and neural networks. We conducted simulated CAT tests on empirical data. Results of these tests are presented for each model separately and compared.
In this paper we follow our previous research in the area of Computerized Adaptive Testing (CAT). We present three different methods for CAT. One of them, the item response theory, is a well established method, while the other two, Bayesian and neural networks, are new in the area of educational testing. In the first part of this paper, we present the concept of CAT and its advantages and disadvantages. We collected data from paper tests performed with grammar school students. We provide the summary of data used for our experiments in the second part. Next, we present three different model typ…
Institute of Information Theory and Automation（信息理论与自动化研究所）；Academy of Sciences of the Czech Republic（捷克科学院）
Computerized adaptive testing (CAT) is an interesting and promising approach to testing human abilities. In our research we use Bayesian networks to create a model of tested humans. We collected data from paper tests performed with grammar school students. In this article we first provide the summary of data used for our experiments. We propose several different Bayesian networks, which we tested and compared by cross-validation. Interesting results were obtained and are discussed in the paper. The analysis has brought a clearer view on the model selection problem. Future research is outlined …
Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic（捷克共和国科学院信息理论与自动化研究所）
In our previous work we have shown how Bayesian networks can be used for adaptive testing of student skills. Later, we have taken the advantage of monotonicity restrictions in order to learn models fitting data better. This article provides a synergy between these two phases as it evaluates Bayesian network models used for computerized adaptive testing and learned with a recently proposed monotonicity gradient algorithm. This learning method is compared with another monotone method, the isotonic regression EM algorithm. The quality of methods is empirically evaluated on a large data set of the…
Computerized adaptive testing (CAT) refers to a form of tests that are personalized to every student/test taker. CAT methods adaptively select the next most informative question/item for each student given their responses to previous questions, effectively reducing test length. Existing CAT methods use item response theory (IRT) models to relate student ability to their responses to questions and static question selection algorithms designed to reduce the ability estimation error as quickly as possible; therefore, these algorithms cannot improve by learning from large-scale student response da…
AgentCAT: Simulating Computerized Adaptive Testing via Multi-Agent Large Language Models
Institute for Clarity in Documentation（文档清晰度研究所）；Inria Paris-Rocquencourt（法国国家信息与自动化研究所巴黎-罗康库尔中心）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕默研究实验室）；State Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology, Institute of Physical Science and Information Technology, Anhui University（光电信息获取与防护技术国家重点实验室，物理科学与信息技术学院，安徽大学）；School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）；State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China, the Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（认知智能国家重点实验室，中国科学技术大学，人工智能研究院，合肥综合性国家科学中心）
University of Science and Technology of China（中国科学技术大学）；Anhui University（安徽大学）；IFLYTEK Research（科大讯飞研究院）
Computerized Adaptive Testing (CAT) is emerging as a promising testing application in many scenarios, such as education, game and recruitment, which targets at diagnosing the knowledge mastery levels of examinees on required concepts. It shows the advantage of tailoring a personalized testing procedure for each examinee, which selects questions step by step, depending on her performance. While there are many efforts on developing CAT systems, existing solutions generally follow an inflexible model-specific fashion. That is, they need to observe a specific cognitive model which can estimate exa…
Leveraging Computerized Adaptive Testing for Cost-effective Evaluation of Large Language Models in Medical Benchmarking
Institute of Medical Education, Health Science Center, Peking University（北京大学医学教育研究院、健康科学中心）；School of Public Health, Peking University（北京大学公共卫生学院）；Peking University Health Science Center-Chaoxing Joint Laboratory for Digital and Smart Medical（北京大学健康科学中心-超星数字与智能医疗联合实验室）；Graduate School of Education, Peking University（北京大学教育学院）
Diffusion-Inspired Cold Start with Sufficient Prior in Computerized Adaptive Testing
Anhui University（安徽大学）；The Institutes of Physical Science and Information Technology, Anhui University（安徽大学物理科学与信息技术研究院）；School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）
Axiomatization has been widely used for testing logical implications. This paper suggests a non-axiomatic method, the chase, to test if a new dependency follows from a given set of probabilistic dependencies. Although the chase computation may require exponential time in some cases, this technique is a powerful tool for establishing nontrivial theoretical results. More importantly, this approach provides valuable insight into the intriguing connection between relational databases and probabilistic reasoning systems.
University of Kentucky（肯塔基大学）；Duke University（杜克大学）
We examine the computational complexity of testing and finding small plans in probabilistic planning domains with succinct representations. We find that many problems of interest are complete for a variety of complexity classes: NP, co-NP, PP, NP^PP, co-NP^PP, and PSPACE. Of these, the probabilistic classes PP and NP^PP are likely to be of special interest in the field of uncertainty in artificial intelligence and are deserving of additional study. These results suggest a fruitful direction of future algorithmic development.
In this paper, we propose a simple yet effective method to deal with the violation of the Closed-World Assumption for a classifier. Previous works tend to apply a threshold either on the classification scores or the loss function to reject the inputs that violate the assumption. However, these methods cannot achieve the low False Positive Ratio (FPR) required in safety applications. The proposed method is a rejection option based on hypothesis testing with probabilistic networks. With probabilistic networks, it is possible to estimate the distribution of outcomes instead of a single output. By…
Cognitive Systems Laboratory（认知系统实验室）；Computer Science Department（计算机科学系）；UCLA（加州大学洛杉矶分校）
This paper concerns the probabilistic evaluation of the effects of actions in the presence of unmeasured variables. We show that the identification of causal effect between a singleton variable X and a set of variables Y can be accomplished systematically, in time polynomial in the number of variables in the graph. When the causal effect is identifiable, a closed-form expression can be obtained for the probability that the action will achieve a specified goal, or a set of goals.
Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
Trismik；Leverhulme Centre for the Future of Intelligence, University of Cambridge（Leverhulme智能未来研究中心，剑桥大学）；Universitat Politècnica de València（瓦伦西亚理工大学）；University of Cambridge（剑桥大学）
We propose a new probabilistic graphical model that jointly models the difficulties of questions, the abilities of participants and the correct answers to questions in aptitude testing and crowdsourcing settings. We devise an active learning/adaptive testing scheme based on a greedy minimization of expected model entropy, which allows a more efficient resource allocation by dynamically choosing the next question to be asked based on the previous responses. We present experimental results that confirm the ability of our model to infer the required parameters and demonstrate that the adaptive te…
Arizona State University（亚利桑那州立大学）；Yuan Ze University（元智大学）；Lockheed Martin（洛克希德·马丁公司）
Requirements driven search-based testing (also known as falsification) has proven to be a practical and effective method for discovering erroneous behaviors in Cyber-Physical Systems. Despite the constant improvements on the performance and applicability of falsification methods, they all share a common characteristic. Namely, they are best-effort methods which do not provide any guarantees on the absence of erroneous behaviors (falsifiers) when the testing budget is exhausted. The absence of finite time guarantees is a major limitation which prevents falsification methods from being utilized …
