延迟-响应理论模型：通过响应准确性和思维链长度评估大语言模型
Latency-Response Theory Model: Evaluating Large Language Models via Response Accuracy and Chain-of-Thought Length
AI总结：
针对大语言模型评估问题，提出延迟-响应理论（LaRT）模型，联合考虑响应准确性和思维链长度，通过引入关键参数建模，推导高效算法估计参数，经理论和模拟研究验证其优势，实际数据评估中表现优于项目反应理论（IRT）。
摘要：
大语言模型的扩散需要有效的评估方法。项目反应理论（IRT）模型与计算机自适应测试最近成为通过响应准确性评估大语言模型的有前景框架。除了简单的响应准确性，大语言模型的思维链长度是其推理能力的重要指标。为利用思维链长度信息辅助大语言模型评估，我们提出延迟-响应理论（LaRT）模型，通过引入潜在能力和潜在速度之间的关键相关参数，联合建模响应准确性和思维链长度。我们推导了用于参数估计的高效随机近似期望最大化算法，建立了潜在能力和潜在速度参数的严格可识别性结果。通过理论渐近分析和模拟研究，证明LaRT在潜在特质估计的准确性和置信区间方面优于IRT。在实际数据中评估发现，LaRT在多个关键评估指标上优于IRT且产生不同的大语言模型排名。代码和数据可获取。
英文摘要：
The proliferation of Large Language Models (LLMs) necessitates valid evaluation methods to provide guidance for both downstream applications and actionable future improvements. The Item Response Theory (IRT) model with Computerized Adaptive Testing has recently emerged as a promising framework for evaluating LLMs via their response accuracy. Beyond simple response accuracy, LLMs' chain of thought (CoT) lengths serve as a vital indicator of their reasoning ability. To leverage the CoT length information to assist LLM evaluation, we propose the \textbf{La}tency-\textbf{R}esponse \textbf{T}heory (LaRT) model, which jointly models both the response accuracy and CoT length by introducing a key correlation parameter between the latent ability and the latent speed. We derive an efficient stochastic approximation Expectation-Maximization algorithm for parameter estimation. We establish rigorous identifiability results for the latent ability and latent speed parameters to ensure the statistical validity of their estimation. Through both theoretical asymptotic analyses and simulation studies, we demonstrate LaRT's advantages over IRT in terms of superior estimation accuracy and shorter confidence intervals for latent trait estimation. To evaluate LaRT in real data, we collect responses from diverse LLMs on popular benchmark datasets. We find that LaRT yields different LLM rankings than IRT and outperforms IRT across multiple key evaluation metrics including predictive power, item efficiency, ranking validity, and LLM evaluation efficiency. Code and data are available at https://github.com/Toby-X/Latency-Response-Theory-Model
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
