Generating rationales that justify scoring decisions has been a promising way to facilitate explainability in automated scoring systems. However, existing methods do not match the accuracy of classifier-based methods. Plus, the generated rationales often contain hallucinated information. To address these issues, we propose a novel framework capable of generating more faithful rationales and, more importantly, matching performance with classifier-based black-box scoring systems. We first mimic the human assessment process by querying Large Language Models (LLMs) to generate a thought tree. We then summarise intermediate assessment decisions from each thought tree path for creating synthetic rationale data and rationale preference data. Finally, we utilise the generated synthetic data to calibrate LLMs through a two-step training process: supervised fine-tuning and preference optimization. Extensive experimental results demonstrate that our framework achieves a 38% assessment performance improvement in the QWK score compared to prior work while producing higher-quality rationales, as recognised by human evaluators and LLMs. Our work sheds light on the effectiveness of performing preference optimization using synthetic preference data obtained from thought tree paths. Data and code are available at https://github.com/lijiazheng99/thought_tree_assessment.
From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment
Technical University of Munich（慕尼黑技术大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Lund University（吕勒奥大学）；University of Tübingen（图宾根大学）；Stanford Graduate School of Education（斯坦福大学教育研究生院）；Harvard Graduate School of Education（哈佛大学教育研究生院）
PRISM: Calibrating Black-Box Judges with Counterfactual Probes
Massachusetts Institute of Technology（麻省理工学院）；Sakana AI；Lab for Information and Decision Systems（信息与决策系统实验室）；Statistics and Data Science Center（统计与数据科学中心）；EECS（电气工程与计算机科学系）
PsychePass: Calibrating LLM Therapeutic Competence via Trajectory-Anchored Tournaments
School of Computer Science and Engineering, Central South University（中南大学计算机科学与工程学院）；Lingxin AI（灵心AI）；South China Normal University（华南师范大学）；CoAI Group, DCST, IAI, BNRIST, Tsinghua University（清华人工智能研究院）
Self-Calibrating Neural-Probabilistic Model for Authorship Verification Under Covariate Shift
Ruhr University Bochum（波鸿鲁尔大学）；Bucknell University（巴克内尔大学）
我们正在解决两个基本问题，在作者身份验证（AV）：主题的变化和误校准。两个有争议的文本的主题的变化是大多数AV系统错误的主要原因。此外，观察到由深度学习AV机制产生的潜在概率估计常常与相应训练数据中的实际案例计数不匹配。因此，概率估计的校准很差。我们正在从PAN 2020扩展我们的框架，以包括贝叶斯因子评分（BFS）和不确定性适应层（UAL）来解决这两个问题。使用2020/21 PAN AV共享任务数据的实验表明，所提出的方法显着降低了对主题变化的敏感性，并显着提高了系统的校准。
Generating Chain-of-Thoughts with a Pairwise-Comparison Approach to Searching for the Most Promising Intermediate Thought
Center for Advanced Intelligence Project, RIKEN（理化学研究所先进智能项目中心）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；Graduate School of Frontier Sciences, The University of Tokyo（东京大学前沿科学研究生院）
Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of New South Wales（新南威尔士大学）；Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）
