The rapid development and dynamic nature of large language models (LLMs) make it difficult for conventional quantitative benchmarks to accurately assess their capabilities. We propose report cards, which are human-interpretable, natural language summaries of model behavior for specific skills or topics. We develop a framework to evaluate report cards based on three criteria: specificity (ability to distinguish between models), faithfulness (accurate representation of model capabilities), and interpretability (clarity and relevance to humans). We also propose an iterative algorithm for generating report cards without human supervision and explore its efficacy by ablating various design choices. Through experimentation with popular LLMs, we demonstrate that report cards provide insights beyond traditional benchmarks and can help address the need for a more interpretable and holistic evaluation of LLMs.
Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives
Department of Computer Science, Columbia University（哥伦比亚大学计算机科学系）；Department of Applied Physics and Applied Mathematics, Columbia University（哥伦比亚大学应用物理与应用数学系）；Department of Psychology, Northwestern University（西北大学心理学系）
Large Language Models for Large-Scale, Rigorous Qualitative Analysis in Applied Health Services Research
Department of Computer Science（计算机科学系）；Stanford University（斯坦福大学）；Department of Biomedical Data Science（生物医学数据科学系）；Stanford School of Medicine（斯坦福医学院）；Harvard T.H. Chan School of Public Health（哈佛T.H. Chan公共卫生学院）；Impactivo LLC
Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: A Benchmark Study
Indian Institute of Technology Delhi（印度理工学院德里分校）；Indraprastha Institute of Information Technology Delhi（英德拉普拉斯塔信息技术学院）；YourDOST；All India Institute of Medical Sciences, Rishikesh（全印医学科学学院里希盖什分校）；All India Institute of Medical Sciences, New Delhi（全印医学科学学院新德里分校）
SmartBook: AI-Assisted Situation Report Generation for Intelligence Analysts
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Calgary（卡尔加里大学）；Northwestern University（西北大学）；Army Research Laboratory（陆军研究实验室）
CineMR: Tool-Integrated Vision-Language Reasoning for Quantitative Cardiac MRI Assessment
Institute for Artificial Intelligence, University of Central Florida（中佛罗里达大学人工智能研究所）；Nemours Cardiac Center, Nemours Children’s Hospital, Florida（佛罗里达州内穆尔斯儿童医院内穆尔斯心脏中心）；Children’s Heart Center, WVU Golisano Children’s, West Virginia（西弗吉尼亚大学戈利萨诺儿童医院儿童心脏中心）；Department of Clinical Sciences, College of Medicine, University of Central Florida（中佛罗里达大学医学院临床科学系）
