iScore：用于解释语言模型如何自动为摘要评分的可视化分析工具
iScore: Visual Analytics for Interpreting How Language Models Automatically Score Summaries
AI总结：
针对大语言模型用于教育摘要评分时透明度不足、难建立信任的问题，本研究通过用户中心设计开发了交互式可视化分析工具iScore，支持多摘要对比、评分溯源与权重可视化，经部署验证可提升模型准确率并助力工程师理解信任LLM。
摘要：
近年来大语言模型（LLM）的快速普及，促使学习工程师将其整合到能自动为摘要写作评分的自适应教育工具中。在关键学习环境中部署LLM之前，理解和评估这类模型至关重要，但其前所未有的规模和不断增长的参数量降低了透明度，且在模型表现不佳时会阻碍用户建立信任。通过与多位构建并部署摘要评分LLM的学习工程师开展以用户为中心的协作式设计流程，我们明确了模型可解释性相关的核心设计挑战与目标，包括大文本输入的聚合、评分溯源追踪，以及LLM可解释性方法的规模化扩展。为解决这些问题，我们开发了iScore——一款交互式可视化分析工具，可供学习工程师同时上传、评分并对比多篇摘要。工具内紧密整合的多个视图支持用户迭代修改摘要措辞、追踪LLM评分的对应变化，并在多个抽象层级可视化模型权重。为验证我们的方法，我们让三名学习工程师使用iScore为期一个月。我们展示了一个案例研究：通过与iScore交互，一名学习工程师将其LLM的评分准确率提升了3个百分点。最后，我们对学习工程师开展了定性访谈，结果显示iScore帮助他们在部署过程中理解、评估LLM并建立对模型的信任。
英文摘要：
The recent explosion in popularity of large language models (LLMs) has inspired learning engineers to incorporate them into adaptive educational tools that automatically score summary writing. Understanding and evaluating LLMs is vital before deploying them in critical learning environments, yet their unprecedented size and expanding number of parameters inhibits transparency and impedes trust when they underperform. Through a collaborative user-centered design process with several learning engineers building and deploying summary scoring LLMs, we characterized fundamental design challenges and goals around interpreting their models, including aggregating large text inputs, tracking score provenance, and scaling LLM interpretability methods. To address their concerns, we developed iScore, an interactive visual analytics tool for learning engineers to upload, score, and compare multiple summaries simultaneously. Tightly integrated views allow users to iteratively revise the language in summaries, track changes in the resulting LLM scores, and visualize model weights at multiple levels of abstraction. To validate our approach, we deployed iScore with three learning engineers over the course of a month. We present a case study where interacting with iScore led a learning engineer to improve their LLM's score accuracy by three percentage points. Finally, we conducted qualitative interviews with the learning engineers that revealed how iScore enabled them to understand, evaluate, and build trust in their LLMs during deployment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
