我的模型出了什么问题？用语义数据切片识别系统性问题
What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing
AI总结：
针对传统数据切片依赖现有特征难以定位模型系统性问题的局限，提出支持语义数据切片的SemSlicer框架，借助大语言模型按用户自定义标准生成切片，可低成本准确识别性能不佳的问题数据切片。
摘要：
机器学习模型会出错，但有时很难识别错误背后的系统性问题。从业者会开展误差分析、测试、审计、红队测试等各类工作，以提出模型可能（或已经）出问题的假设。为验证这些假设，从业者使用数据切片来识别相关样本。然而，传统数据切片受限于可用特征与可编程切片函数。本研究提出SemSlicer（语义切片器），这是一个支持语义数据切片的框架，无需依赖现有特征即可识别语义一致的切片。SemSlicer利用大语言模型标注数据集，并根据任意用户自定义切片标准生成切片。实验表明，SemSlicer能以低成本生成准确的切片，可在不同设计维度间灵活权衡，能可靠识别性能不佳的数据切片，还能帮助从业者找到反映系统性问题的有用数据切片。
英文摘要：
Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including error analysis, testing, auditing, and red-teaming, to form hypotheses of what can go (or has gone) wrong with their models. To validate these hypotheses, practitioners employ data slicing to identify relevant examples. However, traditional data slicing is limited by available features and programmatic slicing functions. In this work, we propose SemSlicer, a framework that supports semantic data slicing, which identifies a semantically coherent slice, without the need for existing features. SemSlicer uses Large Language Models to annotate datasets and generate slices from any user-defined slicing criteria. We show that SemSlicer generates accurate slices with low cost, allows flexible trade-offs between different design dimensions, reliably identifies under-performing data slices, and helps practitioners identify useful data slices that reflect systematic problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
