理解与支持 Multiverse Analysis 中的调试工作流
Understanding and Supporting Debugging Workflows in Multiverse Analysis
AI总结:
本文针对多重宇宙分析中调试反馈滞后、元数据诊断规模大的问题,原型化 Multiverse Debugger,并通过 n=13 的定性研究揭示调试工作流挑战与系统设计启示。
摘要:
Multiverse analysis(多重宇宙分析)是一种统计分析范式,它并行考虑合理分析选择的所有组合,有望提升透明度与可复现性。尽管近期工具可帮助分析人员指定多重宇宙分析,但它们在实践中仍难以使用。在本研究中,我们将调试识别为一个关键障碍,其原因在于从运行分析到发现缺陷之间存在延迟,并且诊断缺陷需要处理大规模元数据。为应对这些挑战,我们原型化了一个命令行界面工具 Multiverse Debugger,用于帮助诊断多重宇宙中的缺陷并传播修复。在一项定性实验室研究(n=13)中,我们将 Multiverse Debugger 作为探针,构建了调试工作流模型,并识别出具体挑战,包括难以理解多重宇宙的组成。我们最后总结了对未来多重宇宙分析创作系统的设计启示。
英文摘要:
Multiverse analysis, a paradigm for statistical analysis that considers all combinations of reasonable analysis choices in parallel, promises to improve transparency and reproducibility. Although recent tools help analysts specify multiverse analyses, they remain difficult to use in practice. In this work, we identify debugging as a key barrier due to the latency from running analyses to detecting bugs and the scale of metadata processing needed to diagnose a bug. To address these challenges, we prototype a command-line interface tool, Multiverse Debugger, which helps diagnose bugs in the multiverse and propagate fixes. In a qualitative lab study (n=13), we use Multiverse Debugger as a probe to develop a model of debugging workflows and identify specific challenges, including difficulty in understanding the multiverse's composition. We conclude with design implications for future multiverse analysis authoring systems.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。