Design study is an established approach of conducting problem-driven visualization research. The academic visualizationcommunity has produced a large body of work for reporting on design studies, informed by a handful of theoretical frameworks, andapplied to a broad range of application areas. The result is an abundance of reported insights into visualization design, with anemphasis on novel visualization techniques and systems as the primary contribution of these studies. In recent work we proposeda new, interpretivist perspective on design study and six companion criteria for rigor that highlight the opportunities for researchersto contribute knowledge that extends beyond visualization idioms and software. In this work we conducted a year-long collaborationwith evolutionary biologists to develop an interactive tool for visual exploration of multivariate datasets and phylogenetic trees. Duringthis design study we experimented with methods to support three of the rigor criteria:ABUNDANT,REFLEXIVE, andTRANSPARENT. As aresult we contribute two novel visualization techniques for the analysis of multivariate phylogenetic datasets, three methodologicalrecommendations for conducting design studies drawn from reflections over our process of experimentation, and two writing devices forreporting interpretivist design study. We offer this work as an example for implementing the rigor criteria to produce a diverse range ofknowledge contributions.
We develop a new perspective on research conducted through visualization design study that emphasizes design as a method of inquiry and the broad range of knowledge-contributions achieved through it as multiple, subjective, and socially constructed. From this interpretivist position we explore the nature of visualization design study and develop six criteria for rigor. We propose that rigor is established and judged according to the extent to which visualization design study research and its reporting are INFORMED, REFLEXIVE, ABUNDANT, PLAUSIBLE, RESONANT, and TRANSPARENT. This perspective and…
College of Computing and Informatics, Drexel University（德雷塞尔大学计算与信息学院）
We present a focused analysis of user studies in explainable artificial intelligence (XAI) entailing qualitative investigation. We draw on social science corpora to suggest ways for improving the rigor of studies where XAI researchers use observations, interviews, focus groups, and/or questionnaires to capture qualitative data. We contextualize the presentation of the XAI papers included in our analysis according to the components of rigor described in the qualitative research literature: 1) underlying theories or frameworks, 2) methodological approaches, 3) data collection methods, and 4) dat…
Game-Based versus Training-Based Virtual Reality for Cognitive Rehabilitation in Mild Cognitive Impairment and Dementia: A Systematization of Design Paradigms, Outcomes, and Evaluation Rigor
Chasing Meaning and/or Insight? A Survey on Evaluation Practices at the Intersection of Visualization and the Humanities
National Distance Education University (UNED)（西班牙国立远程教育大学）；University for Continuing Education（继续教育大学）；University of Luxembourg（卢森堡大学）；University of Technology Chemnitz（开姆尼茨工业大学）；King’s College London（伦敦国王学院）
Puzzle-based training is a common type of hands-on activity accompanying formal and informal cybersecurity education, much like programming or other IT skills. However, there is a lack of tools to help the educators with the post-training data analysis. Through a visualization design study, we designed the Training Analysis Tool that supports learning analysis of a single hands-on session. It allows an in-depth trainee comparison and enables the identification of flaws in puzzle assignments. We also performed a qualitative evaluation with cybersecurity experts and students. The participants ap…
Technology is influencing education, providing new delivery and assessment models. A combination between online and traditional course, the hybrid (blended) course, may present a solution with many benefits as it provides a gradual transition towards technology enabled education. This research work provides a set of definitions for several course delivery approaches, and evaluates five years of data from a course that has been converted from traditional face-to-face delivery, to hybrid delivery. The collected experimental data proves that the revised course, in the hybrid delivery mode, is at …
Despite the ubiquity of communicative visualizations, specifying communicative intent during design is ad hoc. Whether we are selecting from a set of visualizations, commissioning someone to produce them, or creating them ourselves, an effective way of specifying intent can help guide this process. Ideally, we would have a concise and shared specification language. In previous work, we have argued that communicative intents can be viewed as a learning/assessment problem (i.e., what should the reader learn and what test should they do well on). Learning-based specification formats are linked (e…
Decision support tools enable improved decision-making for challenging decision problems by empowering stakeholders to process, analyze, visualize, and otherwise make sense of a variety of key factors. Their intentional design is a critical component of the value they create. All decision-support tools share in common that there is a complex decision problem to be solved for which decision-support is useful, and moreover, that appropriate analytics expertise is available to produce solutions to the problem setting at hand. When well-designed, decision support tools reduce friction and increase…
