Abstract

Reproducibility and replicability are pillars of the scientific method used to build confidence in scientific findings. In the wake of a replication crisis, more attention has been brought to these pillars within computer science and, specifically, the subfield of visualization. However, visualization covers a spectrum of approaches, from quantitative approaches such as algorithms development and perception studies to design-oriented or qualitative work in which the subjective, situated nature of the work is not intended to be reproducible. An open question remains within the visualization community: how do we, as a research community, make nonreproducible work scrutinizable? The primary contribution of this dissertation is a definition and characterization of traceability as complimentary to reproducibility for scrutinizing non-empirical work and an investigation in supporting a traceable research process. This dissertation also contributes a software prototype that implements user interfaces and visualizations supporting these methods. Four projects described in this dissertation informed understanding of traceability for design-oriented visualization research. The first project was a motivation to challenge the status quo for the design study process and reporting, as current best practices did not support process transparency. Diverging from current practices in subsequent work, this work focused on methodological experimentation with criteria for rigor to improve transparency. This project was the impetus for defining traceability. The third project conceptualized traceability to support transparency for design-oriented visualization research and a vision of how to support traceability through a visualization tool. Finally, this dissertation investigated how traceability transfers to applications outside of design-oriented visualization research in the final piece of work. Conjointly, these projects sketch a path for how the research community can make design-oriented and qualitative work rigorous, traceable, and scrutinizable.

Citation

BibTeX

@phdthesis{2023-thesis-rogers,
  title = {Traceability in Design-Oriented Visualization Research},
  author = {Jen Rogers},
  school = {University of Utah},
  month = {May},
  year = {2023}
}