Audit Trail

We developed this framework through a two year, cross-institution, international collaboration. Here, we summarize our research process and provide an audit trail of documents produced in our reflective analysis.

Our reflective collaboration evolved over a period of two years. It started in Nov, 2015 with a seemingly simple question about two workshops: what could we do better next time? Its scope gradually expanded as we explored ideas about why and how to use visualization CVO workshops. During this time, we ran additional workshops, intertwining reflective analysis and action. Our research process is summarized in this timeline.


Throughout this time we created diverse artifacts, including pre-paper talks, a PhD proposal, tens of paper drafts, and many other documents. To help establish the credibility of our analysis, we assembled these artifacts into the following curated audit trail. We have not edited the documents for consistency, but we have removed personal information and deleted some sensitive content.

2010 — 2015.11
  • JD and others used a series of imagination exercises with GIS analysts [P1].
  • Inspired by success of that project, JD worked with SG, SJ, and others to use a series of workshops in a project with energy analysts [P2].
  • Later, JD and others used series of workshops with defense analysts [P3].
2015.06 — 2015.07
  • EK and MM planned, executed, and analyzed a full day workshop with neuroscientists [P4] based on SG’s paper [P2].
  • The neuroscience workshop inspired development of tools validated through neuroscience and visualization papers.


2015.10 — 2015.10
  • EK, SG, JD, and MM discussed recent experiences using CVO workshops [P2, P4].
  • Tried to answer the question: what could we do better next time?
  • SG was planning to use workshop with constraint programmers [P5].
2015.10 — 2016.03
  • SG planned, executed, and analyzed a CVO workshop with constraint programmers [P5].


2016.05 — 2016.06
analysis + writing
  • SG, JD, EK, MM, and SJ (with other colleagues) tried to analyze three CVO workshops [P2, P4, P5].
  • Focused on low-level details, such as number of participants and location.
  • Proposed pitfalls for future workshops, including:
    • Not staying on top of the photography during the workshop can mean that idea flow maybe broken in post workshop analysis.
    • Not being prepared for unexpected circumstances, like having no wifi in the room. Prepare offline options and printouts
    • Ensure Post-it sizes and paper is right for the different activities. Pilot the activities with your materials
  • Collectively decided that the ideas were not yet ready for publication.
Summary of comparison (png) that focused on describing what happened within workshops instead of how to use workshops.
Screenshot of a 17 page written reflection (redacted) that documented our use of three workshops with similar methods.
2016.08 — 2016.09
analysis + writing
A slide summarizing contributions from our pre-paper talk (pdf) Here, we are grasping at the concept of the framework and workshop process (“when and why to consider…”) but have not yet articulated it.
2016.09 — 2016.10
workshop + analysis
  • EK and MM co-organized and co-facilitated DiscoveryJam 2016.
  • EK and SG analyzed workshop experience in-person at IEEE Vis.
  • Created detailed notes (redacted) about how methods can be tailored to the specific context of a workshop.
2016.10 — 2016.12
  • Collaboration paused to incubate and to work on other projects.
  • EK and MM published results of neuroscience workshop [P4].
Screenshot of the how we described the workshop in our paper. Although the CVO workshop profoundly influenced the direction of our project, we reported it in one sentence.
2017.01 — 2017.03
  • Reviewed literature on the origins of CVO workshops in software engineering and creative problem-solving.
  • Examined workshops used in diverse projects with different goals [P1 - P5] and DiscoveryJam.
  • Proposed a first draft of the framework (pdf) that identified the process of using workshops.
A slide summarizing an early version of the framework. It identified stages of the process model by their input, output, and motivation. We revised this framework many times. In one change, we realized scoping and planning are mutually influential and completed with an iterative process. We merged them into a single stage in the final framework: Before the Workshop: Define & Design.
2017.03 — 2017.04
workshops + analysis
  • EK and MM assisted in designing and facilitating two workshops:
    • A workshop to understand analysis needs of genealogists [P7].
    • A workshop to find funded collaboration with phylogenetic analysts [P8].
  • We iteratively improved the framework by reflecting on our experience with these two workshops.
2017.05 — 2017.07
writing + analysis
  • Created a first draft (pdf) of a Ph.D. dissertation proposal that identified a set of 34 key questions for the framework.
  • EK, JD, and SJ met in person for three days of discussion and analysis.
Screenshot of the proposed research methods that shows how we wrestled with how to learn from our collective experience. Here, we describe our process as collective reflection because we have not yet discovered the methodology of critically reflective practice. We have also not yet focused our work on visualization workshops. Hence, we cite 12 projects that we winnowed to 10 in the final paper.
2017.05 — 2017.07
Slides from the proposal defense. We identified the relevant workshop data to analyze through qualitative analysis methods. We have not yet winnowed the scope of our analysis to CVO workshops.
2017.08 — 2017.09
analysis + writing
  • Created a draft of the framework, summarized below, based on the aforementioned Ph.D. proposal.
  • Key ideas in this draft include:
    • It referred to the use of critically reflective practice.
    • It characterized workshops based on their role in the design process.
    • It did not yet capture the iterative nature of workshop design.
  • Reflecting on this draft revealed shortcomings of our ideas, which we subsequently addressed with more formal analysis methods.
A graphical summary of an early framework draft. This would ultimately become the three-stage process model for CVO workshops.
We tried to characterize workshops by how they influence visualization design and decision models. Ultimately, we focused domain characterization and requirements workshops, which we termed CVO workshops.
2017.09 — 2017.10
analysis + writing
  • Analyzed CVO workshop methods (xlsx) using frameworks for educational and creativity workshops.
  • Listed 80+ observations (xlsx) about the process and design of workshops.
    • Connected each observation to our collective experience or existing theory.
    • Considered many approaches to group or rank the observations.
  • Applied methods from design — observations-to-insights and insight sorting — to generate a list of actionable insights (pdf).
  • Assisted planning and facilitating DiscoveryJam again.
We tried to analyze methods in a number of ways. Here, we categorized workshop methods according to a framework for educational workshops. Although we decided that this framework was not directly applicable to CVO workshops, we include it to show one of the many alternatives that we considered.
Some ideas that do appear in our final framework are shown in this early matrix of effects in which we tried to characterize methods by their impact on the workshop ideaspace. After further thinking, some of these characteristics were integrated into the tactics for effective workshops.
We collaboratively connected workshop recommendations with our experience and the existing literature. Some of the ideas contained in this insight appear in our final framework.
2017.10 — 2017.11
analysis + writing
Summary of the process model with insights for each of the stages. Each insight is connected to our experience and creativity literature.
One insight from our process model. Here, we have started to identify key ideas that would ultimately become part of the workshop opening in our final framework.
As part of our reflective process, we individually wrote about ways to improve our ideas. One suggestion is shown here. It ultimately inspired the structure of our framework’s two interconnected models: the workshop process and the workshop structure.
2017.11 — 2018.01
analysis + writing
  • Rewrote the paper draft (pdf) based on the collaborative reflection_
  • Tried many ways to split the workshop process from the workshop structure:
  • Assembled an early version of this audit trail. Repeatedly revisited it in our subsequent analysis and writing.
Two early diagrams of the workshop process and structure. In the process (left), we show the iterative nature of workshop design. In the structure (right), we have identified the opening, core, and closing but have not yet figured out how to effectively represent it. Here, it looks like an exploding styrofoam takeout container.
We also wrestled with how to summarize our workshop experience as shown in these two tables. Here, we tried to display all of our workshop experience, including workshops that were used as part of a series. Later, we would focus explicitly on the first workshop used in a series.
2018.01 — 2018.02
analysis + writing
  • Updated a draft (pdf) without using explicit recommendations because we were unsure if the recommendations helped or hindered our understanding of CVO workshops.
  • Shared the draft with two colleagues who planned workshops with it. Rewrote the draft (pdf) based on their feedback.
A diagram of the workshop process and structure that closely resembles what appears in our final paper. Here, we define the key aspects of the process and structure.
An early version of the tactics were called cacti factors. We added the topic to emphasize the critical role of data, visualization, and analysis in CVO workshops.
2018.02 — 2018.03
analysis + writing
  • Critically examined all the ideas in the paper in a full day meeting (EK, MM).
  • Added guidelines back to the framework as a way to organize our ideas and to make the framework more actionable.
  • Created this website to describe the workshop methods because we did not have room for them in the paper.
  • Prepared the paper for submission.
2018.03 — 2018.06
analysis + writing
  • Incubated on ideas contained in the submitted paper (pdf) and supplemental material (zip) while it was under review.
  • Received conditional acceptance.
  • Completed a collective, written reflection (pdf) to address points raised by the reviewers.
  • Prepared the next draft of the paper using ideas from the reflection.
  • Received definitive acceptance. Cleaned up the final paper and materials.