Audit Trail

• JD and others used a series of imagination exercises with GIS analysts [P1].
• Inspired by success of that project, JD worked with SG, SJ, and others to use a series of workshops in a project with energy analysts [P2].
• Later, JD and others used series of workshops with defense analysts [P3].

References

• [P1] Dykes, J., Wood, J. & Slingsby, A. (2010). Rethinking Map Legends with Visualization Design IEEE Transactions on Visualization and Computer Graphics, 16(6).
• [P2] Goodwin, S., Dykes, J., Jones, S., Dillingham, I., Dove, G., Duffy, A., Kachkaev, A., Slingsby, A. & Wood, J. (2013). Creative User-Centered Visualization Design for Energy Analysts and Modelers. IEEE Transactions on Visualization and Computer Graphics, 19(12).
• [P3] Walker, R., Slingsby, A., Dykes, J., Xu, K., Wood, J., Nguyen, P., Stephens, D., Wong, W. & Zheng, Y. (2013). An Extensible Framework for Provenance in Human Terrain Visual Analytics. IEEE Transactions on Visualization and Computer Graphics, 19(12).

• EK and MM planned, executed, and analyzed a full day workshop with neuroscientists [P4] based on SG’s paper [P2].
• The neuroscience workshop inspired development of tools validated through neuroscience and visualization papers.

References

• [P4] Kerzner, E., Lex, A., Sigulinsky, C. L., Urness, T., Jones, B. W., Marc, R.E., Meyer, M. (2017). Graffinity: Visualizing Connectivity In Large Graphs. Computer Graphics Forum 36(3).
• [P4] Lauritzen, J.S., Sigulinsky, C. L., Anderson, J. R., Kalloniatis, M., Nelson, N. T., Emrich, D. P., Rapp, C., McCarthy, M., Kerzner, E., Meyer, M., Jones, B. W., Marc, R. E. (2016). Rod-cone Crossover Connectome of Mammalian Bipolar Cells. Journal of Comparative Neurology

• EK, SG, JD, and MM discussed recent experiences using CVO workshops [P2, P4].
• Tried to answer the question: what could we do better next time?
• SG was planning to use workshop with constraint programmers [P5].

• SG planned, executed, and analyzed a CVO workshop with constraint programmers [P5].

Reference

• [P5] Goodwin, S., Mears, C., Dwyer, T., de la Banda, M. G., Tack, G., Wallace, M. (2017) What do Constraint Programming Users Want to See? Exploring the Role of Visualization in Profiling of Models and Search. IEEE Transactions on Visualization and Computer Graphics 23(1).

• SG, JD, EK, MM, and SJ (with other colleagues) tried to analyze three CVO workshops [P2, P4, P5].
• Focused on low-level details, such as number of participants and location.
• Proposed pitfalls for future workshops, including:
  • Not staying on top of the photography during the workshop can mean that idea flow maybe broken in post workshop analysis.
  • Not being prepared for unexpected circumstances, like having no wifi in the room. Prepare offline options and printouts
  • Ensure Post-it sizes and paper is right for the different activities. Pilot the activities with your materials
• Collectively decided that the ideas were not yet ready for publication.

Summary of comparison (png) that focused on describing what happened within workshops instead of how to use workshops.

Screenshot of a 17 page written reflection (redacted) that documented our use of three workshops with similar methods.

• Examined visualization workshop methods (pdf) in larger context of creativity and design literature.
• Collaboratively worked to outline a paper in written outline (pdf) and presentation pre-pre-paper talk (pdf).
• Again, decided that the ideas were not yet ready for publication.

A slide summarizing contributions from our pre-paper talk (pdf) Here, we are grasping at the concept of the framework and workshop process (“when and why to consider…”) but have not yet articulated it.

• EK and MM co-organized and co-facilitated DiscoveryJam 2016.
• EK and SG analyzed workshop experience in-person at IEEE Vis.
• Created detailed notes (redacted) about how methods can be tailored to the specific context of a workshop.

• Collaboration paused to incubate and to work on other projects.
• EK and MM published results of neuroscience workshop [P4].

Screenshot of the how we described the workshop in our paper. Although the CVO workshop profoundly influenced the direction of our project, we reported it in one sentence.

• Reviewed literature on the origins of CVO workshops in software engineering and creative problem-solving.
• Examined workshops used in diverse projects with different goals [P1 - P5] and DiscoveryJam.
• Proposed a first draft of the framework (pdf) that identified the process of using workshops.

A slide summarizing an early version of the framework. It identified stages of the process model by their input, output, and motivation. We revised this framework many times. In one change, we realized scoping and planning are mutually influential and completed with an iterative process. We merged them into a single stage in the final framework: Before the Workshop: Define & Design.

• EK and MM assisted in designing and facilitating two workshops:
  • A workshop to understand analysis needs of genealogists [P7].
  • A workshop to find funded collaboration with phylogenetic analysts [P8].
• We iteratively improved the framework by reflecting on our experience with these two workshops.

References

• [P7] Nobre, C., Gehlenborg, N., Coon, H., and Lex, A. (2018). Visualizing Multivariate Clinical Data in Genealogy Graphs. IEEE Transactions on Visualization and Computer Graphics (to appear).

• Created a first draft (pdf) of a Ph.D. dissertation proposal that identified a set of 34 key questions for the framework.
• EK, JD, and SJ met in person for three days of discussion and analysis.

Screenshot of the proposed research methods that shows how we wrestled with how to learn from our collective experience. Here, we describe our process as collective reflection because we have not yet discovered the methodology of critically reflective practice. We have also not yet focused our work on visualization workshops. Hence, we cite 12 projects that we winnowed to 10 in the final paper.

• Formally proposed the creation of the framework in a Ph.D. dissertation proposal (pdf).
• Summarized the framework in a proposal defense (pdf).
• Used feedback on Ph.D. committee members to improve the framework.

Slides from the proposal defense. We identified the relevant workshop data to analyze through qualitative analysis methods. We have not yet winnowed the scope of our analysis to CVO workshops.

• Created a draft of the framework, summarized below, based on the aforementioned Ph.D. proposal.
• Key ideas in this draft include:
  • It referred to the use of critically reflective practice.
  • It characterized workshops based on their role in the design process.
  • It did not yet capture the iterative nature of workshop design.
• Reflecting on this draft revealed shortcomings of our ideas, which we subsequently addressed with more formal analysis methods.

A graphical summary of an early framework draft. This would ultimately become the three-stage process model for CVO workshops.

We tried to characterize workshops by how they influence visualization design and decision models. Ultimately, we focused domain characterization and requirements workshops, which we termed CVO workshops.

• Analyzed CVO workshop methods (xlsx) using frameworks for educational and creativity workshops.
• Listed 80+ observations (xlsx) about the process and design of workshops.
  • Connected each observation to our collective experience or existing theory.
  • Considered many approaches to group or rank the observations.
• Applied methods from design — observations-to-insights and insight sorting — to generate a list of actionable insights (pdf).
• Assisted planning and facilitating DiscoveryJam again.

We tried to analyze methods in a number of ways. Here, we categorized workshop methods according to a framework for educational workshops. Although we decided that this framework was not directly applicable to CVO workshops, we include it to show one of the many alternatives that we considered.

Some ideas that do appear in our final framework are shown in this early matrix of effects in which we tried to characterize methods by their impact on the workshop ideaspace. After further thinking, some of these characteristics were integrated into the tactics for effective workshops.

We collaboratively connected workshop recommendations with our experience and the existing literature. Some of the ideas contained in this insight appear in our final framework.

• Collaboratively created a 30+ page reflection document (pdf) that organized insights into a workshop process model.
• Created an early draft of the paper (pdf) based on the reflection document.
• Individually reflected on the paper draft and then collated our ideas into a reflection response (pdf).

Summary of the process model with insights for each of the stages. Each insight is connected to our experience and creativity literature.

One insight from our process model. Here, we have started to identify key ideas that would ultimately become part of the workshop opening in our final framework.

As part of our reflective process, we individually wrote about ways to improve our ideas. One suggestion is shown here. It ultimately inspired the structure of our framework’s two interconnected models: the workshop process and the workshop structure.

• Rewrote the paper draft (pdf) based on the collaborative reflection_
• Tried many ways to split the workshop process from the workshop structure:
  • Applied bottom-up analysis (pdf); identified traits of methods from existing literature; decided this was not actionable.
  • Framed methods as creativity support tools (pdf); these ideas would become the tactics for effective workshops.
  • Tried top-down analysis (pdf); described the opening, core, and closing; ultimately inspiring the workshop structure.
• Assembled an early version of this audit trail. Repeatedly revisited it in our subsequent analysis and writing.

Two early diagrams of the workshop process and structure. In the process (left), we show the iterative nature of workshop design. In the structure (right), we have identified the opening, core, and closing but have not yet figured out how to effectively represent it. Here, it looks like an exploding styrofoam takeout container.

We also wrestled with how to summarize our workshop experience as shown in these two tables. Here, we tried to display all of our workshop experience, including workshops that were used as part of a series. Later, we would focus explicitly on the first workshop used in a series.

• Updated a draft (pdf) without using explicit recommendations because we were unsure if the recommendations helped or hindered our understanding of CVO workshops.
• Shared the draft with two colleagues who planned workshops with it. Rewrote the draft (pdf) based on their feedback.

A diagram of the workshop process and structure that closely resembles what appears in our final paper. Here, we define the key aspects of the process and structure.

An early version of the tactics were called cacti factors. We added the topic to emphasize the critical role of data, visualization, and analysis in CVO workshops.

• Critically examined all the ideas in the paper in a full day meeting (EK, MM).
• Added guidelines back to the framework as a way to organize our ideas and to make the framework more actionable.
• Created this website to describe the workshop methods because we did not have room for them in the paper.
• Prepared the paper for submission.

• Incubated on ideas contained in the submitted paper (pdf) and supplemental material (zip) while it was under review.
• Received conditional acceptance.
• Completed a collective, written reflection (pdf) to address points raised by the reviewers.
• Prepared the next draft of the paper using ideas from the reflection.
• Received definitive acceptance. Cleaned up the final paper and materials.