I'd love to see a creative, dynamic workshop at which we work collectively in a manner that aims to :
- encourage VIS participants to replicate existing studies
- advise on designing studies to reproduce results
- explore mechanisms for interpreting and making use of studies that revisit existing experiments
Here are some thoughts on how things might work ...
- anyone who wants to know more about running experiments and analysing outputs.
- day 1 - pm - session 1 - reports
on vis experiments that have been re-run in advance - how do results look?
solicit / open call these in advance
- day 1 - pm - session 2 - experiment
re-run a selected experiment multiple times in parallel
someone describes experimental structure & how AMT works
set (multiple) experiments running overnight - some similar, some with deliberate differences
- day 2 - am - next day - group analysis of results
discussion of p-values vs other (graphical) approaches
graphical analysis of overnight results (with different N, etc.)
determine - what do we know? About the experiment and the experimental methods.
can we perform meta-analyses?
if we are feeling ambitious ... agree on 5 things the VIS community should do to address the reproducibility crisis
- IEEE VIS, Baltimore 16
- some funding for AMT experiment
- to select a few experiments that we can invite people to replicate in advance
- to prepare a different experiment that can be run by groups overnight?
- other preparation? Commitment from organisers to run experiments with students?
- what is relationship with DC? Does there need to be one?
VIS has a role to play both in addressing these concerns in the broad scientific community and using appropriate approaches to advance its own discipline. I think we have the people in the community who can effect change within VIS and beyond. We have enthusiasm, statistical nous, cutting edge approaches to interactive graphics and capabilities in collaboration. This seems like a good opportunity. Can we configure the community into at least considering working together to address key scientific problems collectively? Co-creating data and conducting a meta analysis may well help at least put this on the agenda and giving people experience of some collective analysis might make it more feasible in the future. Workshops at which we work collaboratively to educate and analyse seem a reasonable mechanism for doing this kind of work.
I can't offer much. I want to learn to do studies and to contribute to the new way of doing things - in terms of doing VIS well and using VIS to do science well. I was inspired by Pierre's BELIV talk in Paris 2015 and subsequent reading and discussion. But I need lots of help. I'm a consumer / participant and post workshop contributor in all of this rather than a workshop leader!