Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

How effective is your research programming workflow?

Inspired by The Joel Test, here is a simple 12-step checklist to determine whether you have an effective research programming workflow.

For my Ph.D. dissertation, I investigated research programming, a common type of programming activity where people write computer programs to obtain insights from data. Millions of professionals in fields ranging from science, engineering, business, finance, public policy, and journalism, as well as numerous students and computer hobbyists, all perform research programming on a daily basis.

Inspired by The Joel Test for rating software engineering teams, here is my informal “Philip test” to determine whether your research programming workflow is effective:

  1. Do you have reliable ways of taking, organizing, and reflecting on notes as you're working?

  2. Do you have reliable to-do lists for your projects?

  3. Do you write scripts to automate repetitive tasks?

  4. Are your scripts, data sets, and notes backed up on another computer?

  5. Can you quickly identify errors and inconsistencies in your raw data sets?

  6. Can you write scripts to acquire and merge together data from different sources and in different formats?

  7. Do you use version control for your scripts?

  8. If you show analysis results to a colleague and they offer a suggestion for improvement, can you adjust your script, re-run it, and produce updated results within an hour?

  9. Do you use assert statements and test cases to sanity check the outputs of your analyses?

  10. Can you re-generate any intermediate data set from the original raw data by running a series of scripts?

  11. Can you re-generate all of the figures and tables in your research paper by running a single command?

  12. If you got hit by a bus, can one of your lab-mates resume your research where you left off with less than a week of delay?

These questions are listed in order of difficulty: Any respectable researcher must answer “Yes” to the first few, but almost nobody (myself included) can answer “Yes” to the last few.

I won't make any specific tool recommendations in this article, since the optimal choice of tools will depend on your field and colleagues. However, I recommend for all computational researchers to learn about Software Carpentry online materials and bootcamps as a starting point.

Created: 2013-07-22
Last modified: 2013-07-22
Related pages tagged as research: