Quality Control and Software Development Best Practices

This week’s café conversation posed the questions "How can you be sure your statistical analysis or software is doing what you expect?" and "How can you make sure others will be able to use it and reproduce your results?" As a Research Software Engineer, my answers would be to make use of practices and tools such as testing, version control, continuous integration and containerisation, but these concepts can be daunting to those unfamiliar with them, and will not be applicable across all domains.

As our conversations are cross-disciplinary, with participants present from departments including Anthropology, Education, Computer Science and Geography, as well as from Advanced Research Computing (ARC) and the library systems team, we started the session with a short survey to try to gauge the sorts of tools and programming languages people are using, and how likely they were to use various tools/practices.

Our participants use a wide range of tools, from programming languages (and libraries) to predominantly GUI-based tools. The tools used will clearly have an impact on whether standard software development practices can be used, but can we adapt these practices to make them applicable to other tools?

 

Next, participants were asked to rate how likely they were to use various practices, from 0 (Never heard of it) to 5 (Extremely likely). The average scores for testing and version control were reasonably high, though all three scored some 0s and a handful of 1s. Continuous integration and containerization, unsurprisingly, had low scores.

We discussed how participants can be sure how their code or analysis does what it sets out to do. Most participants use some some sort of manual checks of their analyses, for example checking the outcome for specific, known results. In numerical sciences, testing is the norm: one participant mentioned he used manual testing as a stop-gap to automated testing. In this field, although journal policies vary, some reviewers will automatically reject papers that do not include sufficient proof or due diligence, of which testing is an important part. By contrast, in the humanities, journals are now asking to see the data but not necessarily the analysis or code that produced the results; participants felt that it was only a matter of time before they followed the lead of journals in the sciences and required authors to provide the same level of detail about code.

A question was posed: "What should we be testing for? Is it more important to make sure that the software does not crash, or that it produces the right results?" Whilst defensive programming to avoid crashes can be important in industry (where software will is more likely to be used by non-expert users), in research, crashes can just be a useful signal that something has gone wrong: silently carrying on and producing a wrong result could be far more dangerous.

Version control was the next topic of discussion. Versioning is built in to Dropbox and Microsoft Office 365, and is also included in wiki software used by some teams for their documentation. For more sophisticated use cases, there are tools more traditionally used for programming such as Git. Several participants mentioned that they end up figuring out their own method for keeping track of their code and data, but were not convinced that they were going about things the right way. Some participants had previously attended ARC’s course on Git and GitHub, but had not yet worked out how to fit it in to their everyday work.

As well as using version control to keep track of our own code and documents, how can we keep track of versions of dependencies (e.g. the software packages we use in our analysis or code), to ensure that our results can be reproduced? One participant makes use of DOIs to keep track of the specific versions of software used in a paper, using a Zenodo record to list all of the DOIs; other participants may not have considered keeping track of such things. We mentioned containerisation as one possible solution to this, though it has a steep learning curve; this could perhaps be overcome using tools such as mybinder. (Other solutions for dependency management, which were not discussed, are listed below.)

Time did not allow us to discuss continuous integration (CI) in detail, but we gave a brief explanation of CI to those who were unaware of it: "automat[ing] doing the repetitive tasks you used to do by hand" (as one participant succinctly put it). We pointed out that CI tools such as CircleCI or GitHub Actions can be used for something as simple as ensuring your code compiles or passes syntax checks; it does not need to use a sophisticated workflow.

So what are the barriers to using these tools and practices? Firstly there is the pressure to move on to the next new thing: writing tests and using version control can be seen as boring or a waste of time, particularly if there is no requirement from journals to share code or to ensure analyses can be replicated. Secondly, there is a lack of training and knowledge: some participants felt there was just too big a hurdle to get over before they can use these tools; others had some knowledge but were unsure how to embed these practices into their own workflows. Could ARC produce more domain-specific training for individual departments? Thirdly, there is the lack of motivation: for example, undergraduate Computer Science students are taught how and why to use version control but still rarely make use of it. One participant suggested that we need to show the value of the tools in ways that researchers would understand, for example using version control and blockchain to stop plagiarism. 

This week’s discussion encapsulated some of the issues that are being tackled by groups such as the Software Sustainability Institute and the Society of Research Software Engineers. From my point of view it was fascinating to get a more local insight into these issues, and I hope it will prove useful as we consider the future direction of the training courses run by ARC.

Useful links

Thanks to the participants for sharing many of these links. 

Testing

• Unit testing in R
• Testing in python

Version control

• Software Carpentry git course
• Introduction to GitHub
• Learn git branching 

Executable notebooks to help reproducibility

• Introduction to Juypter notebooks
• Introduction to Jupyter Book
• Introduction to rmarkdown 

Dependency tracking and containerisation

• Defending your code against dependency problems
• binder (automatically dockerises your jupyter notebooks)
• Hands-on docker tutorials

Continuous Integration

• Introduction to GitHub Actions

Advanced Research Computing Training

• Training Courses
• Resources

This blog expresses the author’s views and interpretation of comments made during the conversation. Any errors or omissions are the author’s.

Alison Clarke is a Research Software Engineer at Advanced Research Computing, Durham University.