Copy Me If You Can! Part 2

June 15, 2015 (Vol. 35, No. 12)

It’s Time to Implement Practical Measures to Mitigate Irreproducibility

Irreproducibility, a Contemporary Concern for Science

In the first half of this article Copy Me If You Can, Part 1, published in the June 15 print issue of GEN, we outlined a potentially worrying picture around irreproducibility in science. We introduced results from the second annual State of Translational Science Research survey conducted by Sigma-Aldrich,¹ which closely reflect the picture painted in recent publications on the topic.^2-4 In this second half, we look at a number of the practical steps that can be taken to mitigate some of these issues.

Tackling Irreproducibility

Clearly there are a range of different issues that no one person or group can possibly resolve. As a community though, it behooves us to tackle these at all levels; the good news is that the changes are already happening.

For example, in 2014, the U.S. National Institutes of Health (NIH) convened a meeting with the editors of over 30 journals from the Nature and Science publishing houses to look for practical solutions that the scientific publishing arena could implement to enhance rigor around reproducibility.⁵ The meeting resulted in a set of five principles:

1. Rigorous statistical analysis
2. Transparency in reporting
   1. Standards
   2. Replicates
   3. Statistics
   4. Randomization
   5. Blinding
   6. Sample-size estimation
   7. Inclusion and exclusion criteria
3. Data and material sharing
4. Consideration of refutations
5. Consider establishing best practice guidelines for imaging and biological material identification

Importantly, these principles cover the majority of the topics highlighted by the commentaries referenced in this article, as well as those identified in the survey, and some additional concerns, such as the sharing of software and best practices around image manipulation. This is a very encouraging step as it makes it clear what editors expect to receive in submissions, and therefore should encourage scientists to adopt these as a core aspect of their research. The principles also provide for a better system of notifying the scientific community when published papers are later refuted, and enable much more space for the inclusion of full methods sections. This should significantly help to ensure the accurate transfer of information and exchange of ideas. On this retraction topic, a more direct approach to removing publishers’ inertia has been suggested by the Retraction Watch blog, which proposes some kind of penalty, maybe related to the impact factor of the journal, could be implemented.

This critical reproducibility issue has also led to the emergence of a new research validation industry. These services (such as validation.scienceexchange.com) enable researchers to pay for an independent and unbiased validation of their findings. Although at the level of the individual research project, this cost may be high, in the grand scheme of things it could be a small price to pay for reducing more expensive failure later on.

Arguably the biggest positive impact that any individual scientist or research team can have on this issue is to assess how well they have implemented (if at all) a range of practical steps in their laboratories. These can clearly improve the reproducibility of their own studies and of course the overall validity of their research. There are many steps in this, and most of them are covered in the papers referenced herein and within the survey. Therefore, one of the best places to start is to ensure familiarity with this work and associated literature. It is worthwhile looking briefly at the causes and products that researchers themselves listed as the biggest barriers in the survey, and consider the following approaches in mitigating some of the issues:

1. Experimental design and quality control – Begley & Ellis,³ Ioannidis,⁶ and Nuzzo,⁷ highlight a number of important aspects around experimental structure, including the need to design experiments of suitable sample size using carefully considered statistical power calculations. It is essential to couple this with appropriate positive and negative experimental controls as well as instrument/measurement controls.
2. Animal studies – Perrin⁴ outlines some important steps to take here, including splitting littermates, balancing genders, determining actual cause of death in all animals, and tracking gene inheritance in progeny.
3. Cell culture: Cell lines and mycoplasma – With ~15% of the cell lines used worldwide misidentified,⁹ it is essential to use tools such the International Cell Line Authentication Committee (ICLAC) database to check that the cells you have are what they are supposed be. Once the cell line identity is confirmed, the actual cell culture process itself can be impacted by mycoplasma contamination – estimated to affect 15-35% of cell lines.¹⁰ Mycoplasma contamination can only be checked by testing the cell lines as there is no visual way to identify it. According to the translational research survey,¹ more than 70% of respondents using cell lines did not test for mycoplasma at, or greater than, the recommended monthly assessment.
4. Validating reagents – Although clear, industry-wide standards are still in short supply, it is important to establish the validity of any reagent, especially biologically active ones, before use. For example, is your antibody behaving as expected, are your nucleic acid primers and probes exactly what you ordered, or are your buffers what you thought they were? In the translational research survey, the most commonly validated reagents were antibodies, however by fewer than half of the respondents that used them.¹ Notably, the concept of validation itself is controversial since there are differing opinions on what constitutes validation.  Record keeping – As we look to find ways to ensure reproducibility, it is essential to record every aspect of each step of an experiment, from experimental setup and replicates, down to the batch numbers of all the reagents and serial numbers of the instruments. Traditional paper-based or basic electronic records are by far still the prevailing method for record keeping.  However, the value of electronic laboratory notebooks (ELNs) and laboratory information management systems (LIMS) cannot be underestimated, especially as translational research is so often the basis of important decisions around progression to human studies.
5. Blinding and third party validation – Clinical trials are generally constructed with multiple blinding to reduce bias. While this is more difficult in the preclinical setting, it is possible and a good way to reduce the impact of bias. For example, the lead researcher could be blinded to the test and control groups in animal studies, or the statistics could be conducted by an independent colleague with no knowledge of which data are from test or control. As mentioned above, paid-for third-party validation is also becoming increasingly available, but it is also possible to ask nonauthor colleagues to validate primary data or replicates before manuscript submission.
6. Reproduce, critically analyse, and retraction check – We have noted throughout these articles that the published literature can be a minefield for the unaware. If your research relies heavily on previously published results, it is worthwhile taking some positive actions before assuming the results and conclusions are accurate. The first step is to check for any retractions and consider the validity of any data in retracted manuscripts. It would then be ideal to attempt to reproduce the experiments in your laboratory to confirm reproducibility. Obviously this is very time and resource consuming and may not be possible; therefore, at the very least it would be a good idea to cast a critical eye over the study from premise and methods to statistics and conclusions.

Conclusions

Across these two articles, we have outlined some of the main issues that lead to irreproducibility as well as some reasonably straight forward steps that can be implemented to improve reproducibility.

Of all the areas of human endeavour that should be capable of critical analysis and self-correction, science should be at the very top. After all, scientific advance continues to move forward breaking new ground and producing solutions to some of the world’s most intractable problems, so it can’t be all bad.

To read the first half of this article, click here.

¹ Sigma-Aldrich Corporation. 2014. Second Annual State of Translational Research 2014 Survey Report. Available at: www.sigma-aldrich.com/translational-survey.  (Accessed 1 March 2015)
² Prinz, F., et al. 2011. Nat. Rev. Drug Disc. 10:712
³ Begley, C.G. & Ellis L.M. 2012. Nature 483:531-533
⁴ Perrin, S. 2014. Nature 507:423-425
⁵ www.nih.gov/about/reporting-preclinical-research.htm (Accessed March 15 2015)
⁶ Ioannidis, J.P.A. 2005. PLoS Med. 2(8):e124
⁷ Nuzzo, R. 2014. Nature 506:150-152
⁸ Grieneisen, M.L. & Zhang, M. 2012. PLoS ONE 7(10): e44118
⁹ Masters, J.R. 2012. Nature 492:186
¹⁰ Marx, V. 2014. Nat Meth 11:483

Sean Muthian, Ph.D. ([email protected]), is director of strategic marketing and collaborations at Sigma-Aldrich.