February 1, 2010 (Vol. 30, No. 3)

Kumar Bala

Data from Three Global Initiatives Shows that Standardization and Reproducibility Are Possible

Proteomics has been in vogue for more than a decade with the promise of identifying biomarkers that could be of value for drug development, as well as diagnostics. This promise has fueled multibillion dollar global initiatives including the National Heart Lung and Blood Proteomics Institute initiative, which commenced in September 2002 and focused on technology development at 10 participating centers.

A wet blanket was thrown on this rosy forecast in 2007 at the annual conference held by Human Proteome Organization (HUPO), the flagship organization dedicated to proteomics research. At the meeting in Seoul, “proteomicologists” took to task what they called a “sloppy science,” and aptly termed the biomarker field “chaotic.”

This observation laid bare the Achilles heel of proteomics—the inability of many proteomics researchers to reproduce their data. The result was a unified cry among influential scientific leaders: If you can’t reproduce your data, then don’t publish. In response, both public and private sectors undertook new global initiatives with the goal of developing standards and demonstrating the reproducibility of proteomics research.

There are several sources of variability: biological (arising from the sample), pre-analytical (sample handling and study design), technical, user-related, and statistical (meaningful data analysis).

Carefully minimizing these sources of error will help the field of proteomics make significant strides in producing data that is reliable, repeatable, and comparable such that researchers in different labs can come to the same conclusions and have those conclusions be based on fundamental biological processes.

The Proteomics Research Group of the American Biomolecular Research Facilities (ABRF) has been involved in sharing knowledge about the analysis of proteins since 2002. This group sponsors annual research studies that examine current techniques and capabilities and aims to raise awareness, knowledge, and education about modern methods for protein analysis.

Following in the footsteps of ABRF, this article will elaborate on three of the most significant global initiatives and how they have contributed to the quest for standardization and reproducibility in proteomics. 

To address the question of reproducibility in LC-MS based proteomics, HUPO created a test sample working group led by John Bergeron, professor of anatomy and cell biology at McGill University. The group undertook a controlled study involving 27 different labs.

These labs each received the same sample consisting of an equimolar mixture of 20 human proteins and unique peptides of high purity. The primary task of these labs was to identify all 20 human proteins and all unique peptides.

The initial results demonstrated that only seven labs identified all 20 human proteins. Subsequent analysis of the raw data by an independent group, however, revealed that all 20 proteins and most of the peptides had been correctly detected and identified in all 27 labs.

Missed identifications, environmental contamination, incorrect database matching, and incomplete curation of protein identifications were identified as the sources of problems. It was concluded that adequate training and education could help overcome these problems.

The National Cancer Institute launched the Clinical Proteomic Technology Assessment for Cancer (CPTAC) initiative in 2006 by awarding $35.5 million to five teams for a period of five years to evaluate proteomics technologies with applications to cancer research.

According to Steve Carr, director of the proteomics platform at Broad Institute and senior author on the CPTAC study, the issue is that discovery experiments do not lead to biomarkers in clinical proteomics.  They lead to hypotheses or candidates that need to be further credentialed. He said that the aim of the CPTAC study was to bridge the gap between discovery and a list of markers so that one has confidence as they move toward clinical validation.

In this study, a multisite collaboration showed that multiple reaction monitoring, coupled with isotope dilution mass spectrometry (SID-MRM-MS) used for quantifying candidate biomarkers in plasma, can be performed reproducibly across different labs. In other words, different labs achieved the same data using common materials and standardized protocols.

For 9 out of the 10 peptides tested across the labs, the researchers reported interlaboratory coefficients of variation ranging from 4–14% for study 1, 4–13% for study 2, and 10–23% for study 3 at a limit of quantitation of 2.92 femtomoles per milliliter. From study 1 to study 3, the coefficient of variation progressively increased, indicating that assay variability was due to sample preparation rather than instrument variability.

Carr and his co-researchers also found differences in assay performance for different peptides and recommended that the final selection of signature peptides for SID-MRM-MS biomarker assays be based on multisite studies “so as to ensure the most robust performance.”

The Fixing Proteomics Initiative is an industry-led campaign coordinated by Novartis, Nonlinear Dynamics, CIL Biotech, and Bio-Rad Laboratories, which involves 17 global labs. This follows in the footsteps of the HUPO Phase I study, the outcome of which was presented at “HUPO 2008” in Amsterdam.

This 2-D gel electrophoresis study focused on establishing that 2-D gel analysis can be performed in a reproducible manner within and across global labs. This is of value because proteomics traces its roots to use of 2-D gels in combination with mass spectrometry based identification for biomarker discovery.

Researchers followed a standard 2DE protocol and used a HeLa cell lysate to determine if analysis could achieve both intra- as well as interlab reproducibility. In the final outcome, the study was able to achieve high intralab reproducibility and 10 out of 17 participant labs were able to generate gel images that fell within 95% confidence bound at low stringency.

“This 60 percent success rate may seem modest, but the value of the study is that for the first time, the 2-D gel community will have a reference standard to do an experiment that can reproduce results within a set limit,” said study participant Maxey Chung, associate professor of biochemistry at the National University of Singapore. 

This project illustrates that a complex proteomics method such as 2DE can benefit from availability of reference materials that reduce bias. With the immediate feedback on the quality of their gels, users can gain confidence in being able to generate reproducible data with precious samples. Under these circumstances, even the best labs will see user-dependent variation when they don’t follow the protocols precisely.

Outcomes from these three global initiatives employing different proteomics technologies show that standardization and reproducibility can be achieved. Insofar as practioners of proteomics learn lessons from these studies, they will build confidence and deliver on the promise of proteomics.

In fact, the first proteomics-based test approved by the FDA, the OVA1 ovarian cancer triage test (Quest Diagnostics), shows that the promise of proteomics is already being fulfilled. Alas, a small step for the proteomicologists but a giant leap for proteomics.


Peer-to-peer education leads to quality improvement: After difficulties with the reference sample (A), observed inconsistencies led to suggestions toward improvement (B).

Kumar Bala ([email protected]) is marketing manager at Bio-Rad Laboratories. Web: www.bio-rad.com. He is also the co-chair of HUPO-Industrial Advisory Board.

Previous articleU87MG Glioblastoma Cell Line Sequenced in Under a Month
Next articleRoche Takes a Stab at Getting mAbs Under the Skin