This Article Full Text (PDF) Submit a scholarly reply Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Siddiqui, F. Articles by Backman, S. B. Search for Related Content PubMed PubMed Citation Articles by Siddiqui, F. Articles by Backman, S. B.

Potential pitfalls of interim analysis

Royal Victoria Hospital and McGill University Health Center, Montreal, Canada, E-mail: steven.backman{at}muhc.mcgill.ca

To the Editor:

In the evaluation of medical treatments, randomized trials are the current gold standard. According to the Mayo Clinic Clinical Trials glossary,¹ randomization minimizes the differences among groups by equally distributing people with particular characteristics to the trial arms. The goal of randomization is to minimize bias in the study design. We write to report the results of our randomization efforts that were identified during an interim analysis of a recent study conducted at our institution. This study assesses the effect of an oral cannabinoid on nausea scores, the incidence of vomiting and pain scores after gynecologic laparoscopy.

With the cooperation of the drug’s manufacturer, we obtained 120 tablets of study medication and 120 placebo tablets that were indistinguishable. Each patient was to receive two pills, with the intention to study 60 treatment patients and 60 control patients. The pills were separated into pre-numbered plastic bags, two identical pills in each. A nurse not involved in data collection then selected individual bags at her discretion containing either study medication or placebo, and placed them into correspondingly numbered identical paper envelopes. The 120 envelopes were then mixed and placed in a plastic box. The code identifying the contents of each bag/envelope was stored on a sheet of paper, and filed in a sealed envelope. Envelopes were selected by one of the investigators on the day they were to be administered.

The protocol was designed for an interim analysis after study of approximately 40 patients. This was done to help ensure that patients in either the control or treatment group were not being put at risk unknowingly. After 42 patients were investigated the study was un-blinded. Much to our surprise, 37 of 42 patients received study medication and 5 received placebo.

The probability P of drawing 37 treatment and 5 placebo envelopes is given by the formula P = (combination of 37 treatments · combination of 5 placebos) ÷ combination of 42 total.² The formula for the possible number of combinations C is

where r objects are taken from a total of n objects.³ Thus, the probability of drawing 37 treatment and five placebo envelopes is given by

The probability of this combination is approximately three in 10 billion, which is extremely rare indeed! We reviewed the drug randomization and although selection of the envelopes was blinded, we cannot discount a systematic error in the envelope sorting process.

Our experience highlights one of the inherent hazards of interim analysis that manifests as an unequal number of patients in the study arms. Therefore, unanticipated risks in each study population may be difficult to determine at this point in the study. Moreover, marked skewing of the randomization process, as reported, may for all intents and purposes un-blind the remainder of the study. This experience demonstrates that although the randomization process will guarantee equal numbers in study groups after all patients are collected, interim analysis may be unwittingly affected by a biased distribution of subjects.

Footnotes

Accepted for publication April 24, 2006.

Editor’s note
The important observations raised in this letter should be of interest to all clinical investigators and readers as well. There are several potential pitfalls relating to interim analyses, for which I have invited Dr. Peter Choi, the Journal’s Consultant Epidemiologist, to address in the commentary which appears below.

Donald R. Miller, MD Editor-in-Chief

References

1 Mayo Clinic Clinical Trials Website http://clinicaltrials.mayo.edu/glossary.cfm.

2 http://regentsprep.org/Regents/math/combin/probcomb.htm.

3 http://www.chem.qmul.ac.uk/software/download/qmc/ch5.pdf.

This Article Full Text (PDF) Submit a scholarly reply Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Siddiqui, F. Articles by Backman, S. B. Search for Related Content PubMed PubMed Citation Articles by Siddiqui, F. Articles by Backman, S. B.