| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
* From the Department of Anesthesia, St. Joseph's Healthcare and
McMaster University,
Hamilton, Ontario, Canada; the Departments of Anesthesia and Health Administration, University Health Network, University of Toronto,
Toronto, Ontario, Canada; and the Department of Anesthesia, University of Rochester, Rochester, New York, USA.
Address correspondence to: Dr. Peter T.-L. Choi, Department of Anesthesia, McMaster University, 1200 Main Street West, Room HSC-2U5, Hamilton, Ontario L8N 3Z5, Canada. Phone: 905-521-2100 ext. 75174; Fax: 905-523-1224; E-mail: choip{at}mcmaster.ca
 |
Abstract |
|---|
 TOP Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Methods: Computerized bibliographic searches, citation review, and hand searches were conducted to find all relevant citations on incidence, clinical course, prevention, or treatment of PDPH in parturients. The study design and topic(s) covered by each study were evaluated. Case-control studies and cohort studies were evaluated using the Quality Index; clinical trials were evaluated using the Jadad scale.
Results: One hundred ninety-six relevant citations were published between 1949 and 1999. Research on PDPH has been increasing rapidly with the majority of studies published in the 1990's. Incidence and prevention were the focus of over half of all citations. Optimal study designs were infrequently utilized. The methodological quality was poor for observational studies (Quality Index 10/29) and clinical trials (Jadad scale 2/5).
Conclusion: Although the amount of research on PDPH in parturients is increasing, use of optimal study designs and improvement in methodology is needed.
|
Introduction |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Inherent in the framework of evidence-based health care is the ability to obtain high quality information upon which decisions may be based. A comprehensive bibliographic database is important for several reasons. First, comprehensiveness minimizes the random error and various biases.3 Second, a comprehensive database permits assessment of patterns and trends in publication (type of study, reporting quality, methodological quality), clinical practice (types of intervention, changes in incidence or prevalence, perspectives of patients and clinicians), and research (direction of research efforts). Third, a comprehensive bibliographic database can be a valuable information resource to patients, clinicians, researchers, and policy makers by providing information in a single source and by minimizing duplication of search efforts. Fourth, such databases are the foundation for systematic reviews and meta-analysis, which are heavily dependent on comprehensive acquisition of all relevant data to generate valid conclusions.
Anesthesia-specific bibliographic databases are relatively infrequent. For example, over 100 years of literature on postdural puncture headache (PDPH) exists. Tourtellotte et al.4 systematically compiled a bibliography of the literature up to the 1960's; since then, no comprehensive compilation has been undertaken. In this paper, we report the development of a bibliographic database of the literature on obstetrical PDPH, describe the research architecture in this field, and evaluate the quality of observational and experimental studies of PDPH.
|
Methods |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
) was initially designed for the MEDLINE database and then tailored to the others. The reference list of each citation was reviewed for potential additional citations. Further citations were also sought from abstracts and conference proceedings of major anesthesia meetings from the year of their first publication to 1998 (Table II). We included citations that were not published in English. All citations were recorded in a Reference Manager® (Research Information Systems, Carlsbad, CA, USA) version 9 bibliographic database. The search was completed on February 19, 1999 and the database was closed nine days later.
|
|
). Definitions for each criteria were developed by consensus by four of the authors (PTC, SEG, SL, LT) and field tested over three sessions using randomly selected citations from the literature search. A citation was included if it met all four criteria for any of the four topics; it was possible for a citation to meet all the inclusion criteria for more than one topic. Once the inclusion criteria were established, two of us (PTC, SEG) reviewed all the citations identified by the literature search. For non-English citations, translation was obtained when needed. Disagreements were resolved by consensus. The extent of agreement was recorded and the inter-rater reliability (expressed as kappa ± standard error)5 was calculated for the four topics.
|
Evaluation of quality
Two reviewers (PTC, SEG) independently evaluated the quality of observational and experimental studies listed in MOPED. In the event of duplicate publication, the publication with the most methodological information was used. For case-control studies and cohort studies, the Quality Index6 was used. The Quality Index is a 27-item, partially validated checklist that assesses the reporting quality, generalizability, internal validity, and power of non-randomized and randomized studies.6 For non-randomized studies, scores may range from 0 (poor) to 29 (excellent). Controlled clinical trials were evaluated with the Jadad scale,7 which is a validated three-item scale that evaluates randomization, blinding, and description of withdrawals. Possible scores range from 0 (not randomized, not double-blind, no description of withdrawals) to 5 (randomized appropriately, double-blinded appropriately, withdrawals described).
Reviewers were not blinded to the author(s), journal, or year of publication. Disagreements between the two reviewers were resolved by consensus. The extent of agreement was recorded and the inter-rater reliability was calculated using generalizability theory5 for the two instruments used for rating quality. As the distributions of the quality scores were not expected to be normal, the median and range were chosen as the measures of central tendency and dispersion. We compared the quality scores of abstracts and full papers using the Mann-Whitney U nonparametric test. All statistics were performed using SPSS© 8.0 for Windows (SPSS Inc, Chicago, IL, USA).
|
Results |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Research architecture
One hundred ninety-six citations (73.7% of retrieved citations) met selection criteria for at least one topic.a Information on incidence, clinical course, prevention, and treatment was present in 75.5%, 5.3%, 52.0%, and 15.3% of the citations meeting selection criteria respectively. Inter-rater reliability was high for all topics (incidence k 0.82 ± 0.04; prevention k 0.91 ± 0.03; treatment k 0.98 ± 0.02) except clinical course (k 0.24 ± 0.18).
The classification of the selected citations by topic and study type is outlined in Table IV. The classification has been delineated by categories similar to the "levels of evidence" described by Sackett.8 Only one-third of citations (34.7%) were systematic reviews or randomized controlled trials (RCTs). The majority of citations (53.1%) were observational reports.
|
Publication details
Citations were published in 27 journals and seven languages (English, Finnish, French, German, Norwegian, Spanish, and Chinese). The majority of citations (176 of 196; 89.8%) were published in anesthesia journals. Over three-quarters of all selected citations (151 of 196; 77.0%) were published in the six anesthesia journals with the highest citation impact factors (CIF): Anesthesiology, Anesthesia and Analgesia, Regional Anesthesia and Pain Medicine, British Journal of Anaesthesia, Anaesthesia, and Canadian Journal of Anesthesia (in descending order of CIF). Most citations (187 of 196; 95.4%) were published in English.
Table V classifies the citations by the decade in which they were published and by the category of evidence. The number of citations in each category increased over each decade. The majority of citations (140 of 196; 71.4%) were published in the 1990's. Experimental studies (RCTs and non-RCTs) have increased more than observational studies although the latter still constituted more than 50% of all selected citations in the 1990's. MEDLINE and the Cochrane Library indexed selected citations from 1972 to 1999 and 1950 to 1998 respectively. Citation review and manual searches found selected citations from 1949 to 1995 and 1975 to 1999 respectively. Thirty-eight (29.4%) of the 129 selected citations found by citation review and manual searches were present in MEDLINE but unidentified by the computerized search strategy.
|
Quality assessment of case-control and cohort studies
There were 24 case-control or cohort studies; three duplicate publications were excluded. Twenty-one case-control or cohort studies (14 full papers, seven abstracts) were assessed using the Quality Index. The scores ranged from 2 to 19 with a median score of 10. Table VI enumerates the median and the range for each section of the Quality Index. The inter-rater reliability was 0.68, which is similar to the inter-rater reliability obtained by the original developers of the index.6
|
). No study discussed issues of power or adjusted for confounding in data analyses. Less than half of the 21 studies clearly described characteristics of study participants and those lost to follow-up nor reported all important adverse events. A similar proportion did not report estimates of random variability or the actual probability values for the main outcomes. Issues relating to external validity, blinding, and adjustment for differing lengths of time or follow-up were also poorly addressed. The remaining issues were addressed adequately in over half of the studies: description of the intervention was the item addressed most frequently (95.2%).
Quality assessment of randomized clinical trials
There were 65 RCTs; ten duplicates were excluded. Fifty-five RCTs (38 full papers, 17 abstracts) were assessed using the Jadad scale. The scores ranged from 0 to 5 with a median score of 2. Table VII describes the proportion of responses in each category per question. The inter-rater reliability was 0.58, which is consistent with the reliability obtained by Jadad et al.7
|
|
Discussion |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The findings of the various strategies employed to seek out relevant information reveal the danger of utilizing only a single strategy. The computerized search of MEDLINE found 113 citations, in which 60 were selected ("true positives"). Thirty-eight citations found by citation review or hand search were indexed in MEDLINE but were missed by the computerized search ("false negatives"). The remaining 91 citations obtained by citation review or hand search were abstracts or articles from journals not cited in any of the four computerized bibliographic databases. The sensitivity of the MEDLINE search strategy for all MEDLINE-indexed citations was only 61.2% (60 / 98). For all selected citations, the sensitivity of the MEDLINE strategy dropped to 30.6% (60 / 196). Most of the citations included in MOPED would have been missed if reliance had been placed on the MEDLINE search alone.
Our computerized searches did not include the EMBASE or LILACS databases. EMBASE and LILACS indexes many European and Latin American journals respectively. At the time of this study, neither EMBASE nor LILACS were freely available to the public and the costs of performing the searches were beyond our allotted budget for this project. (LILACS can be accessed now, free of charge, under "Scientific literature" at the website http://www.bireme.br/bvs/l/ihome.htm ). There is a possibility that we missed relevant European or Latin American literature during the development of MOPED; however, this is unlikely at least for RCTs and systematic reviews as the Cochrane Library routinely searches the two databases for citations through its different Cochrane Centers across the world.
In selecting citations for inclusion in MOPED, individuals with no experience in research methodology were trained to read and evaluate the contents of papers identified by the literature search. Other researchers have found that valid and reliable results can be obtained by training students to screen for methodological content.13 The inter-rater reliability for selecting articles on incidence, prevention, and treatment all were high (k 0.82, 0.92, and 0.98 respectively), which is consistent with near perfect agreement.14 Reliability was low (k 0.24; fair agreement)14 for selecting articles on clinical course. There are likely two reasons for the fair strength of agreement. First, differences existed amongst the two reviewers in the interpretation of the outcome used in the inclusion criteria for clinical course despite extensive field testing. At the consensus meeting, both reviewers were able to resolve disagreements very quickly once the difference in interpretation was brought to light. Second, the k statistic is a function of sensitivity, specificity, and prevalence.15 For any given sensitivity and specificity, a decrease in prevalence will deflate the k value.15,16 The low prevalence of citations on clinical course may have contributed to the low k value observed in this topic.
The type of studies published in each topic was also revealing. Optimally, the incidence and clinical course of PDPH would be examined using prospective follow-up of cohorts (with survival-type analysis for the latter topic) and interventions for prevention and treatment would be studied using RCTs. A survey of the literature indicates that the optimal study designs for incidence, clinical course, and treatment were infrequently utilized. Prevention of PDPH was an exception with the majority of studies being RCTs. This observation highlights a problem in the practice of evidence-based medicine with respect to PDPH. Already, the clinician is faced with a hurdle: most of the available information upon which reliance is placed for evidence-based decision making regarding PDPH is not optimal in the study design.
Attempts to minimize language bias were made by including citations of all languages although it is possible that relevant citations not published in English may have been missed using the described search strategy. Few citations were published in other languages.
Nearly one-fifth of the citations in our database contained duplicate data. Most of the duplicates involved abstracts that were subsequently published as full papers. Of the 67 abstracts, 12 were eventually found as full papers. These observations raise an interesting dilemma. Meeting abstracts are often the earliest reports of studies and can contain important information,17 especially since only half of all abstracts are published in full.18,19 However, the peer review process may be less stringent than for full manuscripts and the quality of abstracts may be low.20 As well, the findings of abstracts may differ drastically from the subsequent papers. In a subset of obstetrical anesthesia abstracts, Halpern et al. found a large number of discrepancies between the abstracts and the full papers.21 The choice to include or exclude abstracts remains controversial.
Our findings echo previous observations of the validity and statistical concerns in anesthesia evidence. There is room for improvement in the quality of non-randomized and randomized studies of PDPH in obstetrical patients. Median quality scores were low for both types of studies. There were no differences in quality between full papers and meeting abstracts.
This study is the first to evaluate the quality of observational studies in anesthesia. Both reporting and methodological quality (based on external validity, bias, confounding, and power) were poor with median scores less than half of the maximum possible score per category. No observational study in the examined set evaluated power. This observation reiterates the concerns raised previously by other methodologists who have evaluated anesthesia studies.22,23 Similarly, the results of these studies were infrequently generalizable. Issues with bias and confounding were also poorly addressed.
Based on these observations, we would suggest the following changes to improve quality of observational studies in this field:
1. Subjects should be representative of the entire population of potential participants.
2. Where possible, patients of different cohorts (for cohort studies) or cases and controls (for case-control studies) should be recruited from the same population over the same time period to minimize confounders relating to population and time.
3. Where possible, subjects and individuals measuring the outcomes should be blinded. The same individuals should measure all groups.
4. Patient characteristics, possible confounders, interventions, and all relevant outcomes and adverse effects should be reported clearly. A flow diagram may assist the reader in tracking the fate of patients.
5. Follow-up should be reported along with a description of patients who withdraw or are lost to follow-up.
6. Analyses need to adjust for confounders and time factors.
7. Power analysis should be performed to ensure an adequate sample size for the effect that one wishes to detect. The analysis should be reported.
The weaknesses observed in the RCTs in MOPED were similar to those reported by Bender et al.,24 whose set of studies was in a similar population (obstetrical patients). Again, the majority of RCTs gave inadequate description of the randomization process (72%), blinding (63%), and withdrawals (54%). Our study, which evaluated RCTs from 1949 to 1999, confirms that further work is needed to improve the quality of RCTs.
As mentioned previously, part of the difficulty in evaluating methodological quality is due to poor reporting. Often for blinding and withdrawal, we could not differentiate between failure to perform blinding and track withdrawals or failure to report them. None of the RCTs were reported using standardized reporting criteria. Given the inadequacies of reports examined in anesthesia so far, we advocate the adoption of standardized criteria such as the CONSORT statement25 to improve quality of RCTs.
In summary, we have compiled a bibliographic database that covers the literature on obstetrical PDPH from 1949 to 1999. We hope to update this database on a regular basis. Examination of the architecture of obstetrical PDPH research and the quality of observational and experimental studies indicate that further work is needed to improve the research in this field.
|
Acknowledgments |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
|
Footnotes |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Revision received September 26, 2001. Accepted for publication August 27, 2001.
|
References |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
2 McKibbon A, Eady A, Marks S. PDQ Evidence-Based Principles and Practice. Hamilton: BC Decker Inc., 1999.
3
Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 1997; 127: 380–7.
4 Tourtellotte WW, Haerer AF, Heller GL, Somers JE. Post-Lumbar Puncture Headaches. Springfield, IL: Charles C. Thomas, 1964.
5 Streiner DL, Norman GR. Health Measurement Scales. A Practical Guide to Their Development and Use, 2nd ed. Oxford: Oxford University Press, 1995.
6 Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality of both randomised and non-randomised studies of health care interventions. J Epidemiol Community Health 1998; 52: 377–84.[Abstract]
7 Jadad AR, Moore A, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996; 17: 1–12.[Medline]
8 Sackett DL. Rules of evidence and clinical recommendations. Can J Cardiol 1993; 9: 487–9.[Medline]
9 Jadad-Bechara AR. Meta-Analysis of Randomised Clinical Trials in Pain Relief [D. Phil. Thesis]. Oxford: University of Oxford, 1994.
10 Jadad AR, McQuay HJ. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol 1996; 49: 235–43.[Medline]
11 Halpern SH, Jadad AR, Choi PT-L. Evidence based practice in anaesthesia – how good is the evidence? In: Tramèr MR (Ed.). An Evidence Based Resource in Anaesthesia and Analgesia. London: BMJ Books, 2000: 27–44.
12
Choi PT-L, Halpern SH, Malik N, Jadad AR, Tramèr MR, Walder B. Examining the evidence in anesthesia literature: a critical appraisal of systematic reviews. Anesth Analg 2001; 92: 700–9.
13 Sands ML, Murphy JR. Use of kappa statistic in determining validity of quality filtering for meta-analysis: a case study of the health effects of electromagnetic radiation. J Clin Epidemiol 1996; 49: 1045–51.[Medline]
14 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–74.[Medline]
15 Spitznagel EL, Helzer JE. A proposed solution to the base rate problem in the kappa statistic. Arch Gen Psychiatry 1985; 42: 725–8.[Abstract]
16 Grove WM, Andreason NC, McDonald-Scott P, Keller MB, Shapiro RW. Reliability studies of psychiatric diagnosis. Theory and practice. Arch Gen Psychiatry 1981; 38: 408–13.[Abstract]
17 Kelly JA. Scientific meeting abstracts: significance, access, and trends. Bull Med Libr Assoc 1998; 86: 68–76.[Medline]
18
Yentis SM, Campbell FA, Lerman J. Publication of abstracts presented at anaesthesia meetings. Can J Anaesth 1993; 40: 632–4.
19 Scherer RW, Dickersin K, Langenberg P. Full publication of results initially presented in abstracts. A meta-analysis. JAMA 1994; 272: 158–62.[Abstract]
20 Rubin HR, Redelmeier DA, Wu AW, Steinberg EP. How reliable is peer review of scientific abstracts? Looking back at the Meeting of the Society of General Internal Medicine. J Gen Intern Med 1993; 8: 255–8.[Medline]
21 Halpern SH, Palmer S, Angle P, Tarshis J. Published abstracts in obstetrical anesthesia: full publication rates and data reliability. Anesthesiology 2001; 94(1A): A69 (abstract).
22
Goodman NW, Hughes AO. Statistical awareness of research workers in British anaesthesia. Br J Anaesth 1992; 68: 321–4.
23
Goodman NW, Powell CG. Could do better: statistics in anaesthesia research (Editorial). Br J Anaesth 1998; 80: 712–4.
24
Bender JS, Halpern SH, Thangaroopan M, Jadad AR, Ohlsson A. Quality and retrieval of obstetrical anaesthesia randomized controlled trials. Can J Anaesth 1997; 44: 14–8.
25 Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996; 276: 637–9[Medline]
This article has been cited by other articles:
 |
|
 |
|
  D. H. Dyson and S. C. Sparling Delay in Final Publication Following Abstract Presentation: American College of Veterinary Anesthesiologists Annual Meeting J Vet Med Educ, January 1, 2006; 33(1): 145 - 148. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 P. Angle, S. L. T. Tang, D. Thompson, and J. P. Szalai Expectant management of postdural puncture headache increases hospital length of stay and emergency room visits: [Le traitement symptomatique de la cephalee post-ponction durale augmente la duree du sejour hospitalier et les visites a la salle d'urgence] Can J Anesth, April 1, 2005; 52(4): 397 - 402. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Y.C. Wong Is PDPH from a 25-gauge Whitacre needle always short-lasting and self-resolving? Can J Anesth, June 1, 2004; 51(6): 637 - 637. [Full Text] [PDF]
|
|
|
|
|
|
P. T. Choi, S. E. Galinski, L. Takeuchi, S. Lucas, C. Tamayo, and A. R. Jadad PDPH is a common complication of neuraxial blockade in parturients: a meta-analysis of obstetrical studies: [Les cephalees post-ponction durale sont une complication courante du bloc neuraxial chez les parturientes : une meta-analyse d'etudes obstetricales] Can J Anesth, May 1, 2003; 50(5): 460 - 469. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
