This Article Résumé de cet Article Full Text Full Text (PDF) Additional Material Submit a scholarly reply Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Related articles in CJA Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Jacobsohn, E. Search for Related Content PubMed PubMed Citation Articles by Jacobsohn, E.

Poor inter-rater reliability on mock anesthesia oral examinations

[Le pauvre coefficient d’objectivité d’examens oraux en anesthésie simulés]

Eric Jacobsohn, MBCHB MHPE FRCPC^*, P. Alan Klock, MD, Michael Avidan, MBBCH^* and the Oral Examinations Group

^* From the Departments of Anesthesiology and Surgery, Washington University School of Medicine, St. Louis, MO; and the
Anesthesia and Critical Care, University of Chicago, Chicago, Illinois, USA;
(see Appendix II).

Address correspondence to: Dr. Eric Jacobsohn, Department of Anesthesiology, Washington University School of Medicine, 660 S. Euclid, MC 8054, St. Louis, MO 63110, USA. Phone: 314-747-4155; Fax: 314-362-4551; E-mail: jacobsoe{at}msnotes.wustl.edu

Purpose: Inter-rater reliability (IRR) and communication skills are both important factors that have been shown to affect oral examination scores. This study was designed to test: 1) IRR of a group of American Board of Anesthesiology (ABA) and the Royal College of Physicians and Surgeons of Canada (RCPSC) graders who graded in true isolation; 2) the effect of teaching residents examination techniques.

Methods: This was a randomized, pretest-posttest trial. Twenty-five residents did an initial oral examination (E1) resembling the ABA examination. They were then randomized into two groups, a routine education group, and an intervention group that was taught oral examination skills. Six weeks later they did another oral examination (E2). The videotaped examinations were subsequently scored by six experienced RCPSC and ABA graders.

Results: There was very poor IRR on E1 (weighted Kappa = 0.166, intraclass correlation coefficient 0.243), which improved only slightly on E2 (weighted Kappa = 0.275, P = NS; intraclass correlation coefficient = 0.405, P < 0.01). Pass rate for graderpairs increased from E1 to E2 (15% vs 43%, P = 0.01). The improved pass rate on E2 occurred in both the routine education group and in the intervention group. There was no significant difference between RCPSC and ABA graders. Teaching examination skills per se did not improve performance, but this conclusion may be limited by the poor IRR. Practice orals do appear to improve performance on future examinations.

Conclusions: Inter-rater reliability may be poor when graders score an oral examination in true isolation. Teaching candidates an oral examination communication and presentation technique did not appear to improve performance. Oral examination practice may be of value in training for future examinations.

Related articles in CJA:

The oral examination process – gold standard or fool’s gold/Le processus d’examen oral - un vrai ou un faux étalon-or ?

Patricia Houston, Ramona A. Kearney, and Georges Savoldelli
CJA 2006 53: 639-642. [Full Text]  

This article has been cited by other articles:

				S. C. Hall Poor inter-rater reliability on mock anesthesia oral examinations Can J Anesth, December 1, 2006; 53(12): 1268 - 1269. [Full Text] [PDF]

				P. Houston, R. A. Kearney, and G. Savoldelli The oral examination process - gold standard or fool's gold/Le processus d'examen oral - un vrai ou un faux etalon-or ? Can J Anesth, July 1, 2006; 53(7): 639 - 642. [Full Text] [PDF]