This Article Résumé de cet Article Full Text Full Text (PDF) Submit a scholarly reply Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Kearney, R. A. Articles by Skakun, E. N. Search for Related Content PubMed PubMed Citation Articles by Kearney, R. A. Articles by Skakun, E. N.

The inter-rater and intra-rater reliability of a new Canadian oral examination format in anesthesia is fair to good

[La fiabilité interexaminateurs et intra-examinateurs d'un nouveau modèle d'examen oral canadien en anesthésie est de moyenne à bonne]

Ramona A. Kearney, MD FRCPC^*, Stephen A. Puchalski, MD FRCPC, Homer Y.H. Yang, MD FRCPC and Ernest N. Skakun, PhD^*

^* From the Departments of Anesthesia University of Alberta, Edmonton, Alberta, and
McMaster University, The Division of Studies in Medical Education, Hamilton, Ontario, Canada.

Dr. Ramona A. Kearney, Department of Anesthesiology and Pain Medicine, University of Alberta, Clinical Sciences Building, Room 8-120G, Edmonton, Alberta T6G 2B7, Canada. Phone: 780-407-2689; Fax: 780-407-7461; E-mail: rkearney{at}ualberta.ca

Purpose: In response to the Royal College's request to improve the validity and reliability of oral examinations, the Examination Board in anesthesia proposed a structured oral examination format. Prior to its introduction, we studied this format in two residency programs to determine reliability of the examiners.

Methods: Twenty faculty and 26 residents from two Canadian residency programs participated (Sites A and B). Pairs of examiners scored five or six residents examined consecutively on two standardized questions using global rating scales with anchored performance criteria. Residents' performances were scored independently during the examination (Time 1) and later from a videotaped recording (Time 2). Correlations between scores of the pairs of examiners and between scores of each examiner were determined.

Results: Correlations demonstrating inter-rater agreement between examiners at Site A ranged from -.324 to .915 (mean .506) at Time 1. At Time 2, correlations ranged from .64 to .887 (mean .791). At Site B correlations ranged from .279 to .989 (mean .707) at Time 1 and at Time 2 correlations ranged from -.271 to .924 (mean .477).

Correlations demonstrating intra-rater agreement of examiners at Site A ranged from .054 to .983 (mean .723) and at Site B correlations ranged from -.055 to .974 (mean .662).

Correlations >0.4 were seen in 80% of the scores and >0.7 in 50% indicating fair to good intra-rater and inter-rater reliability using this format.

Conclusions: Despite the limitations of our study our results compare favourably with those previously reported in anesthesia. We recommend the adoption of this format to the Royal College of Physicians and Surgeons of Canada Examination Board.

This article has been cited by other articles:

				P. Houston, R. A. Kearney, and G. Savoldelli The oral examination process - gold standard or fool's gold/Le processus d'examen oral - un vrai ou un faux etalon-or ? Can J Anesth, July 1, 2006; 53(7): 639 - 642. [Full Text] [PDF]

				E. Jacobsohn, P. A. Klock, M. Avidan, and the Oral Examinations Group Poor inter-rater reliability on mock anesthesia oral examinations: [Le pauvre coefficient d'objectivite d'examens oraux en anesthesie simules]. Can J Anesth, July 1, 2006; 53(7): 659 - 668. [Abstract] [Full Text] [PDF]