This Article Résumé de cet Article Full Text Full Text (PDF) Submit a scholarly reply Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Morgan, P. J. Articles by Herold, J. Search for Related Content PubMed PubMed Citation Articles by Morgan, P. J. Articles by Herold, J.

Validity and reliability of undergraduate performance assessments in an anesthesia simulator

Pamela J. Morgan, MD CCFP FRCPC^*, Doreen M. Cleave-Hogg, BA MA PhD, Cameron B. Guest, MD FRCPC MED^* and Jodi Herold, BSc (Pt) MA

^* From the Department of Anesthesia, Sunnybrook andthe Centre for Research in Education,
Women's College Health Sciences Centre, University of Toronto and University of Toronto, Toronto, Ontario, Canada.

Address correspondence to: Dr. P.J. Morgan, Department of Anesthesia, Sunnybrook & Women's College Health Sciences Centre, Women's College Campus, 76 Grenville Street, Toronto, Ontario M5S 1B2 Canada. Phone: 416-323-6400 Ext. 4349; Fax: 416-323-6307; E-mail: pam.morgan{at}utoronto.ca

Purpose: To examine the validity and reliability of performance assessment of undergraduate students using the anesthesia simulator as an evaluation tool.

Methods: After ethics approval and informed consent, 135 final year medical students and 5 elective students participated in a videotaped simulator scenario with a Link-Med Patient Simulator (CAE-Link Corporation). Scenarios were based on published educational objectives of the undergraduate curriculum in anesthesia at the University of Toronto. During the simulator sessions, faculty followed a script guiding student interaction with the mannequin. Two faculty independently viewed and evaluated each videotaped performance with a 25-point criterion-based checklist. Means and standard deviations of simulator-based marks were determined and compared with clinical and written evaluations received during the rotation. Internal consistency of the evaluation protocol was determined using inter-item and item-total correlations and correlations of specific simulator items to existing methods of evaluation.

Results: Mean reliability estimates for single and average paired assessments were 0.77 and 0.86 respectively. Means of simulator scores were low and there was minimal correlation between the checklist and clinical marks (r=0.13), checklist and written marks (r=0.19) and clinical and written marks (r=0.23). Inter-item and item-total correlations varied widely and correlation between simulator items and existing evaluation tools was low.

Conclusions: Simulator checklist scoring demonstrated acceptable reliability. Low correlation between different methods of evaluation may reflect reliability problems with the written and clinical marks, or that different aspects are being tested. The performance assessment demonstrated low internal consistency and further work is required.

This article has been cited by other articles:

				M. B. Brett-Fleegler, R. J. Vinci, D. L. Weiner, S. K. Harris, M.-C. Shih, and M. E. Kleinman A Simulator-Based Tool That Assesses Pediatric Resident Resuscitation Competency Pediatrics, March 1, 2008; 121(3): e597 - e603. [Abstract] [Full Text] [PDF]

				P. J. Morgan, J. Lam-McCulloch, J. Herold-McIlroy, and J. Tarshis Simulation performance checklist generation using the Delphi technique: [Generation d'une liste de verification de la performance simulee a l'aide de la methode Delphi] Can J Anesth, December 1, 2007; 54(12): 992 - 997. [Abstract] [Full Text] [PDF]

				H. Berkenstadt, G. S. Kantor, Y. Yusim, N. Gafni, A. Perel, T. Ezri, and A. Ziv The Feasibility of Sharing Simulation-Based Evaluation Scenarios in Anesthesiology Anesth. Analg., October 1, 2005; 101(4): 1068 - 1074. [Abstract] [Full Text] [PDF]

				A. K. Wong Full scale computer simulators in anesthesia training and evaluation: [Des simulateurs informatises grandeur nature pour la formation et l'evaluation en anesthesie] Can J Anesth, May 1, 2004; 51(5): 455 - 464. [Abstract] [Full Text] [PDF]

				P. J. Morgan, D. Cleave-Hogg, S. DeSousa, and J. Tarshis High-fidelity patient simulation: validation of performance checklists Br. J. Anaesth., March 1, 2004; 92(3): 388 - 392. [Abstract] [Full Text] [PDF]

				J. M. Weller, M. Bloch, S. Young, M. Maze, S. Oyesola, J. Wyner, D. Dob, K. Haire, J. Durbridge, T. Walker, et al. Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists Br. J. Anaesth., January 1, 2003; 90(1): 43 - 47. [Abstract] [Full Text] [PDF]

				P. J. Morgan and D. Cleave-Hogg A worldwide survey of the use of simulation in anesthesia: [Une enquete mondiale sur l'usage de la simulation en anesthesie] Can J Anesth, August 1, 2002; 49(7): 659 - 662. [Abstract] [Full Text] [PDF]