| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
* From the Departments of Anesthesia, St. Michael’s Hospital,
Mount Sinai Hospital,
and the Sunnybrook and Women’s College Health Sciences Centre, University of Toronto, Toronto, Ontario, Canada.
Address correspondence to: Dr. Viren N. Naik, Department of Anaesthesia, St. Michael’s Hospital, University of Toronto, 30 Bond Street, Toronto, Ontario M5B 1W8, Canada. Phone: 416-864-5071; Fax: 416-864-6014; E-mail: naikv{at}smh.toronto.on.ca
 |
Abstract |
|---|
 TOP Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Methods: Residents unfamiliar with epidural anesthesia kept a log of their labour epidural successes and failures during a six-month hospital rotation. Failure was defined as a dural puncture or relinquishing the procedure to staff. Cusum analysis was performed using an acceptable failure rate of 10%. Residents were deemed competent when their graph remained below the calculated cusum boundary.
Results: Eleven anesthesia residents were recruited. The number of epidural attempts over six months ranged from 75 to 128. Ten residents attained competency by cusum between one and 85 attempts. One resident failed to achieve competency by cusum after 75 attempts.
Conclusion: After a period of training, residents are expected to perform the skill of labour epidural insertion independently. This study illustrates that some residents may need as many as 75 attempts to ensure proficiency. Training programs could use cusum to track the progress of their residents’ technical skills in order to guarantee an adequate experience.
|
Introduction |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Logbooks are commonly used to track a trainee’s progress with a particular skill. For example, some certifying bodies require that trainees perform a minimum number of procedures to be competent.2 Despite the limitations of defining competency with log books, they continue to be widely used because of their simplicity and feasibility.
Cumulative sum (cusum) analysis is a statistical and graphical tool that can be used to track the success and failure at a technical skill and examines trends over time.3 It can be used to demonstrate proficiency in a newly learned technical skill or as a measure of quality assurance once a technical skill has been mastered. When compared against a standard, cusum provides an objective tool that utilizes the data gathered through log books to determine whether or not a resident has achieved competency in a particular skill.4 Cusum has been previously described as a means of quality control in medical procedures such as colonoscopies.4 More recently, it has been used as a measure of competence for technical procedures learned by postgraduate anesthesia trainees.5
In our training program, we teach labour epidural analgesia to anesthesiology residents during a six-month rotation in their first year of clinical anesthesia. Supervision of this procedure decreases with increasing experience and success. We expect residents to be proficient in performing this skill independently at the end of this time. Clearly, an objective assessment of proficiency would be beneficial to declare competency.
The purpose of this study was to determine the number of epidural attempts required to attain proficiency as defined by cusum and determine the number of residents in our sample who obtained proficiency in the six-month period.
|
Methods |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
All subjects received a formal 45-min lecture on performing an epidural for labour analgesia. All subjects also received an informal one-to-one didactic session with a staff anesthesiologist, which included one demonstration on a patient before their first attempt in the clinical setting. These interventions are standard prior to the first patient attempt in our training program. Subjects were then observed directly and coached verbally by staff anesthesiologists during their first several attempts. The degree and duration of supervision varied depending on a subjective evaluation made by the staff anesthesiologist.
During their six-month rotation, subjects performed labour epidurals for all patients requesting epidural analgesia. Consent was not required from patients because labour epidural analgesia by trainees is part of routine clinical care and patients were not considered "subjects" of the study.
The subjects recorded the success or failure of each labour epidural attempt sequentially in a log book immediately after each procedure. We defined a successful epidural as an independently placed epidural catheter that provided some degree of analgesia, without physical assistance from a staff anesthesiologist. Advice on the choice of epidural site and verbal instruction was allowed. We defined a failed epidural as a dural puncture or if physical assistance by a staff anesthesiologist was required. In our training program, residents must ask for staff assistance after three unsuccessful attempts on the same patient. We did not consider "inadequate analgesia" as a failure because of its multifactorial etiology in this setting. The subjects returned their log books anonymously to the investigators at the end of the rotation.
The primary objective of the study was to determine the number of epidural attempts required to attain proficiency over the six-month training period. In addition, we wished to determine the number of subjects who became proficient in this period as defined by cusum.
Statistical analysis
We used Minitab 12.2 (Minitab version 12.2, Minitab Inc., State College, PA, USA) for all statistical and graphical analyses. Subject demographics were analyzed using descriptive statistics.
Cusum analysis
To perform a cusum analysis, an acceptable and unacceptable failure rate must be determined. For this study, an acceptable labour epidural failure rate (fo) of 10% (0.10) was determined by departmental consensus using a modified Delphi approach.6 Given an acceptable failure rate of 10%, the unacceptable failure rate (f1) was set by the statistical software at 15% (0.15).
The cusum graphical trend is summarized as follows:
Sn = 
(Xn - fo) where Xn = 0 for success and 1 for failure, n is the number attempted, and fo = the acceptable failure rate.5,7
We then constructed a graph for each subject such that they increased by a value of 1-fo (0.90) with failures and decreased by a value of fo (0.10) with successes. For example, three epidural failures followed by a success would take the values 0.9, 1.8, 2.7, and 2.6.
Graphical boundary lines were calculated as follows:7
where P = ln (f1/fo), Q = ln (1-fo)/(1-f1), a = ln (1-ß)/
, and b = ln (1-)/ß
Cusum declares competence or incompetence when the graphical trend crosses the lower boundary limit (ho) or the upper boundary limit (h1). Type I error () is the risk of declaring competence when it is not achieved. Type II error (ß) is the risk of not declaring competence when it has been attained. To calculate our boundary lines, the probability of type I (), and II (ß) were both set at 0.1. By setting the type I and II error rates to be equal, ho = h1, and subsequent boundary lines can be constructed as multiples of ho.4 Competency is achieved when a subject’s graphical trend fell below two adjacent calculated boundary lines (ho, 2ho, 3ho, 4ho,...). Competency was lost if the graphs again ascended and crossed two calculated boundary lines.
Individual cusum graphs were plotted for each student.
|
Results |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
.
|
and 2 are examples of cusum graphs for two trainees. The point at which each graph crossed and remained below two boundary lines was recognized as the critical number of attempts for competence.
|
|
).
|
|
Discussion |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Logbooks are utilized by many subspecialties including anesthesiology. Their popularity is based primarily on their simplicity and feasibility because direct observation is not always possible. However, educators often criticize logbooks because committees decide by consensus rather than experimentation the number of procedures a trainee requires to be deemed competent.11
Cusum analysis may serve to improve the validity of logbooks for several reasons. First, it has the potential of providing the trainee with timely graphical feedback. This graphical representation can function as a powerful kind of formative assessment throughout the training process for both the trainee and the educational program. Cusum analysis can be used to identify students with persistent difficulties early in their education. More importantly, it can identify a training program that does not provide students with adequate exposure to a particular procedure. Pietroni argued strongly that if a formative assessment is done well "it identifies strengths and weaknesses not only in the trainee, but also in the trainer and the institution."12 Cusum analysis is a useful tool in promoting both of these goals of assessment.
Trainees who obtained competency by cusum did so within a wide range of epidural attempts. One trainee with no prior experience, never failed to correctly place a labour epidural catheter. As many as 9% of residents may continue to have difficulty performing labour epidurals after six months of regular exposure to the skill. One trainee failed to achieve competency after as many as 75 epidural attempts. One other study has examined labour epidural proficiency using the cusum method. In this study, five of 12 residents obtained competency after 29 to 185 procedures with a 5% acceptable failure rate.5 This wide variation is similar to our findings and similar to the variation found with other intraoperative neuraxial procedures that have been analyzed using cusum.5,13
There were several limitations in this study. Patient characteristics were not recorded and as such, it is possible that subjects were exposed to different patient populations during their six-month rotation. Having said that, our patient population is similar to those encountered in both primary and tertiary centres. Secondly, catheter placement in the epidural space cannot be confirmed with absolute confidence from a patient’s subjective assessment of analgesia. By specifically examining the technical skill of labour catheter placement as opposed to catheter management, we were unable to use analgesia as an end-point for success. Repeating the study using nerve stimulation via the catheter to confirm proper placement would verify our results.14 Lastly, logbooks are criticized because self-reporting by trainees may not be truthful.2 All of our data was collected in an anonymous fashion and as such, may be more of an accurate reflection of the residents’ experience.
While cusum analysis is a powerful tool for formative assessment, its usefulness for summative evaluations requires further investigation. Specifically, further studies need to be undertaken that strengthen the validity of cusum analysis by comparing its results with other more valid measures of technical skills assessment, such as direct observation with task specific checklists and global rating assessments.2,15 With further validation, cusum analysis could provide educators with a practical measure of competency for this and other skills. Furthermore, cusum analysis could also be used to assess the efficacy of future teaching interventions including practice on models and virtual reality training.16,17
In conclusion, this study illustrates that residents at our institution achieved labour epidural proficiency by cusum anywhere between one and 85 attempts. More importantly, it shows that some residents may require as many as 75 attempts to ensure proficiency. Training programs could consider cusum as a useful tool to track the progress of their residents’ technical skills.
|
Acknowledgments |
|---|
|
Footnotes |
|---|
Revision received April 25, 2003. Accepted for publication November 28, 2002.
|
References |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
2 Watts J, Feldman WB. Assessment of technical skills. In: Neufeld VR, Norman GR (Eds.). Assessing Clinical Competence. New York: Springer Publishing Company; 1989.
3 Altman DG, Royston JP. The hidden effect of time. Stat Med 1988; 7: 629–37.[Medline]
4 Williams SM, Parry RP, Schlup MMT. Quality control: an application of the cusum. Br Med J 1992; 304: 1359–61.
5 Kestin IG. A statistical approach to measuring the competence of anaesthetic trainees at practical procedures. Br J Anaesth 1995; 75: 805–9.
6 Stewart J, O’Halloran C, Harrigan P, Spencer JA, Barton JR, Singleton SJ. Identifying appropriate tasks for the preregistration year: modified Delphi technique. BMJ 1999; 319: 224–9.
7 Davies OL. The Design and Analysis of Industrial Experiments. London: Longman; 1978.
8 Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 1997; 84: 273–8.[Medline]
9 Muzzin LJ, Hart L. Oral examination. In: Neufeld VR, Norman GR (Eds). Assessing Clinical Competence. New York: Springer Publishing Co.; 1985: 71–93.
10 Gates EA. New surgical procedures: can our patients benefit while we learn. Am J Obstet Gynecol 1997; 176: 1293–9.[Medline]
11 Norris TE, Cullison SW, Fihn SD. Teaching procedural skills. J Gen Intern Med 1997; 12: S64–9.
12 Pietroni M. The assessment of competence in surgical trainees. Ann R Coll Surg Engl 1993; 75: 200–2.[Medline]
13 Filho GR. The construction of learning curves for basic skills in anesthetic procedures: an application for the cumulative sum method. Anesth Analg 2002; 95: 411–6.
14 Tsui BC, Gupta S, Finucane B. Confirmation of epidural catheter placement using nerve stimulation. Can J Anaesth 1998; 45: 640–4.
15 Zeller RA. Validity. In: Walbey HJ, Haertel GD (Eds.). The International Encyclopedia of Education Evaluation. Oxford: Pergamon Press; 1990: 251–9.
16 Prystowsky JB, Regehr G, Rogers DA, Loan JP, Hiemenz LL, Smith KM. A virtual reality module for intravenous catheter placement. Am J Surg 1999; 177: 171–5.[Medline]
17 Reznick RK. Teaching and testing technical skills. Am J Surg 1993; 165: 358–61.[Medline]
This article has been cited by other articles:
 |
|
 |
|
  D. J Biau, M. Resche-Rigon, G. Godiris-Petit, R. S Nizard, and R. Porcher Quality control of surgical and interventional procedures: a review of the CUSUM Qual. Saf. Health Care, June 1, 2007; 16(3): 203 - 207. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J Taitz, B Wyeth, R Lennon, P D Torre, T Yen, B Harrison, and M Cattell Effect of the introduction of a lumbar-puncture sticker and teaching manikin on junior staff documentation and performance of paediatric lumbar punctures. Qual. Saf. Health Care, October 1, 2006; 15(5): 325 - 328. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
