| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
From the Departments of Anesthesiology and Pain Medicine, Division of Studies in Medical Education, University of Alberta and University of Ottawa, Canada.
Address correspondence to: Dr. R.A. Kearney, Department of Anesthesiology and Pain Medicine, University of Alberta, Room 3B2.32, 8440 – 112 Street, Edmonton, Alberta T6G 2B7 Canada. Phone: 780-407-2689; Fax: 780-407-3200; E-mail: rkearney{at}gpu.srv.ualberta.ca
 |
Abstract |
|---|
 TOP Abstract IntroductionMethodsResultsDiscussionConclusionReferences |
|---|
Methods: All English speaking Canadian residency programs in Anesthesia were surveyed. Results of the ABA-ASA ITE of former residents who had completed RCPSC certification were collected as percentile scores according to level of training. Level of training was based on the number of months of anesthesia training and classified according to American residency program nomenclature. The ABA-ASA ITE scores were correlated with success on the RCPSC written and oral examinations. The probability of success on the RCPSC examinations was determined by calculating the cutoff score with the best sensitivity and specificity as determined by Receiver Operating Characteristic (ROC) curves for each year in which the examination was taken and for both the written and oral examinations.
Results: Nine residency programs provided information on 165 residents. A weak positive correlation was found between scores on each year of the ABA-ASA exam. Scores > 50th percentile for any year were highly predictive of success in the written component ( > 60th percentile for the oral component). Scores < 20th percentile were predictive of failure on both the written and oral components of the RCPSC examination.
Conclusion: The ABA-ASA ITE is a useful tool in predicting performance on the RCPSC examination.
|
Introduction |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
For many years, Canadian residency programs in Anesthesia have taken advantage of the American Board of Anesthesiology-American Society of Anesthesiologists (ABA-ASA) in-training examination (ITE). This is a multicentre multiple choice examination administered annually and residents obtain a percentile ranking according to level of training. This is a high quality examination which can assess the resident's knowledge base. Program directors try to infer from the examination results whether the resident will be successful on the Canadian certification examinations. The relevance of the percentile scores to the RCPSC examination is unknown, making predictions difficult. The purpose of this study is to investigate the usefulness of the ABA-ASA ITE in predicting success on the RCPSC certification examinations.
|
Methods |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
Because the American residency programs are four years long and have a different nomenclature, Canadian residents were classified according to the American system based on the number of months of training in Anesthesia. The American designations are as follows: clinical base year (CB) = 0–1 month of anesthesia training, first clinical anesthesia year (CA-1) = 2–12 months of anesthesia training, second clinical anesthesia year (CA-2) = 13–24 months of anesthesia training, third clinical anesthesia year (CA-3) = 25–36 months of anesthesia training.
Percentile scores on the ABA-ASA ITE were grouped into five distinct score ranges and the pass rate on the RCPSC examinations for each range was calculated. To determine the relationship between the ABA-ASA ITE percentile scores and success on the RCPSC written and oral examinations, Pearson product – moment correlations were computed. This type of correlation was selected because success on the RCPSC written and oral examinations is a dichotomous variable.
Because we used a continuous score to predict a dichotomous outcome, the best cutoff score needed to be determined. Establishing optimal cutoff scores or cut points uses a technique dating from the early days of radar and sonar detection called the Receiver Operating Characteristic curve (ROC). This technique operates on the premise that as the sensitivity of the radio is increased, we pick up both the desired sound as well as static. Initially the signal increases faster than the static but at some point there is a cross-over and the static grows faster than the signal. The optimal setting is where we detect the largest ratio of signal to static.8 In this study the "signal" is the number of residents passing the RCPSC examinations; the "static" is the number of residents who fail the RCPSC examinations but scored above the cutoff on the ABA-ASA ITE. Arbitrary cut points in the scores were chosen and 2x2 tables were used to calculate their sensitivity and specificity. These pairs of sensitivities and specificities were then graphed on an ROC where the cutoff point with the best discriminating ability is determined (most signal, least static). This is indicated by the cutoff point nearest the upper left corner on the graph. This was completed for each year in which the examination was taken and for both the written and oral examinations. Likelihood ratios were calculated to determine the post-test probabilities of success given the pretest probability of success of the cohort. The probability of success of a resident in the RCPSC exams according to the year in which the ABA-ASA ITE is taken was determined.
|
Results |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
The success rates on the written component of the RCPSC examination for various percentile ranges on the ABA-ASA ITE show that scores higher than the 60th percentile are associated with a high likelihood of success in the RCPSC examinations, whereas scores less than the 20th percentile are associated with a high likelihood of failure. (Table I
) For example, all Canadian residents designated CA-1 who obtained a score above the 60th percentile on the ABA-ASA ITE passed the RCPSC written component, whereas 42% of residents scoring below the 20th percentile failed the RCPSC examination.
|
) For example, most Canadian residents designated CA-2 writing the ABA-ASA ITE who obtained a score above the 60th percentile passed the RCPSC oral component but the failure rate at lower scores was paradoxically greater with scores between the 21st and 40th percentile than with those scoring below the 20th percentile.
|
) The cutoff scores chosen indicate the ABA-ASA ITE score at which the resident has approximately a 90% chance of passing the written component of the RCPSC examination. The likelihood of failing the RCPSC examination if this score is not achieved lies between 25–67%. For example, if a resident writes the ABA-ASA ITE in his CB year and achieves a score at the 55th percentile, then he has a 94% chance of passing the RCPSC written examination. If this resident scores at the 15th percentile then his chance of failure is 67%.
|
) Cutoff scores were chosen again to approximate a 90% chance of success. The likelihood of failing the RCPSC examination if this score is not achieved is between 31–38%. For example, if a resident writes the ABA-ASA ITE in his CB year and achieves a score at the 55th percentile, then he has an 89% chance of passing the RCPSC oral examination. If this resident scores at the 15th percentile then his chance of failure is 38%.
|
|
Discussion |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
Most Canadian Anesthesia programs have utilized the ABA-ASA ITE for reasons previously mentioned. Additional benefits include the ability to compare a resident's progress with that of his/her cohort and to compare a resident's progress from year to year. Unfortunately, only the global score is provided in percentile form. As a formative tool, a profile of performance on subscales of the exam, which is currently provided only as a raw score, would be more useful in percentile form. Regardless, consistently high scores in this examination tend to reassure both the program committee and the resident that success on the RCPSC examinations is likely, while variable and poor scores may have the opposite effect. Our data indicate that useful information can be obtained from this examination.
Correlation coefficients were poor to moderate. This may be due to the comparison of percentile ranks with dichotomous scores (pass-fail), to the small number of residents in the study or to a true difference between the subject matter on the American and Canadian examinations. We were unable to obtain the actual scores of the RCPSC examination as the data were collected retrospectively and the location of these former residents was often unknown making it impossible for scores to be released.
The numbers in the study are small and the correlations were affected by a number of residents who failed the RCPSC examination while obtaining ABA-ASA ITE scores greater than the 80th percentile. For example the 92% pass rate on the RCPSC written examination for those residents obtaining an ABA-ASA ITE score greater than the 80th percentile is the failure of one out of 13 residents. Success rates on either component of the RCPSC examination ranged from 76%–92% depending upon level of training at the time of the exam.
Whether there is a true difference between the funds of knowledge of residents in Canadian training programs vs American training programs is unknown. Residents in Canadian programs take an additional year of training in general internal medicine which may influence the content of the examination. When the American Board of Pediatrics in-training examination was administered to pediatrics residents in Italy a difference in scores was seen. It was felt that the emphasis of the examination was different from that of the Italian training program.8 They did note that the Italian residents scores were different from those typically seen in American and Canadian residents implying that there was no difference between residents scores of the latter two countries.
Observation of the pass rates on the RCPSC examination indicates a high likelihood of success when percentile scores on the ASA-ABA ITE were high. Determination of the post-test probabilities of success confirm this observation. The cutoff scores were chosen based on the assumption that their position at the upper left–hand corner of the ROC curve minimized error. This is so when the pretest probability of success is close to 50%. Success rates on the RCPSC exam ranged from 76% – 92% as mentioned above. When the post-test probabilities were recalculated taking into account the corrected pretest probability, the results did not differ.
|
Conclusion |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
|
Acknowledgments |
|---|
Accepted for publication June 13, 2000.
|
References |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionConclusionReferences |
|---|
2
Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in Internal Medicine: a review. Ann Intern Med 1998; 129: 42–8.
3 Waxman H, Braunstein G, Dantzker D, et al. Performance on the Internal Medicine second-year residency In-training Examination predicts the outcome of the ABIM certifying examination. J Gen Intern Med 1994; 9: 692–4.[Medline]
4 Grossman RS, Fincher R-ME, Layne RD, Seelig CB, Berkowitz LR, Levine MA. Validity of the In-Training Examination for predicting American Board of Internal Medicine certifying examination scores. J Gen Intern Med 1992; 7: 63–7.[Medline]
5 Leigh TM, Johnson TP, Pisacano NJ. Predictive validity of the American Board of Family Practice In-Training Examination. Acad Med 1990; 65: 454–7.[Medline]
6 Shetler PL. Observations on the American Board of Surgery In-Training Examination, board results, and conference attendance. Am J Surg 1982; 144: 292–4.[Medline]
7
Webb LC, Juul D, Reynolds CF III, et al. How well does the Psychiatry residency In-Training Examination predict performance on the American Board of Psychiatry and Neurology Part I examination? Am J Psychiatr 1996; 153: 831–2.
8 Streiner DL, Norman GR. Health Measurement Scales, 2nd ed. New York: Oxford University Press Inc., 1995.
9 Butzin DW, Guerin RO, Oliver TK, Stockman JA, Da Dalt L, Perilongo G. Administering the American Board Of Pediatrics In-Training Examination in a European Pediatrics residency (Letter). Acad Med 1996; 71: 393–4.
This article has been cited by other articles:
 |
|
 |
|
  A. M.A. Boker NEW SCALE TO IMPROVE INTER-EXAMINER RELIABILITY DURING ORAL EXAM Can J Anesth, June 1, 2005; 52(suppl_1): A85 - A85. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
