| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
* From the Department of Community Health Sciences, the Faculty of Medicine, University of Calgary; and
the Office of Continuing Medical Education, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
Address correspondence to: Dr. Jocelyn Lockyer, Continuing Medical Education and Professional Development, Faculty of Medicine, University of Calgary, 3330 Hospital Drive N.W., Calgary, Alberta, T2N 4N1, Canada. Phone: 403-220-4248; Fax: 403-270-2330; E-mail: lockyer{at}ucalgary.ca
 |
Abstract |
|---|
 TOP Abstract IntroductionMethodResultsDiscussionReferences |
|---|
Methods: Surveys with 11, 19, 29 and 29 items were developed for patients, coworkers, medical colleagues and self, respectively, using five-point scales with an ‘unable to assess’ category. The items addressed communication skills, professionalism, collegiality, continuing professional development and collaboration. Each anesthesiologist was assessed by eight medical colleagues, eight coworkers, and 30 patients. Feasibility was assessed by response rates for each instrument. Validity was assessed by rating profiles, the percentage of participants unable to assess the physician for each item, and exploratory factor analyses to determine which items grouped together into scales. Cronbach’s alpha and generalizability coefficient analyses assessed reliability.
Results: One hundred and eighty-six physicians participated. The mean number and percentage return rate of respondents per physician was 17.7 (56.2%) for patients, 7.8 (95.1%) for coworkers, and 7.8 (94.6%) for medical colleagues. The mean ratings ranged from four to five for each item on each scale. There were relatively few items with high percentages of ‘unable to assess’. The factor analyses revealed a two-factor solution for the patient, a two-factor solution for the coworker and a three-factor solution for the medical colleague survey, accounting for at least 70% of the variance. All instruments had a high internal consistency reliability (Cronbach’s 
> 0.95). The generalizability coefficients were 0.65 for patients, 0.56 for coworkers and 0.69 for peers.
Conclusion: It is feasible to develop multi source feedback instruments for anesthesiologists that are valid and reliable.
|
Introduction |
|---|
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
In other disciplines of medicine, multi source feedback (MSF) or 360° evaluation is being used to assess and provide feedback to physicians about a broad range of competencies5,6 by licensing authorities,7–9 professional organizations10,11 and health care facilities.12 This type of assessment relies on questionnaires completed by patients, medical colleagues, and coworkers to provide feedback to physicians about their communication skills, interpersonal skills, collegiality, medical expertise, and the ability to learn continually and improve practice patterns.5–12 There have been a number of studies in which anesthesiologists receive feedback from surgeons13 and patients.14–16 Unlike family medicine,7 internal medicine,9–12 and surgery,8 however, questionnaires from multiple groups (e.g., patients, peers and coworkers) have not been used simultaneously to assess and provide performance data to anesthesiologists.
Studies of MSF show that reliable and valid instruments (questionnaires) can be developed.5–12 It appears feasible to develop quality improvement programs in which most physicians within a given discipline can be assessed by eight to ten coworkers and medical colleagues, and 25 patients. This number of raters produces an acceptable reliability for both the overall instrument and the physician being assessed.7,9,10–12 Furthermore, given that the intent of MSF is to guide professional development, studies show that participating physicians will use their feedback data to make changes in their practices.6,8,10,17
The practice of anesthesia is clearly different from the practice of a consultant internist, surgeon, or psychiatrist. For same-day admit and surgical day care patients who are not evaluated by an anesthesiologist in a preoperative assessment unit, patient encounters with the anesthesiologist are often brief, and may be compromised by patient stress and/or the effect of anxiolytics administered preoperatively.
The main purpose of the present study was to assess the feasibility, validity, and reliability of a MSF system for the practice of anesthesia. We sought to address the following specific questions: 1) What is the feasibility of an assessment system for practicing anesthesiologists which provides feedback from patients, coworkers, medical colleagues and self? 2) What questions about an anesthesiologist’s practice can patients, coworkers, and medical colleagues answer? 3) What are the score profiles for each of the items (i.e., mean, and standard deviation) on the surveys? 4) Do the items on a survey group together into meaningful scales to guide performance improvement direction? 5) Are the instruments reliable for both the practice of anesthesia and for the individual anesthesiologist?
|
Method |
|---|
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
A working group was recruited to develop a set of instruments for these physicians that would be similar to previously developed instruments which had been found to be reliable and valid.7–9 The committee was charged with the mandate of reviewing previous instruments but realigning and redesigning items to assess anesthesia practice. After the working group determined the items for inclusion on the instruments, copies of the instruments were sent to all of the anesthesiologists on the CPSA register for review and feedback as part of the face validation process. The working group made adjustments to the instruments based on this feedback. All previous instruments had included patient surveys and there was lengthy discussion about the advantages and disadvantages of maintaining a patient instrument for anesthesiologists. A small pilot project involving four anesthesiologists indicated that patient responses could be obtained, but would be challenging. In the end, the committee retained an abbreviated patient survey along with questionnaires for medical colleagues, coworkers and self.18
The final instrument for patients consisted of 11 items (Table I
). Assessors were asked to use a five-point rating scale (1 = strongly disagree to 5 = strongly agree). The instrument for medical colleagues (Table II) and co-workers (Table III) consisted of 19 and 29 items, respectively, with a five-point rating scale (1 = among the worst to 5 = among the best). The self-assessment instrument (Table IV) was identical to the medical colleague instrument except that all items were written in the first person. All questionnaires provided respondents with the option of being able to indicate they were unable to assess the physician on the item.
|
|
A number of statistical analyses were undertaken to address the research questions posed. Response rates were used to determine feasibility for each of the respondent groups (Question 1). For each item on each survey, the percentage of ‘unable to assess’ along with the mean and standard deviation was computed in order to determine the viability of items and the score profiles (Questions 2 and 3, respectively). When the percentage of ‘unable to assess’ items exceeds 20% on a survey, it may suggest a need to examine the item for revision or deletion. We used exploratory factor analysis to determine which items on each survey belonged together and became a ‘factor’ (or scale) such as communication (Question 4). This analysis allowed us to identify the factors and the number of factors for each survey and to describe the relative variance accounted for by each factor within the whole instrument. These factors or scales could then be used within the feedback report to guide overall direction for improvement. Individual items within the factor would provide more precise data about specific behaviors which the physician might change. As the self data is reported in feedback reports to the physician along-side the medical colleague data, a factor analysis of the self assessment questionnaire was not done. Finally, reliability was assessed (Question 5). Internal consistency reliability was examined using the Cronbach’s coefficient for each of the rater groups and for each of the scales/factors for each rater group. This would tell us whether the instruments have overall stability. This analysis was followed by a generalizability analysis to determine the Ep2 to ensure there were sufficient numbers of items and raters to provide stable data for each individual anesthesiologist on each instrument. Normally, an Ep2 of 0.70 suggests data are stable.1,5,8 Too low an Ep2 suggests the need for modifications to the measurement procedure (i.e., more raters or more items).
The study received approval from the Conjoint Health Research Ethics Board of the University of Calgary. The University researchers received anonymous data to conduct the psychometric assessment. The data did not contain any physician characteristics (e.g., location of practice, year or school of graduation) that could be associated with any practitioner.
|
Results |
|---|
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
The majority of items on the questionnaires could be answered by respondents. As presented in Tables I
to IV (Tables III and IV available as Additional Material at www.cja-jc.org) the assessment of ‘unable to assess’ items showed that two items (of 11) on the patient survey, one on the coworker (of 19), and five (of 29) on the medical colleague survey had unable to respond rates > 15%. The mean ratings for all items on the patient, medical colleague and peer surveys were between 4 and 5.
The factor analysis identified two factors on the patient survey that accounted for 77.6% of the variance, professionalism and communication. The factor analysis identified two factors (communication and collaboration) on the coworker instrument as accounting for 67.5% of the variance. The medical colleague assessment identified three factors accounted for 74.5% of the variance, clinical performance, communication and professionalism, and continuing professional development.
A Cronbach’s was calculated to determine the internal reliability of the instruments. Patient surveys had an alpha of 0.93, coworker surveys of 0.95, medical colleagues of 0.97 and the self assessment was 0.97. The generalizability coefficients (Ep2) were 0.65, 0.56, and 0.69 for patient, coworker, and the medical colleague surveys, respectively.
|
Discussion |
|---|
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
The PAR program is mandatory and the response rates were high as expected (except for the patient surveys). As such, these rates are consistent with the response rates for other groups of physicians who have been studied.7–9 As noted, the most challenging component was the patient survey. Unlike other groups7,8 in which 25 patient surveys were given to the physician to be distributed in the office and virtually all are collected, the mean response rate was < 60% and each anesthesiologist was provided with a minimum of 30 surveys. Despite this, we believe these data show that it is feasible to design a MSF program for anesthesia practice which includes a medical colleague, coworker, patient, and self components. The patient component, however, does need to be regularly reviewed for feasibility.
The majority of the items could be answered by the anesthesiologist’s assessors. There were some items which proved difficult for respondents to assess. In some cases the items which were less likely to be answered may have simple explanations which relate to the opportunity to observe the behavior assessed by that item. For example, three of these items on the medical colleague questionnaire were within the area of professional development and participation in quality improvement activities. For anesthesiologists who selected surgeons as respondents, this information may not have been known. On the patient survey, items related to decision making and anesthetic options may not have been recalled subsequently when patients were responding.
The range and the mean ratings were similar to that of other groups with most physicians receiving all of their ratings between 4 and 5.7–9 While these scores are high, they are consistent with the range of scores found in most assessments of residents and medical students as well as practicing physicians. Similarly, the self ratings were lower than those provided by medical colleagues, a finding which is consistent with other studies of this nature.7,8
The factor analysis was helpful in identifying the factors (scales) for each of the surveys. Our analysis found two factors for the patient and the coworker surveys and three for the medical colleague survey. These factors of professionalism, communication, collaboration, clinical performance, and continuing professional development allowed us to confirm that our tools had assessed the key domains which the CPSA wanted to examine. It also showed that our scales were similar to those from other PAR instruments,6–9 thus maintaining the integrity of the PAR assessment. The creation of scales through factor analysis also offers the advantage that the scales can be used to guide physicians in global areas (e.g., communication) as well as on an item by item basis (e.g., ‘talked with me about anesthetic options’). Participants received feedback about their own performance as well as data for the entire group of anesthesiologists who participated in the study.
The reliability analysis indicates that overall, the instruments are stable. The Cronbach’s was high and the generalizability coefficient, while not as robust as found in our previous work,7 was comparable to that found with the American Board of Internal Medicine work.10 These data suggest that the mix of items and number of raters on the surveys is appropriate and that practitioners can be confident that their feedback is reliable (stable).
Overall, we believe that it is possible to develop high quality MSF instruments for the practice of anesthesia. While this method cannot substitute for direct observation of anesthesiologists2,3 or the opportunities to assess clinical performance, teamwork and collaboration provided by simulation methods2 or profiling techniques,4 these instruments provide unique information about communication skills and professionalism. Moreover, it is a relatively inexpensive method of providing feedback about communication skills, professionalism, collaboration and continuing professional development interest from those who have first hand experience with the clinician, and observe these behaviours directly.
|
Acknowledgments |
|---|
|
Footnotes |
|---|
Accepted for publication July 19, 2005. Revision accepted August 26, 2005.
|
References |
|---|
|
TOPAbstractIntroductionMethodResultsDiscussionReferences |
|---|
2 Fletcher GC, McGeorge P, Flin RH, Glavin RJ, Maran NJ. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth 2002; 88: 418–29.
3 Slagle J, Weinger MB, Dinh MT, Brumer VV, Williams K. Assessment of the intrarater and interrater reliability of an established task analysis methodology. Anesthesiology 2002; 96: 1129–39.[Medline]
4 St Jacques PJ, Patel N, Higgins MS. Improving anaesthesiologist performance through profiling and incentives. J Clin Anesthesiol 2004, 16: 523–8.[Medline]
5 Evans R, Elwyn G, Edwards A. Review of instruments for peer assessment of physicians. BMJ 2004, 328: 1240.
6 Lockyer J. Multisource feedback in the assessment of physician competencies. J Contin Educ Health Prof 2003; 23: 4–12.[Medline]
7 Hall W, Violato C, Lewkonia R, et al. Assessment of physician performance in Alberta: The physician achievement review. CMAJ 1999; 161: 52–7.
8 Violato C, Lockyer J, Fidler H. Multisource feedback: a method of assessing surgical practice. BMJ 2003; 326: 546–8.
9 Lockyer J, Violato C. An examination of the appropriateness of using a common peer assessment instrument to assess physician skills across specialties, Acad Med 2004; 79(10 Suppl.): S5–8.[Medline]
10 Lipner RS, Blank LL, Leas BF, Fortna GS. The value of patient and peer ratings in recertification. Acad Med 2002; 77(10 Suppl.): S64–6.[Medline]
11 Ramsey PG, Wenrich MD, Carline JD, Inui TS, Larson EB, LoGerfo JP. Use of peer ratings to evaluate physician performance. JAMA 1993; 269: 1655–60.[Abstract]
12 Ramsey PG, Carline JD, Blank LL, Wenrich MD. Feasibility of hospital-based use of peer ratings to evaluate the performances of practicing physicians. Acad Med 1996; 71: 364–70.[Medline]
13 Le May S, Dupuis G, Harel F, Taillefer MC, Dube S, Hardy JF. Clinimetric scale to measure surgeons’ satisfaction with anesthesia services. Can J Anesth 2000; 47: 398–405.
14 Carnie J. Patient feedback on the anaesthetist’s performance during the pre-operative visit. Anaesthesia 2002; 57: 697–701.[Medline]
15 Thoms GM, McHugh GA, Lack JA. What information do anaesthetists provide for patients. Br J Anesth 2002; 89: 917–9.
16 Dexter F, Aker J, Wright WA. Development of a measure of patient satisfaction with monitored anesthesia care. The Iowa Satisfaction with Anesthesia Scale. Anesthesiology 1997; 87: 865–73.[Medline]
17 Fidler H, Lockyer JM, Toews J, Violato C. Changing physicians’ practices: the effect of individual feedback. Acad Med 1999; 74: 702–14.[Medline]
18 College of Physicians and Surgeons of Alberta, Physician Achievement Program. Available from URL; http://www.par-program.org/ (accessed July 21, 2005).
This article has been cited by other articles:
 |
|
 |
|
  J L Campbell, S H Richards, A Dickens, M Greco, A Narayanan, and S Brearley Assessing the professional performance of UK doctors: an evaluation of the utility of the General Medical Council patient and colleague questionnaires Qual. Saf. Health Care, June 1, 2008; 17(3): 187 - 193. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
