CJA
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
This Article
Right arrow Abstract Freely available
Right arrow Résumé de cet Article
Right arrow Full Text (PDF)
Right arrow Submit a scholarly reply
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by VanDenKerkhof, E. G.
Right arrow Articles by Milne, B.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by VanDenKerkhof, E. G.
Right arrow Articles by Milne, B.
Canadian Journal of Anesthesia 51:449-454 (2004)
© Canadian Anesthesiologists' Society, 2004
General Anesthesia

In Canada, anesthesiologists are less likely to respond to an electronic, compared to a paper questionnaire

[Au Canada, les anesthésiologistes sont moins enclins à répondre à un questionnaire électronique qu’à un questionnaire sur papier]

Elizabeth G. VanDenKerkhof, DRPH, Joel L Parlow, MD MSc, David H. Goldstein, MB BCH MSc and Brian Milne, MD MSc

From the Department of Anesthesiology, Queen’s University, Kingston General Hospital, Kingston, Ontario, Canada.

Address correspondence to: Dr. Elizabeth VanDenKerkhof, Department of Anesthesiology, Queen’s University, Kingston General Hospital, 76 Stuart Street, Kingston, Ontario K7L 2V7, Canada. Phone: 613-549-6666, ext. 3964; Fax: 613-548-1375; E-mail: ev5{at}post.queensu.ca


   Abstract
TOP
 Abstract
Introduction
 Methods
 Results
 Discussion
 References
 
Purpose: A randomized unblinded controlled trial was used to assess the utility of electronic questionnaires in a survey of Canadian anesthesiologists.

Methods: Postal or electronic questionnaires were sent between November 2001 and March 2002 to 1,333 anesthesiologists registered with the Canadian Anesthesiologists’ Society. The primary outcome measure was the difference in response rates between electronic and postal questionnaires. Secondary outcome measures included a comparison of demographic characteristics, cost, and knowledge and practice regarding prophylactic perioperative beta blockade.

Results: The overall response rate was 52%. E-mail participants were half as likely as postal participants to respond to the questionnaire (35% vs 69%, relative risk = 0.51, 95% confidence interval 0.45–0.58). Respondents who provided an e-mail address were younger and more likely to be affiliated with an academic institution. There were no significant differences in responses to knowledge and practice questions. The electronic arm was faster than the postal arm and the cost per reply was one-third the cost of the postal arm ($2.50 vs $8.02).

Conclusions: Electronic surveys are a means of acquiring information from a large number of individuals in a rapid, efficient and cost-effective manner. This methodology may be particularly valid and useful in surveys of participants with similar backgrounds and internet access. However the lower response rates achieved as compared with postal surveys indicates a need to use vigilance when generalizing results to a broader population.


   Introduction
TOP
 Abstract
 Introduction
Methods
 Results
 Discussion
 References
 
THE electronic questionnaire can be an efficient, inexpensive means of doing survey research, but a review of the literature indicates that response rates are poor1–4 (21%–34%). Electronic surveys yielding better response rates tend to be targeted at convenience or self-selected samples rather than random samples of the population being studied.5 Selection bias is likely to exist not only due to the inadequacy of such sample designs, but also due to the "digital divide," which is thought to have developed along age, racial groups, income levels, and geographic settings (http://www.ntia.doc.gov/ntiahome/net2/falling.html). Therefore, a frequently cited disadvantage of electronic surveys is representativeness, yet few studies have systematically assessed the representativeness of respondents to electronic surveys.6 The purpose of this study was to assess the utility of electronic questionnaires in a survey of Canadian anesthesiologists. The primary objective was to assess the difference in response rate between electronic and postal questionnaires. The secondary objective was to compare demographic characteristics between Canadian Anesthesiologists’ Society (CAS) members with and without an e-mail address registered with the CAS, and to compare knowledge and practice regarding prophylactic perioperative beta blockade and cost between the two survey methods. To our knowledge this is the first randomized trial in medicine that assesses, in a systematic fashion, the outcomes of electronic vs paper questionnaires with respect to response rate and internal validity.


   Methods
TOP
 Abstract
 Introduction
 Methods
Results
 Discussion
 References
 
Following approval from the University’s Research Ethics Board, and permission from the CAS, the CAS membership list of specialist trained anesthesiologists practicing both within Canada and abroad was used as the sampling frame (n = 1634). Sample size calculations were carried out for the purpose of the original study relating to anesthesiologists’ knowledge and practices regarding prophylactic perioperative beta blockade7 (post hoc power calculation provided in the results). As a subcomponent of the original study, and for the purpose of this article, anesthesiologists were stratified into those who provided the CAS with an e-mail address (n = 1033) and those who did not (n = 551). The physician data provided by the CAS were in a Microsoft Excel spreadsheet. The data were divided into two files based on whether or not an e-mail address was present. A column of random numbers was generated in each database for the purpose of random sampling. The columns were sorted in ascending order, thereby generating a randomized list. The first 300 anesthesiologists in the database of those who did not provide an e-mail address were sent a postal questionnaire. An additional random sample of 300 anesthesiologists who provided an e-mail address were also sent postal questionnaires. This latter group was selected to assess for potential differences in postal respondents with and without an e-mail address registered with the CAS. The remaining anesthesiologists with an e-mail address registered with the CAS were sent an e-mail containing a cover letter identical to the postal cover letter, but with instructions on how to complete the electronic questionnaire and a link to the web page containing an identical electronic version of the questionnaire (n = 733).

The questionnaire comprised 21 closed-ended questions dealing with knowledge, belief and practices regarding prophylactic perioperative beta blocker administration in patients with risk factors or known coronary artery disease. Demographic characteristics, practice characteristics, and information regarding internet and e-mail use were also included in the questionnaire. A copy of the questionnaire can be found elsewhere.7 Response categories in the electronic questionnaire were provided in drop-down lists if only one response was required, and as check boxes if more than one response could be selected. The web page was created using active server pages (®Microsoft Inc., Richmond, VA, USA) as the development tool. When the participant completed the survey and hit the submit key, the questionnaire responses were processed into a Microsoft Access (®Microsoft Inc., Richmond, VA, USA) database. The structure was such that all processing was done by the web server. Both versions of the questionnaire took approximately five minutes to complete.

Both questionnaire formats were sent out on the same day. Non-responders were followed using a modified version of the Dillman technique.8 The electronic group was sent a follow-up e-mail with a link to the questionnaire at approximately one week, two weeks and three weeks after the initial mailing (Table IGo). Non-respondents to the postal questionnaire were sent a follow-up postcard at four and 12 weeks, and another copy of the questionnaire at 18 weeks. Data analysis included frequencies, percents, Chi square and relative risk (RR) to assess for differences in demographic and computer use characteristics between respondents who provided an e-mail address to the CAS and those who did not. Frequencies and percents were used to assess for differences in response rate between postal and electronic questionnaires. Cost analysis included total cost, cost per returned questionnaire and theoretical cost per returned questionnaire assuming a 100% response rate. Electronic responses were merged with the postal in a Microsoft Excel (®Microsoft Inc., Richmond, VA, USA) spreadsheet. SPSS (®SPSS, Chicago, IL, USA) was used to calculate frequencies, percents, Chi squares, and P values. EpiInfo was used to calculate RR and 95% confidence intervals (CI). RR was used to assess for the likelihood of responding between the two groups.


View this table:
[in this window]
[in a new window]
  		TABLE I Response rate for electronic and postal questionnaires
 

   Results
TOP
 Abstract
 Introduction
 Methods
 Results
Discussion
 References
 
The first mailing took place on November 2, 2001. The last follow-up on the electronic arm was sent November 28, 2001 and the website was shut down December 20, 2001. The last follow-up on the postal arm was sent March 2, 2002. One hundred and sixty-five anesthesiologists on the electronic arm were excluded after the initial mail out, due to invalid electronic addresses. Table IGo outlines the response rates obtained in the three arms of the study. The overall response rate for the study was 52%. Participants in the e-mail group were half as likely as participants in the postal arm to respond to the survey (RR = 0.51, 95% CI 0.45–0.58). The response rate for postal participants who provided an e-mail address was 72% compared to 65% for the postal participants who did not provide an e-mail address. The results of a post hoc power calculation indicate that a sample size of 1,168, a criterion of significance (alpha) of 0.05, and a two-tailed test, result in 100% power to detect a difference in response rate of this magnitude (0.69 vs 0.35). Using a power of 90% and a sample size of 1,168, the smallest significant detectable effect would be a 9% difference in response rates between the two methods.

The distribution of responses over time can be found in Table IIGo. The first questionnaire was returned within an hour of the electronic mail out. The overall response rate was lower at each mailing in the electronic arm compared to the postal arm. With respect to returned questionnaires, 53% (n = 106) of the 200 returned electronic questionnaires arrived before the first follow-up reminder (one week). Of the postal questionnaires that were returned 36% (n = 150) arrived before the first follow-up reminder (one month).


View this table:
[in this window]
[in a new window]
  		TABLE II Distribution of response rates by questionnaire follow-up period
 
Table IIIGo presents demographic characteristics of study participants, stratified by whether or not they provided the CAS with an e-mail address. The majority of the participants were male (76%) and over 40 yr of age (77%). Sixty-three percent were affiliated with an academic institution. Eighty percent reported using both e-mail and the internet frequently or daily. The likelihood of providing an e-mail address to the CAS decreased significantly with increasing age ({chi}2 for trend = 4.06, P = 0.04). Anesthesiologists who provided an e-mail address were more likely to be affiliated with an academic institution than anesthesiologists who provided a postal address only (88% vs 51%) and were also more likely to use the e-mail (92% vs 54%) or internet (90% vs 58%) frequently or daily ({chi}2 = 115, P = 0.00 and {chi}2 = 84, P = 0.00).


View this table:
[in this window]
[in a new window]
  		TABLE III Demographic characteristics of survey respondents stratified by whether they provided the CAS with a postal address only or a postal and an e-mail address
 
Table IVGo presents a selection of questions participants were asked to respond to with respect to knowledge and practice regarding prophylactic perioperative beta blockade.7 No statistically significant differences were found with respect to knowledge and practice between electronic and postal questionnaires. Similar findings were identified when only postal questionnaires were stratified by whether or not the participant provided an e-mail address to the CAS (data not shown). The cost analysis of the survey is presented in Table VGo.


View this table:
[in this window]
[in a new window]
  		TABLE IV Comparison of electronic and postal responses to questions related to the use of beta blockers in the perioperative period7*
 

View this table:
[in this window]
[in a new window]
  		TABLE V Cost analysis for the electronic vs the postal survey
 

   Discussion
TOP
 Abstract
 Introduction
 Methods
 Results
 Discussion
References
 
Compared to postal questionnaires, electronic questionnaires were found to be a rapid, efficient and relatively inexpensive means of conducting a survey to assess the knowledge and practices of anesthesiologists. Response rates in the postal arm were approximately double that of the electronic arm, yet responses to questions about knowledge and practice patterns were not significantly different between the two groups. The cost of the electronic arm was one-third the cost of the postal arm. Participants who provided an e-mail address to the CAS tended to be younger and to be affiliated with an academic institution. In spite of differences in demographic characteristics, responses to knowledge and practice based questions did not differ between the postal and the electronic questionnaires. However, we are reluctant to imply that this finding can be generalized to all surveys. In order to assess the validity of electronic questionnaires, future electronic surveys should consider including a postal component for data validation purposes.

Some of the weaknesses of this study relate to potential generalizabilty and include low response rate, exclusion of anesthesiologists who are not members of the CAS and lack of information about non-respondents. A comparison of demographic characteristics with a recent census of Canadian anesthesiologists found a similar male to female ratio and age structure, but there was greater representation from academic centres in our study compared to the census.9

Strengths of this study include the systematic approach to comparing electronic and postal questionnaires, the use of random sampling to select potential study participants, the random allocation of e-mail and postal questionnaires, and the consistent and comparable follow-up process. All of these aspects allow for comparison between the two arms while minimizing bias. Few studies in the literature have compared these two methods in this way and as a result suffer to an even greater degree from selection bias than does our study.

In spite of the rapid evolution and uptake of electronic communication, the response rate of 35% in the electronic group remains consistent with the literature.3,4,10,11 Response rates for electronic surveys increase if incentives are provided,12 or self selection or convenience samples rather than random samples are used to identify subjects,5 however these methods result in even greater selection bias. Response rates for electronic surveys also appear to be on the decline since they were first introduced in the mid-1980’s.6 In spite of the low and different response rates and the differences in demographic characteristics between electronic and postal questionnaires, knowledge and practice responses were similar. Hence the demographic characteristics of electronic survey participants may not be representative of all potential respondents, but the response to knowledge and practice questions may be reflective of the population. This finding is supported in the literature.13

Major benefits of electronic surveys are reduced costs, the immediate availability of responses, and the ability to have data downloaded directly into a database for the purpose of analyses, thereby reducing delay, manpower resources and data entry errors. However, a number of lessons were learned which may be of benefit in future electronic surveys: i) it is important to test the validity of e-mail addresses before calculating sample size. One hundred and sixty-five (23%) e-mail addresses proved to be invalid, but 109 (66%) of these were due to insolvency declared by a popular internet provider; ii) provide drop down lists with "select one" as the default option rather than an actual response, as the latter could create bias; iii) even if only one response is required, participants must be given the opportunity to select more than one response or to provide a comment in a text box; iv) provide a progression bar so participants can determine how far along they are in the questionnaire; v) presentation of one question per screen would simplify the visual field and make the interface more user-friendly; and vi) as unsolicited e-mail becomes more common, e-mail surveys may erroneously be considered as "junk mail"11 and deleted, therefore an initial contact is important to the success of electronic surveys. A postcard sent via regular post, indicating that an electronic questionnaire will follow might increase response rates but will also increase the cost of electronic surveys.

In summary, electronic surveys are a means of acquiring information from a large number of individuals in an efficient and cost-effective manner. This methodology may be particularly valid and useful in multinational surveys of individuals with similar backgrounds and internet access. The ease and speed at which electronic surveys can be administered make them particularly useful for pilot studies. However the lower response rates achieved as compared with postal surveys indicates a need to use vigilance when generalizing results to a broader population.


   Acknowledgments
 
We would like to acknowledge the assistance of the Canadian Anesthesiologists’ Society and its members for participating in this study. We would also like to thank Nicole Avery for her assistance in data collection, Jeff Babcock and Mike Rimmer of the Queen’s University Anesthesiology Informatics Laboratory (QUAIL) for their assistance with web and database management. Funding was provided through a Queen’s University Advisory Research Committee grant and Dr. VanDenKerkhof’s Queen’s University Research Initiation Grant.


   Footnotes
 
Accepted for publication November 24, 2003. Revision accepted February 20, 2004.


   References
TOP
 Abstract
 Introduction
 Methods
 Results
 Discussion
 References
 
1 Singarella T, Baxter J, Sandefur RR, Emery CC. The effects of electronic mail on communication in two health sciences institutions. J Med Syst 1993; 17: 69–86.[Medline]

2 Sands DZ, Safran C, Slack WV, Bleich HL. Use of electronic mail in a teaching hospital. Proc Annu Symp Comput Appl Med Care 1993; 306–10.

3 Chang HY, Sharma VK, Howden CW, Gold BD. Knowledge, attitudes, and practice styles of North American pediatric gastroenterologists: helicobacter pylori infection. J Pediatr Gastroenterol Nutr 2003; 36: 235–40.[Medline]

4 Eigenmann PA, Zamora SA. An internet-based survey on the circumstances of food-induced reactions following the diagnosis of IgE-mediated food allergy. Allergy 2002; 57: 449–53.[Medline]

5 Yeaworth RC. Use of the internet in survey research. J Profes Nurs 2001; 17: 187–93.

6 Mavis BE, Brocato JJ. Postal surveys versus electronic mail surveys. The tortoise and the hare revisited. Eval Health Prof 1998; 21: 395–408.[Abstract/Free Full Text]

7 VanDenKerkhof EG, Milne B, Parlow JL. Knowledge and practice regarding prophylactic perioperative beta blockade in patients undergoing noncardiac surgery: a survey of Canadian anesthesiologists. Anesth Analg 2003; 96: 1558–65.[Abstract/Free Full Text]

8 Dillman DA. Mail and Internet Surveys. The Tailored Design Method, 2nd ed. New York: John Wiley and Sons Inc.; 2000.

9 Ghazar N, Morewood GH, Engen D, Ashbury T, VanDenKerkhof EG, Wang L. Gender differences in the Canadian Anesthesia Workforce. Can J Anesth 2003; 50: 109A (abstract).[Free Full Text]

10 Fyfe S, Leonard H, Gelmi R, Tassell A, Strack R. Using the internet to pilot a questionnaire on childhood disability in Rett syndrome. Child Care Health Dev 2001; 27: 535–43.[Medline]

11 Jones R, Pitt N. Health surveys in the workplace: comparison of postal, email and World Wide Web methods. Occup Med (Oxford) 1999; 49: 556–8.

12 Pealer LN, Weiler RM, Pigg RM Jr, Miller D, Dorman SM. The feasibility of a web-based surveillance system to collect health risk behavior data from college students. Health Educ Behav 2001; 28: 547–59.[Abstract/Free Full Text]

13 Pettit FA. A Comparison of World-Wide Web and paper-and-pencil personality questionnaires. Behav Res Methods Instrum Comput 2002; 34: 50–4.[Medline]




This article has been cited by other articles:


Home page
 Field MethodsHome page
T.-H. Shih and Xitao Fan
Comparing Response Rates from Web and Mail Surveys: A Meta-Analysis
Field Methods, August 1, 2008; 20(3): 249 - 271.
[Abstract] [PDF]

Home page
 Eval Health ProfHome page
L. A. Crane, M. F. Daley, J. Barrow, C. Babbel, B. L. Beaty, J. F. Steiner, A. Kempe, L. Miriam Dickinson, and S. Stokley
Sentinel Physician Networks as a Technique for Rapid Immunization Policy Surveys
Eval Health Prof, March 1, 2008; 31(1): 43 - 64.
[Abstract] [PDF]

Home page
 Eval Health ProfHome page
J. B. VanGeest, T. P. Johnson, and V. L. Welch
Methodologies for Improving Response Rates in Surveys of Physicians: A Systematic Review
Eval Health Prof, December 1, 2007; 30(4): 303 - 321.
[Abstract] [PDF]

Home page
 Anesth. Analg.Home page
P. L. Bailey, L. G. Glance, M. P. Eaton, B. Parshall, and S. McIntosh
A Survey of the Use of Ultrasound During Central Venous Catheterization
Anesth. Analg., March 1, 2007; 104(3): 491 - 497.
[Abstract] [Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Résumé de cet Article
Right arrow Full Text (PDF)
Right arrow Submit a scholarly reply
Right arrow Alert me when this article is cited
Right arrow Alert me when eLetters are posted
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Right arrow reprints & permissions
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by VanDenKerkhof, E. G.
Right arrow Articles by Milne, B.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by VanDenKerkhof, E. G.
Right arrow Articles by Milne, B.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS