| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
From the Department of Anesthesia and Resuscitation, Maisonneuve-Rosemont Hospital, University of Montreal, Montreal, Quebec, Canada.
Address correspondence to: Dr. Pierre Drolet, Département d’Anesthésie-Réanimation, Hôpital Maisonneuve-Rosemont, 5415 Boul. de l’Assomption, Montréal, Québec H1T 2M4, Canada. E-mail: pdrolet{at}aei.ca
 |
Abstract |
|---|
 TOP Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
Methods: Peer review material was analyzed from 213 submissions to the CJA. Characteristics of accepted and rejected manuscripts were compared. Reviewers’ comments were classified according to editorial criteria used by the journal and the distribution of the different types of comments amongst accepted and rejected submissions was compared.
Results: Characteristics of 213 manuscripts and comments from 405 reviewers were analyzed. Overall, 57% of manuscripts submitted to the CJA were accepted. The type of research (study vs case report, clinical vs laboratory science) had no impact on the fate of the manuscripts; however, frequency of acceptance differed between articles originating from different geographic regions (P < 0.0001) with Canadian submissions posting the highest frequency (86%). Comment analysis suggests that the relationship between the experimental design, the results, and the conclusion was the main determinant of an article’s fate. Lack of originality or inappropriate experimental design were likely to be associated with rejection. Conversely, aspects involving the presentation of manuscripts (tables, figures, references) were rarely cited as reasons to justify acceptance or rejection.
Conclusion: Although articles are judged on many criteria, authors need to be aware that some aspects of a manuscript, namely the relationship between experimental design, results, and conclusions, the originality, and the use of an appropriate study design, are the most important features with regard to its acceptance or rejection.
|
Introduction |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The main goals of this study were to identify the characteristics of the manuscripts submitted to the Canadian Journal of Anesthesia (CJA) that were associated with their acceptance or rejection and to classify all of the peer reviewers’ comments in order to analyze their impact on the decision to publish. This article should shed light on how reviewers use the different forms provided by the editor-in-chief to facilitate the peer review process and give authors and readers an in-depth look at some of the steps preceding publication.
|
Methods |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
The manuscripts included in this study were classified according to their geographical origin (Canada, United States, Europe, Asia, or other), their design (observational or experimental study vs case report), and their emphasis (clinical vs laboratory). The association between these characteristics and the fate of the manuscript was evaluated.
It is the policy of the editorial board to send manuscripts submitted to the CJA to two peer reviewers for analysis. Whether they are members of the editorial board or guests, peer reviewers are chosen for their knowledge and expertise with regard to the subject of the article. Reviewers receive unblinded copies of the article and they advise the editor on acceptance or rejection. Reviewers are asked to proceed using three forms:
); ); and
|
|
. When particular comments for authors could not be assigned to one of the 13 criteria, they were noted as unclassified. Unclassified comments were also analyzed and assigned, if possible, to new categories. Because of their small number, comments for the editor were not divided according to the 13 criteria suggested in the review process. The classification of the comments was accomplished primarily by the first author (C.T.). Although the assignment was, by nature, a subjective process, less than 20 comments proved difficult to assign. Those few comments were classified following discussion and agreement between the investigators. The characteristics of the peer review material were compared according to the fate of the manuscripts (acceptance or rejection). Data from the checklist and characteristics such as geographical origin and type of submissions (studies vs case reports; clinical vs laboratory) were analyzed with the Chi-square test. The number of comments for editor and for authors, the latter being divided according to the 13 criteria suggested in the review process, was analyzed with Fisher’s exact test or the Chi-square test for trends with Bonferroni’s correction for multiple comparisons. The Spearman correlation coefficient was used to measure the strength of the association between formal recommendations submitted by both reviewers regarding acceptance or rejection. To do so, each of the four possible recommendations ("accept as is," "accept following appropriate revision," "possibly accept following revision," and "reject") was treated as an item on an ordinal scale. A P value < 0.05 was considered significant.
|
Results |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
). A significant difference was also noted in the distribution of accepted and rejected manuscripts by geographical origin of the submissions (Table III). The difference stemmed from a high proportion of accepted manuscripts from Canada (86%) and a lower proportion from Asia (41%). The frequency of acceptance of case reports (32/60; 53%) did not differ significantly from that of studies (89/153; 58%; P = 0.45). Similarly, the frequency of acceptance of laboratory reports (12/30; 40%) was not significantly different from that of clinical matters (109/183; 59%; P = 0.12).
|
|
shows the mean number of comments for authors divided into the 13 criteria suggested for the review of manuscripts. Reviews for accepted and rejected manuscripts contained an average of 1.4 and 0.9 comments respectively that were not classifiable into the 13 criteria. Most of these comments were of a general nature (e.g., "enjoyed reading," "worth nothing"). The only new criterion that could be extrapolated from these unclassified comments pertained to the clinical relevancy of the submitted manuscripts. As a criterion, clinical relevancy was the object of 0.1 comments amongst reviews for accepted and rejected manuscripts. Table V identifies criteria for which there were differences between accepted and rejected submissions with regard to the number of comments for authors.
|
|
The correlation between the formal recommendations to accept or reject a manuscript provided by the two reviewers (last item of the checklist) was low (r = 0.17; 95% confidence interval –0.02 to 0.29).
|
Discussion |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
) and suggests that the checklist is discriminative and useful. Thus, it is important for authors to write manuscripts that can comply with the criteria of the checklist. However, the poor correlation between reviewers regarding the recommendation to publish or reject highlights the difficulty for an editor to rely on a single instrument. Others have noted the lack of agreement between peer reviewers.8 Hence, other tools are necessary. To be useful, these tools must allow the reviewers to form and communicate opinions toward the manuscript that are both clear and systematically presented. Even divergent opinions, if they are clearly stated and argued, will help the editor decide what to do with a manuscript.9 A few months after data were collected for this study, the CJA started using an expanded checklist with more items. In addition to the items studied in this article, reviewers are now asked to comment on the hypothesis and objectives stated in the submitted manuscript and to suggest if a statistical or an ethics review is required.
The "comments for authors" section was the most elaborate part of the peer review documents. Each peer review analyzed for this study contained a mean of nine comments for authors. Peer reviewers were asked to comment on the 13 criteria proposed by the editorial board (Table I). The 3,722 comments for authors found in the 405 peer reviews were classified according to these criteria by the investigators of this study (mostly by C.T.). Since comments were written in prose, the classification was subjective by nature, entailed decisions based on judgment, and is one of the limitations of this study since we cannot exclude that other researchers might have opted to assign some comments to a different category. Whenever a comment could belong to more than one category, it was assigned to the category best dictated by the context in which it appeared. Only two criteria, "methods used are adequately described and appropriate?" and "discussion appropriate and complete?," were the object of more than one comment, on average, in all reviews, whether the manuscripts were accepted or not. Three criterias, "conclusions appropriate in relation to experimental design?," "results clearly described? Tables, figures appropriate?" and the unclassified comments, received more than one comment from peer reviewers, but only for accepted manuscripts. In contrast, two criteria ("any concern about ethics?," "CJA appropriate for this manuscript?") were the subject of less than 0.1 comments, on average.
If Table IV shows the criteria that were addressed most often by peer reviewers, Table V identifies the most discriminating ones with regard to acceptance or rejection. "Conclusions appropriate in relation to experimental design and results?" was the only criterion showing differences regarding the number of comments, both positive and negative, between accepted and rejected manuscripts. Surprisingly, more negative comments were seen amongst accepted articles than amongst rejected ones. Other criteria showed differences in the number of comments between accepted and rejected manuscripts, but only with regard to either positive (unclassified comments) or negative comments ("information original, in total or in part?" and "experimental design appropriate?"). Analysis of these results suggests that the relationship between the study design, the results, and the conclusions may be of greatest interest to peer reviewers when they are commenting on a manuscript and deciding its fate. On the other hand, some aspects of a manuscript’s presentation (tables and figures) were unlikely to play a major role in the decision to accept or reject a submission, although they were the object of many comments. Table V also shows that a majority of peer reviewers chose to not address systematically all the specific criteria proposed by the editorial board. Whether the review process could benefit from eliminating or modifying some of the less discriminating or less addressed criteria is beyond the scope of this study.
References have been found to be notoriously inaccurate in many published articles.10,11 This issue has been addressed by some journals, including the CJA, which requests authors to send the title page of all references with the final version of accepted manuscripts. Neither were references a popular topic amongst reviewers nor did they impact significantly on the fate of a manuscript.
Clinical relevancy, the only criterion that could be extrapolated from the unclassified comments, was also unlikely to be singled out as a cause for acceptance or rejection although other investigators of medical publication have emphasized the importance for a study to be clinically relevant.12 Conversely, the originality of the material submitted and the quality of the experimental design, even if they were not particularly praised in accepted manuscripts, were identified as reasons for recommending rejection by reviewers when deemed faulty.
It was customary for reviewers to comment negatively on the relationship between conclusions, study design, and results. Surprisingly, these negative comments seemed to reflect favourably on the decision to publish. This suggests that, although many authors had the tendency to overinterpret their results, this did not seem to be a major flaw.4 Instead, our analysis suggests that when reviewers took time to comment on what could be construed as an excess of enthusiasm in interpreting results, the article probably deserved to be published after due revision. Others have suggested that even negative comments can contribute to the improvement of a submission, if they are formulated in a constructive manner.13
The "comments for editor" section was used sparingly by peer reviewers. On average, it contained only two comments (positive or negative). Still, it was quite discriminating since both types of comments, positive and negative, showed differences between accepted and rejected submissions. The importance of reviewers’ comments can hardly be overemphasized since a study conducted amongst North American editors showed that about half of them relied exclusively on these comments to decide the fate of manuscripts.14
This study shows that, in spite of some criticisms and reservations appearing in recent literature regarding the place of case reports,15,16 they are accepted as frequently as other studies in the CJA. The clinical nature of the CJA and of anesthesiology may facilitate the acceptance of case reports.17 Link suggested that reviewers may evaluate papers submitted by authors coming from their own country more favourably.18 Our data do not allow us to comment or speculate on the impact of this phenomenom on the high proportion for Canadian submissions accepted for publication.
In conclusion, manuscripts submitted to a clinical journal like the CJA are subjected to a complex process of peer review. The forms provided to reviewers need to be discriminating while promoting a systematic approach. Analysis of the responses provided by peer reviewers to the CJA showed that many factors are influential in the decision to publish a manuscript. These factors have to be taken into account by authors who wish to publish an article. The relationship between the study design, the results, and the conclusions may be the most important element amongst the peer review criteria and should generate enough interest so that the reviewers will take time to comment constructively on this topic. Some presentation elements (tables, figures, references) are unlikely to be cited as causes for acceptance or rejection. On the other hand, lack of originality or inappropriate study design will influence reviewers to recommend rejection of a manuscript.
|
Acknowledgments |
|---|
|
Footnotes |
|---|
|
References |
|---|
|
TOPAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
2 Evans AT, McNutt RA, Fletcher SW, Fletcher RH. The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med 1993; 8: 422–8.[Medline]
3 Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA 1998; 280: 231–3.
4 Bordage G. Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports. Acad Med 2001; 76: 889–96.[Medline]
5 Joint Task Force of Academic Medicine and the GEA-RIME Committee. Review criteria for research manuscripts. Acad Med 2001; 76: 897–978.[Medline]
6 Bligh J. What happens to manuscripts submitted to the Journal? (Editorial) Med Educ 1998; 32: 567–70.
7 Nylenna M, Riis P, Karlsson Y. Multiple blinded reviews of the same two manuscripts. Effects of referee characteristics and publication language. JAMA 1994; 272: 149–51.[Abstract]
8 Cullen DJ, Macaulay A. Consistency between peer reviewers for a clinical specialty journal. Acad Med 1992; 67: 856–9.[Medline]
9 Eckberg DL. When nonreliability of reviews indicates solid science. Behav Brain Sci 1991; 14: 145–6.
10 Eichorn P, Yankauer A. Do authors check their references? A survey of accuracy of references in three public health journals. Am J Public Health 1987; 77: 1011–2.
11 Evans JT, Nadjari HI, Burchell SA. Quotational and reference accuracy in surgical journals. A continuing peer review problem. JAMA 1990; 263: 1353–4.[Abstract]
12 Rosenfeld RM. The seven habits of highly effective data users (Editorial). Otolaryngol Head Neck Surg 1998; 118: 144–58.[Medline]
13 Purcell GP, Donovan SL, Davidoff F. Changes to manuscripts during the editorial process. Characterizing the evolution of a clinical paper. JAMA 1998; 280: 227–8.
14 Wilkes MS, Kravitz RL. Policies, practices, and attitudes of North American medical journal editors. J Gen Intern Med 1995; 10: 443–50.[Medline]
15 Fox R. Writing a case report: an editor’s eye view. Hosp Med 2000; 61: 863–4.[Medline]
16 Wright SM, Kouroukis C. Capturing zebras: what to do with a reportable case. CMAJ 2000; 163: 429–31.
17 Vandenbroucke JP. In defense of case reports and case series. Ann Intern Med 2001; 134: 330–4.
18 Link AM. US and non-US submissions. An analysis of reviewer bias. JAMA 1998; 280: 246–7.
This article has been cited by other articles:
 |
|
 |
|
  D. A.E. Shephard The changing pattern of anesthesia, 1954-2004: a review based on the content of the Canadian Journal of Anesthesia in its first half-century: [La transformation du modele de l'anesthesie, 1954-2004 : une revue fondee sur le contenu du premier demi-siecle du Journal canadien d'anesthesie] Can J Anesth, March 1, 2005; 52(3): 238 - 248. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
