A New Process for Peer-Reviewed Journals

Matthias Grossglauser and Jennifer Rexford

(an idea based on a presentation in the "outrageous opinions session" at ACM SIGCOMM 2000)

Scientific journals are the main vehicles for the dissemination and archiving of scholarly information. A good journal ensures the high quality of articles on behalf of its readers. The quality of a published article depends essentially on three aspects: (i) technical correctness; (ii) novelty and significance; (iii) timeliness. We argue that current journals emphasize technical correctness over significance, and that turnaround times are too long. We believe that this is a direct result of the paper evaluation process used by journals, because the most critical resource a journal has to manage is the time of peer reviewers. The traditional evaluation process wastes this resource unnecessarily, because it relies on a written peer review to evaluate both technical correctness and significance. As a result, reviewers are swamped and take a long time to complete reviews.

We propose a new reviewing process that is based on two assumptions: (a) novelty and significance of a paper can be typically assessed in a much shorter time than its correctness; (b) on the other hand, significance is a more subjective quality than correctness; therefore, peer reviewers can be expected to disagree much more on the significance of a paper than on its correctness. In our proposed reviewing process, all submitted papers go into a pool, where they remain for a fixed amount of time. Members of the editorial board browse this pool and search for papers that seem to make a novel and significant contribution. They then select a fixed number of papers per cycle (e.g., a month) for an in-depth peer review. Therefore, only a subset of prescreened papers get reviewed for correctness. As each paper may be selected by any member of the editorial board, the "noise" in the significance estimation is averaged out. This process conserves reviewer time, makes reviewing more interesting, and reduces the noise in the significance assessment.

The second novelty of our journal architecture is the process to appoint the editorial board. The role of the editorial board is to run the entire reviewing process, including prescreening of papers, selection of reviewers, and making the ultimate editorial decision; editorial boards are usually self-appointed. In our journal architecture, the editorial board also has the additional responsibility of selecting significant papers out of the pool of submitted papers. We therefore believe that the editorial board should be elected by the readership, to ensure that the board accurately represent the interests and focus of the readership. The editorial board could be elected, for example, on an annual basis, where each candidate for board membership would submit a written statement defining the scope of papers he or she will focus on.

In summary, the proposed process reduces the workload on reviewers, by prescreening papers for significance and novelty by the entire editorial board. This makes reviewing more enjoyable, and reduces turnaround time. Also, paper selection is more fair, because an elected editorial board jointly decides on the set of significant papers to review; the reviewers only verify technical correctness.