We would like to discuss Stodolsky's "consensus journals" model (Stodolsky, 1990; 1992). We will quickly show that the context in which such a model can be applied has to be precisely defined. Examples will then be given to illustrate our views and show both the scientific and pragmatic limits of this proposal. For us, Stodolsky's attempt to automatize electronic publication could be considered as an additional approach for article submission rather than one which will replace those existing ones.
2. In paragraph 1 the author suggests that "the intensity of evaluation is much greater with electronic journals". According to Stodolsky, quality of judgement will thus be the result of a transition from paper to electronic publication, i.e. only by changing the medium of communication. No explanation is given as to how this will change the quality of evaluation, mearly the possibility of sending an article to hundreds of referees instead of three. We agree that electronic communication has sped up diffusion of both submitted and published articles. Nevertheless, in spite of facilities to send a submitted article to twenty reviewers, the editorial board of an electronic journal can decide to send it to only three of them. We thus do not consider sending a submission to a limited number of referees as "inappropriate in the computer network environment". The main weakness of a two or three referee evaluation is its main strength. Randomness can play a role in publication and give an unpopular theory a chance to be diffused. Of course, this system can be discussed on many points, but the switch from a book-like to an electronic publication mode is not a good argument to change the evaluation process of a journal. Nowadays, most of the journals ask authors to send an electronic version of their submitted article in order to speed up the exchange of information with their referees.
3. A review is not better or done quicker because a referee receives the article a day after the editor. A good review takes time and still will. Besides, to each one its due: the role of an editor is to choose between referees to review one submitted article. The role of referee is to review the article. Such tasks are time demanding because they involve both editor's and reviewer's responsibilities. As written, "[a consensus journal] requires employing more people in the editorial process"(paragraph 8). We are not sure such a requirement will make this new kind of article's submission popular in the short run. We are even less sure that it would improve objectivity of scientific publication. Stodolsky tries to change the role of both editors and reviewers. Editors send articles to reviewers, this would not be the case if reviewers have to withdraw articles from a database. Referees would switch from being chosen by editors to withdrawing articles that call their attention.
4. This proposal to send a short summary of the article before submitting it as a whole seems incomplete. If you have an experiment that you want to publish, both method and data analysis can evolve in time while writing or previewing results. The reviewer will only have a slight idea of what the article could bring to the research domain. This leads to unanswered questions when you read the commented article. Does Stodolsky's suggestion really differ from the main goal of seminars or international conferences: through free discussions and criticism, an improvement of papers and therefore of the global theory? Could any editor make a real difference between a lot of "one paragraph reactions to the target article" (paragraph 1-4)? What kind of scientific credibility could a one paragraph proposal have? Especially when you bear in mind that this paragraph is not about a whole article but a short version withdrawn from a database. Finally, what could be the level of confidentiality of such invitational journal if every body can react on a subject?
5. Although many points in paragraph 2 could be discussed, we would like to focus on the attempt to find a way to measure performance through statistics in order to achieve the "fully automatic invitational journal". We consider this concept as very odd if not dangerous for science. Such statistical methods to decide about publication lead to uniformity and don't seem to be the best way to test methods and concepts. This is not the case of the "invitational journal based upon editorial consensus" proposed by Stodolsky which is a pretty good idea on its own. The "manual approach" enables alternate articles to have a chance to be published. This would not be the case in "fully automatic invitational journal". Without any philosophical discussion, just note that scientific revolution often started from unpopular concepts firstly expressed by isolated scientists.
6. Most of the problems on objectivity related to publishing would be solved by using an "objective consensus model based upon statistics" (Stodolsky, 1990). According to Thomas Khun (1970), history of science can be defined as shifts from one dominant theory to another. One theory can have the consensus for a couple of decades while the impact of a 'minor' theory grows until the 'minor' becomes dominant and so on. A good example is the raise of dynamical systems approach applied to motor control as described by Abernethy and Sparrow (1992). By using consensus as the criteria for publishing, editors will re-enforce dominant theories and slow down the rise of new ones. Two opposing theories can easily coexist. A good example is what happened between Albert Einstein and Niels Bohr. They have mutual respect for each other as scientists but no consensus was possible between them on quantum theory. It was impossible for Einstein to admit a Bohr's probabilistic conception of reality. Einstein can be thus considered as the last classical physicist, i.e. the symbol of the fall of a exclusive theory.
7. In conclusion, we would like to point out that Stodolsky's proposal is simple and seducing. Objectivity has always been the grail of science. This might explain why he is trying to elaborate universal rules in terms of publishing instead of simply considering his model as a new way to publish. When submitting an article, you have to choose whether it is going to live on its own or whether it is a target article written in order to make people react. 'Classical' journals will mix old and new theories whereas Stodolsky's consensus would only show what the dominant theoretical paradigm is at the moment.
Abernethy, B. & Sparrow, W.H. (1992). 'The rise and the fall of dominant paradigms in motor behavior research'. In J. J. Summers (Ed.), Approaches to the study of motor control and learning (pp. 3-45). North-Holland.
Kuhn, T.S. (1970). The structure of scientific revolutions. The University of Chicago Press.
Stodolsky, D.S. (1990). Consensus Journals: Invitational journals based upon peer consensus. PSYCOLOQUY 1(15) ftp://ftp.princeton.edu/pub/harnad/Psycoloquy/1990/psyc.arch.1.15.90
Stodolsky, D.S. (1992) Invitational Journals Based Upon Editorial Consensus. PSYCOLOQUY 3(67) ftp://ftp.princeton.edu/pub/harnad/Psycoloquy/1992.volume.3/ psyc.92.3.67.consensus-journals.1.stodolsky http://www.cogsci.soton.ac.uk/cgi/psyc/newpsy?3.67