Objectivity has long been both an ideal in scientific communication and the basis for accreditation claims by authors of journal articles, yet the evaluation of articles for publication proceeds on a mostly subjective basis. This has been particularly true in the case of "invitational journals," where an unassisted editorial judgement may determine whether or not something is published. A "consensus journal" has been proposed, that uses a rigorous measurement model based on multidimensional peer judgements to generate invitations, but this model assumes an infrastructure that is not currently available. An invitational system based on similarity judgements could also yield objective performance measurements. Such a system could be a transitional one between currently existing editorial mechanisms and those of a consensus journal.
2. Stodolsky (1990) presented a role-free model for invitational journals based upon peer consensus. This model is limited in that it assumes preexisting dimensions for evaluative judgments and the availability of a robust statistical procedure for calculating consensus positions among reviewers. The objective of issuing invitations to the most knowledgeable proponents of different consensus positions can also be achieved by creating an editorial role. This may require a redefinition of what is meant by a "consensus" position and a "most knowledgeable" proponent. It is not clear that the same performance can be achieved with this new model. However, the new model can be used to build the set of preexisting dimensions needed to operate the "fully automatic" invitational journal. The approach discussed here is "manual," because editorial intervention is required before invitations can be issued. However, as with the earlier model, statistical measures of performance are available. We can call this new model an "invitational journal based upon editorial consensus."
3. For the purpose of this discussion, we assume that articles persist in a database until they are withdrawn. Editorial expertise is measured by analyzing, in the first instance, the agreement of editors in issuing invitations. The final criteria depend on what articles are actually submitted, whether any are determined to be redundant, and how long articles persist before they are withdrawn.
4. After reading a published article, potential authors submit short (e.g., one-paragraph) reactions to the target article. These reactions are treated as proposals for new articles and directed to a jury of editors (this could be done by a set of corresponding editors if more then one jury was available). The editors then independently select a subset of the proposals judged to be mutually exclusive and representing the most competent opinion. The judgment of mutual exclusivity can be performed by sorting proposals into groups. Within each group, the potential authors would be addressing the same question. Between groups there would be a significant difference in what question was being treated (or how it was to be treated). Discrepancies among groupings by editors would require a "consensus" set of groupings to be calculated. Editors' performance could be calculated from their judgements using rater reliability statistics. Later articles and responses to them would be the final criterion as to whether two articles were, in the final analysis, distinctive (addressing different questions or the same question in a different way).
5. When two or more proposals were sorted into the same group, an author would be selected on the basis of expertise. In the simplest case, the editorial choice could also include a judgement of author expertise, based upon the proposal submitted. A more objective mechanism would base choice on previously demonstrated performance. For example, an author might be issued a credential for each week a submitted article was maintained in the database. Then, in the case of nonexclusive proposals, the author with the largest number of credentials (in that subject area) would automatically be issued an invitation. A further consideration could be the past performance of the author in responding to invitations. For example, an author who failed to respond to invitations 10% of the time might lose future invitations one of ten times to less competent peers.
6. Similar objective mechanisms could be applied to editorial responses, allowing editors' judgments to be weighted according to previous performance. A system of credentials and selection mechanisms could also be developed, as suggested above for authors. The final criterion for editorial performance could be the number of article-weeks generated by invitations.
7. Given a history of operation for the journal structure described, the groupings made by editors could be used as a basis for generating dimensions for evaluative judgments. A reliable set of such dimensions would permit more efficient sorting of proposals within the framework of an invitational journal based upon editorial consensus; it would also permit the testing of the more powerful invitational structure based upon peer consensus.
8. Thus, the invitational journal based upon editorial consensus can be viewed as a stepping stone between current invitational journals such as sci.psychology.digest (PSYCOLOQUY's Usenet Edition) and the more advanced invitational structures proposed by Stodolsky (1990). The transition from current practice requires employing more people in the editorial process. However, it also permits greater automation in the administration of a journal. A very important benefit is the quantitative measurement of editorial performance that could be used to support claims of journal quality.
Stodolsky, D. S. (1990). Consensus Journals: Invitational journals based upon peer consensus. PSYCOLOQUY 1(15) psyc.arch.1.15.90. Also appeared in: Datalogiske Skrifter (Writings on Computer Science) No. 29 1990. Roskilde University Centre, Institute of Geography, Socioeconomic Analysis, and Computer Science (ISSN 0109-9779-29).