Simon Buckingham Shum (2000) Research Needed on Online Usage and Peer Review. Psycoloquy: 11(094) Electronic Journals (7)

Volume: 11 (next, prev) Issue: 094 (next, prev) Article: 7 (next prev first) Alternate versions: ASCII Summary
Topic:
Article:
PSYCOLOQUY (ISSN 1055-0143) is sponsored by the American Psychological Association (APA).
Psycoloquy 11(094): Research Needed on Online Usage and Peer Review

RESEARCH NEEDED ON ONLINE USAGE AND PEER REVIEW
Book Review of Tenopir & King on Electronic-Journals

Simon Buckingham Shum
Knowledge Media Institute
The Open University
Milton Keynes
MK7 6AA
United Kingdom
www.kmi.open.ac.uk/sbs

sbs@acm.org

Abstract

We know little about how scientists search for articles, read them, annotate them, share them, or review them, and how they wish things might be in the future. How could ejournals and other network resources better support these activities? We need to know not only what and how much scientists do, but how they accomplish this work individually and collectively in their day to day lives. A detailed discussion of the pros and cons of new, electronically mediated peer review models would also be welcome.

Keywords

copyright, citation impact, digital library, electronic archives, electronic publishing, electronic journals, peer review, publication costs, research funding
1. The quotes on this book's cover from eminent figures do indeed reflect it's contribution as a compendium (488 pages) of facts, figures and analysis on the production and use of scholarly journals. King and Tenopir (2000a,b) have been working in the field of scientific knowledge dissemination for many years (King's earliest listed reference going back to 1971), and bring a carefully weighed perspective on both paper and digital technologies for research dissemination.

2. The subtitle, "Realities for Scientists, Librarians and Publishers" reflects their concern to represent the interests of all the key stakeholders in the situation in which we now find ourselves. Moreover, their understanding of these concerns helps to explain the inertia of an industry which has evolved over 300 years from the first journals in order to satisfy the varied agendas that are in play today. They emphasise throughout that innovations in the process or product of scientific publishing will have to clearly prove their worth to be sustained in the long term. This may seem like stating the obvious, but amidst the charged, frictionless rhetoric of the internet and e-business, this is a welcome reality check.

3. For someone relatively new to the publishing industry (ejournals are but one of many implications of the knowledge and collaboration technologies that are my research field), the book was extremely useful as an introduction to the field's players, scale and dynamics, as well as a treasure trove of references, especially to early efforts to develop network-based alternatives to paper dissemination. The NSF and British Library initiatives of the 1960s-80s are not cited much, giving the impression that preprint archives and ejournals are a Web phenomenon. This is, however, anything but a book concerned only with ejournal technology. The title "Towards Electronic Journals" reflects its historical perspective, providing us with what is probably the best factual foundation for understanding paper journals today, as well as the potential impact of new technologies. A glance at the list of Tables gives a flavour of what is to come, e.g. "Average importance ratings of factors used by authors for selecting journals, 1976, 1979", "Break-even point in readings between personally subscribing to journals and using the library at various journal pricings and distance to library", Total journal costs by academic and special libraries, projected to 1998", Start-up ongoing costs for an electronic journal".

4. Tenopir & King's (2000b) extended precis in PSYCOLOQUy gives a useful overview for prospective readers. The rest of this review will therefore overlay a particular lens, namely, that of an ejournal co-founder and editor (of what I discovered is a "Level 6" ejournal in terms of its functionality; p.327). Specifically, I wanted (1) to better characterise my own ejournal's niche in the broader scheme of scholarly publishing, and (2) find expert assessment of the claims often made about the economic and dissemination improvements that networked dissemination (not limited to just ejournals) can make.

5 The "Executive Summary" provided in Chapter 1 lists the authors' key conclusions on the potential impact of ejournals, details of which are then unpacked in later chapters. A key conclusion seems to be that ejournals could reduce costs: "Perhaps the greatest challenge and best opportunity for electronic journals involves future pricing. It is clear that past strategies have not been successful and that new, truly innovative approaches will be required." (p.44). This is qualified with emphasis on factors such as journal ciculation, publisher size, and whether publication is dual media (paper and electronic) or electronic only.

6. The authors present an excellent critique of the commonly encountered argument that the average article in a paper journal is hardly read (hence we need new dissemination routes, and even to disband the whole concept of the journal). The whole of Chapter 7 is devoted to the qualitative and quantitative readership of journals (drawing attention to different patterns by academic and industry scientific communities), and making the case that (in general) journals are both regularly read and beneficial (there are even figures suggesting that the most successful scientists read more journal articles than the average).

7. Specifically on electronic publishing, Chapters 15-18 present a valuable collection of data, although Chapter 16 (Economic aspects of the internet) is, inevitably, already dated, and better served by a related website that can be updated (the number of ISPs is estimated from a 1996 report; the value of the network services market is estimated from a 1995 report).

8. A lot of the authors' work has examined costs at every stage in the dissemination process, and publishers and librarians should find it a valuable reference point. Occasionally, I was a little bemused by some of the analyses, such as the following cost exercise:

    "The second element of cost [to readers] involves the time of
    readers... We estimate scientists' time at $0.805 per minute and
    other such expenses as equipment and facilities at $0.06 per
    minute.  Printing is estimated to cost $0.53 per article ($0.045 X
    11.7 pages).  Thus, the reader cost per reading for obtaining an
    electronic article online from a subscription is approximately...
    $7.00 per reading, versus $5.10 for reading paper versions"
    Therefore it costs less for the electronic version up to 10.5
    readings (or more), while it costs more electronically above that
    amount, although the difference is not enough to be of great
    concern." (p.377).

I have to confess to wondering how useful such quantifications will be, based as they are on a particular set of assumptions. Often, the data for the assumptions comes from questionnaires to researchers about their practices, but there was no critical discussion, e.g. in discussing reading habits, of whether scientists might not be serving several agendas (only one of which is to give the researcher the truth) when reporting how much they read of what.

9. I was, on occasion therefore, left feeling unconvinced. Obviously, a book cannot do everything, and the author's meticulous attention to costing the full lifecycle of documents served to highlight for me a limit to this book's contribution, where future research must take up the baton. Quantitatively, the book scores highly on describing the terrain over which journals have travelled, but there is little which will help ejournal publishers design their journals, the terrain we have yet to cross. Qualitatively, we seem to know relatively little about how scientists search for articles, read them, annotate them, share them, or review them -- the stuff of everyday life -- and how they wish things might be in the future. How could ejournals and other network resources better support these activities?

10. The book contributes high level descriptions of new functionality that ejournals could offer, based on a few examples, but there are no evaluation reports of such ejournals in terms of user interaction or satisfaction. The main reason is likely to be that such data simply do not yet exist, and this limitation is a function of the book's timing in this transitional phase of publishing history. Given the correct importance that the authors place on genuine, value-added innovation in dissemination media, this research vacuum stands out all the more. We need to know not only what and how much scientists do, but how they accomplish this work individually and collectively in their day to day lives. Without such naturalistic, design-oriented knowledge, software developers cannot be blamed for thinking they know what researchers do, only to perpetuate the pattern of providing tools that are never really integrated into everyday working life.

11. A second non-trivial limit to the book's coverage concerns the peer review process. Whilst the book makes evident that journals serve multiple functions, to the most important stakeholders -- namely the researchers -- the quality control function is what distinguishes journals from the vanity press and anything else that can be placed on the internet. In the whole book, we find three and a half pages dedicated to peer review (pp. 136-139), summarising some research on the amount of time scientists spend as referees, and a couple of proposals for peer review ratings over networks. There are occasional appearances of the topic elsewhere (e.g. citing studies that confirm the value that researchers place on peer review). There is, however, a substantial research literature into the effectiveness of the anonymous peer review system which does not receive any mention. This is surprising, since elsewhere the authors note that the journal system is often criticised for its delays in publishing (due in part to stalled peer review processes): but the key point is that journals as a concept are most clearly distinguished by the freely donated refereeing of researchers.

12. If there is a second edition of the book, then a detailed discussion of the pros and cons of new, electronically mediated peer review models would be a prime candidate for inclusion. For instance, radical proposals that envisage the replacement of journals with a networked repository of unrefereed literature with voluntary ratings are technically implementable, but fail to engage many of the human dimensions: When we're all too busy, why would anyone bother to rate something? What does a number represent, and do individuals use ratings comparably? How to prevent abuse of the system? Do studies critical of the fairness of current review processes support the idea of simple digital replication, or (as we have done in our e-journal) is this an opportunity to promote such 'radical' ideas as the author's right of reply during review?

13. Finally, the book comes to a rather abrupt end after a discussion of alternative pricing models. Perhaps the authors were content to let the chapters' overviews and summaries stand without further elaboration, but some concluding comments would have been welcomed (or is the relentless desire for closure merely a vestigial legacy of the codex before we learn to write open hypertexts?!).

14. In summary, for anyone interested in scholarly publishing, electronic or paper, the authors are to be congratulated on coherently organising and discussing a vast amount of data from their own and others' work. This provides a sobre, balanced foundation on which to build the next generation socio-technical infrastructures we require to access timely, quality research information.

REFERENCES

Tenopir, Carol, and Donald W. King (2000a) Towards Electronic Journals: Realities for Scientists, Librarians, and Publishers. Washington, D.C.: Special Libraries Association. http://www.sla.org

Tenopir, Carol, and Donald W. King (2000b) Precis of: "Towards Electronic Journals" PSYCOLOQUY 11(084) ftp://ftp.princeton.edu/pub/harnad/Psycoloquy/2000.volume.11/ psyc.00.11.084.electronic-journals.1.tenopir http://www.cogsci.soton.ac.uk/cgi/psyc/newpsy?11.084


Volume: 11 (next, prev) Issue: 094 (next, prev) Article: 7 (next prev first) Alternate versions: ASCII Summary
Topic:
Article: