Open Peer Review & Argumentation:
Loosening the Paper Chains on Journals
Tamara Sumner and
Simon Buckingham Shum of the Knowledge
Media Institute of the Open University, describe
the design and implementation of
the Journal of Interactive Media in Education,
as well as some of the issues behind the approach that this journal takes to the process of scholarly
review.
Contents
The emergence of the internet and the World Wide Web (WWW) have potentially profound implications
for scholarly practice, particularly in the submission, review, and publication of articles in journals.
However to date, much of the impact of these new technologies on journals has been on digitising the
products of journal publication; the scholarly processes involved in reviewing
articles remain unchanged and unsupported. We are using computer-supported collaborative argumentation
(CSCA) tools to rethink and redesign the process of scholarly debate at the heart of journal reviewing. This
paper describes the design principles behind our approach, how they are currently being realised in the
context of a specific new journal, and discusses some of the issues that this approach raises.
Keywords: digital journals, argumentation, electronic publishing, hypermedia,
WWW
The emergence of the internet and electronic publishing have far reaching implications for the way in which
knowledge is disseminated and sanctioned within scholarly communities, which since the Gutenberg
revolution in printing has been shaped by the affordances of paper. Currently, we are witnessing the
beginnings of a shift from paper to digital media in the submission, review and publication of scholarly
work. In many journals, it is now standard procedure for reviews and submissions to be transmitted
electronically. Sometimes, automated systems are in place for matching reviewers with submissions based
on keyword analysis. In a few cases, the final version is even published electronically as a set of HTML
files available on the WWW. Thus to date, this shift from paper to digital media has mainly affected the
products of publication. Various documents are becoming digital and some activities are automated, but the
actual process of scholarly work has not changed to a great extent.
What remains completely unsupported is the intellectual 'meat' of scholarly publication, the review process
itself. This remains a private affair involving a small number of reviewers, and in most paper-based
journals, the gap between initial submission and article publication can be dragged out over a year. The
potential of internet technologies for opening up and revitalising the scholarly debate process has yet to be
realised.
What is the nature of intellectual work in the peer review process? Reviewing a submission involves
drawing on certain criteria (e.g. theoretical content; empirical content; presentation quality; appropriateness
for the publication) to evaluate the quality of reasoning and evidence provided, to probe for weaknesses,
acknowledge strengths, and question background assumptions. In effect, reviewing is an argumentative
process where reviewers are engaging in an imaginary debate with distant authors who are not present to
respond to their analysis. This paper-based review model has shortcomings in that questions go unanswered;
confusions go unclarified; criticisms go undefended. The dynamic cut-and-thrust of debate normally found
in face-to-face contexts such as workshops or conferences is not supported by the paper-based review
processes, nor is it yet being realised in the new electronic media.
We are currently rethinking the review process to use new technologies in order to recapture the best
features of a dynamic scholarly debate. This rethinking is guided by existing research into hypertext-based,
computer-supported collaborative argumentation. Argumentation research is concerned with developing
notations and tools to facilitate public debate and negotiation.
The remainder of this paper will begin by reviewing previous work in argumentation and distilling
important design guidelines for creating argumentation tools. Next, we will describe how these guidelines
are being followed in the context of a specific new, tool-supported journal publication. Finally, we discuss
some of the issues which may be encountered in the new review model and its supporting technology.
CSCA research has focused on designing notations to support debate processes [1, 3] , creating
computational tools to support using the notations [4, 6, 9] and understanding the organisational contexts
and work practices necessary for such notations and tools to succeed [5, 11]. Key lessons have been learned
from previous research that point to design guidelines in each of these three areas:
- Lesson 1: Avoid elaborate and rigid notations. Much research has focused
on finding the "right" notation to support debate. Often, the resulting notations require people to express
their thoughts using elaborate sets of provided distinctions such as positions, issues, comments, pros, and
cons. This, however, runs the risk of burdening people with excessive representational overhead by forcing
them to categorise and commit their thoughts to rigid notations before they are ready. Empirical findings
indicate that people are often unwilling and sometimes even unable to do this [1, 2, 10].
- Lesson 2: Computational tools must integrate argumentation with the
artifacts being discussed. Early Csca approaches separated the argumentation from other artifacts (i.e., the
papers and drawings being discussed). This separation hindered users from (i) quickly accessing relevant
argumentation when it was most needed during problem-solving, and (ii) easily adding new argumentation.
Later work has shown that tools must support users to bridge the separation and move seamlessly in both
directions between the artifact and argumentation [4, 8].
- Lesson 3: Work practices must be redesigned so that argumentation is
integral to the task being performed. Empirical studies indicate that people often do not contribute to
argumentation because it is perceived as extra work over and above what they are already required to do
[5]. Successful argumentation approaches have redesigned work practices to make contributing to
argumentation integral to the overall task being performed [11].
Our goal is to develop a CSCA environment for scholarly review of journal submissions which takes into
account these three lessons. This environment is being developed to support the reviewing process of a new
scholarly journal, the Journal for Interactive Media in
Education (JIME).
The JIME Web environment has been carefully designed in accordance with three design principles
deriving from the three lessons introduced above.
Principle 1: Begin with a simple argumentative notation
The notation used to support reviewing in JIME has one primary class of contribution called
comments. Optionally, debaters may choose to categorise comments as
agreements (flagged as 
) or
disagreements (
). The power of the Web means
that reviewers may elect to include pointers to evidence in their comments; evidence can be
either plain text (e.g., "See Conklin & Begeman, 1988") or a Web hyperlink direct to an online source.
Thus, a body of secondary evidence may grow around an article under review, providing a valuable
resource for both readers and authors.
Principle 2: Integrate the publication with the argumentation
Prior to publishing for open debate, JIME submissions are marked-up to include special argumentation tags
(
). Each major document section and the interactive
demonstrations are prefaced with these navigation tags. Clicking on one takes the reader to the
argumentation about that document section, in the Review Window. Conversely, from the Review Window,
debaters can easily switch back to any part of the publication.
Figure 1 shows the user interface for reading an article in the JIME Web environment. Reviewers can also
download an Acrobat version since we believe that there is still a strong preference for being able to read a
paper version. Thus, a reader/reviewer has two main 'work spaces' in JIME - the Article Window for
reading, and the Reviews Window for making review comments, which are automatically opened and
brought to the front as needed.
Figure 1: Reading an article in JIME. In the lower left frame, the Comment icon takes the
reader to the top of associated Review Window (see Figure 3 below). The printer icon allows the reader to
download an Adobe Acrobat version for printing. In the main frame, the Comment icon takes the reader to
the review comments about this figure and associated demonstration. The Demo icon takes the reader to a
Macromedia Shockwave demo of the system (see Figure 2).
Figures 2-4 show screens for trying out a demonstration (Fig. 2), displaying the Review Debate about an
article (Fig. 3), and reading Reviews about a particular section of the article (Fig. 4). The Review
argumentation is added and manipulated using an environment which we have created by tailoring the
NCSA HyperNews system [7].
Figure 2: Authors describing a new system must submit demonstrations of it using an
appropriate technology. The above example shows an interactive demo using the Macromedia 'Shockwave'
plug-in. This allows readers and reviewers to interact with the system described in the article, as though
they were students using the original application.
Figure 3: Overview of a review discussion in JIME. The example shows the outline view with
headings for general categories of comment (e.g. Clarity of Goals; Credibility of Results), followed by
headings which match the headings in the article. Section-sepcific comments are placed under
these.
Figure 4: Details of a Review discussion in JIME. The example shows review comments about
a specific section in the article (3.3).
Principle 3: Make argumentation integral to the review process
The proposed review model for JIME submissions is shown in Figure 5. This model changes the role of
participants in the process to directly involve generating and manipulating argumentation. Reviewers return
their comments to the editor in the JIME argumentative format. The editor pulls together all the reviews to
seed the argumentative debate. The publisher marks-up the publication and the initial argumentation to
include the special cross-tool navigation tags. The article under review and the reviewers' initial comments
are then published on the Web, and the review process moves into a phase of open peer review, in which
authors, reviewers and readers can engage in debate. The editor then decides whether the article should be
accepted, and formulates change requirements for the authors. We also wish to allow for the possibility that
interesing discussion threads may arise during the review process which could be distilled into
commentaries for publication with the final article.
Figure 5: Lifecycle of a JIME article under review.
Once the article is published, debate about it will continue. Wth conventional journals, it is only at this point
that debate would begin, perhaps a year or more after initial submission. Even then, the debate is not tightly
linked to the article, but located in other articles, often published in other journals, at substantially later
times. In JIME, discussion can continue directly linked to the published article, or via external links to
related articles at other sites.
To summarise our design philosophy:
- JIME is not simply making the conventional media of paper-based
articles available on the Net, but addressing the interactive
dimensions which are the essence of the new media;
- JIME is not simply adding an e-mail listserver to allow peripheral
discussion about published articles, but developing
a journal review environment in which submitted articles
and open peer review are tightly integrated with
each other, and central to the journal's operation.
This project is still in its early design stages and the details of the current interface will undoubtedly change
as it undergoes further implementation and user testing. However, we feel it is important to discuss work
such as this in its early stages in order to encourage more participation and dialogue between scholars
concerning future publishing practices. We believe the review model presented here offers several benefits
for opening up and revitalising scholarly debate. However, there remain many open issues to be resolved.
Some issues relate to the user interface for supporting collaborative, Web-based argumentation. For
instance, the Web introduces new concurrency challenges regarding timely updating of reviewers' screens
with new contributions to the discussion. Additionally, Web-based systems offer impoverished degrees of
interactivity compared to what we now expect from direct manipulation interfaces, limiting our ability to
take advantage of new developments in graphical argumentation tools.
Another issue raised by our proposed model is the perceived risk which authors may feel when exposing
publications to large audiences at the review stage. Submissions will be critiqued by many more reviewers
than is currently possible in the conventional review process, and this will take place in a public rather than
private forum. The willingness of authors to do this depends greatly upon the professionalism and netiquette
exhibited by reviewers. The philosophy behind this model is that perceived risks of this sort will be
outweighed by the benefit to authors of quicker, more extensive, and more relevant feedback, resulting
ultimately in higher quality publications.
To conclude this brief article, we should not be surprised that electronic journals meet with resistance from
some quarters, since they force to the surface huge issues which entail rethinking how scholarly knowledge
is disseminated and sanctioned. This is the time to reflect radically and creatively on the 'papyrocentric'
scholarly practices which have until recently been taken for granted. JIME is pushing the boundaries of
electronic journals one step further with its Web-based open peer review. This seems to us the logical
direction in which to take journals. We await your reactions with interest.
- Buckingham Shum, S., "Analyzing the Usability of a Design Rationale Notation," in Design Rationale:
Concepts, Techniques, and Use, T. P. Moran and J. M. Carroll, Ed., LEA, Hillsdale, NJ, 1996, pp. 185-215.
(Abstract :: Preprint - postscript :: Request reprint)
- Buckingham Shum, S. and N. Hammond, "Argumentation-Based Design Rationale: What Use at What
Cost?," International Journal of Human-Computer Studies, Vol. 40, pp. 603-652, 1994. (Abstract :: Preprint - postscript :: Request reprint)
- Conklin, J. and M. L. Begeman, "gIBIS: A Hypertext Tool for Exploratory Policy Discussion," ACM
Transactions on Office Information Systems, Vol. 6, pp. 303-331, 1988.
- Fischer, G., A. C. Lemke, R. McCall and A. I. Morch, "Making Argumentation Serve Design,"
Human-Computer Interaction, Vol. 6, pp. 393-419, 1991.
- Grudin, J., "Evaluating Opportunities for Design Capture," in Design Rationale: Concepts,
Techniques, and Use, T. P. Moran and J. M. Carroll, Ed., LEA, Hillsdale, NJ, 1996, pp. 453-470.
- Hair, D. C., "LEGALESE: A Legal Argumentation Tool," ACM SIGCHI Bulletin, Vol. 23, pp. 71-74,
1991.
- LaLiberte, D., "HyperNews,". National Center
for Supercomputing Applications, University of Illinois at Urbana-Champaign.
- McKerlie, D. and A. MacLean, "Reasoning with Design Rationale: Practical Experience with Design
Space Analysis," Design Studies, Vol. 15, pp. 214-226, 1994.
- Schuler, W. and J. Smith, "Author's Argumentation Assistant (AAA): A Hypertext-Based Authoring
Tool for Argumentative Texts," in Hypertext: Concepts, Systems and Applications, A. Rizk, N. Streitz and
J. André, Ed., Cambridge University Press, Cambridge, 1990.
- Shipman, F. and R. McCall, "Supporting Knowledge-Base Evolution with Incremental Formalization,"
Proceedings of ACM CHI'94, Boston, Mass., 1994, pp. 285-291.
- Terveen, L. G., P. G. Selfridge and M. D. Long, "From "Folklore" to "Living Design Memory"," in
Proceedings of ACM/IFIP INTERCHI'93, 1993, pp. 15-22.
The Journal of Interactive Media in Education is at: http://www-jime.open.ac.uk/jime/
Open peer review of the first submission to JIME begins on 2nd Sept.'96. We invite all Ariadne readers to
participate and leave comments, either on the article itself if this is an area in which you are qualified, or on
the journal's design more broadly.
An earlier version of this article was presented at HCI'96, Annual Conference of the
British Computer Society on Human-Computer Interaction, London, 20-23 August, 1996.
Material on this page is copyright Ariadne/original
authors. This page last updated on September 11th 1996