UC San Diego New-Media Expert Pushes Peer Review into the 21st Century

San Diego, CA, March 21, 2008 -- The populist blog could be the future of academic publishing. In a daring attempt to rewrite – or at least edit – the rules of scholarly peer-review publishing, a professor at the University of California, San Diego this week posted the last chapter sections from his upcoming book on a web log – opening it up to comments from anyone, anywhere.

Streaming video clips of Noah Wardrip-Fruin on:
- The genesis of his blog-based peer review
experiment: click here. Length: 1:36
- The surprise benefits of peer review via blogging:
click here. Length: 3:26
[Windows Media and broadband connection required]

Communication professor Noah Wardrip-Fruin posted the manuscript of "Expressive Processing: Digital Fictions, Computer Games and Software Studies" in daily installments, averaging 2,000 words, over a ten-week period on Grand Text Auto. The GTxA blog is co-authored by Wardrip-Fruin and five other artists, game designers and academics. Its blend of commentary and high-minded discussion on videogame culture and new media attracts roughly 35,000 unique visitors each month.

"I wanted to experiment – and take an existing academic blog, where people are already talking about the kinds of things you care about, and make them part of the peer review process," said the UCSD academic. "My editor at MIT Press was very supportive of doing something like this as a way of exploring the space, but he was skeptical that I would learn anything much of value. He was also annoyed that it was positioned as a competition between blog-based and traditional peer review."

As his editor, Doug Sery, told the Chronicle of Higher Education, "the reputation of MIT Press, or any good academic press, is based on a peer review model." So even as he was giving Wardrip-Fruin the green light to post eight chapters of the manuscript on the blog, Sery sent the same chapters to three peer experts who would submit blind reviews.

Final 'meta' post of Chapter 8 from the as-yet-unpublished-in-print "Expressive Processing", by Noah Wardrip-Fruin, on the Grand Text Auto blog

A crucial element of the experiment was the development of software tools to allow readers to comment via an existing blog on a paragraph-by-paragraph basis. With funding from the UCSD Academic Senate, Wardrip-Fruin approached Ben Vershbow at the Institute for the Future of the Book about adapting the institute's CommentPress tool for inserting digital notes in the margin of a blog. They collaborated with UCSD researcher Jeremy Douglass and WordPress developer Mark Edwards. Starting in January, Grand Text Auto incorporated the new plug-in into its existing site and invited visitors to submit comments.

"My argument for blog-based peer review is that it's a good thing to do in an existing online community, where people who know a lot about the subject (inside and outside the academy) are already part of the community," said Wardrip-Fruin.

"I was impressed at how important conversation turned out to be, although that should have been expected," continued Wardrip-Fruin. "In one of the first postings, someone said he felt there was something wrong with the way I was discussing The Sims™. I responded that I could probably just massage a couple of sentences and fix it. Then somebody else came in a few paragraphs later and said the problem was more with the way I connected The Sims to Chris Crawford's idea of process [vs. data] intensity. That was insightful, and I probably wouldn't have discovered that in the one-way, one-step response that you get from traditional peer review."

During the 10-week experiment, hundreds of comments were attached to the manuscript on the blog, primarily from roughly 60 core commenters.

Before the experiment began, the author feared "a lot of noise" from comments that would not really move the conversation forward. "As it turns out, there has been very low noise, and I think that has to do with the fact that we're doing it within this existing academic community that has already set the standards for what it means to comment on the Grand Text Auto blog," explained Wardrip-Fruin. "So I've got the best of both worlds: a relatively low noise threshold, but a lot more voices than I would have gotten otherwise."

Wardrip-Fruin scrutinizes the latest blog comments on his book manuscript.

From the author's perspective, blog-based peer review has other advantages. "When I write traditional reviews for journal articles and books, I tend to focus on the high level argument, and the shape of that argument, and whether the authors are making their case," he said. "Unless something is glaringly wrong, you just don't take the time in one of those three-page reviews to say, by the way, they get this example subtly wrong. So one of the advantages of blog-based peer review is that people really engage with each section of the manuscript."

Another benefit, according to Wardrip-Fruin, was the variety of people submitting comments: "It was important to get the views of people in the videogame industry for example, who would never participate in a traditional academic peer review."

"Expressive Publishing" explores the world of videogames, game culture and interactive fiction through the lens of an emerging critical field called 'software studies'. Wardrip-Fruin is co-director of a new Software Studies Initiative based in the UCSD division of the California Institute for Telecommunications and Information Technology (Calit2) and supported by the Center for Research in Computing and the Arts (CRCA), which hosts the GTxA blog.

Wardrip-Fruin in the gallery@calit2 in san diego

Wardrip-Fruin plays with [giantJoystick] in the gallery @ calit2, an interactive sculpture created by artist Mary Flanagan, one of Wardrip-Fruin's co-authors on the Grand Text Auto blog.

Now that the blind reviews are in, Wardrip-Fruin is beginning to digest what the reviews are saying, although he also expects to incorporate comments from the blog that come in over the next few weeks. "I'm having a very interesting experience," he said. "The two groups see the project very differently and have very different sorts of comments. But they point to exactly the same major revision for the manuscript!"

Specifically, he says, the reviewers on the blog felt that they lost sight of the overarching themes of the book, and suggested that the broad argument – now primarily detailed in the first chapter – be threaded through the individual chapters more.

The traditional reviews reached the same conclusion, but for different reasons. "All three of the blind peer reviewers had a very clear sense of the overall argument of the book," said Wardrip-Fruin. "But they strongly disliked the first chapter, which lays out the argument in a compressed, rapid-fire way, depending on readers to get through one challenging chapter and then remember it for the rest of the book." The blind peer review comments therefore also point toward presenting the main body of the argument gradually, over the course of the book's chapters, rather than compressing so much of it into the first chapter.

The author believes that blog-based peer review will never completely take the place of the traditional kind. But, he argues, the blog format works well "in an existing online community where people who know a lot about the subject are already part of the community." The blog-based review is particularly useful, Wardrip-Fruin adds, in fields where non-academics such as game developers, industry researchers and community historians can offer insights that might never occur to an academic.

In those cases, he argues, blog-based and traditional peer review can be complementary and valuable.

"Frankly, I hope I won't be the last person to do this," concluded Wardrip-Fruin. "I'm hoping we'll see more experiments of this sort once we release the software for other people to use. When that happens, we'll hopefully be able to write up some recommendations for when this kind of peer review can be a powerful addition to traditional peer review, and what sort of audiences we think should pursue it and how."

Until then, the UC San Diego professor can pride himself on making a believer of one important reviewer who left at least one comment on the blog. That reviewer: Doug Sery – his editor at MIT Press.

Media Contacts

Media Contact: Doug Ramsey, 858-822-5825, dramsey@ucsd.edu

Newsroom > News Release

UC San Diego New-Media Expert Pushes Peer Review into the 21st Century