There’s an interesting conversation going on at Historiann’s blog about peer review. It’s especially interesting to librarians I think as a peek behind the curtain of academic publishing – at least a glimpse of what it’s like in certain disciplines.
I have always wondered how closely my experiences writing in the peer-reviewed library science literature match the reality of publishing in other disciplines – and I’ve assumed that they don’t match that well. It’s a little heartening to see some of my frustrations echoed here – especially where she says “Journals also seem to have no shared rules or system of peer review.” The amount and type of peer review I’ve gotten for my articles has varied pretty dramatically, and the extent to which I have been able to see and incorporate the reviewers’ thoughts has varied as well.
This is my favorite part – even though most of the post is about why this is an idealistic picture of a process that doesn’t look like this very often:
What’s not to like, with a fair and humane group of supportive senior scholars freely sharing their wisdom with their (usually junior) colleagues? Furthermore, having one’s work reviewed by supportive senior scholars is a really great way of making new friends and influencing influential people. I’ve had that experience a few times–and I’m truly grateful to the people who lent their time and expertise to make me a better historian.
I like this because I think it’s easy to forget this reason for the peer review process. At this point, with a huge structure of scholarly publishing, employment and reputation that both supports and relies upon our faith in the peer review process as a guarantor of quality – I think it’s easy to forget the “peer” part of peer review. And I also think that remembering it can help us as we figure out how to talk about peer review in a time where the landscape of scholarly communication is changing.
Right after I read this, I read the Tenured Radical’s lovely post remembering Charles Tilly, who died on April 29th. The whole piece is a wonderful tribute. This segment particularly struck me after reading Historiann’s description of ideal peer review as “a fair and humane group of supportive senior scholars freely sharing their wisdom with their (usually junior) colleagues?”
I didn’t know that at that moment in time you got Chuck and Louise as a package, and that once you fell into their orbit you never really left. You became part of this network of astonishing people with capacious intellects who came in and out of town, moving through offices that were a hive of activity, research and ideas. Looking at something I had written one day, Chuck said, “Theda Skocpol is coming through next week — let’s have her take a look at it and pick her brain.” Chuck ran a proseminar on the state which was my principle intellectual context during my final years in graduate school: one fall, in the first meeting, I walked in and sitting around the table were E.P. Thompson, Christopher Hill, Bridget Hill, and Eric Hobsbawm.
What’s not to like, indeed?
info-fetishist 
Hey–thanks for the link. I’m sorry that library journals are as disorganized and un-systematic as history journals in their peer-review process!
thanks for the post – I really liked it. I haven’t had any really negative experiences with journal publishing in library science – which is one way I think our experiences are different than those in a lot of more clearly defined disciplines – and some of my experiences have been really great. The journal portal: Libraries and the Academy offers a great mentoring-type relationship between reviewers and authors, for example. And other editors I’ve worked with have been really responsive and helpful. Other experiences, though, have been the professional equivalent of getting a paper back marked “A-/B+” with no comments as an undergrad – not really a bad experience, but unsettling.
To steal from Churchill, peer review is the worst way to make decisions on professional issues except for all the other ways.
I spent a lot of time running peer review, and I came to the conclusion that it is hard to do well. You need people who really understand the issues on which they are asked for opinions, who are willing to spend time and effort to do their jobs well, and who are disinterested, or at least whose interests are declared and known.
The peer review process has to be well managed. People do better in face to face panel meetings rather than mail reviews, but conducting panel meetings involves a lot of work, and heavy demands on reviewer time. I found it useful to avoid pressing for consensus in such meetings. I also found it very useful to have observers in the peer review meeting to evaluate the process and help interpret the advice and decide how to use it.
It is important to recognize that peer reviewers are not giving revealed truth, but only their informed opinion. Think about a probability distribution of reviewer values of a paper or proposal, based on the distribution of such objects. That can be considered an a priori probability distribution for quality. If you get a quality rating from a reviewer, think of Bayes Rule as a means of finding an a posteriori distribution of ratings. Thus the next reviewer may disagree strongly with the first.
We also know that there are all sorts of biases that reviewers have. Some are reluctant to deviate from average judgments, some are unwilling to make other than extreme judgments. Less knowledgeable or less serious reviewers are likely to provide less information in their reviews as to what others might do. I actually found one reviewer who on average disagreed with his peers — if he said to accept something, it was more likely that others would vote to reject it.
Still, how else are you going to review well complex objects that are highly technical?