I spent a lot of time running peer review, and I came to the conclusion that it is hard to do well. You need people who really understand the issues on which they are asked for opinions, who are willing to spend time and effort to do their jobs well, and who are disinterested, or at least whose interests are declared and known.

The peer review process has to be well managed. People do better in face to face panel meetings rather than mail reviews, but conducting panel meetings involves a lot of work, and heavy demands on reviewer time. I found it useful to avoid pressing for consensus in such meetings. I also found it very useful to have observers in the peer review meeting to evaluate the process and help interpret the advice and decide how to use it.

It is important to recognize that peer reviewers are not giving revealed truth, but only their informed opinion. Think about a probability distribution of reviewer values of a paper or proposal, based on the distribution of such objects. That can be considered an a priori probability distribution for quality. If you get a quality rating from a reviewer, think of Bayes Rule as a means of finding an a posteriori distribution of ratings. Thus the next reviewer may disagree strongly with the first.

We also know that there are all sorts of biases that reviewers have. Some are reluctant to deviate from average judgments, some are unwilling to make other than extreme judgments. Less knowledgeable or less serious reviewers are likely to provide less information in their reviews as to what others might do. I actually found one reviewer who on average disagreed with his peers — if he said to accept something, it was more likely that others would vote to reject it.

Still, how else are you going to review well complex objects that are highly technical?