When comments collide

at least in my head.  My head is tired, though, from speaking at this and at this.  So I may not be making any sense.

Last week before I got sucked into preparing stuff I had a brief exchange about peer review with John Daly on the K4D blog and he said something that resonated but I wasn’t able to take time to think about it until now —

Perhaps more importantly, peer review works when it works because experts have learned ways to evaluate information — the clarity of hypotheses, the appropriateness of the sample, the quality of the instruments used, etc. We can all use these same criteria to evaluate information. We don’t usually because we only need information that is “good enough” and it is not worth while to spend more time and effort getting information of higher quality that one needs.

I wonder to what extent this line, “we can all use these same criteria to evaluate information” holds true for undergraduate students.  I mean, we all can if we know how – but do they know how?  When do we learn what those methods are?  I don’t think that many of us in libraries are teaching that – and I’m wondering how much that is being taught throughout the curriculum, particularly at the gen-ed or core-curriculum level?

And I’m even wondering to what extent people really immersed in a discipline would even agree that there are general evaluation skills that we can all learn?  Does the way academia is structured, and the way it supports intellectual specialization work against this idea to a certain extent?

(and when I say “I’m wondering” I mean I’m really wondering – not just passive aggressively suggesting an answer.  It’s been so long since I was immersed in a curriculum myself that I don’t really know)

Daly continues on  to say that “I have been impressed by the failure of my students in the past to see that they can transfer their information literacy from one field to another. Teaching scientific experts to use their skills to evaluate foreign policy information is an example of something that seems easy but is really quite difficult.”

And this echoes something that Barbara Fister said on my Why I am a Librarian post several months ago —

The issue of expertise is fascinating. In some ways, faculty in the disciplines defer tremendously to expertise, even to the point of saying “I can’t comment on that issue, because it’s not my field.” Well …. we didn’t get training in most of what the world’s about, but we did get training in how to read and analyze and respond. Sometimes we have to figure it out even if it’s not familiar.

That’s one of the devil-may-care things you get to do as a librarian, figure it out. And help other people learn how to think for themselves.

That’s been my  experience too – that the more expert one becomes in one field or sub-field the more one feels inexpert in others.   The thing is, even when experts in one field defer to experts in another, they’re not doing so entirely blindly.  And the other thing is — it’s really important that lifelong learners, that informed citizens, figure this stuff out.  In today’s world we do need to be able to evaluate like experts, even if we aren’t going to be generating any original scholarship or research ourselves.

Maybe this is one way to get at the “how to teach peer review” question — it would take a lot longer than I usually have to talk about this issue, but it could get at the question of “why peer review” as well as the question of “have the peer reviewers done their job.”  Normally, this is what I do when asked to help students find peer reviewed articles – I explain what the process of peer review looks like and talk about some cues the students can look for to identify whether or not an article has gone through that process.

This really isn’t very different than the checklist approach I criticized a few posts ago, but it does focus on the students’ actual need, which is to use a source that has been peer reviewed.  I’ve seen a lot of those assignment sheets and they’re not usually required to identify a good scholarly source so much as they’re required to identify a scholarly source.

Given a couple of sources without the names, institutional affiliations, journal names, etc — I wonder how well they’d do coming up with a list of metrics or questions or methods that one could use to evaluate the quality of that information?  Would they think to question which research method the scholar used, or would they think to question whether or not the conclusions matched the results?  Would they think of following up the sources in the lit review?  I might be a Pollyanna, but I think at least some of them would.  And even if they didn’t – I think I’d have a much better sense of how to approach the why peer-review question with other students if I saw some approach that question in this way.