Peer-reviewed Monday – knowing stuff makes you search better

Actually, it doesn’t. But more on that later.

I might have to rethink the peer-reviewed Monday thing because sometimes there are Mondays when I just don’t have anything exciting to talk about. I knew I should have saved the doodling thing, but I just didn’t want to.

So today found me searching. And look what I found – from the psychology realm this time, published in Computers & Education.

Teena Willoughby, S. Alexandria Anderson, Eileen Wood, Julie Mueller, Craig Ross (2009). Fast searching for information on the Internet to use in a learning context: The impact of domain knowledge Computers & Education, 52 (3), 640-648 DOI: 10.1016/j.compedu.2008.11.009

I couldn’t find a pre-print (though, as Barbara pointed out a couple of weeks ago, Elsevier usually allows self-archiving.  And SHERPA/RoMEO says this particular journal allows authors to archive the post-print) so this link is just to the abstract & references.

The basic idea that knowing stuff makes you a better searcher is pretty well known in instruction circles – it’s the thing upon which we base the whole idea that people should “explore broadly before narrowing to a thesis.”  It’s a fundamental part of Kuhlthau’s Information Search Process model, and it’s a fundamental part of the work that we do with our beginning composition program.

This article, however, doesn’t really say that.  It doesn’t say that knowing stuff makes you a better searcher — in fact, it concludes that knowing stuff doesn’t seem to make you a better searcher.  And being a better searcher might not really matter.  Because knowing stuff makes you a better information-user.  If you know stuff, the authors found, you can more easily and quickly and efficiently get stuff out of what you find.  Which means you can more quickly make use of it.

A little bit about the method, especially for those of you who can’t just go read the article – the study population was 150 undergraduate students.  Half were studying science (and would thus have a high level of domain knowledge in biology) and half  were enrolled in an environmental studies program (and would thus have a high level of domain knowledge about urban environments).  All students were asked to write two essays: one about biology, and one about urban environments.  The idea was that everyone would have high domain knowledge for one essay, and low for the other.

The students were broken into four groups:

• 1 group was given basic instruction in Internet searching skills, and used the Internet for 30 minutes before writing their essays (20 science students/ 20 environmental studies students)
• 1 group was not given any instruction in Internet searching skills, and used the Internet for 30 minutes before writing their essays (20 science students/ 20 environmental studies students)
• 1 (control) group wrote the essays immediately, with no access to the Internet (20 science students/ 20 environmental studies students)
• 1 (control) group wrote the essays without using the Internet, but after taking 30 minutes of planning time (15 science students/ 15 environmental studies students)

The essays the students wrote were scored by the number of “factually correct statements or phrases… produced that directly addressed the assigned question.”  So, accuracy and relevance both went into the score but, probably obviously, things like originality, creativity, or the quality of the prose did not.

The only factor measured that played a measurable role in the quality of the students’ essays was their domain knowledge.  The best essays were those written by students with high domain knowledge and Internet access.  When students had low domain knowledge, using the Internet didn’t help – their essays were no better than those produced by the students who wrote without researching first.

More interesting for instruction librarians, teaching students search techniques did not produce better essays.  The authors suggest, interestingly, that this finding might be due to the fact that most of the students are decent searchers, or possibly that Internet search engines are good tools.

The researchers had access to the students’ Internet sessions and they suggest that most students found quality sources on the topics in question — whether or not they had searching instruction.  Similarly, most students were able to *find* decent sources no matter how much domain knowledge they had.  Those who had low levels of domain knowledge, however, apparently couldn’t do much with the sources they found, even when those sources were high-quality and relevant.

Now, of course, the situation created in this study design was very specific – the look something up really fast and then write about it scenario isn’t ideal for student learning.  We all know that.  And I was impressed with the care that the authors took to say that their results did not suggest that novice learners shouldn’t use the Internet.  Instead,

On the contrary, similar to learning from other sources, retrieving information from the Internet and then using that information in a learning context may need to be supported in novices. For example, novices may need to be provided with extra time for searching the Internet (see Desjarlais and Willoughby (2007) for a study supporting that suggestion). In addition, learners may need additional scaffolding such as strategic interventions to facilitate encoding and retrieval of the novel information ([Kuhn et al., 1988] and [Willoughby et al., 1999]).

What’s interesting about this finding to me is in the idea that most students can pretty much handle the searching side of things, especially on the Internet.  This certainly backs up my own anecdotal sense of things, and I’ve heard other instruction librarians suggest the same thing.  I don’t always feel that I have a ton of success convincing others – faculty, or people who work a lot on academic success-related programs – that the finding isn’t the problem though.  Something helping me to argue that my teaching them to find “the good stuff” won’t help all by itself – I like that.

I’m not saying that students don’t need help searching – that they can find the best sources without training.  Of course, I’ve written before about my skepticism about the whole best source concept.  I also don’t think they can search comprehensively without training.  But how often do undergraduates really need to do that?  Anyway, knowing that is harder for them to organize, contextualize, analyze and remember new information when their domain knowledge is low – that’s valuable.  Breaking down some of the assumptions that if they’re not using the good stuff it’s not inherently because they are too lazy or incompetent to find the good stuff – that’s valuable too.