Peer Review 2.0 (2.0) – Ontario Library Association Superconference
January 27th, 2009 § 2 Comments
Peer Review 2.0: Tomorrow’s Scholarship for Today’s Students
Kate Gronemyer & Anne-Marie Deitering
Ontario Library Association Super Conference
January 30, 2009. Toronto, Ontario, Canada.
What is 2.0?
Five Web 2.0 themes — from the ACRL Instruction Section’s Current Issues Discussion Forum, Research Instruction in a Web 2.0 World (Annual, 2006).
We need to deal with the strengths and weaknesses of peer review
Jabberwiki: The Educational Response, Part II (Michael Gorman, Britannica Blog, 6/26/2007)
Peer-to-Peer – a nature.com blog “for peer-reviewers and about the peer-review process.”
Uses of Journals in Scholarly Communities
Ann C. Schaffner (December 1994). “The Future of Scientific Journals: Lessons from the Past.” Information Technology and Libraries, 13(4), pp. 239-247.
Creating an archive of knowledge
PhD comics – Your (real) Impact Factor (12/8/2008)
Citation Systems for New Media (MediaCommons, June 19, 2008) – the issue of measuring impact factor in digital scholarship
Thomas Kuhn (Stanford Encyclopedia of Philosophy)
The Structure of Scientific Revolutions (Kuhn, 1962)
Coping with Peer Rejection (Nature 425, 645. 16 October 2003)
Building a community of scholars
Twitter for Academia (AcademHack, January 23, 2008) – see “track a conference”
- #alamw09 (ALA Midwinter 2009)
- #AHA (American Historical Association)
- #eli09 (EDUCAUSE Learning Initiative)
- #sparc08 (SPARC Digital Repositories Meeting 2008)
- #mla08, #mla140 (Modern Language Association 2008)
MLA Goes Tweet (Scott McLemee, Inside Higher Ed, (12/29/08)
Guaranteeing quality control
In the Matter of J Hendrik Schon (physicsworld.com, 11/01/02)
Retraction (Letters, Science, 11/01/02)
Fiona Godlee, Catharine Gale & Christopher Martyn (1998), “Effect on the Quality of Peer Review of Blinding Reviewers and Asking Them to Sign Their Reports,” JAMA, 280, 237-240. (Link to abstract)
The Purpose of Peer Review (Dorothea Salo, Caveat Lector, 9/3/2006)
Distributing rewards
A few examples of faculty & researchers writing about their experiences with the process:
What Professors Want from Editors and Peer Reviewers (Inside Higher Ed, October 2, 2008).
Peer Review or Smear Review? Reflections on a Rigged System (Historiann, May 2, 2008).
Rescind to Sender (FemaleScienceProfessor, July 1, 2008)
So, what happens if we ignore all this?
Peer Review Education Resource, from Sense About Science. This is a British project designed to help teachers teach students about peer review.
Open Peer Review (which is not the same as open access)
A great discussion of open peer review and a summary of different models for it can be found here -
Mind the Gap: Peer Review Opens Up (Sara Kearns, Talking in the Library, June 24, 2008).
Nature experiments with open peer review – Nature Peer Review Trial and Debate (December 2006).
Models:
BMJ – be sure to read all the Rapid Responses
Connexions lenses – educational materials, not research, but an interesting model.
Expressive Processing, an experiment in blog-based peer review
Classic Science Papers: The 2008 “Challenge” ! (Skulls in the Stars, May 2008). Science bloggers analyze and explain the significance of key papers in their fields in a way accessible to non-scientists.
Open Notebook Science Using Blogs and Wikis (Jean-Claude Bradley). Preprint at Nature Precedings.
freakin’ ALA
January 26th, 2009 § Leave a Comment
Peer-reviewed Monday is going to become Peer-Reviewed Tuesday for this week only.
I’m on what can best be termed a layover at home between two big and crazy conferences — this one, where I was committee-meeting girl and this one, where Kate and I are giving a presentation.
I have the article, I have the will. But I’m too tired to finish writing it up. I don’t have the brain. I will say this, though. There are all kinds of articles tht I think are worth writing about – the good, the bad, and the okay-pretty-bad-but-there’s-this-one-interesting-concept. This one is in another category – the kind of thinking that makes me a fangirl category.
Peer Reviewed Monday – Re-Thinking Information Literacy
January 20th, 2009 § Leave a Comment
The article that starts off this post is this 2004 article from the Journal of Librarianship and Information Science, but it’s behind a paywall, so I’m just going to talk about it briefly and focus instead on this 2007 paper which reports on, and updates, the same research.
So I was motivated to read these because they fit well into some larger questions I’ve been having about the way we think about what we know in libraries – and about the intersections of theory and practice. I didn’t think the 2004 piece really lived up to the promise in the abstract, though it did spark some new thoughts. The 2007 update I liked better – perhaps just because its focus is different. The earlier piece is un-focused, arguing for a new definition or understanding of the concept of information literacy without ever getting very specific (or maybe just without being very convincing about the specifics) about what that might mean for practice. The later piece is clearer about what it might mean for a particular form of practice — research.
Quick summary – librarians need to expand their understanding of what information literacy is and what it means to be “information literate.” She argues that teachers and librarians understand information literacy in a particular context, that of formal education, and that that skews their understanding of what should be a much broader concept. Particularly in the 2004 paper, she spends time building a case that librarians look at information literacy as a set of master-able, measurable skills, that they believe those skills are generic and can be translated into entirely different contexts without any specialized understanding of those contexts. Most importantly, she argues that the skill sets that make up librarian-style IL are way too focused on text – on finding texts, accessing texts, understanding texts, and showing one’s understanding of texts by producing texts.
some rights reserved by xymox (flickr)
Lloyd uses constructionist grounded theory methodology in her research, looking not at students but at people engaged in work. Her first study examined the information literacy practices of 14 firefighters and the second did the same with 15 ambulance workers. In both, she finds that:
- information literacy is context-specific, that it cannot be understood without understanding the discourse and the discursive practice of different communities of practice
- there are different information modalities related to different kinds of knowing and learning: epistemic, corporeal and social. I’ll talk about this a bit more below, but the basic idea is this – that text-based information is important for workers at the training level, when they are learning how to act. That gives way quickly to more experiential forms of information and knowing – as the novice learns, in part by acting, to be a practitioner.
- we need to remember power
She argues that we need to understand how we connect with information holistically, not just with text and not just as individuals. She argues that a holistic understanding of information literacy will make all of those connections we talk about between information literacy and lifelong learning meaningful.
I have some issues with the earlier article – on the one level, I agree with a lot of what she says about how we conceptualize, and particularly assess, information literaccy in higher education. But then she goes on to assert that “many librarians” believe that “library skills sessions, computer literacy and referencing sessions constitute the whole information literacy concept.” I’m sure there are some libarians who think that, but I don’t know them. Or at least, I don’t know them well.
And back to that question of assessment. That was one of the things I really wanted to get out of this article. I have been thinking a lot lately about how we know what we know in libraries, particularly when it comes to what we teach when we teach – what we should be teaching and what our students need to learn. I’ve been wondering if we don’t let other disciplines define what we know too much, and specifically if we don’t let the discourse about assessment, outcomes and standards define what we know way too much. So I was interested to read something connecting these ideas here. But at the end of the day, too much of the Lloyd’s criticism here seems to reduce to the idea that our understanding of information literacy in libraries focused too much on text.
Especially when the main call to action at the end of the article seems to be this -
We need to develop ways of exploring information literacy which will enable the development of a broader concept of information literacy which addresses the learning concerns of education, workplace and community and to construct a generic framework of skills (relating to textual, social and body learning) and competencies. These may then be taught across these settings and facilitate the development for skills and competencies which will enable and enrich learning throughout life.
I’m not seeing this – I’m not sure why the answer is to do this deeply contextualized research and then stick with the same old generic, skills-and-competency focused approach to information literacy.
But I did find some good things to think about in this article, most of which are continued in the later piece. First, I think the idea that novices learn to act using some information modalities and that they learn to be practitioners using others is very interesting, and connects to the idea Caleb and I were discussing the other day about communities of scholars and existing online communities of scholars.
Lloyd argues that novices learn policies, procedures in an objective, abstract, ideal way when they are first becoming part of a community of practice. They mostly connect and learn from texts at this point, and in the process of learning they develop a subjective understanding of themselves in relation to the officially sanctioned practicies and defined policies of the institution or community.
When they actually start doing the job, actually begin to act as practitioners, the abstract or ideal knowing from texts comes up against the knowing they develop by doing. Sometimes the two mesh, sometimes they conflict. Here they are beginning to learn how to be a practitioner by doing, and they are building up a body of knowing they can share with the group. This type of learning is very context-dependent. They begin to connect to the community of practice at this point.
There’s one more kind of learning/knowing to add to what they learn by doing themselves. As they become a practitioner and part of the community of practice, they learn where they can go within the community for that tacit, contextualized, social knowledge that the community holds.
There’s an interesting tension here when it comes to the kind of information literacy people need and learn in school, I think. Lloyd argues that in the academic context, textual knowledge is valued above other forms of knowledge so that is what librarians and educators focus on. So on one level, you could argue that we are helping students learn to act, as scholars but more than that, helping them learn to act in their fields of study. The future teacher accesses texts about sanctioned classroom practice.
On the other hand, scholars themselves access more forms of knowledge than just the textual. There’s a lot of research out there suggesting that scholars prefer informal information sources — accessing social networks of experts and resources and that they develop these networks and the skills to access them over time. There’s research saying that senior scholars access those informal networks more often, while junior scholars use search-based methods of inquiry more frequently.
This all has some fascinating implications for what we do when we teach information literacy. It suggests that just saying “peer review = better” is not good enough. Not that we ever thought it was, but it suggests a different twist on why it’s not good enough. It suggests that we aren’t teaching a set of skills, but instead teaching students to identify the information sources and types that are accepted and valued in a discourse community. So just like senior firefighter telling junior firefighter, “this is the information that real firefighters trust,” we help students understand how to connect to the information that real scholars trust.” Pretending that the peer review thing has inherent value beyond the value that scholars and researchers afford it isn’t good enough.
Which pushes the focus away from the peer review and towards the scholarship. If we are trying to convince students of anything it shouldn’t be that double-blind peer review is the gold standard or objectively awesome. It should be that scholarly inquiry and the research that inquiry produces has value. And value beyond school. Helping students develop the skills and understanding they need to be able to connect those scholarly sources to their out-of-school information needs – that’s kind of exciting to think about.
Lloyd’s research suggests why it’s not automatic, why people don’t go “this peer reviewed study helped me get an A, clearly it will help me in this entirely different context over here.” It also suggests that my intial question – understanding how instruction librarians know what we know – has value as well. At least, it does if I want to be an information literate information literacy librarian.
**********************************
A. Lloyd (2005). Information literacy: Different contexts, different concepts, different truths? Journal of Librarianship and Information Science, 37 (2), 82-88 DOI: 10.1177/0961000605055355
Lloyd, A. (2007). “Recasting information literacy as sociocultural practice: Implications for library and information science researchers” Information Research, 12(4) paper colis34. [Available at http://InformationR.net/ir/12-4/colis34.html]
DIY research
January 17th, 2009 § 5 Comments
Not learning to do stuff from tutorials, though that’s where this started, but more thinking about tutorials by looking at tutorials.
It has been a couple of weeks, but I finally had some time to take a look at the backlog in my “arts and crafts” folder in Google Reader and one of the things I found there was this Top 100 Tutorials of 2008 post at The Long Thread blog.
Rachel and I have talked before about crafty or DIY tutorials because we both like to do crafty and DIY things so we run across a lot of them. And with Karen we talked about crafty tutorials in the context of library tutorials for a bit.
But this Long Thread post got me thinking about that topic again – because I’ve been thinking about tutorials lately in my job sometimes in ways that I find fun, and interesting and sometimes in ways that just make me tired.
A lot about library tutorials makes me tired. I get tired because the process of making them and maintaining them can quickly get so big. I get tired because Camtasia doesn’t work on my Mac, And I get tired using Camtasia and Captivate anyway. I get tired because you can spend all this time making them and then you’re still left with the even more complicated question of how to get people to actually use them.
So I think it’s actually interesting to look at this crafty community, to look at DIY tutorials and think about what tutorials mean in a context where the basic assumption is that people do want to learn how to do this stuff, that they are interested, that they do want to do some of the work themselves. I have looked at almost all of the tutorials on this list, though I won’t claim that I’ve seen every one of the 100 and there are some common threads that are interesting to think about –
(yeah, I didn’t really mean to say common threads there)
#1. They’re kind of at the point of need. They’re kind of not.
There’s no discussion here about how to get these tutorials into someone’s knitting bag or onto someone’s sewing table. The expectation is that people will find them in their normal information flow, that they will get pointed to them by crafty friends, or on blogs they follow. Or, that they will find them on the web.
The former thing, I think about a lot. Without a lot of success. The latter thing, though, I think is something to think about – how searchable are our tutorials? Do our students even think to use Google as a strategy when they don’t know how to do something? I’m really asking – is that how they go about answering those questions? When I need to know how to do something that’s not clear in the directions I have, whether it’s a tool thing, or a software thing, or a cooking thing, my first step is usually to search (Google Reader, Google or delicious) for an answer. I would imagine that is not a generational thing, but I don’t know. Are there students out there Googling “how do I find peer-reviewed journal articles?” or “how do I find newspaper editorials?”
(plus side, by putting those phrases in this post, if they are searching for those things I should start seeing similar phrases in my referral logs soon)
And if they are, what are they finding? Are our tutorials and modules and how-to’s findable? Let’s see. Trying search strings instead of keywords – if I try:
“how do i find op ed pieces using lexis-nexis”
Two of the top four results are from Lexis-Nexis itself. Includng this one which is #1 and right on point. Not surprising, and not bad. The first library result is high, at #3. It’s from Duke and it’s a more general page about finding periodicals online.
But I don’t think many students would actually phrase a search in this way. Maybe if they could cut and paste from their assignment guidelines and their assignment guidelines pointed them to Lexis-Nexis. So how about if we make it more general - “lexis-nexis” becomes “newspaper databases” and “op-ed pieces” becomes “editorials.”
“how do i find editorials in newspaper databases”
This time, a page from UT-Austin comes in at #2, and it is quite useful. There are two other library pages in the top 5 – UNLV at #4 and Long Island University at #3, but in both cases the page in question is simply a list of newspaper databases.
Still, this search requires the user to be pointed to databases, or to know that these sources are likely to be found in newspaper databases. Here’s the search I actually think is most likely:
“how do i find editorials in newspapers”
Nothing comes up here. The whole first page of results is pages explaining what editorials are, or how to write them.
I have more examples looking for how-tos on finding peer reviewed articles or citing sources, but they don’t really suggest anything different. I’m putting the searchability question on the list of stuff to think about more. That seems to be important on a couple of levels – if people are already using this strategy to find out how to do stuff, and we’re not findable there that’s an issue. If our students aren’t using this strategy, they should be. But we should make sure they’ll find the stuff we make there first.
#2. They’re all about how to make something.
This might seem obvious but a lot of our tutorials aren’t about how to make something. They’re about how to do something that will then let you make something. So this is both a “how these tutorials are different than ours so we should be careful about drawing a lot of parallels” – and a “maybe there’s another way to think about our tutorials”part.
There are tons and tons of “how to do this thing” tutorials out there in the world too. The fact that none of them are on this list means that the list is really more about the products that you can make with the tutorials, not the quality of the actual tutorials. But I think it’s worth thinking about how we conceptualize and present some of our tutorials as well – can we identify some things that our users want to make/ need to make and present the tutorials as a “how to” in that way?
Which connects to -
#3. These usually assume some knowledge on the part of the user.
In other words, these aren’t “how to sew a skirt if you’ve never seen thread” tutorials. They are tutorials on how to make stuff for people who already know how to make other stuff using some of the same techniques.
A really extreme example of this is in this tutorial to make these:
The full set of instructions for these bookmarks is this:
I glued little craft floss hair-dos on them and then stuck pipe cleaners in their heads, used more glue to make a felt pipe cleaner sandwich and then whipstitched around the edge.
Now, that is kind of excessively brief, but most of these tutorials assume they can give directions like “whipstitch” or “use your zipper foot” or “use a long-tail cast-on” or “straight stitch” without having to explain every one of those terms.
Which is something I think we can think a lot more about in libraries. On a couple of levels.
First, is the letting stuff go level. I think we do have a tendency to think that the tutorials we create have to be complete and comprehensive. Or maybe this is just in my library. But the truth is that even though we usually pull back from it, in our conversations about the smallest learning objects we initially start having conversations that do this – “but to do X they will need to know A, B. and C.” There’s a couple of linked assumptions there – that they will never click on or search for more help at this moment, and that they might never come across our help again after this one time. I think we mostly know now that we have to let stuff go so that we can focus on real learning of the stuff we do have time to teach or cover. But it’s hard.
On another level, though, our students do spend a lot of time finding stuff using online tools. They do have transferable skills – we can assume they know some stuff.
Back to the craft people, they are also on the Internet. So even if you use an instruction like “whipstitch” and Annie the Sewing Newbie doesn’t know what that means, she’s on the Internet and and Google will find something that will tell her what it is/ show her how to do it. Which is another reason why the concept of making our tutorials as findable as possible is interesting.
Which leads to –
#4 – They are presented using social tools
Almost every single one of these tutorials is a simple combination of blog post + pictures. No fancy video, or branching, or audio involved. I do think this relates to the “people want to use this stuff” assumption that these tutorial designers have. They are not focused on building in cool bells and whistles to engage the users because they can assume their users are already engaged. So they can use the power of today’s social tools to get stuff up there fast.
But even the tutorials that are presented as PDF files are usually delivered via a blog post. Which means that the people using the tutorials can ask questions. So you don’t have to explain every step as if the person will never get any other help ever – they can ask you for it, or they can get that help from the other people who have used the tutorials.
For example in the comments to this tutorial, the comments include several from other other crafters answering a question that the original poster had about the project.
Beyond this, many of these projects develop a second life on Flickr – where a lot of different people can show what they did with the basic concept suggested in the tutorial. This also has implications for library tutorials, I think. Given how complex and dynamic and personal research is, the idea that other users can show how a different application of a basic concept can lead to different resultsc could have a great deal of utility.
As an example of this, in this tutorial (PDF) about making a fabric-covered charging box for your devices, there is a note to add final projects to the author’s flickr group.
Which connects still further to this -
#5 – There’s value added. They do some of the work for you.
So these tutorials are made for people who already know how to do some stuff, and really, there are a lot of tutorials out there that seem kind of obvious. Even on this list of the top 100 there is this tutorial for adding patterned paper to a clear iPod case that seems like it defies the need for instruction. But there are two things to keep in mind here. One is that sometimes all we need is the idea – but that doesn’t mean that it’s not worth telling people about the idea.
The other is that most of these tutorials add some value by doing some of the more fiddly work of crafting for you. There are a lot of tutorials here where I could figure out how to make the thing myself, but it is really nice when someone else has already done all of the necessary math for me:
Hobo lunch bag tutorial (Sewing Notions)
I don’t know how to apply this to libraries, but I think there has got to be some things out there where we could present tutorials on – here’s how you make this thing, and we’ve already figured out some of the fiddly parts for you.
Anyway, it’s not surprising that -
#6 – a lot of the time, they’re marketing tools
The Purl Bee is a great example of this. This is an upscale fabric and other crafts store in NYC. There are tons of useful, cute tutorials on their store website – all helpfully linking users to the materials they will need to make the things in the tutorials, which are all available from… The Purl Bee.
For this pattern for a Quick Bias Tape bib, there are direct links for four different items that you will need to make the (adorable and easy) bib.
For everyone like me who sees a pattern on the Purl Bee site and thinks “wow that is exactly what I need to make something out of that piece of fabric I bought in San Francsico in the 1990′s” there are probably a whole lot more who think “must buy fabric from Purl Bee now.”
That’s something I think we could definitely do – create things we could link from our homepage that tell people how to do things we know they want to to do.
However –
# 7 – These are created in the context of an existing community
Even if we have blogs, I don’t think we have a lot of readers. Even though we have a ton of visitors to our web sites, I don’t think we have a lot of people tracking the changes on those sites. Relying on our students to point each other to our tutorials seems unrealistic no matter how useful they are. This one bears more thinking about.
Now, I have big plans to go make reusable fabric versions of all of the gift bags I will need for the next year. Unfortunately (or luckily) obligations at Midwinter may prevent obsessive crafting from happening, at least immediately.
Kicking off Peer Reviewed Mondays
January 12th, 2009 § 1 Comment
I am apparently not the only theory geek out there. But I realized that I haven’t been doing a great job of putting my money where my mouth is where it comes to the value of peer-reviewed articles. So my Monday evenings this term look like they are, for a variety of reasons, going to lend themselves to this experiment – Peer Reviewed Mondays.
Which means – Mondays I’ll try to at least point to a peer-reviewed article that I think is worth talking about. I’ll also try to explain why. That seems do-able.
So I’ve had the idea for this for a while, which of course means that I haven’t come across anything peer reviewed that really inspired me to write. But today, I was inspired by a presentation I heard on campus to go poking around into journals I haven’t looked at before and I came across this article (PDF) from 2004, which takes a different kind of look at issues of evaluation when it comes to science and research and peer reviewed papers. The meta of that delights me, and I think this paper is a useful addition to the stuff in my brain.
The article is called The Rhetorical Situation of the Scientific Paper and the “Appearance” of Objectivity.
(Useless aside – when the presenter today was being introduced I was unreasonably amused because to fully express the meaning of each journal article, the professor doing the introducing had to read out all of the punctuation – like the quotation marks above. That doesn’t happen in library papers enough. )
Allen’s method is simple. He articulates ideas and theories from the rhetoric/composition discourse, and then analyzes a single scholarly article -
in this essay, I analyze Renske Wassenberg, Jeffrey E. Max, Scott D. Lindgren, and Amy Schatz’s article, “Sustained Attention in Children and Adolescents after Traumatic Brain Injury: Relation to Severity of Injury, Adaptive Functioning, ADHD and Social Background” (herein referred to as “Sustained Attention in Children and Adolescents”), recently published in Brain Injury, to illustrate that the writer of an experimental report in effect creates an exigence and then addresses it through rhetorical strategies that contribute to the appearance of objectivity.
Allen’s initial discussions, that scientific rhetoric is rhetoric, that scholarly objectivity has limits, and that specific rhetorical decisions (like the use, or what might be considered overuse in any other context, of the passive voice) are employed to enhance the impression of objectivity. are interesting but not earth-shattering. Where I found the nuggets of real interest were in the concluding sections. Allen draws upon John Swales, who examined closely how scientists establish the idea that their research question is important.
Swales called the strategy Create a Research Space (CARS) and identified three main rhetorical moves that most scholars make. As Allen describes them:
- Establish centrality within the research.
- Establish a niche within the research.
- Occupy the niche.
So what’s interesting about this to me, is Allen’s conclusion that – scientific authors seek to establish their study’s relevance through the implication that they share the same central assumptions and information base.
The reason this is interesting is that it dovetails neatly with Thomas Kuhn’s idea of normal science – the idea that a shared set of first principles is central to the idea of scholarly discourse, particularly a discourse that advances knowledge. Where that intersects with the idea of peer review is in the idea of what makes it through peer review – the idea that an article or a piece of research needs to be more than just good or interesting research, it also needs to be a good example of what research is in a particular discipline, or discourse community.
Allen compares the relatively ordinary piece of research that he is anaylzing with a far more influential piece and finds that some of the rhetorical strategies that are used to establish the science-ness and the objectivity of the former are present, but far less present in the latter. This – especially the idea that authors proposing truly ground-breaking research might be less likely to use the passive, “objective” voice – might be more likely to refer to themselves as active agents – is simply fascinating.
Allen points out that the very language we use to talk about scholarly research – creating meaning, knowledge, verifying truth claims – implies that the situation in which scientists communicate their findings is rhetorical. He also points out that it is not just scientists – but the rest of us who rely in many ways on the meaning and knowledge scientists create – who need to understand these rhetorical practices. His last sentence, in fact, is as good a justification for information literacy in higher ed as I’ve seen:
Certainly, scientists and researchers should be aware of embedded rhetorical strategies. But given the profound and pervasive influence of science in Western culture, we should all––scientist or not––be attentive to how our knowledge is shaped.
And now on to that delicious meta I love so much.
I love the idea of using this article to talk about what scholarship really is with undergraduates, particularly undergrads working on understanding academic writing. But here’s the thing – the author of this article is himself an undergraduate writer.
Seriously!
The article appears in Young Scholars in Writing: Undergraduate Research in Writing and Rhetoric – a peer-reviewed journal, with an editorial board made up of faculty in rhetoric and composition. The content of the journal is all by undergraduate researchers, and the peer reviewers themselves are undergraduates who have also published in the journal.
The journal reflects some of the practices of open peer review that fascinate me – especially as information literacy resources for students as they learn the practice of scholarly knowledge creation. The review process itself is not made public, but each issue of the journal has a Comments and Responses section where student writers write 2-5 page responses to papers that have been published in the journal.
Which gets to the last piece of meta. The one overwhelming benefit of the peer-review process, that is rarely discussed by us librarians in information literacy classes when we have to talk to students about finding peer reviewed articles – and that is rarely discussed by the faculty who require their students to find peer reviewed articles — the one thing that is pretty much unanimously agreed is that the process of peer review makes the paper better. Maybe not all better, maybe not better in the same way as it would be better if it were reviewed by other people. But better than it would have been had it not gone through that process. This paper is beautifully written – clear and accessible and smart.
I love the idea of digging into the idea of peer review, using student engagement with the peer review process as an entry point — but to be able to do that with a paper that itself should spark new ideas about the value of scholarship, and how to evaluate scholarship – that looks kind of like a gift.
*************
Matthew Allen (2004). The Rhetorical Situation of the Scientific Paper and the “Appearance” of Objectivity Young Scholars in Writing: Undergraduate Research in Writing and Rhetoric, 2
Comments
January 8th, 2009 § 1 Comment
I was naively hoping I would never need to do this, but I decided I probably should do so before I really needed to – I added a brief set of guidelines about comments on my about page.
citation humor FTW
January 7th, 2009 § Leave a Comment
(that’s “for the win”)
(via CrookedTimber) The PostModern Language Association’s new rulings on
alternative source citations – necessary because the MLA doesn’t tell you how to cite tattoos or Magic 8-balls.
So how about that Standard One
January 6th, 2009 § Leave a Comment
According to the ACRL Instruction Section’s Information Literacy Competency Standards for Higher Education (ACRL/IS ILCSHE for short) the first step to information literacy goes like this: the information literate student determines the nature and extent of the information needed.
(Or, as it is usually described in the first-year courses I teach – choosing a topic.)
(The reason this clip is here – starts about 6 minutes in. Remember the “Trask… radio … Trask … radio” scene?)
In the December 19th Chronicle (and again with behind the paywall) there’s a great little essay by Robert Hampel on how scholars choose their research topics. I think there’s a lot in this short article for librarians to think about. Hampel says that for years he advised his graduate students to find their topics this way – “Fill a gap in the literature, identify a problem that has not been studied adequately, and add a brick to the wall of knowledge.”
That really resonated with me, because that is exactly how I was told to come up with topics, from my first can-you-cut-it-as-a-grad-student seminar class to my thesis to my (nonexistent) dissertation. What I like about this article is that Hampel started questioning that advice, thinking that the gap in the literature might provide an answer to why a research topic is worth doing, but on a process-level – that is not how most scholars identify their research topics.
By talking to a bunch of colleagues with years of research, he found that while they had a lot of different ways of generating the good research topic ideas – none of them used the browse the literature and look for gaps strategy. That totally makes sense to me on a process level, that doesn’t seem like a way to spark new ideas. Hampel grouped what he found into four main categories:
- New research comes from your old research – you’re never really done with a topic. Every advance in understanding sparks new questions.
- Research questions come out of your life as you live it. He calls this both “autobiographical” and “me-search.”
- Vygotsky! Or, more precisely, the idea that by talking with others about research you come up with questions you would never come up with on your own.
- Topics that get you funding – or, more broadly, someone else says “a better understanding of topics X, Y, and Z is something that is valuable to me.”
I think these ideas, directed as they are at those who teach the future professoriate have resonance all the way down the academic line to the newest researchers in the picture. It’d be a pretty rare first-year student who could use potential grant money as an incentive to choose a research topic, but any of the other strategies could be valuable ways for them to think about the choosing a topic step in the research process.
How often are undergraduates given the chance to meaningfully revisit a research topic? We frequently encourage them to “choose a topic you’re interested in,” but does that translate into research that solves actual life problems? And how often, do we build in social and collaborative work on the topic-generating stage?
I do think that each of these ideas is potentially valuable, but it would need to be dealt with much more substantively than “revisit this other paper” or “choose something you’re interested in.” After all, no one wants students to turn the same paper in over and over, or to rewrite the same ideas. We want students to choose topics they’re interested in, but sometimes the more interest they have before writing the paper, the harder it is for them to approach their research with a true disposition to critical thinking.
Hampel addresses this last point elegantly, I think – in talking about the academics he knows who draw their inspiration from their own lives he says that “[f]or those colleagues, their lives are inspiration, but not evidence.”
In the last term, I had the chance to work with two writing instructors at OSU who spent serious, intentional time talking about topic generation in a way that inspired a variety of students to choose topics they were genuinely curious about. I do not know if that radically changed the quality of the final papers, but I do know that it improved the quality of those students’ questions about and engagement with the research process.
Female Science Professor asks – is this kind of thing even something we can teach? We’re really talking here about a disposition towards curiosity, and is that even teach-able? I’m not sure that I don’t agree with her about that. But two points I think are important here -
The first is that we already do teach it – from first-years to grad students we already tell them how to find their research topics. We might not, and we probably do not, do so at length or intentionally – but it’s a part of teaching people how to research and how to write about research. Students ask how to do it, students have trouble doing it – so we teach it. So that’s reason enough to be reflective about how we do it.
The second is from Hampel’s own conclusion :
If graduate students cannot see how senior scholars generate and manage their ideas, then their induction is incomplete. Our students dutifully take research-methods courses, but every graduate seminar should discuss the wide range of sources of creative work. Otherwise our students will think in terms of the assignments we give them, when they should really be thinking about the assignments they can give themselves: interesting topics for future study.
I like how he doesn’t come down on the side of telling people how to do it, but instead he advocates making room to discuss the many, many ways of doing it. The sharing piece of this idea is important I think – in that Vygotskian way mentioned above.
There is more sharing about how this is done in the comments section at the Chronicle, and around the web:
- Thus Spake Zuska: Good topics for future research and how you find them – especially the comments!
- Critical Pedagogy: Generating ideas – provides more citations on this topic
But what I think is most important from that quotation is the first line – seeing how “senior scholars generate and manage their ideas.” This resonates with me not just for choosing a topic, but for all of those how to use, or even just appreciate, scholarship things that we want to teach about. And this is where I think we definitely want to take the “graduate” modifier out, and just say “students.” Our undergraduates also need to see what it is their professors do, what they like, what ideas they value, and how they use them.
discovery and creation and… lies!
January 3rd, 2009 § 8 Comments
I’ve never really understood the whole pirate thing. Talk like a pirate day can come and go without my noticing, and despite the presence of Johnny Depp, I didn’t make it through the whole Pirates of the Caribbean trilogy.
So even if I had seen the mentions of the Last American Pirate hoax on the blogs I read all the time, I’m not sure that I would have bothered to follow the links. But maybe I would have. This story does combine two of my favorite things – scholarly uses of social media and history. Still, amidst holiday preparations and Oregon-style snowapocalypses I totally missed the initial stories on the topic.
Which is relevant in that I’m not a disgruntled blog reader feeling taken in. I was not personally hurt in any way by the deliberate historical hoax created by the students of History 389 at George Mason University last term.
And yet. I keep thinking about it and I’m not sure I can really articulate why.
So, quick recap. Professor Mills Kelly of eduwired.org fame taught a class on historical hoaxes last term. Early in the term, he gave advance notice that his class would be perpetuating their own historical hoax. The class created a fake story about the search for the Last American Pirate, a guy named Edward Owens. The search was chronicled by fake student Jane on this fake blog, discussed in these fake interviews on YouTube, and finally reported as fact in this fake Wikipedia article. Some people were taken in by said hoax, most notably a pop culture blogger at USA Today. Kelly reportedly pulled the plug on the hoax when some of his real-world colleagues were taken in and the whole thing was revealed in the December 19th Chronicle of Higher Education in an article only found behind the paywall.
So why do I keep thinking about it? There has been a fair amount of discussion about it, some I really like. Some talking about things I really don’t care about. There are some people that love the experiment. I’m not really moved by any of those arguments. They seem to be mostly focused on the idea that kids today can’t get into traditional historical research, so this is a good, creative alternative.
The criticisms i find most compelling are found here, where Michael Feldstein explains why vandalising Wikipedia for the sake of a lesson is uncool and here in the comments on Dr. Kelly’s reveal post. Commenter Martha, in particular, talks about the impact of this kind of project on trust networks. Given that trust networks are, I think, a crucial part of meaningful information evaluation on the social web and thus a tool any information literate student should know how to use in this context, an assignment that deliberately devalues and damages those networks strikes me as problematic, even if there is some small benefit on the cautionary tale scale.
But that’s not what I keep thinking about except in a tangential way. No, what’s got me thinking is what this project means for teaching information literacy and research — first in terms of the evaluation skills that are an overt, intended outcome articulated in the syllabus but also, and more deeply, in terms of research itself – why we do it and why we want students to do it. These are, I suspect, related, but I’m not sure how. Maybe if I write about it they’ll come together. Maybe this will be in two parts.
Dr. Kelly says at the top that he is hoping for an information-literacy, information evaluation benefit to this assignment.
I’m hoping that this will mean that my students dig in and do some excellent historical research. I’m also hoping that they’ll learn a number of technical skills, will learn to work in a group, and will develop greater “information literacy” as we like to call it here. And, of course, I’m hoping they’ll have fun.
Specifically (from the syllabus – opens in PDF):
I do have some specific learning goals for this course. I hope that you’ll improve your research and analytical skills and that you’ll become a much better consumer of historical information. I hope you’ll become more skeptical without becoming too skeptical for your own good. I hope you’ll learn some new skills in the digital realm that can translate to other courses you take or to your eventual career. And, I hope you’ll be at least a little sneakier than you were before you started the course.
So the quick issue I have with this is that I just don’t see where the information literacy skills here translate into what most students need in their real work with online information sources. Increasingly, I just think that a focus on deliberate hoaxes isn’t a very good way to teach students how to evaluate information.
Now I get that the work done to create the hoax might give the students in this class a greater appreciation for stuff that could make them more information literate, and that knowing specifically what they did to create a fake site might give them some stuff to look for in other sites, but I don’t really see the larger benefit here beyond the reminder that stuff on the Internet can be fake and I honestly don’t think that our students don’t know that full well already.
Because here’s the first thing – helping students learn that there is stuff on the wild, wild web that was put there just to trick them, to punk them or to prank them – well, there’s not a lot of value in that. The punker or the pranker will either be really good at it, in which case all of the abstract stuff we might teach them about how to identify bad information won’t help them because the good pranker isn’t going to do any of that stuff. Or, and this is more likely, the prank won’t be all that good. And our students – I really think they’re very able to identify the obvious crap that exists online.
They don’t need help identifying stuff that is fake or wrong just for the sake of being fake or wrong because there’s not a ton of stuff like that out there. Honestly, our ability to identify stuff that exists for no other reason than to trick us is not a real-world problem that keeps me up at night. Most people who put fake or wrong or misleading information out there on the Internet have an agenda beyond April Fool’s – they’re trying to do more than trick us and what our students need is help identifying those agendas. They need help identifying the information that isn’t flat out lies, but that is a particular kind of truth.
There’s not a lot of historical information TO evaluate on the pieces of this hoax that are available to the public – the blog talks a a lot (I mean, a LOT) about how painful and difficult research in archives and mircofilm collections is – but the details about the sources themselves are pretty light. Most sources are presented as transcripts (“once I found the articles, there was no way to get a copy of them, apparently the machine is broken, so I had to transcribe them by hand,” that kind of thing). The main thing that is presented as a digitized image is a will, not found in any archive or collection that could be investigated further – it is from the private attic-type collection of one of Edward Owens’ “descendants.”
Very clever.
No, what we have to consider here if we are evaluating information is not the quality of the historical sources in question (for the most part). We don’t have the information to evaluate most of the fake sources, and beyond that – most historical sources in the world aren’t on blogs or YouTube so the skills that would help us evaluate them there wouldn’t necessarily translate to evaluating sources in archives. What we really have to evaluate here are the classic foci of Internet evaluation: the authority of the scholar/author herself and the nature of the digital tools used to present that scholarship. And here is where I think it is useful to return to the criticisms mentioned above – the tools we need to use to filter the social web are different than the tools of historical scholarship – and this project made those tools less useful for the rest of us.
Yes, we should remember that our trust networks and Wikipedia pages aren’t infallible. Treating them as if they are is dumb and dangerous, of course. But not starting from the assumption that someone is willing to do all this work just to fake you out? That’s not unreasonable. Creating a hoax like this just for its own sake, after all, is not more fun than the work it takes to do it is not fun. This one took an entire class of students working for a whole term with the great big huge carrot of the GRADE as motivation, after all. When someone, or a class of someones, does deliberately put false information out there – and I’m not talking here about the fake historical documents, but the fake blog posts and tweets and comments and pointers – it makes it harder for all of us to use the skills that really do help us navigate and evaluate the social web.
I think it’s pretty significant that outside of the USA Today blogger, most of the people who got excited about this story – excited enough to blog about it – weren’t excited because of the history beyond the “that’s kind of cool” level. The excitement was about how “Jane” leveraged social media tools to present her research broadly:
Or stated more directly, after the reveal:
And I agree with all of this in theory, but in terms of this specific hoax there is still something missing to me, and it’s an important something. It’s research – and inquiry – and discovery.
I know I am only seeing a tiny portion of what is going on in this classroom – and from the syllabus just the idea that one of the goals of the class is to show that hoaxes can themselves be the topic of serious historical research, just like wars or elections, is something I find fairly awesome. I have no idea how the process of discovery was inculcated in the other projects the students did. All I have is the public pieces of the course – the blogs, the videos, and the rest.
And that’s a piece of this discussion that shouldn’t be missed. By putting this material up on the real web, on the public web, by consciously trying to get people to access and engage with this material the question of what kind of learning experience does this material provide for those of us NOT in the class is a valid one. Is our learning experience supposed to be related to information literacy as well? To history? Or is it just a clever, creative prank?
Because here’s the next thing – I don’t think that there is much of a learning experience for the rest of us in this project – at least not in terms of information literacy.
Don’t get me wrong, I value creation and creativity. I value world-building and imagination. And I don’t think those things are separate from academic research. There is definitely creativity and imagination in scholarly inquiry, in looking at sources and seeing what might have been or what could be and re-searching based on that new potential meaning. Watching a class of students using the social web to extend and communicate such a learning process would itself be valuable in that information literacy context.
And I think there’s room in that picture for fiction as well – in telling a story that you know in your bones to be a kind of truth even though you can’t prove it, at least not in a way that would be recognized as proof, epistemologically speaking. I think there are truths and stories and voices that can only be captured with fiction. So it’s not the made up or false part that gives me pause.
But in the case of this project, as it is laid out for us to see — the public pieces of this class project combine to celebrate what a truly information-literate student can do to take control of their own learning – but all the time that information literacy only exists on the surface.
This is why I have problems thinking about the pirate hoax as a great new way to talk about or teach information literacy. Because beyond the fact that I don’t think hoaxes are a great way to teach evaluation, I’m also not sure they are a great way to talk about research and scholarly creativity. At its heart, I think information literacy is inherently linked to inquiry, and discovery. It’s about the ability to learn from information – not just to find the sources worth learning from but to use that new information to change the way you understand things, and change the way you approach the next question.
“Jane” talks endlessly about the physical pain she feels as a result of days of looking at microfilm:
But, I have no idea how I am functioning right now…I can barely look at the screen without wanting to throw up, my eyes are in so much pain.
And she goes on about how frustrating it is not to find that evidence in the documents that will prove that her pirate existed:
After my failed trip to the town, I was really discouraged. I found out enough information to keep me going, but nothing really substantial. I have not gotten any closer to figuring out a name, and my trips to the library that last four hours at a time to look through the microfilm (I’m convinced I’m causing permanent damage to my eyes), have yielded absolutely no results.
But she never talks about that other kind of pain and frustration that comes with research and learning – one of the big things that makes research hard – feeling stupid, or having to question what you thought you knew before. That’s what I mean when I say “Jane’s” process is all surface-level. She never finds anything in her research that leads her in a new direction. She finds additional things she can use on the path she’s already on, but that’s not the same.
In the end, it is a lucky break that brings Jane’s process to a close. The lucky break isn’t the issue — the real issue is that at the end of the research process described in the blog she finds exactly the single document perfect right source she had been looking for from the start. The perfect right source she imagined might exist that would answer the narrow question she formulated before she even know much about her topic at all. That’s not how research usually works. You could argue that that’s not how good research ever works.
And that’s the last and main thing. At no point does Jane really engage with something that leads her to change her mind about anything, to reevaluate her process, to go back over the same ground with a new understanding or a new set of questions. It’s needle in the haystack searching she does – she has to be creative to find different ways into the haystacks but at the same time she’s not going into the haystacks to find out what’s there. She’s going in to look for that one needle that she thinks/hopes must be there.
And yes, I get that she’s pretend, but the fictional process the real class came up with does suggest that historical research is difficult and tedious and one doesn’t make the great discovery by engaging with sources in an open-minded way. If the class had been engaged in a discovery-based research process I would hope that that would have come through in their fictional avatar’s narrative. It doesn’t. There is no doubt that this group of students were truly engaged – playing with history, creating a new world and the characters to fill it.
I can’t find it now, but when I was reading about this project earlier I was struck by the description of how the topic was selected in the first place – all of the considerations were practical – not too well known, not too likely to inspire a lawsuit if the hoax was discovered, and so on. The reasons for piracy were practical as well – a topic of broad popular interest, local, not likely to be something anyone would already be an expert on, etc. They didn’t talk about discovering the space in the historical record for their hoax to exist, they talked about creating it.
And if it’s mainly about creativity, about the class’ engagement around creating this alternate reality, around engaging with each other, and about engaging with others on the social web, then I’m not sure I see the value in making it a hoax. Except that that was the topic of the rest of the class to which we were not privy. If the skills they were learning were about creativity and world-building it seems like the resulting project could have taken the form of an ARG or a similar project where those creative muscles could be flexed in the service of creating a world for the rest of us to play in, too.
