oh no – is this serious enough?

This morning, on my drive in to work I was thinking about ScienceBlogs (tagline – “the world’s largest conversation about science”) and in particular the “blogging on peer-reviewed research icon” you see on those posts.

I really like the ScienceBlogs project – the whole idea is to get experts to comment on research, but in a way that’s accessible to the general public. As a teacher and as a non-scientist myself, I love this project. Really, this might be the most important way the participatory web connects to scholarship – not in helping scholars communicate with each other, but in helping scholars communicate with the rest of us.

If you want to check it out, I’d recommend Cognitive Daily – cognitive science, just about every post is blogging on peer-reviewed research, and there’s lots and lots of connections to teaching and learning.

Connected to that is this broader effort – ResearchBlogging dot org — trying to make it easy for readers to find serious, thoughtful posts about peer-reviewed research in a whole variety of disciplines. They’re reorganizing right now, and on a kind of hiatus. When they come back, bloggers will be able to register their blogs that (at least sometimes) analyze peer reviewed research. And library science is one of the subtopics (under the main topic Research/Scholarship) bloggers can claim. They mean to aggregate such posts, provide a registry of blogs dealing with research-related topics, and they provide an icon bloggers can use to mark their analytical, research-focused posts.

And that’s what I was thinking about this morning – we should be using that icon when we talk about peer-reviewed research from our field. At least I should be using that icon, right? For posts like this one, and this one over at ⌘f.

But then I started reading the conversations over at researchblogging.org and I started to get nervous! Is the peer-reviewed research I write about peer-reviewed enough? I don’t know. More important, though, do I provide the kind of commentary that counts as thoughtful commentary. I don’t mean this as a call for people to tell me I’m thoughtful – it’s more a rumination on what I am thinking about when I write about research.

I usually say something about the research methods, but not always. I rarely contextualize the research itself within the larger discipline. I raise questions that I have about the method or the conclusions, but I don’t usually go into my analysis thinking “I am going to comment on whether or not I think this is good research.” And when I read the science bloggers – I want them to do these things. I want them to help me evaluate the research and contextualize it.

But I don’t do that myself. More often, I want to share how the research informed or sparked my thinking about the work we all do – the connecting of the theory and the research to my practice as a librarian. I’m influenced a lot by Donald Schon and reflective practice here, and I’m thinking I might write about that sometime soon. But not now. Now I’ll just say that I think there’s value there. Deeply oversimplifying – apologies to Schon – value in the idea that theory is useful when it’s useful, when it’s connected to knowledge generated by lived experience. There’s a lot of people out there who don’t have time or inclination to reflect on research and what it means for practice, but who find it useful to read what others have to say.

So I think that’s kind of interesting, and worth thinking about what the theory we generate, the research we do means for our discipline, which, let’s face it is not the same as biology or sociology or physics. I’ll be registering when knowledgeblogging.org comes back up, and I’ll be using the icon. I hope others do too.  I’m interested in the idea of what peer reviewed research means to our field, and this is one way to think about that.

I never thought I’d really have to update those screenshots

Raise your hand if you thought that del.icio.us was never ever really going to change?

Yeah, me too.

my bookmarks at the new delicious

my bookmarks at the new delicious

No more weird URL — it’s at http://delicious.com

So far, so good?  I’m going to miss the little star that says I have a new fan.  And some other things I haven’t discovered yet.  But after four minutes with it I can say nothing has bugged me excessively.

Why we should read it before we cite it — no, really!

Last week, Female Science Professor wrote a lovely pair of posts about scholars and scholarship, what it feels like when your work has an impact on someone and what it feels like to meet the people who have influenced you in that particular undefinable way where it’s hard to even express what they’ve meant to you. I shared one, saved the other and generally felt very good about being a small part of this world where rock star crushes on ideas and the people who share them are understood and embraced.

Way to ruin everything, Inside Higher Ed.

Okay, not really. But seriously, it’s a lot harder to feel like a rock star because someone has read and used your work if, as Malcolm Wright and J. Scott Armstrong suggest, they probably didn’t read it and if they did, they probably read it wrong.

That might be a little bit strong, but not by much. So what does it mean when a published, peer-reviewed article in a real life journal kicks off its final, concluding paragraph with this sentence – Authors should read the papers they cite.


This isn’t a library tutorial aimed at fifth-graders writing their first research paper, after all. This is a paper talking about what professional scholars, people responsible for the continued development of knowledge in disciplines, should do. It can’t mean anything good. Here’s the original article:

Article at Interfaces – requires subscription

Article at Dr. Anderson’s faculty page – does NOT require a subscription – (opens in PDF)

Nutshell – Dr. Anderson wrote one of the more impact-heavy articles in his discipline, and the only article that analyzes and explains how to correct for non-response bias in mail surveys (that’s bias caused by people who do not respond at all to the survey). By analyzing 1. how often research based on mail surveys includes a citation to this article, and 2. how often the later researchers seem to interpret and apply the original article correctly the authors conclude that many, many researchers are not reading all of the relevant literature. More disturbingly, many, many researchers aren’t even reading all of the articles they themselves cite.

Now, on one level this isn’t a shocker – anyone who has read moderately deeply in any body of literature has probably looked at at least one bloated literature review and said “hey – this person probably didn’t really read all of these books and articles.” This article suggests that it’s more complex than just lit-review padding, that scholarly authors also mis-cite and mis-use the resources they use to support the methods they use and the conclusions they draw.

Working on the assumption that if your research uses a mail survey, you should at least be considering the possibility of nonresponse bias, they found that:

…far less than one in a thousand mail surveys consider evidence-based findings related to nonresponse bias. This has occurred even though the paper was published in 1977 and has been available in full text on the Internet for many years.

Working on the further assumption that someone who makes a claim about nonresponse bias, and who reads, understands and cites an article that outlines a particular method for correcting nonresponse bias to support that claim, will follow the method outlined in the article they cited, the authors conclude that many authors are either not reading or are not understanding the articles they cite:

The net result is that whereas evidence-based procedures for dealing with nonresponse bias have been available since 1977, they are properly applied only about once every 50 times that they are mentioned, and they are mentioned in only about one out of every 80 academic mail surveys.

Most of the research that seriously digs into how well researchers use the sources they cite has come out of the sciences, particularly the medical sciences. This is one of the first articles I’ve seen dealing with the social sciences, and I think it’s worth reading more closely because this very rough and brief summary doesn’t really do justice to the issues it raises. But right now, I want to turn to the authors’ conclusions because I think they get at some of the things we’ve been talking about around here about how new technologies and the read/write web might have an impact on scholarship.

The first two outline author responsibilities:

  • First – read the sources you cite. I think we can take that as a given – a bare-minimum practice not a best practice.
  • Secondly, “authors should use the verification of citations procedure.” Here they’re calling for authors to contact all of the researchers whose work they want to cite to make sure that they’re citing it correctly. I’m going to come back to this one.

The second two put some of the burden on the journals:

  • Journals should require authors to attest that they have in fact read the work they cite and that they have performed due diligence to make sure their citations are correct. That seems a sad, largely symbolic, but not unreasonable precaution.
  • Finally, journals should provide easily accessible webspaces for other people to post additional work and additional research that is relevant to research that has been published in the journal. Going to come back to this one too because I think it’s actually related to the one above.

Basically – both of these recommendations suggest that more communication and more transparency would be more better for knowledge creation. And what is the read/write web about if not communication and transparency, networking and openness?

Some of the commenters on the IHE article expressed, shall we say, polite skepticism that an author should be obligated to contact every person they cite before citing them. These concerns were also raised by one of the formal comment pieces attached to the Interfaces article. And I have to say I agree with these concerns for a few reasons. Anderson made the claim more than once that he does this as an author, with good results, and that the process is not too onerous. But that doesn’t really address the question of how onerous it would be for a prolific or influential author to have to field all of those requests.

And I’ll also admit to having some author is dead reactions to this. What if I contact Author A and say I’m planning to use your work in this way and they say “well I didn’t intend it to be used in that way so you shouldn’t.” Does that really mean I shouldn’t? Really? It’s hard to see this kind of thing not devolving quickly into something that actually hinders the development of new knowledge because it hinders new researchers’ ability to push at and find new connections in work that has come before.

But not to throw everything out with this bathwater – the idea that more and better and faster communication between scholars (more and better and faster than can be provided within journals and the citation-as-communication tradition) makes better scholarly conversations and better scholarship – that’s something I think we need to hold on to. Anderson points out how talking to the researcher who really knows the area described in the thing you’re citing can point you to other, less cited but more useful resources – how they can expand your knowledge of the field you’re talking about:

We checked with Franke to verify that we cited his work correctly. He directed us to a broader literature, and noted that Franke (1980) provided a longer and more technically sophisticated criticism; this later paper has been cited in the ISI Citation Index just nine times as of August 2006.

This is an area where the transparency, speed and networking aspects of the emerging web might have a real impact on the quality of scholarship even if there are no material changes in the practice of producing journal articles. I might not be sure about the idea of making this communication a part of citation verification but it should be a part of knowledge creation. And it’s tied as well with the final recommendation – that journals should provide webspaces for some, not all but some, of this communication to happen.

The types of conversations between similarly interested scholars that Anderson is describing is nothing new – the emerging web offers some opportunities for those conversations to move off the backchannel. Or maybe it’s the idea that it’s still a backchannel, but the back channel being visible is interesting. Whether the journal has its own backchannel for errors, additions, omissions and new ideas to be posted, or whether that backchannel exists on blogs, in online knoweldge communities, or networking spaces doesn’t matter so much as it can exist. We certainly have the technology.

And the journal Interfaces itself I think provides a suggestion as to why this kind of addtional discourse and conversation is valuable. You may have noticed that what looks like a fifteen page article is really an eight page article with six pages of response pieces, followed by an authors’ response. The responses challenge parts of the original article, and enrich other parts with additional information and examples. They illustrate the collaborative nature of knowledge production in the disciplines in a way that citations alone cannot. I couldn’t find anything on the journal’s website about this practice – if it’s a regular thing, how responses are solicited, or more. These responses are a spot of openness in a fairly closed publication.

And that as well points to the last point to make here because this is far too long already – I don’t think we have to change everything to fix the problems raised here – and I don’t think if we did change everything it would fix all of the problems raised here. There’s that scene in Bull Durham where Eppie Calvin gets his guitar taken away because he won’t get the lyrics right. And that’s the connection between FemaleScienceProfessor and Anderson and Wight — who can feel like a rock star if they’re singing your songs but getting them wrong?

There will always be Eppie Calvins out there inside and outside of academia -for them, women are wooly because of the stress. But injecting just some openness, making some communication visible – won’t stop Eppie Calvin, but might keep the next person from replicating his mistakes. And that’s a good thing.

Library Instruction 2.0: Building Your Online Instruction Toolkit

(cross-posted at Notes and Links)

2008 ALA Annual Conference
Rachel Bridgewater, Reed College
Anne-Marie Deitering, OSU Libraries
Karen Munro, University of Oregon

Links to our examples, and many more resources to browse can be found at our Library a la Carte page: Library Instruction 2.0

Web pages, CMS tools, LMS tools

LibGuides (SpringShare)

Library a la Carte (Oregon State University)

Haiku (web-based LMS, free service is limited)



Viviti (still in private beta)

Widgetize-able tools and applications

Sprout – use Sprout to create widgets out of RSS feeds and more






YouTube’s Embeddable Custom Player (you must be signed into YouTube)

VodPod (to include videos from providers other than YouTube)

Resources for Keeping Up

Google Reader

Wikipedia’s page on RSS aggregators – lots to choose from!


Information Aesthetics

Jane’s E-Learning Pick of the Day

Read/Write Web

Blackboard blogs

fun with words! pretty, pretty tagclouds for all

peer review 2.0, the proceedings paper in tagcloud

This is the paper Kate and I submitted along with our LOTW presentation, rendered into this gorgeous tagcloud by Wordle, a new tagcloud generator I saw today on Information Aesthetics.  I love tagclouds anyway – but this one lets you play with layout, fonts and colors in a way I’ve never seen before.  You can upload or copy and paste any text, or hook it into a del.icio.us account as a new way to see those tags.  So much fun.

LOTW follow-up – from people who weren’t there!

Kate and I are still buzzing from the great conversation we had with the people who came to our session at LOEX of the West. It’s always an amazing and kind of surreal experience when you find out that other people are excited by the same ideas you are.

And it seems that other people really are. Almost the second we stopped talking, we started finding other people who were. All over the web.

At ACRLog, Barbara Fister brings up the issue of promotion and tenure, and how many committees find it difficult to evaluate the significance of publications that don’t fit into the traditional scholarly formats — particularly when they are trying to evaluate the impact of scholars from other disciplines. These ideas are strongly connected to the ideas about distributing professional rewards, and we really just got started talking about the question of expertise, and evaluating work outside your discipline at the end there – good to see and think more about it.

Dorothea Salo talks about the differences between informal writing on the participatory web (like blog posts) and scholarly journal writing. She brings up one benefit to scholarly journals that we only hinted at – the way that the lengthy give and take between author and editor in the traditional publication process can make an individual article better. Not bring it up to some objective standard of quality, but make it better than it was. She also talks about something we did spend a lot of time talking about – the archive of knowledge, or the scholarly record. But she goes a lot further than we did talking about the role academic libraries play in that process.

Then today, I saw Tenured Radical’s discussion of the Social Science Research Network. She’s asking why historians aren’t participating in this project, and looking at some of the implications of that lack of participation. The SSRN is a digital archive that has as its goal the rapid dissemination of research in the social sciences. It includes an abstract database (of scholarly working papers and forthcoming papers) and an e-library of downloadable papers. These resources are available to registered members for free; there are also entry points into some proprietary database holdings, for a fee.

Tenured Radical highlights one of the reasons we think it’s so important that we are all having these conversations – not to replace traditional forms of publication, but to make them accessible. Not to encourage scholars to write for the public instead of for each other, but to leverage technological change in ways that can keep that scholarly discourse available to those who want to find it —

An insistence that the only good work has been heavily vetted through our current refereeing practices may be a mistake, much as soliciting the criticisms of others does contribute to producing good work (although it doesn’t always, I’m afraid, as cases where flawed research has slipped through to publication or to a prize demonstrates.) In its current form, it may be a fetish that is doing us more harm than good, and may be something that our professional associations need to review to take advantage of an atmosphere of intellectual vigor offered by electronic and other forms of mass publication.

LOEX of the West presentation, 2008

Peer Review 2.0: Tomorrow’s Scholarship for Today’s Students

LOEX of the West, Las Vegas

Anne-Marie Deitering & Kate Gronemyer


Five Web 2.0 themes — from the ACRL Instruction Section’s Current Issues Discussion Forum, Research Instruction in a Web 2.0 World (Annual, 2006).


{Edit: These didn’t make it into the presentation, but they are examples of some discussions on the web over the last year that started our thinking on this topic.}

danah boyd – open-access is the future: boycott locked-down academic journals

danah boyd – Viewing American Class Divisions through Facebook and MySpace

danah boyd – editing a special issue of JCMC






Current Anthropology

Expressive Processing: An Experiment in Blog-Based Peer Review – Noah Waldrip Fruin on Grand Text Auto

Cognitive Daily blog

ScienceBlogs – The World’s Largest Conversation about Science

BPR3: Bloggers for Peer-Reviewed Research Reporting — Icons


UsefulChem Wiki

Radiology Wiki

Open Notebook Science Using Blogs and Wikis (Jean-Claude Bradley, at Nature Precedings)

Rrresearch Blog

“The Academic Manuscript” — Wicked Anomie: Sociology Run Amok

Welcome to Nature Precedings


Barbara Fister points to this article in the Chronicle:  Certifying Online Research (Gary Olson), about the challenges of evaluating online publications.  See also Barbara’s post at ACRLog: Peer (to Peer) Review.