Peer Review 2.0, revised and updated
April 27th, 2009 § 2 Comments
Watch this space – we may be able to put up a link to the actual talk at some point. This version is being presented online, to librarians and faculty members from Seattle-area community colleges.
Sneak preview:

Why 2.0?
Michael Gorman (Britannica Blog) Jabberwiki: The Educational Response, parts one and two
Shifting perspective – why journals?
Ann Schaffner (1994). The Future of Scientific Journals: A View from the Past (ERIC)
Archive of knowledge
(Skulls in the Stars), Classic Science Papers: The 2008 “Challenge” !
(Female Science Professor) Everyone knows that already
Community building
Sisyphus (Academic Cog), MMAP Update April 13: Publishing Advice from the Professionals
(Historiann), Peer review: Editors versus authors smackdown edition
Clickstream Data Yields High Resolution Maps of Science (PLoS ONE)
Quality control
BBC TV and Radio Follow-Up: The Dark Secret of Henrik Schon
Bell Labs, press release. Bell Labs announces results of inquiry into research misconduct.
Fiona Godlee, et al. Effect on the Quality of Peer Review of Blinding Reviewers and Asking them to Sign their Reports (paywall)
Willy Aspinall (Nature Blogs: Peer-to-Peer), A metric for measuring peer-review performance
(Lounge of the Lab Lemming), What to do when you review?
Distributing rewards
Undine (Not of General Interest), From the Chronicle, Are Senior Scholars Abandoning Journal Publication (also includes a link to the original article behind the Chronicle’s paywall)
(PhD Comics) How professors spend their time
Report of the MLA Task Force on Evaluating Scholarship for Tenure
Openness – access
Directory of Open Access Journals
Openness – scope
Fill His Head First with a Thousand Questions blog
Landes Bioscience Journals, RNA Biology: Guidelines for Authors (requires authors to submit a Wikipedia article)
(Crooked Timber) Seminar on Steve Teles’ The Rise of the Conservative Legal Movement
Henry (Crooked Timber), Are blogs ruining economic debate ?
Collaborative
WikiBooks – Human Physiology
Re-mixed
Iterative
(sometimes) Digital
Stevan Harnad. Creative Disagreement: Open Peer Commentary Adds a Vital Dimension to Review Procedures.
(Peer-to-Peer) Nature Precedings and Open Peer Review, One Year On
Sara Kearns (Talking in the Library), Mind the Gap: Peer Review Opens Up
Miscellaneous
The awesome font we used on the slides is available for free from Typographia:
http://new.typographica.org/2007/type_commentary/saul-bass-website-and-hitchcock-font-are-back/
Photo credit (because it is tiny here) – Surprise. flickr user Jeremy Brooks. http://www.flickr.com/photos/jeremybrooks/3330306480/
DIY tutorials, library style
April 21st, 2009 § 1 Comment
Or kind of. After writing this post last winter, I started thinking about this idea as a way to connect with some of the classes I work with. Quick recap, I was looking at craft tutorials online and came up with some common characteristics they had, that our library tutorials don’t always have:
- They’re kind of at the point of need, they’re kind of not.
- They’re all about how to make something.
- They usually assume some level of knowledge on the part of the user.
- They are presented using social tools.
- There’s value added. They do some of the work for the user.
- A lot of the time, they’re marketing tools.
- They are created within an existing community.
I work with some of our distance education classes, the writing classes for example, and having some very quick and easy “here’s how to get this thing done” how-to’s make so much sense for those students – I tend to answer the same questions over and over and I have access to their class space in the LMS and to their email addresses.
But it’s not just the distance classes that I am thinking about. I taught for a business writing class and it was exactly the kind of class I frequently have trouble with. The students need to do a little bit of very specific kinds of research for every project they have in this class — there’s no way to time a single instruction section so that it works for this class.
To show them how to find the specific types of stuff (information on non-profits, job listings, community statistics, opinion polls, company information, annual reports, and on and on and on) they need to find, in a face-to-face session inherently means spending most of the session doing straight-up how-to demos to support assignments they don’t even have yet. There’s no way around it. There’s no way the instructor could have structured the class any better, and there’s no way that I could make these topics more relevant in a traditional one-shot.
And the stuff is pretty straightforward – it’s mostly a matter of pointing to where the stuff is, and a few tips on the how, and they can take it and with it from there. The complex part of what students need to do in this class is to figure out what kind of evidence they need to write about the project they’ve come up with for the audience they have — that’s good, interesting work but it’s also not well-suited to a one-shot because they have to do this over and over again for every project they do. So multiple one-shot sessions would make no sense for this class either.
What makes sense to me is to connect with this class at the start of the term, by visiting them in person since they are an in-person class. But the quick connection at the start would be pretty easy to replicate online. And once that first connection is made, it makes sense to me to send the class quick how-to information about the stuff they need to find, when they need to find it.

I am also thinking about some of the large general education classes that I would like to support, but which we could never support with face-to-face sessions given current staffing levels. We are already embedded into the First Year Composition curriculum, which is the only course required for all of our undergraduates. But there are a lot of other courses that have a lot of undergraduates enrolled and some of those have assignments that require outside sources. Thinking about the opportunity to reach 500 or so students with some point-of-need help (that reinforces the FYC lessons) in each of those classes, while continuing to reach 700-800 in FYC – that would make me pretty happy about our impact on the first-year experience.
So, is copying the craft tutorials the way to go? Maybe it is – not entirely copying, but there are some opportunities there, I think. Our web developer, Susan McEvoy, put together a blog for me to use just for this – that should let us track the same kind of statistics we track on the overall website. It’s very simple, stripped-down. The posts are just text and images. Because I write fast, putting together one of these takes 20-40 minutes, with most of that taken up uploading images.
So that means I can be really responsive and tailor things to assignments. It’s also easy to send students a link and announcement from within the LMS. In fact, there’s a DIY Tutorial on how to do that.
So how do they match up with the craft tutorials? Do these concepts translate? Sometimes yes, sometimes now.
1. They’re kind of at the point of need, they’re kind of not.
This is true in that they are sent to students at the point of need, and they also persist, so they can be found or re-found later. But I don’t think they’re very searchable now, given that I haven’t done much to make that happen. The images are all on flickr, which is something I think could be utilized better – at first I thought putting together Joe Murphy-style tutorials at the same time as the DIY tutorials made sense, but then I realized that I re-use a lot of the images. But I think the tagging here could connect people to the finished products too, if I think about it more.
2. They’re all about how to make something.
This one, I have trouble with. The bibliographic management ones work in this way – “make a bibliography.” A lot of the others are more process-focused. I tried to focus the titles at least on the thing(s) that could be found with the process, but I think this one needs more work too.
3. They usually assume some level of knowledge on the part of the user.
I did link out to other tutorials when I thought there might be things people didn’t know. But otherwise these are limited to the specific thing they are about, not all of the building blocks knowledge people will need, or the additional questions they might spark.
4. They are presented using social tools.
Yes, and this is important. There is an issue with the comments, since most of our users don’t log in to the system before using it. But putting it on a blog allows for the content to be repurposed easily into our Course Pages:

and our LMS:

5. There’s value added. They do some of the work for the user.
This, I haven’t figured out. Perhaps if I was working in less of a teaching environment this would be easier.
6. A lot of the time, they’re marketing tools.
Absolutely!
7. They are created within an existing community.
To the extent that they are being created and conceptualized entirely within existing classes, yes, this works. To the extent that being of a community makes them findable, I think that is less clear.
So, we’ll see how it goes. I have no plans for assessment at this point beyond web logging information – including the time spent and return visits, so more interesting than straight hit counts. And I have a fairly modest definition of success – these take so little effort to make, I don’t need all of the students in the class to find them useful, or even to try them. I will keep you posted.
talking on the web
April 19th, 2009 § Leave a Comment
What is it about spring term that it always ends up being overloaded? Sometimes it is travel, and this term definitely has its share of that – informational visits to the University of Minnesota and the University of Wisconsin-Madison, WILU in Montreal, and an insane 30-hour total trip across 3 time zones for a college reunion. But unlike other travel-crazy terms, this time around it’s the presentations that have me feeling that “there’s always something more to be working on” feeling.
Upcoming – my first real forays into web-based presentations.
First, there is this one with Rachel: Social Media and the Ethics of Information.
Then a few days later, Kate and I are going to do a version of the Peer Review 2.0 talk as a professional development workshop for community college libraries in Seattle.
Given budget realities for all of us, I would expect that this form of presentation and sharing will only become more important, so I am excited to try it out. But I’m also nervous. I’ve definitely been in on online presentation situations where the content and/or presentation style didn’t translate very well. And it’s not something you can practice, or at least I haven’t figured out how; every practice feels even more artificial than practicing a traditional presentation in front of the mirror.
(Not that I’d know anything about that – I am a practice-while-driving type of presenter)
So yeah, if you’ve ever sat in on a really great webcast presentation, or a really bad one, I’d love to hear what works and what doesn’t.
Not quite peer-reviewed Monday, but related!
April 7th, 2009 § 1 Comment
So slammed, so briefly (well, for me). Via CrookedTimber, a pointed to this post by Julian Sanchez on argumentative fallacies, experts, non-experts and debates about climate change. It’s well worth reading, especially if you are interested in the question of how non-experts can evaluate and use expert information, which is a topic that I think should be of interest to any academic librarian.
Obviously, when it comes to an argument between trained scientific specialists, they ought to ignore the consensus and deal directly with the argument on its merits. But most of us are not actually in any position to deal with the arguments on the merits.
Sanchez argues that most of us have to rely upon the credibility of the author — which is a strategy many librarians also espouse — in part because someone who truly wants to confuse them can do so, and sound very plausible.
Give me a topic I know fairly intimately, and I can often make a convincing case for absolute horseshit. Convincing, at any rate, to an ordinary educated person with only passing acquaintance with the topic.
Further, he suggests that the person who wants to confuse a complex issue actually has an advantage over those who want to talk about the complexity:
Actually, I have a plausible advantage here as a peddler of horseshit: I need only worry about what sounds plausible. If my opponent is trying to explain what’s true, he may be constrained to introduce concepts that take a while to explain and are hard to follow, trying the patience (and perhaps wounding the ego) of the audience:
Come to think of it, there’s a certain class of rhetoric I’m going to call the “one way hash” argument.
And that’s where we get to the evaluation piece. We need to know how much we know to know whether it even makes sense to try and evaluate the arguments. Because if we don’t know enough, trying to evaluate the quality of the actual argument will probably steer us astray more often than using credibility as our evaluation metric.
If we don’t sometimes defer to the expert consensus, we’ll systematically tend to go wrong in the face of one-way-hash arguments, at least our own necessarily limited domains of knowledge.
(Note: I skipped most of the paragraph where he really explains the one-way hash argument – you should read it there)
The thing I really want to focus on is this – that one word, consensus. Because I don’t think we do much with that idea in beginning composition courses, or beginning communication courses, or many other examples of “beginning” courses which often serve as a student’s first introduction to scholarly discourse.
And by “we” here, I’m talking about higher ed in general, not OSU in particular. I think we ask students in these beginning classes to find sources related to their argument; their own argument or interest is the thing that organizes the research they find. They work with that article outside of any context, except which might be presented in the literature review – they don’t know if it’s steadily mainstream, a freakish outlier, or suggesting something really new.
So they go out and find their required scholarly sources, and they read them and they think about the argument in the scholarly paper and how it relates to the argument they are making in their own paper and try to evaluate it – and of course, they evaluate mostly on the question of how well it fits into their paper. And what other option do they have?
Sanchez argues, and it rings true to me, that we usually don’t have the skills to evaluate the quality of the argument or research ourselves. And I know that I am not at all comfortable with the “it was in a scholarly journal so it is good” method of evaluation. Even if they find the author’s bona fides, I’m not sure that helps unless they can find out what their reputation is in the field, and isn’t that just another form of figuring out consensus?
In some fields, meta-analyses would be helpful here, or review essays in others, but so many students choose topics where neither of those tools would be available, that it’s hard to figure out how to use that in the non-disciplinary curriculum.
And perhaps it doesn’t matter – maybe just learning that there are scholarly journals and that there are disciplinary conventions, is enough at the beginning level. But if that’s the case, then maybe we should let that question of evaluation, when it comes to scholarly arguments, go at that level too?
it’s the math
April 1st, 2009 § Leave a Comment
I’m not sure that even my tendency to see information literacy connections everywhere will explain why I’m posting this, but I just thought it was really interesting. This morning, I got pointed to this article (via a delicious network) which argues that hands-on, unstructured, discovery-based learning doesn’t do the trick for many science students at the secondary level. Using preparedness for college science as their definition of success, most students are more successful if their high school science learning is significantly structured for them by their teachers.
Structure More Effective in High School Science Classes, Study Reveals
What jumped out at me here was that the reason seemed to be linked to the math – students with good preparation in the math, did benefit from unstructured, discovery based learning. And then there was another “similar articles to this one link at the bottom of the page, pointing to another study, making another point -which supports this idea too (which is not hugely surprising because both items point to different papers by the same researchers).
You do better in college if you’ve taken high school classes in chemistry, better in physics if you’ve taken physics – but the one big exception to the success in one doesn’t generalize argument? You do better in everything if you’re well-prepared in math.
College Science Success Linked to Math and Same-Subject Preparation
After that there are more “articles like this one links” leading to articles about middle-school math teachers in the US being really ill-prepared, or things about gender and math and science which really got me thinking about further implications of those findings – if math is such a lynchpin. So there is something there about how this dynamic, browsable environment makes your brain work in ways that make research better.
There’s also something there about context – getting the “math teachers aren’t prepared” article in the context of the “math is key” research made the significance of the former clearer, made how I could *use* that research much clearer than it would have been if I came upon it alone. There’s also something there about the power of sites like ScienceDaily (and ScienceBlogs, and ResearchBlogging.org and others) to pull together research, present it in an accessible way in spaces where researchers/readers can make those connections.
And there might even be something there about foundational, cognitive skills that undergird other learning. But mostly, I just found it interesting.
—————
Studies referenced were reported on here:
Sadler, Philip M. & Tai, Robert, H. The two high-school pillars supporting college science (Education Forum) Science 27 July 2007: Vol. 317. no. 5837, pp. 457 – 458. DOI: 10.1126/science.1144214 (paywall)
Tai & Sadler, Same Science for all? Interactive association of structure in learning activities and academic attainment background on college science performance in the USA. International Journal of Science Education, Volume 31, Issue 5 March 2009 , pages 675 – 696. DOI: 10.1080/09500690701750600