Good library assignments, part 1

I’m putting together a workshop tomorrow for teaching librarians about good research assignments — so I went looking to see what else has been written on the topic. I found lots of good stuff (I’ll talk about that later) but mostly what I found were rules — do’s and don’ts — embedded into pages about “when to ask for library instruction.”

(I bet you can predict what the rules are).

But here’s the thing – I break the rules all the time. In the last five years I have:

  • Taught classes without the faculty member present!
  • Said. “okay, sure!” when I was asked for a scavenger hunt activity.
  • Scheduled workshops for classes that don’t have research assignments, and which aren’t going to have research assignments.
  • And in one memorable case – integrated a scavenger hunt into a workshop for a class that was in the library without their instructor, that was a third again too big for every student to have an hands-on computer AND that didn’t have any kind of research assignment.

I mean, I don’t break rules for the thrill of breaking rules. And it’s not like we have anything so structured as “rules” here anyway. But I know them, just like we all know them, which means that even though I had good reasons for doing all of those things, I felt I had to figure those reasons out and justify those choices.

But I realized this morning that … I’m tired of rules. Or, maybe it’s more that rules make me tired. The effort to control and regulate a bunch of external conditions to make the one-shot — which has a bunch of moving parts that are uncontrollable — work is really tiring.

(And the rules have a nasty little unstated flip side — the one that says if all of the rules are followed, then the only reason why the one-shot isn’t awesome is librarian failure. That exhausts me even more.)

So in thinking about “good library assignments” the last thing I feel like doing is coming up with more rules. That’s right, not even “no scavenger hunts.”

I’m trying to pull together 3 pieces of interconnected thinking here. I don’t think I’ll talk about them all today – but I am hoping they’ll cohere if I talk about them. Here they are:

War stories: Thinking over “bad library assignments” I have seen – what are the broader categories?

  1. Assignments that require students to use, locate or manipulate a thing that my library does not have.
  2. Assignments that require students to do a thing in an outdated or inefficient way.
  3. Assignments with no immediate payoff – that serve only an unknown future need.
  4. Mis-matches — between assignment requirements and students’ cognitive development.
  5. Mis-matches — between the assignment requirements and the audience/ rhetorical purpose of the assignment.

Truisms: What are some things that are usually true (from my experience) about research assignments and teaching research?

  1. Saying “use the library” doesn’t make the library useful.
  2. The best way to encourage students to use a research tool or collection is to design a task that is legitimately easier when one uses that tool.
  3. The library is not a shortcut. People who use the library can’t end-run thinking or evaluating.
  4. Requiring something is not the same as teaching it.
  5. Students won’t automatically understand the connections between research assignments and course outcomes.
  6. Research freedom isn’t all it’s cracked up to be.

Expertise: What do we know about how students interact with research assignments that many others on campus do not?

  1. Library anxiety is real, has cognitive consequences, and can’t be fixed by requiring students to enter the building or touch the books.
  2. There are a lot of terrible sources available in library databases and on library shelves.
  3. Students will stick with what they know.
  4. Topic selection is difficult and stressful, and can be a barrier to student success on research assignments.
  5. Sometimes, it’s trying to do the right thing that leads students to do the wrong thing.
  6. Teachers and librarians have had experiences with (and built up a body of knowledge about) research and information that their students have not.

I’m going to dig into this more tomorrow, I think but for now – what do these things have to do with the rules above?

The faculty member present thing – probably nothing.  I agree that an active, involved faculty member makes my sessions better.  But I also have a lot of faculty at this point I’ve been working with for a long time — if someone I’ve assignment-designed with, taught with and published with needs to go to a conference the same week that her students need the library, I’m going to say yes.

But the rest – the rest do relate.  Because basically, I don’t think that a thrown-together research assignment, a mediocre research assignment, or a research assignment that’s separate from the class and will never be talked about again is going to make my session better.

And when we’re thinking beyond my individual session — then, a bad research assignment is going to make things worse.  So at that point, I have a couple of options – do the session without one (which I’ve done) or say, “no thanks, not this term” (which I’ve also done).

Why do I think they make things worse?  Because there are implicit messages buried in each of those “bad assignment” characteristics — let’s revisit?

Assignments that require students to use, locate or manipulate a thing to be successful — and my library does not have that thing (or enough of that thing).

Subtext:  Libraries don’t have what you need.  And perhaps even worse – librarians don’t know what you need and cannot help you.

Assignments that require students to do a thing in an outdated or inefficient way.

Subtext: People who use libraries do so because they don’t know the best way to do things.

Or, as a colleague and I used to say “let’s teach them – whatever you do, DON’T use library resources!”  This actually came from an assignment that never happened.  We wanted students to get an overview of the topic before going to scholarly sources (as you do) and we thought we might be able to embed a discussion about the differences between traditional encyclopedias and Wikipedia in the unit (yeah, yeah, it was 2005.  It was how we thought then).

We opened up our online Encyclopedia Brittanica, took a stack of student research logs, and started plugging in the words and phrases that they’d used in their initial searches.  And OMG were the results ever terrible.  We compared twenty-five student searches (because rigor) but we knew after five that we were never going to send people to the Brittanica because we’d be sending the implicit message – “whatever you do, DON’T use library resources.”

Assignments with no immediate payoff – that serve only an unknown future need.

Subtext: 

Mis-matches — between assignment requirements and students’ cognitive development.
Mis-matches — between the assignment requirements and the audience/ rhetorical purpose of the assignment.

These are two different things, but the subtext I’m worried about is the same:  You have to use these sources, processes, and tools here in school, but once you graduate you’ll never use them again.

So what did I miss?  Plus, more to come.

Research Immersion?

From Lee Bassette in last Thursday’s Inside Higher Edabout the Digital Humanities Summer Institute

I am excited because I am surrounded by people who are interested in similar research and skills as I am. I can talk about my project with people who a) actually know what and who I am talking about and b) are as excited by it as I am. People are interested in me and know me as a researcher, respect me as a researcher. My professional identity at home is so wrapped up in being a teacher, it’s nice to have this important other part of me recognized and, dare I say it, validated. I have a real intellectual community, not just a theoretical one.

I have lots of professor friends on other campuses who talk about this kind of isolation, but not librarians. When we talk about the issues of solo librarianship, we’re usually talking about something else — not about being the only metadata person, or instruction librarian, or archivist in the place.

And as librarians, we have both a (sometimes potential) research self, and a (usually actual) practice self — I could easily imagine that there are some librarians who are alone in their research interest, even with plenty of professional peers. Who needs this kind of shared experience more than us — what would it look like? Research Immersion?

So, according to TechCrunch in 2010, Bill Gates thinks that by 2015, people won’t have to go away to college anymore because

Five years from now on the web for free you’ll be able to find the best lectures in the world…. It will be better than any single university.

Fast forward to this year and Harvard and MIT launch edX, designed to bring an interactive course experience to anyone with an Internet connection (so, not just lectures) – building a “global community of learners” and strengthening programs back on campus as well.

Online education and it’s potential to disrupt college as we know it is a talked-about thing, is what I am saying.

But despite that, I have never really thought about this.

(via Walking Paper)

It’s kind of a longish video with a pace that is measured, or slow – so if you didn’t read it basically it seems to be a platform that manages online course offerings – potential teachers can upload their classes, potential students can find and sign up for classes.  There’s some consistency in offerings – they’re all one-day, in-person workshops that cost $20.

Here’s the thing, I can see this working with enough critical mass — but I’m not sure I can see it working on a college campus.  But I think it should work on a college campus – like, I can see it working on a campus that’s not all that different than the ones we have.  Why?  Well, reasons…

  1. We have a lot of really smart students who know how to do stuff.  We also have a lot of really smart faculty and staff who know how to do stuff, but I haven’t figured out yet if it works better in my head to be something bringing the whole community together – building a learning community that encompasses the physical community — or if it’s better as a student-teaching-students thing.
  2. We have students (and faculty and staff) who have a lot of interests – who want to learn how to do stuff.
  3. We talk a lot about high impact educational practices – those practices that increase  student success and engagement.  What’s important about these practices isn’t so much “if students get these experiences then school will be easier for our students” so much as “if students get these experiences then they’ll develop the networks, resources and resilience to get through the tough parts, stay in school and ultimately figure out how to succeed.  Taking on the teaching role doesn’t directly fit any of these practices, but it seems to fit in spirit — basically, if the teaching feels like it’s part of what makes the community the community, then participating would increase attachment to the community.

But on the other hand, other reasons …

When school pressures hit, there’s very little that survives. Which is what I mean when I say I can see this working on a college that is similar but not exactly the same as what I see outside my window.  (or what I would see if I had a window).  Basically, what I mean here is that I find it hard to see our students finding time for this kind of, well, dabbling a lot of the time — they can use working out or even parties as a legit reason not to study — one keeps you healthy and the other keeps you in friends – but taking a class on fixing your bike?  No, that I can’t see being treated as a legit reason not to focus on the classes and learning you’re actually paying for.

And I’m not sure what that means – I can easily see something like this working with my students just after they leave college.  Well, not easily, but realistically, I can imagine this kind of ecosystem taking root.  In college, on the other hand, it’s a lot harder.  I’m not sure what I think about that.

But here’s the thing – this seems like a great thing for libraries to manage.  This is information literacy, browsing, exploration and curiosity.  Exactly the kind of thing we are all about in college – but think about the ecosystem we build to support it.  What’s missing?  This kind of collaborative sharing of expertise — the people networks.

Which brings it back to he discussions of online learning I started with  – see, I’m pulling it back around.  Seriously, I’m as surprised as you are.

One thing that got me (and really, almost everyone else) two years ago when that Bill Gates quote appeared was just what a top-down, boring view of education it suggested — sitting in front of lectures, absorbing the knowledge =/= education.

And I’m a known lecture defender, but seriously – what made college worth it for me was the people.  And not just the faculty, though they were important, but my peers as well.

Which is why I think, on one level, that I couldn’t stop thinking about this community-teaching model after seeing it this morning.  Because it’s using technology to develop the community, but it gets at something that could only work on campus – that reflects part of why I love our campus community (and all of the campuses and communities I’ve been a part of).  It gets at part of the reason why, even though I had to do a distance library degree, I chose a program where I had classmates.

Of course, I learned a lot from my classmates, and of course I learned a lot from my interactions with faculty.  But even more than that – those relationships (especially with peers) are what created the culture of learning that existed in my college experience — the expectations, the standards, the ideas about what was worth your time and what weren’t – -those things were all social, shared values that we gave each other.  Some campuses did it really well, building a culture that really pushed me beyond where I would have been on my own.  Some, well, showed me how great I used to have it.

Even though I think it wouldn’t work – I keep trying to think about why it would.  Because a college that developed the kind of culture where that kind of sharing and learning was possible, was rewarded, was considered important enough to do even alongside the classes you’re paying for — that would be really cool.

babies, bathwater, whatnot…

Two recent rants have caught my eye, interestingly both from people who seem to have recently attended SXSW, a conference to which I’ve never been (but always wanted to go).  Anyway, one calls for “flipping the conference” and the other wholeheartedly agrees.

Before I start, let me say that I doubt either of these posts is really calling for eliminating all broadcast-type lectures or presentations from all conferences anytime or anywhere forever and ever.  Just like I don’t think that people who call for flipping the classroom – the discourse both of these pieces are riffing on — are talking about replacing lectures so much as using them better.  There’s a big difference between throwing out lecture altogether and recognizing that the teaching shouldn’t end there (and sometimes shouldn’t start there).

So this isn’t a “someone’s wrong,” rant so much as I don’t hear this expressed very much, and I think expressing it is important – there’s a continuum here, after all, and if the endpoints skew to one side then the conversation is affected.

This is the side that I never see expressed.  I like lectures, sometimes.  I think there’s a place for them.  I like some of the things that are “the problem” in these conversations and I am going to tell you why.

1. Remember, we’re all different.  

Both of the above pieces refer to this one shared assumption – that the really important thing about any conference is the “discussion, debates and conversations.”  And I think a lot of people express this, and believe it and for them it’s true, but for me (and not just me, I think) — It’s not.

I’ve come out before as a pretty serious introvert and I’ve got to tell you, two or three days filled with nothing but discussion, debates and conversations sounds like my idea of hell.  Okay, yes, I can imagine many worse things than that so maybe not hell, but seriously, it would take me at least a week to recover from something like that.

(Note.  I am not exaggerating for effect.)

For a quick primer on what I mean when I say “introvert” watch this:

To me, the best part of any conference is many things.  Sometimes it’s an inspiring keynote, sometimes its a meaningful conversation with someone from my home state over drinks half a continent away, sometimes its an idea that came to me in the middle of a talk on a topic I never would have sought out and am only being exposed to because it was the most interesting-looking paper in an underwhelming session.  You see what I mean?  It’s ideas and they come from everywhere and I don’t want to limit where I might run into them.

Lectures can be a super-effective way to get a lot of ideas out there to a lot of people really quickly.  Do I think we should build in time for reflection, time for discussion, time for debate?  Yes, of course I do, but don’t throw out the lectures.  See, I don’t need to do all of my thinking and all of my learning AT the conference.  Sometimes I want to get exposed to as much thinking and as many ideas as humanly possible while I’m there and I’m okay with doing some of my thinking and processing and conversation on my own time.

And here’s the thing, I think there IS value in giving busy professionals some space in their work lives to stop doing everything else, to focus on ideas and content for a few days.  Sometimes that means taking advantage of that shared space to talk, to discuss and debate.  Sometimes it means time to think, to reflect, to listen.  There’s value in providing a space where people who are passionate about the same things can share the experience of hearing new ideas or learning new things.

2. We don’t do the reading 

Because that shared experience is essential to making those conversations, debates and discussions about something, or at least about something new, isn’t it?

And here’s the second thing – I don’t think we’re going to do that in advance, on our own time.  I have been responsible for leading conference events and workshops and discussions that provided people with advance readings before and I am pretty confident in my blanket statement that no one ever does them.  I mean, let’s face it, there’s not gonna be a test.

(Okay, Jane does them, but then she has to catch everyone else up.)

We come to the conference, with our pens (or iPads or laptops), ready to absorb the knowledge* but I’m guessing that prepping in advance rarely goes further than what we can get done on the plane ride, if that.  And lectures are a GOOD way for lots of us to get that shared experience, to absorb that knowledge that can be our jumping off point.

Lectures don’t suck.  Bad lectures do.  Now, are bad lectures worse than bad active learning exercises?  I don’t know, I can get stuff done during a bad lecture without being actively rude, which is something I value.  I can think and reflect on the good ones.  Ask my co-workers how many projects have been started by emails I wrote from bad talks about ideas I got during bad talks.  (Spoiler alert!  A lot)

(Note.  I’m not advocating for bad lectures.  But maybe bad lectures > bad other stuff.)

Bad discussions, though, those are tough.  And a lot of times, they’re bad because there’s no shared content pushing them forward – because my class didn’t do the reading so it’s a series of unconnected “here’s what I think” statements – oh wait, that’s another topic.  But it’s not entirely another topic.  Discussions grounded only in a set of possibly connected individual experiences, not in a shared reading, or talk or idea – those can become deadly unfocused or reflect not much more than the loudest voice in the room.  It takes a lot of skill to facilitate discussion and spark conversation.  We don’t all have that skill.

*Bonus points for catching that reference

3. Let me be inspired

Because just like we all learn in different ways – we all teach in different ways.  Some people are great at facilitating discussion, creating debate and pushing the room to collaboratively developed insights but everyone isn’t.  Some people need time to deliberate practice and prepare to be their most effective and I don’t want to shut any of these people out of the conversation.  Isn’t there room for all?

It’s common in these discussions to talk about how none of the learning that really stuck with us from college came from lectures – it all came from more active work done (usually) outside the classroom, etc.  Well, okay. Some of my most memorable learning in college happened in lecture halls, listening to some truly fantastic speakers.  I think about those historians, political scientists and (in one case) psychologists every time I hear the anti-lecture conversation start up because I think the world is better place with those people doing what they do best.

And here’s the last thing – people like to be inspired and they are INSPIRED by good talks.  Look at the tweets from just about any conference and you’ll see tons of excited OMG I am so inspired comments – occasionally sparked by conversation but more often sparked by speakers.  Passionate, skilled, awesome speakers.

 

 

Peer Reviewed Monday – Expertise Reversal Theory

ResearchBlogging.org
Okay. So I am pretty sure that the actual article I am pointing to here (probably behind a pay wall – sorry) is not peer-reviewed.  It is the editors’ introduction to a special issue of the journal Instructional Science.  In this introduction they tell us that there are five empirical research reports and two commentary pieces in the issue, but this piece is neither empirical research nor peer-reviewed.

So, take a look at the whole issue, if your access to Springer journals is sufficient.  If not, I will summarize!

So, expertise reversal theory.  Sounds fancy.  I have definitely heard much talk about its theoretical context – cognitive load theory.  That shows up all over the place in library instruction circles. I was in an airplane during the opening keynote at WILU last week, but from what I heard (and I heard it a lot – the keynote must have been really good) the “brain guy” talked about the kinds of things that come up in conversations about cognitive load theory.

Expertise reversal theory is a subset of this research.  It suggests that the very same things that reduce cognitive load in novice learners can actually increase cognitive load in expert learners, or in learners with more domain knowledge.

The implications of this for library instruction seem immediate and obvious.

In the classroom.  How many times have we talked about the problem of the class that seems to be equally divided between the students who have never been pointed to a database and students who have been to the library classroom four times this year?  You can’t pitch your presentation to the experts in that scenario, and depending on what you have to do for the novices, you might find yourself saying “well, hearing it again won’t hurt them.”. This theory suggests that maybe it will.

On the web. This is, of course, the really significant place where we have to think about the possibilities this theory raises,  Who makes Beginning and Advanced online help possible?  (okay, maybe Zotero but who else?)

I have been thinking about my tutorial posts from last year, because they were part of a larger process that Hannah and I presented on at WILU.  One of the main takeaways I took from the craft tutorials was the way that they assumed that the people using the tutorials brought with them a body of knowledge, and that idea runs throughout this discussion of expertise reversal theory.  The authors argue that the most important cognitive factor in learning is prior knowledge:

Studies of expert-novice differences in cognitive science have clearly demonstrated that learner knowledge base is the most important and fundamental cognitive characteristic that influences learning and performance.

Basically, the way I read this overview is like this – novices don’t have a set of mental models, body of domain knowledge or prior experiences to structure their interaction with new information, tools, etc.  They need help – and to reduce their cognitive load, we provide  help that gives them that structure, whether it be text presented together with images, detailed step-by-step instructions, or whatever the case may be.  More experienced learners have those mental models in place.  When you give them that structuring information, it becomes something they have to wade through, and it might actually impede their ability to access their own prior knowledge.

At the most basic level, they need the opportunity to “opt out” of the extra help you put in there for novices.

A couple of the papers in the special issue are of interest, though I haven’t read them closely enough to analyze yet – they look at how well expertise reversal theory holds up in messy domains (specifically literary criticism and writing-to-learn in psychology) instead of focusing on “well-structured” domains like math or physics — and as instruction librarians, messy domains are where we usually live, right?

Kalyuga, S., & Renkl, A. (2009). Expertise reversal effect and its instructional implications: introduction to the special issue Instructional Science, 38 (3), 209-215 DOI: 10.1007/s11251-009-9102-0

Zotero assignment revisions

So, in the end the Zotero assignment worked very well on the Zotero side, and less well on the information literacy side.  So I’m spending this week revising it and designing some new activities.  A few quick takeaways:

The assignment was trying to do too much.  It was the main way to assess:

  • Students’ ability to recognize different source types and explain where the fit into the scholarly process.
  • Students’ ability to track down those different source types.
  • Students’ understanding of what the scholarly and creative output of their department (and by extension the scope of intellectual activity within their discipline).
  • Students’ ability to use research tools to organize and manage their sources.

Way too much.  Illustrated mainly by the fact that there were a few students to managed to do all of those things in their work.  That made it very clear what others were missing and made me want to figure out a way for all students to be able to get to where the few did in this class.

So here’s the thing – the first two outcomes up there were the problem, not the technology or logistics of syncing libraries and the like.  The bibliography project should really be about the 3rd and 4th outcomes.  The collaborative nature of the bibliography (and ability to see the breadth of what our faculty produces) was lost on students who had to work to hard to meet all of the format requirements that were in place to measure the first two outcomes.  All of the format requirements I put in to meet the first two outcomes took away from the authenticity of the experience, and of the evaluation and contextualization I had hoped the students would be able to do.

So this term, I am planning to get at those first two outcomes in different ways, and then make some changes to the bibliography assignment:

  1. drop the number of sources required in the annotated bibliography from 5 to 3.
  2. increase the emphasis on evaluation (and multiple methods of evaluation) in the annotations.
  3. change the workflow a bit – have students create a broad, pre-evaluated body of resources in a personal library and then have them select their 3 sources from that larger pool, annotate them and add them to the collaborative bibliography.
  4. build in a required conference so that I talk directly to each student about the process fairly early on.
  5. drop the format requirements altogether and allow students to add any 3 resources they want (while increasing their responsibility to justify those choices in multiple ways in their annotations).
  6. push the due date for the sources up a week, add a week between the final sources due date and the final reflection due date, and target and focus the scope final reflection essay significantly.

(Big hat tip to my students.  Many of these changes were also articulated by them when I asked them to help – in some cases their input was what really allowed me to put my finger on the problems).

What about the tech?

In the end, syncing did cause problems for a few, and Zotero hurdles did cause problems for a few.  Students who were, for whatever reason, not able to spend a focused amount of time at some point earlier in the term learning the mechanics of Zotero found it very challenging to manage finding sources and figuring out Zotero in the context of a last-minute scramble.

I had thought that my students would have to do the bulk of their Zotero work at home because of having to re-download and sync Zotero every time in the classroom.  MY Zotero library was still very difficult to sync in the classroom (I assume the hugeness is a factor) but the students rarely had to wait for more than 2-3 minutes.  Clearly, I can and should rely a lot more on classroom time as a place where students can be working with Zotero.

Most students were very positive about Zotero.  A few found it cumbersome.  There was a clear pattern though that I found interesting, but troubling in that there is nothing I can do with it.  The pattern was this — those students who had reason to use Zotero for real, for a real research project, during the term were much, much clearer in their evaluation of its value.  And by extension, I believe that they are the ones most likely to keep using it.

My class is a 1-credit class.  I can’t assign an authentic student-y scholarly research project that would take that little work.  But whether or not they have reason to use it in another class is nothing I can control.  It’s troubling because it points to a deeper issue about this class’ place within the major – issues we all know about but aren’t sure how to fix.

Yes, we did write that up

Finally!

Kate and I finally got an article related to our LOEX of the West presentation (from 2008!) finished and published.  This peer-reviewed article delay had nothing to do with publishing cycles and everything to do with writing process.  But it’s available (in pre-print) now, and I pretty much like it.

Beyond Peer-Reviewed Articles: Using Blogs to Enrich Students’ Understanding of Scholarly Work

Critical Literacy for Research – Sort of Peer-Reviewed Friday

Unexpectedly it’s Peer Reviewed Friday.  Well sort of.  Harvard Educational Review is a student-run journal, with an editorial board made up of graduate students deciding which articles get published.

I was teaching a class in our small classroom – where I never teach – so I went up early to make sure that I still knew how to work the tech.  It’s on the 5th floor, where the L’s are shelved, so I was flipping through the Fall 2009 issue while I waited for them to show up.  This article caught my eye — well worth reading, both for the content/ideas and because it is very enjoyably written.

Harouni, Houman (Fall 2009). High School Research and Critical Literacy: Social Studies with and Despite Wikipedia. Harvard Educational Review, 79:3. 473-493.

It’s a reflective, case-study type description of the author’s experiences reworking his research assignments in high school social studies classes. There’s a ton here to talk about – the specific exercises he developed and describes, the way the piece works as an example of critical reflective practice — but mainly I want to unpack this bit, which I think is the central theme of the work:

If students do not engage in the process of research inside the classroom, then it is natural for them to view the assignment in a results-oriented manner — the only manifestation of their work being their final paper and presentation.  It is not surprising then, that they are willing to quickly accept the most easily accessible and seemingly accurate information that satisfies the assignment and spares them the anxiety of questioning their data.  And when their final products did not meet my expectations, the students responded not by rethinking the research process itself but by simply attempting to adjust the product in light of what they perceived to be personal preferences. (476-77)

(emphasis mine)

Basically, the narrative he lays out says that his research projects had been unsuccessful for a while, but it wasn’t until he noticed his students’ heavy and consistent reliance on Wikipedia as a source that he started digging into why, what that meant, what he really wanted to teach, and what he really wanted students to learn.  And he changed stuff based on those reflections.

Harouni’s thinking about information literacy (which he calls “critical literacy for research”) was initially sparked by students who were not evaluating sources or showing any sign of curiosity as they researched, but it was further sparked when his first attempts at addressing student gaps didn’t work, sparked by students who were trying, and failing, to evaluate texts they weren’t yet ready to evaluate.

Along the way, he talks about the limitations of a checklist, or “algorithmic” approach to evaluation — limitation he discovered when he reflected on what his students actually did when he tried to use that approach in his classroom:

Two observations confirmed the shallowness of the learning experience created through the exercise: first, the students did not apply their learning unless I asked them to do so; second, they remained dependent on the list of rules and questions to guide their inquiry. (480)

In other words, they could do the thing he asked them to do (apply the checklist to information sources) but it didn’t affect their actual practice as researchers, nor did it change how they viewed the information they were getting from Wikipedia.

And also why it is important to help students understand the openness and dynamism of Wikipedia, but that that itself is not enough:  “knowledge of the uncertainties of a source does not automatically translate into an awareness of one’s relationship with the information (477).”

This piece is, I think, essential at getting at what I think is the real value of his insights and experience — many of our students want to find certainty in their research processes.  They want to know that a source is good or bad.  Wikipedia bans feed that.  Checklists feed that too, especially when they are not taught as an initial step in an evaluation process, but as the process itself.  What we really want students to be able to do when they research is to manage uncertainty — to say I know this is uncertain and I can figure out what it means for me as I try to answer my real, important, and complex question.

Harouni’s process his is an excellent reminder of how teachers want clarity too – and how they have to be willing to embrace uncertainty themselves if they are to guide students through a process of authentic inquiry:

In teaching critical literacy for research, I have had to separate research from its dry, academic context and consider it as an everyday practice of becoming informed about issues that have an impact on students’ lives.  I must value not answers but instead questions that represent the continued renewal of the search.  I must value uncertainty and admit complexity in the study of all things. (490)

In this, he knocks on the door of a question that I frequently have as an instruction librarian (one which I think many instruction librarians have — how much can I really accomplish as a teacher on my own).  If the classroom instructor – the person creates, assigns, explains, and evaluates the research assignment isn’t actively engaged with the students’ research process – are there limits to what I can do?  I do think there are.  I don’t think those limits means that I should do nothing, far from it – but I do think those limits affect what I think I should be trying to accomplish on my own and affect the other ways I should be thinking about furthering my goals for students, inquiry and learning.

At the end of the day, one of Harouni’s basic assumptions is that it is part of his job as a social studies teacher to foster inquiry and curiosity in his students, “[f]or two semesters, research projects remained a part of my curriculum — not because they were wonderful learning experiences, but because I could not justify, to myself, a social studies class that did not work to improve the way students navigated the ocean of available information (474-5).”  In other words, he believes that teaching information literacy is an essential part of what he does.   And that is key.  You can’t have that perspective and also value coverage – of content information – above all else.  It’s one or the other.  (is it?  Yeah, I think it is).

Every faculty member isn’t going to have that idea of what their job is.  And every librarian isn’t either – but I think maybe for instruction librarian it should be.  It is true that rules and clarity make coverage easier.  There was a question on ILI-L yesterday from someone (responding to an ongoing discussion about teaching web evaluation)  asking “how do you even have time to talk about web evaluation when you have to cover all this other stuff.”

Rules make it easier to “cover” web evaluation.  Faculty want us to “cover” lots of different tools.  WE want to “cover” lots of different tools.

(N.B. I am not suggesting that everyone who engaged in the “web evaluation” discussion just “covers” it and doesn’t teach it.  Nor am I suggesting that the people who worry about covering what the faculty want them to cover are only interested in coverage.  I do think though that the pressure to “cover” is as true for us as it is for people in the disciplines and these discussions spark reminders of that)

But if we want students to think about research as a process, if we want research to BE a learning process, then we have to engage in teaching the process.  And that’s extra hard for us – we can’t do that in the one-shot by ourselves.  And we can’t do it if we’re worried about coverage — about covering everything the library has to offer.  And I’m not just saying that about “we can’t teach everything about the library in a one-shot” — I think we all know that.  I think I am saying that it can’t be about that at all – that the point has to be about the process, about authenticity, about this -

I now understand that whatever research strategies students use in their day-to-day lives, which no doubt will vary depending on who the learners are, must be investigated and taken into account by their teacher.  Neither this goal nor the goal of improving these strategies can be attained unless students have time to engage in research while they are in the classroom.  And inviting students to the computer lab and remaining attentive to their interaction with online sources is as important as accompanying students to the library. (490)

And maybe this means not worrying about teaching research as a recursive learning process in the one-shot.  Maybe this means rethinking what and where we teach and maybe it’s work with faculty that gets at that overarching goal.  I don’t know.  I do know, though, that I have some great ideas for rethinking my credit class next term.

Classroom activities to promote critical literacy for research:

1. A (relatively innocuous) vandalism example demonstrated in class.  He didn’t change the content of pages, just the accompanying photo to illustrate the process of editing.

2. Students work in pairs to evaluate a Wikipedia article on a topic they know a lot about (for example, one student used the article about her former high school). Through this exercise he was able to teach about:  skepticism & its place in the research process, identifying controversial claims in a text, citations and footnotes, and verifying claims by checking outside sources.

3. Judging a book by its first sentence. He brought in 5 history textbooks, showed the covers and provided the first sentence.  Then he asked students to describe what they could figure out about the book from that first sentence.  With this exercise he was able to teach: authorial bias or point of view; finding the author’s voice.

4. Research beyond the first sentence.  When they tried to apply these critical skills to the texts they found in their research projects, though, they still had trouble because they didn’t know enough about the stuff they were researching.  So he looked for a way through this problem. Enter Wikipedia.  He provided a list of pages identified by Wikipedia editors as biased or lacking a neutral point of view, and asked the students to choose an article on a somewhat familiar topic and write a brief essay, with specific references to the text, with suggestions for improving the piece to meet the Wikipedia’s neutrality standard.

5. Contributing as an author.  Similar to other projects like this, it was one option for his students as a final project.  Interesting in that he collaboratively developed the assignment and rubric with interested students.

Citations 101

This term our first-year seminar/orientation classes (called U-Engage) have given me the opportunity to do some different things, teaching-wise.  One of the sections asked for resources for a “Citations 101″ unit.  This is what I’ve put together so far.

Does this work because it has a workable focus, and because it treats citing like something that has value, instead of something to do to avoid getting in trouble (or because no one respects the ideas of college students, which is a message that I think some students take away from lessons about the rhetorical uses of outside sources).  I do think the rhetorical uses are crucial, but they were beyond the scope here – and I think would have taken the focus beyond “workable.”

ALS 199: Citations 101

(Built with our Library a la Carte tutorials extension.)

Making one-shots better – what the research says (Peer Reviewed Monday, part 2)

ResearchBlogging.org

And now, on to Peer-Reviewed Monday, part two but still not Monday.

Mesmer-Magnus, J., & Viswesvaran, C. (2010). The role of pre-training interventions in learning: A meta-analysis and integrative review☆ Human Resource Management Review, 20 (4), 261-282 DOI: 10.1016/j.hrmr.2010.05.001

As I said earlier this week, this was started by a link to this article, a meta-analysis trying to dig deeper into the questions: which of the pre-practice interventions examined in the Cannon-Bowers, et al study are most effective?  For what type of learning outcomes?  And under what conditions?

The first part of the paper reviews what each of the pre-training interventions are, and presents hypotheses about what the research will reveal about their effectiveness.

METHOD

They reviewed 159 studies, reported in 128 manuscripts.  For this work, they considered only studies that met all of the following conditions:

  • they involved the administration of a pre-training intervention
  • the study population included adult learners
  • the intervention was part of a training program
  • the study measured at least one learning outcome
  • the study provided enough information to compute effect sizes.

The studies were coded for: the type of pre-practice intervention; the type of learning outcome; the method of training delivery; and the content of the training.

The codes for pre-practice intervention were drawn from Cannon-Bowers, et al: attentional advice, metacognitive strategies, advance organizers, goal orientation, and preparatory information.

The codes for learning outcomes were drawn from the Kraiger, et al (1993) taxonomy:

  • Cognitive learning (can be assessed at 3 stages: verbal knowledge, knowledge organization and cognitive strategies)
  • Skill-based learning (also assessed at 3 stages: skill acquisition, skill complication, and skill automaticity)
  • Affective learning (attitudinal outcomes, self-efficacy outcomes and disposition outcomes)

Training methods coded were very relevant to information literacy instruction: traditional classroom; self-directed or distance learning or simulations, such as role-plays or virtual reality.

Training content was coded as: intellectual, interpersonal, task-related or attitude.

RESULTS & DISCUSSION — so, what does the research say:

For attentional advice — this was one that I was able to immediately think of one-shot related applications for, so it was particularly interesting to me that medium to large positive effects were found for both skill-based and cognitive outcomes, with the largest gains found for skill-based outcomes — given that so much of what is taught in one-shots is skill-based, intended to promote success on particular assignments.  These effects are strongest when general, not specific, advice is given.

Metacognitive strategies –

The authors identified two main forms of meta-cognitive strategies that were studied: strategies that involved the learner asking why questions, and strategies where the learner was prompted to think aloud during learning activities.

The research shows that meta-cognitive strategies seem to promote all levels of cognitive and skill-based learning.  Why-based strategies had more consistent effects for all levels of cognitive learning, which supports the authors’ initial hypothesis — but think-aloud strategies do a better job of supporting skill-based outcomes, which does not.

Advance organizers —

Positive results were found for these for both cognitive and skill-based outcomes.  Of particular note for instruction librarians is this finding:  “stronger results were found for graphic organizers than text-based ones across all levels of skill-based outcomes.”

Goal orientation —

When compared with situations were no overt goal was provided to the learners, goal orientations seem to support all types of learning: cognitive, skill-based and affective, with the strongest effects (just by a little bit) in the affective domain.

The authors also hypothesized that mastery goals would be better than performance goals.  The findings suggest this hypothesis is true for skill-based learning and for affective learning.  They were not able to test it for cognitive learning.  They did find something odd with regards to affective learning – when they compared performance goals and mastery goals separately against no-goal situations, then performance goals showed greater effects.  But when they compared mastery goals and performance goals, stronger effects were found for mastery goals.

Preparatory information –

This showed positive effects for skill-based and affective learning, but they weren’t able to test it for cognitive learning outcomes.

SO WHAT ELSE COULD HAVE AN EFFECT?

The training conditions and content were coded to see if those things had an effect on which pre-practice interventions were most effective.  Of particular interest to me were the finding that stronger effects for cognitive learning were found for advance organizers paired with self-directed training (e.g. tutorials) than for traditional classrooms or simulations.  (Of course, it’s important to remember that those showed positive effects too).

RESULTS BY TYPE OF OUTCOME

This turned out to be the most interesting way to think about it for me, so I’m going to include all of these probably at a certain level of length…

For skill-based outcomes, broken down – the strategies that work best seem to be:

  • skill acquisition: mastery goals & graphic advance organizers.
  • skill compilation: think-aloud meta-cognitive strategies, attentional advice and goals.
  • skill automaticity: graphic organizers and pre-training goals.

This seems to suggest pretty strongly that librarians should find a way to communicate goals to students prior to the one-shot.  Obviously, the best way to do this would probably be via the classroom faculty member, which is why this also makes me think about the implicit message in the goals we do send to students – most specifically, I mean the implicit message sent by requirements like “find one of these, two of these, three of these and use them in your paper.  It does seem like this could be considered a performance goal more than a mastery goal and even if the main impact on students is added stress to perform, is that stress that is serving any purpose or should it be eliminated?

For cognitive outcomes, also broken down – these strategies emerged from the literature:

  • verbal knowledge: specific attentional advice, why-based meta-cognitive strategies, and graphic advance organizers had the largest effect.
  • knowledge organization: general attentional advice and think-aloud metacognitive strategies
  • development of cognitive strategies: why-based strategies and attentional advice.

This is interesting, of course, because while we know that teaching on this cognitive-outcome level is pretty hard in 50 minutes, a lot of the topics we’re asked to address in the one shot are really asking students to perform in that domain.  Ideas like information ethics, intellectual honestly, scholarly communication, identifying a good research article – these all require more than a set of skills, but also require a way of thinking.  So in this area, I am thinking okay, we can’t teach this in 50 minutes, but if we can prep them in advance, maybe we have a better chance of getting to something meaningful in that time.

For affective outcomes –

  • Overall, a pre-training goal orientation and attentional advice were most effective in this domain.

These might not seem relevant in the one-shot, but they really are.  We’re talking in many cases about teaching them something with the hope that they’ll use it later, when they really get to that stage of their research process, their confidence and self-efficacy is clearly relevant, as is their disposition to believe that you’re teaching them something valuable!  In fact, I think this might be as worth or more worth focusing on that cognitive outcomes.  So that makes these findings particularly interesting:

  • post training self-efficacy AND disposition toward future use of the training material were most influence when a performance goal orientation was used.
  • Attentional advice, mastery goals and preparatory information are also promising here.