Peer Reviewed Monday – Expertise Reversal Theory

ResearchBlogging.org
Okay. So I am pretty sure that the actual article I am pointing to here (probably behind a pay wall – sorry) is not peer-reviewed.  It is the editors’ introduction to a special issue of the journal Instructional Science.  In this introduction they tell us that there are five empirical research reports and two commentary pieces in the issue, but this piece is neither empirical research nor peer-reviewed.

So, take a look at the whole issue, if your access to Springer journals is sufficient.  If not, I will summarize!

So, expertise reversal theory.  Sounds fancy.  I have definitely heard much talk about its theoretical context – cognitive load theory.  That shows up all over the place in library instruction circles. I was in an airplane during the opening keynote at WILU last week, but from what I heard (and I heard it a lot – the keynote must have been really good) the “brain guy” talked about the kinds of things that come up in conversations about cognitive load theory.

Expertise reversal theory is a subset of this research.  It suggests that the very same things that reduce cognitive load in novice learners can actually increase cognitive load in expert learners, or in learners with more domain knowledge.

The implications of this for library instruction seem immediate and obvious.

In the classroom.  How many times have we talked about the problem of the class that seems to be equally divided between the students who have never been pointed to a database and students who have been to the library classroom four times this year?  You can’t pitch your presentation to the experts in that scenario, and depending on what you have to do for the novices, you might find yourself saying “well, hearing it again won’t hurt them.”. This theory suggests that maybe it will.

On the web. This is, of course, the really significant place where we have to think about the possibilities this theory raises,  Who makes Beginning and Advanced online help possible?  (okay, maybe Zotero but who else?)

I have been thinking about my tutorial posts from last year, because they were part of a larger process that Hannah and I presented on at WILU.  One of the main takeaways I took from the craft tutorials was the way that they assumed that the people using the tutorials brought with them a body of knowledge, and that idea runs throughout this discussion of expertise reversal theory.  The authors argue that the most important cognitive factor in learning is prior knowledge:

Studies of expert-novice differences in cognitive science have clearly demonstrated that learner knowledge base is the most important and fundamental cognitive characteristic that influences learning and performance.

Basically, the way I read this overview is like this – novices don’t have a set of mental models, body of domain knowledge or prior experiences to structure their interaction with new information, tools, etc.  They need help – and to reduce their cognitive load, we provide  help that gives them that structure, whether it be text presented together with images, detailed step-by-step instructions, or whatever the case may be.  More experienced learners have those mental models in place.  When you give them that structuring information, it becomes something they have to wade through, and it might actually impede their ability to access their own prior knowledge.

At the most basic level, they need the opportunity to “opt out” of the extra help you put in there for novices.

A couple of the papers in the special issue are of interest, though I haven’t read them closely enough to analyze yet – they look at how well expertise reversal theory holds up in messy domains (specifically literary criticism and writing-to-learn in psychology) instead of focusing on “well-structured” domains like math or physics — and as instruction librarians, messy domains are where we usually live, right?

Kalyuga, S., & Renkl, A. (2009). Expertise reversal effect and its instructional implications: introduction to the special issue Instructional Science, 38 (3), 209-215 DOI: 10.1007/s11251-009-9102-0

Zotero assignment revisions

So, in the end the Zotero assignment worked very well on the Zotero side, and less well on the information literacy side.  So I’m spending this week revising it and designing some new activities.  A few quick takeaways:

The assignment was trying to do too much.  It was the main way to assess:

  • Students’ ability to recognize different source types and explain where the fit into the scholarly process.
  • Students’ ability to track down those different source types.
  • Students’ understanding of what the scholarly and creative output of their department (and by extension the scope of intellectual activity within their discipline).
  • Students’ ability to use research tools to organize and manage their sources.

Way too much.  Illustrated mainly by the fact that there were a few students to managed to do all of those things in their work.  That made it very clear what others were missing and made me want to figure out a way for all students to be able to get to where the few did in this class.

So here’s the thing – the first two outcomes up there were the problem, not the technology or logistics of syncing libraries and the like.  The bibliography project should really be about the 3rd and 4th outcomes.  The collaborative nature of the bibliography (and ability to see the breadth of what our faculty produces) was lost on students who had to work to hard to meet all of the format requirements that were in place to measure the first two outcomes.  All of the format requirements I put in to meet the first two outcomes took away from the authenticity of the experience, and of the evaluation and contextualization I had hoped the students would be able to do.

So this term, I am planning to get at those first two outcomes in different ways, and then make some changes to the bibliography assignment:

  1. drop the number of sources required in the annotated bibliography from 5 to 3.
  2. increase the emphasis on evaluation (and multiple methods of evaluation) in the annotations.
  3. change the workflow a bit – have students create a broad, pre-evaluated body of resources in a personal library and then have them select their 3 sources from that larger pool, annotate them and add them to the collaborative bibliography.
  4. build in a required conference so that I talk directly to each student about the process fairly early on.
  5. drop the format requirements altogether and allow students to add any 3 resources they want (while increasing their responsibility to justify those choices in multiple ways in their annotations).
  6. push the due date for the sources up a week, add a week between the final sources due date and the final reflection due date, and target and focus the scope final reflection essay significantly.

(Big hat tip to my students.  Many of these changes were also articulated by them when I asked them to help – in some cases their input was what really allowed me to put my finger on the problems).

What about the tech?

In the end, syncing did cause problems for a few, and Zotero hurdles did cause problems for a few.  Students who were, for whatever reason, not able to spend a focused amount of time at some point earlier in the term learning the mechanics of Zotero found it very challenging to manage finding sources and figuring out Zotero in the context of a last-minute scramble.

I had thought that my students would have to do the bulk of their Zotero work at home because of having to re-download and sync Zotero every time in the classroom.  MY Zotero library was still very difficult to sync in the classroom (I assume the hugeness is a factor) but the students rarely had to wait for more than 2-3 minutes.  Clearly, I can and should rely a lot more on classroom time as a place where students can be working with Zotero.

Most students were very positive about Zotero.  A few found it cumbersome.  There was a clear pattern though that I found interesting, but troubling in that there is nothing I can do with it.  The pattern was this — those students who had reason to use Zotero for real, for a real research project, during the term were much, much clearer in their evaluation of its value.  And by extension, I believe that they are the ones most likely to keep using it.

My class is a 1-credit class.  I can’t assign an authentic student-y scholarly research project that would take that little work.  But whether or not they have reason to use it in another class is nothing I can control.  It’s troubling because it points to a deeper issue about this class’ place within the major – issues we all know about but aren’t sure how to fix.

Zotero assignment update

So the first mini-deadline on the Zotero assignment has come and gone, and I’m pretty happy with the results so far.  They’re not very impressive to look at, but when you compare what is actually happening with what I thought could happen, I think we are well on our way to getting this done.

For the first section, which has 21 students:

  • 11 successfully added a scholarly source to their Zotero library AND successfully synced to the group library.  Another one got the sync to work, but what got saved isn’t in very good shape yet.  Three more are waiting on ILL to decide which article they want to save to the bibliography.
  • Of those 11, 6 have added an original annotation and tags.

There are a few who added something in another format (and I’m not sure if that is a result of still not knowing how to find a scholarly article for their person, or if it is a matter of the best sources authored by their person not being scholarly articles)  I’ll find out more about that in class this week.

In the second, which has 24 students registered:

  • 13 successfully added a source to their Zotero library AND successfully synced.  Another one did the sync okay, but what got added was wonky.  There is one person who has added two things.  There is also an example article that I added still in there.
  • And there is a weird article from the medical literature that is still mysterious.  The author doesn’t share a last name with one of our target authors, so I am thinking maybe it was left in one of the classroom computers’ Zotero libraries and accidentally got dragged into our group library?
  • Nine have added original annotations.
  • Another handful are waiting on their articles from ILL.

Most of these have wonky notes/ attachments from the databases, and some need some of their metadata cleared up.  Batting 500+, though, was more than I expected at this point.  Why?  A few reasons, actually -

  1. First, these students have never used Zotero before at all.  Most of them have never used any kind of Firefox plugin.  That whole process of downloading and installing Firefox, then the plugin, was conceptually something new.  I expected this to be a hurdle in and of itself, before we even got to the the group library and syncing piece of the puzzle.  And it was, for sure, for some.  But not for most – most got themselves set up with Firefox no problem, and got the plug in working just fine.
  2. I want to be really clear here – it’s not that I thought these students weren’t intelligent enough to do this nor did I think it was really hard – I just thought it was going to be new and made more difficult by the fact that I asked them to do most of this new thing on their own on their own computers.  I did this mostly because I wasn’t at all certain that syncing the classroom computers to the Zotero group library would work with any kind of reliability.  So it comes down to -  I thought that showing them in class and then asking them to do the work at home was not necessarily setting them up for success (for all that that is how homework usually works).
  3. I really didn’t give them much instruction on how to do this at all.  We went over Zotero on the first day of class, and then I asked them to test different features of it along the way.  But here’s the thing – most of them didn’t do that along the way stuff because I wasn’t grading it and it wasn’t on the syllabus.  It was mostly a “please do this for your own good” thing and wasn’t at the top of anybody’s priority list.  So that .500+ batting average comes from students figuring stuff out with the tutorial I provided and what they could find in their notes and on the Zotero website.
  4. Some of the problems that have happened are undoubtedly not about Zotero at all, but are about navigating library systems and databases and the difficulties that come up during the process of finding scholarly articles — those are the primary reason for this class, after all!
  5. The syncing with the classroom computers is working really well – or at least it has for the last two sessions.  I have to tell you that I was worried about this with good reason.  Every time I have attempted to show this in the classroom, the sync has churned and churned and churned without any end (or any sync) in sight.  So when the students were having no trouble syncing the Zotero libraries in the classroom to their group accounts in class two weeks ago and again last week, I was shocked.  But what this means is that this week we can treat the classroom like a lab and troubleshoot most of the remaining problems together.

Onward!

Citations 101

This term our first-year seminar/orientation classes (called U-Engage) have given me the opportunity to do some different things, teaching-wise.  One of the sections asked for resources for a “Citations 101″ unit.  This is what I’ve put together so far.

Does this work because it has a workable focus, and because it treats citing like something that has value, instead of something to do to avoid getting in trouble (or because no one respects the ideas of college students, which is a message that I think some students take away from lessons about the rhetorical uses of outside sources).  I do think the rhetorical uses are crucial, but they were beyond the scope here – and I think would have taken the focus beyond “workable.”

ALS 199: Citations 101

(Built with our Library a la Carte tutorials extension.)

Zotero group bibliography assignment

I decided before the start of this term, the first term in which I would be teaching a credit class in almost eight years (and I’m teaching 2!) that my Library Skills for English Majors class would collaboratively create an annotated bibliography in Zotero for their main group project.

I want them to develop some facility with Zotero, and this seems like a good way to do this.  The ins and outs of working with metadata on Zotero connects back to a lot of the course themes, making even those that are a little abstract seem more concrete.  At least I hope so.

I’ve barely explored the Zotero group settings for all that I have been there for a while (and for all that I have group libraries and everything) so I was not at all sure how well it would work or even if it would work for students in this class.  I’m still not sure because I’d like them to do a lot of the work in-class, and they don’t have their own computers there.  It should be possible for them to sync what they do on library machines to their online libraries, but until we try it, I just won’t know.

So yeah, that’s the reason why I was so happy to find that I’m not the first person to try this -(Profhacker author) Brian Croxall at Clemson did it before and he did it for English and he wrote about it extensively which is so amazingly awesome.  I drew heavily from it even when it didn’t directly – it’s amazing how working with someone else’s assignment online is like talking it through, having someone to get you thinking about the stuff you’re forgetting.

Anyway, so the theme they’re building this bibliography around is the scholarly and creative output of their own faculty,  This is only a 1-credit class (more on the challenges of doing anything meaningful in a 1 credit class later, I promise) and they don’t have a common research assignment in other classes (or any research assignment, in many cases) so it’s really hard to make it relevant.  I am hoping that this focus will add a note of relevance to a kind of abstract skills-for-skills-sake class.  I am also fascinated by what our faculty are producing and will enjoy what the students find and choose to add in any event.

The full text of the assignment is under the cut.

Continue reading

tutorials a la carte

So the thing about DIY tutorials is that even though the technology bar required to build them is pretty low, there are some back-end logistical issues that have to be navigated.  When I decided I wanted to try these with some of my distance learning classes, I had to figure out how to create a locally hosted blog, and we ended up doing it in a way that hadn’t been tried before (using Drupal) so we had to go through some rounds of figuring out settings and styling.

Don’t get me wrong, this wasn’t a bad thing to go through for me, or for the library.  And when colleagues wanted a similar tool, we were that much further down the road and had some good reasons to try a different tool.  Still, when the goal is “I want to get a tutorial up” these kinds of barriers aren’t necessarily what you want to go through.

I realize that a local option wasn’t the only option – if all I cared about was putting up a blog for tutorials quick and fast there are obviously tons of options out there.  But that wasn’t the only problem I wanted to solve.  I didn’t just want to make tutorials myself, I wanted to do something that would make it easy for all of us to create tutorials and other instructional aids.

The issue of how to share the burden of creating instructional materials has been around for a long time.  Learning objects repositories like MERLOT offer lots of options for teachers – the ability to borrow online instructional modules, and the ability to comment on and rate those tools.  Shared tutorial projects like ANTS let librarians share tutorials, collaborate on prioritizing and creating tutorials, and also provide a social space to talk about tutorial-related issues.  Here, locally, we have  CLIP (Collaborative Library Instruction Project) which takes a good step in the right direction – making source files available and shareable, letting librarians share some but not all of the work of tutorial-building.

CLIP is just getting off the ground, so I don’t know what will happen there.  But here’s the thing.  Whenever I have talked to anyone involved with projects like these, or heard them speaking about the larger issues, they say the same thing – getting participation from educators is really, really hard.  They spend a lot of time gathering the content for the repositories themselves, they have a core group of committed people sharing, but getting the concept of sharing — of taking the time to participate in these types of projects and communities – to be part of the normal workflow for teachers is super difficult.

My own experience with the Library Instruction Wiki (now offline due to basement server room flood) left me feeling pessimistic about that project’s stop reinventing the wheel tagline.  I have come to believe some things – some things that got me thinking that instead of sharing the final product of tutorials, the way to go is to figure out ways groups of teaching librarians can share a process:

1. Teachers like reinventing the wheel.

This is something Jean Caspers said to me once, and its stuck with me because I think it’s essentially true.  Not that teachers don’t want to borrow and adapt and take advantage of other people’s cool ideas and good work, but to really feel comfortable going into a classroom and teaching a group of students something, a lot of us need to feel like we’ve made the stuff we’re teaching ours.  And the only way to do that is to adapt, and reshape, and refine.  So we don’t want to just point to other people’s handouts and tutorials (sometimes we do, but go with it).  We want to make them ours.

And an important part of the cognitive process that a lot of us go through preparing to teach is preparing the materials.  I know that when I create a course page for a class, I’m thinking about how I am going to present the material in class, and about how I am going to transition from one topic to the next.  And when I make a tutorial for an online class, I want to tailor it to their assignment, and the process of putting it together helps me clarify what they need to know/do for that assignment.

So, it’s not a bad thing that teachers want to reinvent the wheel, not at all.  And I think it’s a need teachers have that should be considered by anyone trying to help them work collaboratively.

2. The hard part of developing tutorials isn’t technical, it’s in the content.  And that’s the hardest piece to share.

One piece of this is practical – try teaching anyone how to do anything research-related on your campus and see how long it takes before there is some local quirk that you need or want to explain.  It doesn’t take long.  There’s a reason why the tutorials we’re most likely to share are on topics like plagiarism or citations — things where we are all working on basically the same standards and rules, defined outside of our local institutions.

One piece of this is related to the above point.  We want the content to be tailored to the students’ needs – we have strong opinions about how to do research and about how to teach it and about what students need to know about it.  I don’t know, I haven’t talked to many people who say things like “I would totally borrow X’s tutorial, but it’s built in Camtasia and I would rather it be in Captivate. The people I talk to are more likely to forgo borrowing a tutorial because they don’t like something about it — they’d like to change the way it explains a concept, or to add just one more piece.

3.  There is one sure-fire way to make the technical part hard – that is to tell everyone they have to use the same tool at every stage of the process.

Now this may seem to undercut the whole process-sharing thing I mentioned above but bear with me – I’m really talking about all of the component parts of the final product.  If you tell everyone that they have to use the same tool to build webpages, that’s going to leave you with a few people upset yes.  You’ll also have a lot of people that don’t really care.

But if you tell them that they have to use the same thing to take screenshots, the same thing to do screencasts, the same thing to create word clouds, or to display bookmarks, or to push useful links — then you’re going to start getting the kind of resistance that makes people decide not to create the thing at all.  This is especially true if people have been left on their own to figure out their own best way to do those things before.

So, to get to the point already…

So, Hannah and I were working on redoing our big tutorial – and one of the problems we wanted to address was the bottleneck that occurs when only a few people can edit or make changes to a tool.  We also wanted to de-Blackboard our beginning composition assignments, and make them more lightweight and dynamic.  Thinking to kill two birds with one stone, we decided to look at content-management-izing our tutorial building process.

The brain trust behind Library a la Carte (an open-source, lightweight CMS for building course pages and subject guides) works down the hall from us, so we had a place to go with this problem.  We were initially open to a variety of approaches (including Drupal and WordPress) for building these tutorials, but we ended up deciding that extending LALC to include a tutorial-building function made the most sense.

We launched it barely-alpha, with the fall term beginning composition students.  We found a bunch of things that wanted fixing, but even in this sub-optimal situation, there was an awful lot of it that worked well.   Now it is in barely-beta, and with about 40 sections of beginning composition using it, the reports we have had about it not working for the students have been in the single digits.

This is what the tutorials look like:

(Here’s a link if you want to look at the whole tutorial.  This one is cool because it has modules that feature: images, cartoons, videos and text included.  You have to log in to see the quizzes, but not to see the rest)

There’s a few things I really like about it -

First, it allows our teaching librarians to share at a pretty granular (modular) level – I can borrow Hannah’s catalog-using screencast, and put it into one tutorial that is really about how distance students can have books delivered to them by mail, and also into another tutorial that is really about the serendipitous process of browsing for books on the shelves.

And even at that granular level, it lets us borrow-and-then-tweak — keeping things right in most teachers’ comfort zone.

Using an incredibly scientific data gathering method (n=1; n=me) I have determined that instruction librarians just may be more likely to borrow from each other and to remember to share with each other in this format.

Secondly, it solves the problem of where everything is going to live.  Because it is integrated with our subject and class guides, the system lets us create tutorials and then automatically puts them on the website, with no pesky decisionmaking steps beyond what to call the thing.  Which admittedly, can stymie me for a while, but…

Using the same data collection method (this time n=me and Hannah) I have determined that with this tool available, we are more likely to include tutorials and learning modules in the things we do for our classes, whether we meet with them face-to-face or not.

It also lets the librarian pull in content from elsewhere, so they can use any method they are comfortable with to create the content initially.  If I want to store my photos on flickr, and edit with Picnik, that works fine. But so does uploading the photos that I edited with Photoshop.  If I want to embed a video I edited with iMovie, I do the same thing I would do to drop in a screencast video I created with Camtasia.

Editing a video module:

It also lets us easily use dynamic content (which sometimes breaks the styling but, live and learn) so if I want to embed a delicious linkroll to recommend links to three different classes of students, I can, and then of course I can update all 3 class pages at the same time when I update my Delicious.  Or if I want to do the same thing by embedding a Twitter feed, no problem.

Again, because this is integrated with our class pages system, we’ve already made a lot of decisions about where to share some of our resources.  But even though we have a flickr archive, lots of us don’t use that for building the modules.  And even though we have a YouTube channel, the videos there can be created in lots of ways.

I have no idea at this point if anyone else is going to start using this, or if it will mostly be used to update our main tutorial and in beginning composition.  But I do like the concept of process-sharing, and I think this might be a way that idea makes sense.