Image by Wokandapix from Pixabay

So last month when I did a webinar for the GLA Carterette Series on some of my ideas for incorporating the contextual nature of research into information literacy instruction, there were a lot of great questions at the end about assessment. In answering them, I realized that this was something of a hole in my discussion of this topic and I wanted to see if I could address some of it here.

First, it might help to know why assessment is such a blind spot for me. Basically, the culture around assessment in my current institution is a lot different from what I think the norm is for most libraries. I experienced something closer to that norm at my previous institution, where we were asked to constantly assess student learning and some part of the library’s value (not to mention our value as a reference and instruction department within the library) was directly tied to our program-level learning outcomes and how well our students met those outcomes. All of this was, in turn, very closely tied to questions of student retention and the role the library played in the institution’s retention efforts.

Where I am now, there is certainly interest in making sure that what we teach contributes toward student learning and student retention. And there are conversations about finding a way to assess our teaching in order to speak to our value both in the library and on campus. But because instruction responsibilities here are so fragmented, any assessment effort on this level would require buy-in across several departments in the library. As you can imagine, there would be some difficulty there. For now, everyone just kind of does their own thing. That’s been a big part of what’s allowed me to take more creative approaches to my teaching, which is an aspect of my job that I’m very grateful for.

But these more creative approaches aren’t exactly useful if students don’t learn anything as a result. Hence: it’s time to talk about assessment.

I’ve mentioned before that part of the reason the ACRL Standards focused on basic research skills was because those are the things we can assess. It’s much easier to assess whether a student can successfully identify a scholarly source in a library database than it is to assess a change in their way of thinking. How do you measure something like that?

Of course, this is a question we’ve all been struggling with to one degree or another since the advent of the ACRL Framework, which uses threshold concepts instead of learning outcomes. Threshold concepts are literally all about changing someone’s way of thinking.

Teaching about the contextual nature of research is in a large sense about changing the way students think about research. It’s asking them to recognize that the conventions and methods of research are going to be different depending on the context in which research is taking place. Not just disciplinary contexts, but contexts outside of academia as well.

No matter what context of research you’re working with, there are going to be skills involved. So one idea for assessing the contextual nature of research is to determine what the skills associated with the context(s) you’re teaching are and assessing students’ ability to not only perform those skills but recognize the appropriate context for those skills. For example, if a student is searching for or citing a peer-reviewed source when you’ve asked them to perform the type of research associated with a non-scholarly or non-academic context, they’re showing that they have good research skills but that they’re not applying them to the correct context.

This is something that can be captured in a number of ways. You can observe a student’s information behavior to judge whether it’s appropriate to a given task. You can have the student create a research product and judge how well they show awareness of the conventions of a particular type of research. You can create a video that explains the conventions of a particular research context and then quiz students on their understanding of what they watched.

Of course, being able to judge whether students are using skills and following conventions appropriate to a particular context requires establishing what those appropriate skills and conventions even are. Not to mention establishing what the contexts of research might be.

In my own work, I’ve suggested a few very broad categories or “genres” of research, including academic, scholarly, personal, professional, scientific, and creative research. I even outlined some of the characteristics of these genres in my article introducing these ideas. But this outline was meant to illustrate a point rather than act as a guide. Clearly, more work needs to be done here.

But that doesn’t mean you can’t teach the contextual nature of research until that work is done.

In my own classes, I have quizzes that students take after reading or listening to a lecture that I’ve written on a given topic (it’s an online class). These lectures address the contextual nature of research in mostly general terms and I test students’ understanding of this concept by including questions like the following on the associated quiz:

What type of research are scholarly, peer-reviewed articles most appropriate for?

• Academic/scholarly research
• Personal research
• Professional research
• Creative research
• All research, no matter the context

 

It’s a simple question that tells me a lot about how much students understand about this concept even without a lot of specifics about the conventions of each type of research. Students who get it right have shown me what they’ve learned. Students who get it wrong—like the surprising number who try to argue that peer-reviewed sources should be considered appropriate for all types of research because their other professors have always told them that they are the “gold standard” of credibility even after I’ve told them all the reasons this isn’t actually the case—show me that there’s still a ways to go before they cross that threshold of understanding.

I also had an experience recently where I participated on a committee whose charge was to create and implement a library research award for undergraduate students. As part of that work, the committee had to come up with a way to evaluate the work we were seeing, which could come from any discipline being studied on campus. We wanted to make sure the award process was open not just to students who had completed standard research papers but also those who had done research in connection to more creative projects and we needed a rubric to reflect that.

We ended up adapting a rubric (with permission) from one that had been used by several other institutions. But where the original rubric mentioned skills appropriate to a particular discipline, we substituted the phrase “appropriate to the context.”  That might seem like a small change, but not all research takes place within an academic discipline. We also wanted to make sure that students who had conducted their research in more creative contexts knew that they were eligible for the award as well. Either way, the wording is a way to capture that an excellent research project is one in which the student applies skills and conventions appropriate to the context of the research.

So there’s not as much concrete information about assessment here as I would like. Like I said, assessment tends to be a little under my radar for a variety of reasons but this is something I’m going to continue to think about and share some thoughts on in the future. If anyone else has thoughts, I’d be interested in hearing those as well.