Peer Assessment in Online Education: Training Students to Know What's 'Good'
By Tiffany Fox, (858) 246-0353, tfox@ucsd.edu
San Diego, Calif., Nov. 20, 2013 — Is it possible to teach classes with creative, open-ended projects on a massive scale?
|
Difficult, but not impossible, argues the University of California, San Diego's Scott Klemmer. Massive online courses benefit from the 'wisdom of the crowd,' which can actually enhance a crucial element of creative coursework: peer assessment.
The challenge, says Klemmer, is that peer assessment requires students to know what ‘good’ is, and that they have a way of providing feedback in an effective way – something that can be difficult when students are sitting behind computer screens scattered all over the world. The path is smoother now that he and his colleagues have developed a set of best practices that have been used in more than 100 massive online classes, from world music to nutrition to mathematical thinking..
Klemmer recently joined the faculty of the Cognitive Science and Computer Science departments at UC San Diego, where he’s leading a research effort at the Qualcomm Institute (QI) called Design at Large to empower more people to design, program, learn and create using online media. Klemmer’s research is informed by his collaboration with Coursera in 2012 to launch the first massive-scale class — a design course — with self and peer assessment.
“My middle school math teacher used to tell us that teaching is learning,” says Klemmer, who will speak on peer assessment and online learning as part of the Design at Large lecture series tomorrow at 3 p.m. in the Calit2 Auditorium in Atkinson Hall.
“The first person you teach is yourself, and the second is your friend. That’s essentially what self- and peer-assessment are. But they’re surprisingly difficult to do well and there are many subtle pitfalls.”
To address these pitfalls, Klemmer and his colleagues developed a set of Seven Habits for Highly Effective Peer Assessment, which he says are equally useful in both online and in-person classes. Klemmer will elaborate this Friday on the “Seven Habits”, which include:
- Assignment-specific rubrics and training
- Iteration before release (pre and during)
- Assessing self after peers
- Using staff grades as ‘ground truth’
- Aggregating grades adaptively
- Using cued prompts to offer written feedback
- Closing the loop; giving assessors’ feedback
In sum, providing training to students on how to assess one another is one of the most important elements of any course that adopts peer review, says Klemmer. For this reason, he and others begin their online courses with a “calibration” phase that requires students to rate examples that have already been assessed, and then train students further by comparing their assessments to assessments given by teaching staff (i.e. ‘ground truth’).
“The greatest transformation that comes about in teaching with peer assessment is that the instructor has to articulate what good is,” Klemmer explains. “For most classes the knowledge about what excellent work is is implicit in the professor’s head. A radical transformation that happens with peer assessment is that it forces this change.”
Klemmer adds that it’s also essential to use an assessment rubric for each assignment. Assignment-specific rubrics “help scaffold students’ ability to assess excellence on a particular topic,” he notes.
“While experts know how to generalize principles to novel situations (that’s what makes them experts) novices don’t. If you’re asking somebody to assess, you’re asking them to behave with expertise. One reason people worry about peer assessment is a worry that we’re letting the inmates run the asylum, so to speak. People ask how it’s possible that students can do assessments, and the answer is: they’re learning the skills for assessment by doing it.”
That said, Klemmer emphasizes it’s difficult to write rubrics that “express and honor the diversity of student creativity without being overly vague or overly trite.”
“And that’s why classes that use peer assessment well have been honing it for years,” he adds. “Every time I teach, after every assignment, I meet with my teaching staff to debrief one another, and based on the successes or struggles people have or cool ideas students come up with, we revise the assignment and rubric for the next offering. The materials I used have evolved over a half dozen years now through six offerings of an in-person class and four online, and I’m amazed by how much they’re still evolving.
“It’s really hard and takes several years to build learning materials that are appropriate for your domain,” adds Klemmer, “but it’s extremely rewarding. Students report peer assessment to be one of the most valuable aspects of a course.”
In his talk this Friday, Klemmer will also discuss other ways to take online education from a platform where students learn “alone together” to one that includes small group video discussion, social networks and global meet-ups — techniques that can be used for any massive online class. The talk is part of a series organized by QI’s Technology Enhanced Learning initiative and has also featured talks on “Digital Rhetoric and Institutional Reform” (Elizabeth Losh, UCSD Sixth College); “Understanding and Bridging the Wikipedia Language Gap” (Darren Gergle, Northwestern University); and “The Power of Friends in Big Data” (James Fowler, UCSD). Marti Hearst of UC Berkeley will give the final talk in the series, which takes place Dec. 5.
To see the other lectures in the series, visit Calit2ube.
Media Contacts
Tiffany Fox, (858) 246-0353, tfox@ucsd.edu
Related Links