Usability Study: Coursera

I trained in Usability Testing yesterday as part of a UX Conference being conducted by Nielsen Norman Group. We formed groups of 3-4 attendees and conducted our own usability study. We only had time for 3 tasks and a single participant. I was surprised at how much I learned from a single session.

The Client

We chose the Coursera website as our test product.  We needed a website that was general enough so that we can pick participants from our roomful of attendees.

The Goal

To find out how effective the client’s website is on getting a potential customer to sign up for an online class.

 The Participant

We had a roomful of people who were attending full-day training classes they discovered online so we actually had the perfect group of participants for our usability test. People who would be open to taking an online class.

Our participant was a female professional who likes mastering courses she studies and is an alumni of UC San Diego.

The Setting

We pretended to be in our usability study lab aka a conference room in our office.  We set up for a moderated in-lab type of study.

The Tasks

We gave our participant 2 tasks that would help us reach our overall goal:

  1. Give 3 adjectives that best describe the homepage of the website. This will tell us about the overall perception of the website.
  2. After completing this Usability Testing course, you want to expand your UX skills.  Find a course that you want to enroll in and fits your schedule. This will show us what obstacles potential customers encounter when trying to enroll in an online class.

We were supposed to have one more task, but our brilliant instructor, Hoa Loranger, said that our second task was broad enough to touch on many of the site’s features that we may not need the third task.

The Facilitation

This was the hard part.  As a facilitator you have to encourage your participant to think out loud – and our participant did very well.  She read the tasks aloud and was very vocal with what she was thinking as she performed the tasks.  The facilitator has to remain silent for the most part.

As the facilitator, you can not help or direct the participant in any way even when you see them struggling.  You cannot answer UX questions from them.  Your task is to observe and note down what you are learning.

The Observers

The observer’s task was to take notes.  Where was the participant navigating to?  What were the issues?  What was the participants saying?  What was happening?  Did the participant get frustrated?  Were they able to complete the task?

It was a bit hard to take down everything we were observing, the participant had a lot of helpful inputs and insight.

The Observations / Findings

  • The Search box was the first tool used to find a course
  • Familiarity with the institution offering the course (UC San Diego in this case, since she was an alumni) led the participant to view more details about a class
  • The specialization flags on certain classes attracted the participant to view more details about a class
  • The number of classes in a specialization discouraged the participant from enrolling in a specialization
  • The Upcoming Session dates and the Commitment section caused confusion.  Actual days of the week when the classes would occur was expected instead of the data given.
  • The information in the course detail page seemed repetitive and the amount of data in the page made it difficult for the participant to find the answer she needed.
  • Not finding the schedule information expected (days of the week of the classes), the participant was encouraged to sign up to find out if doing so will reveal the information to her.

The Recommendation

I realize this is not a complete Usability Study lacking the right amount of participants (4-6), but the issue encountered by the participant garnered enough frustration and confusion to warrant inclusion in the recommendation.

The on-demand nature of the classes needs to be better explained by additional explanatory content in the Upcoming Sessions section or changes in the Commitment content.  For example, changing the Commitment content to say: the length of lecture videos and reading materials are approximately 20 hours.  Adding Course materials available on-demand from at the beginning of content in Upcoming Sessions may help potential students understand how online classes are offered.

Author’s Commentary

I actually am enrolled in a Coursera specialization at the time of this study, which led me to suggest their website when a course website we initially planned to use proved to need a very specific type of user.

I remember finding the concept of class sessions confusing when I first enrolled.  The course materials are available on-demand – whenever you have time to view them.  To fit your schedule, you must have the amount of hours in the Commitment section to devote to watching all the lecture videos, readings, course projects, peer reviews and quizzes.

They introduced an even more confusing feature recently.  Specialization courses now have a subscription-model payment structure.  So instead of paying for 1 class at a time when you want to take the class, you pay a certain amount per month until you finish the specialization or cancel your subscription.

I always only take 1 class at a time and I used to pause for a few months at a time.  Since January, I’ve continuously enrolled in class after class. I want to complete my specialization so this was what I planned to do, but I am now discouraged from resting for a few months before taking the next class.   I am too lazy to cancel the subscription and then resubscribe again.

I’m on my last paid class this month – the Capstone Project is free.  So now I’m confused again.  Should I unsubscribe after my current class or will the subscription automatically end?  Contact customer service, I know.  But these issues should be explained in their content to avoid these confusions!  Time for another usability study?