Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

Demographic Differences in How Students Navigate Through MOOCs

research paper summary
Demographic Differences in How Students Navigate Through MOOCs. Philip J. Guo and Katharina Reinecke. ACM Conference on Learning at Scale, 2014.
The current generation of Massive Open Online Courses (MOOCs) attract a diverse student audience from all age groups and over 196 countries around the world. Researchers, educators, and the general public have recently become interested in how the learning experience in MOOCs differs from that in traditional courses. A major component of the learning experience is how students navigate through course content.

This paper presents an empirical study of how students navigate through MOOCs, and is, to our knowledge, the first to investigate how navigation strategies differ by demographics such as age and country of origin. We performed data analysis on the activities of 140,546 students in four edX MOOCs and found that certificate earners skip on average 22% of the course content, that they frequently employ non-linear navigation by jumping backward to earlier lecture sequences, and that older students and those from countries with lower student-teacher ratios are more comprehensive and non-linear when navigating through the course.

From these findings, we suggest design recommendations such as for MOOC platforms to develop more detailed forms of certification that incentivize students to deeply engage with the content rather than just doing the minimum necessary to earn a passing grade.
@inproceedings{GuoLAS2014navigation,
 author = {Guo, Philip J. and Reinecke, Katharina},
 title = {Demographic Differences in How Students Navigate Through MOOCs},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {21--30},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566247},
 doi = {10.1145/2556325.2566247},
 acmid = {2566247},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {massive open online course, mooc, navigation strategy, non-linear learning},
}

Millions of students from around 200 countries are now learning on MOOC platforms such as edX. Do student demographics such as age and country of origin affect how they navigate through these courses?

To explore this question, Katharina Reinecke and I performed an empirical study and published a paper at the new ACM Learning at Scale conference. This article summarizes our paper.

Methodology

We extracted demographic information and navigation patterns of over 110,000 students taking four edX courses in Fall 2012, 10% of whom earned certificates of completion.

We focused on demographic variables such as age, education level, and country of origin. One interesting variable is the student-teacher ratio of the country of origin. This is a widely-used indicator of educational quality and level of individualized student attention. (However, it is also correlated with economic indicators such as per-capita GDP and median household income, so any observed effects might not be due simply to pedagogical quality.)

One interesting form of navigation is a backjump, which occurs when a student navigates backwards from a resource to another one that was released earlier in the term (e.g., jumping from Lecture 6 to Lecture 3).

Main Findings

EdX students come from diverse backgrounds and navigate through the courseware in many different ways. In particular:

  1. Certificate-earning students are, on average, more highly educated than the general student population.

  2. Certificate-earning students view only 78% of learning sequences (e.g., lectures or assessments). This means that they skip 22% of the course content, on average, and yet still earn a certificate.

  3. Certificate earners engage in lots of non-linear navigation behavior, often jumping backward to revisit earlier lectures.

  4. Backjumps from assessments to lectures are more common than lecture-to-lecture backjumps. One possible explanation is that students are opportunistically working backward to learn old content in order to pass the assessments.

  5. Older students participate more actively in discussion forums.

  6. Older students visit and repeat more learning sequences than younger students.

  7. Students from countries with lower student-teacher ratios (e.g., the U.S. and Eastern European countries) visit and repeat more learning sequences than those from countries with higher student-teacher ratios (e.g., India, Kenya). See the graph below for details.

  8. However, the effect of age is stronger than that of country: Older students from countries with higher student-teacher ratios behave more like their similarly-aged counterparts in countries with lower student-teacher ratios.

The graph above (click to enlarge) shows the mean number of backjumps per visited sequence for all certificate-earning students in the 30 countries with the most edX students. There is a negative correlation between student-teacher ratio and backjumps. One possible explanation is that students from countries with a higher student-teacher ratio are more accustomed to a linear, teacher-led instructional style and hence do not make as many backjumps when taking MOOCs.

Design Implications

Right now, MOOC user interfaces try to be “one size fits all,” but we have found that students navigate quite differently based on both demographics and enrollment intent. Here are some interface design ideas that could help MOOCs serve a broader audience:

  • To encourage greater engagement with learning sequences, MOOCs should augment certificates (which indicate only pass/fail) with richer metrics such as discussion forum participation, peer ratings, or time-on-task. When properly designed, these new reward mechanisms could motivate students to cover more sequences and not just do the minimum required to get a certificate.

  • MOOC user interface designers could experiment with adaptive interfaces that adjust based on, say, a student's age or country of origin. Prior educational technology research has shown that certain types of students prefer a more linear structure and explicitly stated learning goals while others prefer more freedom in navigation.

  • Social dashboards (e.g., “other learners usually spent 45 minutes on this part”) could motivate students and keep them on track as they progress through a several-month-long MOOC. However, these systems must be carefully designed so as not to alienate students from certain demographics.

One ideal to strive for is to present each student with a personalized user interface catered to their individual preferences and learning needs.


Read the full paper for details:

Demographic Differences in How Students Navigate Through MOOCs. Philip J. Guo and Katharina Reinecke. ACM Conference on Learning at Scale, 2014.
The current generation of Massive Open Online Courses (MOOCs) attract a diverse student audience from all age groups and over 196 countries around the world. Researchers, educators, and the general public have recently become interested in how the learning experience in MOOCs differs from that in traditional courses. A major component of the learning experience is how students navigate through course content.

This paper presents an empirical study of how students navigate through MOOCs, and is, to our knowledge, the first to investigate how navigation strategies differ by demographics such as age and country of origin. We performed data analysis on the activities of 140,546 students in four edX MOOCs and found that certificate earners skip on average 22% of the course content, that they frequently employ non-linear navigation by jumping backward to earlier lecture sequences, and that older students and those from countries with lower student-teacher ratios are more comprehensive and non-linear when navigating through the course.

From these findings, we suggest design recommendations such as for MOOC platforms to develop more detailed forms of certification that incentivize students to deeply engage with the content rather than just doing the minimum necessary to earn a passing grade.
@inproceedings{GuoLAS2014navigation,
 author = {Guo, Philip J. and Reinecke, Katharina},
 title = {Demographic Differences in How Students Navigate Through MOOCs},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {21--30},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566247},
 doi = {10.1145/2556325.2566247},
 acmid = {2566247},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {massive open online course, mooc, navigation strategy, non-linear learning},
}
Created: 2014-03-12
Last modified: 2017-10-02