Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

How MOOC Video Production Affects Student Engagement

edX research paper

A few months ago I blogged about my summer 2013 research on optimal video length for student engagement. Recently Juho Kim, a Ph.D. student at MIT, Rob Rubin, the VP of Engineering at edX, and I extended that work and published the following paper at the new ACM Learning at Scale conference:

Philip J. Guo, Juho Kim, Rob Rubin. How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. ACM Conference on Learning at Scale, March 2014.
[BibTeX citation]

This blog post summarizes our paper.

Videos are central to the student learning experience in many kinds of MOOCs.

The above figure shows four main kinds of videos on the edX platform: a.) a recorded classroom lecture, b.) an instructor's talking head, c.) a Khan-style digital tablet drawing (popularized by Khan Academy), and d.) a PowerPoint slideshow.

Research Question: How does video production affect student engagement in MOOCs?

Methodology: We measured engagement by how long students watched each video and also whether they attempted to answer post-video assessment problems.

We took all 862 videos from four edX courses offered in Fall 2012 and hand-classified each one based on its type (e.g., traditional lecture, problem-solving tutorial) and production style (e.g., PowerPoint slides, Khan-style tablet drawing, talking head). We automatically extracted other features such as length and speaking rate (words per minute). We then mined the edX server logs to obtain over 6.9 million video watching sessions from almost 128,000 students.

To our knowledge, this is the largest-scale study of video engagement to date.

Findings and Recommendations

Here are our seven main findings and corresponding recommendations for creators of online educational videos:

1. Shorter videos are much more engaging. Engagement drops sharply after 6 minutes.

Recommendation: Invest heavily in pre-production lesson planning to segment videos into chunks shorter than 6 minutes. This is the most significant recommendation!

2. Videos that intersperse an instructor's talking head with PowerPoint slides are more engaging than showing only slides.

Recommendation: Invest in post-production editing to display the instructor's head at opportune times in the video. But don't go overboard because sudden transitions can be jarring. Picture-in-picture might also work well.

3. Videos produced with a more personal feel could be more engaging than high-fidelity studio recordings.

Recommendation: Try filming in an informal setting such as an office to emulate a one-on-one office hours experience. It might not be necessary to invest in big-budget studio productions.

4. Khan-style tablet drawing tutorials are more engaging than PowerPoint slides or code screencasts.

Recommendation: Introduce motion and continuous visual flow into tutorials, along with extemporaneous speaking so that students can follow along with the instructor's thought process.

5. Even high-quality prerecorded classroom lectures are not as engaging when chopped up into short segments for a MOOC.

Recommendation: If instructors insist on recording traditional classroom lectures, they should still plan lectures with the MOOC format in mind and work closely with instructional designers who have experience in online education.

6. Videos where instructors speak fairly fast and with high enthusiasm are more engaging.

Recommendation: Coach instructors to bring out their enthusiasm and reassure them that they do not need to purposely slow down. Students can always pause the video if they want a break.

7. Students engage differently with lecture and tutorial videos.

Recommendation: For lectures, focus more on the first-time watching experience. For tutorials, add more support for rewatching and skimming, such as inserting subgoal labels in large fonts throughout the video.

Read the full paper for details.

Created: 2014-03-12
Last modified: 2014-04-06