Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos

research paper summary
How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Philip J. Guo, Juho Kim, Rob Rubin. ACM Conference on Learning at Scale, 2014.
Videos are a widely-used kind of resource for online learning. This paper presents an empirical study of how video production decisions affect student engagement in online educational videos. To our knowledge, ours is the largest-scale study of video engagement to date, using data from 6.9 million video watching sessions across four courses on the edX MOOC platform. We measure engagement by how long students are watching each video, and whether they attempt to answer post-video assessment problems.

Our main findings are that shorter videos are much more engaging, that informal talking-head videos are more engaging, that Khan-style tablet drawings are more engaging, that even high-quality pre-recorded classroom lectures might not make for engaging online videos, and that students engage differently with lecture and tutorial videos.

Based upon these quantitative findings and qualitative insights from interviews with edX staff, we developed a set of recommendations to help instructors and video producers take better advantage of the online video format.
@inproceedings{GuoLAS2014vidprod,
 author = {Guo, Philip J. and Kim, Juho and Rubin, Rob},
 title = {How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {41--50},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566239},
 doi = {10.1145/2556325.2566239},
 acmid = {2566239},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {mooc, online education, video engagement},
}

Videos are central to the learner experience in many MOOCs:

The above figure shows four main kinds of videos on the edX platform: a.) a recorded classroom lecture, b.) an instructor's talking head, c.) a Khan-style digital tablet sketch drawing (popularized by Khan Academy), and d.) a PowerPoint slideshow.

Since we're now in the early days of the widespread global popularity of video-based MOOCs, we were curious about the following question: How does MOOC video production style affect student engagement?

Why care about engagement? Because it's often a prerequisite for learning. Although it's very hard to assess actual learning from MOOC server logs without designing an experiment to measure learning outcomes, engagement is something that can reasonably be measured at scale on existing MOOC data.

For this study, we measured engagement by how long students watched each video and also by whether they attempted to answer post-video assessment problems. (Note: We can't measure true learner engagement at scale, since we don't know how much learners were actually paying attention to these videos while watching, but these metrics are a reasonable first proxy.)

We took all 862 videos from four edX courses offered in Fall 2012 (Intro. CS, Statistics, Artificial Intelligence, and Solid State Chemistry) and hand-classified each one based on its type (e.g., traditional lecture, problem-solving tutorial) and production style (e.g., PowerPoint slides, Khan-style tablet sketch drawing, talking head). We automatically extracted other features such as length and speaking rate (words per minute). We then mined the edX server logs to obtain over 6.9 million video watching sessions from almost 128,000 students. To our knowledge, this is the largest-scale study of video engagement to date.

Findings and Recommendations

Here are our seven main findings and corresponding recommendations for creators of online educational videos:

1. Shorter videos are much more engaging. Engagement drops sharply after 6 minutes.

Recommendation: Invest heavily in pre-production lesson planning to segment videos into chunks shorter than 6 minutes. This is the most significant recommendation!

2. Videos that intersperse an instructor's talking head with PowerPoint slides are more engaging than showing only slides.

Recommendation: Invest in post-production editing to display the instructor's head at opportune times in the video. But don't go overboard because sudden transitions can be jarring. Picture-in-picture might also work well.

3. Videos produced with a more personal feel could be more engaging than high-fidelity studio recordings.

Recommendation: Try filming in an informal setting such as an office to emulate a one-on-one office hours experience. It might not be necessary to invest in big-budget studio productions.

4. Khan-style tablet drawing tutorials are more engaging than PowerPoint slides or code screencasts.

Recommendation: Introduce motion and continuous visual flow into tutorials, along with extemporaneous speaking so that students can follow along with the instructor's thought process.

5. Even high-quality prerecorded classroom lectures are not as engaging when chopped up into short segments for a MOOC.

Recommendation: Some MOOC videos are created by simply chopping up old classroom lecture videos (often recorded in prior years) and putting them online. Those videos are less engaging than ones recorded specifically with the MOOC format in mind. But if instructors still insist on recording traditional classroom lectures, they should purposefully plan lectures with the MOOC format in mind and work closely with instructional designers who have experience in online education.

6. Videos where instructors speak fairly fast and with high enthusiasm are more engaging.

Recommendation: Coach instructors to bring out their enthusiasm and reassure them that they do not need to purposely slow down. Students can always pause the video if they want a break.

7. Students engage differently with lecture and tutorial videos.

Recommendation: For traditional lectures, focus more on the first-time watching experience. For step-by-step tutorials, add more support for rewatching and skimming, such as inserting subgoal labels in large fonts throughout the video.


Read the full paper for details:

How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Philip J. Guo, Juho Kim, Rob Rubin. ACM Conference on Learning at Scale, 2014.
Videos are a widely-used kind of resource for online learning. This paper presents an empirical study of how video production decisions affect student engagement in online educational videos. To our knowledge, ours is the largest-scale study of video engagement to date, using data from 6.9 million video watching sessions across four courses on the edX MOOC platform. We measure engagement by how long students are watching each video, and whether they attempt to answer post-video assessment problems.

Our main findings are that shorter videos are much more engaging, that informal talking-head videos are more engaging, that Khan-style tablet drawings are more engaging, that even high-quality pre-recorded classroom lectures might not make for engaging online videos, and that students engage differently with lecture and tutorial videos.

Based upon these quantitative findings and qualitative insights from interviews with edX staff, we developed a set of recommendations to help instructors and video producers take better advantage of the online video format.
@inproceedings{GuoLAS2014vidprod,
 author = {Guo, Philip J. and Kim, Juho and Rubin, Rob},
 title = {How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {41--50},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566239},
 doi = {10.1145/2556325.2566239},
 acmid = {2566239},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {mooc, online education, video engagement},
}

(This article was adapted from my post on the edX blog.)

Created: 2014-03-12
Last modified: 2017-10-02