Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

HappyFace: Identifying and Predicting Frustrating Obstacles for Learning Programming at Scale

research paper summary
HappyFace: Identifying and Predicting Frustrating Obstacles for Learning Programming at Scale. Ian Drosos, Philip J. Guo, Chris Parnin. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2017.
Unnecessary obstacles limit learning in cognitively-complex domains such as computer programming. With a lack of appropriate feedback mechanisms, novice programmers can experience frustration and disengage from the learning experience. In large-scale educational settings, the struggles of learners are often invisible to the learning infrastructure and learners have limited ability to seek help. In this paper, we perform a large-scale collection of code snippets from an online learn-to-code platform, Python Tutor, and collect a frustration rating through a light-weight learner feedback mechanism. We then devise a technique that can automatically identify sources of frustration based on participants labeling their frustration levels. We found 3 factors that best predicted novice programmers' frustration state: syntax errors, using niche language features, and understanding code with high complexity. Additionally, we found evidence that we could predict sources of frustration. Based on these results, we believe an embedded feedback mechanism can lead to future intervention systems.
@inproceedings{DrososVLHCC2017,
  author={Drosos, Ian and Guo, Philip J. and Parnin, Chris},
  title={{HappyFace}: Identifying and Predicting Frustrating Obstacles for Learning Programming at Scale},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '17},
  year={2017},
  month={Oct}
}

Wouldn't it be awesome if a programming instructor or course designer could magically peek into the minds of learners to see exactly when they're getting frustrated? That way, they can know what kinds of code are confusing for learners, when to best intervene to help out, and how they can improve instructional materials in the future.

How might we go about achieving this goal? We could attach electrodes to everyone's heads and monitor their brain activity. We could use 3D sensors like a Kinect to detect their body posture and facial expressions. We could attach eye trackers to everyone's computer monitor and use that data to estimate frustration. We could imagine doing all sorts of monitoring, but the problem with all of these ideas is that they're expensive and thus don't scale. How are we ever going to attach these devices onto hundreds of students (or tens of thousands, in the case of MOOCs)? Who will calibrate or maintain them? What if they get lost or stolen?

Instead of relying on high-tech gizmos, we decided to take the most low-tech possible approach: What if we just asked learners to tell us when they're frustrated? This method seems so simple and naive; does it have any chance of working? We created a prototype called HappyFace to find out. HappyFace is a five-level pain scale embedded within the bottom of my Python Tutor code editor (click screenshot to enlarge):

Whenever the learner wants, they can click one of the five faces (ranging from really happy to really sad), and the system will capture their selection, along with a snapshot of the code they're currently working on. The learner can also optionally write in why they feel happy or sad at the moment.

The design of HappyFace was directly inspired by medical pain scales where patients self-report their pain levels to their healthcare providers. Here's an example from the UCSD urgent care clinic that I recently had the pleasure of visiting!

The basic idea behind pain scales is that they provide a simple and intuitive way for people to report how they're feeling at the moment. One of the best ways to scale something is to have it be super-simple with the fewest possible moving parts. That's what HappyFace tries to provide for programming-related materials.

To test whether this straightforward approach can actually identify known causes of programmer frustration, we deployed it to the Python Tutor website for 2.5 months and collected 2,385 self-reported clicks from learners around the world. We analyzed the associated code and found three major factors that correlated with higher frustration levels: syntax errors, using idiosyncratic Python language features, and trying to understand code with high complexity. These three factors are consistent with what prior studies have found to be frustrating for novice programmers, which indicates that our simple approach can collect useful data at a much lower cost than, say, using fancy physical sensors.

In sum, HappyFace provides a simple and scalable foundation on which to build follow-up systems that allow instructors to provide real-time targeted help and to refine their instructional materials using fine-grained data about common learner frustrations.


Read the full paper for details:

HappyFace: Identifying and Predicting Frustrating Obstacles for Learning Programming at Scale. Ian Drosos, Philip J. Guo, Chris Parnin. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2017.
Unnecessary obstacles limit learning in cognitively-complex domains such as computer programming. With a lack of appropriate feedback mechanisms, novice programmers can experience frustration and disengage from the learning experience. In large-scale educational settings, the struggles of learners are often invisible to the learning infrastructure and learners have limited ability to seek help. In this paper, we perform a large-scale collection of code snippets from an online learn-to-code platform, Python Tutor, and collect a frustration rating through a light-weight learner feedback mechanism. We then devise a technique that can automatically identify sources of frustration based on participants labeling their frustration levels. We found 3 factors that best predicted novice programmers' frustration state: syntax errors, using niche language features, and understanding code with high complexity. Additionally, we found evidence that we could predict sources of frustration. Based on these results, we believe an embedded feedback mechanism can lead to future intervention systems.
@inproceedings{DrososVLHCC2017,
  author={Drosos, Ian and Guo, Philip J. and Parnin, Chris},
  title={{HappyFace}: Identifying and Predicting Frustrating Obstacles for Learning Programming at Scale},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '17},
  year={2017},
  month={Oct}
}
Created: 2017-10-11
Last modified: 2017-10-11
Related pages tagged as computing education: