Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

Codechella, Codeopticon, and Codepourri: The 2015 Trilogy

research paper summary
Codechella: Multi-User Program Visualizations for Real-Time Tutoring and Collaborative Learning. Philip J. Guo, Jeffery White, Renan Zanelatto. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
An effective way to learn computer programming is to sit side-by-side in front of the same computer with a tutor or peer, write code together, and then discuss what happens as the code executes. To bring this kind of in-person interaction to an online setting, we have developed Codechella, a multi-user Web-based program visualization system that enables multiple people to collaboratively write code together, explore an automatically-generated visualization of its execution state using multiple mouse cursors, and chat via an embedded text box. In the past nine months of live deployment on an educational website, people from 296 cities across 40 countries have started 299 Codechella sessions for both tutoring and collaborative learning. 57% of sessions connected participants from different cities. 69% of actions were visualization interactions, which indicates high engagement with program visualizations. Finally, participants showed signs of learning at the lower three levels of Bloom's taxonomy: remembering, understanding, and applying knowledge.
@inproceedings{GuoVLHCC2015codechella,
  author={Guo, Philip J. and White, Jeffery and Zanelatto, Renan},
  title={Codechella: Multi-user program visualizations for real-time tutoring and collaborative learning},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={79-87},
  doi={10.1109/VLHCC.2015.7357201},
  month={Oct}
}
Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming. Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2015.
One-on-one tutoring from a human expert is an effective way for novices to overcome learning barriers in complex domains such as computer programming. But there are usually far fewer experts than learners. To enable a single expert to help more learners at once, we built Codeopticon, an interface that enables a programming tutor to monitor and chat with dozens of learners in real time. Each learner codes in a workspace that consists of an editor, compiler, and visual debugger. The tutor sees a real-time view of each learner's actions on a dashboard, with each learner's workspace summarized in a tile. At a glance, the tutor can see how learners are editing and debugging their code, and what errors they are encountering. The dashboard automatically reshuffles tiles so that the most active learners are always in the tutor's main field of view. When the tutor sees that a particular learner needs help, they can open an embedded chat window to start a one-on-one conversation. A user study showed that 8 first-time Codeopticon users successfully tutored anonymous learners from 54 countries in a naturalistic online setting. On average, in a 30-minute session, each tutor monitored 226 learners, started 12 conversations, exchanged 47 chats, and helped 2.4 learners.
@inproceedings{GuoUIST2015,
 author = {Guo, Philip J.},
 title = {Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming},
 booktitle = {Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '15},
 year = {2015},
 isbn = {978-1-4503-3779-3},
 pages = {599--608},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2807442.2807469},
 doi = {10.1145/2807442.2807469},
 acmid = {2807469},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {computer programming, learning at scale, remote tutoring}
}
Codepourri: Creating Visual Coding Tutorials Using A Volunteer Crowd Of Learners. Mitchell Gordon and Philip J. Guo. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
A popular way to learn is by studying written tutorials. However, tutorials for computer programming can be tedious to create, since a static text-based format cannot visualize what happens as code executes. We created a system called Codepourri that enables people to easily create visual coding tutorials by annotating steps in an automatically-generated program visualization. Using Codepourri, we developed a crowdsourcing workflow where learners who are visiting an educational website collectively create a tutorial by annotating individual steps and then voting on the best annotations. Since there are far more learners than experts, using learners as a crowd is a potentially more scalable way of creating tutorials. Our experiments with 4 expert judges and 101 learners adding 145 raw annotations to Python code show the learner crowd's annotations to be accurate, informative, and containing some insights that even experts missed.
@inproceedings{GordonVLHCC2015codepourri,
  author={Gordon, Mitchell and Guo, Philip J.},
  title={Codepourri: Creating visual coding tutorials using a volunteer crowd of learners},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={13-21},
  doi={10.1109/VLHCC.2015.7357193},
  month={Oct}
}

(This summary was adapted from my 2015 O'Reilly Ideas article, Learning Programming at Scale.)

Millions of people around the world – from aspiring software engineers to data scientists – now want to learn programming. One of the best ways to learn is by working side-by-side with a personal tutor. A good tutor can watch you as you code, help you debug, explain tricky concepts on demand, and provide encouragement to keep you motivated. However, very few of us are lucky enough to have a tutor by our side. If we take a class, there might be 25 to 50 students for every teacher. If we take a MOOC (Massive Open Online Course), there might be 1,000 to 10,000 students for every professor or TA. And if we’re learning on our own from books or online tutorials, there’s no tutor or even fellow learners in sight. Given this reality, how can computer-based tools potentially bring some of the benefits of face-to-face learning to millions of people around the world who do not have access to an in-person tutor?

I've begun to address this question by building open-source tools to help people overcome a fundamental barrier to learning programming: understanding what happens as the computer runs each line of a program’s source code. Without this basic skill, it's impossible to start becoming fluent in any programming language. For example, if you’re learning Python, it might be hard to understand why running the code below produces the following three lines of output (click to enlarge):

A tutor can explain why this code prints what it does by drawing the variables, data structures, and pointers at each execution step. However, what if you don't have a personal tutor?

To mimic how a human tutor would illustrate code execution, I've developed Python Tutor (pythontutor.com).

Using this tool, you can:

  1. Choose your programming language and write code in your browser. Despite its legacy name, the tool currently supports Python, Java, JavaScript, TypeScript, Ruby, C, and C++.
  2. Run your code and use the slider to step forward and backward to any execution step.
  3. See a visualization of the execution state at each step, featuring stack frames, variables, data structures, and pointers.
  4. See the terminal output up to the current step.

This tool helps people build robust mental models of what their code is doing behind the scenes, which is crucial for learning any programming language. So far, over 3.5 million people in over 180 countries have used it to visualize over 30 million pieces of code [numbers updated in 2017], often as a supplement to learning from MOOCs, digital textbooks, and online tutorials.

Codechella – multi-user code visualizations

Visualizing execution is helpful, but what would be better is to have a tutor sit by your side and explain those visualizations to you. To support this use case, I've developed a tool called Codechella that enables multiple people to connect to a single Python Tutor session. (Click to enlarge screenshots.)

To use Codechella:

  1. Click the “Start shared session” button (now called “Start private chat session“) at the top of the Python Tutor interface. Then send the unique session URL to a tutor, which will bring them into your session.
  2. Now the two of you can write code together in a collaborative text editor. Also, you will both see the same synchronized view of the visualization so that you can explore it together.
  3. Chat with each other using the embedded chat box.
  4. See each other's mouse cursors in real time.

Codechella approximates the intimate experience of people huddling around a single computer to learn together. In its first nine months of deployment in 2014–2015, people from 296 cities across 40 countries participated in 299 sessions, chatting in 27 different languages. 57% of sessions had participants who joined from different cities, and 12% from different countries, which shows that Codechella connected people who were otherwise unable to get together to learn face-to-face.

Codeopticon – one-to-many remote tutoring

Codechella works well for one-on-one tutoring, but there are usually far fewer tutors than learners. For instance, in a large university course or a MOOC, a TA might be responsible for helping hundreds or even thousands of learners. There is no way that they can devote the time to tutoring everyone individually. So to make the most of their scarce time, I've developed a real-time dashboard interface called Codeopticon.

Each tile in the interface (eight shown here) displays one learner’s actions in real time as they are writing, running, and visualizing code on pythontutor.com. The most active learners’ tiles are always shown at the top. A tutor can:

  1. See each learner's code edits in real time, with insertions in green and deletions in red.
  2. See code errors in red, which could reveal learner struggles.
  3. Use a slider to see a history of each learner's actions, which could also reveal struggles.
  4. Start a chat with any number of learners directly in their tiles. Each learner sees an embedded chat box in their Python Tutor session, similar to using Codechella.

Codeopticon scales up a single tutor's scarce attention so that they can monitor and help multiple learners at once. During user testing, we found that tutors could keep an eye on around 50 learners during a 30-minute session, start 12 chat conversations, and concurrently help 3 learners at once. Tutors felt surprisingly immersed in the experience as though they were helping students face-to-face in the computer lab.

Codepourri – crowdsourced tutorial generation

Codeopticon is useful for making the most of a single tutor's limited time, but what if you’re learning on your own without a tutor in sight? If you’re taking a MOOC or using online resources such as StackOverflow or pythontutor.com, there are up to hundreds of other learners on that same website at the same time you are. Can you somehow call out to them for help? My student Mitchell Gordon and I built a tool called Codepourri that explores this possibility.

Using Codepourri, you can upload your code to the Python Tutor website and let a crowd of people collectively create a tutorial for it by annotating each execution step in the visualization. For example, here are four steps of a tutorial explaining how a simple “while” loop works:

We deployed Codepourri live on pythontutor.com and recruited a crowd of learners there to volunteer to annotate two pieces of Python code from a textbook. The crowd wrote 145 total annotations, and a panel of four Python experts judged 65% of those annotations as correct and 17% of the correct ones as containing surprising insights that even the experts did not consider. The crowd then voted on the best annotations for each step to include in a final tutorial, and the experts judged the learner-created tutorials as comparable to those that experts would create. This experiment shows that, at least for simple code, Codepourri allows you to summon a crowd of fellow learners to explain how a piece of code works, even when there are no experts around.

To recap, Python Tutor and the three extensions built upon it – Codechella, Codeopticon, and Codepourri – bring some of the benefits of face-to-face learning to millions of people around the world who want to learn programming but who do not have access to an in-person tutor.


Read the full papers for details:

Codechella: Multi-User Program Visualizations for Real-Time Tutoring and Collaborative Learning. Philip J. Guo, Jeffery White, Renan Zanelatto. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
An effective way to learn computer programming is to sit side-by-side in front of the same computer with a tutor or peer, write code together, and then discuss what happens as the code executes. To bring this kind of in-person interaction to an online setting, we have developed Codechella, a multi-user Web-based program visualization system that enables multiple people to collaboratively write code together, explore an automatically-generated visualization of its execution state using multiple mouse cursors, and chat via an embedded text box. In the past nine months of live deployment on an educational website, people from 296 cities across 40 countries have started 299 Codechella sessions for both tutoring and collaborative learning. 57% of sessions connected participants from different cities. 69% of actions were visualization interactions, which indicates high engagement with program visualizations. Finally, participants showed signs of learning at the lower three levels of Bloom's taxonomy: remembering, understanding, and applying knowledge.
@inproceedings{GuoVLHCC2015codechella,
  author={Guo, Philip J. and White, Jeffery and Zanelatto, Renan},
  title={Codechella: Multi-user program visualizations for real-time tutoring and collaborative learning},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={79-87},
  doi={10.1109/VLHCC.2015.7357201},
  month={Oct}
}
Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming. Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2015.
One-on-one tutoring from a human expert is an effective way for novices to overcome learning barriers in complex domains such as computer programming. But there are usually far fewer experts than learners. To enable a single expert to help more learners at once, we built Codeopticon, an interface that enables a programming tutor to monitor and chat with dozens of learners in real time. Each learner codes in a workspace that consists of an editor, compiler, and visual debugger. The tutor sees a real-time view of each learner's actions on a dashboard, with each learner's workspace summarized in a tile. At a glance, the tutor can see how learners are editing and debugging their code, and what errors they are encountering. The dashboard automatically reshuffles tiles so that the most active learners are always in the tutor's main field of view. When the tutor sees that a particular learner needs help, they can open an embedded chat window to start a one-on-one conversation. A user study showed that 8 first-time Codeopticon users successfully tutored anonymous learners from 54 countries in a naturalistic online setting. On average, in a 30-minute session, each tutor monitored 226 learners, started 12 conversations, exchanged 47 chats, and helped 2.4 learners.
@inproceedings{GuoUIST2015,
 author = {Guo, Philip J.},
 title = {Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming},
 booktitle = {Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '15},
 year = {2015},
 isbn = {978-1-4503-3779-3},
 pages = {599--608},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2807442.2807469},
 doi = {10.1145/2807442.2807469},
 acmid = {2807469},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {computer programming, learning at scale, remote tutoring}
}
Codepourri: Creating Visual Coding Tutorials Using A Volunteer Crowd Of Learners. Mitchell Gordon and Philip J. Guo. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
A popular way to learn is by studying written tutorials. However, tutorials for computer programming can be tedious to create, since a static text-based format cannot visualize what happens as code executes. We created a system called Codepourri that enables people to easily create visual coding tutorials by annotating steps in an automatically-generated program visualization. Using Codepourri, we developed a crowdsourcing workflow where learners who are visiting an educational website collectively create a tutorial by annotating individual steps and then voting on the best annotations. Since there are far more learners than experts, using learners as a crowd is a potentially more scalable way of creating tutorials. Our experiments with 4 expert judges and 101 learners adding 145 raw annotations to Python code show the learner crowd's annotations to be accurate, informative, and containing some insights that even experts missed.
@inproceedings{GordonVLHCC2015codepourri,
  author={Gordon, Mitchell and Guo, Philip J.},
  title={Codepourri: Creating visual coding tutorials using a volunteer crowd of learners},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={13-21},
  doi={10.1109/VLHCC.2015.7357193},
  month={Oct}
}
Created: 2015-08-13
Last modified: 2017-11-09
Related pages tagged as human-computer interaction:
Related pages tagged as programming:
Related pages tagged as computing education: