Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

Selected Publications

Check out who is citing these papers. For a complete list of publications, see my Curriculum Vitae.

Note that in many areas within computer science and human-computer interaction, conferences (not journals) are the primary venues for peer-reviewed publications.

Short PDF summary of my recent publications.

Summary: I research scalable ways to help people learn computer programming and data science.

2017

Older Adults Learning Computer Programming: Motivations, Frustrations, and Design Opportunities. Philip J. Guo. ACM Conference on Human Factors in Computing Systems (CHI), 2017.
(Honorable Mention Paper Award)
Many people over age 60 want to learn to code but get frustrated by cognitive and social factors.
Computer programming is a highly in-demand skill, but most learn-to-code initiatives and research target some of the youngest members of society: children and college students. We present the first known study of older adults learning computer programming. Using an online survey with 504 respondents aged 60 to 85 who are from 52 different countries, we discovered that older adults were motivated to learn to keep their brains challenged as they aged, to make up for missed opportunities during youth, to connect with younger family members, and to improve job prospects. They reported frustrations including a perceived decline in cognitive abilities, lack of opportunities to interact with tutors and peers, and trouble dealing with constantly-changing software technologies. Based on these findings, we propose a learner-centered design of techniques and tools for motivating older adults to learn programming and discuss broader societal implications of a future where more older adults have access to computer programming -- not merely computer literacy -- as a skill set.
@inproceedings{GuoCHI2017,
 author = {Guo, Philip J.},
 title = {Older Adults Learning Computer Programming: Motivations, Frustrations, and Design Opportunities},
 booktitle = {Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '17},
 year = {2017},
 isbn = {978-1-4503-4655-9},
 location = {Denver, Colorado, USA},
 pages = {7070--7083},
 numpages = {14},
 url = {http://doi.acm.org/10.1145/3025453.3025945},
 doi = {10.1145/3025453.3025945},
 acmid = {3025945},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {computational literacy, learning programming, older adults},
}
CodePilot: Scaffolding End-to-End Collaborative Software Development for Novice Programmers. Jeremy Warner and Philip J. Guo. ACM Conference on Human Factors in Computing Systems (CHI), 2017.
CodePilot is a web IDE that simplifies Git, GitHub, and real-time collaborative coding for novices.
Novice programmers often have trouble installing, configuring, and managing disparate tools (e.g., version control systems, testing infrastructure, bug trackers) that are required to become productive in a modern collaborative software development environment. To lower the barriers to entry into software development, we created a prototype IDE for novices called CodePilot, which is, to our knowledge, the first attempt to integrate coding, testing, bug reporting, and version control management into a real-time collaborative system. CodePilot enables multiple users to connect to a web-based programming session and work together on several major phases of software development. An eight-subject exploratory user study found that first-time users of CodePilot spontaneously used it to assume roles such as developer/tester and developer/assistant when creating a web application together in pairs. Users felt that CodePilot could aid in scaffolding for novices, situational awareness, and lowering barriers to impromptu collaboration.
@inproceedings{WarnerCHI2017,
 author = {Warner, Jeremy and Guo, Philip J.},
 title = {CodePilot: Scaffolding End-to-End Collaborative Software Development for Novice Programmers},
 booktitle = {Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '17},
 year = {2017},
 isbn = {978-1-4503-4655-9},
 location = {Denver, Colorado, USA},
 pages = {1136--1141},
 numpages = {6},
 url = {http://doi.acm.org/10.1145/3025453.3025876},
 doi = {10.1145/3025453.3025876},
 acmid = {3025876},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {collaborative ide, novice programmers, pair programming},
}
Hack.edu: Examining How College Hackathons Are Perceived By Student Attendees and Non-Attendees. Jeremy Warner and Philip J. Guo. ACM International Computing Education Research conference (ICER), 2017.
At college hackathons, students learn to code incidentally, opportunistically, and from peers.
College hackathons have become popular in the past decade, with tens of thousands of students now participating each year across hundreds of campuses. Since hackathons are informal learning environments where students learn and practice coding without any faculty supervision, they are an important site for computing education researchers to study as a complement to studying formal classroom learning environments. However, despite their popularity, little is known about why students choose to attend these events, what they gain from attending, and conversely, why others choose *not* to attend. This paper presents a mixed methods study that examines student perceptions of college hackathons by focusing on three main questions: 1.) Why are students motivated to attend hackathons? 2.) What kind of learning environment do these events provide? 3.) What factors discourage students from attending? Through semi-structured interviews with six college hackathon attendees (50% female), direct observation at a hackathon, and 256 survey responses from college students (42% female), we discovered that students were motivated to attend for both social and technical reasons, that the format generated excitement and focus, and that learning occurred incidentally, opportunistically, and from peers. Those who chose not to attend or had negative experiences cited discouraging factors such as physical discomfort, lack of substance, an overly competitive climate, an unwelcoming culture, and fears of not having enough prior experience. We conclude by discussing ideas for making college hackathons more broadly inclusive and welcoming in light of our study's findings.
@inproceedings{WarnerICER2017,
 author = {Warner, Jeremy and Guo, Philip J.},
 title = {Hack.edu: Examining How College Hackathons Are Perceived By Student Attendees and Non-Attendees},
 booktitle = {Proceedings of the 2017 International Conference on International Computing Education Research},
 series = {ICER '17},
 year = {2017},
 location = {Tacoma, Washington, USA},
 publisher = {ACM},
 address = {New York, NY, USA},
}
DS.js: Turn Any Webpage into an Example-Centric Live Programming Environment for Learning Data Science. Xiong Zhang and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2017.
(Honorable Mention Paper Award)
DS.js transforms any existing webpage into a live programming environment for data science.
Data science courses and tutorials have grown popular in recent years, yet they are still taught using production-grade programming tools (e.g., R, MATLAB, and Python IDEs) within desktop computing environments. Although powerful, these tools present high barriers to entry for novices, forcing them to grapple with the extrinsic complexities of software installation and configuration, data file management, data parsing, and Unix-like command-line interfaces. To lower the barrier for novices to get started with learning data science, we created DS.js, a bookmarklet that embeds a data science programming environment directly into any existing webpage. By transforming any webpage into an example-centric IDE, DS.js eliminates the aforementioned complexities of desktop-based environments and turns the entire web into a rich substrate for learning data science. DS.js automatically parses HTML tables and CSV/TSV data sets on the target webpage, attaches code editors to each data set, provides a data table manipulation and visualization API designed for novices, and gives instructional scaffolding in the form of bidirectional previews of how the user's code and data relate.
@inproceedings{ZhangUIST2017,
 author = {Zhang, Xiong and Guo, Philip J.},
 title = {{DS.js}: Turn Any Webpage into an Example-Centric Live Programming Environment for Learning Data Science},
 booktitle = {Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '17},
 year = {2017},
 publisher = {ACM},
 address = {New York, NY, USA},
}
Omnicode: A Novice-Oriented Live Programming Environment with Always-On Run-Time Value Visualizations. Hyeonsu Kang and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2017.
Omnicode constantly visualizes the full history of all numeric values in your code as you're coding.
Visualizations of run-time program state help novices form proper mental models and debug their code. We push this technique to the extreme by posing the following question: What if a live programming environment for an imperative language always displays the entire history of all run-time values for all program variables all the time? To explore this question, we built a prototype live IDE called Omnicode ("Omniscient Code") that continually runs the user's Python code and uses a scatterplot matrix to visualize the entire history of all of its numerical values, along with meaningful numbers derived from other data types. To filter the visualizations and hone in on specific points of interest, the user can brush and link over the scatterplots or select portions of code. They can also zoom in to view detailed stack and heap visualizations at each execution step. An exploratory study on 10 novice programmers discovered that they found Omnicode to be useful for debugging, forming mental models, explaining their code to others, and discovering moments of serendipity that would not have been likely within an ordinary IDE.
@inproceedings{KangUIST2017,
 author = {Kang, Hyeonsu and Guo, Philip J.},
 title = {Omnicode: A Novice-Oriented Live Programming Environment with Always-On Run-Time Value Visualizations},
 booktitle = {Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '17},
 year = {2017},
 publisher = {ACM},
 address = {New York, NY, USA},
}
Torta: Generating Mixed-Media GUI and Command-Line App Tutorials Using Operating-System-Wide Activity Tracing. Alok Mysore and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2017.
Torta lets you make programming/sysadmin tutorials by simply demonstrating command-line/GUI actions.
Tutorials are vital for helping people perform complex software-based tasks in domains such as programming, data science, system administration, and computational research. However, it is tedious to create detailed step-by-step tutorials for tasks that span multiple interrelated GUI and command-line applications. To address this challenge, we created Torta, an end-to-end system that automatically generates step-by-step GUI and command-line app tutorials by demonstration, provides an editor to trim, organize, and add validation criteria to these tutorials, and provides a web-based viewer that can validate step-level progress and automatically run certain steps. The core technical insight that underpins Torta is that combining operating-system-wide activity tracing and screencast recording makes it easier to generate mixed-media (text+video) tutorials that span multiple GUI and command-line apps. An exploratory study on 10 computer science teaching assistants (TAs) found that they all preferred the experience and results of using Torta to record programming and sysadmin tutorials relevant to classes they teach rather than manually writing tutorials. A follow-up study on 6 students found that they all preferred following the Torta tutorials created by those TAs over the manually-written versions.
@inproceedings{MysoreUIST2017,
 author = {Mysore, Alok and Guo, Philip J.},
 title = {Torta: Generating Mixed-Media GUI and Command-Line App Tutorials Using Operating-System-Wide Activity Tracing},
 booktitle = {Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '17},
 year = {2017},
 publisher = {ACM},
 address = {New York, NY, USA},
}
HappyFace: Identifying and Predicting Frustrating Obstacles for Learning Programming at Scale. Ian Drosos, Philip J. Guo, Chris Parnin. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2017.
HappyFace uses a five-level pain scale to identify causes of frustration when learning programming.
Unnecessary obstacles limit learning in cognitively-complex domains such as computer programming. With a lack of appropriate feedback mechanisms, novice programmers can experience frustration and disengage from the learning experience. In large-scale educational settings, the struggles of learners are often invisible to the learning infrastructure and learners have limited ability to seek help. In this paper, we perform a large-scale collection of code snippets from an online learn-to-code platform, Python Tutor, and collect a frustration rating through a light-weight learner feedback mechanism. We then devise a technique that can automatically identify sources of frustration based on participants labeling their frustration levels. We found 3 factors that best predicted novice programmers' frustration state: syntax errors, using niche language features, and understanding code with high complexity. Additionally, we found evidence that we could predict sources of frustration. Based on these results, we believe an embedded feedback mechanism can lead to future intervention systems.
@inproceedings{DrososVLHCC2017,
  author={Drosos, Ian and Guo, Philip J. and Parnin, Chris},
  title={{HappyFace}: Identifying and Predicting Frustrating Obstacles for Learning Programming at Scale},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '17},
  year={2017},
  month={Oct}
}

2016

Paradise Unplugged: Identifying Barriers for Female Participation on Stack Overflow. Denae Ford, Justin Smith, Philip J. Guo, Chris Parnin. ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE), 2016.
Female programmers face unique challenges when contributing to discussion forums and Q&A sites.
It is no secret that females engage less in programming fields than males. However, in online communities, such as Stack Overflow, this gender gap is even more extreme: only 5.8% of contributors are female. In this paper, we use a mixed-methods approach to identify contribution barriers females face in online communities. Through 22 semi-structured interviews with a spectrum of female users ranging from non-contributors to a top 100 ranked user of all time, we identified 14 barriers preventing them from contributing to Stack Overflow. We then conducted a survey with 1470 female and male developers to confirm which barriers are gender related or general problems for everyone. Females ranked five barriers significantly higher than males. A few of these include doubts in the level of expertise needed to contribute, feeling overwhelmed when competing with a large number of users, and limited awareness of site features. Still, there were other barriers that equally impacted all Stack Overflow users or affected particular groups, such as industry programmers. Finally, we describe several implications that may encourage increased participation in the Stack Overflow community across genders and other demographics.
@inproceedings{FordFSE2016,
 author = {Ford, Denae and Smith, Justin and Guo, Philip J. and Parnin, Chris},
 title = {Paradise Unplugged: Identifying Barriers for Female Participation on {Stack Overflow}},
 booktitle = {Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering},
 series = {FSE 2016},
 year = {2016},
 isbn = {978-1-4503-4218-6},
 location = {Seattle, WA, USA},
 pages = {846--857},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/2950290.2950331},
 doi = {10.1145/2950290.2950331},
 acmid = {2950331},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {Barriers, Females in Computing, Online Communities, Social Q\&\#38;A},
}
Understanding Conversational Programmers: A Perspective from the Software Industry. Parmit K. Chilana, Rishabh Singh, Philip J. Guo. ACM Conference on Human Factors in Computing Systems (CHI), 2016.
Conversational programmers are professionals who learn to code to communicate better with engineers.
Recent research suggests that some students learn to program with the goal of becoming conversational programmers: they want to develop programming literacy skills not to write code in the future but mainly to develop conversational skills and communicate better with developers and to improve their marketability. To investigate the existence of such a population of conversational programmers in practice, we surveyed professionals at a large multinational technology company who were not in software development roles. Based on 3151 survey responses from professionals who never or rarely wrote code, we found that a significant number of them (42.6%) had invested in learning programming on the job. While many of these respondents wanted to perform traditional end-user programming tasks (e.g., data analysis), we discovered that two top motivations for learning programming were to improve the efficacy of technical conversations and to acquire marketable skillsets. The main contribution of this work is in empirically establishing the existence and characteristics of conversational programmers in a large software development context.
@inproceedings{ChilanaCHI2016,
 author = {Chilana, Parmit K. and Singh, Rishabh and Guo, Philip J.},
 title = {Understanding Conversational Programmers: A Perspective from the Software Industry},
 booktitle = {Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '16},
 year = {2016},
 isbn = {978-1-4503-3362-7},
 location = {Santa Clara, California, USA},
 pages = {1462--1472},
 numpages = {11},
 url = {http://doi.acm.org/10.1145/2858036.2858323},
 doi = {10.1145/2858036.2858323},
 acmid = {2858323},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {conversational programmers, non-CS majors, programming literacy, technical conversations},
}

2015

Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming. Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2015.
Codeopticon lets instructors monitor and chat with dozens of students at once while they are coding.
One-on-one tutoring from a human expert is an effective way for novices to overcome learning barriers in complex domains such as computer programming. But there are usually far fewer experts than learners. To enable a single expert to help more learners at once, we built Codeopticon, an interface that enables a programming tutor to monitor and chat with dozens of learners in real time. Each learner codes in a workspace that consists of an editor, compiler, and visual debugger. The tutor sees a real-time view of each learner's actions on a dashboard, with each learner's workspace summarized in a tile. At a glance, the tutor can see how learners are editing and debugging their code, and what errors they are encountering. The dashboard automatically reshuffles tiles so that the most active learners are always in the tutor's main field of view. When the tutor sees that a particular learner needs help, they can open an embedded chat window to start a one-on-one conversation. A user study showed that 8 first-time Codeopticon users successfully tutored anonymous learners from 54 countries in a naturalistic online setting. On average, in a 30-minute session, each tutor monitored 226 learners, started 12 conversations, exchanged 47 chats, and helped 2.4 learners.
@inproceedings{GuoUIST2015,
 author = {Guo, Philip J.},
 title = {Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming},
 booktitle = {Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '15},
 year = {2015},
 isbn = {978-1-4503-3779-3},
 pages = {599--608},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2807442.2807469},
 doi = {10.1145/2807442.2807469},
 acmid = {2807469},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {computer programming, learning at scale, remote tutoring}
}
Codechella: Multi-User Program Visualizations for Real-Time Tutoring and Collaborative Learning. Philip J. Guo, Jeffery White, Renan Zanelatto. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
Codechella lets multiple people write code, visualize run-time state, debug, and chat in real time.
An effective way to learn computer programming is to sit side-by-side in front of the same computer with a tutor or peer, write code together, and then discuss what happens as the code executes. To bring this kind of in-person interaction to an online setting, we have developed Codechella, a multi-user Web-based program visualization system that enables multiple people to collaboratively write code together, explore an automatically-generated visualization of its execution state using multiple mouse cursors, and chat via an embedded text box. In the past nine months of live deployment on an educational website, people from 296 cities across 40 countries have started 299 Codechella sessions for both tutoring and collaborative learning. 57% of sessions connected participants from different cities. 69% of actions were visualization interactions, which indicates high engagement with program visualizations. Finally, participants showed signs of learning at the lower three levels of Bloom's taxonomy: remembering, understanding, and applying knowledge.
@inproceedings{GuoVLHCC2015codechella,
  author={Guo, Philip J. and White, Jeffery and Zanelatto, Renan},
  title={Codechella: Multi-user program visualizations for real-time tutoring and collaborative learning},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={79-87},
  doi={10.1109/VLHCC.2015.7357201},
  month={Oct}
}
Codepourri: Creating Visual Coding Tutorials Using A Volunteer Crowd Of Learners. Mitchell Gordon and Philip J. Guo. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
Codepourri lets an anonymous online crowd create step-by-step coding tutorials together.
A popular way to learn is by studying written tutorials. However, tutorials for computer programming can be tedious to create, since a static text-based format cannot visualize what happens as code executes. We created a system called Codepourri that enables people to easily create visual coding tutorials by annotating steps in an automatically-generated program visualization. Using Codepourri, we developed a crowdsourcing workflow where learners who are visiting an educational website collectively create a tutorial by annotating individual steps and then voting on the best annotations. Since there are far more learners than experts, using learners as a crowd is a potentially more scalable way of creating tutorials. Our experiments with 4 expert judges and 101 learners adding 145 raw annotations to Python code show the learner crowd's annotations to be accurate, informative, and containing some insights that even experts missed.
@inproceedings{GordonVLHCC2015codepourri,
  author={Gordon, Mitchell and Guo, Philip J.},
  title={Codepourri: Creating visual coding tutorials using a volunteer crowd of learners},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={13-21},
  doi={10.1109/VLHCC.2015.7357193},
  month={Oct}
}
Perceptions of Non-CS Majors in Intro Programming: The Rise of the Conversational Programmer. Parmit K. Chilana, Celena Alcock, Shruti Dembla, Anson Ho, Ada Hurst, Brett Armstrong, Philip J. Guo. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
College students learn some coding to understand technical jargon even if they don't want to code.
Despite the enthusiasm and initiatives for making programming accessible to students outside Computer Science (CS), unfortunately, there are still many unanswered questions about how we should be teaching programming to engineers, scientists, artists or other non-CS majors. We present an in-depth case study of first-year management engineering students enrolled in a required introductory programming course at a large North American university. Based on an inductive analysis of one-on-one interviews, surveys, and weekly observations, we provide insights into students' motivations, career goals, perceptions of programming, and reactions to the Java and Processing languages. One of our key findings is that between the traditional classification of non-programmers vs. programmers, there exists a category of conversational programmers who do not necessarily want to be professional programmers or even end-user programmers, but want to learn programming so that they can speak in the "programmer's language" and improve their perceived job marketability in the software industry.
@inproceedings{ChilanaVLHCC2015,
  author={Chilana, Parmit K. and Alcock, Celena and Dembla, Shruti and Ho, Anson and Hurst, Ada and Armstrong, Brett and Guo, Philip J.},
  title={Perceptions of non-CS majors in intro programming: The rise of the conversational programmer},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={251-259},
  doi={10.1109/VLHCC.2015.7357193},
  month={Oct}
}
Toward a Domain-Specific Visual Discussion Forum for Learning Computer Programming: An Empirical Study of a Popular MOOC Forum. Joyce Zhu, Jeremy Warner, Mitchell Gordon, Jeffery White, Renan Zanelatto, Philip J. Guo. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
Text-based discussion forums are inefficient for teaching programming; they need to be more visual.
Online discussion forums are one of the most ubiquitous kinds of resources for people who are learning computer programming. However, their user interface -- a hierarchy of textual threads -- has not changed much in the past four decades. We argue that generic forum interfaces are cumbersome for learning programming and that there is a need for a domain-specific visual discussion forum for programming. We support this argument with an empirical study of all 5,377 forum threads in Introduction to Computer Science and Programming Using Python, a popular edX MOOC. Specifically, we investigated how forum participants were hampered by its text-based format. Most notably, people often wanted to discuss questions about dynamic execution state -- what happens "under the hood" as the computer runs code. We propose that a better forum for learning programming should be visual and domain-specific, integrating automatically-generated visualizations of execution state and enabling inline annotations of source code and output.
@inproceedings{ZhuVLHCC2015csforums,
  author={Zhu, Joyce and Warner, Jeremy and Gordon, Mitchell and White, Jeffery and Zanelatto, Renan and Guo, Philip J.},
  title={Toward a Domain-Specific Visual Discussion Forum for Learning Computer Programming: An Empirical Study of a Popular MOOC Forum},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={101-109},
  doi={10.1109/VLHCC.2015.7357193},
  month={Oct}
}
How High School, College, and Online Students Differentially Engage with an Interactive Digital Textbook. Jeremy Warner, John Doorenbos, Bradley N. Miller, Philip J. Guo. International Conference on Educational Data Mining (EDM), short paper, 2015.
Digital textbooks complement both classroom and MOOC lessons by being self-paced and interactive.
Digital textbooks have been growing popular as a lower-cost and more interactive alternative to paper books. Despite the recent rise in adoption, little is known about how people use these resources. Prior studies have investigated student perceptions of digital textbooks in the classroom via interviews and surveys but have not quantified actual usage patterns. We present, to our knowledge, the first large-scale quantitative study of digital textbook usage. We mined 6.8 million log events from over 43,000 people interacting with How To Think Like a Computer Scientist, one of the most widely-used Web-based textbooks for learning computer programming. We compared engagement patterns among three populations: high school students, college students, and online website viewers. We discovered that people made extensive use of interactive components such as executing code and answering multiple-choice questions, engaged for longer when taking high school or college courses, and frequently viewed textbook sections out of order.
@inproceedings{WarnerEDM2015,
 author = {Warner, Jeremy and Doorenbos, John and Miller, Bradley N. and Guo, Philip J.},
 title = {How High School, College, and Online Students Differentially Engage with an Interactive Digital Textbook},
 booktitle = {Proceedings of the International Conference on Educational
 Data Mining},
 series = {EDM '15},
 year = {2015},
}
Wait-Learning: Leveraging Wait Time for Second Language Education. Carrie J. Cai, Philip J. Guo, James Glass, Robert C. Miller. ACM Conference on Human Factors in Computing Systems (CHI), 2015.
WaitChatter lets people learn foreign languages during brief pause times when instant messaging.
Competing priorities in daily life make it difficult for those with a casual interest in learning to set aside time for regular practice. In this paper, we explore wait-learning: leveraging brief moments of waiting during a person's existing conversations for second language vocabulary practice, even if the conversation happens in the native language. We present an augmented version of instant messaging, WaitChatter, that supports the notion of wait-learning by displaying contextually relevant foreign language vocabulary and micro-quizzes just-in-time while the user awaits a response from her conversant. Through a two week field study of WaitChatter with 20 people, we found that users were able to learn 57 new words on average during casual instant messaging. Furthermore, we found that users were most receptive to learning opportunities immediately after sending a chat message, and that this timing may be critical given user tendency to multi-task during waiting periods.
@inproceedings{CaiCHI2015,
 author = {Cai, Carrie J. and Guo, Philip J. and Glass, James R. and Miller, Robert C.},
 title = {Wait-Learning: Leveraging Wait Time for Second Language Education},
 booktitle = {Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems},
 series = {CHI '15},
 year = {2015},
 isbn = {978-1-4503-3145-6},
 location = {Seoul, Republic of Korea},
 pages = {3701--3710},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2702123.2702267},
 doi = {10.1145/2702123.2702267},
 acmid = {2702267},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {micro-learning, second language learning, wait-learning},
}
OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale. Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip J. Guo, Robert C. Miller. ACM Transactions on Computer-Human Interaction (TOCHI), 2015.
OverCode lets instructors view and comment on thousands of student solutions to coding assignments.
In MOOCs, a single programming exercise may produce thousands of solutions from learners. Understanding solution variation is important for providing appropriate feedback to students at scale. The wide variation among these solutions can be a source of pedagogically valuable examples, and can be used to refine the autograder for the exercise by exposing corner cases. We present OverCode, a system for visualizing and exploring thousands of programming solutions. OverCode uses both static and dynamic analysis to cluster similar solutions, and lets teachers further filter and cluster solutions based on different criteria. We evaluated OverCode against a non-clustering baseline in a within-subjects study with 24 teaching assistants, and found that the OverCode interface allows teachers to more quickly develop a high-level view of students' understanding and misconceptions, and to provide feedback that is relevant to more students' solutions.
@article{GlassmanTOCHI2015,
 author = {Glassman, Elena L. and Scott, Jeremy and Singh, Rishabh and Guo, Philip J. and Miller, Robert C.},
 title = {OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale},
 journal = {ACM Trans. Comput.-Hum. Interact.},
 issue_date = {March 2015},
 volume = {22},
 number = {2},
 month = mar,
 year = {2015},
 issn = {1073-0516},
 pages = {7:1--7:35},
 articleno = {7},
 numpages = {35},
 url = {http://doi.acm.org/10.1145/2699751},
 doi = {10.1145/2699751},
 acmid = {2699751},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {Programming education, learning at scale},
}

2014

Data-Driven Interaction Techniques for Improving Navigation of Educational Videos. Juho Kim, Philip J. Guo, Carrie J. Cai, Shang-Wen (Daniel) Li, Krzysztof Z. Gajos, Robert C. Miller. ACM Symposium on User Interface Software and Technology (UIST), 2014.
LectureScape is a MOOC video player that lets viewers see what thousands of prior viewers focus on.
With an unprecedented scale of learners watching educational videos on online platforms such as MOOCs and YouTube, there is an opportunity to incorporate data generated from their interactions into the design of novel video interaction techniques. Interaction data has the potential to help not only instructors to improve their videos, but also to enrich the learning experience of educational video watchers. This paper explores the design space of data-driven interaction techniques for educational video navigation. We introduce a set of techniques that augment existing video interface widgets, including: a 2D video timeline with an embedded visualization of collective navigation traces; dynamic and non-linear timeline scrubbing; data-enhanced transcript search and keyword summary; automatic display of relevant still frames next to the video; and a visual summary representing points with high learner activity. To evaluate the feasibility of the techniques, we ran a laboratory user study with simulated learning tasks. Participants rated watching lecture videos with interaction data to be efficient and useful in completing the tasks. However, no significant differences were found in task performance, suggesting that interaction data may not always align with moment-by-moment information needs during the tasks.
@inproceedings{KimUIST2014,
 author = {Kim, Juho and Guo, Philip J. and Cai, Carrie J. and Li, Shang-Wen (Daniel) and Gajos, Krzysztof Z. and Miller, Robert C.},
 title = {Data-driven Interaction Techniques for Improving Navigation of Educational Videos},
 booktitle = {Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '14},
 year = {2014},
 isbn = {978-1-4503-3069-5},
 location = {Honolulu, Hawaii, USA},
 pages = {563--572},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2642918.2647389},
 doi = {10.1145/2642918.2647389},
 acmid = {2647389},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {MOOCs, interaction peaks, multimedia learning, video content analysis, video learning, video summarization},
}
A Direct Manipulation Language for Explaining Algorithms. Jeremy Scott, Philip J. Guo, Randall Davis. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), short paper, 2014.
A visual language makes it easier to create CS algorithm animations than manually drawing diagrams.
Instructors typically explain algorithms in computer science by tracing their behavior, often on blackboards, sometimes with algorithm visualizations. Using blackboards can be tedious because they do not facilitate manipulation of the drawing, while visualizations often operate at the wrong level of abstraction or must be laboriously hand-coded for each algorithm. In response, we present a direct manipulation (DM) language for explaining algorithms by manipulating visualized data structures. The language maps DM gestures onto primitive program behaviors that occur in commonly taught algorithms. We performed an initial evaluation of the DM language on teaching assistants of an undergraduate algorithms class, who found the language easier to use and more helpful for explaining algorithms than a standard drawing application (GIMP).
@inproceedings{ScottVLHCC2014,
 author = {Scott, Jeremy and Guo, Philip J. and Davis, Randall},
 title = {A Direct Manipulation Language for Explaining Algorithms},
 booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
 series = {VL/HCC '14},
 publisher = {IEEE},
 year = {2014},
 pages = {45--48},
}
Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. Juho Kim, Phu Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, Krzysztof Z. Gajos. ACM Conference on Human Factors in Computing Systems (CHI), 2014.
(Honorable Mention Paper Award)
Crowdsourcing can effectively add annotations of step-by-step instructions to tutorial videos.
Millions of learners today use how-to videos to master new skills in a variety of domains. But browsing such videos is often tedious and inefficient because video player interfaces are not optimized for the unique step-by-step structure of such videos. This research aims to improve the learning experience of existing how-to videos with step-by-step annotations.

We first performed a formative study to verify that annotations are actually useful to learners. We created ToolScape, an interactive video player that displays step descriptions and intermediate result thumbnails in the video timeline. Learners in our study performed better and gained more self-efficacy using ToolScape versus a traditional video player.

To add the needed step annotations to existing how-to videos at scale, we introduce a novel crowdsourcing workflow. It extracts step-by-step structure from an existing video, including step times, descriptions, and before and after images. We introduce the Find-Verify-Expand design pattern for temporal and visual annotation, which applies clustering, text processing, and visual analysis algorithms to merge crowd output. The workflow does not rely on domain-specific customization, works on top of existing videos, and recruits untrained crowd workers. We evaluated the workflow with Mechanical Turk, using 75 cooking, makeup, and Photoshop videos on YouTube. Results show that our workflow can extract steps with a quality comparable to that of trained annotators across all three domains with 77% precision and 81% recall.
@inproceedings{KimCHI2014toolscape,
 author = {Kim, Juho and Nguyen, Phu Tran and Weir, Sarah and Guo, Philip J. and Miller, Robert C. and Gajos, Krzysztof Z.},
 title = {Crowdsourcing Step-by-step Information Extraction to Enhance Existing How-to Videos},
 booktitle = {Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems},
 series = {CHI '14},
 year = {2014},
 isbn = {978-1-4503-2473-1},
 location = {Toronto, Ontario, Canada},
 pages = {4017--4026},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556288.2556986},
 doi = {10.1145/2556288.2556986},
 acmid = {2556986},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {crowdsourcing, how-to videos, video annotation.},
}
Demographic Differences in How Students Navigate Through MOOCs. Philip J. Guo and Katharina Reinecke. ACM Conference on Learning at Scale, 2014.
People from different countries and age groups consume MOOC content at different levels of detail.
The current generation of Massive Open Online Courses (MOOCs) attract a diverse student audience from all age groups and over 196 countries around the world. Researchers, educators, and the general public have recently become interested in how the learning experience in MOOCs differs from that in traditional courses. A major component of the learning experience is how students navigate through course content.

This paper presents an empirical study of how students navigate through MOOCs, and is, to our knowledge, the first to investigate how navigation strategies differ by demographics such as age and country of origin. We performed data analysis on the activities of 140,546 students in four edX MOOCs and found that certificate earners skip on average 22% of the course content, that they frequently employ non-linear navigation by jumping backward to earlier lecture sequences, and that older students and those from countries with lower student-teacher ratios are more comprehensive and non-linear when navigating through the course.

From these findings, we suggest design recommendations such as for MOOC platforms to develop more detailed forms of certification that incentivize students to deeply engage with the content rather than just doing the minimum necessary to earn a passing grade.
@inproceedings{GuoLAS2014navigation,
 author = {Guo, Philip J. and Reinecke, Katharina},
 title = {Demographic Differences in How Students Navigate Through MOOCs},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {21--30},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566247},
 doi = {10.1145/2556325.2566247},
 acmid = {2566247},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {massive open online course, mooc, navigation strategy, non-linear learning},
}
How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Philip J. Guo, Juho Kim, Rob Rubin. ACM Conference on Learning at Scale, 2014.
The most engaging MOOC videos are very short, show talking heads, and use hand-made tablet drawings.
Videos are a widely-used kind of resource for online learning. This paper presents an empirical study of how video production decisions affect student engagement in online educational videos. To our knowledge, ours is the largest-scale study of video engagement to date, using data from 6.9 million video watching sessions across four courses on the edX MOOC platform. We measure engagement by how long students are watching each video, and whether they attempt to answer post-video assessment problems.

Our main findings are that shorter videos are much more engaging, that informal talking-head videos are more engaging, that Khan-style tablet drawings are more engaging, that even high-quality pre-recorded classroom lectures might not make for engaging online videos, and that students engage differently with lecture and tutorial videos.

Based upon these quantitative findings and qualitative insights from interviews with edX staff, we developed a set of recommendations to help instructors and video producers take better advantage of the online video format.
@inproceedings{GuoLAS2014vidprod,
 author = {Guo, Philip J. and Kim, Juho and Rubin, Rob},
 title = {How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {41--50},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566239},
 doi = {10.1145/2556325.2566239},
 acmid = {2566239},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {mooc, online education, video engagement},
}
Understanding In-Video Dropouts and Interaction Peaks in Online Lecture Videos. Juho Kim, Philip J. Guo, Daniel T. Seaton, Piotr Mitros, Krzysztof Z. Gajos, Robert C. Miller. ACM Conference on Learning at Scale, 2014.
When many people pause/rewatch the same part of MOOC videos, they are either confused or interested.
With thousands of learners watching the same online lecture videos, analyzing video watching patterns provides a unique opportunity to understand how students learn with videos. This paper reports a large-scale analysis of in-video dropout and peaks in viewership and student activity, using second-by-second user interaction data from 862 videos in four Massive Open Online Courses (MOOCs) on edX. We find higher dropout rates in longer videos, re-watching sessions (vs first-time), and tutorials (vs lectures). Peaks in re-watching sessions and play events indicate points of interest and confusion. Results show that tutorials (vs lectures) and re-watching sessions (vs first-time) lead to more frequent and sharper peaks. In attempting to reason why peaks occur by sampling 80 videos, we observe that 61% of the peaks accompany visual transitions in the video, e.g., a slide view to a classroom view. Based on this observation, we identify five student activity patterns that can explain peaks: starting from the beginning of a new material, returning to missed content, following a tutorial step, replaying a brief segment, and repeating a non-visual explanation. Our analysis has design implications for video authoring, editing, and interface design, providing a richer understanding of video learning on MOOCs.
@inproceedings{KimMOOC2014,
 author = {Kim, Juho and Guo, Philip J. and Seaton, Daniel T. and Mitros, Piotr and Gajos, Krzysztof Z. and Miller, Robert C.},
 title = {Understanding In-video Dropouts and Interaction Peaks In Online Lecture Videos},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {31--40},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566237},
 doi = {10.1145/2556325.2566237},
 acmid = {2566237},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {in-video dropout, interaction peaks, mooc, online education, peak detection., video analysis},
}

2013

Online Python Tutor: Embeddable Web-Based Program Visualization for CS Education. Philip J. Guo. ACM Technical Symposium on Computer Science Education (SIGCSE), 2013.
Python Tutor lets anyone write code in their web browser and visualize step-by-step run-time state.
This paper presents Online Python Tutor, a web-based program visualization tool for Python, which is becoming a popular language for teaching introductory CS courses. Using this tool, teachers and students can write Python programs directly in the web browser (without installing any plugins), step forwards and backwards through execution to view the run-time state of data structures, and share their program visualizations on the web.

In the past three years, over 200,000 people have used Online Python Tutor to visualize their programs. In addition, instructors in a dozen universities such as UC Berkeley, MIT, the University of Washington, and the University of Waterloo have used it in their CS1 courses. Finally, Online Python Tutor visualizations have been embedded within three web-based digital Python textbook projects, which collectively attract around 16,000 viewers per month and are being used in at least 25 universities. Online Python Tutor is free and open source software, available at pythontutor.com
@inproceedings{GuoSIGCSE2013,
 author = {Guo, Philip J.},
 title = {{Online Python Tutor}: Embeddable Web-based Program Visualization for {CS} Education},
 booktitle = {Proceedings of the 44th ACM Technical Symposium on Computer Science Education},
 series = {SIGCSE '13},
 year = {2013},
 isbn = {978-1-4503-1868-6},
 location = {Denver, Colorado, USA},
 pages = {579--584},
 numpages = {6},
 url = {http://doi.acm.org/10.1145/2445196.2445368},
 doi = {10.1145/2445196.2445368},
 acmid = {2445368},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {CS1, program visualization, python},
}

2012

Characterizing and Predicting Which Bugs Get Reopened. Thomas Zimmermann, Nachiappan Nagappan, Philip J. Guo, Brendan Murphy. ACM/IEEE International Conference on Software Engineering (ICSE), Software Engineering In Practice (SEIP) track, 2012.
(Best Paper Award)
Software bugs get closed then reopened due to problems in reproducibility, priorities, and process.
Fixing bugs is an important part of the software development process. An underlying aspect is the effectiveness of fixes: if a fair number of fixed bugs are reopened, it could indicate instability in the software system. To the best of our knowledge there has been on little prior work on understanding the dynamics of bug reopens. Towards that end, in this paper, we characterize when bug reports are reopened by using the Microsoft Windows operating system project as an empirical case study. Our analysis is based on a mixed-methods approach. First, we categorize the primary reasons for reopens based on a survey of 358 Microsoft employees. We then reinforce these results with a large-scale quantitative study of Windows bug reports, focusing on factors related to bug report edits and relationships between people involved in handling the bug. Finally, we build statistical models to describe the impact of various metrics on reopening bugs ranging from the reputation of the opener to how the bug was found.
@inproceedings{ZimmermannReopen2012,
 author = {Zimmermann, Thomas and Nagappan, Nachiappan and Guo, Philip J. and Murphy, Brendan},
 title = {Characterizing and Predicting Which Bugs Get Reopened},
 booktitle = {Proceedings of the 2012 International Conference on Software Engineering},
 series = {ICSE 2012},
 year = {2012},
 isbn = {978-1-4673-1067-3},
 location = {Zurich, Switzerland},
 pages = {1074--1083},
 numpages = {10},
 url = {http://dl.acm.org/citation.cfm?id=2337223.2337363},
 acmid = {2337363},
 publisher = {IEEE Press},
 address = {Piscataway, NJ, USA},
}
Burrito: Wrapping Your Lab Notebook in Computational Infrastructure. Philip J. Guo and Margo Seltzer. USENIX Workshop on the Theory and Practice of Provenance (TaPP), 2012.
Burrito automatically records scientists' computer activities and makes lab notebooks out of them.
Researchers in fields such as bioinformatics, CS, finance, and applied math have trouble managing the numerous code and data files generated by their computational experiments, comparing the results of trials executed with different parameters, and keeping up-to-date notes on what they learned from past successes and failures.

We created a Linux-based system called Burrito that automates aspects of this tedious experiment organization and notetaking process, thus freeing researchers to focus on more substantive work. Burrito automatically captures a researcher's computational activities and provides user interfaces to annotate the captured provenance with notes and then make queries such as, "Which script versions and command-line parameters generated the output graph that this note refers to?"
@inproceedings{GuoBurrito2012,
  author = {Guo, Philip J. and Seltzer, Margo},
  title = {BURRITO: Wrapping Your Lab Notebook in Computational Infrastructure},
  booktitle = {Proceedings of the 4th USENIX Workshop on the Theory and Practice of Provenance},
  series = {TaPP'12},
  year = {2012},
  location = {Boston, MA},
  url = {http://dl.acm.org/citation.cfm?id=2342875.2342882},
  acmid = {2342882},
  publisher = {USENIX Association},
  address = {Berkeley, CA, USA},
}
Software Tools to Facilitate Research Programming. Philip J. Guo. Ph.D. dissertation, Department of Computer Science, Stanford University, 2012.
Data scientists code in different ways than software engineers, so they need new kinds of tools.
Research programming is a type of programming activity where the goal is to write computer programs to obtain insights from data. Millions of professionals in fields ranging from science, engineering, business, finance, public policy, and journalism, as well as numerous students and computer hobbyists, all perform research programming on a daily basis.

My thesis is that by understanding the unique challenges faced during research programming, it becomes possible to apply techniques from dynamic program analysis, mixed-initiative recommendation systems, and OS-level tracing to make research programmers more productive.

This dissertation characterizes the research programming process, describes typical challenges faced by research programmers, and presents five software tools that I have developed to address some key challenges. 1.) Proactive Wrangler is an interactive graphical tool that helps research programmers reformat and clean data prior to analysis. 2.) IncPy is a Python interpreter that speeds up the data analysis scripting cycle and helps programmers manage code and data dependencies. 3.) SlopPy is a Python interpreter that automatically makes existing scripts error-tolerant, thereby also speeding up the data analysis scripting cycle. 4.) Burrito is a Linux-based system that helps programmers organize, annotate, and recall past insights about their experiments. 5.) CDE is a software packaging tool that makes it easy to deploy, archive, and share research code.

Taken together, these five tools enable research programmers to iterate and potentially discover insights faster by offloading the burdens of data management and provenance to the computer.
@PhdThesis{GuoPhD2012,
  author = {Guo, Philip J.},
  title = {Software Tools to Facilitate Research Programming},
  school = {Stanford University},
  month = May,
  year = 2012,
}
HAMPI: A Solver for Word Equations over Strings, Regular Expressions and Context-free Grammars. Adam Kiezun, Vijay Ganesh, Shay Artzi, Philip J. Guo, Pieter Hooimeijer, Michael D. Ernst. ACM Transactions of Software Engineering Methodology (TOSEM), 2012.
HAMPI is a constraint solver for strings that makes automatic bug-finding tools more efficient.
Many automatic testing, analysis, and verification techniques for programs can be effectively reduced to a constraint-generation phase followed by a constraint-solving phase. This separation of concerns often leads to more effective and maintainable software reliability tools. The increasing efficiency of off-the-shelf constraint solvers makes this approach even more compelling. However, there are few effective and sufficiently expressive off-the-shelf solvers for string constraints generated by analysis of string-manipulating programs, so researchers end up implementing their own ad-hoc solvers.

To fulfill this need, we designed and implemented HAMPI, a solver for string constraints over bounded string variables. Users of HAMPI specify constraints using regular expressions, context-free grammars, equality between string terms, and typical string operations such as concatenation and substring extraction. HAMPI then finds a string that satisfies all the constraints or reports that the constraints are unsatisfiable.

We demonstrate HAMPI's expressiveness and efficiency by applying it to program analysis and automated testing. We used HAMPI in static and dynamic analyses for finding SQL injection vulnerabilities in Web applications with hundreds of thousands of lines of code. We also used HAMPI in the context of automated bug finding in C programs using dynamic systematic testing (also known as concolic testing). We then compared HAMPI with another string solver, CFGAnalyzer, and show that HAMPI is several times faster. HAMPI's source code, documentation, and experimental data are available at http://people.csail.mit.edu/akiezun/hampi.
@article{KiezunHampi2012,
   author = {Adam Kiezun and Vijay Ganesh and Shay Artzi and Philip J.
  Guo and Pieter Hooimeijer and Michael D. Ernst},
   title = {{HAMPI}: A solver for word equations over strings, regular
  expressions, and context-free grammars},
   journal = {ACM Transactions on Software Engineering and Methodology},
   volume = {21},
   number = {4},
   pages = {25:1--25:28},
   month = nov,
   year = {2012}
}
CDE: A Tool for Creating Portable Experimental Software Packages. Philip J. Guo. IEEE Computing in Science and Engineering, 2012.
CDE lets users run any set of Linux programs on other Linux machines without any installation.
One technical barrier to reproducible computational science is that it's hard to distribute scientific code in a form that other researchers can easily execute on their own computers. To help eliminate this barrier, the CDE tool packages all software dependencies required to rerun Linux-based computational experiments on other computers.
@article{GuoCdeCISE2012,
  author = {Guo, Philip J.},
  title = {{CDE}: A Tool for Creating Portable Experimental Software Packages},
  journal = {Computing in Science and Engineering},
  volume = {14},
  number = {4},
  issn = {1521-9615},
  year = {2012},
  pages = {32-35},
  doi = {http://doi.ieeecomputersociety.org/10.1109/MCSE.2012.36},
  publisher = {IEEE Computer Society},
  address = {Los Alamitos, CA, USA},
}

2011

Proactive Wrangling: Mixed-Initiative End-User Programming of Data Transformation Scripts. Philip J. Guo, Sean Kandel, Joseph M. Hellerstein, Jeffrey Heer. ACM Symposium on User Interface Software and Technology (UIST), 2011.
A data wrangling UI uses proactive suggestions to help users transform data into relational formats.
Analysts regularly wrangle data into a form suitable for computational tools through a tedious process that delays more substantive analysis. While interactive tools can assist data transformation, analysts must still conceptualize the desired output state, formulate a transformation strategy, and specify complex transforms. We present a model to proactively suggest data transforms which map input data to a relational format expected by analysis tools. To guide search through the space of transforms, we propose a metric that scores tables according to type homogeneity, sparsity and the presence of delimiters. When compared to "ideal" hand-crafted transformations, our model suggests over half of the needed steps; in these cases the top-ranked suggestion is preferred 77% of the time. User study results indicate that suggestions produced by our model can assist analysts' transformation tasks, but that users do not always value proactive assistance, instead preferring to maintain the initiative. We discuss some implications of these results for mixed-initiative interfaces.
@inproceedings{GuoProWrangler2011,
 author = {Guo, Philip J. and Kandel, Sean and Hellerstein, Joseph M. and Heer, Jeffrey},
 title = {Proactive Wrangling: Mixed-initiative End-user Programming of Data Transformation Scripts},
 booktitle = {Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '11},
 year = {2011},
 isbn = {978-1-4503-0716-1},
 location = {Santa Barbara, California, USA},
 pages = {65--74},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2047196.2047205},
 doi = {10.1145/2047196.2047205},
 acmid = {2047205},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {data analysis, data cleaning, data transformation, end-user programming, mixed-initiative interfaces},
}
Using Automatic Persistent Memoization to Facilitate Data Analysis Scripting. Philip J. Guo and Dawson Engler. International Symposium on Software Testing and Analysis (ISSTA), 2011.
IncPy speeds up scientists' iteration times by automatically caching results of long computations.
Programmers across a wide range of disciplines (e.g., bioinformatics, neuroscience, econometrics, finance, data mining, information retrieval, machine learning) write scripts to parse, transform, process, and extract insights from data. To speed up iteration times, they split their analyses into stages and write extra code to save the intermediate results of each stage to files so that those results do not have to be re-computed in every subsequent run. As they explore and refine hypotheses, their scripts often create and process lots of intermediate data files. They need to properly manage the myriad of dependencies between their code and data files, or else their analyses will produce incorrect results.

To enable programmers to iterate quickly without needing to manage intermediate data files, we added a set of dynamic analyses to the programming language interpreter so that it automatically memoizes (caches) the results of long-running pure function calls to disk, manages dependencies between code and on-disk data, and later re-uses memoized results, rather than re-executing those functions, when guaranteed safe to do so. We created an implementation for Python and show how it enables programmers to iterate faster on their data analysis scripts while writing less code and not having to manage dependencies between their code and datasets.
@inproceedings{GuoIncPy2011,
 author = {Guo, Philip J. and Engler, Dawson},
 title = {Using Automatic Persistent Memoization to Facilitate Data Analysis Scripting},
 booktitle = {Proceedings of the 2011 International Symposium on Software Testing and Analysis},
 series = {ISSTA '11},
 year = {2011},
 isbn = {978-1-4503-0562-4},
 location = {Toronto, Ontario, Canada},
 pages = {287--297},
 numpages = {11},
 url = {http://doi.acm.org/10.1145/2001420.2001455},
 doi = {10.1145/2001420.2001455},
 acmid = {2001455},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {dependency management, scientific workflows},
}
Sloppy Python: Using Dynamic Analysis to Automatically Add Error Tolerance to Ad-Hoc Data Processing Scripts. Philip J. Guo. International Workshop on Dynamic Analysis (WODA), 2011.
SlopPy lets existing Python data analysis scripts tolerate and recover from errors without crashing.
Programmers and data analysts get frustrated when their long-running data processing scripts crash without producing results, due to either bugs in their code or inconsistencies in data sources. To alleviate this frustration, we developed a dynamic analysis technique that guarantees scripts will never crash: It converts all uncaught exceptions into special NA (Not Available) objects and continues executing rather than crashing. Thus, imperfect scripts will run to completion and produce partial results and an error log, which is more informative than simply crashing with no results. We implemented our technique as a "Sloppy" Python interpreter that automatically adds error tolerance to existing scripts without any programmer effort or run-time slowdown.
@inproceedings{GuoSlopPy2011,
 author = {Guo, Philip J.},
 title = {{Sloppy Python}: Using Dynamic Analysis to Automatically Add Error Tolerance to Ad-hoc Data Processing Scripts},
 booktitle = {Proceedings of the Ninth International Workshop on Dynamic Analysis},
 series = {WODA '11},
 year = {2011},
 isbn = {978-1-4503-0811-3},
 location = {Toronto, Ontario, Canada},
 pages = {35--40},
 numpages = {6},
 url = {http://doi.acm.org/10.1145/2002951.2002961},
 doi = {10.1145/2002951.2002961},
 acmid = {2002961},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {data processing, fault tolerance, scripting},
}
CDE: Run Any Linux Application On-Demand Without Installation. Philip J. Guo. USENIX Large Installation System Administration Conference (LISA), 2011.
CDE lets users run any set of Linux programs on other Linux machines without any installation.
There is a huge ecosystem of free software for Linux, but since each Linux distribution (distro) contains a different set of pre-installed shared libraries, filesystem layout conventions, and other environmental state, it is difficult to create and distribute software that works without hassle across all distros. Online forums and mailing lists are filled with discussions of users' troubles with compiling, installing, and configuring Linux software and their myriad of dependencies. To address this ubiquitous problem, we have created an open-source tool called CDE that automatically packages up the Code, Data, and Environment required to run a set of x86-Linux programs on other x86-Linux machines. Creating a CDE package is as simple as running the target application under CDE's monitoring, and executing a CDE package requires no installation, configuration, or root permissions. CDE enables Linux users to instantly run any application on-demand without encountering "dependency hell".
@inproceedings{GuoCdeLisa2011,
 author = {Guo, Philip J.},
 title = {{CDE}: Run Any {Linux} Application On-demand Without Installation},
 booktitle = {Proceedings of the 25th International Conference on Large Installation System Administration},
 series = {LISA'11},
 year = {2011},
 location = {Boston, MA},
 url = {http://dl.acm.org/citation.cfm?id=2208488.2208490},
 acmid = {2208490},
 publisher = {USENIX Association},
 address = {Berkeley, CA, USA},
}
CDE: Using System Call Interposition to Automatically Create Portable Software Packages. Philip J. Guo and Dawson Engler. USENIX Annual Technical Conference, short paper, 2011.
CDE lets users run any set of Linux programs on other Linux machines without any installation.
It can be painfully hard to take software that runs on one person's machine and get it to run on another machine. Online forums and mailing lists are filled with discussions of users' troubles with compiling, installing, and configuring software and their myriad of dependencies. To eliminate this dependency problem, we created a system called CDE that uses system call interposition to monitor the execution of x86-Linux programs and package up the Code, Data, and Environment required to run them on other x86-Linux machines. Creating a CDE package is completely automatic, and running programs within a package requires no installation, configuration, or root permissions. Hundreds of people in both academia and industry have used CDE to distribute software, demo prototypes, make their scientific experiments reproducible, run software natively on older Linux distributions, and deploy experiments to compute clusters.
@inproceedings{GuoCdeUsenix2011,
 author = {Guo, Philip J. and Engler, Dawson},
 title = {{CDE}: Using System Call Interposition to Automatically Create Portable Software Packages},
 booktitle = {Proceedings of the 2011 USENIX Annual Technical Conference},
 series = {USENIX'11},
 year = {2011},
 location = {Portland, OR},
 url = {http://dl.acm.org/citation.cfm?id=2002181.2002202},
 acmid = {2002202},
 publisher = {USENIX Association},
 address = {Berkeley, CA, USA},
}
"Not My Bug!" and Other Reasons for Software Bug Report Reassignments. Philip J. Guo, Thomas Zimmermann, Nachiappan Nagappan, Brendan Murphy. ACM Conference on Computer Supported Cooperative Work (CSCW), 2011.
People reassign software bugs to teammates since it can be hard to find the best person to fix them.
Bug reporting/fixing is an important social part of the software development process. The bug-fixing process inherently has strong inter-personal dynamics at play, especially in how to find the optimal person to handle a bug report. Bug report reassignments, which are a common part of the bug-fixing process, have rarely been studied.

In this paper, we present a large-scale quantitative and qualitative analysis of the bug reassignment process in the Microsoft Windows Vista operating system project. We quantify social interactions in terms of both useful and harmful reassignments. For instance, we found that reassignments are useful to determine the best person to fix a bug, contrary to the popular opinion that reassignments are always harmful. We categorized five primary reasons for reassignments: finding the root cause, determining ownership, poor bug report quality, hard to determine proper fix, and workload balancing. We then use these findings to make recommendations for the design of more socially-aware bug tracking systems that can overcome some of the inefficiencies we observed in our study.
@inproceedings{GuoCSCW2011,
 author = {Guo, Philip J. and Zimmermann, Thomas and Nagappan, Nachiappan and Murphy, Brendan},
 title = {{"Not My Bug!"} and Other Reasons for Software Bug Report Reassignments},
 booktitle = {Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work},
 series = {CSCW '11},
 year = {2011},
 isbn = {978-1-4503-0556-3},
 location = {Hangzhou, China},
 pages = {395--404},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/1958824.1958887},
 doi = {10.1145/1958824.1958887},
 acmid = {1958887},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {bug reassignment, bug tracking, bug triaging},
}

2010

Characterizing and Predicting Which Bugs Get Fixed: An Empirical Study of Microsoft Windows. Philip J. Guo, Thomas Zimmermann, Nachiappan Nagappan, Brendan Murphy. IEEE International Conference on Software Engineering (ICSE), 2010.
Software bugs reported by more respected and socially-proximal people are more likely to get fixed.
We performed an empirical study to characterize factors that affect which bugs get fixed in Windows Vista and Windows 7, focusing on factors related to bug report edits and relationships between people involved in handling the bug. We found that bugs reported by people with better reputations were more likely to get fixed, as were bugs handled by people on the same team and working in geographical proximity. We reinforce these quantitative results with survey feedback from 358 Microsoft employees who were involved in Windows bugs. Survey respondents also mentioned additional qualitative influences on bug fixing, such as the importance of seniority and interpersonal skills of the bug reporter.

Informed by these findings, we built a statistical model to predict the probability that a new bug will be fixed (the first known one, to the best of our knowledge). We trained it on Windows Vista bugs and got a precision of 68% and recall of 64% when predicting Windows 7 bug fixes. Engineers could use such a model to prioritize bugs during triage, to estimate developer workloads, and to decide which bugs should be closed or migrated to future product versions.
@inproceedings{GuoICSE2010,
 author = {Guo, Philip J. and Zimmermann, Thomas and Nagappan, Nachiappan and Murphy, Brendan},
 title = {Characterizing and Predicting Which Bugs Get Fixed: An Empirical Study of {Microsoft Windows}},
 booktitle = {Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1},
 series = {ICSE '10},
 year = {2010},
 isbn = {978-1-60558-719-6},
 location = {Cape Town, South Africa},
 pages = {495--504},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/1806799.1806871},
 doi = {10.1145/1806799.1806871},
 acmid = {1806871},
 publisher = {ACM},
 address = {New York, NY, USA},
}
Towards Practical Incremental Recomputation for Scientists: An Implementation for the Python Language. Philip J. Guo and Dawson Engler. USENIX Workshop on the Theory and Practice of Provenance (TaPP), 2010.
IncPy speeds up scientists' iteration times by automatically caching results of long computations.
Computational scientists often prototype data analysis scripts using high-level languages like Python. To speed up execution times, they manually refactor their scripts into stages (separate functions) and write extra code to save intermediate results to disk in order to avoid re-computing them in subsequent runs. To eliminate this burden, we enhanced the Python interpreter to automatically memoize (save) the results of long-running function executions to disk, manage dependencies between code edits and saved results, and re-use memoized results rather than re-executing those functions when guaranteed safe to do so. There is a ~20% run-time slowdown during the initial run, but subsequent runs can speed up by several orders of magnitude. Using our enhanced interpreter, scientists can write simple and maintainable code that also runs fast after minor edits, without having to learn any new programming languages or constructs.
@inproceedings{GuoTaPP2010,
 author = {Guo, Philip J. and Engler, Dawson},
 title = {Towards Practical Incremental Recomputation for Scientists: An Implementation for the {Python} Language},
 booktitle = {Proceedings of the 2nd Workshop on the Theory and Practice of Provenance},
 series = {TAPP'10},
 year = {2010},
 location = {San Jose, California},
 url = {http://dl.acm.org/citation.cfm?id=1855795.1855801},
 acmid = {1855801},
 publisher = {USENIX Association},
 address = {Berkeley, CA, USA},
}

2009

HAMPI: A Solver for String Constraints. Adam Kiezun, Vijay Ganesh, Philip J. Guo, Pieter Hooimeijer, Michael D. Ernst. International Symposium on Software Testing and Analysis (ISSTA), 2009.
(ACM SIGSOFT Distinguished Paper Award)
HAMPI is a constraint solver for strings that makes automatic bug-finding tools more efficient.
Many automatic testing, analysis, and verification techniques for programs can be effectively reduced to a constraint-generation phase followed by a constraint-solving phase. This separation of concerns often leads to more effective and maintainable tools. The increasing efficiency of off-the-shelf constraint solvers makes this approach even more compelling. However, there are few, if any, effective and sufficiently expressive off-the-shelf solvers for string constraints generated by analysis techniques for string-manipulating programs.

We designed and implemented HAMPI, a solver for string constraints over bounded string variables. HAMPI constraints express membership in regular languages and bounded context-free languages. HAMPI constraints may contain context-free-language definitions, regular-language definitions and operations, and the membership predicate. Given a set of constraints, HAMPI outputs a string that satisfies all the constraints, or reports that the constraints are unsatisfiable.

HAMPI is expressive and efficient, and can be successfully applied to testing and analysis of real programs. Our experiments use HAMPI in: static and dynamic analyses for finding SQL injection vulnerabilities in Web applications; automated bug finding in C programs using systematic testing; and compare HAMPI with another string solver. HAMPI's source code, documentation, and the experimental data are available at http://people.csail.mit.edu/akiezun/hampi/.
@inproceedings{KiezunHampi2009,
 author = {Kiezun, Adam and Ganesh, Vijay and Guo, Philip J. and Hooimeijer, Pieter and Ernst, Michael D.},
 title = {{HAMPI}: A Solver for String Constraints},
 booktitle = {Proceedings of the Eighteenth International Symposium on Software Testing and Analysis},
 series = {ISSTA '09},
 year = {2009},
 isbn = {978-1-60558-338-9},
 location = {Chicago, IL, USA},
 pages = {105--116},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/1572272.1572286},
 doi = {10.1145/1572272.1572286},
 acmid = {1572286},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {context-free languages, regular languages, string constraints},
}
Linux Kernel Developer Responses to Static Analysis Bug Reports. Philip J. Guo and Dawson Engler. USENIX Annual Technical Conference, short paper, 2009.
Developers like to first investigate easier-to-fix bug reports that affect newer and smaller files.
We present a study of how Linux kernel developers respond to bug reports issued by a static analysis tool. We found that developers prefer to triage reports in younger, smaller, and more actively-maintained files, first address easy-to-fix bugs and defer difficult (but possibly critical) bugs, and triage bugs in batches rather than individually. Also, although automated tools cannot find many types of bugs, they can be effective at directing developers attentions towards parts of the codebase that contain up to 3X more user-reported bugs.

Our insights into developer attitudes towards static analysis tools allow us to make suggestions for improving their usability and effectiveness. We feel that it could be effective to run static analysis tools continuously while programming and before committing code, to rank reports so that those most likely to be triaged are shown to developers first, to show the easiest reports to new developers, to perform deeper analysis on more actively-maintained code, and to use reports as indirect indicators of code quality and importance.
@inproceedings{GuoUsenix2009,
 author = {Guo, Philip J. and Engler, Dawson},
 title = {Linux Kernel Developer Responses to Static Analysis Bug Reports},
 booktitle = {Proceedings of the 2009 USENIX Annual Technical Conference},
 series = {USENIX'09},
 year = {2009},
 location = {San Diego, California},
 url = {http://dl.acm.org/citation.cfm?id=1855807.1855829},
 acmid = {1855829},
 publisher = {USENIX Association},
 address = {Berkeley, CA, USA},
}
Automatic Creation of SQL Injection and Cross-site Scripting Attacks. Adam Kiezun, Philip J. Guo, Karthick Jayaraman, Michael D. Ernst. IEEE International Conference on Software Engineering (ICSE), 2009.
Ardilla automatically finds SQL injection and cross-site scripting vulnerabilities in PHP web apps.
We present a technique for finding security vulnerabilities in Web applications. SQL Injection (SQLI) and cross-site scripting (XSS) attacks are widespread forms of attack in which the attacker crafts the input to the application to access or modify user data and execute malicious code. In the most serious attacks (called second-order, or persistent, XSS), an attacker can corrupt a database so as to cause subsequent users to execute malicious code.

This paper presents an automatic technique for creating inputs that expose SQLI and XSS vulnerabilities. The technique generates sample inputs, symbolically tracks taints through execution (including through database accesses), and mutates the inputs to produce concrete exploits. Ours is the first analysis of which we are aware that precisely addresses second-order XSS attacks.

Our technique creates real attack vectors, has few false positives, incurs no runtime overhead for the deployed application, works without requiring modification of application code, and handles dynamic programming-language constructs. We implemented the technique for PHP, in a tool Ardilla. We evaluated Ardilla on five PHP applications and found 68 previously unknown vulnerabilities (23 SQLI, 33 first-order XSS, and 12 second-order XSS).
@inproceedings{KiezunArdillaICSE2009,
 author = {Kiezun, Adam and Guo, Philip J. and Jayaraman, Karthick and Ernst, Michael D.},
 title = {Automatic Creation of {SQL} Injection and Cross-site Scripting Attacks},
 booktitle = {Proceedings of the 31st International Conference on Software Engineering},
 series = {ICSE '09},
 year = {2009},
 isbn = {978-1-4244-3453-4},
 pages = {199--209},
 numpages = {11},
 url = {http://dx.doi.org/10.1109/ICSE.2009.5070521},
 doi = {10.1109/ICSE.2009.5070521},
 acmid = {1555036},
 publisher = {IEEE Computer Society},
 address = {Washington, DC, USA},
}
Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code. Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontcheva, Scott R. Klemmer. ACM Conference on Human Factors in Computing Systems (CHI), 2009.
(Honorable Mention Paper Award)
Programmers use the web to learn on-demand, to clarify existing knowledge, and as external memory.
This paper investigates the role of online resources in problem solving. We look specifically at how programmers -- an exemplar form of knowledge workers -- opportunistically interleave Web foraging, learning, and writing code. We describe two studies of how programmers use online resources. The first, conducted in the lab, observed participants Web use while building an online chat room. We found that programmers leverage online resources with a range of intentions: They engage in just-in-time learning of new skills and approaches, clarify and extend their existing knowledge, and remind themselves of details deemed not worth remembering. The results also suggest that queries for different purposes have different styles and durations. Do programmers' queries "in the wild" have the same range of intentions, or is this result an artifact of the particular lab setting? We analyzed a month of queries to an online programming portal, examining the lexical structure, refinements made, and result pages visited. Here we also saw traits that suggest the Web is being used for learning and reminding. These results contribute to a theory of online resource usage in programming, and suggest opportunities for tools to facilitate online knowledge work.
@inproceedings{BrandtOpportunisticCHI2009,
 author = {Brandt, Joel and Guo, Philip J. and Lewenstein, Joel and Dontcheva, Mira and Klemmer, Scott R.},
 title = {Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code},
 booktitle = {Proceedings of the SIGCHI Conference on Human Factors in Computing Systems},
 series = {CHI '09},
 year = {2009},
 isbn = {978-1-60558-246-7},
 location = {Boston, MA, USA},
 pages = {1589--1598},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/1518701.1518944},
 doi = {10.1145/1518701.1518944},
 acmid = {1518944},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {copy-and-paste, opportunistic programming, prototyping},
}
Opportunistic Programming: Writing Code to Prototype, Ideate, and Discover. Joel Brandt, Philip J. Guo, Joel Lewenstein, Mira Dontcheva, Scott R. Klemmer. IEEE Software, 2009.
Opportunistic programmers prioritize speed and ease of iteration over code quality and robustness.
Five principles of opportunistic programming can help guide the development of tools that explicitly support prototyping in code.
@article{BrandtOpportunisticIEEE2009,
 author = {Brandt, Joel and Guo, Philip J. and Lewenstein, Joel and Dontcheva, Mira and Klemmer, Scott R.},
 title = {Opportunistic Programming: Writing Code to Prototype, Ideate, and Discover},
 journal = {IEEE Software},
 issue_date = {September 2009},
 volume = {26},
 number = {5},
 month = sep,
 year = {2009},
 issn = {0740-7459},
 pages = {18--24},
 numpages = {7},
 url = {http://dx.doi.org/10.1109/MS.2009.147},
 doi = {10.1109/MS.2009.147},
 acmid = {1608693},
 publisher = {IEEE Computer Society Press},
 address = {Los Alamitos, CA, USA},
 keywords = {debugging, opportunistic programming, opportunistic programming, prototyping, debugging, software engineering, prototyping, software engineering},
}

2008

Opportunistic Programming: How Rapid Ideation and Prototyping Occur in Practice. Joel Brandt, Philip J. Guo, Joel Lewenstein, Scott R. Klemmer. Workshop on End-User Software Engineering (WEUSE), 2008.
Opportunistic programmers prioritize speed and ease of iteration over code quality and robustness.
At times, programmers work opportunistically, emphasizing speed and ease of development over code robustness and maintainability. They do this to prototype, ideate, and discover; to understand as quickly as possible what the right solution is. Despite its importance, opportunistic programming remains poorly understood when compared with traditional software engineering. Through fieldwork and a laboratory study, we observed five characteristics of opportunistic programming: Programmers build software from scratch using high-level tools, often add new functionality via copy-and-paste, iterate more rapidly than in traditional development, consider code to be impermanent, and face unique debugging challenges because their applications often comprise many languages and tools composed without upfront design. Based on these characteristics, we discuss future research on tools for debugging, code foraging and reuse, and documentation that are specifically targeted at this style of development.
@inproceedings{BrandtOpportunisticWEUSE2008,
 author = {Brandt, Joel and Guo, Philip J. and Lewenstein, Joel and Klemmer, Scott R.},
 title = {Opportunistic Programming: How Rapid Ideation and Prototyping Occur in Practice},
 booktitle = {Proceedings of the 4th International Workshop on End-user Software Engineering},
 series = {WEUSE '08},
 year = {2008},
 isbn = {978-1-60558-034-0},
 location = {Leipzig, Germany},
 pages = {1--5},
 numpages = {5},
 url = {http://doi.acm.org/10.1145/1370847.1370848},
 doi = {10.1145/1370847.1370848},
 acmid = {1370848},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {end-user software engineering, opportunistic programming, prototyping},
}

2007

The Daikon system for dynamic detection of likely invariants. Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, Chen Xiao. Science of Computer Programming, 2007.
Daikon finds likely program invariants, which can improve documentation, tests, and code analyses.
Daikon is an implementation of dynamic detection of likely invariants; that is, the Daikon invariant detector reports likely program invariants. An invariant is a property that holds at a certain point or points in a program; these are often used in assert statements, documentation, and formal specifications. Examples include being constant (x = a), non-zero (x != 0), being in a range (a <= x <= b), linear relationships (y = ax+b), ordering (x <= y), functions from a library (x = fn(y)), containment (x E y), sortedness (x is sorted), and many more. Users can extend Daikon to check for additional invariants.

Dynamic invariant detection runs a program, observes the values that the program computes, and then reports properties that were true over the observed executions. Dynamic invariant detection is a machine learning technique that can be applied to arbitrary data. Daikon can detect invariants in C, C++, Java, and Perl programs, and in record-structured data sources; it is easy to extend Daikon to other applications.

Invariants can be useful in program understanding and a host of other applications. Daikon's output has been used for generating test cases, predicting incompatibilities in component integration, automating theorem proving, repairing inconsistent data structures, and checking the validity of data streams, among other tasks.

Daikon is freely available in source and binary form, along with extensive documentation, at http://pag.csail.mit.edu/daikon/.
@article{ErnstDaikonSCP2007,
  author = {Michael D. Ernst and Jeff H. Perkins and Philip J. Guo and Stephen McCamant and Carlos Pacheco and Matthew S. Tschantz and Chen Xiao},
  title = {The {Daikon} system for dynamic detection of likely invariants},
  journal = {Science of Computer Programming},
  volume = {69},
  number = {1--3},
  pages = {35--45},
  month = Dec,
  year = {2007}
}

2006

Dynamic Inference of Abstract Types. Philip J. Guo, Jeff H. Perkins, Stephen McCamant, Michael D. Ernst. International Symposium on Software Testing and Analysis (ISSTA), 2006.
DynComp infers semantically-meaningful types by running code & tracking run-time value interactions.
An abstract type groups variables that are used for related purposes in a program. We describe a dynamic unification-based analysis for inferring abstract types. Initially, each run-time value gets a unique abstract type. A run-time interaction among values indicates that they have the same abstract type, so their abstract types are unified. Also at run time, abstract types for variables are accumulated from abstract types for values. The notion of interaction may be customized, permitting the analysis to compute finer or coarser abstract types; these different notions of abstract type are useful for different tasks. We have implemented the analysis for compiled x86 binaries and for Java bytecodes. Our experiments indicate that the inferred abstract types are useful for program comprehension, improve both the results and the run time of a follow-on program analysis, and are more precise than the output of a comparable static analysis, without suffering from overfitting.
@inproceedings{GuoDyncompISSTA2006,
 author = {Guo, Philip J. and Perkins, Jeff H. and McCamant, Stephen and Ernst, Michael D.},
 title = {Dynamic Inference of Abstract Types},
 booktitle = {Proceedings of the 2006 International Symposium on Software Testing and Analysis},
 series = {ISSTA '06},
 year = {2006},
 isbn = {1-59593-263-1},
 location = {Portland, Maine, USA},
 pages = {255--265},
 numpages = {11},
 url = {http://doi.acm.org/10.1145/1146238.1146268},
 doi = {10.1145/1146238.1146268},
 acmid = {1146268},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {C, C++, Java, abstract types, dynamic analysis, interaction, mixed-level analysis, type inference, units, values and variables},
}
Automatic Inference and Enforcement of Data Structure Consistency Specifications. Brian Demsky, Michael D. Ernst, Philip J. Guo, Stephen McCamant, Jeff H. Perkins, Martin Rinard. International Symposium on Software Testing and Analysis (ISSTA), 2006.
Automatically finding invariants and repairing data structures can prevent buggy code from crashing.
Corrupt data structures are an important cause of unacceptable program execution. Data structure repair (which eliminates inconsistencies by updating corrupt data structures to conform to consistency constraints) promises to enable many programs to continue to execute acceptably in the face of otherwise fatal data structure corruption errors. A key issue is obtaining an accurate and comprehensive data structure consistency specification.

We present a new technique for obtaining data structure consistency specifications for data structure repair. Instead of requiring the developer to manually generate such specifications, our approach automatically generates candidate data structure consistency properties using the Daikon invariant detection tool. The developer then reviews these properties, potentially rejecting or generalizing overly specific properties to obtain a specification suitable for automatic enforcement via data structure repair.

We have implemented this approach and applied it to three sizable benchmark programs: CTAS (an air-traffic control system), BIND (a widely-used Internet name server) and Freeciv (an interactive game). Our results indicate that (1) automatic constraint generation produces constraints that enable programs to execute successfully through data structure consistency errors, (2) compared to manual specification, automatic generation can produce more comprehensive sets of constraints that cover a larger range of data structure consistency properties, and (3) reviewing the properties is relatively straightforward and requires substantially less programmer effort than manual generation, primarily because it reduces the need to examine the program text to understand its operation and extract the relevant consistency constraints. Moreover, when evaluated by a hostile third party "Red Team" contracted to evaluate the effectiveness of the technique, our data structure inference and enforcement tools successfully prevented several otherwise fatal attacks.
@inproceedings{DemskyInferISSTA2006,
 author = {Demsky, Brian and Ernst, Michael D. and Guo, Philip J. and McCamant, Stephen and Perkins, Jeff H. and Rinard, Martin},
 title = {Inference and Enforcement of Data Structure Consistency Specifications},
 booktitle = {Proceedings of the 2006 International Symposium on Software Testing and Analysis},
 series = {ISSTA '06},
 year = {2006},
 isbn = {1-59593-263-1},
 location = {Portland, Maine, USA},
 pages = {233--244},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/1146238.1146266},
 doi = {10.1145/1146238.1146266},
 acmid = {1146266},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {data structure repair, dynamic invariant detection},
}
A Scalable Mixed-Level Approach to Dynamic Analysis of C and C++ Programs. Philip J. Guo. Master of Engineering (M.Eng.) thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 2006.
(MIT Charles and Jennifer Johnson Award for Outstanding Computer Science Master of Engineering Thesis)
Combining source- and binary-level data makes it easier to build tools that analyze C and C++ code.
This thesis addresses the difficult task of constructing robust and scalable dynamic program analysis tools for programs written in memory-unsafe languages such as C and C++, especially those that are interested in observing the contents of data structures at run time. In this thesis, I first introduce my novel mixed-level approach to dynamic analysis, which combines the advantages of both source- and binary-based approaches. Second, I present a tool framework that embodies the mixed-level approach. This framework provides memory safety guarantees, allows tools built upon it to access rich source- and binary-level information simultaneously at run time, and enables tools to scale to large, real-world C and C++ programs on the order of millions of lines of code. Third, I present two dynamic analysis tools built upon my framework -- one for performing value profiling and the other for performing dynamic inference of abstract types -- and describe how they far surpass previous analyses in terms of scalability, robustness, and applicability. Lastly, I present several case studies demonstrating how these tools aid both humans and automated tools in several program analysis tasks: improving human understanding of unfamiliar code, invariant detection, and data structure repair.
@mastersthesis{GuoMEngThesis2006,
   author = {Philip J. Guo},
   title = {A Scalable Mixed-Level Approach to Dynamic Analysis of {C} and
  {C++} Programs},
   school = {MIT Department of Electrical Engineering and Computer Science},
   address = {Cambridge, MA},
   month = May,
   year = {2006}
}