Philip Guo (Phil Guo, Philip J. Guo, Philip Jia Guo, pgbovine)

Research Design Patterns

Summary
In the spirit of software design patterns, here are some examples of design patterns for academic research, especially in engineering and technology-related fields.

I often get asked, “How do people come up with research ideas?” Everyone has their own techniques, but I've noticed some common patterns throughout the years. In the spirit of software design patterns, here are some design patterns for academic research that I've seen in fields near mine. Although there are many great examples from others, I'll use some of my own papers as examples here because I don't want to speculate about how others came up with their ideas; all I know is how I've come up with mine!

Lots of research advice basically boils down to “come up with something new that nobody else has thought of yet.” This kind of advice is way too vague, so I'm trying to make it more concrete by listing a set of design patterns. Note that these patterns don't dictate a specific type of research contribution (e.g., an algorithm, toolkit, programming language, system, empirical study, data analysis, or theoretical analysis), although they were developed with engineering and technology research in mind. Here are my top ten:

  1. Seinfeld
  2. Inside-Out
  3. All the Things
  4. Best of Both Worlds
  5. Back to the Future
  6. Horizontal Transfer
  7. Vertical Transfer
  8. Mr. Beast
  9. Force of Nature
  10. Nelson - HA HA!

Seinfeld

Jerry Seinfeld made his comedy career by asking, “What's the deal with ...” (even though he didn't actually say that line much).

To apply the Seinfeld pattern in your research, dig deep into something that's right there in front of everyone but others haven't investigated yet. For instance, I noticed that pretty much everyone around me focused on younger people (e.g., children, college students) learning programming but clearly people of all ages were learning. So I asked, “What's the deal with older adults learning programming?”

Older Adults Learning Computer Programming: Motivations, Frustrations, and Design Opportunities. Philip J. Guo. ACM Conference on Human Factors in Computing Systems (CHI), 2017.
(Honorable Mention Paper Award)
Computer programming is a highly in-demand skill, but most learn-to-code initiatives and research target some of the youngest members of society: children and college students. We present the first known study of older adults learning computer programming. Using an online survey with 504 respondents aged 60 to 85 who are from 52 different countries, we discovered that older adults were motivated to learn to keep their brains challenged as they aged, to make up for missed opportunities during youth, to connect with younger family members, and to improve job prospects. They reported frustrations including a perceived decline in cognitive abilities, lack of opportunities to interact with tutors and peers, and trouble dealing with constantly-changing software technologies. Based on these findings, we propose a learner-centered design of techniques and tools for motivating older adults to learn programming and discuss broader societal implications of a future where more older adults have access to computer programming -- not merely computer literacy -- as a skill set.
@inproceedings{GuoCHI2017,
 author = {Guo, Philip J.},
 title = {Older Adults Learning Computer Programming: Motivations, Frustrations, and Design Opportunities},
 booktitle = {Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '17},
 year = {2017},
 isbn = {978-1-4503-4655-9},
 location = {Denver, Colorado, USA},
 pages = {7070--7083},
 numpages = {14},
 url = {http://doi.acm.org/10.1145/3025453.3025945},
 doi = {10.1145/3025453.3025945},
 acmid = {3025945},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {computational literacy, learning programming, older adults},
}

And my Ph.D. student Sean Kross was like (in a Seinfeld voice), “What's the deal with all these people now teaching data science? I mean, who ARE these people? What ARE they doing?”

Practitioners Teaching Data Science in Industry and Academia: Expectations, Workflows, and Challenges. Sean Kross and Philip J. Guo. ACM Conference on Human Factors in Computing Systems (CHI), 2019.
(Honorable Mention Paper Award)
Data science has been growing in prominence across both academia and industry, but there is still little formal consensus about how to teach it. Many people who currently teach data science are practitioners such as computational researchers in academia or data scientists in industry. To understand how these practitioner-instructors pass their knowledge onto novices and how that contrasts with teaching more traditional forms of programming, we interviewed 20 data scientists who teach in settings ranging from small-group workshops to large online courses. We found that: 1) they must empathize with a diverse array of student backgrounds and expectations, 2) they teach technical workflows that integrate authentic practices surrounding code, data, and communication, 3) they face challenges involving authenticity versus abstraction in software setup, finding and curating pedagogically-relevant datasets, and acclimating students to live with uncertainty in data analysis. These findings can point the way toward better tools for data science education and help bring data literacy to more people around the world.
@inproceedings{KrossCHI2019,
 author = {Kross, Sean and Guo, Philip J.},
 title = {Practitioners Teaching Data Science in Industry and Academia: Expectations, Workflows, and Challenges},
 booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
 series = {CHI '19},
 year = {2019},
 isbn = {978-1-4503-5970-2},
 location = {Glasgow, Scotland Uk},
 pages = {263:1--263:14},
 articleno = {263},
 numpages = {14},
 url = {http://doi.acm.org/10.1145/3290605.3300493},
 doi = {10.1145/3290605.3300493},
 acmid = {3300493},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {data science education, teaching programming},
}

To make this pattern work well, you need to be uniquely positioned to get insights that others can't, and your topic must be something timely that others care about now (probably nobody in 2019 cares about, What's the deal with Windows 3.11?). For my older adults paper, I had access to thousands of daily visitors to my Python Tutor site from all age demographics. For Sean's paper, he had been teaching data science for years and had direct access to those at the leading edge of teaching this topic.

Risks: This pattern is great for getting started on research and for inspiring follow-up ideas, but it can be hard to build an entire portfolio out of just asking what's the deal with a bunch of different topics (unless you're actually Seinfeld). It risks coming off as superficial and scattershot if you do it too frequently.

Inside-Out

This is one of my favorite patterns, but I don't have a concise name for it. It comes down to: turn commonly-accepted assumptions inside-out in an extreme way and run with it.

For example, everyone knows that to do data science, you need to bring your data to your code, which means setting up a coding environment then downloading data sets or using APIs to pull data into where your code is located. But let's turn that common assumption inside-out: what if instead you bring your code to your data? What does that even mean? How would it ever work? And what would it be good for? We explored these ideas in DS.js:

DS.js: Turn Any Webpage into an Example-Centric Live Programming Environment for Learning Data Science. Xiong Zhang and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2017.
(Honorable Mention Paper Award)
Data science courses and tutorials have grown popular in recent years, yet they are still taught using production-grade programming tools (e.g., R, MATLAB, and Python IDEs) within desktop computing environments. Although powerful, these tools present high barriers to entry for novices, forcing them to grapple with the extrinsic complexities of software installation and configuration, data file management, data parsing, and Unix-like command-line interfaces. To lower the barrier for novices to get started with learning data science, we created DS.js, a bookmarklet that embeds a data science programming environment directly into any existing webpage. By transforming any webpage into an example-centric IDE, DS.js eliminates the aforementioned complexities of desktop-based environments and turns the entire web into a rich substrate for learning data science. DS.js automatically parses HTML tables and CSV/TSV data sets on the target webpage, attaches code editors to each data set, provides a data table manipulation and visualization API designed for novices, and gives instructional scaffolding in the form of bidirectional previews of how the user's code and data relate.
@inproceedings{ZhangUIST2017,
 author = {Zhang, Xiong and Guo, Philip J.},
 title = {DS.Js: Turn Any Webpage into an Example-Centric Live Programming Environment for Learning Data Science},
 booktitle = {Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '17},
 year = {2017},
 isbn = {978-1-4503-4981-9},
 location = {Qu\&\#233;bec City, QC, Canada},
 pages = {691--702},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/3126594.3126663},
 doi = {10.1145/3126594.3126663},
 acmid = {3126663},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {data science, live programming, novice programmers},
}

This is one of my favorite brainstorming techniques: List the most obvious and commonly-accepted assumptions in your domain ... then turn each one inside-out in the most absurd way possible and see what you can do with it. Most of the time, nothing. But a gem might emerge if conditions are right.

Risks: If not done well, this risks coming off as trying to be annoyingly contrarian just for the sake of it.

All the things

ALL THE THINGS was a popular meme in 2010. To apply this pattern to your research, thing about stretching an idea to the most ridiculous extreme to encompass ALL THE THINGS. Two examples from my own work come to mind:

Example 1: Code debugging and visualization tools usually display a small number of variable values at once at certain time steps. It's impractical to display more values on screen due to lack of space. But my student Hyeonsu and I wanted to display more values on screen at once. How many more? Well, how about all of them? We wanted our tool to DISPLAY ALL THE VALUES!

This idea seems ridiculously impractical, but we built a prototype called Omnicode (based on Python Tutor) that tried to do this:

Omnicode: A Novice-Oriented Live Programming Environment with Always-On Run-Time Value Visualizations. Hyeonsu Kang and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2017.
Visualizations of run-time program state help novices form proper mental models and debug their code. We push this technique to the extreme by posing the following question: What if a live programming environment for an imperative language always displays the entire history of all run-time values for all program variables all the time? To explore this question, we built a prototype live IDE called Omnicode ("Omniscient Code") that continually runs the user's Python code and uses a scatterplot matrix to visualize the entire history of all of its numerical values, along with meaningful numbers derived from other data types. To filter the visualizations and hone in on specific points of interest, the user can brush and link over the scatterplots or select portions of code. They can also zoom in to view detailed stack and heap visualizations at each execution step. An exploratory study on 10 novice programmers discovered that they found Omnicode to be useful for debugging, forming mental models, explaining their code to others, and discovering moments of serendipity that would not have been likely within an ordinary IDE.
@inproceedings{KangUIST2017,
 author = {Kang, Hyeonsu and Guo, Philip J.},
 title = {Omnicode: A Novice-Oriented Live Programming Environment with Always-On Run-Time Value Visualizations},
 booktitle = {Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '17},
 year = {2017},
 isbn = {978-1-4503-4981-9},
 location = {Qu\&\#233;bec City, QC, Canada},
 pages = {737--745},
 numpages = {9},
 url = {http://doi.acm.org/10.1145/3126594.3126632},
 doi = {10.1145/3126594.3126632},
 acmid = {3126632},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {always-on visualizations, live programming},
}

Example 2: An expert tutor can do a good job helping a single student at once in real time. If they're good, maybe they can switch back and forth between a handful of students. But what about an entire class of students? With proper tool support, how many students could a single tutor help? 10? 20? 50? 100? 1000? I wanted to let a single tutor HELP ALL THE STUDENTS!

This idea also seems ridiculously impractical on its face, but I built a prototype called Codeopticon (again based on Python Tutor) that tried to do this:

Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming. Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2015.
One-on-one tutoring from a human expert is an effective way for novices to overcome learning barriers in complex domains such as computer programming. But there are usually far fewer experts than learners. To enable a single expert to help more learners at once, we built Codeopticon, an interface that enables a programming tutor to monitor and chat with dozens of learners in real time. Each learner codes in a workspace that consists of an editor, compiler, and visual debugger. The tutor sees a real-time view of each learner's actions on a dashboard, with each learner's workspace summarized in a tile. At a glance, the tutor can see how learners are editing and debugging their code, and what errors they are encountering. The dashboard automatically reshuffles tiles so that the most active learners are always in the tutor's main field of view. When the tutor sees that a particular learner needs help, they can open an embedded chat window to start a one-on-one conversation. A user study showed that 8 first-time Codeopticon users successfully tutored anonymous learners from 54 countries in a naturalistic online setting. On average, in a 30-minute session, each tutor monitored 226 learners, started 12 conversations, exchanged 47 chats, and helped 2.4 learners.
@inproceedings{GuoUIST2015,
 author = {Guo, Philip J.},
 title = {Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming},
 booktitle = {Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '15},
 year = {2015},
 isbn = {978-1-4503-3779-3},
 pages = {599--608},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2807442.2807469},
 doi = {10.1145/2807442.2807469},
 acmid = {2807469},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {computer programming, learning at scale, remote tutoring}
}

I love this pattern because it provides an aspirational starting point for ideas. Even if you don't end up at the ultimate end goal, thinking about the extreme of achieving ALL THE THINGS at the largest possible scale can push you to come up with novel designs.

(I tried to slide in a Blink-182 All the Small Things reference here but couldn't quite make it work.)

Risks: You risk overpromising and underdelivering if you talk big but show only a partial solution with incremental improvements.

Best of Both Worlds

In one of the most epic episodes of Star Trek: TNG, Captain Picard gets captured and turned into a human-machine hybrid with the best of both his human intuitions and the collective computational powers of the Borg:

To apply this pattern in your research, think about two existing things that each have complementary benefits and drawbacks and try to combine them into a hybrid with the best aspects of both. For instance, I noticed that when people teach programming or give code-related talks, they often use one of two formats:

  • Slide presentations:
    • Benefits: well-organized, visually clean
    • Drawbacks: can't run the code shown on slides, too rigid, can't improvise
  • Live coding:
    • Benefits: can run code live, authentic, can easily improvise
    • Drawbacks: disorganized, visually messy, lacks context

To get the best of both worlds, my master's student Charles and I designed a hybrid code presentation system that combines PowerPoint-style slides and live coding in a new way:

Improv: Teaching Programming at Scale via Live Coding. Charles Chen and Philip J. Guo. ACM Conference on Learning at Scale, 2019.
Computer programming instructors frequently perform live coding in settings ranging from MOOC lecture videos to online livestreams. However, there is little tool support for this mode of teaching, so presenters must now either screen-share or use generic slideshow software. To overcome the limitations of these formats, we propose that programming environments should directly facilitate live coding for education. We prototyped this idea by creating Improv, an IDE extension for preparing and delivering code-based presentations informed by Mayer's principles of multimedia learning. Improv lets instructors synchronize blocks of code and output with slides and create preset waypoints to guide their presentations. A case study on 30 educational videos containing 28 hours of live coding showed that Improv was versatile enough to replicate approximately 96% of the content within those videos. In addition, a preliminary user study on four teaching assistants showed that Improv was expressive enough to allow them to make their own custom presentations in a variety of styles and improvise by live coding in response to simulated audience questions. Users mentioned that Improv lowered cognitive load by minimizing context switching and made it easier to fix errors on-the-fly than using slide-based presentations.
@inproceedings{ChenLAS2019,
 author = {Chen, Charles and Guo, Philip J.},
 title = {Improv: Teaching Programming at Scale via Live Coding},
 booktitle = {Proceedings of the Sixth Annual ACM Conference on Learning at Scale},
 series = {L@S '19},
 year = {2019},
 numpages = {10},
 url = {https://doi.org/10.1145/3330430.3333627},
 doi = {10.1145/3330430.3333627},
 publisher = {ACM},
 address = {New York, NY, USA},
}

Risks: If not done well, this risks coming off as a clunky “duct tape” approach where you just throw different components together haphazardly without blending them into a hybrid.

Back to the Future

We're going back ... back to the future! This pattern involves taking older ideas in your field that were hot a few decades ago and adding a modern twist to them. For instance, web mashups were hot around the mid to late 2000s (when Web 2.0 started), but then interest in them simmered down. My student Xiong and I noticed that everyone was simply mashing up web data but instead what if we mashed up the actual user interfaces of web apps? While this idea was in theory feasible back in the day, the cornucopia of modern web apps in 2019 makes it a much more compelling environment to implement such an idea. The time felt right for Fusion:

Fusion: Opportunistic Web Prototyping with UI Mashups. Xiong Zhang and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2018.
Modern web development is rife with complexity at all layers, ranging from needing to configure backend services to grappling with frontend frameworks and dependencies. To lower these development barriers, we introduce a technique that enables people to prototype opportunistically by borrowing pieces of desired functionality from across the web without needing any access to their underlying codebases, build environments, or server backends. We implemented this technique in a browser extension called Fusion, which lets users create web UI mashups by extracting components from existing unmodified webpages and hooking them together using transclusion and JavaScript glue code. We demonstrate the generality and versatility of Fusion via a case study where we used it to create seven UI mashups in domains such as programming tools, data science, web design, and collaborative work. Our mashups include replicating portions of prior HCI systems (Blueprint for in-situ code search and DS.js for in-browser data science), extending the p5.js IDE for Processing with real-time collaborative editing, and integrating Python Tutor code visualizations into static tutorials. These UI mashups each took less than 15 lines of JavaScript glue code to create with Fusion.
@inproceedings{ZhangUIST2018,
 author = {Zhang, Xiong and Guo, Philip J.},
 title = {Fusion: Opportunistic Web Prototyping with UI Mashups},
 booktitle = {Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '18},
 year = {2018},
 isbn = {978-1-4503-5948-1},
 location = {Berlin, Germany},
 pages = {951--962},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/3242587.3242632},
 doi = {10.1145/3242587.3242632},
 acmid = {3242632},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {opportunistic programming, ui mashups, web prototyping},
}

This isn't the best example, but it's one that I've personally worked on. Better examples include applying classic research ideas to emerging new domains such as augmented reality, virtual reality, 3D fabrication, embedded devices, multimodal interfaces, etc. The key is to have a novel take on a classic idea that's made possible because of conditions today that didn't exist in the past.

Risks: If you don't put a fresh enough spin on classic ideas, others might perceive your research as being too old-school and already done decades ago, since they ARE old ideas, after all.

Horizontal transfer

To use this pattern, transfer a well-known idea in one field horizontally to another field. My favorite personal example is Porta, where we took the idea of code profiling from software engineering and transferred it to the HCI / online learning world by applying it to improve online tutorials. We called the resulting technique tutorial profiling:

Porta: Profiling Software Tutorials Using Operating-System-Wide Activity Tracing. Alok Mysore and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2018.
(Best Paper Award)
It can be hard for tutorial creators to get fine-grained feedback about how learners are actually stepping through their tutorials and which parts lead to the most struggle. To provide such feedback for technical software tutorials, we introduce the idea of tutorial profiling, which is inspired by software code profiling. We prototyped this idea in a system called Porta that automatically tracks how users navigate through a tutorial webpage and what actions they take on their computer such as running shell commands, invoking compilers, and logging into remote servers. Porta surfaces this trace data in the form of profiling visualizations that augment the tutorial with heatmaps of activity hotspots and markers that expand to show event details, error messages, and embedded screencast videos of user actions. We found through a user study of 3 tutorial creators and 12 students who followed their tutorials that Porta enabled both the tutorial creators and the students to provide more specific, targeted, and actionable feedback about how to improve these tutorials. Porta opens up possibilities for performing user testing of technical documentation in a more systematic and scalable way.
@inproceedings{MysoreUIST2018,
 author = {Mysore, Alok and Guo, Philip J.},
 title = {Porta: Profiling Software Tutorials Using Operating-System-Wide Activity Tracing},
 booktitle = {Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '18},
 year = {2018},
 isbn = {978-1-4503-5948-1},
 location = {Berlin, Germany},
 pages = {201--212},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/3242587.3242633},
 doi = {10.1145/3242587.3242633},
 acmid = {3242633},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {activity tracing, software tutorials, tutorial profiling},
}

People who come from interdisciplinary backgrounds are really well-positioned to do horizontal transfer: I spent my early research career in software engineering and then transferred to HCI / online learning, so it's no surprise that I applied that to my own research too. This design pattern is extremely powerful since you don't need to transfer a mindblowing idea – code profiling is a pretty mundane idea in 2019, but when applied to profiling online tutorials, it suddenly becomes fresh again!

Risks: If the domains you're transferring ideas between are too far apart, then people might not get what you're trying to do.

Vertical transfer

Since I mentioned horizontal transfer, it makes sense to follow with vertical transfer. I'm not sure how well this one generalizes, but here we go: Vertical transfer means using lower-level machinery to achieve higher-level goals, thus transferring ideas from lower to higher levels of abstraction.

Our Burrito and Torta (food-inspired) systems are my favorite personal examples of vertical transfer. For both, we implemented low-level operating-system-wide tracing functionality, which is usually used for low-level tasks such as debugging, profiling, and error reporting at the operating system level. However, we “lifted” that data up to a higher level of abstraction to generate electronic lab notebooks for scientists (for Burrito) and step-by-step software tutorials (for Torta), respectively.

Burrito: Wrapping Your Lab Notebook in Computational Infrastructure. Philip J. Guo and Margo Seltzer. USENIX Workshop on the Theory and Practice of Provenance (TaPP), 2012.
Researchers in fields such as bioinformatics, CS, finance, and applied math have trouble managing the numerous code and data files generated by their computational experiments, comparing the results of trials executed with different parameters, and keeping up-to-date notes on what they learned from past successes and failures.

We created a Linux-based system called Burrito that automates aspects of this tedious experiment organization and notetaking process, thus freeing researchers to focus on more substantive work. Burrito automatically captures a researcher's computational activities and provides user interfaces to annotate the captured provenance with notes and then make queries such as, "Which script versions and command-line parameters generated the output graph that this note refers to?"
@inproceedings{GuoBurrito2012,
  author = {Guo, Philip J. and Seltzer, Margo},
  title = {BURRITO: Wrapping Your Lab Notebook in Computational Infrastructure},
  booktitle = {Proceedings of the 4th USENIX Workshop on the Theory and Practice of Provenance},
  series = {TaPP'12},
  year = {2012},
  location = {Boston, MA},
  url = {http://dl.acm.org/citation.cfm?id=2342875.2342882},
  acmid = {2342882},
  publisher = {USENIX Association},
  address = {Berkeley, CA, USA},
}
Torta: Generating Mixed-Media GUI and Command-Line App Tutorials Using Operating-System-Wide Activity Tracing. Alok Mysore and Philip J. Guo. ACM Symposium on User Interface Software and Technology (UIST), 2017.
Tutorials are vital for helping people perform complex software-based tasks in domains such as programming, data science, system administration, and computational research. However, it is tedious to create detailed step-by-step tutorials for tasks that span multiple interrelated GUI and command-line applications. To address this challenge, we created Torta, an end-to-end system that automatically generates step-by-step GUI and command-line app tutorials by demonstration, provides an editor to trim, organize, and add validation criteria to these tutorials, and provides a web-based viewer that can validate step-level progress and automatically run certain steps. The core technical insight that underpins Torta is that combining operating-system-wide activity tracing and screencast recording makes it easier to generate mixed-media (text+video) tutorials that span multiple GUI and command-line apps. An exploratory study on 10 computer science teaching assistants (TAs) found that they all preferred the experience and results of using Torta to record programming and sysadmin tutorials relevant to classes they teach rather than manually writing tutorials. A follow-up study on 6 students found that they all preferred following the Torta tutorials created by those TAs over the manually-written versions.
@inproceedings{MysoreUIST2017,
 author = {Mysore, Alok and Guo, Philip J.},
 title = {Torta: Generating Mixed-Media GUI and Command-Line App Tutorials Using Operating-System-Wide Activity Tracing},
 booktitle = {Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology},
 series = {UIST '17},
 year = {2017},
 isbn = {978-1-4503-4981-9},
 location = {Qu\&\#233;bec City, QC, Canada},
 pages = {703--714},
 numpages = {12},
 url = {http://doi.acm.org/10.1145/3126594.3126628},
 doi = {10.1145/3126594.3126628},
 acmid = {3126628},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {mixed-media tutorials, software tutorials, tutorial generation},
}

Just like horizontal transfer, coming from an interdisciplinary background helps when trying to do vertical transfer. You're in a position to see related ideas at different levels of the technology stack that others who work at a single level don't notice.

Risks: If the higher-level goal isn't compelling, then it might come off as you bringing out the big guns of overly-complex low-level machinery just for the sake of doing so.

Mr. Beast

Mr. Beast is a YouTuber known for putting an absurd number of hours into seemingly-mundane things, like counting from 1 to 100,000. More broadly, he prides himself for being willing to put in 10x or more hours than everyone else in producing his videos. Check out a snippet of this great interview starting at 11:50 ...

To lightly paraphrase: “If other people are spending 1 hour thinking of video ideas and 5 hours filming, I want to spend 10 hours thinking of ideas and several days filming. It's not just spending more time on ideas, it's more time on everything.”

You can channel Mr. Beast in your research by being willing to spend an absurd amount of time doing what your peers won't. Paradoxically, by grinding on seemingly mundane tasks, you can actually get deeper insights that others overlook simply because they haven't put in the raw hours. I've gotten many great ideas this way, not because I'm any smarter than my peers, but just because I was willing to put in the hours when others weren't.

For instance, when I was a visiting summer researcher at edX in the early MOOC days, I spent countless hours watching 862 lecture videos from four courses and taking notes on how each was structured. This absurd amount of work both gave me some unique insights for one of my currently most-cited papers ...

How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos. Philip J. Guo, Juho Kim, Rob Rubin. ACM Conference on Learning at Scale, 2014.
Videos are a widely-used kind of resource for online learning. This paper presents an empirical study of how video production decisions affect student engagement in online educational videos. To our knowledge, ours is the largest-scale study of video engagement to date, using data from 6.9 million video watching sessions across four courses on the edX MOOC platform. We measure engagement by how long students are watching each video, and whether they attempt to answer post-video assessment problems.

Our main findings are that shorter videos are much more engaging, that informal talking-head videos are more engaging, that Khan-style tablet drawings are more engaging, that even high-quality pre-recorded classroom lectures might not make for engaging online videos, and that students engage differently with lecture and tutorial videos.

Based upon these quantitative findings and qualitative insights from interviews with edX staff, we developed a set of recommendations to help instructors and video producers take better advantage of the online video format.
@inproceedings{GuoLAS2014vidprod,
 author = {Guo, Philip J. and Kim, Juho and Rubin, Rob},
 title = {How Video Production Affects Student Engagement: An Empirical Study of MOOC Videos},
 booktitle = {Proceedings of the First ACM Conference on Learning @ Scale Conference},
 series = {L@S '14},
 year = {2014},
 isbn = {978-1-4503-2669-8},
 location = {Atlanta, Georgia, USA},
 pages = {41--50},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/2556325.2566239},
 doi = {10.1145/2556325.2566239},
 acmid = {2566239},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {mooc, online education, video engagement},
}

... but more importantly, it gave me deep intuitions about how educational videos are made, which ended up leading to follow-up research ideas over the ensuing years. I remember my manager at edX generously offering to pay for lower-cost temp workers to look through those videos and label them for me since he didn't get why I wanted to go through such a tedious grind myself; it just didn't seem cost-effective! But I refused because I knew that I had to go through that grind to gain those insights myself.

Then during my first year as a professor, I continued this tradition by getting my first group of students together and reading all 5,377 discussion forum posts from an online programming course. Again this resulted in unique research findings for a paper ...

Toward a Domain-Specific Visual Discussion Forum for Learning Computer Programming: An Empirical Study of a Popular MOOC Forum. Joyce Zhu, Jeremy Warner, Mitchell Gordon, Jeffery White, Renan Zanelatto, Philip J. Guo. IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2015.
Online discussion forums are one of the most ubiquitous kinds of resources for people who are learning computer programming. However, their user interface -- a hierarchy of textual threads -- has not changed much in the past four decades. We argue that generic forum interfaces are cumbersome for learning programming and that there is a need for a domain-specific visual discussion forum for programming. We support this argument with an empirical study of all 5,377 forum threads in Introduction to Computer Science and Programming Using Python, a popular edX MOOC. Specifically, we investigated how forum participants were hampered by its text-based format. Most notably, people often wanted to discuss questions about dynamic execution state -- what happens "under the hood" as the computer runs code. We propose that a better forum for learning programming should be visual and domain-specific, integrating automatically-generated visualizations of execution state and enabling inline annotations of source code and output.
@inproceedings{ZhuVLHCC2015csforums,
  author={Zhu, Joyce and Warner, Jeremy and Gordon, Mitchell and White, Jeffery and Zanelatto, Renan and Guo, Philip J.},
  title={Toward a Domain-Specific Visual Discussion Forum for Learning Computer Programming: An Empirical Study of a Popular MOOC Forum},
  booktitle = {Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)},
  series = {VL/HCC '15},
  year={2015},
  pages={101-109},
  doi={10.1109/VLHCC.2015.7357193},
  month={Oct}
}

... but more importantly, it gave us intuitions about how learners interacted online, which powered years of follow-up work.

Research is about seeing what others don't; one way to do so is to put in more time than everyone else doing mundane tasks that others think is beneath them. A shining example of such a mundane task is looking at the raw data (look at Tip 2 on this page). People are often in a huge rush to write scripts to clean, process, and analyze their data; instead, first spend significant amounts of time looking at it, like we did with the 862 lecture videos and 5,377 forum posts. You'll get insights that others won't.

Risks: Obviously this can be a gigantic waste of time and demoralizing if not done well. Not all mundane tasks are fruitful.

Force of nature

Force of Nature was one of the earliest power creatures in Magic: The Gathering. Here's the beautiful Alpha printing of the card:

It wasn't subtle: you get an 8/8 creature that can deal 8 damage with trample every turn to utterly demolish your opponents ... but you had to pay 4 green (G) mana of upkeep every turn or else it would demolish YOU with 8 damage.

I'm a fan of finesse and elegance in research, but sometimes you just need to bust out the Force of Nature and get 8/8 trample on your side. For me, it's the Python Tutor platform that I've built up over the past decade, growing it to a site that tens of thousands of people use every day to learn programming. This platform has powered several research papers and, recently, my NSF CAREER proposal. Writing that proposal was not an exercise in subtlety.

This approach works when you have access to a piece of technology or infrastructure that others don't, so you're well-positioned to do certain kinds of research that your peers can't. But you need to pay the 4 mana of upkeep every turn to maintain it or else it deals 8 damage to you!

Risks: Force of Nature projects can have a tremendously high upkeep and maintenance costs, which can eat you alive if you don't manage them well. For instance, most of the time that I've sunk into keeping the Python Tutor site up and running throughout the past decade of growth provides no research value.

Nelson - HA HA!

Nelson from The Simpsons has a great catchphrase: Ha Ha!

Nelson research happens when you have access to some unique data set that others don't. Then it's relatively easy to crank out some simple analysis on that data and get it published, at which point you channel your inner Nelson and say, Ha Ha!

My 2009 Microsoft Research internship project followed this pattern to a tee. Our team had access to a cornucopia of internal proprietary data about software development processes at Microsoft that we chopped and pureed into a trilogy of papers:

Characterizing and Predicting Which Bugs Get Fixed: An Empirical Study of Microsoft Windows. Philip J. Guo, Thomas Zimmermann, Nachiappan Nagappan, Brendan Murphy. IEEE International Conference on Software Engineering (ICSE), 2010.
We performed an empirical study to characterize factors that affect which bugs get fixed in Windows Vista and Windows 7, focusing on factors related to bug report edits and relationships between people involved in handling the bug. We found that bugs reported by people with better reputations were more likely to get fixed, as were bugs handled by people on the same team and working in geographical proximity. We reinforce these quantitative results with survey feedback from 358 Microsoft employees who were involved in Windows bugs. Survey respondents also mentioned additional qualitative influences on bug fixing, such as the importance of seniority and interpersonal skills of the bug reporter.

Informed by these findings, we built a statistical model to predict the probability that a new bug will be fixed (the first known one, to the best of our knowledge). We trained it on Windows Vista bugs and got a precision of 68% and recall of 64% when predicting Windows 7 bug fixes. Engineers could use such a model to prioritize bugs during triage, to estimate developer workloads, and to decide which bugs should be closed or migrated to future product versions.
@inproceedings{GuoICSE2010,
 author = {Guo, Philip J. and Zimmermann, Thomas and Nagappan, Nachiappan and Murphy, Brendan},
 title = {Characterizing and Predicting Which Bugs Get Fixed: An Empirical Study of {Microsoft Windows}},
 booktitle = {Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1},
 series = {ICSE '10},
 year = {2010},
 isbn = {978-1-60558-719-6},
 location = {Cape Town, South Africa},
 pages = {495--504},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/1806799.1806871},
 doi = {10.1145/1806799.1806871},
 acmid = {1806871},
 publisher = {ACM},
 address = {New York, NY, USA},
}
"Not My Bug!" and Other Reasons for Software Bug Report Reassignments. Philip J. Guo, Thomas Zimmermann, Nachiappan Nagappan, Brendan Murphy. ACM Conference on Computer Supported Cooperative Work (CSCW), 2011.
Bug reporting/fixing is an important social part of the software development process. The bug-fixing process inherently has strong inter-personal dynamics at play, especially in how to find the optimal person to handle a bug report. Bug report reassignments, which are a common part of the bug-fixing process, have rarely been studied.

In this paper, we present a large-scale quantitative and qualitative analysis of the bug reassignment process in the Microsoft Windows Vista operating system project. We quantify social interactions in terms of both useful and harmful reassignments. For instance, we found that reassignments are useful to determine the best person to fix a bug, contrary to the popular opinion that reassignments are always harmful. We categorized five primary reasons for reassignments: finding the root cause, determining ownership, poor bug report quality, hard to determine proper fix, and workload balancing. We then use these findings to make recommendations for the design of more socially-aware bug tracking systems that can overcome some of the inefficiencies we observed in our study.
@inproceedings{GuoCSCW2011,
 author = {Guo, Philip J. and Zimmermann, Thomas and Nagappan, Nachiappan and Murphy, Brendan},
 title = {{"Not My Bug!"} and Other Reasons for Software Bug Report Reassignments},
 booktitle = {Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work},
 series = {CSCW '11},
 year = {2011},
 isbn = {978-1-4503-0556-3},
 location = {Hangzhou, China},
 pages = {395--404},
 numpages = {10},
 url = {http://doi.acm.org/10.1145/1958824.1958887},
 doi = {10.1145/1958824.1958887},
 acmid = {1958887},
 publisher = {ACM},
 address = {New York, NY, USA},
 keywords = {bug reassignment, bug tracking, bug triaging},
}
Characterizing and Predicting Which Bugs Get Reopened. Thomas Zimmermann, Nachiappan Nagappan, Philip J. Guo, Brendan Murphy. ACM/IEEE International Conference on Software Engineering (ICSE), Software Engineering In Practice (SEIP) track, 2012.
(Best Paper Award)
Fixing bugs is an important part of the software development process. An underlying aspect is the effectiveness of fixes: if a fair number of fixed bugs are reopened, it could indicate instability in the software system. To the best of our knowledge there has been on little prior work on understanding the dynamics of bug reopens. Towards that end, in this paper, we characterize when bug reports are reopened by using the Microsoft Windows operating system project as an empirical case study. Our analysis is based on a mixed-methods approach. First, we categorize the primary reasons for reopens based on a survey of 358 Microsoft employees. We then reinforce these results with a large-scale quantitative study of Windows bug reports, focusing on factors related to bug report edits and relationships between people involved in handling the bug. Finally, we build statistical models to describe the impact of various metrics on reopening bugs ranging from the reputation of the opener to how the bug was found.
@inproceedings{ZimmermannReopen2012,
 author = {Zimmermann, Thomas and Nagappan, Nachiappan and Guo, Philip J. and Murphy, Brendan},
 title = {Characterizing and Predicting Which Bugs Get Reopened},
 booktitle = {Proceedings of the 2012 International Conference on Software Engineering},
 series = {ICSE 2012},
 year = {2012},
 isbn = {978-1-4673-1067-3},
 location = {Zurich, Switzerland},
 pages = {1074--1083},
 numpages = {10},
 url = {http://dl.acm.org/citation.cfm?id=2337223.2337363},
 acmid = {2337363},
 publisher = {IEEE Press},
 address = {Piscataway, NJ, USA},
}

The data analyses we did weren't mindblowingly complex at all; anyone else could've done them if only they had access to our data. But they didn't. So, Ha Ha!

The difference between Nelson and Force of Nature is that Nelson feels more like an easy cheap win: just get privileged access to some data and swoop in to publish using it. In contrast, Force of Nature involves spending years of labor maintaining a beast of a platform or infrastructure (such as Python Tutor) before being able to reap any tangible research benefits from it.

I wouldn't recommend building your entire portfolio out of Nelson-style research, but it can be fun once in a while :)

Risks: If you don't provide any meaningful generalizable insights, this type of research can easily be dismissed as superficial.

Appendix

Here are other design patterns that didn't quite make the top cut:

  • Imperfectly automate: Can you automate something that people currently have to do manually, even if your automation isn't perfect? My IncPy and CDE projects fall into this category.
    • a related pattern here is reduce preconditions: can you reduce the amount of work that's required from a user, or the amount of assumptions about the inputs?
  • Multi-user: Can you take a single-user system and extend it for multiple users? Would doing so involve some non-trivial conceptual leaps or novel designs? My Codechella, Codepourri, and CodePilot projects fall into this category.
  • Lettuce: something so crisp, crisp as lettuce. It's hard to pin down, but many good research ideas are so so CRISP when someone tells it to you, as though it was obvious in hindsight.
  • Sequel: Proactive Wrangler was a straight-up sequel to the original Wrangler paper. Sequels sometimes make sense, but they're often overshadowed by their predecessors so I wouldn't recommend trying for them too often.

Here are some other good resources for thinking about how to come up with research ideas, especially in STEM fields:

Finally, ideas are just a starting point. Good execution is critical! Some thoughts on execution:

Keep this website up and running by making a small donation.

Created: 2019-05-14
Last modified: 2019-05-14
Related pages tagged as research: