June 2012 (perspective of a Ph.D. student)
Year Four: Reboot
The Ph.D. Grind
Throughout my second year of Ph.D. and my summer 2009 internship at MSR, I performed empirical software measurement research by writing computer programs in a popular language called Python to process, analyze, and visualize data sets. After writing hundreds of these sorts of Python programs, I noticed that I kept facing similar inefficiencies over and over again. In particular, I had to tediously keep track of a multitude of data files on my computer and which programs they depended upon; I also needed to make my programs unnecessarily complex so that they could execute (run) quickly after each round of incremental changes to my analyses. After letting these observations simmer in the back of my mind during the summer, the following idea suddenly came to me on that quiet July afternoon at MSR: By altering the Python run-time environment (called an interpreter) in some innovative ways, I could eliminate many of these inefficiencies, thereby improving the productivity of computational researchers who use Python. I named my proposed modification to the Python interpreter “IncPy,” which stands for Incremental Python.
Like any pragmatic researcher, the first thing I did after coming up with my IncPy idea (after calming down from the initial excitement) was to frantically search the Web for research papers describing related work. Thankfully, nobody had created exactly what I was planning to create, but there were some past research projects with a similar flavor. That was fine, though; no idea is truly original, so there will always be related projects. However, in order to eventually publish, I had to make a convincing case for how IncPy was different enough from similar projects. Within a few days, I had sketched out an initial project plan, which included arguments for why IncPy was unique, innovative, and research-worthy.
Since I still had over a month left at MSR, I took advantage of the fact that I was around lots of smart colleagues who were also performing similar kinds of data analysis programming for their research. I grabbed coffee with several colleagues and interviewed them about their data analysis work habits and inefficiencies. I then pitched them my IncPy idea and discussed possible refinements to make it both more useful and also more interesting from a research perspective. These early conversations helped boost my confidence that I was on the right track, since other people shared my frustrations with performing data analysis and my enthusiasm for the ideas that IncPy embodied.
For the rest of the summer, I spent my nights and weekends at coffee shops refining my fledgling IncPy idea, strengthening its “marketing pitch,” and getting more feedback from MSR colleagues. I emailed drafts of my idea to Dawson, but I didn't actually care how enthusiastic he was about it since this was going to be my own undertaking. I wasn't asking for his permission; I was just informing him about what I planned to do once I returned to Stanford in the fall.
I rebooted my Ph.D. career as I began my fourth year at Stanford, severing my ties to the previous three years and starting anew. No more working on already-established research projects, no more trying to scheme up ways to align with the supposed interests of professors and senior colleagues, and no more worrying about what kinds of research the academic establishment liked to see. I was now hell-bent on implementing my new IncPy idea, turning it into a publishable paper, and making it the first part of my dissertation.
Although I was filled with newfound excitement and passion, a part of me was scared because I was breaking away from the establishment to do something new and unproven without any professor support. Most Ph.D. students in my department work on projects that their advisors or other professors are interested in, since it's much easier to publish papers with the help of professors' enthusiasm and expertise. However, even without professor backing, my hunch was that IncPy could become a publishable idea; I had accumulated enough battle scars from the past three years of research failures to develop better intuitions for which ideas might succeed. Trusting this gut instinct became the turning point of my Ph.D. career.
On a pragmatic note, I still kept Dawson as my advisor since he was okay with me doing my own project and occasionally giving me high-level feedback on it. Since I was still self-funded by fellowships, I didn't have an obligation to continue working on Klee like the rest of Dawson's students did. My relationship with Dawson was much more hands-off throughout the rest of my Ph.D. years. We probably met less than a dozen times to talk about research; I was mostly working on my own or seeking out collaborations with others. I didn't want to formally switch advisors, because then I would need to “pay my dues” all over again with a new research group. Besides, I didn't know of any professors in my department who were excited by IncPy-like ideas, or else I might have contemplated switching.
To eventually graduate, I needed to form a thesis committee consisting of three professors who agree to read and approve my dissertation. Most students simply have their advisors choose committee members for them, since they are working on “official” advisor-sanctioned projects. Since I was going rogue and not working on Dawson's Klee project, I didn't have that luxury. I tried hard to find professors both in my own department and in other related departments (e.g., statistics, bioinformatics) to serve on my committee. I cold-emailed a few professors and attempted to reach others via their current students. I even made a PowerPoint slide deck to pitch my dissertation proposal to potential committee members, but unfortunately, no professor was interested in meeting with me. However, instead of giving up and crawling back to a traditional project with more institutional support, I took a risk by marching forward with IncPy and hoping that the thesis committee issue would sort itself out later.
Taking inspiration from my HCI (Human-Computer Interaction) work with Scott and Joel during my second year, as soon as I returned to Stanford in the fall of 2009, I began interviewing colleagues who wrote Python programs to analyze data for their research. My goal was to discover what their programming-related inefficiencies were, and how IncPy could eliminate those inefficiencies. I also had some friends arrange for me to give talks on IncPy—which was only a half-baked idea at the time—at their lab group meetings. My initiative in conducting interviews and making presentations at this preliminary stage was helpful both for providing fresh ideas and also for refining IncPy's “marketing pitch.” I'm immensely grateful for the friends who helped me get my project off the ground when I had nothing to show except for a few shabby PowerPoint slides.
I grew more and more encouraged as I found out that researchers in a variety of computation-based fields such as machine learning, pharmacology, bioengineering, bioinformatics, neuroscience, and ocean engineering all performed similar sorts of data analysis for their research and could benefit from a tool such as IncPy. After a few weeks of interviews and subsequent refinements to my design plans, I felt confident that I had a good enough pitch to “sell” the project convincingly in a future paper submission. The argument I wanted to make was that lots of computational researchers in diverse fields struggle with several common inefficiencies in their daily programming workflow, and IncPy provides a new kind of fully automated solution to such inefficiencies that nobody else has previously implemented. This initial pitch would later evolve into the theme of my entire dissertation.
With an overall strategy in place, I was ready to begin the thousands of hours of hard labor—grinding—required to turn IncPy from an idea into a real working prototype tool. I had spent the end of my summer playing the “professor role” of sketching out high-level designs, giving talks, and refining conceptual ideas. Now I was ready to play the “student role” of massively grinding throughout the next year to implement IncPy.
By now it should be clear that having a good idea is necessary but nowhere near sufficient for producing publishable research. Junior researchers—usually Ph.D. students—must spend anywhere from hundreds to thousands of hours “sweating the details” to bring that idea to fruition. In computer science research, the main form of labor is performing computer programming to build, test, and evaluate new software-based prototype tools and techniques. The nearly ten thousand hours I had spent doing many types of programming over the past decade—in classroom, hobby, research lab, and engineering internship settings—prepared me to endure the intensely intricate programming required to implement research ideas such as IncPy.
Implementing IncPy involved some of the grimiest programming grind that I had ever done. If I didn't have all of those hours of trial-by-fire training over the past decade, then I would have never even attempted such a labor-intensive project. Other people have undoubtedly recognized the same inefficiencies that I observed in computational research workflows, but what set me apart from would-be competitors was that these people didn't have the deep programming expertise required to create a fully automated solution such as IncPy. At best, they might create semi-automated solutions that would not be substantive enough to publish in a respectable conference.
Even though I wasn't in love with my previous research projects earlier in grad school and during college, the technical skills and judgment that I gained from those experiences made it possible for me to now implement my own ideas that I truly cared about. Over the next three years (2009 to 2011), I grinded non-stop on creating five new prototype tools to help computational researchers (IncPy was the first), published one or more first-author papers describing each tool, and then combined all of that work together into a Ph.D. dissertation that I was extremely proud of. Those three years—my latter half of grad school—were the most creative and productive of my life thus far, in stark contrast to my meandering first half of grad school.
I became fiercely self-driven by an enormous amount of productive rage. I turned steely, determined, and ultra-focused. Every time I reflected back on the inefficiencies, failures, and frustrations that I had endured during my first three years of grad school, I would grow more enraged and push myself to grind even harder; I was motivated by an obsessive urge to make up for supposedly lost time. Of course, those early years weren't actually lost; without those struggles, I wouldn't have gained the inspiration or abilities to create the five projects that comprised my dissertation.
However, as I was about to begin my fourth year of Ph.D. in September 2009, I had no sense of what awaited me in the future and definitely no grand plans for a five-project, 230-page dissertation. I didn't even know whether any professors would take my unproven ideas seriously enough to agree to serve on my thesis committee. All I wanted to do was implement IncPy, try to get it published, and then figure out my next move when the time came.
I was about to start implementing (programming) IncPy when an unexpected bit of fortune struck. At the time, I had no idea that this one minor event would lead to a cascade of other good luck that would pave the way for my graduation. One day at lunch, my friend Robert told me he was planning to submit a paper about his new research project to a workshop whose deadline was in 2.5 months.
Normally, I wouldn't have thought twice about such an announcement for two reasons: First, Robert's research topic (in a subfield called data provenance) was unrelated to IncPy, so where he planned to submit papers had no relevance to me. Second, a published workshop paper usually does not “count” as a dissertation contribution in our department. Workshop papers are meant to disseminate early ideas and are not scrutinized as rigorously as conference papers. Whereas a conference paper submission has an 8 to 30 percent chance of acceptance, a workshop paper submission has a 60 to 80 percent chance. Most professors in our department don't encourage students to submit to workshops, since if the paper gets accepted (which is likely), the professor must pay around $1,500 of travel, hotel, and registration costs using their grant money for the student to attend and give a talk at the workshop. It costs about as much to send a student to a workshop as to a conference, and conference papers are much more prestigious both for the student and the professor. Thus, top-tier computer science professors strongly encourage students to publish more selective conference papers and eschew workshops altogether.
I asked Robert why his advisor encouraged him to submit his early-stage idea to a workshop rather than developing it further and submitting to a conference at a later date. He said that it was partly out of convenience, since the workshop was located in nearby San Jose. His entire research group planned to attend since it was only 20 miles from Stanford, so he also wanted to present a paper there.
Out of curiosity, I looked at the workshop's website to see what topics were of interest to the organizers. Although it was a workshop on data provenance (Robert's research area), the topics list included a bullet point entitled “efficient/incremental recomputation.” I thought to myself: Hmmm, IncPy provides efficient incremental recomputation for Python programs, so if I pitch my paper properly, then maybe it's sort of appropriate for this workshop! Since I didn't need to travel to attend the workshop, I wasn't afraid to ask Dawson to pay for my registration fee if my paper was accepted. I emailed Dawson my plans to submit a paper to this workshop, and he wrote back with a lukewarm response. As expected, he wasn't enthusiastic about workshops in general, but he was okay with me submitting since he respected the workshop's program committee co-chair, a Harvard professor named Margo (who would later play a pivotal role in helping me to graduate).
The timing was perfect: I now had 2.5 months to super-grind on implementing a first working prototype of IncPy and then write a workshop paper submission. Since I knew that the bar for workshop paper acceptance was a lot lower than for a conference paper, I wasn't too stressed. I just needed to get a basic prototype working reasonably well and anecdotally discuss my experiences. I discovered that this strategy of finding and setting short-term deadlines for myself would work wonders in keeping me focused throughout the rest of my Ph.D. years. Without self-imposed deadlines, it becomes easy to fall into a rut and succumb to chronic procrastination.
My paper was accepted with great reviews, and I attended the workshop in February 2010 to give a 30-minute talk on it. In particular, I was honored that Margo, the program committee co-chair, personally praised my paper and mentioned how it was related to a new Python-based project that her student Elaine was starting. Since Elaine wasn't at the workshop, Margo gave me Elaine's email address and suggested for us to get in touch.
At first, I was afraid that Elaine would be my direct competition and publish a conference paper before I did, which would make it harder for me to publish a conference paper on IncPy: Since being first is highly valued in academia, once another researcher publishes a similar idea and “scoops” you, then it becomes much harder to get your own idea published. But after exchanging some reconnaissance emails and video chatting with her, I was relieved to find out that she didn't have plans to gear up for a conference paper submission. Also, her tool lacked the fully automatic capabilities that set IncPy apart from related work. Once we realized that we weren't rivals, we quickly became friends. I'm glad that Elaine and I kept in touch over the next few years, since she would be partially responsible for reuniting me with Margo towards the end of my Ph.D. journey.
Although giving a talk on my IncPy workshop paper was great for getting feedback and especially for meeting Margo, that paper didn't count as a “real” publication for my dissertation. I knew that I still needed to publish this work in a conference that professors in my department recognized as legitimate. The biggest difference between a workshop and a conference paper is that a conference paper must have a convincing experimental evaluation that shows the effectiveness of the tool or technique being described in the paper. A paper's evaluation section can come in many flavors ranging from measurements of run-time performance to a controlled laboratory study of user behavior. Since many researchers come up with similar ideas, reviewers carefully scrutinize how those ideas are implemented and experimentally evaluated when deciding which papers to accept or reject.
Even at the start of my IncPy project, I knew that it would be difficult to present a convincing evaluation because the argument I wanted to make—that IncPy can improve the productivity of computational researchers—was a subjective and fuzzy notion. After reading several other papers that presented similar productivity improvement claims, I devised a two-part evaluation strategy:
The next relevant top-tier conference submission deadline was in seven months, so I aimed to have a paper ready by then. I spent lots of time trying to find case study subject programs and potential users for deployment. Without those, there would be no evaluation, no conference publication, no dissertation, and no graduation. (Of course, I still spent the majority of my waking hours doing grimy programming, debugging, and testing to implement IncPy.)
I learned to be part-salesman and part-beggar, persistently asking colleagues whether they had Python programs I could use for case studies or, even better, whether they would be willing to install and use IncPy on a daily basis for their research and report their impressions to me. As expected, I mostly got “No” replies, but I politely asked for recommendations of other people I could contact. I also invited myself to give several more talks at various lab group meetings to drum up interest in IncPy. After a few months of “begging,” I obtained Python programs from half a dozen researchers in a diverse variety of fields, which was enough to begin my case studies. I appreciated everyone who helped me out during that time, since they had nothing to gain besides the goodwill of an unknown Ph.D. student.
Although case studies might have been sufficient for the evaluation section of my paper submission, what I truly craved was deployment—getting real researchers to use IncPy. Not only did I feel that deployment would make for a stronger evaluation, but more importantly, I genuinely believed that IncPy could improve the day-to-day productivity of computational researchers. The majority of research prototype tools are built to demonstrate a novel idea and then discarded, but I wanted IncPy to be practical and enduring. I didn't just dream up IncPy in a vacuum for the sake of publishing papers; it was inspired by real-world problems I faced when performing programming for my research, so I wanted real-world people to actually use it.
In spite of my idealism, I understood why almost no research prototype tools get successfully deployed. The underlying reason is that people don't want to try a new tool unless they can instantly see major benefits without any drawbacks; researchers simply don't have the time or resources to get their prototypes working well enough to meet these stringent requirements. In particular, my target users are happy enough using regular Python—despite its inefficiencies—that they don't want to take a risk by switching to IncPy. To convince someone to try IncPy, I would need to guarantee that it works better than regular Python in all possible scenarios. While that's true in theory, in reality IncPy is a research prototype tool being maintained by one student, so it's bound to have numerous bugs. In contrast, the official Python project is over twenty years old and being maintained by hundreds of veteran programmers, so it's much more stable and reliable. As soon as someone hits a single IncPy bug, they will immediately dismiss it as lame and switch back to using regular Python. I knew the odds were against me, but I didn't care.
After failing to find anybody at Stanford who was willing to install and use IncPy, I looked for leads at nearby universities. In March 2010, I cold-emailed a few Ph.D. students at UC Davis (a two-hour drive from Stanford) who were friends with one of my former MSR intern colleagues. I appealed to their sense of altruism at helping out a fellow grad student in need and managed to invite myself over for a visit to advertise IncPy. A few gracious professors even agreed to meet with me, including one I had met at the data provenance workshop. Although I received more helpful feedback, I didn't manage to find any subjects for case studies or deployment.
I spent the night at UC Davis and planned to return to Stanford the next morning. Then I thought of an impulsive idea—cold-emailing Fernando, a research scientist at UC Berkeley who was passionate about the use of Python in computational research. A few months earlier, a fellow grad student who attended one of my IncPy talks emailed me a link to one of Fernando's blog posts, and I jotted down a reminder to contact Fernando sometime in the future. Now seemed like a good time: Since UC Berkeley was located between UC Davis and Stanford, I could potentially drop by his office on the drive back home. I cold-emailed Fernando and asked if he had time to chat the next morning. It was a long shot, but he agreed to meet with me. I had a wonderful initial one-hour meeting with Fernando; it felt great to see a well-established senior scientist support my IncPy idea.
The most significant outcome of our first meeting was that Fernando invited me to return to UC Berkeley the following month to give a talk on IncPy. He told me that my audience would consist of neuroscientists who were using Python to run their computational research experiments. When I gave my one-hour talk, I was a bit taken aback by three researchers continually interrupting and pestering me with detailed questions about how well IncPy worked in practice. At first, I thought those guys were being overly pedantic, but afterwards they came up to me and expressed a strong interest in trying IncPy. They lamented about how they're currently suffering from the exact kinds of inefficiencies that inspired me to create IncPy in the first place! It turned out that they weren't trying to be annoying during my talk; they just wanted to understand the details to assess whether it was feasible to deploy IncPy on their research lab computers.
I was thrilled by the possibility of getting my first set of real users. I exchanged some emails with them and offered to drive up to UC Berkeley again to help them install and set up IncPy. My first email started with a cautiously optimistic tone:
Thanks for expressing an interest in becoming the first real user of my IncPy memoizing Python interpreter! I'm really enthusiastic about getting IncPy to a state where you can use it in your daily workflow. I think the main issues will be mostly grunge work with installation / setup / configuration, which I am more than happy to deal with in order to give you the smoothest possible user experience.
By the time I tried installing IncPy on the Berkeley neuroscientists' computers, I had been building and testing it for seven months, so I felt reasonably confident that it would work for them. However, shortly after installation, we discovered that IncPy was not compatible with dozens of third-party Python add-ons (called extension modules) that these scientists were using in their daily work. In my own private testing, I tested only the basic use cases without any extension modules. I was hit with a lesson in the harshness of real-world deployment: failures come in unexpected forms, and once the user gets a bad first impression, then it's over! Like most researchers creating prototypes in an ivory tower, I could have never predicted that this unforeseen banal extension module issue would completely derail my first deployment attempt.
I wasn't about to give up, though. I spent the next few weeks redesigning and reimplementing a critical portion of the IncPy code so that it now worked perfectly with any arbitrary Python extension modules. I emailed the UC Berkeley neuroscientists again to ask for a second chance, but I got no response. I had one shot, and I blew it.
This disappointment spurred me to keep improving IncPy in practical ways: Along with fixing many subtle bugs, I created an IncPy project website containing a short video demo, documentation, and beginner tutorials. None of these hundreds of hours of effort made any improvements to my original research contribution, but they were necessary if I wanted to get real user anecdotes for an evaluation section of a future paper submission.
After a few months passed, three strangers from different parts of the world found IncPy via its website, downloaded a copy, and used it to speed up some of their research programming activities. Although three is a pathetically small number of users, at least it's more than zero, which is the number of users most research prototypes have. I felt satisfied that IncPy was in a polished enough state—thanks to my post-Berkeley improvements—that strangers were now able to use it without my guidance. These three people emailed me anecdotes about how they found IncPy to be mildly useful for some parts of their work. Their anecdotes weren't strong evidence of effectiveness, but they were better than nothing.
As my fourth year of Ph.D. came to an end in September 2010, I submitted my IncPy paper to a top-tier conference with an evaluation consisting of case studies and three brief deployment anecdotes. Only 14 percent of papers submitted to that same conference last year were accepted. Thus, I knew that my paper would most likely be rejected, both due to those meager odds and especially since IncPy didn't neatly fit into any traditional academic subfield. Nonetheless, it was still wise to first aim for the top tier and then resubmit to a second-tier conference if necessary, since a top-tier publication would carry more weight in my future graduation case.
I still didn't have a clearly paved path to graduation, but at least I was able to start taking charge of my own research agenda rather than tagging along on other people's projects. I was happy that within the past year, I had turned IncPy from the spark of an idea in my mind into a semi-practical tool that benefited three strangers who downloaded it from the Internet. Even though this minor accomplishment wouldn't help my graduation case at all—professors respect real-world deployment far less than theoretical novelty—it was a somewhat satisfying end to my fourth year.
Copyright © 2012 Philip Guo
Keep this website up and running by making a small donation.