On reviewing research papers
Today, I finished off two reviews for PLDI 2011 (http://pldi11.cs.utah.edu/), one of the biggest conferences in programming languages research. I enjoy opportunities to perform these reviews, even though they still take me somewhere between six and eight hours each. Why?
Professional debt
Each paper is reviewed by 3-5 people, depending on the conference. So, if you submit a paper, you accrue a debt of 3-5 papers (admittedly, split with your co-authors). As a young graduate student, you can’t provide concrete feedback. Even at my stage, later in my graduate career but still a year or so away from completion, I can only provide reviews in a few very narrow areas. Given the number of people who will submit, graduate, and never pay off their debt, just keeping the whole research publication game alive requires you to review quite a few per year!
Programming languages reviews are fantastic
It always impresses the other University of Chicago graduate students in areas like supercomputing and theory when I tell them about the quality of reviews in PL. Overwhelmingly, every review we’ve gotten for Manticore has been thorough, deep, and insightful. Not just one long and detailed response per submission, but nearly every individual review, from major conferences to workshops to journals to even NSF grant proposals! I enjoy attempting to meet that high bar set by my peers and find the kinds of persnickety technical details, key missing related work, and insights about wider applicability or opportunities for extension that we’ve been given on our project. Seeing my review next in its place next to 2-4 other reviews of a paper can be humbling, especially when I’ve missed a glaring error or come away with a different position on the paper than all the rest of the reviewers.
Really understand the submission
Short of implementing the meat of a paper or reproducing its proofs by hand, writing a thorough and constructive review is the best way for me to believe I really understand what the paper is presenting. What does it build on? Do I believe its claims? How did they evaluate their results and is that a meaningful measure of their claims?
