Picking Papers, Part 4
(be sure to read parts one, two and three so you know where we are in the story)
I wish I could say that we held the program committee meeting in some marble-adorned Hall of Justice edifice but it was, in truth, a standard hotel conference room in San Francisco. There I and most of the program committee members got down to the serious business of deciding which papers were to make the cut.
The first step of this process was relatively easy. The initial accept/cut round started off by looking at a graph I had brought to the meeting. A couple of days before we met I wrote a small Perl script to scrape the final paper scores from the webreview system and plot them via GD:Graph on a graph that looked very similar to this:
[note: I don’t have a problem showing you the scores like this because it would be impossible for an author or anyone else to guess a paper’s score given the rest of the process we’re about to see.]
From this graph we were able to determine a high cutoff score and a low cutoff score. Papers at the high cutoff or better were moved into the initial accept pile; papers at the low cutoff or below were moved into the initial reject pile. The program committee members were then asked to make sure there weren’t papers rejected by score that deserved consideration and papers provisionally accepted that shouldn’t be there for any reason.
With the first chunk of accepts and rejects done, we then moved on to considering the contenders in the middle. This typically takes the most time because the decisions tend to get harder and harder as you proceed (with a commensurate increase in the amount of discussion time per paper). This year’s meeting was no exception, most of the day was spent working on this set of submissions.
If you’ve never been on a conference program committee before, you may not realize just how carefully and seriously each submission is considered in this process. Example weighty questions that get asked and answered include “How similar is this paper to ones we’ve already accepted? Does this paper have lasting value to the field? Is this paper too specific? Not specific enough? Will it spark a useful discussion in the community? Can it be the start of an interesting line of investigation by future papers?”
We worked our way down the list from the highest scoring papers to the lowest until all but about a fifth of the remaining slots had been filled. At that point the process changed and the program committee members were asked to go through the remaining pool, and if they chose, through any of the previous decisions made, to mark any papers they deemed worthy of more discussion irrespective of score. We then worked out way through this final pool and filled the remaining spots in the program.
It’s a hard job that comes with considerable responsibility, but I think this year’s program committee (with the help of our external reviewers) did a superb job with the selection of papers. I’m very thankful I could be part of this process.
And so that, my friends, is how papers were picked at LISA 2005.