How Papers Get Accepted
September 6th, 2006Every year approximately 30 papers are accepted at a LISA conference for presentation and publication in the proceedings. These papers present the latest research efforts in all corners of our industry. Authors come from all types of organizations and backgrounds. We have many universities and research groups present their work, as well as corporations and independent consultants. This is where you will find the ideas and technologies that will become commonplace in the years ahead.
So how do we choose these papers? You might think that a fortune teller and ouija board are involved, but things aren’t quite that mystic. At the beginning of the year I selected twelve experts in system administration to create the core of my program committee. In February we sent out a call for papers and by the May deadline we had received 49 submissions.
Each submission needed to be thoroughly read and reviewed. So as soon as the deadline had passed I began to scan and categorize every submission. These categories are very important to the process. I know that I have one fantastic committee with some of the most knowledgeable people in the industry. But not everyone knows everything, and it is important to match papers with those who are qualified to review them. The categories make this matching possible. I made sure that every paper was reviewed by at least four people, while at the same time limiting each individual to no more than 10 papers. Reviewing papers takes time, and I didn’t want to drive any of my committee members to the brink of insanity by giving them too much to read. I also added a few readers to the list of reviewers. These are folks who are not on the committee but help us out by doing some of the reviews.
Last year the program committee, David Blank-Edelman, used some ultra-powerful fancy perl script to perform this matchup. But I had some reviewers who were a little slow choosing their categories. That and a number of other issues made it simpler for me ot just assign them by hand. But it was a bit like a jigsaw puzzle making everything fit.
Once reviewers are assigned they typically have about 4 weeks to read the papers, type up comments, and assign scores. Each paper is scored in a number of areas, including technical and editorial quality, as well as suitability for the conference. The reviewers’ scores are averaged and the papers are sorted by these scores.
On a cold Saturday in June, the program committee was locked in an undisclosed and top secret bunker in the side of a mountain. Well, okay, the Saturday was warm, we weren’t locked in, the bunker was really a conference room at a hotel, and it was in the flatlands of suburban Chicago. But the other way sounded better. We spent the day discussing each paper individually, even the ones that ranked the lowest. The scores from the review process were used as a guideline, but were not the final deciding factor. A decision was made on each paper based on the score, reviewers comments, and the discussion of the entire committee. We were able to reach a concensus on almost every paper. There were a few papers where individual committe members had strong and opposing views. Hard to imagine such a thing with this group, isn’t it? So in cases where we could not reach concensus I just overrode everyone and made my own decision. In the end we accepted 24 good quality papers. Once the meeting was over we all went out for some stiff drinks.