Reviewing Papers
David did such a great job in last year’s conference blog describing the process by which papers are reviewed, discussed, and ultimately accepted. The process is mostly the same from year to year, but I wanted to expand on certain aspects of the process.
Committees use a web-based system to examine and review the submissions. After the submission deadline I scanned every paper and assigned a gneral category to each. The categories vary each year depending on what types of papers are submitted. This year we had 16 categories: Applications, Autonomics, Backups, Clustering and distributed computing, Configuration and Change Management, Databases, E-commerce, High availability and redundancy, Mail systems, Networking, People management, Security, Spam, Telecom integration, Troubleshooting, and Virtualization. Each paper was tagged with one or more of these categories. Then I asked the committee members to indicate their preference for each category: will review, will not review, or don’t care. Using this information I was able to assign papers to members of the committee.
It soon became clear that I would need more people reviewing papers if I didn’t want to overwhelm the committee members. So I began recruiting readers to supplement the committee’s work. A reader reviews a paper in exactly the same way that a committee member does. He or she grades the paper and adds comments. However, readers do not participate in the program committee meeting and have no input in to the final discussions of paper acceptance. By using the readers I was able to get a wide variety of input on each paper without requiring every reviewer to read a dozen or more papers. A few ambitious reviewers managed to read and comment on every paper that was submitted. That was quite an effort!
To review a paper, a reviewer thoroughly reads it, then grades it in each of four areas:
- overall quality
- technical quality: the quality of the technology being presented,
- editorial quality: the quality of the presentation (how good is the paper as a paper),
- suitability: does it fit a LISA conference.
In addition, each reviewer rates himself on his or her own confidence in the subject matter of the paper. These scores are added and adjusted based on the reviewer’s confidence and form a weighted average. This gives the committee a quantifiable view of the paper.
In addition to the scores, each reviewer can submit comments about the paper. These comments are separated in to three categories: solely for the chair, only for the committee, for the authors. This structure allows the reviewer to be as candid as possible for the appropriate audience. Authors never see their scores, but when acceptance and rejection notices go out, the author comments are also sent. It is hoped that these comments can provide some constructive feedback that will enable the authors to improve the quality of their papers not only for this year but for future submissions as well.
This year paper submissions were due on May 23. By May 31 I had all the papers assigned to reviewers and I let them loose on the papers. I urged them all to have their reviews done by June 20. During those twenty days I reviewed as many papers as I could and I regularly reminded everyone else how close the deadline was! I’m sure most of the committee got sick of my e-mail messages. In reality I didn’t get to do the final summation of scores until late on the 21st, so that gave the reviewers a little extra time. However, I was scheduled to leave for Chicaog on the 22nd so that I had a day to visit relatives. It was a tight squeeze, but I did manage to get everything prepared in time for my trip.
The committee met face-to-face in Chicago on June 24. The scores were used to provide the order in which we would discuss papers. The comments to the committee were used as the basis for conversation, and those that had read a particular paper would offer their insights. But the final decision was based solely on the discussion, not the scores.
In the end, out of 49 submissions we accepted 24 (that’s 49% for those of you without a calculator). I believe that the resulting slate of papers is excellent this year, and I am looking forward to reading and hearing as many of these papers as I can.