The LISA ‘05 Conference Blog

Continuing where we left off after parts one and two, the reviewers are just to about to get to work. They log in to the web review system and see the list of papers assigned to them. They also see a much longer list of papers they could review should they have time and energy after the assigned papers are completed.

The actual reviews entered for each submission consists of two parts: scores and comments. Let’s look at each separately:

Reviewers are asked to assign several scores to each submission:

• Overall marks
• Confidence Level: how confident is the reviewer in her or his ability to rate that paper.
• Technical Quality: The quality of the technology being documented in the paper.
• Editorial Quality: how good is the paper as a paper?
• Suitability: does this paper fit the LISA conference?

These scores are only used by the program committee as part of the selection process. Authors never actually see their scores.

What authors do see, however, are the comments each reviewer submits for a paper. Reviewers can submit three kinds of comments:

• comments to the author(s)
• comments to the program committee
• comments to the program chair (me)

This lets each reviewer be as candid as possible. They can leave good process comments like “recommend only accepting this paper if it gets heavy editing help” or “important topic but the paper is a bit weak. Only accept if there is not another paper on this topic.”

As program chair, my primary job while all of this reviewing is going on is to make sure things are going smoothly–namely, all papers receiving the proper amount of attention (in the form of completed reviews). The webreview system can show the reviews received for a each paper but it doesn’t have a good heads-up display. As you probably guessed, I wound up writing yet another Perl script to scrape the info from the webreview system and generated a simple HTML status table. The table showed green, yellow and red cells to show which paper had and had not received adequate coverage. This script was run every 15 minutes from cron to generate a status page I could check at will. I let the reviewers see this page as well so they would know where to focus their attention next after completing their assigned submissions.

After several weeks of hard work by the reviewers, it was time to close down the review process and begin to get ready for the meeting where the final decisions would be made. That meeting will be the subject of our next entry on this topic…

This entry was posted on Monday, July 25th, 2005 at 4:34 pm and is filed under Papers. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.