The LISA ‘06 Conference Blog

It’s always risky to pick out a few papers and say “I really like these,” because such an act tends to alienate and discourage authors of those that did not get chosen. So I have to say up front that I like all the papers we chose for this year’s conference. In fact, if I didn’t like them then they would not be in the conference.

But as we read through the submissions there were a few papers that really stood out. These are papers that really make me want to go see the presentations.

A collection of scientists from the Netherlands will be speaking on a hardware platform for the security and privacy administration of RFID tags. RFID is quickly permeating the marketplace. These tiny things are showing up everywhere, including US Passports. They’re ubiquitous and generally promiscuous, which in my mind is not a good combination. Some very clever people have come up with a way to provide security and privacy for RFID tags. The idea is timely, the execution is great, and the presentation is fantastic. The only concern we had with this paper was the appropriateness for LISA. But in this case the importance of the subject and the excellent research trumped concerns about the subject being a stretch for the audience. Their presentation will be in the Security session, conference Wednesday at 4 pm.

Dan Klein was looking through his MRTG graphs one day and discovered that his system was being hijacked by spammers. He immediately went in to an investigative mode and began collecting as much data as he could. The result is an amazingly thorough forensic analysis of this particular kind of spammer attack. His presentation is in the Electronic Mail session on conference Wednesday at 11 am.
People who don’t pay much attention to networking may not even know what a network flow is, but anyone who has ever tried to troubleshoot a network problem will certainly understand the concept. Rather than looking at a file full of packets sorted by time, one can collect up and examine packets by connection, or flow. Using flows can greatly simplify lots of tedious tasks in network adinistration and troubleshooting. The problem is, every manufacturer has a different way of storing and examining flows. Well someone from CERT (Brian Trammel) and from CA Labs (Carrie Gates) have come up with a suite of tools that lets you examine and manipulate flows. The result is great! The NetSA Aggregated Flow Suite does for network flows what ImageMagick does for digital images. Their presentation is in the Visualization session on conference Thursday at 4 pm.

Large installations have large problems, especially when part of the infrastructure fails and other parts have to be changed to take up the slack. Most installations use either manual or ad hoc mechanisms to detect these changes and take corrective actions. The state of the art in this area is policy-based management systems, but even these don’t always scale well to large installations. Interdependencies of components can cause a flood of changes caused b a series of reactions to a single stimulus. Some scientists from UIUC and from HP think that part of the problem is the way in which policies are specified. Their paper does an excellent job of presenting the current widely accepted mechanism for policy specification, called Event-Condition-Action (ECA), and proposing an enhancement to it which will scale better in larger installations. This is an excellent presentation of sound theoretical work with immediate practical application. Their presentation is in the Theory session on conference Thursday at 9 am.

There are many more excellent papers that you may find interesting. I wish I had time to write about each and every one. Look through the conference schedule and see what papers sound promising. When you arrive at the conference, get your copy of the Proceedings right away, then read through those paper that interest you. If you like what you read, go to the paper’s presentation, hear what the author has to say, and you will even have a chance to ask questions, either as part of the audience or one-on-one when the session is over.

I hope you enjoy this year’s papers as much as I do.

David did such a great job in last year’s conference blog describing the process by which papers are reviewed, discussed, and ultimately accepted. The process is mostly the same from year to year, but I wanted to expand on certain aspects of the process.

Committees use a web-based system to examine and review the submissions. After the submission deadline I scanned every paper and assigned a gneral category to each. The categories vary each year depending on what types of papers are submitted. This year we had 16 categories: Applications, Autonomics, Backups, Clustering and distributed computing, Configuration and Change Management, Databases, E-commerce, High availability and redundancy, Mail systems, Networking, People management, Security, Spam, Telecom integration, Troubleshooting, and Virtualization. Each paper was tagged with one or more of these categories. Then I asked the committee members to indicate their preference for each category: will review, will not review, or don’t care. Using this information I was able to assign papers to members of the committee.

It soon became clear that I would need more people reviewing papers if I didn’t want to overwhelm the committee members. So I began recruiting readers to supplement the committee’s work. A reader reviews a paper in exactly the same way that a committee member does. He or she grades the paper and adds comments. However, readers do not participate in the program committee meeting and have no input in to the final discussions of paper acceptance. By using the readers I was able to get a wide variety of input on each paper without requiring every reviewer to read a dozen or more papers. A few ambitious reviewers managed to read and comment on every paper that was submitted. That was quite an effort!

To review a paper, a reviewer thoroughly reads it, then grades it in each of four areas:

• overall quality
• technical quality: the quality of the technology being presented,
• editorial quality: the quality of the presentation (how good is the paper as a paper),
• suitability: does it fit a LISA conference.

In addition, each reviewer rates himself on his or her own confidence in the subject matter of the paper. These scores are added and adjusted based on the reviewer’s confidence and form a weighted average. This gives the committee a quantifiable view of the paper.

In addition to the scores, each reviewer can submit comments about the paper. These comments are separated in to three categories: solely for the chair, only for the committee, for the authors. This structure allows the reviewer to be as candid as possible for the appropriate audience. Authors never see their scores, but when acceptance and rejection notices go out, the author comments are also sent. It is hoped that these comments can provide some constructive feedback that will enable the authors to improve the quality of their papers not only for this year but for future submissions as well.

This year paper submissions were due on May 23. By May 31 I had all the papers assigned to reviewers and I let them loose on the papers. I urged them all to have their reviews done by June 20. During those twenty days I reviewed as many papers as I could and I regularly reminded everyone else how close the deadline was! I’m sure most of the committee got sick of my e-mail messages. In reality I didn’t get to do the final summation of scores until late on the 21st, so that gave the reviewers a little extra time. However, I was scheduled to leave for Chicaog on the 22nd so that I had a day to visit relatives. It was a tight squeeze, but I did manage to get everything prepared in time for my trip.

The committee met face-to-face in Chicago on June 24. The scores were used to provide the order in which we would discuss papers. The comments to the committee were used as the basis for conversation, and those that had read a particular paper would offer their insights. But the final decision was based solely on the discussion, not the scores.

In the end, out of 49 submissions we accepted 24 (that’s 49% for those of you without a calculator). I believe that the resulting slate of papers is excellent this year, and I am looking forward to reading and hearing as many of these papers as I can.

Every year approximately 30 papers are accepted at a LISA conference for presentation and publication in the proceedings. These papers present the latest research efforts in all corners of our industry. Authors come from all types of organizations and backgrounds. We have many universities and research groups present their work, as well as corporations and independent consultants. This is where you will find the ideas and technologies that will become commonplace in the years ahead.

So how do we choose these papers? You might think that a fortune teller and ouija board are involved, but things aren’t quite that mystic. At the beginning of the year I selected twelve experts in system administration to create the core of my program committee. In February we sent out a call for papers and by the May deadline we had received 49 submissions.

Each submission needed to be thoroughly read and reviewed. So as soon as the deadline had passed I began to scan and categorize every submission. These categories are very important to the process. I know that I have one fantastic committee with some of the most knowledgeable people in the industry. But not everyone knows everything, and it is important to match papers with those who are qualified to review them. The categories make this matching possible. I made sure that every paper was reviewed by at least four people, while at the same time limiting each individual to no more than 10 papers. Reviewing papers takes time, and I didn’t want to drive any of my committee members to the brink of insanity by giving them too much to read. I also added a few readers to the list of reviewers. These are folks who are not on the committee but help us out by doing some of the reviews.

Last year the program committee, David Blank-Edelman, used some ultra-powerful fancy perl script to perform this matchup. But I had some reviewers who were a little slow choosing their categories. That and a number of other issues made it simpler for me ot just assign them by hand. But it was a bit like a jigsaw puzzle making everything fit.

Once reviewers are assigned they typically have about 4 weeks to read the papers, type up comments, and assign scores. Each paper is scored in a number of areas, including technical and editorial quality, as well as suitability for the conference. The reviewers’ scores are averaged and the papers are sorted by these scores.

On a cold Saturday in June, the program committee was locked in an undisclosed and top secret bunker in the side of a mountain. Well, okay, the Saturday was warm, we weren’t locked in, the bunker was really a conference room at a hotel, and it was in the flatlands of suburban Chicago. But the other way sounded better. We spent the day discussing each paper individually, even the ones that ranked the lowest. The scores from the review process were used as a guideline, but were not the final deciding factor. A decision was made on each paper based on the score, reviewers comments, and the discussion of the entire committee. We were able to reach a concensus on almost every paper. There were a few papers where individual committe members had strong and opposing views. Hard to imagine such a thing with this group, isn’t it? So in cases where we could not reach concensus I just overrode everyone and made my own decision. In the end we accepted 24 good quality papers. Once the meeting was over we all went out for some stiff drinks.