The LISA ‘06 Conference Blog

At this year’s conference you will have a very unique opportunity to hear from one of the leading researchers in the field talk about visions of the future. Autonomic computing, outsourcing, and service oriented architectures are all poised to radically change the way we do things. Are they really going to have a big impact on our profession? Are they going to make all of your knowledge and experience obsolete?

This year at LISA 06 we are trying something a little different. We will be presenting a lunch-time talk, and we will be inviting attendees to go grab lunch and bring it back to this presentation. It is a unique format for a unique talk. Dr. Alva Couch will be presenting his thoughts on the future of system administration, and how you can best prepare yourselves professionally to take the most advantage of these emerging technologies and business models. It may be controversial, and it will certainly be enlightening. So I hope you will join him at this year’s special lunch and learn on conference Friday from 1 to 1:45. The morning sessions will end at 12:30, and that leaves you just enough time to race out to a fast food place, grab something on the go, and bring it back to the lunch and learn.

Truth be told, this isn’t really the first “lunch and learn” that we’ve done at LISA. Last year (LISA 05) one of our speakers had an unexpected conflict and was unable to make his presentation on the date and time we had scheduled. We held his talk during the lunch hour on the next day. So although this year’s lunch and learn isn’t the first, it is the first time we have done one intentionally.

Invited talks are quite a bit different than refereed papers, and the process used to accept them is also quite a bit different. Two members of the program committee are specifically designated at the IT co-ordinators. This year we are very fortunate to have two excellent folks as co-ordinators. Last year’s conference chair, David Blank-Edelman along with Doug Hughes headed up the effort this year to select two tracks of invited talks.

The original intent of the invited talks track was that the committee would invite prominent individuals to speak at the conference. That’s why they’re called “invited”. But as the conference expanded it was decided that we would be served well by actually soliciting proposals from those who were interested in speaking. We still invite speakers, but we also accept proposals. In order to keep everything fair, we ask those we invite to submit a proposal. This allows us to review all the possible talks in the same manner.

The proposal is not that complicated. We ask for a speaker’s biography, a description of the talk, and an outline of what will be presented. With these proposals the committee needs to choose enough talks to fill two full tracks of 10 slots each, for a total of 20 presentations. Sometimes this number will be a little less. If the committee finds a particularly strong or important presentation that we think everyone needs to year, we will create a plenary session on Thursday.

When choosing talks, we look at several characteristics. We want the subject matter to be relevant. A talk that explains the proof of Fermat’s Last Theorem would be interesting, but hardly relevant. We also want variety among the presentations. Although Spam prevention is important, we don’t want four separate presentations about it. We want to ensure that the presentation is thorough and backed by sound facts and research. Finally, we look at the quality of the speaker.

You can see how much this differs from selecting papers. A paper is selected solely on its own merit, regardless of the authors or their abilities as speakers. The focus is on the research presented in the paper: the presentation is secondary. But an invited talk lasts nearly 90 minutes, and the presentation is the final product. So it is appropriate to consider the presenter’s abilities as a public speaker. But bear in mind that it isn’t the only criteria. Committees are willing to consider average or below average speakers if their topic is particularly important or relevant. And the converse is also true: we would consider a weak topic if it was presented by a strong speaker.

After the submission deadline had passed and we were sure all the lingering proposals had been submitted, the two co-ordinators got together on a conference call and discussed the proposals, generally classifying each as “definitely”, “maybe”, and “no way”. This provides an idea of how many “maybe” talks need to be accepted to fill out the slate. These choices are made based on the criteria listed above: what talks look good and relevant, what provides the best variety, and which speakers appear to be strong. The end result is a list of those talks we want to accept and a short list of alternates. This list was presented to me for final approval. There were a few choices that the co-ordinators were not able to reach a concensus on, so I ended up making the final decision. This year I feel very fortunate that we were able to get a strong slate of talks, due to the tireless efforts of our fantastic IT co-ordinators.

Once the choices were finalized, we contacted the speakers to make sure that they were still interested in presenting. If anyone had backed out, we had alternates to fall back on. This year, it wasn’t necessary to use any of our alternates.

One interesting problem we faced this year was a presentation proposed by one of the most prominent researchers in our community on a topic that the committee felt was particularly important and relevant. However, the speaker told us that he didn’t think he could fill more than 45 minutes. That makes for a very short presentation unless we could find something to pair it with. Unfortunately, we couldn’t, and we ended up in a difficult position. We really wanted to accept this talk, but there was no place on the program to put it. We don’t really have a position for a 45 minute presentation unless we invent one. So that is eventually what we did: we turned it in to the Friday lunchtime presentation, where Dr. Alva Couch will be telling all of us how autonomics will change our industry. I will post more about that at a later date.

Dan Klein has very generously taken the time to create some ICS format calendars for the conference schedule. I put together the raw information and he magiccally turned it in to a set of ICS files. These should be usable with iCal, evolution, and any number of other on-line calendar systems. As of now there is no location information in the entries. As soon as I get a list of rooms from the conference staff we will update the calendars with location information for each presentation.

To subscribe, go to this page: http://ical.usenix.org

After a day of arguing over papers at the program committee meeting in Chicago, everyone was ready for some nourishment. Although a few people had to leave that evening, most of us were able to stay and went out for a very enjoyable evening at a restaurant in Schaumburg. The place we went to is a Brazilian churrascaria called Sal y Carvão, which I think is Portugese for “stuff yourself silly with meat”. It’s like an all-you-can-eat-buffet, except they bring the meat to you. The dinner was absolutely wonderful and the company was lots of fun.

After everyone was completely sated with meat, salad, meat, dessert, more meat, and drink (and some more meat) it was time to return to the hotel. We had enough cars that we only needed to call one cab, but we did have to request a van. The van showed up and one individual in the group, who shall remain nameless, decided to open the hatchback to put a bag in the back. The cabbie came around the back to help out just as our illustrious committee member started to slam the overhead hatch closed, and the door came down very hard on the poor cabbie’s shoulder. He was clearly in pain and we feared his reaction, but he got back in the driver’s seat and was ready to go. Perhaps he sought revenge, or perhaps he was simply dazed with pain, but the trip back to the hotel was one I really wish I had not experienced. He drove slow. He refused to take the tollway. For awhile I wasn’t really sure he was taking us in the right direction. But the topper was the red light that he completely ignored. This wasn’t like stretching a yellow. This light had been red for awhile when our cabbie just rolled on through it as if it didn’t exist. A car came out of the side street at the same time and came very close to hitting us. We all started looking at each other with the same thought: “we are all going to die!”

The good news is that we did make it back to the hotel safely, and we were never sued by the cab driver. But it did make for an exciting conclusion to a very busy day.

David did such a great job in last year’s conference blog describing the process by which papers are reviewed, discussed, and ultimately accepted. The process is mostly the same from year to year, but I wanted to expand on certain aspects of the process.

Committees use a web-based system to examine and review the submissions. After the submission deadline I scanned every paper and assigned a gneral category to each. The categories vary each year depending on what types of papers are submitted. This year we had 16 categories: Applications, Autonomics, Backups, Clustering and distributed computing, Configuration and Change Management, Databases, E-commerce, High availability and redundancy, Mail systems, Networking, People management, Security, Spam, Telecom integration, Troubleshooting, and Virtualization. Each paper was tagged with one or more of these categories. Then I asked the committee members to indicate their preference for each category: will review, will not review, or don’t care. Using this information I was able to assign papers to members of the committee.

It soon became clear that I would need more people reviewing papers if I didn’t want to overwhelm the committee members. So I began recruiting readers to supplement the committee’s work. A reader reviews a paper in exactly the same way that a committee member does. He or she grades the paper and adds comments. However, readers do not participate in the program committee meeting and have no input in to the final discussions of paper acceptance. By using the readers I was able to get a wide variety of input on each paper without requiring every reviewer to read a dozen or more papers. A few ambitious reviewers managed to read and comment on every paper that was submitted. That was quite an effort!

To review a paper, a reviewer thoroughly reads it, then grades it in each of four areas:

• overall quality
• technical quality: the quality of the technology being presented,
• editorial quality: the quality of the presentation (how good is the paper as a paper),
• suitability: does it fit a LISA conference.

In addition, each reviewer rates himself on his or her own confidence in the subject matter of the paper. These scores are added and adjusted based on the reviewer’s confidence and form a weighted average. This gives the committee a quantifiable view of the paper.

In addition to the scores, each reviewer can submit comments about the paper. These comments are separated in to three categories: solely for the chair, only for the committee, for the authors. This structure allows the reviewer to be as candid as possible for the appropriate audience. Authors never see their scores, but when acceptance and rejection notices go out, the author comments are also sent. It is hoped that these comments can provide some constructive feedback that will enable the authors to improve the quality of their papers not only for this year but for future submissions as well.

This year paper submissions were due on May 23. By May 31 I had all the papers assigned to reviewers and I let them loose on the papers. I urged them all to have their reviews done by June 20. During those twenty days I reviewed as many papers as I could and I regularly reminded everyone else how close the deadline was! I’m sure most of the committee got sick of my e-mail messages. In reality I didn’t get to do the final summation of scores until late on the 21st, so that gave the reviewers a little extra time. However, I was scheduled to leave for Chicaog on the 22nd so that I had a day to visit relatives. It was a tight squeeze, but I did manage to get everything prepared in time for my trip.

The committee met face-to-face in Chicago on June 24. The scores were used to provide the order in which we would discuss papers. The comments to the committee were used as the basis for conversation, and those that had read a particular paper would offer their insights. But the final decision was based solely on the discussion, not the scores.

In the end, out of 49 submissions we accepted 24 (that’s 49% for those of you without a calculator). I believe that the resulting slate of papers is excellent this year, and I am looking forward to reading and hearing as many of these papers as I can.

I’ve been working hard with Dan Klein in the past few days to put together an iCal calendar that contains the session information for the conference. I’ve never put one of these together before, so I’m still making some beginner’s mistakes. Dan and I settled on a simple multi-line format for the records that will be easy to maintain. Dan wrote a perl script that converts this format to ICS and from there (as I understand it) iCal compatibility is a snap. For now we don’t have location information on any of the sessions. Once we get room names we will add that stuff to the calendar. I am also currently musing about whether or not to include the invited talk descriptions. Right now it is just speaker and talk title. But I may add the descriptions as well before we make the calendar live. I’m not much of an iCal person, so this is all still pretty new to me.

I shall keep you posted.

The program committee met at the end of June at the O’Hare Hyatt Regency in suburban Chicago. I arrived in town a day early to visit relatives and drove out to the hotel the night before the meeting. I didn’t get to the hotel until very late, around 11:30 pm. You probably know what a hotel lobby is like late at night. The skeleton crew is behind the desk: there are maybe two people on duty. You walk in and go right up to one of them and check in. You’re done and on your way in 5 minutes, maybe less. Not that night! When I got to the front desk there was a line a mile long and it didn’t seem to be moving very fast. I checked my watch to see if I had suddenly fallen through a time warp, but it still said 11:30. How weird. As I was standing around waiting for the line to move, eager to get to my room and rest up before the big meeting, I wondered aloud why there were so many people. The lady standing next to me was kind enough to share her story. Apparently a United flight to Phoenix had been cancelled, and she suspected several other flights had suffered the same fate. Most of the people waiting in line were airline refugees, sent there with the promise of a room for the night and a $7 meal voucher. I wasn’t really sure what sort of meal $7 was going to buy in the Hyatt Regency, especially since the only place still open was the sports bar. $7 of room service might get you a napkin and a glass of water, but only if you pay for the mandatory gratuity yourself.

I was sure glad that I had a reservation, and was afraid they might decide to give my room away anyway, considering how late my arrival was. But my fears were unfounded. When I finally got to one of the frazzled employees behind the front desk, I cheerily said “Hi! My name is William LeFebvre and I actually have a reservation.” He sighed and replied, “Thank God!”

Every year approximately 30 papers are accepted at a LISA conference for presentation and publication in the proceedings. These papers present the latest research efforts in all corners of our industry. Authors come from all types of organizations and backgrounds. We have many universities and research groups present their work, as well as corporations and independent consultants. This is where you will find the ideas and technologies that will become commonplace in the years ahead.

So how do we choose these papers? You might think that a fortune teller and ouija board are involved, but things aren’t quite that mystic. At the beginning of the year I selected twelve experts in system administration to create the core of my program committee. In February we sent out a call for papers and by the May deadline we had received 49 submissions.

Each submission needed to be thoroughly read and reviewed. So as soon as the deadline had passed I began to scan and categorize every submission. These categories are very important to the process. I know that I have one fantastic committee with some of the most knowledgeable people in the industry. But not everyone knows everything, and it is important to match papers with those who are qualified to review them. The categories make this matching possible. I made sure that every paper was reviewed by at least four people, while at the same time limiting each individual to no more than 10 papers. Reviewing papers takes time, and I didn’t want to drive any of my committee members to the brink of insanity by giving them too much to read. I also added a few readers to the list of reviewers. These are folks who are not on the committee but help us out by doing some of the reviews.

Last year the program committee, David Blank-Edelman, used some ultra-powerful fancy perl script to perform this matchup. But I had some reviewers who were a little slow choosing their categories. That and a number of other issues made it simpler for me ot just assign them by hand. But it was a bit like a jigsaw puzzle making everything fit.

Once reviewers are assigned they typically have about 4 weeks to read the papers, type up comments, and assign scores. Each paper is scored in a number of areas, including technical and editorial quality, as well as suitability for the conference. The reviewers’ scores are averaged and the papers are sorted by these scores.

On a cold Saturday in June, the program committee was locked in an undisclosed and top secret bunker in the side of a mountain. Well, okay, the Saturday was warm, we weren’t locked in, the bunker was really a conference room at a hotel, and it was in the flatlands of suburban Chicago. But the other way sounded better. We spent the day discussing each paper individually, even the ones that ranked the lowest. The scores from the review process were used as a guideline, but were not the final deciding factor. A decision was made on each paper based on the score, reviewers comments, and the discussion of the entire committee. We were able to reach a concensus on almost every paper. There were a few papers where individual committe members had strong and opposing views. Hard to imagine such a thing with this group, isn’t it? So in cases where we could not reach concensus I just overrode everyone and made my own decision. In the end we accepted 24 good quality papers. Once the meeting was over we all went out for some stiff drinks.

Greetings. My name is Bill LeFebvre and I have the privilege of being your host and conference chair for this year’s LISA conference. This will be the 20th Large Installation Systems Administration conference, and it is sure to be one to remember. The committee has worked very hard this year so that we can provide you with a fantastic conference. So I hope that you will be joining us in Washington DC this December. If you want to see all the official details you will find them over here.

Over the next few months I will be using this blog to tell you about some of the special things that will be happening at the conference, and to bring you some stories from behind the scenes. I started working on the conference in January of this year and it has been a very rewarding experience for me. So please come back and visit this blog regularly to get the latest scoop. I will leave comments enabled so that you can join in the fun if you wish.

Finally, let me state for the record that anything I put here is just from me. Don’t take it to be anything official from Usenix, because it’s not. I will try not to embarrass too many people.