This year our Guru-is-in Coordinator did such an amazing job at finding great speakers that we found ourselves with a overflow of great speakers on topics we knew attendees would want to hear about.

We know that people really like these sessions because it gives them a chance to interact with some of the heavy-hitters in an area. Imagine getting advice on (to pick some at random):

• backups from Curtis Preston
• Samba from Jerry Carter
• OSX from Jordan Hubbard
• OpenSSL from Ben Laurie
• Solaris 10 features (dtrace, zones, ZFS) from key members of the Sun engineering team

This level of access to the best in the field was too cool for us to limit, so we decided to do everything we needed to make sure we could accommodate the bonanza. As a result, you’ll find that we’ve split some of the sessions and added extra session rooms to the schedule wherever we could. Going to be a heck of a track this year.

Register for the conference and the tech program here.

Each program chair gets to choose her or his method for a) keeping track of all of the kerjillions of details associated with planning the program and b) coordinating all the communication that has to take place between the various volunteers during the planning process.

I knew email alone wasn’t going to cut it. Sure, we exchanged tons of it (and continue to keep port 25 hot) but email doesn’t provide an easy repository of information for the entire set of people involved to reference/edit. A collection of email also doesn’t do a good job of representing the current state at any one time (e.g. a concise summary of what was decided on a particular subject). We really needed a canonical data store and rendezvous point for all of the activity taking place.

I chose to put up a Wiki for the organizers to use because it seemed to be a perfect application for that model. Originally I planned to use MediaWiki because that’s what Wikipedia uses but a test instance ran way too slow for me when I tried it. (This speed problem, I’m certain, was a reflection of the particular way I had it set up and has nothing to do with the software itself). At that point I found TWiki and decided to use it instead because it looked fairly mature and my Perl background would be helpful if I needed to hack/debug it.

In general we used a stock version of the production release with only a few small tweaks beyond the normal configuration work:

• added the calendar plugin
• added the mailer contrib plugin (for better notification of changed topics)
• added the EditTable plugin(for a reason you’ll see in one sec)

So how well has this worked for us? Here’s my impressions as program chair, other conference organizers who were in the process may have different opinions:

Bad: several of the committee members despise wikis (for reasons that I never quite investigated), but they managed to use it through gritted teeth just to humor me. I thank them for this, though I wasn’t happy using a method that other people didn’t like. (On the flip side, several committee members really dig wikis, so perhaps it all evened out?).

Less good: As an experiment, I tried using the wiki as a annotation tool. I spent a bunch of time converting the results of the 2004 LISA attendees survey into wiki format, commented the heck out of it, and then asked other people to add their own annotations. The wiki was not the ideal tool for this, probably because its (intentional) lack of tools to deal with structured content. Ideally you’d like to be able to freeze the source document and just let people freely edit comments on it. I’m sure I could have hacked TWiki into doing a better job of this ala QuickTopic Document review, but I didn’t spend the time. [As an aside: yes, we really do pay attention to the results we get back from the survey we ask attendees to fill out. I spent a lot of time pouring over it to help me understand ways to improve this year’s conference. Please fill out the 2005 version so my successor also has that kind of data.]

Less good: The wiki got used at times as a discussion/conversation forum about various topics. People would leave comments on the content (sometimes interleaved into the content itself) or in response to other people’s comments. While this worked ok, my experience was this tended to get unwieldy or just graphically unpleasant. Perhaps this was just bad typographical conventions on our part, but my experience with wikis in general suggests they don’t function nearly as well as threaded discussion forums do for this purpose. We never got to the point where one had to perform archaeological forays to find things (as one does sometimes on TWiki.org) but that was a likely endpoint. I took a very strong hand at times and shuffled off past discussion to separate wiki pages whenever it seemed appropriate.

Good: Besides the issue with discussions mentioned in the last item, the wiki functioned well for general brainstorming (e.g. where people contribute to a list of items).

Very good: The wiki functioned well in two cases: scheduling and status reporting. I find the process (especially with people this busy) of scheduling meetings and conference calls via email to be pretty painful. It worked very well to ask everyone to go to a wiki page and edit a pre-made page where they could indicate their availability. Yes, this could have been done with either polling software or calendar software, but this was a quick and easy (for me) solution that didn’t require a bunch of round-trip email transactions. Similarly, we used the EditTable plugin mentioned above to allow people to easily report back the status of papers as they moved through the shepherding process.

Very good: (all of the stuff regarding ad-hoc content creation/editing wikis are good for)

I had thoughts about bringing in other cool web-based planning software (e.g. Basecamp) into the picture, but I ran out of time to deal with the infrastructure (vs. just planning the conference). Still, I’m pretty happy with what we did use and I plan to apply the knowledge I gained from working with the software to my day job.

One of the things which makes LISA special and different from other conferences is the research and investigation into system administration that gets presented. I don’t know of any place else that publishes peer reviewed work like this designed to advance the field.

Here’s a few examples I’m pulling randomly from the technical session listing:

• Fast User-Mode Rootkit Scanner for the Enterprise, Yi-Min Wang and Doug Beck, Microsoft Research (seen anything in the news about rootkits lately?)
• A Case Study in Configuration Management Tool Deployment, Narayan Desai, Rick Bradshaw, Scott Matott, Sandra Bittner, Susan Coghlan, Rémy Evard, Cory Lueninghoener, Ti Leggett, John-Paul Navarro, Gene Rackow, Craig Stacey, and Tisha Stacey, Argonne National Laboratory (what is deploying this stuff really like?)
• Reducing Downtime Due to System Maintenance and Upgrades, Shaya Potter and Jason Nieh, Columbia University (reducing downtime something you’ve been asked to do at your job?)
• A1: Spreadsheet-based Scripting for Developing Web Tools, Eben M. Haber, Eser Kandogan, Allen Cypher, Paul P. Maglio, and Rob Barrett, IBM Almaden Research Center (if you’ve suspected spreadsheets could be useful for something beyond number crunching, here’s how to do sysadmin with them)
• Manage People, Not Userids, Jon Finke, Rensselaer Polytechnic Institute (identity management issues buzzing louder at your workplace these days?)

Register for the conference and tech sessions here.

That’s the thinking behind a special evening program on Monday night we’ve arranged for this year’s LISA. We’ve invited two frequent contributors to the magazine (one of which is on their tech advisory board) to come and talk to us about their work and the philosophy behind the new resurgence of do-it-yourself-ing behind MAKE. They’ll also be bringing a bunch of stuff for attendees to see and play with after their talks.

(added bonus: the first 100 people attending the talk will receive a free copy of the magazine, courtesy of the nice people at MAKE magazine. If you’ve never seen the magazine, here’s a special online sampler for people reading this blog.)

Here’s the info on the talks for this special evening:

Talk I: Tweaking, Bending, and Making: Stories of a Hardware Hacker
Joe Grand, Grand Idea Studio, Inc.

Never before has the do-it-yourself ethos been so popular. Bolstered by loose-knit communities of curious tinkerers and O’Reilly’s new quarterly MAKE magazine, tweaking, hacking, and bending have all but reached the mainstream. Behind the projects lie individuals with the drive to make something better, to modify a product to do something it was never intended to do, or to just create something out of the ordinary. This approach to problem solving should be familiar to the USENIX community. 


In this fun and light-hearted session, Joe Grand, electrical engineer and obsessed inventor, will tell his story and that of MAKE magazine. Armed with some interesting, wacky, and/or curious hardware hacks, Joe will provide a show-and-tell that will hopefully motivate you to embrace the Maker mindset in your own lifestyle. 


Joe Grand is the President of Grand Idea Studio, Inc. (www.grandideastudio.com), a San Diego-based product research, development, and licensing firm, where he specializes in the invention and design of consumer electronics, video game accessories, and toys. Joe is the author of several books, including Hardware Hacking: Have Fun While Voiding Your Warranty and Game Console Hacking. He is on the Technical Advisory Board and is a Contributing Writer for MAKE magazine. 


Joe is also a globally recognized figure in computer security. He has testified before the United States Senate Governmental Affairs Committee and is a former member of the legendary hacker collective L0pht Heavy Industries. Joe holds a Bachelor of Science degree in Computer Engineering from Boston University.

Talk II: Hacking Silicon: Secrets From Behind the Epoxy Curtain
Bunnie Huang, bunnie studios, LLC

I’ll talk about basic methods and theory behind silicon hacking:

• motivation
• examples of silicon-based security
• overview of methods for decapsulating silicon chips
• methods for imaging chips
• theory behind deciphering silicon chips (briefest introduction)
• practical example of hacking a PIC microcontroller to recover data from security fused regions

Bunnie Huang (www.bunniestudios.com) has a strong background in silicon design and reverse engineering. bunnie completed his PhD at MIT on computer architecture, with an emphasis on the big-picture silicon implementation issues of large scale parallel machines. During the course of his studies, bunnie reverse engineered cryptographic keys out of the Xbox hardware and published his findings in CHES (Cryptographic Hardware and Embedded Systems) and in a book titled Hacking the Xbox. bunnie’s professional experience in silicon design (which includes 802.11b/Bluetooth radios, 10 Gigabit transceivers, CMOS photonics, and various prototype chips for silicon devices research) combined with his reverse engineering expertise gives him a unique perspective on silicon hacking.

Register for the conference here.

Ben Laurie has agreed to be a speaker in our guru-is-in track. You probably know his name because he’s been key contributor in several of the projects you deal with daily (Apache/Apache-SSL and OpenSSL ring any bells?). He’s also done some tremendously cool security related work like the Bluetooth attacks (see his blog and homepage for other examples).

Here’s the official bio:

Ben Laurie is the Director of Security at The Bunker Secure Hosting. He is the author of Apache-SSL as well as serving as an Apache core team and board member, and an OpenSSL core team member.

Register for the conference and the tech program here.

More info on the training class here. Register for the conference and this class here.

Now you have the chance. Gerry Carter, one of the hardest working people in the training biz, is teaching a class on just this subject (Ethereal and the Art of Debugging Networks). He’s been a core member of the Samba team (who know more than anyone should have to know about protocols on a wire) and a LDAP/Kerberos guru so he has tremendous Ethereal chops.

More information on the training class here. Register for the conference and this class here.

Dan Klein, the training program coordinator, does his best to make sure that each conference brings with it new and exciting classes. This year’s LISA is no exception so I though I would highlight some of the new classes that caught my eye in this blog entry and the next few entries.

This year I noticed that Mark Burgess (the only full professor of network and system administration I know of), is teaching a new intro class in configuration management (Understanding Configuration Management). This is like offering a beginning animation class with Will Eisner (RIP) or Hayao Miyazaki. Burgess is the author of cfengine and has been an active researcher at the forefront of the configuration management for many years. It’s a heck of an opportunity if you are interested in configuration management at all.

More info on the training class here. Register for the conference and this class here.

Hubbard was one of the co-founders of FreeBSD and one of the reasons why that project has developed into the well-respected operating system it is today [just fyi, the words you are reading are being served off FreeBSD boxes].

I still remember when it was announced in 2001 that Hubbard joined Apple to work on Darwin. It was at that point that I knew we were in for some interesting and substantial stuff out of Apple. Turns out I was right [he says while typing on an Apple Powerbook].

For the Guru-is-In track, we do our best to bring someone who really knows their stuff on a topic. For Mac OSX I’m sure you’ll agree we found the very epitome of the term “guru” when you hear that we’re privileged to have Hubbard as the Mac OSX guru-is-in speaker at LISA 2005.

Here’s Jordan Hubbard’s bio from our program:

Jordan Hubbard is the Director of UNIX Technology, CoreOS, at Apple Computer. He has been a software developer since the late 70’s and is a longtime contributor to the open source community, from the earliest days of USENET’s comp.sources.unix group, through MIT’s X11 contributed software collection, to the FreeBSD Project, which he co-founded in 1993. These days, he focuses on the day-to-day development of Mac OS X and, more generally, on Apple’s open source strategy and its relationship with traditional UNIX developers and administrators. His current pet count, for those who follow such things, is 10 cats and 4 dogs.

I was curious about what it took to provide a network for a conference full of sys and netadmins so I asked Tony to describe the setup he uses for LISA. Here’s what Tony, the sysadmin’s sysadmin, wrote back:

David asked me to talk a little about the USENIX conference LAN. I’ve tried to limit the following to the bits that are somewhat unique to a conference network, and LISA at the Town and Country specifically though most of it applies to every USENIX conference LAN.

The most crucial bit of the LISA ‘05 conference LAN is the internet connection, without which there really isn’t much point in having a network. Attendees used to corporate LANs or cable modems don’t think twice about downloading ISOs or pulling large chunks of code from CVS while at a conference, so having plenty of bandwidth is an obvious primary concern. ISP contracts being what they are, USENIX can’t order up a T3 for a week, or even a month, so we’re largely reliant on what the venue has to offer. The T&C has a shiny new T3 which is wonderful compared to the T1 we’ve used in previous years.

Second to the connection is the site network infrastructure and how much leave I have to use and alter it. The T&C is a property (meeting planners call hotels “properties”) USENIX has visited many, many times and, unlike some properties I’ll not mention, is very accommodating in granting access to its infrastructure. The T&C’s ethernet isn’t great, but isn’t non-existent either. There are always challenges in making a network that is designed to work a certain way work the way I need it to. Most of the resolutions to those challenges involve me on my hands and knees taping down several hundred feet of CAT5. Don’t walk barefoot at a conference. Trust me. The T&C requires three such runs of cable to work around the way the room the router sits in is wired. Why not move the router to some central location you ask? Access. The main wiring closet of the hotel is in a locked cage that only a few people have unrestricted access to, and I don’t number amoung them.

A bit on the hardware and software I use. The “router” for the conference LAN in past years has been an 800 PIII Mhz Dell laptop with three interfaces running OpenBSD. A little over a year ago I discovered the hard way that PCMCIA cards are pretty limited in the amount of traffic they can handle. Thus the current “router” is a 700Mhz PIII desktop with a gigabit interface for the conference LAN and a four port Soekris card for the internet connection and registration LAN. A note on the Soekris card: it buffer underruns under load. I have a cron job that ifconfigs the active interfaces up and down every five minutes. The next conference router will not have a four port Soekris card.

The “router” runs the usual collection of network software: Bind 9, ISC dhcpd (the OpenBSD version), Squid, and an ftp proxy. NAT, packet filtering and redirection is done by OpenBSD’s packet filter, PF. A laptop provides a second dns server for the network and doubles as a router and firewall for hands-on security training classroom. The Squid proxy has been voluntary at past conferences but became transparent for our Security conference. 400 people on a 1.1Mbit DSL line without caching is not pretty.

Wireless hardware is a collection of four old Aironet 4800 series access points, five Cisco 1200 series access points, and an Airport Base Station for small isolated meetings. The T&C presents more of a challenge than most venues because it is so spread out, thus requiring more hardware than any other property. The LISA conference format recently changed such that training and technical sessions happen on the same days which requires additional hardware. In short, I need more access points for LISA at the T&C than any other USENIX conference, and I don’t have them. I’ve tried using borrowed SOHO access points, but they fall apart with more than about 10 active connections. The Ciscos can handle 40 to 50 active connections on a single power outlet and ethernet connection. If you have spare Cisco gear laying idle you’d like to lend to the network please let me know.

What I do is based on the work of many others, my own experience, and the suggestions of attendees at each USENIX conference. LISA is the most challenging USENIX conference as its attendees use the most bandwidth, use “security evaluation” tools the most, and have the highest percentage of laptop usage. LISA is also the USENIX conference I learn the most at, and have the most fun at. This year a network team is forming to provide additional services on the Conference LAN. If you are interested in helping or have an idea for a service to provide, please send e-mail to wireless at usenix dot org.