Trip Report SANS IV Conference System Administration, Networking, and Security Washington, DC April 26-28, 1995 A copy of the Proceedings is in Melissa's office. BOFs: ----- - SAGE We reviewed what SAGE does, including their web site http://www.sage.usenix.org/sage/. Apparently they will be providing tutorials for UniForum. Upcoming booklets (to follow their Job Descriptions one) will include one on legal issues for the sysadmin, and one on site policies. Suggestions for what else SAGE should do (including petitions by local groups for funds) can be sent to sage-board@usenix.org. - Majordomo and Large Email Sites This BOF dealt with assorted email administration issues, with a lengthy discussion (2.5 hours) of Majordomo, talk about POP, and how to administer aliases files. Seemed like a BOF that would be popular at many conferences. The current developer for Majordomo was present, and told us about a number of features that will be available in v1.94 (currently in alpha testing). It will not slow down as much when dealing with a large number of aliases. It will have access lists for just about everything. There will be an option to prepend the listname (intelligently) to the subject of each message being processed. Someone runs an autobounce program, that handles mailing list recipients who are bouncing their mail. It generates "approve-unsubscribe" messages to the list maintainer, and moves the bouncing address to a bouncer list. They will be sent regular reminders of the fact that their address bounced. One university site has 30,000 users, and they receive 60-80,000 messages a day. Sun sold them a package with a SparcCenter 1000, Online Disk Suite, and NIS (NIS+?). But NIS dies on a host receiving a lot of mail, so they bought an HP for their NIS server. There was some discussion of listproc. Apparently, it keeps track of message IDs and checksums (or perhaps adds an X-Listproc: header) to prevent loops. We discussed the "Content-Length:" header, which can be broken by manipulations like this, causing the message to be unreadable by some MUAs. The easiest solution is to remove that header. There is a product called Unameit (or possible Uname*It) which will help manage alias files as well as the rest of the "enterprise name space". Apparently Tivoli's product (like Uname*It? is Uname*It?) now works for 3 million users. Someone suggested that each alias, in addition to a list of mail addresses, should forward all messages to "|head -1 >> counters/aliasname", in order to keep track of whether the list is in use. There was rumor of something called APOP which is like POP but allows for a pool of mail hubs to service the requests. In a similar vein, MX records can be used to spread the incoming mail load to a number of hosts. There is a package called MLA, mail-list-archiver, which generates HTML forms for browsing mail archives. It is from the author of flex-fax, and is allegedly available from ftp.sgi.com. - Women As usual, the women's BOF was attended by about a dozen women and one token man. The woman who had suggested the BOF took a list of names and may start a mailing list. She offered to contact SAGE and see if there is previous womens-bof info or need for a mailing list or SAGE women's page. We discussed how women get into technical positions, many issues common to all system administrators (getting more money, appreciation for our work, etc), and took a few informal surveys. We discussed support for young women studying science and computers. Apparently there is some sort of women's page at MIT. We also discussed trouble ticket systems and help desks. Keynote: -------- - Katie Hafner, Newsweek Associate Editor and author of Cyberpunk: Outlaws and Hackers on the Computer Frontier. Katie Hafner's book Cyberpunk had three sections, one of which was about Kevin Mitnick. Given recent Mitnick events, her keynote was very timely. She is writing an article, possibly appearing in Esquire, to update the Mitnick story. This will also appear as an epilogue in a new edition of Cyberpunk. She re-iterated part of the original Mitnick story, and gave us an update on recent events. She has interviewed many of the major players in these incidents, and had many insights into what drives them (to hack, or to turn each other in). She gave a book-signing after her talk. SysAdmin Track: --------------- - Goals of System Administration, by Elizabeth Zwicky, Silicon Graphics. Or, how to tell if you're doing a good job. Elizabeth divides sites into three levels of support: average: what most sites have; acceptable: the least she can stand to work with; and excellent, which is rare. She points out that a company has certain assets. Usually the most important asset is the people, and the next most important is the data. Backups protect the data from accident. Security protects the data from intentional harm. Networks allow the data to get where it needs to. She emphasized the importance of support (an official, recognized way to get help), and gave examples of the support levels for resource sharing (of files and printers), maintainability (upgrade paths, keeping things working), and backups, security and networks. Also important are human resources issues -- keeping the sysadmins (since employees are an asset). - The 1994 SANS Salary Survey, presented by Rob Kolstad, BSDI. The survey results were compiled and graphed by Alan Paller of Computer Associates, but he couldn't make it so Rob presented the results. The top 25% of sysadmins are making an average of $64k, the next 25% $49k, then $40k, then $30k. Rob points out that this last group must include some part-timers at something like $10k. One of the top earners was someone in New York City making $125k, with a beeper on 24-hour call. 29% of those surveyed are making between $40k and $50k. Annual salary increases average 5.5%. Sysadmins with one year or less experience average $38k, though Rob guesses this is so how because they must have switched careeres recently. One part of the survey addressed how to get better raises, and the most popular answer was to get a new employer. One person went so far as to reach the exit interview, where he told them he was leaving to get more money, and they promptly made him a better offer, so he stayed. - People Skills for the System Administrator, by P. David Parks, IEEE (or maybe Glaxow Pharmaceuticals). David discussed various things that ought to be common sense, but apparently often aren't. He reminded us to use tact, to carefully consider what we are saying in email, and to use good manners in our dealings with other people. Common problems with email include flaming, and sending mail to an entire mailing list instead of one person. He emphasized that a sysadmin should try to properly answer questions, rather than answering with rhetorical questions, stating that they have no time to explain, or giving cryptic answers that do not help. - How SysAdmins Spend their Time, by Rob Kolstad, BSDI. I missed the beginning of this, but due to a missing presentation, Rob apparently re-presented the results of his 1992 survey (seen previously at the 1993 LISA). It seems we spend most of our time dealing with other people, and very little time doing backups. We spend our time doing many different things -- planning, fixing, helping people, etc. He reminds us that reading netnews can be considered personal growth (depending on the newsgroup), but that we might re-evaluate the time we spend and what we are getting from each newsgroup. - How to Hire Good System Administrators, by Michelle Crabb, Sterling/NASA Ames. Old style interviewing was usually done by a hiring manager, but these days it is often helpful to leave managers out of the loop. Michelle strongly recommends phone interviews to do an initial screen of the candidates before inviting them over and having a lot of people spend time with them. Each candidate should be taken on a tour of the facility. Michelle gives the candidates a quiz (12 easy questions are included in the proceedings and online, see below) testing simple skills like writing a for loop in the Bourne shell, or asking what 5 commands the candidate uses most often. At her site, they have a number of people interview the candidate one-on-one, then get together to discuss the results -- this is also important for finding out if the candidate was taking comments from their first interviewer and feeding them back to the next interviewer. More info is at http://www.nas.nasa.gov/NAS/RelatedPapers/SANS95/, for all the papers Michelle presented at SANS IV. - "Best of SysAdmin Magazine" -- Books You Really Want to Read, by Elizabeth (Betty) Zinkum, SysAdmin Magazine. Betty writes a book column for SysAdmin Magazine, and listed her favorites. She had no slides, so these titles are approximate. First, she likes "Preventing Computer Injury, a Hand Book", published by Ergonomy. The author is a pianist, who first became concerned when she was at Julliard and her roommate came back from practice sessions unable to turn a doorknob -- 25% of Julliard students suffer from Carpal Tunnel Syndrome. Betty also listed common favorites like "The UNIX Programming Environment", "Advanced Programming in the Unix Environment", and Nemeth, et al's "Unix System Administration Handbook". Not surprisingly, she listed several O'Reilly books, including "Starting UNIX Users" (?), "UNIX Power Tools" (with CD), "Essential System Administration", "System Performance Tuning", "TCP/IP Network Administration", and the new "Managing Internet Information Servers". She also recommended Peter Salus's new "Casting the Net", a history of the Internet. Also mentioned were Cheswick's "Firewalls and Internet Security" and one or two "dragon books" by Bruce and Karen Hunter called "Unix Systems Advanced Administration and Management Handbook" and "UNIX Networks: a Guide for System Administrators", which she found very readable and complete. - From the Other Side of the Desk: Observations of Consumers from Executives in the Software and Service Industries, by Rob Kolstad, BSDI, and E. Scott Menter, Enterprise Systems Management. Rob and Scott are Executives from the Software (BSD) and Service (sys admins) industries, respectively. They attempted to give us a reality check on what the goals of vendors are -- that it is not a personal relationship, but a business one. Rob and Scott see different types of customers that need to be dealt with differntly. Early adopters require little training, appear in trade magazines, and have a good attitude. Middle-of-the-road types don't want to fall behind, they read the trades with a grain of salt, and usually deal with conservative vendors. Lastly, dinosaurs believe the trades, and like to avoid risk. Some companies can belong to different categories on different days. They also discussed what the various players think they're doing. Executives feel they are creating value; they are non-technical, but look at the big picture and long-term results. Sales people feel they are problem solvers; that they are the only money-makers at the company; paradoxically, they report to management and to the consumer. Engineers see themselves as problem solvers who create the product; they take the short view. Those in support feel they are producing satisfaction; they get no respect but are usually very technical, very focused on each call; they take a very short view. Lastly, the consumers' goal is to improve productivity, to stay competitive, and to spend money. Most frustration in the vendor/customer relationship comes from mismatched expectations, e.g. when the customer forgets that the vendor is trying to make money or when the vendor exaggerates. Recommendations include looking for volume purchases which might require less media or packaging; know who provides support; know your customer ID before calling, as well as your configuration; try to maintain mutual respect when you call, expecting assistance and not training; don't put them on hold. - Advanced Perl 1, by Tom Christiansen. Tom told us about Perl 5 -- how it differs from Perl 4, how it is better, and assorted cool features. He showed a lot of examples, and pointed us to ftp://ftp.cis.ufl.edu/pub/perl, which has most perl items. He also mentioned ftp://ftp.perl.com/pub/perl, but asked us to be gentle and use it sparingly, since it is at the end of a very narrow pipe. Somewhat orthogonal to perl, Tom recommended we use html for forms entry, and pointed us to webstorm.com's pub/perl/ext/CGI*, which does stateful CGI programming. Perl 5 was a major rewrite. It is "object oriented" (including inheritance), and many modules are broken out from the main perl executable to be dynamically loaded. For example, there is a Tk package, so Tk programs can now be written in perl instead of tcl. Perl 5 has anonymous functions, references (like & in C), transparent access to C code, enhanced diagnostics (the -w switch and a sysdiag man page), and a lot of documentation (man pages). Multidimensional arrays now just work. - Building an Ethical and Policy Infrastructure, by Rob Kolstad, BSDI. Rob defined "ethics" as an acceptable code of behavior, versus "law", a formalized code of behavior. The implicit goal of ethics is to keep or create a better society. Rob moderated a lively discussion about anonymous remailers. They provide a service in that people can talk about personal issues (being molested) or things for which they might be persecuted (alternate lifestyles -- the favored example being people who keep ferrets in states where it is currently illegal). However, they pose a problem when it comes to tracking down illegal activity, like posting of copyrighted information or sending threats or harassment. Rob asked what we as sys admins should do about it. Is the current status okay? Should we make it easier to be anonymous or harder? Points people brought up included: an analogy to paper mail, which can also be forged or untraceable; citizens working to change ferret-owning laws; how useful the internet is to schools, despite the problems; and harrassing phone messages. Networking: ----------- - A History of UNIX and the Internet, by Peter Salus. Peter based this talk on his two books: A Quarter-Century of UNIX (from 1994) and Casting the Net: The ARPANET, the Internet and Beyond, just released. In the late 1950s, the government was scared by Russians, so they commissioned a report on packet-switching communications between computers. The first IMPs (Interface Message Processors) were not made until 1967, though in 1969 the first four sites were connected -- three in California and one in Utah. They originally had only 6 bits of address space, for a maximum of 63 hosts. Peter showed us some of the first RFCs, which were typewritten and some even handwritten, since printers were not as readily available as they are today. He also had the first network map, consisting of two boxes connected by a single line. In 1972 a satellite connection was made to Hawaii, while UNIX became more and more popular even without official support. In 1974 the first UNIX user's meeting was held at Columbia University, hosted by Lou Katz and Reidar Bornholdt -- some two dozen attended. In 1977 the name USENIX was adopted, with the statement that "UNIX is a trademark" while "USENIX is not a trademark". UNIX and the Internet continued to accelerate, to the point where the number of hosts on the Internet has doubled in each of the last three years. Networking: ----------- - PONG: A Flexible Network Services Monitoring System, Helen Harrison, SAS Institute. As always, Helen presented a practical solution to a common sysadmin problem. We all use "ping" to monitor hosts and see if they are up, but ping is insufficient for monitoring the services on those hosts. At SAS they wrote "pong", a modular program which connects to services and sees if they are working. Built-in modules include tests of ICMP echo (aka plain-old ping), inetd (via the "echo" service), NFS (via a nullRPC call), and AFS. It can be expanded to test any service by specifying a short no-op interaction with that service. Pong reads a configuration file which specifies which services to monitor on which hosts, and intervals for detecting outage and return to service. This requires tuning to detect problems before too many users do, without adding too much to system load. Notification is done via logs, mail, and xmessages to operations staff. Subsidiary programs include pongstat to query the server, and KingPong, a graphical interface to pongstat with color-coded buttons -- yellow if a service was down and is back up, red if a service is currently down. Pong, including a postscript version of the paper, is available at ftp://www.sas.com/outgoing/. - Performance Monitoring for Network Preemptive Maintenance, F.B. Motahdi, G.E. Telephone Operations. Mr. Motahdi prefaced his talk by pointing out the difference between people talking about phone networks and us computer folks (he wore a tie), and apologized if he spoke a different language. His task is to monitor T1 lines, and to ensure service reliability by detecting and remedying failures before the customer notices. Optimally, they prefer to prevent problems, rather than recovering from them. The theory is that an intermittent problem becomes a soft failure, and this then becomes a hard failure. So they are working on pattern recognition to identify cyclical errors, degrading errors, and faults. They are in the initial data collection phase. - The Netscanner Network Monitoring Tool, Jack Stewart, California Institute of Technology. At CalTech's Center for Advanced Computational Research, they have a unique collection of unusual machines. One of them is a Touchstone Delta, with 576 nodes, whose primary diagnostic was a lot of lights. They created a program called xpartinfo to pictorially show which nodes were free, which became heavily used. With more machines added to their site, they expanded xpartinfo to NetScanner. NetScanner contacts hosts in sequence, and is configurable. It monitors file servers, terminal servers, and supercomputers. Security: --------- - The Pros and Cons of Vulnerability Information Disclosure, by Gene Spafford, Purdue. The hypothetical for this talk was: you discover a flaw, what do you do? If you keep it a secret, does it matter to anyone else? Who will get hurt? Perhaps you reveal it to trusted people -- but who do you trust? Will it get fixed? Do you publicize the problem in hopes that it will generate a fix more quickly? Some vendors have stated they will not hurry a fix when threatened in this way. The fix that results from publicizing a flaw may be too hasty and buggy. Before the fix comes out, the flaw may be used against you. - The Basics of UNIX Vulnerabilities: Case Studies and Analyses or UNIX Security: Threats and Solutions, by Matt Bishop, University of California at Davis. Matt searched for the root of today's security problems, and sees many as stemming from the fact that UNIX was designed by programmers for programmers, and is very open. There are several types of common flaws. For example, the "wiz" attack on sendmail is the result of: inadequate validation of peer (anyone who can say "wiz"), incorrect configuration (compiling with DEBUG), and the fact that such a large program is just too complex. It also suffers from organizational flaws: the developer was not given sufficient access to debug problems, and the vendors did not disable the back door. Since UNIX was not designed with security in mind, it can be hard to backpatch. Also, there tends to be a lack of knowledge of older holes, which results in repeats of the same problems. The problems with a setuid login falling victim to dynamic loadable libraries can be seen as the same as the old "." in your PATH problem -- improper choice of protection domain. Another oft-repeated hole is the time of check vs time of use problem which causes so many race conditions. Matt considers apathetic users to be a real threat, and security test programs like SATAN or cops to be a false threat. - A Network Security FAQ (Frequently Agonizing Quandaries), by Marcus Ranum, Trusted Information Systems. Marcus offered many quandaries for which he finds no easy solutions. However, one easy solution he suggests is for executives who want to hook their most private internal systems up to the Internet so they can browse the web -- often, they can be readily served by getting a plain old public access account, unrelated to their office computer. Questions include: who can connect to the network? What is the security policy? What are the mission critical services, and how does this influence the need for security? Often, workers suffer from a "get the job done" perspective, with no time for security. Marcus finds that some problems have different owners, with inconsistent answers -- like an IP network behind a firewall, with unprotected modems to the internal side.