Chapter 2 THE EVOLUTION OF USENET: THE POOR MAN'S ARPANET By Ronda Hauben rh120@columbia.edu Part I -------- In Fall of 1992, an undergraduate college student had a term project to do. The assignment required that the project be the result of using resources beyond research from books. His professor proposed that students consider interviewing people, sending letters, etc. The student had done some reading and found a source that claimed computer networks had become "the largest machine that man has ever constructed -- the global telecommunications network"(1) The student decided that he would do his research on this international computer network that now spans the globe and that many computer users have access to. He planned to use the network as much as possible to conduct his research. After reading some of the few books and articles that he could find to describe the global computer network, he gathered a few significant quotes and wrote a brief introduction stating that he was trying to determine the subject for a term paper. He asked if the quotes seemed accurate and if readers had any advice. Some of the quotes were from a journal article discussing how the disintegration of Eastern Europe was in part due to the lack of free speech impeding computer development.(2) Also, the student asked if there was any evidence that the Berlin Wall had fallen because of new developments in computer communications. He raised several other questions and included quotes from his reading. Then he took this research proposal and posted it on the computer network news system called Usenet. Posting is the word used to indicate that one has sent an article to be propagated around the world. Usenet is like an electronic news magazine or a world town meeting. It has various newsgroups organized by different topic areas, in a variety of languages and on many different subjects. And the number of all is continually growing. Net users can post articles to any of the newsgroups, can respond to someone elses article or can send a message in response to the author of an article via e-mail. The student posted his questions to a number of newsgroups. In his post, he wrote: "`The Largest Machine': Where it came from and its importance to Society" He explained: "I propose to write a paper concerning the development of `The Net.' I am interested in exploring the forces behind its development and the fundamental change it represents over previous communications media. I will consult with people who have been involved with Usenet from its beginnings, and the various networks that comprise the Computer Network around the world. I wish to come to some understanding of where the Net has come from, so as to be helpful in figuring out where it is going to."(3) Within a few hours, responses began to arrive via electronic mail to him. Among the responses he received was one from a teacher in Russia who explained how pressure for the free flow of information was a force for change in Russia. "Hello," the teacher wrote, "I would also consider another side of the coin: the world is divided on people who use the possibility of computer-mediated communications and the ones who do not. But I am not a specialist in your field." "And as one from the East," he continued, "I know well that the Internet is the first and only connection to the rest of the world for us in Russia. But unfortunately," he reported "to get it there is not too easy.... If you have some questions you think I could answer - please send me email...."(4) The student sent the Russian teacher some excerpts from an article about the lack of free speech in Eastern Europe and its effects on computer development. And he asked the teacher some questions about the excerpts. In response to the questions, the teacher from Russia answered: "[The] first time I saw a computer was in 1985, when our institute got one. It was [an] Apple II. At that time, I had no idea about what a network is, and it was a time when PCs just started to appear in the environment of [the] `normal Russian'... As you can see, it was already Gorbachev's time, and communists were stopping (or already were unable) to keep a very strong control on information flow in the society. So it was easier to access the PC in our institute than to get permission to use photocopy devices. Then the number of PCs was fast growing, and now we have more then 15 PCs, but still no Internet connection....Most scientific institutes now have access to the net. But usually it is restricted to the possibility of using the electronic mail system." "I would say," observed the teacher, "that in the past networks had no direct effect on the life of people there, and now they become more and more important. One of the points is that it is practically the only way to communicate with the West. Telephone lines are so bad that to send a FAX message is almost impossible, conventional mail will reach the address with [a] probability of 50% and it will take at least one month," but the teacher wrote, "e-mail...will be received in 12-24 (!) hours. I have used it for the last year and never had any problems. I am lucky that one of my relatives has e-mail! I guess, you understand how the possibility to communicate is important for [the] scientific community...," the Russian teacher ended. The student received many other responses. One from a German student who described how the Berlin Wall had fallen because of the increased communication made possible by computer networks. The German student pointed out that accurate information about events like the Chernobyl nuclear power plant explosion had become available to people who were no longer dependent upon government channels as their only source of information. Also, there were responses from a teacher in Australia, a businessman in California, a net pioneer, and many others. The student decided to prepare another post. He had become interested in how far and wide the network reached and who would have access to the post he was making. He posted the following message: Subject: I want to hear from the four corners of the Net - that means YOU! I would like to hear from EVERYONE on the Net-Frontier. If you think you are weird or abnormal (or special) in terms of net-connections or usenet connections, please tell me about it.... To the further expansion of the Net! :) He received answers from over 50 people around the world from France to India and Africa. A response from Japan explained: "Yes, I believe I'm connected through some sort of hokey mechanism, but that's just because I'm in Japan. Connectivity doesn't register highly on the importance scale here. Takes a few hours for mail to get from one side of Tokyo to the other. "So what makes me so `special'," wrote the correspondent from Japan, "as far as net connections go? A few things. I can not receive most newsgroups and can not post to any. Yet a friend of mine in the same building as me (on another floor) receives a mostly different set of newsgroups and can post to a few. The interesting bit about any group we both get is that we don't always get the same articles. Japan," he explained, "the `leader' of technology, doesn't know a thing about actually using computers. Just my opinion, of course -- my company won't listen to me anyway! Hope this adds to your research ..." The student received a response from an employee in a large American company. The writer explained: "Not too strange, but I work for a big company that leeches off two small `service providers' for free mail and news feeds. Kind of funny, really....Hey, Usenet broke... and I can't receive mail from the Internet anymore, although I can send it." He described how the company told him `Sorry... the problem is with our feeds. We'll try to get them to fix it.' "Strange enough," the writer sarcastically reported "these small services...[a medical school and a public access usenet site] wouldn't drop everything to fix our problem. How dare they! Of course MY suggestion," he explained, "PAY THEM SOME MONEY," was completely ignored." He went on to explain that he had been told that his company "won't let us have a direct connection to the Internet for security concerns. I understand, but it doesn't make me happy," he ended. A response from Krakow, Poland explained that their site in the Department of Physics at Warsaw University was one of the first four sites in Poland to have access to Usenet. A response from a French user explained how the government charged a lot of money for an Internet connection in France and thus discouraged usage: "It's cheaper to send a `hello' to someone in the US than to someone 5 kilometers from my desk!," the French user wrote, "If you have a `stupidity chapter' in your paper, this could fill a few lines." From Wellington, New Zealand, the student learned that there was a "burgeoning Net Community in Wellington, as there were two Internet connections, one by a private net.enthusiast, and another run by the Wellington City Council on an old PDP-11 computer." They offered the citizens of Wellington "free ftp, telnet, IRC, archie, gopher, E-mail, and Usenet - and all the 1935 locally carried newsgroups." A response from someone who works for Bell Telephone Laboratories wrote: "Some people say that many of us at Bell are on the fringe, but we're probably in the core of things in the Internet. :-)" Other responses came from university students and hobbyists in the U.S., from Net users in Germany, Italy, India, etc. The student also got offers of help finding information, suggestions of books to look at, and offers to send him articles or reports that would be helpful with his research. Also many people wrote describing the unusual or interesting net connections they used to connect to Usenet, A user in South Africa told how he distributed news and e-mail and was trying to gain access to a satellite in order to set connections up with the interior of Africa that lacked the otherwise needed infrastructure. There were many other stories of unusual or pioneering efforts to make connections possible for people to Usenet. Many people wrote asking for a copy of the paper when it was written. In response to requests that he post the draft of the paper before it was completed, the student wrote a draft and posted it on Usenet. He received several helpful comments. He wrote the final draft and handed it in to his teacher and posted it on Usenet. A lively discussion ensued because the student's paper had maintained that the ability of users to post on Usenet was a sign that Usenet was democratic, whereas a document describing Usenet to new users which was posted in the new users newsgroup, maintained that Usenet is an anarchy. The discussion raised the question of whether there can be an official statement maintaining that something is an anarchy. Also, a number of people wrote the student asking him if they could distribute the paper more broadly, quote the paper in their upcoming book, etc. Another student who was writing a proposal for his Master's thesis cited the paper as an important source in his proposal. This all occurred within 2 weeks of the paper being posted. This experience is but one example of the important educational possibilities represented by Usenet and the worldwide communications network it is part of. Yet there are many people who still know nothing of Usenet, and many who are on Usenet do not realize the important potential it makes possible. In a similar way, many people do not know how Usenet developed or the obstacles the pioneers were continually faced with in their efforts to create and nourish Usenet. Since the details of how Usenet was created can provide helpful insight into how to deal with the problems one encounters today, following is an account of how Usenet was created. Part II --------- Usenet was born in 1979 when Tom Truscott and Jim Ellis, graduate students at Duke University, conceived of creating a computer network to link together those in the Unix community. They met and discussed their idea with other interested students, including Steve Bellovin, who was a graduate student at the neighboring University of North Carolina at Chapel Hill. Using homemade auto dial modems and the Unix to Unix copy program (called UUCP), the Unix shell and the find command that were being distributed with the Unix operating system, Version 7, Bellovin, wrote some simple shell scripts to have the computers automatically call each other up and search for changes in the date stamps of the files. If there were such changes, the changed files were copied from one computer to the other. Soon three computer sites, "duke" at Duke University, "unc" at the University of North Carolina at Chapel Hill and "phs" at the Physiology Department of the Duke Medical School, were hooked together and a simple program was running connecting the three sites. Gregory G. Woodbury, a Usenet pioneer from Duke University, describes how "News allowed all interested persons to read the discussion, and to (relatively) easily inject a comment and to make sure that all participants saw it."(5) The program was slow so the students enlisted Stephen Daniel, also a graduate student at Duke, to rewrite the program code in the C programming language. Stephen Daniel writes that "a news program written I believe by Steve Bellovin as a collection of shell scripts was already working, but it was slow, taking upwards of a minute of time on an unloaded PDP 11/70 to receive an article. I got involved when I happened to drop in on a conversation between Tom Truscott and Jim Ellis who were complaining about how slow this news program was. I suggested that if it was so slow it could easily be rewritten in C to run faster. I soon found myself volunteering to do just that."(6) Daniel agreed to write the program in C with help from Tom Truscott. This became the first released version of Usenet in the C programming language, which came to be known as A News. Other people at Duke and the University of North Carolina took part in getting the network debugged. Once the program was functioning on their respective machines, Jim Ellis went to a meeting of what was then the academic Unix users group known as Usenix. In the following account, Tom Truscott describes what happened: "James Ellis (jte) gave a short talk and handed out a 5 page `Invitation to a General Access UNIX Network' at the January 1980 Usenix Conference in Boulder Colorado. We made up 80 copies and they were gobbled up (not surprising, there were a record-smashing 400 attendees)....Afterwards, jte mentioned that the audience particularly enjoyed his description of Duke's two home-built 300 baud autodialers."(7) The invitation they distributed explains: "The initially most significant service will be to provide a rapid access newsletter. Any node can submit an article, which will in due course propagate to all nodes. A `news' program has been designed which can perform this service. The first articles will probably concern bug fixes, trouble reports, and general cries for help. Certain categories of news, such as `have/want' articles, may become sufficiently popular as to warrant separate newsgroups. (The news program mentioned above supports newsgroups.)"(8) The Invitation urged: "This is a sloppy proposal. Let's start a committee. No thanks! Yes, there are problems. Several amateurs collaborated on this plan. But let's get started now. Once the net is in place, we can start a committee. And they will actually use the net, so they will know what the real problems are." Several months later, the software for the A News program for Usenet was put on the conference tape for general distribution at the Delaware Summer 1980 Usenix meeting. The handout distributed at the conference explained, "A goal of USENET has been to give every UNIX system the opportunity to join and benefit from a computer network (a poor man's ARPANET, if you will)...."(9) Daniel explains why the term "poor man's ARPANET" was used. He writes, "I don't remember when the phrase was coined, but to me it expressed exactly what was going on. We (or at least I) had little idea of what was really going on on the ARPANET, but we knew we were excluded. Even if we had been allowed to join, there was no way of coming up with the money. It was commonly accepted at the time that to join the ARPANET took political connections and $100,000. I don't know if that assumption was true, but we were so far from having either connections or $$ that we didn't even try. The `Poor man's ARPANET' was our way of joining the Computer Science community and we made a deliberate attempt to extend it to other not-well-endowed members of the community. It is hard to believe in retrospect," he concludes, "but we were initially disappointed at how few people joined us. We attributed this lack more to the cost of autodialers than lack of desire."(10) The ARPANET that Daniel is referring to pioneered the networking technology that serves as the foundation of today's global Internet. The first host connected to the ARPANET was the SDS Sigma-7 on Sept. 2, 1969 at the UCLA (University of California in Los Angeles) site. It began passing bits to other sites at SRI (SDS-940 at Stanford Research Institute), UCSB (IBM 360/75 at University of California Santa Barbara), and Utah (Dec PDP-10 at the University of Utah). There were many unexpected problems and obstacles, but through the collaborative work by the participants using the network they were creating, the number of sites steadily expanded and by 1977 the ARPANET extended to more than 50 sites from Hawaii to Norway. Originally funded under the US Department of Defense's (DoD) program for Advanced Research Projects Agency (ARPA), however, only those academic computer science departments with DOD funding had the possibility of access to the ARPANET. Usenet, however, was available to all who were interested as long as they had access to the Unix operating system (which in those days was available at a very low cost to the academic and research computer community.) And posting and participating in the network was possible at no cost to the individuals who participated, besides the cost of their own equipment and the telephone calls to receive or send Netnews (as Usenet was called). Therefore, the joys and challenges of being a participant in the creation of an ever expanding network, the experience available to an exclusive few via the ARPANET, was available via Usenet to those without political or financial connections -- to the commonfolk of the computer science community. As Daniel notes, Usenet pioneers report that they were surprised at how slowly Usenet sites expanded at first. But when the University of California at Berkeley (UCB) joined Usenet, links began to be created between Usenet and the ARPANET. The University of California at Berkeley was a site on the ARPANET. At first, it is reported, mailing lists of discussions among Arpanauts (as ARPANET users were called by those on Usenet) were poured into Usenet.(10) This first connection between the ARPANET and Usenet, Daniel reports, only contributed to "the sense of being poor cousins." Daniel explains: "It was initially very hard to contribute to those lists, and when you did you were more likely to get a response to your return address than to the content of your letter. It definitely felt second class to be in read-only mode on human-nets and sf-lovers, which were two popular ARPANET mailing lists."(12) Daniel also clarifies the different philosophy guiding the development of Usenet as opposed to that of the ARPANET. He explains that "Usenet was organized around netnews, where the receiver controls what is received. The ARPANET lists were organized around mailing lists, where there is a central control for each list that potentially controls who receives the material and what material can be transmitted. I still strongly prefer the reader-centered view," he concludes. With the increasing connections to the ARPANET from Usenet, the number of sites on Usenet grew. A map from June 1981 shows the number of different sites on Usenet during this early period: !- Uucp links : Berknet links @ Arpanet links pdp (Misc) ! (NC) (Misc) decvax sii reed phs--unc--grumpy duke34 utzoo cincy teklabs ! ! ! ! ! ! ! ! ! ! ! +--+----+-----+-+--+-------------+-------+------+ ! ! ! ! ! duke ! ! ! ! ! +------+---+-----------------------+--------+ ! ! ! ! ! ! ! ucbopt ! hocsr--mhtsa----research mh135a harpo-----chico : ! ! ! ! ucbcory ! ! eagle ihnss vax135 (Bell Labs) (UCB) : ! ! ! ! ! ucbvax--++----------+--+--+-----+--+------+--------+ : @ ! ! ! (Silicon Valley) ucbarpa @ (UCSD) sdcsvax ! menlo70--hao : @ sdcattb-----+ ! ! ! ucbonyx @ +-----ucsfcgl sytek sri-unix @ phonlab-----+ cca-unix sdcarl There are many stories of frustrations as Usenet developed,(13) Despite such frustrations, there were many who helped Usenet to grow and develop. Unix enthusiasts and pioneers at some large organizations like AT&T's Bell Labs did whatever they could to provide support for Usenet. At one point, AT&T realized that it would save millions of dollars if it worked out the bugs to have internal e-mail. In the process it gave support to the pioneers of Usenet who were trying to develop more efficient e-mail routing programs. Also, Digital Equipment Corporation (DEC) supported Usenet in various ways and the spread of Usenet and Unix encouraged the sale of the Unix based computers from DEC. And, Usenet newsgroups provided much needed technical help for the folks using Unix and Unix based computers. By 1982, the continuing explosion of Usenet surprised even its most dedicated fans. One of those active on Usenet from its earliest days to the present, Gregory G. Woodbury, describes the shock that was experienced when the pioneers realized how Usenet was taking a totally unexpected course of development. He writes: "I do not recall that anyone was quite expecting the explosion that followed," recounts Woodbury, "What developed took everybody by surprise. When the direction of evolution took an unexpected turn, and a continental network emerged, spanning the continent from California to North Carolina, and Toronto to San Diego, it was sort of a shock to realize what had happened."(14) Statistics another Usenet pioneer, Gene Spafford presented at an IETF (Internet Engineering Task Force) meeting in 1988 showed the tremendous growth and development that Usenet experienced. Usenet developed from 2 articles a day posted at three sites in 1979 to 1800 articles a day posted at 11,000 sites by 1988.(15) Year Number of Sites Articles/Day Megabytes/Day 1979 3 ~2 - 1980 15 ~10 - 1981 150 ~20 - 1982 400 ~50 - 1983 600 ~120 - 1984 900 ~225 - 1985 1,300 ~375 1+ 1986 2,500 ~500 2+ 1987 5,000 ~1000 2.5+ 1988 11,000 ~1800 4+ Today Usenet continues to grow in the number of sites participating and in the number of posts it carries and in the number of newsgroups. Usenet is transported by uucp connections and via nntp (Netnews Transfer Protocol) along the Internet, which is the child of the old ARPANET. Many times, pioneers of Usenet have been convinced that the load of posts or the number of sites was becoming too great and that further growth couldn't be sustained. The fear is now facetiously referred to as the "imminent death of the net is predicted." Although each time the problems have seemed insurmountable, they have been investigated and solutions found to deal with them through the hard work of many net participants (referred to on Usenet as "net.citizens" or netizens). In the past few years a system of Free-Nets and community networks has begun to develop, many utilizing the Netnews software to make Usenet available to community people for free or at a very low cost. Cleveland Free-Net, sponsored by Case Western Reserve University and other community organizations in Cleveland, Ohio, was the first Free-Net. It used the Netnews software to create a set of local newsgroups reflecting the different community services in the Cleveland area like the hospitals, public schools, public libraries, museums, etc. Cleveland Free-Net users also have access to the worldwide newsgroups of Usenet. The software used by such community networks makes it relatively easy to read and post on Usenet and in a variety of local discussion groups. A number of community networks have come online around the US and in Canada. Many others are in the planning stages. Also there is a Free-Net in Erlangen, Germany and in Finland. And people with telnet access can join many of these free of charge and thus have e-mail and Usenet access through these community networks. The ARPANET pioneered important breakthroughs in computer networking technology. It also pioneered the ability to collaborate and to utilize dispersed resources -- both people and computers. Usenet represents the continuation of this tradition by making access to these collaborative research relationships available to the commonfolk. The extension of Usenet has also required a great deal of pioneering effort and technical development, but the folks participating in Usenet have been there to solve the problems. Writing in 1968 before the ARPANET network began, J.C.R Licklider, who has been called the Father of the ARPANET, and Robert W. Taylor predicted the challenge that would face society with the development of computer networks. "First, life will be happier for the on-line individual because the people with whom one interacts most strongly will be selected more by commonality of interests and goals than by accidents of proximity. Second, communication will be more effective and productive, and therefore more enjoyable. Third, much communication and interaction will be with programs and programming models, which will be...both challenging and rewarding. And, fourth, there will be plenty of opportunity for everyone (who can afford a console) to find his calling, for the whole world of information, with all its fields and disciplines, will be open to him, with programs ready to guide him or to help him explore." "For the society, the impact will be good or bad depending mainly on the question: Will `to be on line' be a privilege or a right? If only a favored segment of the population gets a chance to enjoy the advantage of `intelligence amplification,' the network may exaggerate the discontinuity in the spectrum of intellectual opportunity." "On the other hand, if the network idea should prove to do for education what a few have envisioned in hope, if not in concrete detailed plan, and if all minds should prove to be responsive, surely the boon to humankind would be beyond measure." "Unemployment would disappear from the face of the earth forever, for consider the magnitude of the task of adapting the networks software to all the new generations of computers coming closer and closer upon the heels of their predecessors until the entire population of the world is caught up in an infinite crescendo of on-line interactive debugging."(16) Their vision of an ever growing part of the population of the world being needed to participate in the debugging and development of the network that will make a new world possible is still a helpful vision. Thus I want to invite you all to help to contribute to the spread of the Wonderful World of Usenet news. It is a world that needs and will reward your participation. --------------------------------- Notes for CHAPTER 2 (1) From Ithiel de Sola Pool, Technology Without Boundaries, edited by Eli Noam, Cambridge, Massachusetts, 1990, p. 56. (2) See for example "The Information Technologies and East European Society" by Gary L. Geipel, A. Tomatz Jarmoszko, and Seymour Goodman, in East European Politics and Society, vol 5 no 3, pp. 394-438. (3) Michael Hauben, post November, 1992. (4) E-mail correspondence from Dima (5) Gregory G. Woodbury, "Net Cultural Assumptions," reprinted in Amateur Computerist, vol 6 no 2/3, Fall/Winter 1994-5. (6) E-mail correspondence from Stephen Daniel. (7) From Usenet History Archives, October 12, 1990, ftp://weber.ucsd.edu/pub/usenet.hist/nethist.901012.Z (8) From "Invitation to a General Access Unix Network" by Tom Truscott, Duke University, Durham, North Carolina. (Copy made available by B. Jones.) (9) Copy from Usenet History Archives. ftp://weber.ucsd.edu/pub/usenet.hist/ (10) Usenet History Archives, January 26, 1993. See nethist.930126.Z. Also by 1979-80, UCB was under contract to ARPA to provide a version of Unix (Berkeley Systems Distribution) for the ARPA contractors that were going to be upgraded to VAX computers. (11) E-mail correspondence from Stephen Daniel. (12) E-mail correspondence from Stephen Daniel. (13) One of the most outstanding is told by Amanda Walker who remembered how it was necessary to send an e-mail message across the continent twice using three networks to get it from the Computer Science Department to the Computer Center on the Case Western Reserve University campus. Usenet History Archives, October 16, 1990. See nethist.901016.Z. (14) Gregory G. Woodbury in "Net Cultural Assumptions." (15) From Gene Spafford, Usenet History Archives, nethist.901011.Z from the Mailing List, Gene Spafford, October 11, 1990 based on data from Adams, Spencer, Horton, Bellovin and Reid. Updating these statistics in 1993, David Lawrence estimates that in a 2 week period ending March 9, 1993, about 26,000 articles per day were posted to 4902 groups for 65 total megabytes (52 without headers). (16) In Memoriam: J.C.R. Licklider 1915-1990, Aug. 7, 1990, p. 40, reprinted by Digital Research Center; originally published as "The Computer as a Communication Device," in Science and Technology, April, 1968. -------------------- Special thanks to the Usenet pioneers and to Bruce Jones for the history materials they have gathered and made available. Last Updated: June 5, 1996 =============================================================== This article is a draft chapter from Michael Hauben's and Ronda Hauben's netbook titled "Netizens: On the History and Impact of Usenet and the Internet." *Commercial use is prohibited* Please send us any comments about this draft. Send comments to both hauben@columbia.edu and rh120@columbia.edu.