Hermit Project

The Hermit Project was a collaboration between Columbia University and Digital Equipment Corporation (DEC), similar to some of the larger projects that DEC had with other universities such as MIT and CMU. It was active from 1984 to 1987. Columbia was chosen on the strength of its Kermit protocol and software work as well as its close ties with DEC. Columbia received equipment valued at $3,000,000.00, including four minicomputers, thirty workstations, various impact and laser printers, tape drives, a high-speed network and storage for our central mainframes, and Columbia's first high-speed backbone network, as well as an on-site DEC engineer. The goal was a distributed computing environment with centralized identities and file storage on both IBM and DEC mainframes, supporting IBM, DEC, and Apple workstations on the desktop (DOS, Unix, and Mac OS), able to operate over diverse communication media ranging from serial ports to Ethernet.

- fdc, August 2011

HERMIT PROJECT PROGRESS REPORT

Frank da Cruz
Principal Investigator

23 January 1987

Here is a summary of the components and capabilities of Hermit, and the current status of each component. There are three tiers to the Hermit architecture: the back end (our central systems), the cluster server (an intermediate MicroVAX Unix system), and personal computers. The cluster servers are attached to the University backbone Ethernet cable and communicate with the central systems using session, presentation, and application protocols built upon TCP transport protocols.

BACK END: The user's real identity resides on a "back end" -- a central system. So far, only the DEC-20s serve this purpose.

File server. Gets user's identity from a login message from the cluster, verifies it on central system, feeds user's files to cluster, locks them against simultaneous updating, maintains database of users' file status, receives and applies updates from cluster. Status: DONE.
Telnet server. Standard TCP/IP virtual terminal service. Status: AVAILABLE.
Print server. Honors or routes network printing requests. Can print on shared network printers, but cannot route jobs to other PC-attached printers. Status: For now, regular host-based print spoolers are used, allowing access to all network printers. So far, no access or limit checking.
Name server. When there is more than one backend host file system, the name server will find out which one the user's home is on. Status: NOT DONE. The idea is to allow not only multiple DEC-20 hosts, but also UNIX and VM/CMS (and possibly VAX/VMS) systems to serve as hosts simultaneously.
Accountant. Keeps records & statistics of Hermit accesses. Status: Sort of done (some information is kept, but not used by accounting system).
Mail server. Provides mail messages to cluster user, based on various selection criteria, allowing messages to be edited, forwarded, etc, from cluster, then applies updates. Status: NOT DONE.
Finger server. Provides finger listings, or forwards to selected host. Status: DONE.
Message server. Routes terminal messages to selected user. Status: DONE.
Library info. The host's interface into CLIO. Requires that CLIO become a host-accessible network service, in addition to a terminal query system. Status: NOT DONE.
Communications. Standard TCP/IP Ethernet services. Status: AVAILABLE.

CLUSTER SERVER: The cluster server has "two faces". With one face, it provides services to the users who come in from the PCs; with the other, it relays information or requests for service to the back end.

File/disk service. The cluster gets files from the back end, reformats them according to the user PC's operating system, and provides virtual disk service to the PC. Upon termination of a session, the user's disk is converted back into back-end files and sent back to the central file server. Status: DONE for MS-DOS format, mostly done for Macintosh and Pro/380.
Mail service. Analogous to file/disk service, but handles mail messages rather than files and disks. Also allows the user to send mail directly from the cluster. Status: Partly done. Talks to client program through Unix pipes rather than over network, does not yet check central mail files out from and into the mainframe.
Telnet service. Acts more or less as a transparent pipe to the virtual terminal service of the back end. Status: DONE.
Print service. Prints user's files on local cluster printers or else routes them to the back end for printing on any selected printer. Status: DONE.
Finger service. Shows who's connected to the cluster, or else passes the request along to the named host and relays the results to the user. Status: NOT DONE.
Message server. Sends messages within the cluster, or routes them to the central system. Status: NOT DONE.
Library server. Go-between for PC & CLIO network service. Status: NOT DONE, not even designed yet.
Communications. Layered according to ISO OSI model (the parts of it which don't conflict with TCP/IP), to allow Hermit code to fit in any layer where no existing service (like TCP/Ethernet) is available. Datalink and transport layers developed locally on top of RS-232, to allow MS-DOS PCs take part in Hermit network services as peers with Ethernet PCs. Status: RS-232 datalink and transport DONE, but not debugged. Layering on top of Ethernet UDP done.
Bidding. When several clusters serve the same local network, they may bid among themselves to serve a new client PC when asks to establish a new session. Status: NOT DONE.

PCs -- MS-DOS (Rainbow, PC/AT; Macintosh & Unix workstations come later)

The Login program allows the user to establish and shut down a Hermit session. Status: DONE.
Disk driver. The standard disk device driver is replaced by a Hermit disk driver, which fetches and updates disk sectors over the communication medium, transparently to the system or application. Status: DONE.
Print driver. The printer driver is replaced by a Hermit driver, to allow routing of print requests to shared cluster-wide or site-wide printers, transparently to the system or application. Status: DONE.
Telnet. Virtual terminal program makes a connection with the Telnet server on the cluster level, which in turn makes a virtual terminal connection with the specified host computer. Status: DONE, but no particular terminal is emulated, except on the Rainbow which has built-in VT102 firmware. Performance is unavoidably jerky.
Mail. A full-blown MM-like mail program, providing message read, send, forwarding, and management functions, through interaction with the Hermit mail server. Status: protocol is designed, cluster-to-PC portion is written. PC-based client program currently runs on VAX, needs to be ported to MS-DOS.
Library service. The user interface to CLIO network service. Status: NOT DONE.
Finger, Message. Simple programs that send requests to the cluster and display the answers on the screen. Status: NOT DONE.

Implementation:

C language used wherever possible.
All PC-based application programs (except on Mac) have DEC20-like user interface, ccmd, coded portably in C, freely movable between VAX/UNIX, Pro/380/UNIX, MS-DOS, and possibly even VAX/VMS.
MS-DOS Hermit service currently running and being debugged as standalone monolithic application. Once debugged, it will be picked apart into its component pieces: a clock-interrupt-driver scheduler and queue manager, terminate-and-stay-resident disk and print drivers.

Brief Synopsis of Where Things Stand:

DEC-20 back end file service is complete.
Cluster interaction with DEC-20 file server complete.
Cluster disk service for MS-DOS is complete.
Most MS-DOS applications are complete, but some of the underlying communication software (notably the Hermit session layer) has persistent bugs that prevent the applications from working.
Once MS-DOS services are working, we'll move to the Pro/380s, then to the Macs.

Why the Delays?

C-language portability problems. Code developed under Unix won't compile under DOS because of compiler differences. Much time wasted evaluating different MS-DOS C compilers, ordering new releases, etc.
Poor program development utilities. The 'make' programs available for MS-DOS are far inferior to the Unix version. Much time wasted fighting DOS 'make' limitations, ordering and converting to different 'make' programs. Similarly for the 'm4' macro processor.
Lack of interactive C-language-oriented debugging tools for interrupt-level software in DOS. Debugging is a prolonged, tedious, frustrating process.
Inherent limitations of working in single-process PC environment: long programs slow to compile, difficult to code (e.g. multiprocessing, which is a built-in feature of systems like Unix, must be painfully simulated by interrupt-level programming, in which many common functions are not allowed). The difficulty and tediousness of this sort of programming was underestimated -- we've been spoiled by programming "real computers".
Difficulties with vendors of commercial Unix systems for Pro/350/380. Pro/350 to Pro/380 conversion. Venix to 2.9BSD conversion. Adaptation (done locally) of 2.9BSD to Pro/380.
VAX 8650 installation and development has been taking a lot of Hermit people's time.
Vacations.
History: repeated personnel turnover, both at CU and at DEC. Multiple redesign cycles as cast of characters and technology changed.
Resignation of Delores, who was was doing the mail software.

Prognosis:

A March 1, 1987, deadline has been set for getting the bugs out of what we have so far, to produce a working Hermit environment encompassing the DEC-20, the intermediate VAX cluster server, and MS-DOS machines -- IBM PC's and DEC Rainbows, providing file, print, and virtual terminal service.

DIAGRAM:


HERMIT PROJECT COMPONENTS AND STATUS, SEPTEMBER 1986
                                                                Legend:

+----------------------------------------------------------+    (1) Done by us, debugged (4) Partly done (7) Easy to do     
|                    Common File System (8)                |    (2) Done, being debugged (5) Designed    (8) Existing service
+----------------------+-------+-------+-------------------+    (3) Almost done          (6) Done once   (9) Not designed
                       |       |       |
 DEC-20s               A       B       D                               VAX/UNIX hosts                VM/CMS hosts
+-----------------------------------------------------------+         +-------------------+         +-------------------+
| file    mail    telnet  print   finger   message  library |         | name     other    |         | CLIO     other    |
| server  server  server  server  server   server   info    |         | server   services |         | server   services |
|  (1)     (5)     (8)     (8)     (8)      (8)      (9)    |         |  (9)      (1-9)   |         |  (9)      (1-9)   |
+-----------------------------------------------------------+         +-------------------+         +-------------------+
                       A       B       D                                       C                          A       B
                       |       |       |                                       |                          |       |
Backbone Ethernet      |       |       |                                       |                          |       |                
-----------------------+-------+-------+-----------+---------------------------+--------------------------+----+--+----------
                                                   |                                                           |   
                    MicroVAX Cluster Server        |                                               MicroVAX Cluster Server
                   +-------------------------------+---------------------------------+           +--------------------------+   
Communications     |  TCP/IP/Ethernet (8)                                            |           |                          |   
                   +-----------------------------------------------------------------+           |                          |   
Back End Client    |  (8)    (5)    (8)    (8)      (7)       (1)     (9)            |           |                          |  
Application        | telnet  mail  print  finger  message  file/disk  CLIO   bid (9) | < - - - > |  bid (9)                 |  
PC Server          |  (1)    (1)    (1)    (1)      (1)       (*)     (9)            |           |                          |   
                   +-----------------------------------------------------------------+           |                          |   
Session Layer      | Hermit (2)                                                      |           |                          |   
                   +-----------------------------------------------------------------+           +--------------------------+   
Datalink/          | TCP (8)            :                                            |                                            
Transport Layer    |                    : - - - - - - - - - - - + Hermit (1)         |                                        
                   |                    :         LAT (8)       :                    |
                   +--------------------------------------------+--------------------+                                           
Physical Layer     | Ethernet (8)                               | RS-232 (1)         |
                   +--------------------+-----------------------+------------+--+--+-+
                                        |                                    |  |  |
Local Ethernet                          |                                    |  |
---------------+------------------+-----+------------------+-------          |
               |                  |                        |                 |  * Disk service:
               |            +-----+--------+     +---------+---------+       |    MS-DOS            (1)
               |            | Kinetics (8) |     | DEC LAT Box (8)   |       |    Pro/380 2.9BSD    (6)
               |            +-----+--------+     +---+---+---+---+---+       |    Macintosh         (6)
 Pro/380       |                  | Appletalk        |   |   |   | RS-232    |
 2.9BSD        |            --+---+--+--+------      |                       |
 (1)           |              |      |  |            | Rainbows or PC/ATs    |  MS-DOS
+--------------+--------+   +-+---+           +------+-----------------------+----------------------+
| Using standard TCP/IP |   | Mac |           | RS-232 (1)                                          | Physical Layer
| applications for now  |   | (6) |           +-----------------------------------------------------+
| (ditto for VS2)  (8)  |   |     |           | Hermit transport (1)                                | Datalink/Transport
| Hermit disk driver(6) |   +-----+           +-----------------------------------------------------+
+-----------------------+                     | Hermit session (2)                                  | Session Layer
                                              +-----------------------------------------------------+
Tools:                                        | login  print   mail  telnet  finger message   disk  | Application Layer
  ccmd (MS-DOS, Unix) (1)                     |  (1)    (1)    (1)    (1)     (1)     (7)     (1)   |
  TCP/IP interface functions (Unix) (1)       +-----------------------------------------------------+

Epilog (Jan 2001)

(Not long afterwards we filled in all the pieces, including the Macintosh and PC clients. We gave a series of successful demos to the funding bodies at Digital Equipment Corporation, including transparent file service, terminal sessions, sending and receiving mail and terminal messages, pressing "print screen" keys on the Mac or PC and having graphics images come out on the shared dot-matrix Printronix printer that was connected to the MicroVAX cluster server, etc. But by this time the DEC-20 line had been canceled and all our work was made obsolete by NFS and similar products. Out of the Hermit Project we got our first campus backbone Ethernet network, our first PC lab, and our UNIX version of the MM e-mail client, which was to serve us and others well for many years to come.)

Frank da Cruz / fdc@columbia.edu / Jan 2001