Frank da Cruz
[email protected]
20 January 2006
Most recent update: Mon Feb 17 08:16:35 2014
Beta.03
2006/01/09File locking reinstated, using the same kind of kernel locks as before. This is not final but it's better than no locking at all; i.e. it's better to protect the mail file from corruption than it is to be locked out of it by an orphaned MM process. If it turns out that no other mail client uses the same locking convention, maybe we can convert to file-based locks, which users can clean up without divine intervention. Also: REMAIL command fixed to not rewrite the From: address to be the remailer's address, and to write the Resent-Date correctly. Also: some files were renamed to allow MM to built on case-insensitive file systems like Mac OS X. Source code: ftp://kermit.columbia.edu/kermit/mm/test/mm-ccmd-0.95-20060110.tar.gz Beta.02
2005/12/07MM's version number took a big leap from 0.91 to 0.95 today after I integrated Nelson Beebe's University of Utah version 0.94 with my version 0.91; 0.91 + 0.94 = 0.95. The benefit is that this version builds not only on Solaris but also Linux (I just tried it) and presumably also all the other platforms (such as Mac OS X, FreeBSD, HP-UX, etc) where Nelson claims it builds and runs. I tested the Solaris and Linux versions with Methods 0 through 2 and both seem to work fine. IF ANYBODY AT COLUMBIA IS INTERESTED IN TESTING IT ON LINUX, MAC OS X, OR OTHER PLATFORMS, PLEASE LET ME KNOW and I'll set up access to the binaries and scripts. Note: as an artifact of today's work, MM no longer locks the mail file; this will be addressed in some fashion in a subsequent edit. Source code: ftp://kermit.columbia.edu/kermit/mm/test/mm-ccmd-0.95-20051208.tar.gz Beta.01
2005/12/04POP interface added. The actual POP interface is external, like movemail, but MM needed a lot of adaptation to the idea that mail is coming from a (possibly slow and overloaded) network server, rather than from a locally visible spool file.
Columbia MM is a text-based email client, developed at Columbia but used at many other sites, that runs on our central Cunix machines. In some ways it is similar to Pine, in others not so much. Columbia MM is an adaptation of the original ARPANET MM program that was used for email on our much-loved DEC-20s (1977-88). Columbia MM was written to give our users the same familiar email client when they migrated from the DEC-20s to Unix; it was the near-universal email client at Columbia until the mid-1990s (IBM mainfame users had another home-grown MM clone, VMM, written as a Wylbur Exec).
At Columbia, many of us have been using MM for 20 years on our central Unix systems without bothering much about the changing landscape. MM uses the traditional Unix-based mail delivery system, in which each user's incoming mail is deposited by the mail server in a "spool file", such as /var/spool/mail/f/d/fdc, which MM checks from time to time. When it finds new messages, it moves them to your own mail file, where you can read them, reply to them, delete them, etc.
MM doesn't follow this model and neither do its users. But the number of diehard MM users is dwindling to the point where it is no longer practical to support the traditional mail delivery method supported by MM alongside the new one demanded by everybody else. The issue came to a head when demand for our mailservers outstripped their capacity, forcing a radical change in mail delivery: a switch to a higher-perfomance multi-server design from Carnegie Mellon University called Cyrus. The new servers and software have been running at Columbia for some months, alongside the traditional mail spool. Each user is on one system or another. Users are moved in groups from the old system to the new one. The cutover is proceeding rapidly and will be complete in early 2006. Once the cutover is complete, the traditional mail spool will be decomissioned.
_________________________
| * | This document was written before the cutover. In fact all this has changed as of late 2006, when the cutover was completed, and I have been using MM exclusively as my email client, served by POP as described below, ever since. |
An interface between MM and the POP server has been written using Columbia's other well-known tool, Kermit, and is presently ready for testing (a new version of Kermit is needed for this). But unfortunately, migration is not completely transparent because (a) POP (like IMAP) requires a login, and (b) POP (like IMAP) can be slower – sometimes a lot slower – than copying a file out of the mail spool.
If the new POP script were simply installed as a replacement for MM's movemail utility, you would have to type your password every single time MM checked for new mail. Furthermore, MM itself could "hang" for minutes at a time as the POP server is being accessed. (Supposedly this will not happen with Cyrus, but all resources are finite and can be overloaded.)
One of the reasons Kermit was chosen as the scripting language is that Kermit can make the kind of secure connection – in this case TLS (Transport Layer Security) – required by the server. The connection is encrypted and should be well protected from any kind of attack. Thus, when you are prompted for a password to be sent to the POP server, your user ID and password are sent on the encrypted channel, not in the clear.
Instructions for testing the new version follow. NOTE: During the development stage, messages are issued from MM and the POP script so we can watch what's happening. Later, these can be removed.
If you have problems or suggestions, send me email at [email protected].
[ C-Kermit Daily Build ] [ Kermit POP Script ] [ New MM Source Code ]
set movemail-path /p/kd/fdc/mm-pop/pop.ksc
touch ~/.mm-use-pop
it doesn't matter what, if anything, is in the .mm-use-pop file. If it is present at all, it tells MM that it will be using a POP server, rather than local spool files.
export PATH=/p/kd/fdc/mm-pop/:$PATH
This directory contains my working copy of C-Kermit, MM, and the two scripts. You can type this command at the shell prompt for temporary effect, or put it in your .bash_profile for permanent effect.
Now, whenever MM would have checked your mail spool, it logs in to the POP server instead.
NOTE: If you change your password after you start MM, you will need to exit from MM, start it again, and give it the new password.
Later, when you are migrated to Cyrus, a ~/.cyrustt file will appear in your home directory as part of the migration process, and MM and the POP script will automatically switch to the Cyrus server. Don't touch this file; if you remove it or rename it, you won't be able to read new mail at all using MM or Pine.
You can adjust the interval at which MM contacts the POP server with MM's SET CHECK-INTERVAL n command (where n is the number of seconds between checks). The default is 300 (5 minutes). As a good network citizen, you shouldn't make it much less than that, but in any case MM will not contact the server more than once in 30 seconds.
Using MM in this mode is not bad at all when the POP server is responsive; otherwise it can be distinctly unpleasant. That's why there's also a Method 2.
By the way, you can also run the pop.ksc script standalone. In that case it will prompt you for your password and then it will download new messages a file in your home directory called .pop_yyyymmdd_secondssincemidnight_pid.txt.
[ View the pop.ksc script ] [ Back Out ]
set movemail-path
That is, give an empty movemail path.
rm -f ~/.mm-use-pop
export PATH=/p/kd/fdc/mm-pop/:$PATH
mailchk.ksc
which prompts for your password when you start it, and then checks the POP server periodically for new mail without your having to touch it again. Leave this window alone until you are ready to log out.
NOTE: If you change your password after you start the mailchk.ksc script, you will need Ctrl-C out of the script, start it again, and give it the new password.
[ View the mailcheck.ksc script ] [ Back Out ]
You don't have to change your PATH; the new MM should work fine with traditional mailspool delivery if you follow these steps. However, in case the test version has a bug and you want to run the regular version of MM instead of the test version, remove the /p/kd/fdc/mm-pop/ directory from your PATH in your ~.bash_profile (if you put it there), or at the shell prompt type (or paste):
export PATH=`echo $PATH | sed -e "s|/p/kd/fdc/mm-pop/:||"`
NOTE: When you log in to the POP server, if you have mail waiting in the spool, the POP server transfers it to its own space. From that moment, the only way to get this new mail is with a POP (or, presumably, IMAP) client.
Mark Crispin, one of the original ARPANET MM authors, has a version of MM called MS. It uses Columbia CCMD (the DEC20-like command parser), but MS itself was written from scratch. It looks like ARPANET MM. Mark used it, along with a companion program called MailManager for the NeXT, for a few years prior to switching to Pine.
MS is simpler than MM; some people might like it for that reason alone (for example, there is no SET command and no options). Others will like it because, as an IMAP client, it won't have any problems with locked mailboxes as we have had with MM. MS uses the same c-client library that Pine uses, so MS and Pine are completely compatible and MS can access all the same types of protocols and mailbox formats that Pine can.
With Mark's help, I have configured and built MS to work at Columbia with our imap.columbia.edu server. I'm not sure yet how the Cyrus cutover will be managed; I'll probably have to change the code to look for ~.cyrustt, as has been done here with Pine and MM. Anyway, as of 5 Dec 2005, you can use MS to send mail and to read and manage your mail on the IMAP server. I have put a copy of MS in the same directory as the test version of MM, so if you followed the instructions above about setting your PATH, you can run it by typing "ms" at the shell prompt. If not, just do this:
export PATH=/p/kd/fdc/mm-pop/:$PATH
or this:
alias "ms=/p/kd/fdc/mm-pop/ms"
There is a short Unix "man page" for MS HERE. Anyway MM users should find it familiar enough. The same prompts, most of the same commands and options, the same ?-help and Tab-completion features, etc. But using IMAP itself is a bit different. To access your IMAP mailbox (the parts you type are shown in red):
MS>get {imap.columbia.edu/norsh}INBOX
[Trying IP address [128.59.48.1]]
{imap-vif.cc.columbia.edu/imap} username: fdc
Password:
[[UNSEEN 1] first unseen message in mbox]
Mailbox: {imap-vif.cc.columbia.edu:143/imap/tls/user="fdc"}INBOX,
2 messages, 0 recent
MS>
MS>hEADERS (MESSAGES) unsEEN
U 1) 5-Dec [email protected] Fantastic Bargains! (1650 chars)
U 2) 5-Dec [email protected] Hot Stock Tips! (5995 chars)
MS>read (MESSAGES) NEW (This doesn't read them)
MS>read (MESSAGES) unsEEN (This does)
The GET command connects you to your IMAP inbox. Only give this command
once per session. After authenticating, use the CHECK command to check for
new mail. If you don't access the IMAP server every XXX minutes or less,
it logs you out. You can give another GET command to re-login (in this case
you can just type "get" without the arguments).
[ MS Man Page ] [ IMAP Information Center ] [ C-Client Source Code ] [ MS Source Code ]
http://www.columbia.edu/acis/history/hermit.html
Work on MM by approximately the same crew continued after Hermit was terminated because it had become Columbia's primary email client. Version 0.90 (1990) added various user-friendliness features designed by a committee of bigwigs ("novice" mode, ?-help categorized by topic, etc), and saw heavy service for some years but was gradually overtaken by Pine, and then later by PC- and Web-based mail. Y2K and Y2K1 patches were added in 1999-2001. MM is now used mainly by touch-typing menu-hating die-hards like me, for reasons such as those elaborated here:
http://www.columbia.edu/kermit/safe.html
After the patches came numerous bug fixes and file reorganization from Ken Harrenstein, including new simplified makefiles and conversion of many files to the "new" (1990) OS-dependent (or feature-dependent) structure that started in version 0.90. Ken's changes are documented in HERE. I carried this forward a bit in getting the new version to build on Linux (with help from Chiaki Ishikawa <[email protected]> and Adam Sampson <[email protected]>, so now the Makefile automatically links config.h to the right OS-dependent config.h-blah file. (NOTE: In Beta.02 the mm and ccmd makefiles were replaced by new ones from Nelson Beebe.)
The main "functional" bug fix to 0.91 is a lot of code to check for, protect against, and allow recovery from truncated outgoing messages upon return from the editor. Users will notice that .mm-xxx temporary files are no longer deleted immediately upon return from EMACS or whatever your editor is, so if "display" should reveal that some or all of the outgoing message text is missing, you can rescue it from the .mm-outgoing-pid file, whose name is announced to you each time you return from the editor. This feature is enabled only in Solaris and SunOS, because those are the only platforms where the problem was observed, but can be compiled in to any other platform too (-DFDC_EDITFIX). In any platform that has this feature, the new command:
SET USER-TEMP-FILES-KEEP { ON, OFF }
can be used to turn it on and off.
Columbia MM change summary.
[ Ken Harrenstein's update notes ] [ My update notes ] [ Nelson Beebe's notes ] [ MM and CCMD Source Code ]
I use MM on a Unix host through a Telnet or SSH session (or other kind of connection) from Kermit 95 on Windows, which understands nearly 100 different character sets: the ISO 8859 Latin Alphabets, the ISO 646 7-bit national sets, various Cyrillic KOI encodings, numerous PC code pages and other proprietary sets such as HP-Roman8 and DG International, as well as Unicode UTF-8, the latter including thousands of characters in dozens of scripts (Latin, Greek, Cyrillic, Hebrew, Japanese, you name it – even Runes).
A properly formatted e-mail message identifies its text character set in a header like this:
Content-Type: text/plain; charset=xxx
where xxx is a MIME standard character-set name, such as ISO-8859-1 or UTF-8. If a message arrives and looks like garbage, you can change Kermit's character set to match (you can do this on the K95 2.0 toolbar) and redisplay the message.
When using Kermit to access MM, you should be aware that Kermit's character-set names are not always the same as the names shown in the Content-Type header (because Kermit predates MIME). For example:
NOTE: C-Kermit 9.0 accepts the MIME names, and the next release of Kermit 95 will do so as well.
MIME Name Kermit Name Remarks ISO-8859-1 Latin1 West European Latin ISO-8859-2 Latin2 East European Latin ISO-8859-3 Latin3 Mainly for Turkish ISO-8859-4 Latin4 Mainly for Sami ISO-8859-5 Cyrillic-ISO Cyrillic + ASCII ISO-8859-6 Arabic-ISO Arabic + ASCII ISO-8859-7 Greek-ISO Greek + ASCII ISO-8859-8 Hebrew-ISO Hebrew + ASCII ISO-8859-9 Latin5 ISO 8859-9 is Latin Alphabet 5 ISO-8859-15 Latin9 ISO 8859-15 is Latin Alphabet 9 (has Euro) UTF-8 UTF8 Unicode UTF-8
Luckily if you tell MM to:
set use-editor-always true
the command parser is bypassed and the 8th bit is not stripped. If you are replying to a message and including the original message, its 8-bit characters are preserved. You can enter 8-bit characters yourself using any method allowed by your editor and/or terminal emulator (e.g. Kermit's Compose key). Then before sending the message, you should tell MM to include the headers that identify the character set. This requires a new command that was added in MM 0.91:
content-type xxx
where xxx is a MIME content type (the default, if you give a CONTENT-TYPE command without an argument, is "text/plain; charset=ISO-8859-1"). When you give this command, MM writes out a full Content-Type header containing the text you supplied, e.g.:
Content-Type: text/plain; charset=UTF-8
When you give a CONTENT-TYPE command, MM also inserts two other headers that are required by the recipient to properly interpret the message:
MIME-Version: 1.0 Content-Transfer-Encoding: 8bit
Should it be necessary for you to specify other values for these headings, you can do so with the following commands (also added in MM 0.91):
mime-version text
transfer-encoding text
The MIME-Version value would normally be "1.0" and the Transfer-Encoding value should be something like "8bit" or "7bit", but no verification is done.
If you send a lot of mail in a non-ASCII character set such as ISO 8859-1 or UTF-8, you can have MM include the needed headers in every outgoing message automatically by creating a file containing the desired headers and then telling MM to SET HEADER-OPTIONS-FILE filespec. For example, create the file .mmheaders in your login directory, put the following lines in it:
MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit
and then put the following command in your .mminit file:
set header-options-file ~/.mmheaders
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <html><head><title>Blah blah</title> <META http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> </head><body>(Text goes here with any desired markup)</body> </html>
(In this example I'm assuming you're including 8-bit characters from ISO 8859-1; if you are using some other character set that is not ASCII, you should substitute its MIME name.) Then tell the content type -- html encoded in such-and-such a character set:
content-type text/html; charset=ISO-8859-1
The content-type command automatically adds the following headers:
MIME-Version: 1.0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit
Or (what I usually do)... Just keep an HTML message template called, say, msg.eml that you can edit any time you want to send an HTML message. Here's a sample template. Note the blank line separating the headers from the message body; the blank line is essential:
To: fill this in Subject: fill this in bcc: . MIME-Version: 1.0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <html><head><title> fill this in - same as Subject </title> <META http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> </head><body> Text goes here with any desired markup </body> </html>
Edit the file into the desired message and then tell MM to:
restore-draft msg.eml
For reading HTML mail, see the next section.
To send email with attachments, use Pine (or any other modern mail client).
To decode an incoming mail with attachments, use MM's COPY command to save the message to a file, then run the file through Metamail. Or download the message using MM's DOWNLOAD command (next section) and look at it in your desktop mail client. Windows users should give the downloaded message a filename that ends with .eml, which is associated with the Windows email reader. The default filename for the DOWNLOAD command is mm.eml. If your terminal emulator is Kermit 95, escape back to it (Alt-x) and tell it to "run start name.eml". a handy macro to have defined in K95 would be:
define z run start \v(filename)(Kermit's \v(filename) variable contains the name of the file most recently transferred).
set print-filter metamail | lynx -stdin -dump | moreThat is, run the message through metamail and pipe the result the lynx (the text-mode Web client), and to the 'more' pager.
Version 0.91 of MM has the following new commands for downloading messages. They work exactly like MM's PRINT command, but they download rather than print.
kermit -Ts - -a %s
that is, send standard input (which is fed to Kermit by MM) in text mode under the name given by the replacement string; the default replacement string is "mm.eml".
The download program must be able to send from standard input (as Kermit does with its "-s -" command-line option). XYZMODEM programs (e.g. Omen rzsz or GNU lrzsz) don't seem to allow this, but you can work around them as follows:
set download-filter cat > %s; sz -a %s; rm %s
You can have up to four %s's in the download-filter; each one is replaced by the same string.
You should be able to include almost any printable character except Space in the filename; however, backslashes must be doubled:
DOWNLOAD /FILE:c:\\some_directory\\some_subdirectory\\some_file.txt
But note that when downloading to Kermit 95, you can use forward slash as the directory separator to avoid the ugliness:
DOWNLOAD /FILE:c:/some_directory/some_subdirectory/some_file.txt
When accessing MM from any reasonably modern version of Kermit (such as Kermit 95 on Windows or C-Kermit 7.0 or later on Unix), Kermit's "autodownload" feature kicks in (yes) automatically – as soon as the MM's download-filter program starts sending with either Kermit or Zmodem protocol, the terminal program switches to file-reception mode, and then back to terminal mode when the transfer is complete. Thus the DOWNLOAD command does the entire job – no "escaping back" or re-connecting is needed.
If you are accessing MM from Kermit 95 on Windows, you can download MIME messages that contain enclosures directly into your Windows mail agent. This can be dangerous unless you know what you're doing and have inspected the message carefully in MM to ensure its payload is not lethal (further explanation is beyond the scope of this document). Tell MM to:
DOWNLOAD /FILE:xxx.eml
Then escape back to the K-95> prompt and tell K95 to:
RUN START xxx.eml
This tells Windows to start whatever program is associated with the .EML (e-mail) extension (Outlook, Netscape, whatever) and feed the xxx.eml file to it so you can experience it in all its multimedia glory. The fact that you saw it in MM first allowed you to prescreen it for safety, an opportunity you would not have if you used a PC-based mail client to begin with. HINT: Define a macro in Kermit 95:
define z run start \v(filename)
and then just type "z" at the K-95> prompt to read the message. In fact this works with any kind of file that you download, for which Windows has an associated application (.xls, .doc, .pdf, etc). \v(filename) is a built-in Kermit variable that contains the pathname of the file most recently downloaded.
There is even a way to have Kermit 95, upon receiving a file, automatically start the associated application with the just-received file loaded into it (email, HTML, a JPG image, an MS Word file, a PDF file for Acrobat Viewer, an Excel spreadsheet, etc). Obviously this would be risky so I won't explain it unless somebody asks me.