The Columbia Crown The Kermit Project | Columbia University
612 West 115th Street, New York NY 10025 USA • kermit@columbia.edu
…since 1981
Table of platforms   Book: Using C-Kermit   Download C-Kermit 9.0 Change Log  

Last C-Kermit 9.0 Beta Test
Platform-independent, transport-independent, scriptable communication software

Frank da Cruz
Columbia University
fdc@columbia.edu
  Current test level: 9.0.299 Beta.02
Date: 21 June 2011
This page updated: Tue Jun 28 09:57:39 2011 (Eastern USA time)

Work on C-Kermit has continued, on and off, since the release of C-Kermit 8.0.211 on 10 April 2004. After that, the working version was called 8.0.212, but it will be released as C-Kermit 9.0, which entered Beta test June 15th. The major goals of this release are (a) compatibility with newer OS releases and hardware; (b) support for large files and 64-bit integers on as many platforms as possible; (c) stability; and (d) some VMS-specific improvements. Other new features are listed below. Perhaps the biggest change of all is a...

NEW LICENSE
Beginning in Beta.01 C-Kermit has the Open Source Revised 3-Clause BSD License, as will subsequent Beta tests and the final C-Kermit 9.0 and later releases.
CLICK HERE to access and download the latest build.

Thanks to (at least) Jeff Altman, William Bader, Ian Beckwith, Nelson Beebe, Gerry Belanger, Joop Boonen, Rob Brown, Christian Corti, John Dunlap, Peter Eichhorn, Carl Friedberg, Terry Kennedy, Günter Knauf, Jason Lehr, Arthur Marsh, Lewis McCarthy, Gary Mills, Jonathan Reams, Mike Rechtman, Mark Sapiro, Steven Schweda (SMS), Kinjal Shah, Michael Sokolov, Andy Tanenbaum, Seth Theriault, Zach A. Thomas, Martin Vorländer, and Eric Weaver for assistance, and to Hewlett-Packard Company for support.

Contents

What's New in General

Very briefly, the major items:

What's New in Beta.02

Not much, just minor changes to get it to build on lots more platforms, and verification for many others that hadn't been tried yet. Now the table lists over 130 successful Beta (β) builds. Much work was done on the linux, linux+krb5, linux+ssl, and linux+krb5+ssl targets to get the best possible (or feasible) results on various Debian, Gentoo, SuSE, Slackware, and Ubuntu distributions. Thanks especially to Ian Beckwith (Debian), Peter Eichhorn (HP-UX), and JP Radley (SCO).

What's New in Beta.01

As of C-Kermit 9.0 Beta.01, C-Kermit is Open Source software with the Simplified 3-Clause BSD license. Copyright notices, the COPYRIGHT command, and the COPYING.TXT file have the new copyright text. New features in Beta.01 include:

I've built it on a wide range of platforms spanning 25 years. You can see the list here. Platforms listed β01 or later in the Last column; there are more than 130 of them, from SCO XENIX 2.3.4 to RHEL5.6.

As noted below, SSL/TLS and Kerberos builds are increasingly problematic with the “deprecation” of DES. There is code to detect the presence or absence of DES in the OpenSSL builds, but it doesn't always work because sometimes the SSL libraries are present but routines are missing from them.

Also note that Kerberos support is for the MIT version only, Heimdal and others are not supported (never have been). Of course anybody can pitch in and add or improve support for whatever they want.

What's New in Alpha.09

Meanwhile I've been getting complaints that C-Kermit's "ssh" command doesn't work in various Linux distributions. Turns out that some of them, reportedly including Ubuntu and Centos, have disabled C-Kermit's pseudoterminal support. Solution: get C-Kermit from here and build it yourself with "make linux" (or "make linux+ssl", etc). .

What's New in Alpha.08

What's New in Alpha.07

This is a minor test release, mostly bug fixes:

What's New in Alpha.06

New ability to read and parse Comma-Separated-Value (CSV) files, described HERE.

What's New in Alpha.05

  1. New built-in functions:
    \fcvtcsets(string,cs1,cs2)
    Function to convert a string from one character set to another.
    \fdecodehex(string[,prefix])
    Function to decode a string containing hex escapes.
    \fstringtype(string)
    Function to tell whether a string is 7-bit, 8-bit, or UTF-8.

    For the motivation for these features and an application that uses them to analyze web logs, see the Weblog script below.

  2. MIME Character-Set Names: A new equivalence between MIME names and Kermit names for character sets, with a new table showing the supported sets HERE (this feature is also illustrated in the Weblog script).

  3. Lazy IF Conditions: Third, now you can do this:
    define foo some number
    if foo command

    instead of this:
    define foo some number
    if \m(foo) command
    Of course the old way still works too. But watch out because if the variable name is the same as a symbolic IF condition (for example COUNT), it won't do what you expected. (IF COUNT was used for loop control in early versions of MS-DOS Kermit, before it got true FOR and WHILE loops; it was added to C-Kermit for compatibility, and it can't be removed because it could break existing scripts).

New Demonstrations for Alpha.05 - Webmaster Tools

The Weblog Script
Reads a web log, extracts the Google searches, normalizes the search strings, and prints the top 20 searches, along with their counts. Documented HERE.
The Amazon Script
Reads an Amazon Associate orders report and lists the products according to the number of orders for each, or the number of clicks on each.
Both of the scripts illustrate (among other things) how to read Tab-Separated-Value (TSV) files.

What's New in Alpha.03 and Alpha.04

General improvements and platform-independent fixes: For UNIX: For VMS: In the VMS build procedure, CKVKER.COM, the "i" option in P1 now means don't include the internal FTP client, and the "f" option means do not include "Large File" support. Large File support in VMS really only applies to the file-transfer display and statistics, which would go out of whack as soon as the byte count overflowed 31 bits because this is C-Kermit, built with the C compiler and the C library (runtime system), which did not support long integers until VMS 7.3.

Special thanks to Steven M Schweda, Martin Vorländer, and Rob Brown for most of the VMS improvements in Alpha.03.

Demonstration: HP Switch Configuration Backup

A common use for Kermit software is to make automated backups of the configuration of network switches and routers, such as those made by Cisco or Hewlett-Packard (although tftp can be used for this, it is not available in all such devices; Kermit, however, works with those that have tftp as well as those that don't).

Typically a backup can be done by making a Telnet, SSH, or serial connection to the device with Kermit and giving a command such as "show config" at the command-line prompt of the device with Kermit's session log activated. The result is a list of the commands that were used to establish the current configuration, suitable for feeding back to the device's console (e.g. with C-Kermit's TRANSMIT command) to reestablish the same configuration or to duplicate it on another device.

At an HP installation it was noted, however, that while the HP switches (various ProCurve models) produced the desired list of commands, they were interspersed with escape sequences for special effects, thus rendering the recorded sessions unsuitable for feeding back into the switches.

C-Kermit 9.0 introduces a new feature to strip the offending sequences out of a session log, leaving just the text. The command SET SESSION-LOG TEXT activates this feature. In C-Kermit 9.0 Alpha.02 and earlier, escape sequence stripping occurred only while logging interactive (CONNECT) sessions; beginning with Alpha.03 it is done also for data that is read by INPUT commands and therefore works for scripts too.

A sample HP Switch Configuration Backup script is HERE, and its data file is HERE. This script also illustrates some other new features of Alpha.03:

MESSAGE text
This lets you put debugging messages in your script that can be displayed or not, according to SET DEBUG MESSAGE (below). This way you don't have to change your script for debugging.  Hint:  In Unix, invoke the script like this:
$ DEBUG=1 scriptname arg1 arg2...
and then include the following command in your script:
if defined \$(DEBUG) set debug message on
XMESSAGE text
Like MESSAGE but prints the text with no line terminator, so it can be continued by subsequent messages.
SET DEBUG MESSAGE { ON, OFF, STDERR }
ON means MESSAGE commands should print to standard output; OFF means they shouldn't print anything; STDERR means the messages should be printed to stderr. DEBUG MESSAGE is OFF by default, i.e. unless you SET it to ON or STDERR.
IF DEBUG command
Executes the command if SET DEBUG MESSAGE is not OFF.
The \v(lastcommand) variable
This variable contains the previous command. You can use it in debugging and error message to show (for example) exactly what the command was that just failed, without having to make a copy of the command:
set host somehost.somecompany.com
if fail exit 1 "FATAL - \v(lastcommand)"
which, if the SET HOST command fails, prints "FATAL - set host somehost.somecompany.com" and then exits with status 1 (which normally indicates failure).

Demonstration: HP iLO Blade Configuration

THIS DOCUMENT describes a script in production use at Columbia University for configuring and deploying racks full of HP blade servers through their "integrated Lights Out" (iLO) management interface, bypassing the tedious and error-prone process of configuring the servers one by one through the vendor-provided point-and-click Web interface, which is ill-suited to configuring large numbers of blades. The script illustrates some of C-Kermit 9.0's new features; source code is available through the link. The code is apt to change from time to time as new requirements surface.

Demonstration: IBM/Rolm/Siemens CBX Management

THIS DOCUMENT describes a suite of scripts (some in production, some in development) used to manage the Columbia campus 20,000-line main telephone switch, along with about 10 satellite switches at off-campus locations. These switches are 1980s technology*, their management consoles are serial ports. Access is via Telnet to reverse terminal servers. The scripts allow for interactive sessions as well as automatic production (and in some cases formatting) of different reports required by different groups at different intervals. These scripts replace a whole assortment of ad-hoc ProComm ASPECT scripts that were scattered all over the place, with passwords embedded. The new scripts are intended to be run from a centralized server where there is a single well-secured configuration file, and where they can be used on demand, or in cron jobs. They are modular so code duplication is minimal.
__________________________
Of course the University is deploying new technology but the but the old system will be used in parallel for some time to come.

Platforms

So far C-Kermit 9.0 has been built and tested on Linux, Mac OS X, FreeBSD, NetBSD, OpenBSD, MirBSD, HP-UX, AIX, MINIX, SCO OpenServer, SGI IRIX, SunOS and Solaris, OSF/1 (Tru64 Unix), and DEC/Compaq/HP (Open)VMS, on a variety of 32-bit and 64-bit hardware. The builds done so far on the current code base, C-Kermit 9.0.299 Alpha.01 and later, are shown in Table 1. See the table for details of each build — OS and version, architecture, word size, build procedure, resulting binary size, and any notes.

Large Files

Kermit is, first and foremost, a file-transfer program. One might expect it to be able to transfer any kind of file, but that has been decreasingly the case as file sizes began to cross the 2 gigabyte threshold.

The biggest change since C-Kermit 8.0.211 is support for large files on platforms that support them. A "large file" is one whose size is greater than 231-1 (2,147,483,647) bytes (2GB-1); that is, one whose size requires more than 31 bits to represent. Before now, Kermit was able to access such files only on 100% 64-bit platforms such as Digital Unix, later known as Tru64 Unix. In the new release, Kermit takes advantage of the X/Open Single UNIX Specification Version 2 (UNIX 98) Large File Support (LFS) specification, which allows 32-bit platforms to create, access, and manage files larger than 2GB.

Accommodating large files required code changes in many modules, affecting not only file transfer, but also file management functions from directory listings to local file manipulation, plus the user interface itself to allow entry and display of large numbers. All this had to be done in a way that would not affect pure 32-bit builds on platforms that do not support large files. Large file support is summarized in the Table of Platforms; entries in Yellow (32-bit builds that support 64-bit integers) and Green (64-bit builds) support large files.

Note that VMS C-Kermit and Kermit 95 for Windows have always been able to transfer large files. However their user interface used 32-bit integers for statistics and the file transfer display. In C-Kermit 9.0 Alpha.03, VMS C-Kermit on 64-bit platforms (Alpha and Itanium) should now give correct statistics and progress displays. (We'll see about Kermit 95 later.)

How to Test Large-File Transfer

Several methods are available for testing large-file transfers:

Arithmetic with Large Integers

Because large file support requires the availability of a 64-bit signed integer data type, other aspects of C-Kermit were adapted to use it too, most notably Kermit's algebraic expression evaluator and its S-Expression interpreter, on all platforms that support large files (those listed as 64 or 32/64 in the Word column of the table). In fact, every Kermit command that parses a number in any field can now parse a large number on those platforms.

S-Expressions can now be forced to operate with integers only, without floating-point conversion or having to explicitly truncate each result; as an example. see the revised Easter date calculation script.

Variable Evaluation

Does the strange behavior of Kermit's \%x variables puzzle or annoy you?

Kermit software development has been a collaborative project over the years, with contributions coming in from almost every country and every sector of the economy – academic, corporate, government. Thus not all versions, and not all features of a given version, are a product of systematic design.

One example was the introduction of variables for text substitution, first in a version of MS-DOS Kermit that was sent in by someone somewhere (I could look it up, but no time...) Although the design of the notation for variable names (table below) is mine, the underlying code was contributed. In that code there was only one kind of variable, and if I recall correctly the variable name was a backslash followed by a single letter, for example \a, \b, etc. The contributed code evaluated these variables recursively, meaning if the definition of a variable contained variable references, then these were resolved when dereferencing the variable, and the process would continue as deep down as necessary to resolve the thing fully.

This was sometimes handy, but it had one severe drawback: There was no way to use variables in a straightforward way to represent strings that contained literal backslashes; for example, DOS or Windows pathnames. This gave rise to all kinds of quoting rules and conventions (e.g. doubling backslashes or forcing single-level evaluation with \\fcontents()), and also to the introduction of other kinds of variables that were evaluated one level deep, rather than recursively.

To accommodate coexistence of different kinds of variables as well as "escape sequences" for representing control and 8-bit characters, the syntax for variable names was extended to include three elements: the leading backslash, then a single character indicating the type of variable, and then the name of the variable in a format corresponding to the type designator, as shown in this somewhat simplified table:

Table 1. Variable-name Syntax in Kermit
Notation Meaning
\000 – \255 8-bit character constant (decimal)
\d000 – \d255 Alternative notation for 8-bit character (byte) constant (decimal)
\o000 – \o377 8-bit character constant (octal)
\x00 – \xff 8-bit character constant (hexadecimal)
\%a – \%z Scalar variable, evaluated recursively.
\%0 – \%9 Macro argument, scalar, evaluated recursively.
\&a – \%& Array name
\&a[x] Array reference, evaluated recursively (x is any constant or variable)
\v(name) Built-in scalar variable, evaluated one level deep.
\m(name) User-defined scalar variable, evaluated one level deep.
\$(name) An environment variable, evaluated one level deep.
\s(name[n:m]) Compact substring notation, evaluated one level deep.
\fname(args...)) Built-in function with zero or more arguments.
\\ Literal backslash
\N OUTPUT command only: NUL, ASCII 0
\B OUTPUT command only: BREAK
\L OUTPUT command only: Long BREAK

Variable names in Kermit are case-independent. The simplifications in the table are that the notation for decimal and octal bytes can have from one to three digits, and can include braces to separate them from text digits, e.g. \7, \{123}, \o{50}. Hex bytes too, except they must always have exactly two hex digits, 0-9a-f. Array indices must be, or must evaluate to, numbers (floating point numbers are truncated). Associative arrays are also available (dynamic arrays with arbitrary text as subscript), but they are really just a variation on \m() variables (read about associative arrays here). Also, there are some alternative notations for compact substring notation.

We didn't want to have lots of "distinguished" characters, as the UNIX shell does; one is enough, clarity over brevity. Although the notation can be a bit cumbersome, we can use the \m(name) form to circumvent the overevaluation in most contexts. But macro arguments are always assigned to the \%0-9 variables, and thus always evaluated recursively, making it difficult and confusing to pass (e.g.) Windows pathnames as arguments to macros. The same is true for array elements, especially in contexts where they are used to return results from built-in functions (for example, \fsplit() used to return the elements of a comma-separated value list if any of the values contained backslashes). An even worse scenario is when macro arguments are passed from one macro to another; for some graphic illustrations see Taming the Wild Backslash – Part Deux from the C-Kermit 7.0 Update Notes.

We can't just change how variables are evaluated because that would break existing scripts. But we can always add Yet Another SET Command:

SET COMMAND VARIABLE-EVALUATION { RECURSIVE, SIMPLE }
This applies only to \%a-z and \%0-9 variables and to \&a-z[] arrays (since all other kinds of variables are evaluated only one level deep). The default, of course, for backwards compatibility, is RECURSIVE. SIMPLE forces the evaluation of these variables to return their literal contents, without further evaluation:

Here's a short script for illustration:

define path c:\users\fdc\somefile.txt
define test1 {        # Normal recursive argument evaluation
  echo \%0: arg=\%1
}
define test2 {        # Simple argument evaluation
  set var simple
  echo \%0: arg=\%1
}
test1 \m(path)
test2 \m(path)
exit

And here's the result:

?<ERROR:NO_SUCH_FUNCTION:\fdc\somefile.txt()>
test2: arg=c:\users\fdc\somefile.txt

The first line might seem surprising, but under the normal rules (see table above) \f indicates a function call, with the letters following the 'f' being the name of the function. But there is no function by that name... and if there were, you probably didn't intend to call it!

SET COMMAND VARIABLE-EVALUATION SIMPLE has no effect on constants, only on variables. Note how \m(path) is defined. The DEFINE command assigns the literal value of its argument to the named variable (see Table 3 below), thus in this case no special syntax is needed. But in other contexts, you must double the backslashes or use the \fliteral() function to use literal backslashes in data:

test2 c:\\users\\fdc\\somefile.txt
test2 \fliteral(c:\users\fdc\somefile.txt)

C-Kermit 9.0 adds a new notation for \fliteral() which also has certain advantages over it: \q(string):

test2 \q(c:\users\fdc\somefile.txt)

Since \fliteral() is a function, its argument list (the text within parentheses) has special syntax of its own, in which commas and braces are treated specially and introduce another set of quoting problems. \q(string) doesn't have these problems. The only consideration is that parentheses must be balanced or else quoted (preceded by backslash), or represented as numeric character entities (left paren = \40, (right paren = \41).

Or else hold the value in a simple variable as we did with \\m(path) above.

SET COMMAND VARIABLE-EVALUATION SIMPLE is a big change and might have repercussions that didn't show up in the initial tests; a lot more testing is needed.

On the topic of variables, let's summarize in one place the ways in which values can be explicitly assigned to variables. There is nothing new here except the table itself:

Table 2. Variable Assignment in Kermit
Command Shorthand Explanation
DEFINE name value .name = value The literal value becomes the contents of the named variable; variables names in the value are copied without evaluation. This command is for defining macros that take parameters, as well as for defining simple variables, especially if the values contain backslashes.
_DEFINE name value   Like DEFINE but the name is evaluated before use.
ASSIGN name value .name := value The value is evaluated and the result becomes the contents of the named variable.
_ASSIGN name value   Like ASSIGN but the name is evaluated before use.
EVALUATE name expression .name ::= value The expression (in regular algebraic notation) is evaluated arithmetically and the result becomes the contents of the named variable. If the expression contains any variables they are evaluated first.
_EVALUATE name expression   Like EVALUATE but the name is evaluated before use.
INCREMENT name expression   Evaluates the variables in the expression, then evaluates the expression arithmetically, and then adds the value to the contents of the named variable, which must be a number or an algebraic expression. If the expression is empty, a value of 1 is used.
_INCREMENT name expression   Like INCREMENT but the name is evaluated before use.
DECREMENT name expression   Evaluates the variables in the expression, then evaluates the expression arithmetically, and then subtracts the value from the contents of the named variable, which must be a number or an algebraic expression. If the expression is empty, a value of 1 is used.
_DECREMENT name expression   Like DECREMENT but the name is evaluated before use.
DECLARE name = list   An array declaration can include an initializer list; items in the list are evaluated before assignment. This can be defeated by doubling any backslashes or enclosing individual arguments in \fliteral().
DO name arguments name arguments When invoking a macro with a DO command (or an implied one), the arguments are evaluated, then assigned to \%1, \%2, etc, and the macro's name to \%0.
(SETQ name value)   Kermit also includes a mini-LISP interpreter

Variables are evaluated automatically in Kermit commands simply by referencing them, according to rules given in Table 1. The following functions can be used to change how a a particular variable is evaluated:

Table 3. Kermit Functions for Evaluating Variables
Function Argument Description
\fcontents() \%x or \&x[y] Evaluates the variable or array element (which normally would be evaluated recursively) one level deep.
\fdefinition() name If the argument is a \%x variable or an array element, it is evaluated to get the name; otherwise the argument is the name. Its definition is returned with no recursion.
\m() name Equivalent to \fdefinition().
\recurse() \m(name) Forces recursive evaluation of a macro definition (a.k.a. long variable name). NOTE: \frecurse() can operate on any kind of variable as well as on any string containing any mixture of variables.

FORCE-3 Packet Protocol

The Kermit protocol has proven itself over the past 30 years to be robust in terms of surviving harsh transmission environments and delivering the data correctly and completely. In these times of Internet everywhere and error-correcting modems in the few places where the Internet isn't, few people even recall the kinds of difficult conditions that were common when the Kermit protocol was first developed: noisy telephone lines, serial interfaces that drop characters, lack of transparency to control or 8-bit characters, absence of flow control, "bare" modems without error correction.

But the Internet is not everywhere, and not all modems are error-correcting. Perhaps the most difficult trial so far for Kermit or any other protocol is the EM-APEX project, in which floats are dropped into the ocean from an aircraft into the path of a hurricane; these floats dive into the water measuring current, temperature, and salinity at different depths and then surfacing to phone home, sending the data to land stations using Kermit protocol over non-error-correcting 300bps Iridium satellite modems, with high seas and winds battering the floats and heavy (sometimes electrical) storms between the modem and the satellite.

Because of the transmission speed and long distances involved, the transfers were very slow. The Kermit software in the floats is Embedded Kermit, which did not implement sliding windows, which would have sped up the flow considerably. John Dunlap, engineer at the University of Washington's Applied Physics Laboratory, undertook the task of adding sliding windows to E-Kermit. For testing, he rigged up a simulator in which Kermit transfers take place over a connection with different amounts of noise and delay. He found that occasionally, a transfer would appear to succeed, but the received file would be corrupt.

According to the Kermit protocol definition, the first packet always has block-check type 1, a 6-bit checksum, which is the only block check type that all Kermit implementations are required to support; thus any Kermit partner can process this packet. This packet itself can negotiate a higher level of checking, such that subsequent packets have (say) block-check type 3, a 16-bit cyclic redundancy check (CRC) encoded as three printable 7-bit ASCII characters. The 16-bit CRC can catch all errors of certain kinds (single-bit, double-bit, bursts of 16 bits or less), and more than 99.9984741210937% of all other possible errors.

John's simulations revealed that file corruption could occur undetected when the initial packet was corrupted in such a way that a parameter or capability byte was changed and the checksum also changed to make the packet appear to be correct, thus allowing the transfer to proceed with the two Kermit partners out of sync as to packet encoding and interpretation (the chances of two such errors producing a seemingly valid packet are about 1 in 6000 when using the 6-bit checksum). For example the compression technique might be misnegotiated and then the receiver might store incoming data without decompressing it.

The solution is a new option, selected by:

BLOCK-CHECK TYPE 5

to require a type 3 block check (16-bit CRC) on every packet, including the initial ones, thus reducing the probability of a misnegotiation by many orders of magnitude. THIS PARAMETER CAN NOT BE NEGOTIATED. Each Kermit program must be given the "set block 5" command prior to transfer. That's because normally every Kermit program expects the first packet to have a 6-bit checksum, and if the first packet has a 3-byte, 16-bit CRC, the packet receiver will think it is corrupted.

In practice, however, it is possible to code the packet receiver "cheat" by reading the packet data before verifying the block check. Thus when the receiver is C-Kermit 9.0 Beta.01 or later or E-Kermit 1.7 or later, it is only necessary to give the "set block 5" command to the file sender, and the receiver will check for a FORCE-3 first packet. If the receiver does not support this feature, however, the initial packet will be be rejected (after several retries) and the file transfer will not take place. There is no attempt to "back off" to normal behavior.

Table 4. Kermit Protocol Packet Block Check Types
Type Command Bytes Status Explanation
1 SET BLOCK 1 1 Required in all Kermit implementations. Negotiated. 6-bit checksum, suitable for good connections.
2 SET BLOCK 2 2 Optional, negotiated. 12-bit checksum. 64 times stronger than type 1.
3 SET BLOCK 3 3 Optional, negotiated. 16-bit CRC.
BLANK-FREE-2 SET BLOCK 4 2 Optional, negotiated. 12-bit checksum, two nonblank bytes.
FORCE-3 SET BLOCK 5 3 Optional, not negotiated. 16-bit CRC forced all packets.
E-Kermit 1.7

Other New Features

See the C-Kermit Daily Builds page for details. Very briefly:

Incompatibilities

A top priority for new Kermit software releases has always been backwards compatibility. A script written for a previous Kermit release should run the same way in the new release.

There's one exception this time. The \fsplit() function is incredibly handy, it can do almost anything, up to and including parsing a LISP program (the underlying code is the basis of the S-Expression interpreter). But did you ever try to use it to parse (say) a Tab-Separated-List (TSV file) or Comma-Separated-List (CSV)? It works as expected as long as the data contains only 7-bit characters. But if your data contains (say) Spanish or German or Russian text written in an 8-bit character set such as ISO 8859-1, every 8-bit character (any value 128-255) is treated as a break character. This is fixed in C-Kermit 9.0 by treating all 8-bit bytes as "include" characters rather than break characters, a total reversal of past behavior. I don't think it will affect anyone though, because if this had happened to anyone, I would have heard about it!

Since most standard 8-bit character sets have control characters in positions 128-160, it might have made sense to keep 128-160 in the break set, but with the proliferation of Microsoft Windows code pages, there is no telling which 8-bit character is likely to be some kind of text, e.g. smart quotes or East European or Turkish accented letters.

What's Not In C-Kermit 9.0

Some large projects that were contemplated have not been done, including:

And a Loose End...
Using External File-Transfer Protocols on Secure Connections

After C-Kermit 8.0.212 Dev.27 (2006/12/22), I spent a big chunk of time trying to solve a particular problem that some of you have complained about and others might be familiar with: If you use C-Kermit to make a secure Telnet connection to another host (e.g. with Telnet SSL/TLS, Kerberos, or SRP) and then attempt to transfer a file using an external protocol such as Zmodem, it doesn't work.

That's because as coded (through 8.0.211), C-Kermit simply starts the external protocol in a fork with its standard i/o redirected to the connection. This completely bypasses the encryption and decryption that is done by C-Kermit itself, and of course it doesn't work. The same thing occurs if you use the REDIRECT command. The routine that handles this is ttruncmd() in ckutio.c.

In order to allow (say) Zmodem transfers on secure connections, it is necessary for C-Kermit to interpose itself between the external Zmodem program and the connection, decrypting the incoming stream before feeding it to Zmodem and encrypting Zmodem's output before sending out the connection.

In principal, this is simple enough. We open a pseudoterminal pair ("master" and "slave") for Zmodem's i/o and we create a fork and start Zmodem in it; we read from the fork pty's standard output, encrypt, and send to the net; we read from the net, decrypt, and write to the fork pty's standard input.

In practice, it's not so simple. First of all, pseudoterminals (ptys) don't seem to interface correctly with certain crucial APIs, at least not in the OS's I have tried (Mac OS X, Linux, NetBSD, etc), such as select(). And i/o with the pty often – perhaps always – fails to indicate errors when they occur; for example, when the fork has exited.

But, even after coding around the apparent uselessness of select() for multiplexing pty and net, and using various tricks to detect when the external protocol exits and what its exit status is, I'm still left with a show-stopping problem: I just simply can not download (receive) a file with Zmodem, which is the main thing that people would probably want to do. I can send files just fine, but not receive. The incoming stream is delivered to Zmodem (to the pty slave) but upon arrival at the Zmodem process itself, pieces are always missing and/or corrupt. Yet I can receive files just fine if I use Kermit itself (C-Kermit or G-Kermit) as the external protocol, rather than Zmodem.

I can think of two reasons why this might be the case:

  1. Zmodem sends all 8-bit bytes and control codes in the clear, and maybe the pty is choking on them because it thinks it is a real terminal.

But Zmodem puts its controlling terminal into raw mode. And C-Kermit puts the pty into raw mode too, just for good measure. If any 0xFF codes are in the Zmodem data stream, and it's a Telnet session, Kermit does any needed byte stuffing/unstuffing automatically. Anyway, if I tell Zmodem to prefix everything, it makes no difference.

  1. Zmodem is a streaming protocol and perhaps the pty driver can't keep up with a sustained stream of input at network speeds. What would be the method of flow control?

I can vary the size of the i/o buffers used for writing to the pty, and get different effects, but I am not able to get a clean download, no matter what buffer size I use. write()'ing to the pty does not return an error, and I can't see the errors because they happen on the master side. It's as if the path between the pty slave and master lacks flow control; I deliver a valid data stream to the pty slave and the master gets bits and pieces. This impression is bolstered somewhat by the "man 7 pty" page in HP-UX, which talks about some special modes for ptys that turn off all termio processing and guarantee a flow-controlled reliable stream of bytes in both directions – a feature that seems to be specific to HP-UX, and exactly the one we need everywhere.

Well, in Pass One I used C-Kermit's existing pty routines from ckupty.[ch], which are well-proven in terms of portability and of actually working. They are currently used by SET HOST /PTY for making terminal connections to external processes. But these routines are written on the assumption that the pty is to be accessed interactively, and maybe they are setting the fork/pty arrangement up in such a way that that's not suitable for file transfer. The Pass One routine is called xttptycmd() in ckutio.c.

So in Pass Two I made a second copy of the routine, yttptycmd(), that manages the pty and fork itself, so all the code is in one place and it's simple and understandable. But it still doesn't work for Zmodem downloads. In this routine, I use openpty() to get the pty pair, which is not portable, so I can have access to both the master and slave pty file descriptors. This version can be used only a platforms that have openpty(): Linux, Mac OS X, NetBSD, etc.

In Pass Three, zttptycmd(), I tried using pipes instead of ptys, in case ptys are simply not up to this task (but that can't be true because if I make a Telnet or SSH connection into a host, I can send files to it with Zmodem, and the remote Zmodem receiver is, indeed, running on a pty). But pipes didn't work either.

In Pass Four, I extracted the relevant routines into a standalone program based on yttptycmd() (the openpty() version, for simplicity), which I tested on Mac OS X, the idea being to rule out any "environmental" effects of running inside the C-Kermit process. There was no difference -- Kermit transfers (with C-Kermit itself as the external protocol) worked; Zmodem transfers (neither sz or lsz) did not.

Well, it's a much longer story. As the external protocol, I've tried rzsz, crzsz, and lrzsz. We know that some of these have quirks regarding standard i/o, etc, which is one of the reasons for using ptys in the first place, and i/o does work – just not reliably. Anyway, the 1100 lines or so of ckc299.txt, starting just below where it says "--- Dev.27 ---" tell the full story. At this point I have to give up and move on; it might be more productive to let somebody else who has more experience with ptys take a look at it – if indeed anyone still cares about being able to do Zmodem transfers over secure Telnet connections.

C-Kermit 9.0 contains the three new routines (and some auxiliary ones), but they are not compiled or called unless you build it specially:

make targetname KFLAGS=-DXTTPTYCMD (builds with xttptycmd())
make targetname KFLAGS=-DYTTPTYCMD (builds with yttptycmd())
make targetname KFLAGS=-DZTTPTYCMD (builds with zttptycmd())
These are all in ckutio.c. As noted, the second one works only for Linux, FreeBSD, NetBSD, and Mac OS X, because it uses non-POSIX, non-portable openpty(). If you want to try it on some other platform that has openpty(), you can build it like this:
make targetname "KFLAGS=-DYTTPTYCMD -DHAVE_OPENPTY"
(and let me know, so I can have HAVE_OPENPTY predefined for that platform too). The best strategy to get this working, I think, would be to concentrate on yttptycmd(), which is the simpler of the two pty-based routines. If it can be made to work, then we'll see if we can retrofit it to use the ckupty.c routines so it will be portable to non-BSD platforms.

By the way, if you build with any of [XYZ]TTPTYCMD defined, then the selected routine will always be used in place of ttruncmd(). This is to allow testing on all kinds of connections, not just secure ones, in both local and remote mode. Once the thing works, if it ever does, I'll add the appropriate tests and/or commands.

By default, in the initial test release, C-Kermit 9.0 uses ttruncmd() on serial connections and ttyptycmd() on network connections. Even when a network connection is not encrypted, Kermit still needs to handle the network protocol, e.g. the quoting of 0xff bytes on Telnet connections.

– Frank da Cruz   fdc@columbia.edu

C-Kermit 9.0 / The Kermit Project / Columbia University / kermit@columbia.edu / validate