13-Nov-97 13:22:37-GMT,1089;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id IAA24115; Thu, 13 Nov 1997 08:22:36 -0500 (EST) Date: Thu, 13 Nov 97 8:22:36 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: fdc@watsun.cc.columbia.edu Subject: some strange results Cc: jrd@cc.usu.edu, jaltman@watsun.cc.columbia.edu Message-ID: watsun to powerpc2 streaming 349525 cps non-streaming 524288 cps powerpc2 to watsun streaming 524288 cps non-streaming 262144 cps This will have to be looked at with the Sniffer to determine what is going on. Although I think it probably has something to do with the Delayed Acks. --- powerpc2 is AIX 4.1 running on a 150MHz Personal Power system. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 13-Nov-97 14:06:37-GMT,1185;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id JAA02794; Thu, 13 Nov 1997 09:06:36 -0500 (EST) Date: Thu, 13 Nov 97 9:06:36 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: fdc@watsun.cc.columbia.edu Subject: AIX ckcnet.h Cc: jaltman@watsun.cc.columbia.edu Message-ID: Do we have a #define for AIX? the include files and both include defines for: #define TCP_NODELAY 0x1 #define TCP_MAXSEG 0x2 #define TCP_KEEPALIVE 0x8 We either want to conditionally include one of the files or just copy these #defines into ckcnet.h conditionally for AIX. BTW, ftp transfers between watsun and powerpc2 are: watsun to powerpc2 binary >1000cps ascii 792cps powerpc2 to watsun binary 942cps ascii 738cps Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 13-Nov-97 16:12:55-GMT,3354;000000000015 Return-Path: Received: from GRUMPY.USU.EDU (grumpy.usu.edu [129.123.1.86]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id LAA29915; Thu, 13 Nov 1997 11:12:54 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IPYEKBTKHC9EJMLL@cc.usu.edu>; Thu, 13 Nov 1997 09:12:42 -0600 (MDT) Date: Thu, 13 Nov 1997 09:12:42 -0600 (MDT) From: Joe Doupnik Subject: Re: fast stuff To: fdc@watsun.cc.columbia.edu Cc: JALTMAN@watsun.cc.columbia.edu Message-id: <01IPYEKBTP769EJMLL@cc.usu.edu> X-VMS-To: IN%"fdc@watsun.cc.columbia.edu" X-VMS-Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Frank and Jeff, Thinking about the streaming stuff overnight... I suspect this is all way off base so far, including that ZD article (especially that confused piece of writing). Let me expand a little. Kermit ACKs piling up in a reverse channel send buffer do not cause TCP ACKs to occur less frequently than if no Kermit ACKs were present. If anything TCP ACKs will occur more frequently to flush a full MSS to the file sender. A 200ms delayed ACK queue can work a couple of ways. One is to accumulate ACK needs and then five times per second send an ACK. That is what your MS statements suggest. Well, suppose that's the case. Then the file sender could, at max, send a whole TCP window five times per second, 5 x 32KB = 150KB/sec, lots lots less for smaller windows such as in MS products. The other way is to queue one ACK and when another is queued see that a queue is outstanding and do a formal transmission. Thus guys standing in line cause the queue to be flushed. This is the ACK every other packet scheme in most large TCP stacks. There is no limit on the transmission throughput in this case. Every arriving segment with data generates an ACK need, every second generates an ACK transmission. The very first and very last segments *may* experience the 200ms delay, as well as first segments after serious congestion recovery. The TCP window is normally larger than two TCP segments, and thus the two machines go into full max rate streaming. There is one more TCP effect I haven't mentioned yet. Window opening. If the receiver's TCP window closes then in big stacks it is reported as still closed until it is half empty. Hysterisis. In lieu of other requests a window opening transmission will be sent by the receiver when the window suddenly is declared available again. Clearly, the file transmitter is blocked by a closed window, though it can probe if it gets anxious (MSK does). MSK's TCP receiver does not hold down window reports that strongly. What I think is happening is CK is committing too many resources trying to read Kermit ACKs. This is not a TCP problem at all. One can easly see if the file receiver is using ACK plan A or plan B by watching the packet trace timing. One can easily see closed window blockages the same way. One can also see if TCP is blocked in any way rather than CK spinning its wheels, as I think it is. That ZD article has matters all confused. Note, older SUN TCP stacks have a bug in window handling such that they drive themselves into a corner and nibble at the data stream. I have a paper on the matter, and SUN has compendium patches which cure it. Joe D. 13-Nov-97 16:45:15-GMT,1496;000000000005 Return-Path: Received: from GRUMPY.USU.EDU (grumpy.usu.edu [129.123.1.86]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id LAA11536; Thu, 13 Nov 1997 11:45:13 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IPYFYSEN0M9LXGDG@cc.usu.edu>; Thu, 13 Nov 1997 09:44:31 -0600 (MDT) Date: Thu, 13 Nov 1997 09:44:30 -0600 (MDT) From: Joe Doupnik Subject: Re: fast stuff To: fdc@watsun.cc.columbia.edu Cc: JALTMAN@watsun.cc.columbia.edu Message-id: <01IPYFYSEOW89LXGDG@cc.usu.edu> X-VMS-To: IN%"fdc@watsun.cc.columbia.edu" X-VMS-Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Frank, I still suspect it's all in Kermit and the o/s scheduler. Sweet spots and all that jazz. Packet traces will resolve TCP heuristic issues, presuming the traces reveal timing to millisec resolution. Again, FTP runs without high level ACKs and thus generates TCP ACKs *less often* than a scheme which sends back info. This will be plain in the traces. Getting caught looking for, and/or dealing with, returned information is the most likely difficulty in this problem. Test the returned info situation with streaming CK. Read the bytes and discard them at as low a level as you can manage. Ditto, but decimate the look rate. See if the read routine leads to blockages or context switches that steal time. Try again with Kermit ACKs padded to one MSS. Joe D. 13-Nov-97 16:57:33-GMT,2004;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id LAA13603; Thu, 13 Nov 1997 11:57:33 -0500 (EST) Date: Thu, 13 Nov 97 11:57:32 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: Frank da Cruz Cc: Joe Doupnik , jaltman@watsun.cc.columbia.edu Subject: Re: fast stuff In-Reply-To: Your message of Thu, 13 Nov 97 11:27:45 EST Message-ID: I threw the Ziff Davis article in because I found it. It really has nothing to do with what I have observed with the Sniffer. Joe, as you say the Delayed Ack can occur on the very last segment. Here is the scenario that I have seen. I don't have access to the Sniffer until Monday and plan to do much more serious documentation of the behavior at that time. The TCP Window from the Sender is full or close to it. The Receiver sends a Kermit ACK (small segment triggering Nagle) with a TCP ACK piggybacked with a Window size smaller than one half the MSS. The Sender stops sending data. The Receiver does not ACK the last received segment and does not send any more Kermit ACKs because of Nagle and therefore waits to timeout 200ms. The Sender gets the TCP ACK containing a Kermit ACK and the larger Window size but doesn't send anything yet. It waits 200 ms and then TCP ACKs the last ACK and the data starts to flow again. --- Again this is not really a problem with the RTT is close to the order of the delay time but when we are talking orders of magnitude between the delay time and the RTT we hit a serious bottleneck. I will send something much more precise next week. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 13-Nov-97 17:03:17-GMT,1727;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id MAA14630; Thu, 13 Nov 1997 12:01:17 -0500 (EST) Date: Thu, 13 Nov 97 12:01:16 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: Joe Doupnik Cc: fdc@watsun.cc.columbia.edu, jaltman@watsun.cc.columbia.edu Subject: Re: fast stuff In-Reply-To: Your message of Thu, 13 Nov 1997 09:44:30 -0600 (MDT) Message-ID: > Frank, > I still suspect it's all in Kermit and the o/s scheduler. Sweet > spots and all that jazz. > Packet traces will resolve TCP heuristic issues, presuming the > traces reveal timing to millisec resolution. Again, FTP runs without > high level ACKs and thus generates TCP ACKs *less often* than a scheme > which sends back info. This will be plain in the traces. Getting caught > looking for, and/or dealing with, returned information is the most likely > difficulty in this problem. > Test the returned info situation with streaming CK. Read the bytes > and discard them at as low a level as you can manage. Ditto, but decimate > the look rate. See if the read routine leads to blockages or context > switches that steal time. Try again with Kermit ACKs padded to one MSS. > Joe D. > Even so, an ACKless protocol will always be faster than an ACKing protocol on Ethernet because Ethernet is half duplex. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 13-Nov-97 17:26:11-GMT,3914;000000000005 Return-Path: Received: from barney.usu.edu (barney.usu.edu [129.123.1.89]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id MAA19289 for ; Thu, 13 Nov 1997 12:26:10 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IPYH1M80Y09LX9CT@cc.usu.edu>; Thu, 13 Nov 1997 10:25:10 -0600 (MDT) Date: Thu, 13 Nov 1997 10:25:08 -0600 (MDT) From: Joe Doupnik Subject: Re: fast stuff To: jaltman@columbia.edu Cc: FDC@WATSUN.CC.COLUMBIA.EDU Message-id: <01IPYH1M82U29LX9CT@cc.usu.edu> X-VMS-To: IN%"jaltman@columbia.edu" X-VMS-Cc: FDC@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT >Even so, an ACKless protocol will always be faster than an ACKing >protocol on Ethernet because Ethernet is half duplex. Jeff, may I politely suggest "rubbish"? You guys are not hitting the limits of the wire. That means you aren't pushing 1MB/sec of user (Kermit) data. Kermit ACK packets are tiny and use very little wire capacity. I have pushed the wire to the limits, with (IPX) ACKs. And I've done it at both 10Mbps and 100Mbps Ethernets, and the user data rates involved are 1MB/sec and 10+MB/sec. On the other message. >The TCP Window from the Sender is full or close to it. Actually you don't know this. The sender's transmit buffer is not advertised on the wire. Only receive windows are advertised. >The Receiver sends a Kermit ACK (small segment triggering Nagle) with >a TCP ACK piggybacked with a Window size smaller than one half the MSS. >The Sender stops sending data. The file sender is in Kermit sliding windows mode. It presumably has more Kermit data to send. If it stops it either stopped itself at Kermit level for Kermit reasons or it found that the TCP stack declined to accept further data or the TCP stack was blocked from transmitting (and Nagle's algorithm has long since faded from the scene, forget it). The file sender stopped for a reason. It can do so if the receiver says its window is closed. That window size is in every TCP segment, so it's visible in the sniffer. If the window did close then the receiver is falling way behind at the application level. Falling behing reading is part of what I suggested is really wrong; CK gets stuck with reading, perhaps because of o/s side effects. >The Receiver does not ACK the last received segment and does not send >any more Kermit ACKs because of Nagle and therefore waits to timeout >200ms. TCP itself ACKs TCP data, always. Every TCP transmission will carry as much user data as the other side's window, moderated by congestion avoidance, will allow. Nagle's algorithm does not reduce this at all. That arriving file data will cause those pent-up Kermit ACKs to be transmitted; this has nothing at all to do with Nagle. If the Kermit transmitter + its TCP stack decides to stop sending for its own reasons then things get sluggish. Now to your sentence above. So a last segment arrives and its TCP ACK is put on the 200ms timer queue. So what? There are many other previous segments which have been ACK'd, and Kermit ACKs going out with them. All the sentence refers to is the very trailing end of the sliding window material, and the rest of the window has been cleaned up and should be raring to send more data. >The Sender gets the TCP ACK containing a Kermit ACK and the larger >Window size but doesn't send anything yet. It waits 200 ms and then >TCP ACKs the last ACK and the data starts to flow again. So why is the transmitter sitting idle for 200ms? There must be a reason, such as the transmitter does not have control of the machine (scheduler, app busy doing something else, bug in the software such that it has not continued to fill the Kermit sliding window stream, receiver's window is closed and won't say open until that end catches up). Joe D. 13-Nov-97 18:23:03-GMT,2189;000000000005 Return-Path: Received: from barney.usu.edu (barney.usu.edu [129.123.1.89]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id NAA02406 for ; Thu, 13 Nov 1997 13:23:02 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IPYJD1S5S09EJTSO@cc.usu.edu>; Thu, 13 Nov 1997 11:22:27 -0600 (MDT) Date: Thu, 13 Nov 1997 11:22:26 -0600 (MDT) From: Joe Doupnik Subject: Re: fast stuff To: jaltman@columbia.edu Cc: FDC@WATSUN.CC.COLUMBIA.EDU Message-id: <01IPYJD1T7G29EJTSO@cc.usu.edu> X-VMS-To: IN%"jaltman@columbia.edu" X-VMS-Cc: FDC@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT >I will send you summations of the Sniffer logs when I get them back. Rog. That should help. >The reason the sender stops sending is because the TCP receive window >is closed. The Kermit Window is not. Now we are getting somewhere. File receiver falls well behind and tries to catch up. When it does drain most/much/half of its TCP receive buffer a window opening packet should be emitted. Perhaps it is not being emitted, leading to the 200ms deadlock. Here's the MSK piece of code on the matter (translation below it): /* send a window update if crossing a boundry */ if (s->rdatalen < (TCP_RBUFSIZE / 2) && (s->rdatalen + x) >= TCP_RBUFSIZE / 2) tcp_send(s); /* send opening update */ Send absolutely (now) a window opening message if the TCP receive buffer drains across the half-full watermark. If a brand X stack put this message on a 200ms wait queue it should show up in the sniffer logs and delay matters by 200ms. Such a delay would be a design error. >C-Kermit does not just blast data, it checks for socket write >availability with select() before attempting any write. Same for reads. > >The sniffer logs show that this behavior never occurs to FTP streams. >FTP streams are continuous with one TCP ACK for every four TCP Data >Segments. Ok, so if Kermit reading causes delays then we are close to the observations. I'll wait for the sniffer logs and spare us more speculation this morning. Thanks, Joe D. 13-Nov-97 21:28:06-GMT,815;000000000005 Return-Path: Received: (from jrd@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id QAA11625; Thu, 13 Nov 1997 16:28:05 -0500 (EST) Date: Thu, 13 Nov 97 16:28:05 EST From: "Joe R. Doupnik" To: fdc@watsun.cc.columbia.edu, jaltman@watsun.cc.columbia.edu Cc: jrd@watsun.cc.columbia.edu Subject: Solaris TCP window bugs Message-ID: Frank and Jeff, In ~jrd on watsun is file probing.TCP.ps.Z, the paper by Doug Comer on shortcomings of Solaris regarding zero TCP windows. It's 10 pages of paper. Sun has subsequently fixed their TCP stack to overcome this defect (has been observed here in spades with a fellow's Sun speaking eventually over a phone line to a Win95 machine at home). One needs to obtain the Sun patch. Joe D. 14-Nov-97 2:35:48-GMT,2713;000000000005 Return-Path: Received: from SLEEPY.USU.EDU (sleepy.usu.edu [129.123.1.85]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA02168; Thu, 13 Nov 1997 21:35:47 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IPZ00M22MC9LXQPC@cc.usu.edu>; Thu, 13 Nov 1997 19:35:33 -0600 (MDT) Date: Thu, 13 Nov 1997 19:35:33 -0600 (MDT) From: Joe Doupnik Subject: Misc ramblings on streaming tests To: FDC@WATSUN.CC.COLUMBIA.EDU Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU Message-id: <01IPZ00M23K69LXQPC@cc.usu.edu> X-VMS-To: FDC@WATSUN.CC.COLUMBIA.EDU X-VMS-Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Frank and Jeff, A few observations from here on the streaming stuff. My normal Kermit setup, CK on netlab1 to CK on watsun: 25KB/sec. Ftp between the same places: 100KB/sec. Look at Kermit settings and see 2KB x 4. That's 8KB of windowing whereas netlab1's TCP stack has 32KB buffers, a 1:4 dilution by Kermit. MSK to watsun gives the same transfer rates, and MSK has only 4KB TCP buffers. Again, the Kermit protocol side is the throttle so the TCP buffer size doesn't make all that much difference in this test. Check assumptions later in the day, USU to CU again. CKermit 1.5KB x 20 on both ends: 120KB/sec Quick ftp: 180KB/sec. Given CK's formatted screen etc that is close. Conclusion: on long distances we starve in the Kermit buffer arena compared to the underlying TCP transport channel. Not much new about that, but the scaling above does work out. Good old pipe filling. MSK check. Bypass decoding and writing to disk. No significant change when disk is a RAM drive, some change if using a real hard drive. So MSK's decoder is faster than a SCSI Seagate Barracuda under DOS. CK check. Create a Unix RAM drive (memfs). No change for local transfers. Conclusion: I dunno. FTP check. Ftp to adjacent NW file server. 500-700KB/sec either way. About right when that server is only a 486-33. I Had a quick look at the wire while within Windows (ugh), and saw CK send to MSK with pauses of 60-120 millisec periodically. MSK does the same when it sends the file, and that is to read disk, construct Kermit packets, etc. It turns out those pauses are large consumers of time. I won't be able to repeat this until the weekend when I can steal my student lab to run clean DOS machines and the wire snoop independently of the file xfer. The wermit involve here was the regular test item, not the streaming edition. To beat FTP on long distance is no problem: just use enough buffering to fill the pipe. To beat FTP locally don't bother. Joe D. 14-Nov-97 2:47:28-GMT,1027;000000000005 Return-Path: Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA03273; Thu, 13 Nov 1997 21:47:27 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IPZ15DULPS9IB80O@cc.usu.edu>; Thu, 13 Nov 1997 19:46:24 -0600 (MDT) Date: Thu, 13 Nov 1997 19:46:24 -0600 (MDT) From: Joe Doupnik Subject: Rambling, part two To: FDC@WATSUN.CC.COLUMBIA.EDU Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU Message-id: <01IPZ15DV3LU9IB80O@cc.usu.edu> X-VMS-To: FDC@WATSUN.CC.COLUMBIA.EDU X-VMS-Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Frank and Jeff, The last of tonight's "putting the numbers in" experiments. MSK to watsun. 2KB x 4: 22KB/sec. 2KB x 20: 45KB/sec. Modify MSK to use a 31KB TCP buffer, repeat MSK to watsun. 2KB x 20: 125KB/sec So the large TCP buffer let MSK fill the pipe and that was that. Joe D. 14-Nov-97 4:17:30-GMT,2048;000000000005 Return-Path: Received: from barney.usu.edu (barney.usu.edu [129.123.1.89]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id XAA13435; Thu, 13 Nov 1997 23:17:29 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IPZ41WQOMO9IBEKQ@cc.usu.edu>; Thu, 13 Nov 1997 21:17:24 -0600 (MDT) Date: Thu, 13 Nov 1997 21:17:24 -0600 (MDT) From: Joe Doupnik Subject: Streaming, last comment for tonight To: FDC@WATSUN.CC.COLUMBIA.EDU Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU Message-id: <01IPZ41WQY1U9IBEKQ@cc.usu.edu> X-VMS-To: FDC@WATSUN.CC.COLUMBIA.EDU X-VMS-Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Frank and Jeff, Upon a little reflection it seems that the way to compete with FTP on local wiring is to join the club. And that means avoid touching bytes as much as possible. A suggestion is to assure an 8-bit clean channel, use regular Kermit packets for all but the D packet. The D packet has a normal Kermit framing but the real data is raw block stuff. To further reduce touching bytes we need to open a second TCP channel which has no Telnet processor at each end, ftp-data style. Send all the Kermit packets via that channel. Use a linear checksum to save time, a la TCP, to double check byte slippage in parsing frames from the stream. Channel closing needs finesse to keep it open over files within a group, and thus lower session setup overhead. 'B' packet ought to do nicely as a terminator. You've probably thought of this too so I will claim zero originality. We claim advantage of: Kermit attributes, character set handling, and similar bells and whistles, but ftp speed for bulk data. The only problem I have with this I can't afford 32KB TCP buffers very easily, but one can be found to absorb file transfer traffic on a single session at a time. I also have an administrative mess in the code dealing with two sessions simultaneously, but I presume there's a way. Joe D. 14-Nov-97 5:40:56-GMT,1048;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id AAA25610 for fdc; Fri, 14 Nov 1997 00:40:56 -0500 (EST) Date: Fri, 14 Nov 97 0:40:56 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: fdc@watsun.cc.columbia.edu Subject: streaming Message-ID: something that occurred to me. There was something ringing in my ear, something Joe said earlier. He said that when the TCP Window closes it won't open again until the recv buffer is half empty. Well, C-Kermit on Unix only reads at most 4096 bytes at a time in myread(). But the recvbuf for Sun TCP is 24K. We need to increase the read buffer of C-Kermit in ckutio.c. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 14-Nov-97 8:27:36-GMT,972;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id DAA25218 for fdc; Fri, 14 Nov 1997 03:27:35 -0500 (EST) Date: Fri, 14 Nov 97 3:27:35 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: fdc@watsun.cc.columbia.edu Subject: more streaming Message-ID: I increased the MYBUFSIZ in ckutio.c to 32678 and with Streaming on I consistently transmit data from watsun to powerpc2 at 1mb/sec. With streaming off the data rate is 0.5mb/sec. --- you might want to compare the TCP RTO detection algorithm with the one you wrote for Kermit. RFC 1122. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 16-Nov-97 2:06:51-GMT,1007;000000000005 Return-Path: Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA01060 for ; Sat, 15 Nov 1997 21:06:51 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IQ1SFNWEI89ELOQ5@cc.usu.edu> for FDC@WATSUN.CC.COLUMBIA.EDU; Sat, 15 Nov 1997 19:06:48 -0600 (MDT) Date: Sat, 15 Nov 1997 19:06:48 -0600 (MDT) From: Joe Doupnik Subject: Streaming, a possible aid To: FDC@WATSUN.CC.COLUMBIA.EDU Message-id: <01IQ1SFNWJ829ELOQ5@cc.usu.edu> X-VMS-To: FDC@WATSUN.CC.COLUMBIA.EDU X-VMS-Cc: JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Frank, On netlab1 I built tcpdump (/opt/bin/tcpdump) as an aid in your streaming tests. It works best in non-promiscous mode (tcpdump -p ...) but does run in promiscous mode if desired. For the time being I made it suid so it could be run by non-root (yes, I know). Joe D. 20-Nov-97 5:17:01-GMT,972;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id AAA29623 for fdc; Thu, 20 Nov 1997 00:17:01 -0500 (EST) Date: Thu, 20 Nov 97 0:17:00 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: fdc@watsun.cc.columbia.edu Subject: streaming notes Message-ID: -I flag should set 'streaming' to 1 I thhink Streaming mode should be displayed to the user as a 'Message' and not as a protocol type. The results of other protocol negotiations should be displayed as messages as well. This will be important for KUI. Other than that looks really good. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 20-Nov-97 15:50:54-GMT,1132;000000000011 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA02703 for fdc; Thu, 20 Nov 1997 10:50:54 -0500 (EST) Date: Thu, 20 Nov 97 10:50:54 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: fdc@watsun.cc.columbia.edu Subject: streaming Message-ID: I am using the code from last night. wermit -Ig ckoco3.c with my modification to turn streaming on when -I is specificied. The connection is one which drops out periodicly with very long delays. Using autodownload server on K95. K95 reported the file successfully transfered, however, it really only transfered 70059 out of 517768 bytes and wermit only received about 66000 bytes. It is not clear why it failed. I am going to try to reproduce it. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 20-Nov-97 15:55:18-GMT,1223;000000000001 Return-Path: Received: (from fdc@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA03386; Thu, 20 Nov 1997 10:55:16 -0500 (EST) Date: Thu, 20 Nov 97 10:55:16 EST From: Frank da Cruz To: jaltman@columbia.edu Subject: Re: streaming In-Reply-To: Your message of Thu, 20 Nov 97 10:50:54 EST Message-ID: > I am using the code from last night. wermit -Ig ckoco3.c with my > modification to turn streaming on when -I is specificied. > What modification? Streaming *is* used when -I is specified, as long as you haven't changed the default SET STREAMING value, which is AUTO. > The connection is one which drops out periodicly with very long > delays. Using autodownload server on K95. > > K95 reported the file successfully transfered, however, it really only > transfered 70059 out of 517768 bytes and wermit only received about > 66000 bytes. > > It is not clear why it failed. > I am going to try to reproduce it. > Well... packet logs, etc. It might be just a case of not reporting the failure on the file-transfer display -- that's where I left off last night. What does "stat" tell you after such a failure. - Frank 21-Nov-97 2:49:24-GMT,4361;000000000005 Return-Path: Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id VAA00747; Thu, 20 Nov 1997 21:49:23 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IQ8S9BODY89EOBB5@cc.usu.edu>; Thu, 20 Nov 1997 19:49:14 -0600 (MDT) Date: Thu, 20 Nov 1997 19:49:14 -0600 (MDT) From: Joe Doupnik Subject: Re: Streaming - Draft 3 To: fdc@watsun.cc.columbia.edu Cc: JALTMAN@watsun.cc.columbia.edu Message-id: <01IQ8S9BOHPU9EOBB5@cc.usu.edu> X-VMS-To: IN%"fdc@watsun.cc.columbia.edu" X-VMS-Cc: JALTMAN@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Frank, Thanks for the most recent streaming draft. It still bothers me, as you can guess. Let me sketch my concerns without getting into debate mode. ACK's aren't bad, they cost almost nothing in bandwidth except for the particular situation of half duplex style connections. Such might occur with modem technology attempting to use the entire bandwidth one-way to drain transmit buffers, but I don't think this is actually true even for the 56Kbps schemes on the market. Nagle is a red herring. The scenario is a Kermit puts its small ACK into the TCP buffer and Nagle's algorithm says wait. Incoming packets carrying any data, as they do for file transfers, mandate a reply from the receiver and that reply includes all available data up to TCP limits. Thus incoming file packets cause transmission of Kermit ACKs. Of course this brings up the subject of 200ms delayed ACK queues. I think, subject to correction from experiments, that only one entry can remain in the queue, and the second entry causes the queue to be flushed. If this were not true then FTP data would progress at the rate of 5 ACKs/sec, and we know there are many more per second than that even in the absence of data in the ACK. If I am correct then the 200ms delay affects the first and/or last ACK but not the ones in the middle. Yes, every other ACK may be delayed, but only until the next packet arrival and hence well within the window jitter budget. Putting these together says the file transfer rolls full blast, as indeed FTP data does. Kermit ACKs would be free payload fitting easily within the Ethernet padding budget. I suspect error correcting modems also use similar triggered ACKs as automatic flow control. With those ACKs go queued reverse channel data. I don't know this to be the case, but it seems likely. Underlying some of what you are trying to accomplish is avoiding retaining many packets in retry buffers, enough to fill the pipe round trip. I can understand that. Why not let the transport provider supply those buffers, as TCP does already, is what I hear in the proposal. Given TCP the buffer sizes are typically 32KB and less, some cases of 64KB (SGI machines) and a very few cases of larger values. That says Kermit needs matching retry/out-of-order-reception buffering capacity, and when it does the system streams at full rate, with ACKs. I demonstrated that last weekend with MSK to CK on watsun, yielding the same rates as FTP. Thus providing Kermit buffers is not a big deal. Local links have short pipes and the Kermit buffer count remains small, even for fat pipes. Long distance links are most often very skinny indeed and Kermit's bit capacity is still more than that in the pipe. A middle sized pipe, USU to CU on a quiet day, appears to be less than 32KB capacity and we can (I have) fill it with ACK-ful Kermits. Interestingly, Kermits can readily have larger packet storage capacity than an underlying transport provider. 32KB TCP windows are easily exceeded; modems have much less storage. What's the problem? Were I to go about this business I would mimic FTP's behavior of "raw" data in one channel and control in another, and dispense with packet framing in the data channel. That itself takes away a measurable overhead. There is the difficulty, however, of serial comms having only one channel. But on modems I don't think we have a difficulty with even ordinary buffers (4 window slot style) and hence no need to give up ACKs and trust the world. So I remain dubious of the need to remove Kermit ACKs. The case for removal just has not been made as far I am aware. Joe D. 21-Nov-97 2:59:19-GMT,1804;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id VAA01522; Thu, 20 Nov 1997 21:59:17 -0500 (EST) Date: Thu, 20 Nov 97 21:59:17 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: Joe Doupnik Cc: fdc@watsun.cc.columbia.edu Subject: Re: Streaming - Draft 3 In-Reply-To: Your message of Thu, 20 Nov 1997 19:49:14 -0600 (MDT) Message-ID: Joe: Kermit ACKs have almost zero cost on long haul connections, but not when there are extremely short Round Trip Times and there are hard coded Delayed ACKs. Don't test from USU to CU, test from between two machines on the same segment where one of those machines is running Microsoft TCP/IP and the other is something other than MSK. FTP does not result in TCP ACKs for every data segment. When using FTP one ACK is sent for every four data segments. The combination of Nagle and Delayed ACKs can be disasterous. Look at some of John Nagle's postings on the subject in recent months. (I think I have one, if I can find it I will forward it.) As soon as I can get the Sniffer back I will send you log summaries. We are having some serious problems with some equipment from Lucent that is preventing me from being able to borrow it. In the meantime, the demonstrative performance gains that both Frank and I have been able to see by implementing Streaming will have to do as evidence of its worthiness. -Jeff Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 21-Nov-97 3:24:01-GMT,2874;000000000005 Return-Path: Received: from SNEEZY.USU.EDU (sneezy.usu.edu [129.123.1.87]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id WAA04289 for ; Thu, 20 Nov 1997 22:24:00 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IQ8U0400289IHD0I@cc.usu.edu>; Thu, 20 Nov 1997 20:23:57 -0600 (MDT) Date: Thu, 20 Nov 1997 20:23:57 -0600 (MDT) From: Joe Doupnik Subject: Re: Streaming - Draft 3 To: jaltman@columbia.edu Cc: FDC@WATSUN.CC.COLUMBIA.EDU Message-id: <01IQ8U0402W29IHD0I@cc.usu.edu> X-VMS-To: IN%"jaltman@columbia.edu" X-VMS-Cc: FDC@WATSUN.CC.COLUMBIA.EDU,JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Jeff, Thanks for John's comments. I think I have a copy of them in the archives (use them later in my networking course). I wish they applied here but I don't think they do because the file transfer keeps forcing replies, 1:2 or even 1:4, and isn't send-send-wait unless the receive window is two segments long (and even then window opening triggers a reply). I'm groping here, but from what I gather on your observations with the MS Windows TCP/IP stack is it has a small TCP buffer. I should find a way of looking at it, except I never use that item from MS. And, to complete the picture, I presume the buffer is smaller than the ACK queue depth. Is that right? Hence the transmitter sends everything and the ACKs sit in the receiver's 200ms delay queue and there aren't enough ACKs to force the queue to flush without a timeout. Right? If wrong please help me understand the environment. That would put them into the 5 ACKs/sec situation; burst wait, burst wait, etc. If this is about right then enlarging the MS TCP buffer should turn things around. A guess is the buffer is 4KB, about 2.5 full length local Ethernet packets, and hence under the 4 and go threshold for ACKs. Then 8KB should be enough to make it roll. We should be able to measure this effect from the outside, and maybe you have already. Maybe there's yet another MS blunder in the picture: excessive hold down of window opening ACK. Perhaps they sit on each window opening announcement until NT 5 is ready. Sorry, couldn't resist. But I am serious about maybe delaying even that piece of bookkeeping. To my way of thinking the TCP stack part of things is testable by regular FTP, as the slowest of the transfer methods. By slow I mean there is no reverse channel data to stimulate replies quicker than done by arriving file data. I'll stop here to keep the discussion a discussion, and when your data become available perhaps I will learn a thing or two. My problem, you see, is I approach these things rationally, what I would have done in their position, rather than what MS hacks together and declares to be revealed truth. Thanks, Joe D. 21-Nov-97 3:30:37-GMT,1340;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id WAA05284; Thu, 20 Nov 1997 22:30:35 -0500 (EST) Date: Thu, 20 Nov 97 22:30:35 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: Joe Doupnik Cc: FDC@watsun.cc.columbia.edu Subject: Re: Streaming - Draft 3 In-Reply-To: Your message of Thu, 20 Nov 1997 20:23:57 -0600 (MDT) Message-ID: Joe: I thank you for your patience. The MS TCP/IP stack defaults to a 8K in, 8K out set of buffers. I have tried increasing the window size but it does not help significantly. It only puts off the eventual problem. I will get you the logs as soon as I can. (Before the next NT 5 beta is shipped.) In the meantime, if you have Win95 or NT4, the latest cknker.exe (k95 beta) may be found in watsun in ~kermit/os2test/bean/cknker.exe. You can use that to test the performance differences between Streaming and non-streaming on Microsoft TCP/IP. - Jeff Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 21-Nov-97 3:54:13-GMT,2635;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id WAA08664 for fdc; Thu, 20 Nov 1997 22:54:13 -0500 (EST) Resent-Message-Id: <199711210354.WAA08664@watsun.cc.columbia.edu> Received: from mailrelay1.cc.columbia.edu (mailrelay1.cc.columbia.edu [128.59.35.143]) by watsun.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id WAA08193 for ; Thu, 20 Nov 1997 22:47:47 -0500 (EST) Received: from barney.usu.edu (barney.usu.edu [129.123.1.89]) by mailrelay1.cc.columbia.edu (8.8.5/8.8.5) with ESMTP id WAA14418 for ; Thu, 20 Nov 1997 22:47:46 -0500 (EST) Received: from cc.usu.edu by cc.usu.edu (PMDF V5.0-5 #11556) id <01IQ8V4OEC0W9EPFEG@cc.usu.edu> for jaltman@columbia.edu; Thu, 20 Nov 1997 20:47:38 -0600 (MDT) Date: Thu, 20 Nov 1997 20:47:38 -0600 (MDT) From: Joe Doupnik Subject: Re: Streaming - Draft 3 To: jaltman@columbia.edu Message-id: <01IQ8V4OEFSI9EPFEG@cc.usu.edu> X-VMS-To: IN%"jaltman@columbia.edu" X-VMS-Cc: JRD MIME-version: 1.0 Content-type: TEXT/PLAIN; CHARSET=US-ASCII Content-transfer-encoding: 7BIT Resent-To: fdc@watsun.cc.columbia.edu Resent-Date: Thu, 20 Nov 97 22:54:12 EST Resent-From: Jeffrey Altman Jeff, I have Win95 and NT 4 so we are in luck on that part. I'll wait for the packet traces as being more economical of my time rather than testing product to product for half a day initially. Plus I'm out of spare machines to act as a packet snoop on the same wire. Curiously, in class I'm discussing sliding windows flavors. The weekend lab project is to use my special packet zappers in IP and Kermit protocols to view how each deals with losses (go back to N vs selective repeat, etc). Tomorrow introduces pipe measurements, and packet versus byte counting, etc for general nameless protocols. Oh yes, for Frank. Saturday I may have netlab1 up and down a bit while I sort out some details in a beta lan driver from Intel. I'm discussing things with the authors of same. Outages will be to regen the box, hence a few minutes, and I work around existing connections. About 2/3 of the tries result in receive but can't send situations, and then I back away and put in a working configuration for awhile. Thanks, Joe D. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 21-Nov-97 3:55:45-GMT,1043;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id WAA08789; Thu, 20 Nov 1997 22:55:42 -0500 (EST) Date: Thu, 20 Nov 97 22:55:42 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: Joe Doupnik Subject: Re: Streaming - Draft 3 In-Reply-To: Your message of Thu, 20 Nov 1997 20:47:38 -0600 (MDT) Cc: fdc@watsun.cc.columbia.edu Message-ID: I don't expect you to spend half a day performing traces. Try a transfer once with streaming and once without just to prove that the performance difference exists. SET TCP ? may be used to adjust the send and recv buffers if you decide you want to play. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 21-Nov-97 15:17:29-GMT,768;000000000005 Return-Path: Received: (from jaltman@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA13853 for fdc; Fri, 21 Nov 1997 10:17:29 -0500 (EST) Date: Fri, 21 Nov 97 10:17:28 EST From: Jeffrey Altman Reply-To: jaltman@columbia.edu To: fdc@watsun.cc.columbia.edu Subject: streaming Message-ID: skipped files (ie, because of date) show up in the Error Count on the full screen display. Jeffrey Altman * Sr.Software Designer * Kermit-95 for Win32 and OS/2 The Kermit Project * Columbia University 612 West 115th St #716 * New York, NY * 10025 * (212) 854-1344 http://www.columbia.edu/kermit/k95.html * kermit-support@columbia.edu 21-Nov-97 15:34:54-GMT,3420;000000000005 Return-Path: Received: (from fdc@localhost) by watsun.cc.columbia.edu (8.8.5/8.8.5) id KAA18425; Fri, 21 Nov 1997 10:34:44 -0500 (EST) Date: Fri, 21 Nov 97 10:34:44 EST From: Frank da Cruz To: Joe Doupnik Cc: Jeffrey Altman Subject: Streaming Message-ID: Lively correspondence! Can't turn my back for minute... Well, as they say, the proof is in the putting (FTP pun). There are two ways to approach this; bottom-up analysis of many kinds of connections and TCP stacks, or top-down observations of performance. Both are fine, but you guys seem to have the former covered nicely, so I'll concentrate on the latter -- I'm mainly interested in refining the definition, getting it to work as advertised, and then doing a lot of measurements. If I can find situations in which streaming improves matters, then it will not have been a waste of time. After all, it is not the user's fault if the underlying transport is implemented poorly or behaves badly, and even less their responsibility to understand or fix it; Kermit has always tried to give them a way to overcome any nastiness below, and this could be just another way. Perhaps more interestingly, it might be used with IBM mainframe Kermit to achieve performance improvements that could not be easily gained any other way. In any case, it is another knob to turn, among many. "Kermit as lab" gets a new piece of equipment: declare \&s[2] off on set file display brief set control unprefix all set control prefix 0 1 255 set transfer bell off if open write close write open write bench.log for \%w 1 31 1 { set window \%w for \%p 480 9000 480 { for \%i 1 2 1 { set streaming \&s[\%i] remote set rec pack \%p if fail stop 1 {REMOTE SET FAILED: \&s[\%i] \%w \%p} echo S=\&s[\%i] W=\%w P=\%p send x if fail stop 1 {TRANSFER FAILED: \&s[\%i] \%w \%p} writeln file - \flpad(\v(cps),8) - \flpad(\&s[\%i],4) - \flpad(\%w,3) - \flpad(\%p,4) } } } close write Running this now, on fully loaded network and host computers, just the first few passes thru the loop (not surprisingly) show streaming to be way faster than not streaming (on a TCP connection Kermit-to-Kermit, no Telnet server; x is a 1MB Sparc executable). Here is the first bit of the log: CPS Stream W Pkt 33825 off 1 480 209715 on 1 480 <- six times faster 65536 off 1 960 349525 on 1 960 <- 5.3 times faster 69905 off 1 1440 209715 on 1 1440 <- 3 times faster 58254 off 1 1920 174762 on 1 1920 etc etc.. 95325 off 1 2400 349525 on 1 2400 80659 off 1 2880 349525 on 1 2880 87381 off 1 3360 262144 on 1 3360 69905 off 1 3840 349525 on 1 3840 41943 off 1 4320 524288 on 1 4320 <- 9.5 times faster! 55188 off 1 4800 349525 on 1 4800 58254 off 1 5280 209715 on 1 5280 43690 off 1 5760 349525 on 1 5760 52428 off 1 6240 262144 on 1 6240 <- 9.5 times faster... 27594 off 1 6720 Obviously we need more controlled tests (e.g. on a dedicated network with unloaded hosts, on serial connections, etc), as well as sniffer results. But at least we're having some fun for a change :-) - Frank