Cron jobs on the web server hosts

dayrot

Dayrot runs on each web server host. It uses orbit to rotate the web server log files (access and error logs), then it sends a HUP to the server, and uses gzip to compress the previous day's log file. We don't use the compress option in orbit because that takes a few minutes and we want the server to get the HUP right away, so we don't lose any log records. After the log files are compressed, copy-weblogs.pl is called to copy the logs to the appropriate subdirectory for the current week, and then remove the logs from the web server.

Cron jobs that run on a separate host

daily-logres.pl

This job processes the logs from multiple web servers, and it runs on a separate host so it will not affect the performance of our web servers. If your web server is configured to resolve IP addresses into host names (HostnameLookups on) then you don't really need to use logres. But we recommend that you use it anyway, to fix unmatched quotes, truncate really long fields, discard junk records, etc.

weekly-set-date.pl

The new week starts on Saturday morning at 4:30. The weekly-set-date.pl job must run on Saturday (any time before midnight) so that the new subdirectory will be created before logres runs again Sunday morning.

weekly-reports.pl

We run the weekly-reports.pl job on Sunday afternoon, to give the daily-logres.pl job from Saturday enough time to complete. Sometimes logres takes up to 35 hours (we have about a million hits per day). If your daily-logres.pl completes in less time, or if you're not using logres, you might want to schedule the weekly-reports.pl job for Sunday morning, instead.

showstats

After installing showstats in the CGI directory, you need to create the web form that will be used to invoke the script. A sample web form is here. Customize for your site, as appropriate

This page is still under construction.

---

Academic Information Systems 212 854.1919 consultant@columbia.edu
Research and Development Group
last modified on Thursday, 26-Feb-1998 19:38:06 EST