#!/usr/bin/perl -w

# FlowMonitor.pm - A module for flowscan designed to implement usage-based 
# rate-limiting. Keeps track of per-IP bandwidth usage (in terms of bytes)
# for a given day and assigns users to usage classes based on that usage

use strict;

package FlowMonitor;

require 5;
require Exporter;

@FlowMonitor::ISA=qw(FlowScan Exporter);

# convert the RCS revision to a reasonable Exporter VERSION:
'$Revision: 1.41 $ ' =~ m/(\d+)\.(\d+)/ && (( $FlowMonitor::VERSION ) = sprintf("%d.%03d", $1, $2));

=head1 NAME

FlowMonitor is a flowscan module designed to let you specify a bandwidth
usage policy per IP and obtain lists of IPs whose usage violates that
policy. These lists can then be used to do things like rate-limit or
black-hole the offending addresses.

FlowMonitor requires a SQL database backend. Initial revisions were written
with a filesystem backend, but this proved problematic with the usage our
campus was seeing (approximately 10k unique IPs in a 5 minute sample, approx
120k total IPs). We use Oracle, but any DBD module that works with Perl
should also work.

=head1 SYNOPSIS

   $ flowscan FlowMonitor

or in F<flowscan.cf>:
   
   ReportClasses FlowMonitor

=head1 DESCRIPTION

FlowMonitor.pm creates text file lists of IP addresses, one per line, of IP
addresses. Which IP addresses go into which file is determined by your
FlowMonitor.cf file (detailed below)

=head1 CONFIGURATION

FlowMonitor.pm reads FlowMonitor.cf, which should be located in the same
directory as your F<flowscan> binary.

=over 4

=item B<Subnet>

Each B<Subnet> entry in the file is an IP/length pair that represents a local
subnet. E.g.:

	# Subnet for main campus
	Subnet 128.59.0.0/16

Add as many of these as is necessary. FlowMonitor assumes that flows will
either have a source address in a one of the declared Subnets, or a
destination address there, but not both. (Ie, FlowMonitor should be used on
flows collcted from a boundary router whos traffic we are interested in
policing). FlowMonitor will keep records on the usage of all IPs it sees in
flow records whose source or destination IP is in a Subnet statement.

=item B<Router>

A B<Router> statement looks like:

	# Only interested in traffic from nyser-gw
        Router 128.59.1.4

You need to put a B<Router> statement in you FlorMonitor.cf for every
exporter through which you wish to police traffic. Note if you have more
than one Router, your Policy statement will refer to total bandwidth through
all listed Routers. Per-Router policies and violation files may be in a
forthcoming release, but for now, FlowMonitor enforces a single policy over
one or more exporters.

=item B<Policy>

A B<Policy> statement looks like:

	# Policy is limit upload to 100Kb/sec
        Policy Outbound 8640000000 86400

The first argument determines which bytes we count for an individual
IP. Allowed values for this argument are Inbound, Outbound, or Both. The
Inbound policy only counts flows whose destination address is contained in a
range defined in a B<Subnet> statement. The Outbound policy only counts
flows whose source address is defined in a B<Subnet> statement. The Both
policy totals bytes when either the source or the destination address is in
a B<Subnet> statement.

The second argument is the number of bytes allowed per IP. The third is
the number of seconds over which they are allowed to use those bytes. The
above policy says that over 86400 seconds (1 day), an individual IP address
is allowed to use 8,640,000,000 bytes (8.6Gb/day, roughly 100Kb/second).

=item B<DBName>

B<DBName> specifies the name and type of a database used to store per-IP
usage information in. That database should have a table in it created as

create table iplogs ( ipaddr varchar(16),
		      bytes number(16),
		      starttime date,
		      constraint iplogs_ct unique(ipaddr) );

making sure that the user FlowMonitor will be running as has permissions to
insert, delete and modify from this table.

	# Store records in oracle database acisora1
        DBName Oracle acisora1

The first parameter should be the name of the DBI module perl should use to
contact the database whose name is the second parameter. The SQL in this
module has been tested with Oracle and MySQL, but may need to be corrected
for your DB.

=item B<DBUser>

B<DBUser> allows you to set the username to try and connect to the database
specified in B<DBName> as. By default, no username is used.

=item B<DBPass>

B<DBPass> allows you to specify a password to use to connect to the database
specified in B<DBName>. By default, no password is used.

=item B<Violators>

B<Violators> statements are used to log violators of the policy. The syntax
of a B<Violators> statement looks like

	# People at 50% over policy get logged to 50percenters
	Violators 50 /var/log/50percenters
        # People at 100% over policy get logged to 100percenters
        Violators 100 /var/log/100percenters
        # People at 200% or more over policy go to losers
        Violaters 200 /var/log/losers

Each IP will only appear in 1 file, so someone at 100% over the policy (in
the above example) will *not* appear in the 50percenters file, only the
100percenters file. Anyone whose percentage of usage is higher than the
highest percentage in a Violators statement gets logged in the highest
percentage file.

=item B<Carry-Forward>

B<Carry-Forward> is a directive designed to help smooth out hosts going into
and out of violation every time their quota interval expires. If
Carry-Forward is set, an IPs usage is not eliminated every time their quota
interval passes, it is reduced by a Quotas worth of bytes. Ie, if your Quota
is 100 megabytes every hour, and IP 1.2.3.4 has 120 megabytes worth of usage
at the end of its hour, instead of starting the next hour with 0 usage, it
will start with 20 megabytes worth. If it ended its hour with 200 megabytes
used, it would start the next hour with 100 megabytes used, and be in
violation immediately.

Note that Carry-Forward will only carry one intervals worth of quota ahead,
so an IP will never start a new interval with more than 1 intervals
quota. To continue the above example, if 1.2.3.4 finished an hour with 300
megabytes of usage, it will still only begin the next hour with 100
megabytes of usage, as that is one intervals quota. This way of implementing
Carry-Forward was chosen because in an environment where IPs are assigned
dynamically, or more than one person can use a given IP (ie, a computer
lab), we did not want to unduly penalize a given IP for more than 2 quota
intervals. Note that if a given host is continually using more than 2 times
its quota every interval, it will always be in violation.

=item B<Ignore>

An B<Ignore> statement is a netblock that you do not wish to have listed in
the Violators files (because it is a known source of traffic, it is
legitimate, or whatever). These statements take the form

	# Ignore our webserver
        Ignore 10.0.0.1/32
        # Ignore machine room hosts
        Ignore 10.0.1.0/24

You may have as many B<Ignore> statements as you wish.

=item B<IgnoreList>

This statement just makes FlowMonitor write out a list of its currently
Ignored hosts to a text file once per flow. It looks like

	# Ignores to ignorelist.txt
	IgnoreList /var/www/html/ignorelist.txt

This file is truncated before writing; it is not a log file, it is a list of
the current ignores. If you do not include this statement, no list will be
produced.

=item B<Logfile>

A B<Logfile> statement provides a way to track who was in what file at what
time. It takes the form

	# Log bad people to /var/log/mon-history
	Logfile /var/log/mon-history

If you have no Logfile statement, FlowMonitor will not log when people enter
Violator groups.

=back

=cut

use Cflow qw(:flowvars 1.015);  # for use in wanted sub
use Net::Patricia;		# Fast IP/mask lookups
use FindBin;			# To find our executable
use Fcntl;			# Need some constants
use DBI;			# DB Interface!
use Socket;			# We need inet_aton

my($SUBNETS);			# A trie of internal subnets. IPs we 
				# should keep data on.
my(%ROUTERS);			# A hash of ip's we should 
				# police records from
my($INUSAGE);			# A trie for recording the usage of IPs 
				# we find in flow records for inbound data
my($OUTUSAGE);			# A trie for recording the usage of IPs 
				# we find in flow records for inbound data
my($IGNORE);			# A trie of ips we should not list
my($QUOTA);			# Bytes allowed in a given INTERVAL
my($INTERVAL);			# Time between resets of an IPs counter, in
				# units of seconds
my(%VIOLATORS);			# Hash keyed by %age of policy, whose value
				# is a filename to write IPs to
my($DBNAME) = 'dbi:';		# Which DBI module to use
my($DBUSER) = '';		# What username to connect as
my($DBPASS) = '';		# What password to connect with

my($SYSDATE) = '';		# What to select to get the current time
my($DATEUNITS) = '';		# Number of seconds in the units $SYSDATE
				# math is done in. Ie, if $SYSDATE - 1 is
				# 1 day before sysdate, then DATEUNITS will
				# be 86400 (seconds per day)

$FlowMonitor::SUBNETS = 
    new Net::Patricia || die 'Could not create a trie ($!)\n';
$FlowMonitor::IGNORE =
    new Net::Patricia || die 'Could not create a trie ($!)\n';
$FlowMonitor::QUOTA    = -1;	# Uninitialized
$FlowMonitor::INTERVAL = -1;
# Multicast address spec's, taken from CampusIO
$FlowMonitor::MCAST_NET  = unpack('N', inet_aton('224.0.0.0'));
$FlowMonitor::MCAST_MASK = unpack('N', inet_aton('240.0.0.0'));
$FlowMonitor::LOGFILE = undef;
$FlowMonitor::IGNOREFILE = undef;
$FlowMonitor::POLICY = 0;	# 1 = Outbound, 2 = Inbound, 3 = Both
$FlowMonitor::CARRYFORWARD = 0;	# Default: no carry-forward

$ENV{'ORACLE_HOME'} = "/opt/oracle-9.2" unless defined $ENV{'ORACLE_HOME'};
&parseConfig("${FindBin::Bin}/FlowMonitor.cf");	# Read our config file

sub new {
   my $self = {};
   my $class = shift;

   return bless _init($self), $class
}

sub _init {
   my $self = shift;

    # Clear this out
    $FlowMonitor::INUSAGE = new Net::Patricia || 
	die 'Could not create a trie ($!)\n';
    $FlowMonitor::OUTUSAGE = new Net::Patricia || 
	die 'Could not create a trie ($!)\n';

   return $self
}

sub parseConfig {
    my $file = shift;

    open(FH,$file) || die "Could not open $file ($!)\n";

    while(<FH>) {
	next if (/^\s*$/);	# Skip blanks
	next if (/^\s*\#/);	# and comments

	if (/^\s*Subnet (\d+\.\d+\.\d+\.\d+\/\d+)\s*$/) {
	    # Stick this entry into our trie
	    $FlowMonitor::SUBNETS->add_string($1);
	} elsif (/^\s*Router\s+(\d+\.\d+\.\d+\.\d+)\s*$/) {
	    $FlowMonitor::ROUTERS{$1} = 1;
	} elsif (/^\s*Policy\s+(\S+)\s+(\d+)\s+(\d+)\s*$/) {
	    die "Multiple policies defined in $file\n"
		if ($FlowMonitor::QUOTA != -1);

	    $FlowMonitor::QUOTA    = $2;
	    $FlowMonitor::INTERVAL = $3;
	    
	    if ($1 =~ /Outbound/i) {
		$FlowMonitor::POLICY = 1;
	    } elsif ($1 =~ /Inbound/i) {
		$FlowMonitor::POLICY = 2;
	    } elsif ($1 =~ /Both/i) {
		$FlowMonitor::POLICY = 3;
	    } else {
		die "Unknown Policy type $1 in $file ($.)\n";
	    }

	} elsif (/^\s*DBName\s+(\S+)\s+(\S+)\s*$/) {
	    $DBNAME = "dbi:$1:$2";
	    
	    if ($1 =~ /^mysql$/) {
		$SYSDATE = 'sysdate()';
		$DATEUNITS = 1;
	    } else {
		$SYSDATE = 'sysdate';
		$DATEUNITS = 86400;
	    }
        } elsif (/^\s*DBUser\s+(\S+)\s*$/) {
            $DBUSER = "$1";
        } elsif (/^\s*DBPass\s+(\S+)\s*$/) {
            $DBPASS = "$1";
	} elsif (/^\s*Violators\s+(\d+)\s+(\S+)\s*$/) {
	    die "Violator class $1 already exists (redefined on $.)\n"
		if (exists $FlowMonitor::VIOLATORS{$1});

	    $FlowMonitor::VIOLATORS{$1} = $2;
	} elsif (/^\s*Ignore\s+(\d+\.\d+\.\d+\.\d+\/\d+)\s*$/) {
	    # Add to exceptions list
	    $FlowMonitor::IGNORE->add_string($1, $1);
	} elsif (/^\s*IgnoreList\s+(\S+)\s*$/) {
	    # Log ignores to $1
	    $FlowMonitor::IGNOREFILE = $1;
	} elsif (/^\s*Logfile\s+(\S+)\s*$/) {
	    $FlowMonitor::LOGFILE = $1;
	} elsif (/^\s*Carry-Forward\s*$/) {
	    $FlowMonitor::CARRYFORWARD = 1;
	} else {
	    die "Invalid line $. in $file\n\t$_\n";
	}
    }

    die "No Policy defined\n" if ($FlowMonitor::INTERVAL == -1);

    close(FH);
}

sub perfile {
    # Only do this, so we get the filetime from our super-class
    my $self = shift;

    $self->SUPER::perfile(@_);
}

sub wanted {
    my $self = shift;
    my $hr;

    # Skip any extraneous flows
    return if (!(defined $FlowMonitor::ROUTERS{$exporterip}));

    # Can't really bill these :P
    return if 
	(($dstaddr & $FlowMonitor::MCAST_MASK) == $FlowMonitor::MCAST_NET);

    if ($FlowMonitor::SUBNETS->match_integer($srcaddr))	{ # Source inside
	if (!($hr = $FlowMonitor::OUTUSAGE->match_integer($srcaddr))) {
	    $FlowMonitor::OUTUSAGE->add_string($srcip, 
					    { address => $srcip,
					      bytes => $bytes,
					  });
	} else {
	    $hr->{bytes} += $bytes;
	}    
    } elsif ($FlowMonitor::SUBNETS->match_integer($dstaddr)) {	# dest inside
	if (!($hr = $FlowMonitor::INUSAGE->match_integer($dstaddr))) {
	    $FlowMonitor::INUSAGE->add_string($dstip, 
					    { address => $dstip,
					      bytes => $bytes,
					  });
	} else {
	    $hr->{bytes} += $bytes;
	}    
    } else {
	# Weird flow, neither source nor destination in a relevant Subnet
	return;
    }
}

sub report {
    my $self = shift;

    my %FILE;			# Usage db on disk
    my($ip, $time, $resettime);
    my(@classes);
    my(%bad);			# Sorry about that


    # set up stupid classes variable, sorting by reverse numeric order
    foreach (sort {$b <=> $a} (keys %FlowMonitor::VIOLATORS)) {
	push @classes, $_;
    }

    # Connect to the DB
    my($dbh) = DBI->connect($DBNAME,$DBUSER,$DBPASS,{ AutoCommit => 0});

    unless ( defined( $dbh ) ) { # did we get a database connection?
	warn "Coult not connect to $DBNAME\n";
	return 0; # skip out of reporting
    }

    my($cursor,$tmp, $newcnt,$oldcnt,$hr);
    my(@row,$allrows,$rowcnt);
    my(%TABLE,%new);

    %TABLE = ( );
    %new = ( );

    # For each IP we've seen, get current usage
    $cursor = $dbh->prepare("select bytes, ipaddr from iplogs");
    $cursor->execute || die "Could not execute: $dbh->errstr\n";

    ($allrows,$rowcnt) = $cursor->fetchall_arrayref;
    foreach $tmp (@$allrows) {
	$TABLE{ $tmp->[1] } = $tmp->[0];
    }
    $cursor->finish;

    my($updh) = 
	$dbh->prepare('update iplogs set bytes = ? where ipaddr = ?');
    my($insh) =
	$dbh->prepare("insert into iplogs (bytes, ipaddr, starttime) " . 
		      "values (?, ?, $SYSDATE)");
    my($trie);			# Which Trie to climb

    $oldcnt = 0; 
    $newcnt = 0;
    
    # Is our policy Both? If so, add OUTUSAGE to INUSAGE
    if ($FlowMonitor::POLICY == 3) {
	$FlowMonitor::OUTUSAGE->climb ( sub {
	    if (!($hr = 
		  $FlowMonitor::INUSAGE->match_string($_[0]->{address}))) {
		$FlowMonitor::INUSAGE->add_string( $_[0]->{address},
						  { address => 
							$_[0]->{address},
						    bytes => 
							$_[0]->{bytes},
						});
	    } else {
		$hr->{bytes} += $_[0]->{bytes};
	    }    	    
	} );
	$trie = $FlowMonitor::INUSAGE;
    } elsif ($FlowMonitor::POLICY == 2) { # Use Inbound only
	$trie = $FlowMonitor::INUSAGE;
    } elsif ($FlowMonitor::POLICY == 1) { # Use Outbound only
	$trie = $FlowMonitor::OUTUSAGE;
    } else {
	die "No policy defined??? $.\n";
    }

    $trie->climb ( sub {
	if (defined($TABLE{ $_[0]->{address} })) {
	    $TABLE{ $_[0]->{address} } += $_[0]->{bytes};
	    $updh->execute( $TABLE{ $_[0]->{address} }, $_[0]->{address} );
	    $oldcnt++;
	} else {
	    $TABLE{ $_[0]->{address} }    = $_[0]->{bytes};
	    $insh->execute( $_[0]->{bytes}, $_[0]->{address} );
	    $newcnt++;
	}
    } );


    if ($FlowMonitor::CARRYFORWARD > 0) {
	# Do carry-forward. Debit the next INTERVALs quota for excess usage 
	# for this INTERVAL

	# Make sure no one has no more than 2*quota 
	$dbh->do("update iplogs set bytes = " . 
		 $dbh->quote(2 * $FlowMonitor::QUOTA) . 
		 "where starttime < ($SYSDATE - " . $FlowMonitor::INTERVAL . 
		 " / $DATEUNITS) and bytes > " . 
		 $dbh->quote(2 * $FlowMonitor::QUOTA) );

	# Debit violators over-use to the next hour
	$dbh->do("update iplogs set bytes = (bytes - " . 
		 $FlowMonitor::QUOTA .") where " . 	
		 "starttime < ($SYSDATE - " . $FlowMonitor::INTERVAL . 
		 " / $DATEUNITS) " );
	
	# Get rid of all non-violators
	$dbh->do("delete from iplogs where " . 
		 "starttime < ($SYSDATE - " . $FlowMonitor::INTERVAL . 
		 " / $DATEUNITS) " .
		 "and bytes < 0");
	
	# Make all surviving records start now
	$dbh->do("update iplogs set starttime = $SYSDATE where " .
		 "starttime < ($SYSDATE - " . $FlowMonitor::INTERVAL . 
		 " / $DATEUNITS)");
    } else {			# No Carry-forward. Just delete old records
	$dbh->do("delete from iplogs where " .
		 "starttime < ($SYSDATE - " . $FlowMonitor::INTERVAL . 
		 " / $DATEUNITS) ");
    }

    $dbh->commit;

    $dbh->disconnect;

    print STDERR localtime() . 
	" Logged data on $newcnt new IPs, $oldcnt old IPs\n";

    # Now write out Violator files
    foreach(keys %TABLE) {
	# Skip ignores
	next if $FlowMonitor::IGNORE->match_string($_);

	if ($TABLE{$_} > $FlowMonitor::QUOTA) {
	    # Where should we write them?
	    my($ip) = $_;

	    my($ratio) = $TABLE{$ip} / $FlowMonitor::QUOTA;
	    $ratio = ($ratio * 100) - 100; # %age over quota
	    
	    foreach(@classes) {
		if ($ratio > $_) { # Write to the highest applicable one
		    $bad{$_} .= ":$ip";
		    last;
		}
	    }		    
	}
    }
	
    # Write out the %bad hash to the correct files
    foreach(@classes) {
	open(FH,">$FlowMonitor::VIOLATORS{$_}") ||
	    die "Could not open $FlowMonitor::VIOLATORS{$_} ($!)\n";

	foreach (split(/\:/, $bad{$_})) {
 	    next if (/^\s*$/);	# Skip blanks
	    print FH "$_\n";
	}
	
	close(FH);
    }

    # Log to a Logfile?
    if (defined($FlowMonitor::LOGFILE)) {
	open(FH,">>$FlowMonitor::LOGFILE") ||
	    die "Could not open $FlowMonitor::LOGFILE for append ($!)\n";
	
	my($class);
	foreach $class (@classes) {
	    foreach (split(/\:/, $bad{$class})) {	    
		next if (/^\s*$/);	# Skip blanks
		print FH localtime() . " - $_ - " . 
		    $FlowMonitor::VIOLATORS{$class} . "\n";
	    }
	}

	close(FH);
    }

    # Log ignores?
    if (defined($FlowMonitor::IGNOREFILE)) {
	open(FH,">$FlowMonitor::IGNOREFILE") ||
	    die "Could not open $FlowMonitor::IGNOREFILE for write ($!)\n";

	$FlowMonitor::IGNORE->climb ( sub { print FH "$_[0]\n"; } );
	close(FH);
    }
}

=head1 BUGS

It would be nice to have a file that contained the blocks to ignore. Then we
wouldn't need a separate IgnoreList directive.

Database handling is pretty messy. Currently, only have MySQL and Oracle
tested and working.

Carry-Forward should be more configurable.

=head1 AUTHOR

Johan Andersen <johan@columbia.edu>

=head1 REPORT PROBLEMS

Please contact <cuflow-users@columbia.edu> to get help with FlowMonitor.

=cut

1
