HOWTO Install OURMON - A Network Monitoring and Anomaly Detection System
------------------------------------------------------------------------

					Jim Binkley
					jrb@cs.pdx.edu

Version 2.8 Winter 2008

note: this document is mostly unchanged from the previous release.
Information about the new release is in the info.html help file.
Thread information is found in the info.html file.

This is an installation guide and a tutorial for the ourmon network
analysis system.  Please read the entire file in order to get a better
understanding of ourmon.  There are several sources for 
ourmon info, 

1. INDEX, this file,
and 

2. info.html in src/web.html which has information about the various
filters and runtime usage of ourmon.  

3. ourmon.conf - read over the supplied ourmon config file in etc/ourmon.conf.  
There is also a smaller config file for security only features (less user BPFs).

4. README.bsd and README.linux that attempt to give fast information
for installing dependencies before running configure.pl.

If you are a first time ourmon user, read this file.
If you are installing ourmon from a tar blob (not a package),
make 100% sure that pre-config dependencies are installed. 
then run configure.pl.  Note that the FreeBSD port will take
care of dependencies, however consult this document if problems occur.

Outline:

0. unpacking information
1. ourmon network setup overview
	a word of advice. 
2. ourmon files, layout, and parts
3. pre-configure dependencies
4. running configure.pl
5. post-install debugging/testing ourmon plus tricks
6. creating a new user/bpf graph/filter
7. optimizing ourmon.
	parallelism and minimization of packet loss.
	9. Linux front-end probe note.   
8. runtime params for ourmon front-end and omupdate.pl back-end
9. bugs, and debugging

--------------------------------------------------
0.  unpacking ourmon tar blob
--------------------------------------------------

Take the ourmon<version>.tar.gz tar blob, and place it in somewhere.
We prefer to make ourmon a user with no login permissions and place
it in /home/mrourmon, but you may choose to place it in
/usr/local/mrourmon or anywhere for that matter.  (All the internal
work directories are relative to the path BARRING the external
web directory).  We assume there is a web directory somewhere
with a web server like apache which is not part of this system.
/usr/local/mrourmon is another good possibility for installing ourmon.

For the sake of discussion we will assume it is placed in 
/home/mrourmon.

# tar xzvf ourmon<version>.tar.gz 

gives you:

drwxr-xr-x root/wheel        0 Apr 15 12:40 2004 mrourmon/
-rw-r--r-- root/wheel      131 Apr 15 12:24 2004 mrourmon/makeclean.sh
-rw-r--r-- root/1003     49117 Apr 15 10:48 2004 mrourmon/INSTALL
-rw-r--r-- root/1003      1009 Feb 17 14:28 2004 mrourmon/README
-rw-r--r-- root/1003      3735 Feb 21 19:50 2004 mrourmon/CHANGES
... etc.

You should unpack the tar blob so that it goes in the directory of choice.

The ourmon system is self-contained in the mrourmon directory
and does not install/configure parts of itself elsewhere barring
 
1. root crontab extensions, which you may or may not choose to install directly
from configure.pl ( But you MUST install them via crontab somehow to
have ourmon work).  The crontab extensions drive the back-end which
makes graphics, reports, and log entries.

2. init.d/rc.local or equivalent kernel boot startup entries for 
running the front-end at boot.

and 

3. a web directory that is accessable via a web server like apache.  
Some possible examples include:

/usr/local/www/data - on FBSD with apache 1.3
/var/www/ourmon - recent Ubuntu linux release with apache 2.X release.

In other words, make sure you know where on your target system, and web
server offer up ourmon files.

The config script will create various directories INSIDE the mrourmon
home directory during the configuration process including a bin
directory for executables and several directories for logging info.   
Unpacking the tar archive does not make files at runtime outside that directory
or the web directory.  Config files, log files, rrddata files, and
executables all live in this directory.  (Linux could be /home/mrourmon,
however on FreeBSD, this is actually /usr/home/mrourmon).  There
is more information on the mrourmon home directory layout below.

You may as a matter of administrative convenience create a mrourmon
user so that cd ~mrourmon will work.  As a first cut, such a user
should have a disabled password, and a shell entry in the password
file of something like /nonexistent or /sbin/nologin, or /dev/null.

E.g, (from /etc/passwd on FreeBSD)
mrourmon:*:1002:1002:mrourmon:/home/mrourmon:/nothing

If you are optimizing ourmon to install the front-end and back-end
on separate hosts, still follow this procedure.  It is very convenient
to keep ourmon in the same directory on any box you set it up on.

--------------------------------------------------
1.  ourmon network setup/overview
--------------------------------------------------

Our current DMZ setup looks something like this:


		|  gigabit ethernet
	----------------
	| Border Router|
	----------------
	        | gigabit ethernet
	---------------------
	| Ethernet Switch    |
	--------------------- 
	| | | ...           | promiscuous mode port
	  |		---------------------------------	
	  |		| P4 Intel/FreeBSD 5.4 Ourmon box|
	  |		---------------------------------	
	  |		
      ------------- non-DMZ-subnet
		|
		|  
		|
	---------------------------
        | ourmon graphics back-end | 
	---------------------------
 
We are seeing about 55k pps at peak times.  Our current front-end
probe box is a 3ghz P4 hyper-threaded Intel PC running SMP 5.4 BSD.
In high packet processing environments, hyper-threading is a good
thing.  A dual cpu would be a good thing too.  This lets the kernel
read packets and possibly process interrupts to do so, and it lets
the ourmon probe run in parallel with the kernel.  Otherwise they
contend for one CPU.

We assume that the front-end ourmon probe runs in promiscuous mode
(which ourmon turns on, users don't do that) inhaling packets coming
from an Ethernet switch configured to do port mirroring.  You may
of course run ourmon on an end system without port mirroring, but
in a modern switched network it will only see packets headed for
the host itself.  Ourmon has two parts, a front-end probe, and a
back-end graphics engine.  The front-end program is called "ourmon",
written in C.  The back end programs are mostly perl. 
The biggest programmatic part in the backend is called "omupdate.pl", and it
uses the RRDTOOL perl library.  At this point the backend is driven
from a shellscript called omupdate.sh which calls omupdate.pl
and other perl programs to do the work.  

It is possible to run the front-end and back-end on the same box.
We split up the work, because we desire to minimize dropped packets
on the front-end, and centralize network graph producing software
on a more protected back-end.  Our front-end runs in our DMZ, and
we have a non-DMZ graphics engine host all running FreeBSD (currently
5.X).  In general ourmon should run on the fastest hardware you can
get.  Also the front-end by itself on a box is happier on a dual
SMP CPU of any sort, even an Intel hyper-threaded CPU (interrupts
get a CPU to themselves).  This is because interrupts and the ourmon
application both get a CPU to themselves and do not have to contend
for one CPU (contention means the interrupt side of the kernel  
wins and the application loses - this can lead to serious amounts
of dropped packets.).

The Ethernet switch must be capable of having a port configured to
do "port mirroring", or what Cisco sometimes calls SPAN (or RSPAN); i.e., you
tell it to take packets coming in (or going out) some other ports
and mirror them on the promiscuous mode host port.  Thus your PC
can see all the network packets headed through the switch or at
least the packets you want it to see.  This PC is a typical "snooper"
device as a consequence.  Ourmon's front-end is thus setup in the
same way snort or ntop might be setup.  You may want it to have two
Ethernet cards, so that one card is dedicated to sniffing, and one
is used for access to the PC itself.  We are going to leave this
sort of setup consideration to your own judgement, however.  

Another possible architecture for the probe could take advantage
of Linux or BSD Ethernet drivers that support 802.1Q VLANS
in trunk mode.  We will not get into details here, but the front-end
probe basically looks for 802.1Q format in packets and if found "strips it".
This allows the probe to be run under a promiscuous mode interface
that has more than one VLAN's packets coming out of it.   This is a very
useful trick as it can allow a set of servers to be hooked up to
different vlans on one switch and yet still be monitored by one probe.

A word of advice:

We recommend the fastest computer you can lay your hands on for the
front-end box.  Ideally the front-end ourmon probe runs on a computer
by itself.  Said box should not be a router.  It should not run snort
(great zot ... snort needs its own box, as you cannot afford to miss
a single packet with snort, as it may be the packet with the new worm
in it).  Do not complain if the front-end ourmon probe eats the CPU.  
Optimizing ourmon is a way of life actually.   

The ourmon front-end is CPU intensive.  It is a good idea to dedicate
a box to it.  The back-end always needs to run to completion in 30
seconds for 30-second reports obviousally.  The current back-end
will complain to the event log if it cannot complete in 30 seconds
(get a faster computer).  If this happens, get a faster computer
or do less. DNS reverse resolution is not possible in the 30 second
sample period but it is used by back-end summarization programs
that can take one hour to run.

The challenge here is to have enough CPU power that you don't start
dropping packets.  Of course how fast the computer should be depends
on the speed of the interface you are trying to measure.  If you
have a T1 or DSL/cable modem connection, you may not need such a
fast CPU.  The supplied and default "pkts" filter shows BPF packet
counts and drops in packets per second.  It is the most important
filter.  Although ourmon is NOT snort and is not looking for attacks
in individual packets (i.e., where one missed attack packet can
kill you because the new day zero flash worm in the packet is not known
anywhere), it is still important that you minimize packet loss.
Some may occur, and just how much packet loss you can tolerate is
up to you in the final analysis.  Less is better.  See the optimizing
section below for more information on avoiding dropped packets.

rule of thumb:

In an institution of our size in our DMZ (say with 50k pps)

One good reason for minimizing packet loss is that 

IF 
	you do not ordinarily lose packets with the basic pkts/drop graph.  

THEN
	packet loss in the pkts/drop graph by itself
	is a useful indicator of BIG anomalies (DOS attacks)

ourmon philosophy:

If you are dropping packets, you may be trying to do too much.  If
you are NOT dropping packets, you could stand to be doing more :->.

We do not intend to tell you how to configure a FreeBSD kernel.  
However the probe-box/kernel MUST HAVE the bpf pseudo-device enabled.  
This is not a problem with Linux usually.  Linux has a much more 
serious problem as a probe box and although our system works with Linux 
both as a probe and graphics display systems, we do not recommend
Linux at this time for the probe.  See the optimization section
below for a brief discussion here.  

Another rule of thumb:

If tcpdump works, you can install ourmon.

-------------------------------------------------------------------
2. ourmon files, layout, and parts
-------------------------------------------------------------------
Ourmon as unpacked will have most of the following directories and files 
(Files marked with * are created by the configure.pl configuration utility):

INSTALL - this file
README - well, README ...  
ACKs - who has contributed to ourmon
CHANGES - new features, and changes over time according to version.
VERSION - version number
bin* - all runtime executables live here, ourmon, ourmon.sh, omupdate.sh, omupdate.pl, etc.
	configure.pl installs and creates scripts here.
	Executables are compiled and installed here as well by configure.pl

configure.pl - configure script for compiling, installing ourmon front-end
	and graphics-producing back-end.  Can install front-end, back-end separately.

etc - the ourmon config file ./etc/ourmon.conf lives here.  
	The front-end system will use this at runtime.  Read it and understand it.
	You may add your own new BPF set filters as well.
	Note ./etc/crontab.sample in here is an example of what goes into the root
	crontab to make the back-end run.

	There is also a "security filters" only config file called secmin.conf
	that has no user BPF filters in it, no topn flows, no topn ports.
	It does use CBPF filters.  The goal is to provide a security-oriented
	filter file that minimizes network monitoring features and may
	be useful in a more highspeed packet environment.  (The main
	supplied static web page needs to be customized by you ... simply
	get rid of parts of it that don't produce data).

logs* - ourmon log data is stored here.  all logging files that are NOT stored
	in the web directory (direct access reports) are put in here.
	Many of the files in here are used for hourly summarization.  Roughly
	topn filters and other kinds of filters have a week's worth of data stored here.
	See info.html and below for more information.  

rrddata* - directory for storage of rrd database files (.rrd files).
	Filename is based on ourmon.conf filter label/tag. The rrdtool runtime
	error log called "ourmon.log" (back-end error log) is stored
	here.  

rrddata*/ourmon.log - runtime back-end error log for rrd internal errors.
	If there is some problem with RRD picture generation (and omupdate.pl
	is executable), look here first.
	--------------->  ourmon.log <-------------------------------------
	is important.  You may have to muck with the rrdtool setup (in etc/ourmon.conf,
	or simply throw out a .rrd file) if ourmon.log indicates errors in rrd runtime
	generation.  omupdate.pl does all back-end rrd work.  if the rrd library
	encounters errors, they are placed in ourmon.log.  

tmp* - runtime files produced by the front-end including mon.lite and
	the optional tcpworm.txt are stored here.  These are the files that communicate
	data between the front-end and the back-end.  

scripts - misc. sample scripts that may be of use.  these are some useful
	scripts that for example may be used to make sure the front-end is always
	running.

src - src code for frontend and backend and supplied web pages as well.

src/ourmon - C src code for the front-end ourmon sniffer.
src/web.code - C and perl code for the back-end picture making, report generating, 
		and analysis programs.
src/web.html - starter html files for the back-end web display system.

web.pages* - usually you tell the ourmon configure.pl script where
	to store the back-end graphic output and reports in a web directory.
	E.g., perhaps you are running apache and you have htdocs
	set to /usr/local/apache2/htdocs.  ourmon files could be
	configured to be stored in /usr/local/apache2/htdocs/ourmon.
	web.pages is simply a symbolic link set to whereever you
	store your web pages, and exists for administrative
	convenience.  

During configuration, certain programs are created and installed
in the bin directory. Also certain directories are created in the BASE directory
(say /home/mrourmon).  The directories created include bin, logs, rrddata, tmp as outlined above.

Certain files will be produced as the ourmon system runs.  We will give a brief overview here.
There are more details in other sections below.

Let's look at ourmon from this point-of-view:  

1. front-end sniffer, 2. back-end grapher/reporter, 3. back-end top of top reports and analysis tools

The front-end has just a few files associated with it.  The back-end
has MANY files which can be broken down into three categories:
	1. static web pages in the web directory (html)
	2. generated web reports (ASCII)
	3. generated image files (PNG)
	4. log files (ASCII).
The back-end logically consists of a 30-second snapshot, and hourly snapshots
which may be called top of top or summarization reports.  Effectively
we have 30 second reports and hourly reports for the day.  The latter
is rolled over into one week's worth of daily reports every day at midnight.
Logging of raw data is kept and is used in the summarization process.  Summaries
made by the system are put in the web directory.  mrourmon/logs data is raw,
and although it may be analyzed by a knowledgeable human, is not analyzed by the system.

1. front-end probe/sniffer

~mrourmon/bin/ourmon.sh - this is a shellscript "driver" used to
	package parameters and simplify the boot of the front-end ourmon executable when
	a system reboots or when you change ourmon.conf.  
	You can use it to manually start or stop the ourmon probe.  You
	might also modify it to change the interface ourmon uses or to
	supply BPF expression parameters to the ourmon probe (the default
	expression is no expression (NULL) meaning read all packets).
	E.g., one might tell the BPF to ignore certain packets (use not in the expression),
	which will cause the kernel to ignore the packets.  This is not done by default.

	It should be called from /usr/local/etc/rc.d/ourmon.sh or the Linux equivalent
	(e.g.,  possibly /etc/rc.d/rc.local) to make sure that the ourmon front-end always runs at boot.  
	This file is created at configuration time depending upon input
	parameters.  It may be used to start stop or restart ourmon.
	E.g.,

		./ourmon.sh start
		./ourmon.sh stop
		./ourmon.sh restart

	of course, kill -9 or killall ourmon can be used in a pinch.  kill -9
	does not hurt ourmon or do anything bad in terms of system state.

~mrourmon/bin/ourmon - front-end C program called "ourmon".  ourmon will
	be built by the config script and is a compiled C executable.
	Parameters to ourmon are packaged in ourmon.sh.

~mrourmon/etc/ourmon.conf - input configuration file to ourmon.  This
	file specifies the filters ourmon uses.  You may modify this file with the 
	programmable BPF mechanism to create your own filter sets
	(a set of BPF expressions that produce one set of RRDTOOL graphs
	and one rrd) looking at local ports/hosts/networks
	and combinations of the same.  See ourmon.conf for examples.
	You can put all the filters in (by default) or take some out
	if for some reason you don't want them.

~mrourmon/tmp/mon.lite - output file created every 30 seconds by ourmon.
	As ourmon sniffs, every 30 seconds it creates this file,
	which store byte or packets for the set of named filters 
	found in ourmon.conf.  This file in turn is fed 
	to the back-end for the creation of web output, and logging.
	Thus this file is the output for the front-end and the input
	for the back-end. 

~mrourmon/tmp/tcpworm.txt - This is an feature of the topn syn list mechanism
	which gives the topn syn scanners according to the most SYNS seen during
	the sample period.  topn_syn in ourmon.conf enables the topn syn list.
	In addition you can ask the front-end to generate a "wormfile" called
	tcpworm.txt in a supplied directory, which by default is /home/mrourmon/tmp. 
	(Note: topn_syn_wormfile must be put in the config to cause this feature
	to happen, but nowadays the -D switch to the probe ourmon executable causes
	ourmon to put all output files in one directory.  The moral is don't muck
	with the filename in the config file, but you can take the config line out
	if you don't want the feature).
	E.g., in ourmon.conf, you find something like:

	topn_syn 60
	topn_syn_wormfile /home/mrourmon/tmp
	topn_syn_homeip  10.1.0.0   255.255.0.0

	topn_syn_wormfile only works if topn_syn is specified, and tells the front-end
	where to put the tcpworm.txt file, which captures and stores individual IP syn
	scanners and related TCP control counts as a front-end report.

	topn_syn_homeip should be set to your internal network with its netmask,
        ***********************************************************************

	as this is used to distinquish attacks from outside versus attacks from
	the inside in the web tworm.html page.  configure.pl prompts you to do this
	but it is a good idea to check the ourmon.conf file and make sure it is correct.

~mrourmon/tmp/p2pfile.txt - this is also a feature of the topn syn list mechanism
	and gives the set of hosts seen using various P2P apps including
	Bittorrent, Kazaa, Gnutella, and Morpheus/Livewire as well as IRC using hosts.
	It is enabled by the following syntax in ourmon.conf:
	
	topn_syn_p2pfile  /home/mrourmon/tmp

~mrourmon/tmp/irc.txt - this file includes additional information about the set of
	hosts using IRC and the IRC channels.  It is enabled with the following
	ourmon.conf syntax.  IRC information is correlated with syn tuple
	information and may be useful in the detection of IRC channels used by bots
	if those channels are full of hosts that are doing scanning. 

	topn_irc 0
	topn_irc_file /home/mrourmon/tmp

It should also be pointed out that ourmon post boot makes syslog
entries.  Look in /var/log/messages to see if it booted ok.  BSD users
should also note the BPF sizes message.  See the optimization hints
section below for more information on that subject.

2. back-end grapher/logger

The back-end consists of a set of static html web pages, and several
perl and shell scripts including "omupdate.pl" which is the main
back-end driver for 30 second updates of web information.  omupdate.pl
(using RRDTOOL) produces graphics (png files).  There are also files
that are generated and used by these programs and it is important
to understand their functions in order to install the overall ourmon
system and get it working.

From the back-end point-of-view, there are several kinds of graphs and
outputs: 

1. RRDTOOL based graphs.  There are several cases where RRD png
files are generated including the user BPF filter sets and other cases as well.
Typically the front-end supplies stats to the back-end
and omupdate.pl uses the RRD library to generate dynamic PNG graphs,
which are accessed on the web either via supplied .html pages or
new .html pages created by you.  omupdate.pl is the only application
in the back-end that uses RRDs.

2. histograms made for top N talkers.  These histograms are created
more or less by the creation of dynamic web pages that are assumed
to be hooked to the main html page (index.html).  We use iframes
and precreated "bar" graph png images.  All top N histograms use this
system.  For example the top N ourmon ip flows, tcp flows, udp flows,
and icmp flows use this system.


There is an important distinction in terms of baselined data.  RRDTOOL graphs
of course consist of the traditional, day, week, month, year graphs and
are based on the MRTG tradition.  In order words, one year of data
is baselined in the rrds, and shown graphically in the web directory.  
The histogram mechanism is totally different.  All of the topn filters
in the front-end produce: 1. a histogram of the top N sources (this
may be 1 to N pictures depending on the filter configuration, with
typically 10 or so bars per graph), and 2. "syslog" style logging
info (if logging is enabled) in various files in the /home/mrourmon/logs
directory.  So one can see RRDTOOL has a year of baselining (in rrds
and in pictures), and top N filters have one week of baselining in
syslog-style text data.  Some but not all of the top N mechanisms
which produce histograms have perl scripts that run on the hour
and produce summaries.  In a sense RRD summarization then is built
into the RRD graphs.  top_n filters have histograms for the "now",
30-second view, and (mostly but not always) perl scripts for ASCII
hourly graphs which may be rolled over into daily summaries.
Roughly the best case here for logged data is about a week.

3. web-summarization:  These are basically ASCII reports placed
on the web.  They come in two forms, 30-second summarizations (now),
and hourly (and eventually daily) reports.  Some top_n data tuples 
are summarized.  In some cases 30-second reports are generated
(e.g., the syn tuple produces a portsignature report useful
for worm attack analysis "now").  In some cases hourly and hence
daily reports are produced.  For example, irc produces 30-second, hourly
and daily reports.  Usually we find that irc data for "now" is
not useful (not enough context), but it is useful over hours and days
and may reveal botnets.

4. There is also additional logging information which is stored in
the logs directory.  The logs directory is not available over the
web.  You must go to it and typically use grep to look things up.
Some of the hourly summarizations are produced from it (and placed
on the web).  Topn logs have their own files.  mon.lite and tcpworm.txt
front-end files are also stored in their own directories.  Also
both TCP and UDP portreport.txt output files are stored in their
own directories.

Log files may be useful especially when there is no web summarization.
Logging is in general driven by the back-end.

back-end drivers and programs include:

~mrourmon/bin/omupdate.sh - this file is installed and may be
	slightly modified  by configure.pl depending on user
	parameters. It packages parameters to omupdate.pl and other
	scripts.  It may be changed by the user to copy the front-end
	files mon.lite and tcpworm.txt from a front-end system to
	a back-end system.  (We typically use wget to do that and
	put a web-server on the front-end back, constrained by ACLs
	so that the back-end box is the only system that can fetch
	front-end outputs).  Note that we assume this script is
	from crontab once a minute.  Since ourmon runs at a 30
	second interval, and this file is called from crontab
	once per minute, work in this file has the following
	structure:

		loop thru twice
			do task A
			do task B, etc.
		sleep 30

	thus omupdate.sh runs once a minute, and makes omupdate.pl
	and other scripts run twice per minute.  It also complains
	to the ourmon event log if the time exceeds 30 seconds.

	If you need to test omupdate.sh, make a copy of this by hand,
	throw out the loop part ... and simply run it once by hand to
	see if omupdate.pl is complaining about something. 
	(omupdate.pl also has a debug switch which must be set in the perl code).  
	ourmon will not suffer any long lasting corruption problems if
	you run omupdate.sh by hand.  This is a good thing to do for
	initial testing post a new install to make sure everything
	is ok.  In fact you don't need to change the script or make
	a test copy of it.  Just run it like so:

	# sh -x omupdate.sh

	and see if the outputs include any complaints.
	
~mrourmon/bin/omupdate.pl - perl script run via bin/omupdate.sh, 
	This is the main-back end driver and it
	updates the graphics via RRDTOOL and creates
	topn pictures.  omupdate.pl uses mon.lite as input and
	creates rrd databases, and updates rrd databases,
	and makes .png files placed in the web directory.

	It creates new rrds on demand, but never deletes them.
        ******************************************************

	It also does top N logging in the logs directory.
	It does NOT back up mon.lite or tcpworm.txt, which must be 
	done elsewhere (omupdate.sh).

~mrourmon/bin/tcpworm.pl - this program is a separate program for
	processing the tcpworm.txt front-end report.  This file
	takes any topn_syn tuple output file from the front-end and
	processes it in various ways making either 30-second "port
	signature report" format files (sorted by ascending IP
	address) or summarization files of various forms that provide
	a statistical average of various interesting counters and
	weights.  For example, when/if the front-end spits out
	tcpworm.txt, as a raw syn dump, then tcpworm.pl can produce
	two kinds of output files for that including the port
	signature report and a longer more verbose version of the
	port signature report (that can be viewed as tuple translation
	from internal counters to something more human readable).
	tcpworm.txt also updates the event log stored in the web
	pages, and the database used to capture new port signatures
	which is stored in the logs/tworm directory.  tcpworm.pl
	also produces hourly summaries of port report data.  See
	below for more information.  This information can be quite
	useful for detection of automated scanning.

~mrourmon/bin/irc.pl - this program is a separate program for processing the
	front-end irc.txt tuple lists.  It produces reports of IRC information
	and places them in the web directory at both 30 second sample periods
	and at hourly periods for IRC summarizations.  The front-end provides
	information on the set of IPs bound to a channel name, and statistical
	counts of basic IRC message types.  ourmon does not look at IRC
	message data.  It does however correlate IPs to channels and channels
	to TCP syn anomalies.

~mrourmon/bin/wormtolog.pl - this is a small perl program that takes 
	the input tcpworm.txt file and converts it to a "log-like"
	file in the top n logs directories.  Thus the tcpworm.txt
	list of IP addresses is converted to a topn like log, and
	the ombatchsyn.pl report generator works on both the topn_syn
	log and the worm log as their internal tuple format is the
	same.  This is used to produce the allworm_today.txt file,
	which gives an average of the portreport.txt data sorted
	by average work weight over the day.  Essentially IP hosts
	are ranked by the number of TCP syns generated and the work
	weight (used in the port signature report) is also computed
	as a statistical average.  This is essentially a big SYNNER
	report -- and is different from summarization reports
	produced by tcpworm.pl which tends to sort things differently
	(max number of instances that an IP appeared in 30-second
	samples).  Think "BIG SYNNER" for the allworm_today.txt
	file.

~mrourmon/rrddata/*.rrd - hardwired and BPF filters RRDTOOL rrds are
	stored here.  omupdate.pl both creates the rrd databases
	and then updates them.  An error file called ourmon.log is
	put here, in case the RRDTOOL perl library is not happy
	about information passed to it by omupdate.pl.

~mrourmon/tmp - the backend finds the front-end output files here:
	including mon.lite and other files.

~mrourmon/logs - omupdate.pl and omupdate.sh log various files 
	in this directory.  When configure.pl runs, it uses a
	subscript called bin/mklogdir.sh to create needed log
	directories for ourmon.  This is done in the base build
	directory and has the form:

	BASEDIR/mrourmon/logs

	in the log directory one finds the following log files:

	1. topn talker directories:

	Sun Mon Tue Wed Thu Fri Sat  topn_today

	topn talker tuples are stored here on a per day basis.  
	The symlink topn_today is always reset to the current day.  

	E.g.,

	bash-2.05b# ls topn_today
	icmperror_list.log      syn_list.log            topn_icmp.log           udp_ports.log
	ip_portscan.log         syn_worm.log            topn_ip.log             udp_portscan.log
	ip_scan.log             tcp_ports.log           topn_tcp.log            udperror_list.log
	scanner.db              tcp_portscan.log        topn_udp.log

	The above listing shows per topn log files for the various
	topn filters as well and also includes the daily "scanner.db"
	used to remember scans seen previously that day (only new
	scans are put in the scanner log).   The scanner db database
	is created and used by tcpworm.pl to tell if a port signature
	is "new" or old (seen previous) on a daily basis.  New
	information in here is reported in the web page called:
	newportsig_today.txt, which is found on the front index.html
	under the rubric: "new port signatures (TCP scanners) for
	today".

	syn_worm.log is a log like file produced by wormtolog.pl
	and used for a top of the top report that essentially is a
	statically averaged portreport.txt report.  This is used
	to produce the BIG SYNNER report previously mentioned.

	In addition, back end code logs complete ourmon files into
	various sub-directories again according to the day of the week.
	For example, front-end mon.lite files are logged. Each day
	is also linked to a "today" file so that one can easily
	find today's log.  These files are rotated at midnight.
	
	Each file is stored with the date prepended in order to make 
	both makes its time obvious and to make each file unique in 
	terms of its name. 

	See info.html for more information on logging.

HERE

~mrourmon/bin/daily.pl - this script is part of the logging system,
	and is run out of crontab.  It runs at midnight and initializes
	the next day log directories for new logging output.

mklogdir.sh - this script can be used to recreate the logging 
	directories from scratch.  It should be run first, and then 
	daily.pl should be run once.  It is not run on a daily basis.

~mrourmon/bin/monbackup.pl - this program is used by omupdate.sh to back up various 30-second reports
	and log files into the logging system.

And somewhere else on your back-end system, you have a web browser,
likely apache.

/usr/local/apache/htdocs/ourmon/*.html - 
	Various starting html files are placed in this directory
	by configure.pl.  Also dynamically created .png files 
	are stored here by omupdate.pl every thirty seconds.
	Logging summarization reports are also stored here as .txt files.

	index.html is the top-level "glue" html file that gives a
	daily view on back-end MRTG-like html files.  Various html
	files for grouping hardwired and BPF filters exist.  You
	may have to modify these or add to them as you add filters
	(or simply choose to not use filters and remove features
	you are not using). Post-installation, you should change
	and debug the .html files as needed.  Your job is to install
	a web server, and configure.pl will install files in a web
	directory.  NOTE: configure.pl will create the web directory,
	as it needs to know where it is.

3. back-end top of top reports and analysis tools

Creation of runtime summarization or top of top reports are made
hourly for the current day, and then rotated at midnight for a
week's worth of top of top reports on the web page.  These reports
are called web summarizations and are put in the web directory.

3.1 top of top reports

Top of top reports refers to reports made by perl and shell code
automatically run out of crontab placed in the web.pages directory
for ourmon.

There are two principle scripts that work together to drive the 
summarization process:

	1. batchip.sh - run hourly
	2. batchipall.sh - run daily (mostly simply rotates the last daily hourly file
		into the previous day, etc... day-1 to day-2, for a week).

Here is how this sub-system works:

	batchip.sh uses perl scripts (ombatchip.pl, ombatchipsrc.pl,
	ombatchsyn.pl irc.pl and tcpworm.pl) to analyze several sets of topn logs: 

	topn_ip: top N ip flows (all flows)
	topn_udp: top N udp flows
	topn_icmp: top N icmp flows
	topn_syn: top N syns in terms of IP source addresses. 
	rawirc: raw irc front-end probe files
	rawp2p: raw p2p day
	etc...

	This is done per hour on the current day.  The perl scripts in 
	question work like this:

	ombatchip.pl - sort the flows according to which flow generated
	the most bytes so far on that day.  Note the # of instances the flow
	was seen, and record its starting and stopping time (first time seen
	and last time seen).

	ombatchipsrc.pl - similar to above, but sort according to ip source
	and ip dst, thus it attempts to tell us who the top ip src and ip dst
	addresses were in terms of flows.

	Important note:  topn_icmp is often very good at showing persistant
	TCP and UDP scanning, as scanners often generate a long term trace
	of ICMP errors, which may not show up "now" (e.g., in the topn icmp
	histogram graph) but very well may show up in the daily log.  Thus
	this is an important tool for watching for scanning attacks.

	ombatchsyn.pl - analyzes the topn_syn log and produces a daily summary
	of that log in a fashion similar to that produced by ombatchip.pl.
	It sorts on syn counts, but the weight algorithm result is also shown
	(the same sort of weight calculation as used in the topn_syn histogram
	graph is also shown).  This program also takes the tcpworm.txt file
	(via a sneaky trick performed at 30 second sample time by wormtolog.pl),
	and produces a weighted average of the portreport.txt file.  The default
	weight is 100%.  Thus anomalous syn sources (IP sources) who have average
	work weights greater than or equal to 100% will appear in the allworm_today.txt
	report page on the web. (info.html calls this the hourly summarized port signature
	analysis report).

	irc.pl - takes raw irc files and produces an hourly summarization with many
	subcategories including global stats, channel stats, and host stats.
	servers are identified and so are "evil channels".  An evil channel
	is roughly a channel where a majority of the IP hosts seem to be doing
	SYN scanning based on the TCP work weight.

	batchipall.sh is run near midnight and simply takes the previous day's
		batch output in the web directory, and rolls it back one day.
		8 days worth of reports are kept in the log directory. 
	E.g.,
		topn_today.txt is the report for today for topn_ip.
		topn.0.txt is for the previous day.
		topn.1.txt is for the day before that ... etc.
		...
		topn.7.txt is the last day, roughly one week old.

	All the reports in the web directory end in .txt.

~mrourmon/bin/batchip.sh  - driver for hourly log report based on topn_ip.log
	information.  It produces a sorted list of the top IP flows,
	as well as IP src, IP dst, ICMP, UDP flow, syn reports, ETC.  The reports
	are stored in the web directory.    For example, the top
	of the top hourly report is named: topn_today.txt.  In general, you
	access the web reports via index.html which is supplied.

~mrourmon/bin/batchipall.sh - run once a day, it takes files like topn_today.txt in
	the web directory and rotates them back one day according to the scheme
	outlined above.

~mrourmon/bin/ombatchip.pl - perl script called by above shellscript to generate 
	top of top sorts.  This works on topn IP flows (all flows
	tcp/icmp/udp, etc).  We currently are not doing top TCP though.

~mrourmon/bin/ombatchipsrc.pl - perl script called by above shellscript to generate 
	top of top sorts based on ip src and ip destination.

~mrourmon/bin/ombatchsyn.pl - perl script called by above shellscript to generate top of top sort
	on tcp syn logging information.  This guy does TOP SYNNER work.

~mrourmon/bin/irc.pl - perl script to generate summarized hourly irc reports.

~mrourmon/bin/tcpworm.pl - summarizes raw 30 second tcpworm.txt files to produce various
	kinds of output reports.  So basically these are summarizations similar to
	the 30-second tcp port signature report.  In general these summarizations are
	sorted based on the number of instances (how many times has an IP shown up in
	the set of 30-second port reports during the current day, or the previous day total).

	There are 6 default summarizations:
	1. all - no pre-filtering is done.  
	2. 445 worms - only instances mentioning TCP port 445 are displayed.
	3. only weight 40 and above - only instances with a TCP work weight > 40 are included.
	4. p2p summarization only, not worm.  Only instances mentioning P2P traffic (bittorrent,
	kazaa, etc.) are included.
	5. syndump summarization - all IP srcs that are local are mentioned.
	6. email summarization - hosts sending email are mentioned.

It is quite possible that tcpworm.pl could be setup for additional summarization reports.
E.g., we run summarization reports for port 25 traffic, and for various versions of
the above summaries based on either internal or external IPs.  Run ./tcpworm.pl by itself
to see the options and refer to how it is used in batchip.sh.  You may wish to add
your own summarizations.  

-------------------------------------------------------------------
3. pre-configure dependencies:
-------------------------------------------------------------------

Ourmon has dependencies that must be installed before configuration time else the config process and/or its produced executables will fail.
Note: see README.linux, and/or README.bsd for the last word
in dependencies.

1. Front-end sniffer dependencies:

1.1. libpcap must be available.  It is installed by default
on FreeBSD.  Nevertheless, it is findable at:  

http://www.tcpdump.org.

If you have tcpdump, you could be in business, as tcpdump uses
libpcap.  In our experience, sometimes a given UNIX distribution
is not up to date in terms of libpcap.  On both FreeBSD and Linux
if you have a problem you might download a new version of libpcap
and install it in /usr/local/lib/libpcap.a and then build or rebuild
the front-end ourmon probe.

Some linux distros may have such an old libpcap that the libpcap
parser itself is seriously old.  If the front-end probe complains
about problems in the *supplied* ./etc/ourmon.conf, this is the problem.  
Get a new libpcap and relink.

You may test the ourmon front-end by hand by trying to build
ourmon as follows (although the configure.pl script will do this for you).

If FreeBSD:

(cd src/ourmon; make -f Makefile.bsd)

If Linux:

(cd src/ourmon; make -f Makefile.linux)

For the above to work, the Makefile needs to have the right pathes
for include files and the libraries.  configure.pl does this for you.
(it tries to figure them out).

1.2 the front-end needs the Perl Compatible Regulation Expression library
or libpcre.a.   See the README files for more information.

You may have it as lots of tools use it.   E.g., ngrep is
a useful sniffer and it uses pcre. 

2. Back-end dependencies:
 
2.1. omupdate.pl needs RRDTool

findable at: 
http://oss.oetiker.ch/rrdtool/download.en.html

RRDTool is used for graph creation and storage of data.  It must
be fetched and installed before ourmon is configured.
It is strongly recommended to build rrdtool as follows: 
#########################################################

	# make site-perl-install

as this means the rrd perl libraries will be installed in system
places, and you will not have to change /home/mrourmon/bin/omupdate.pl
in terms of its "use lib" statement.  (not a big deal).
E.g., ompdate.pl has a "use lib" statement something like the following:

:use lib qw( /usr/local/rrdtool-1.0.13/lib/perl ../lib/perl );

One way to make sure omupdate.pl works is to simply change this
statement so it can find the needed perl libs.

How to test omupdate.pl:

# perl -d omupdate.pl

if there are complaints about RRD perl libs, you have work to do.
No complaints, then simply exit.  You do not need to run omupdate.pl.
Although it doesn't really hurt anything to run it once by hand
all the way to make sure it works.

Note: it may also be useful to test out the rrd tool install.
E.g cd to ./examples in the rrdtool distribution directory,
and make sure that you can run "bigtops.pl" or other example
scripts and get png pictures.

On FreeBSD, rrdtool is available as a port.  See README.bsd
for more info.

3. It is assumed that you have a web server; e.g., apache.
Use of a web server is simple, we provide .html, and .png pictures.  
There is no CGI functionality in ourmon at this time.  
*****************************************************
You should know where you want the 
web pages and top of top reports to go; e.g., in a directory such as 
/usr/local/apache2/htdocs/ourmon or /usr/local/www/data/ourmon 
or /var/www/htdocs (or wherever the distro people have decided apache
goes this time -- some consistency here would be nice).

A prototype index.html file is supplied, but it may be modified
based on the front-end ourmon.conf filters you actually use.  The
configuration process installs a prototype index.html.  It will
serve to get started.  Edit as needed.  You may also need to modify
or create per-filter html pages if you use the user-bpf functionality.

You should learn to use the user-bpf functionality, else you are wasting
a rather important feature.  E.g., make custom BPFs for subnets
and/or for that set of hosts that should be mail servers.

****************************************************************

See any "bpf*.html" and clone and edit that as needed for new
filters.  Any of the bpf*html files may be used as a template.
There is a section below on how to create a new BPF filter set of
your own.

4. optional dependencies if front-end and back-end are on separate
computers.  You will need a mechanism to copy the mon.lite and
tcpworm.txt files from the front-end to the back-end.  
We suggest wget as it is simple and can be done in-line in omupdate.sh. 
Use iptables or ipfw ACLS to narrow down who can access the front-end, and
possibly micro_http or some other web-server on the probe.  On FBSD,
wget can be found in /usr/ports/ftp/wget.  This is fairly easy to setup.  
See optimizations below for more information.  Of course, you can setup 
your own system here using who knows what, rsync, ftp, ssh, etc.
Certainly you can use secure shell here to do the copies.

------------------------------------------------------
4. running configure.pl
------------------------------------------------------

Configure.pl installs the ourmon system.  Actually it installs the
front-end first, and the back-end second.  Either front-end or
back-end can be installed separately or together.  A beginner should
probably install all of it on one machine.  configure.pl does not
install dependencies.  

In addition to installing ourmon, and omupdate.pl, etc. 
configure.pl installs the front-end ourmon.sh script and other script files
and also creates a suggested root crontab "file" which should be carefully
appended to the root crontab.

These scripts may be examined and tweaked as desired.  ourmon.sh and omupdate.sh
serve as parameter packaging scripts for the front-end ourmon program, and the back-end
omupdate.pl program and other back-end programs.  This makes invoking those programs much easier.
In general, to run the front-end:

	./ourmon.sh start

to run the back-end (for the 30-second graphics, and reports)

	./omupdate.sh

It can be useful to test the latter by doing this:

	sh -x ./omupdate.sh

ourmon (front-end), is compiled and installed in the ourmon
bin directory.  omupdate.pl, and logging, and other scripts are put 
in bin as well.  

Install notes:

In this section we give a rough idea of what configure.pl does:

Note: in terms of command syntax configure.pl can be run in two ways:

# 1. # ./configure.pl
#       installs in current working directory.  NOTE: we assume
#       that this is at the top of the ourmon development tree (src is in ./src).
# 2. # ./configure.pl /dirpath
#       builds ourmon locally (src is still ./src), but makes bin/scripts so
#       that entire built tree can be installed somewhere else via a cp -r.
#       NOTE!!!.  ./mrourmon is appended to the /dirpath, so if you want
#               /usr/local/mrourmon supply /usr/local
#
The first case is typically used for a manual install.  Note ourmon can
be installed in any directory.  We merely use /home/mrourmon as an example.

4.1 front-end

On BSD configure.pl first tries to locate libpcap, in either
/usr/local/lib/libpcap.a or /usr/lib/libpcap.a in that order.  On
linux it just goes with the usual spot.  A new install from
www.tcpdump.org will place libpcap in /usr/local/lib, which is
preferred.  It also finds libpcre.a and tries to compile the ourmon
front-end via the relevant BSD or linux Makefile in src/ourmon.
In a similar manner it will locate a libpcre.a and use both libraries
to build the front-end probe called ourmon which is placed in the 
local /usr/mrourmon/bin directory.

configure.pl needs an input interface for the front-end.  Of course,
this is the Ethernet or wireless interface on which you wish to run ourmon.
E.g., on FreeBSD it might be em0, sk0, or fxp0, or lo0 for testing.
On Linux it might be eth0.  Use netstat -a to find your interface name before
you run configure.pl.   The current version tries to guess the interface.
It may be wrong.

configure.pl asks if you want to install the ourmon.sh script
in /usr/local/etc/rc.d on BSD  or somewhere similar on Linux
(so that ourmon will always be run when the system is rebooted).  

configure.pl asks you what your home network and mask are.
For example if you are running ourmon to monitor the 192.168.1.0/24
subnet, you would enter: 

		192.168.1.0	255.255.255.0

This information is used to patch the ./etc/ourmon.conf file
in order to make sure the topn_syn "tworm" us vs them graph
information is correct:

topn_syn_homeip  10.1.0.0       255.255.0.0

You may skip this step and edit ourmon.conf by yourself before
booting or rebooting ourmon.  In general it is a good idea
to set your notion of home IP or network.  If you are managing
a network, set it to a network address.  If a single host,
then set it to that host's IP, e.g.,

		10.1.2.3	255.255.255.255

At the conclusion of the front-end install, you should have
bin/ourmon, and bin/ourmon.sh.  You can choose here to exit configure.pl
and simply boot the front-end (./ourmon.sh start).  You should get
data in the -D directories (at least mon.lite).  

Remember to boot the front-end the first time:

cd /home/mrourmon/bin;  ./ourmon.sh start

As ourmon.sh is placed in a startup script directory, it should
be booted in the future when the system is rebooted.  

4.2 back-end

Daily logs are placed in the ~mrourmon/logs directory.
Sorts of those logs are done on an hourly basis to produce top of top
web summarizations.

Logging is self-contained within the ourmon back-end and has
nothing to do with the UNIX syslog system at this point.
Note that configure.pl will create the logs directory and its lower-level
directories.  It should be pointed out that RRD logs in the rrddata
do not grow over time.  They are created with a certain size and do
not get any bigger.  After a week, the base ourmon log directory
does not get MUCH bigger either as files are simply rotated from that point on.
Of course in individual port signature file might have 1 host in it, none,
or 1 million, and hence size needs here are fuzzier.

configure.pl asks for a pathname for the web output directory.  The
default assumption is ourmon as a sub-directory (in the web htdocs
directory) in which both dynamically created output and canned .html
files will be placed.  Dynamic output comes from programs like
omupdate.pl.  Top of top reports are also put here.
Canned html files come from the src/web.html directory.  configure.pl
*creates* the base directory (ourmon).  You must make sure that the
path leading up to the base directory exists (e.g.,
/usr/local/apache/htdocs).

configure.pl asks if you want to copy the base files (from src/web.html)
to the web directory.  This should be done on an initial back-end
install.  Post install, you want to edit index.html and possibly other supplied
files (or use them as templates) for putting your base index.html
file together.  You want index.html to match the filters in
~mrourmon/etc/ourmon.conf.  For example, if you have a new BPF filter set
expression graph, then you must create a .html file to go with it.
See adding a BPF graph below in the optimizing section for more
information.

configure.pl will create a local directory for RRDs to be stored
in.  By default, this is rrddata.

You will be asked to set a UDP weight threshold.  This value
is used to determine when the back-end should put exceeded UDP weight
messages in the event log, possibly catching UDP scanners.
The initial default is probably fine to start with.  See info.html
for more information.

You will also be asked to set a home network so that ourmon can make
decisions about what is home and what is "away".  Do this in
the form  network/netmask; e.g., 10.0.0.0/8.   

Finally configure.pl generates 1-4 crontab entries in one of 2 possible
formats (vixie and pre-vixie) as follows:

Assume vixie format here:

############## ourmon crontab entries ###############
# run ourmon back-end omupdate.pl etc. per minute
*/1	*	*	*	*	root	/usr/home/mrourmon/bin/omupdate.sh
# batchip.sh - hourly log summary
0	*	*	*	*	root	/usr/home/mrourmon/bin/batchip.sh
# batchipall.sh - roll over logs at almost midnight
59	23	*	*	*	root	/usr/home/mrourmon/bin/batchipall.sh
# daily.pl - re init logs at midnight
0	0	*	*	*	root	/usr/home/mrourmon/bin/daily.pl /usr/home/mrourmon/logs

(With a non-vixie cron such as used with Linux slackware, there is no
user line; i.e., no root in the above).

If you want the crontab entries are placed in a file in /tmp, and you
can manually arrange for cron to be called.  Or you can have configure.pl
simply append them to /etc/crontab.  Note, that this means omupdate.sh will
start being called immediately.  This may generate some root/cron mail,
but you can simply delete the mail later.  Or comment out the call
to omupdate.sh, etc for awhile until you do some unit testing, and get
the front-end running.

configure.pl also installs various shell scripts and perl scripts 
for analysis and sorting of the topn log files.  Thus batchip.sh is called on an
hourly basis during the current day to generate top of top reports
where the log files in ~mrourmon/logs/topn_today are the inputs,
and the outputs are placed in the web directory (*daily.txt files).
batchipall.sh simply rotates the outputs in the web directory.
daily.pl runs at midnight and prepares the logging directories in
~mrourmon/logs for the next day.

Now we turn to testing the various parts of ourmon.

-----------------------------------------------------
5. debugging/testing ourmon plus tricks
-----------------------------------------------------

At this point we assume you have run configure.pl, and have
a front-end and/or back-end installed.

It is best if we go thru and sanity check various parts of ourmon
in order, from front-end to back-end.  

RRDTOOL databases live in:
	/home/mrourmon/rrddata

The mon.lite and tcpworm.txt and other probe output files are found at:
	/home/mrourmon/tmp/mon.lite 
	/home/mrourmon/tmp/tcpworm.txt
	etc. (p2p and irc may be here)

We assume for the sake of argument that your web pages are living at:
	/usr/local/apache/htdocs/ourmon

5.1 front-end testing

The front-end has ourmon.conf as input, and mon.lite, etc. as output.
Edit ourmon.conf with the ASCII editor of your choice.  For information
on filters (not given here) look at ~mrourmon/etc/ourmon.conf (after
installation) which has filters in it, or src/web.html/info.html
(with a web browser) for information about filters.  

Note that you may design new filters using the BPF filter language (as used with
tcpdump), and create new filters with say 1..6 filters per BPF/RRDTOOL
graph.  There are a number of examples of BPF filter sets supplied
in the ourmon.conf file including the bpf/ports or "ports of interest" filter. 
Some examples in the supplied ourmon.conf file are commented out, at least one is commented in.

Comment filters as desired in/out using a hashmark in the 1st column of 
ourmon.conf as follows:

# commented out
# topn_ip 20

# commented in
topn_ip 20

In the sample ourmon.conf there are a number of BPF-based "graphs" with
multiple BPF expressions.  What you do here is up to you, and must make sense 
in the context of your network.  It is easy to set up filters for local nets or hosts
of interest.  Let's assume you have subnets 10.1/16, 10.2/16,
and 10.3/16.  You could write a very simple BPF graph as follows
that will show you per-subnet traffic for several class B-sized subnets.

# example 1: bpf neighbors (chart some neighbor subnets both in/out)
bpf "neighbor-traffic" "neighbor1" "net 10.1.0.0/16"
bpf-next "neighbor2" "net 10.2.0.0/16"
# leave off extra, else remainder bytes put in
bpf-next "neighbor3" "net 10.3.0.0/16"
# leave off extra, else remainder bytes put in
bpf-noxtra

This gives you a graph called "neighbor-traffic" with 3 lines in it, one
for each subnet.  Since ourmon defaults to doing BPF graphs in bytes,
this will show byte count (packet count is also possible, see ourmon.conf
for more information).

ourmon.sh looks something like this: 

/usr/home/mrourmon/bin/ourmon -s 256 -a 30 -f /home/mrourmon/etc/ourmon.conf -i eth0 -D /home/mrourmon/tmp &

Invoke it via ./ourmon.sh for testing.  E.g.,:

# sh -x ourmon.sh start

If there are errors in ourmon.conf,
ourmon will bail and tell you there is an error.  If it boots, look
with ps -ax (ps -elf) to see it, or look in /var/log/messages for a signon message.
Fix any config errors, and try again.  In 30 seconds (-a 30), you should have
some output in /home/mrourmon/tmp/mon.lite.  If so, you can go on to testing
the backend.  If there are big problems, try using the -d debug switch.

Note that the final parameter to ourmon above is actually the NULL string.  ourmon
takes a BPF expression which can be passed into the kernel to tell the kernel which
packets to accept or ignore.  For example, if for some reason you wanted
to ignore UDP packets, you could tack on: 

/usr/home/mrourmon/bin/ourmon -s 256 -a 30 -f /home/mrourmon/etc/ourmon.conf -i eth0 -D /home/mrourmon/tmp "not udp"&

Or if you have a reasonably secure host (running solaris say) that does news and
is a big network hog, you could choose to ignore it by putting in an expression like

"not host <IP>"  

or you could ignore a network with

"not net 10.0.0.0/8"

This has the security downside of missing attacks to or from that host or net.

5.2 debugging the front-end

So, what can wrong?  Probably more than I comment on here. 
One could always run out of memory due to too many packets
and not enough memory for mallocs.  See your message log
if there are problems.  Send us a backtrace with gdb if there
are core dumps.

5.2.1 debugging bpf graphs and making the RRDTOOL lib happy

Expressions should be tried on tcpdump first.  If they work there,
then they should work in ourmon.conf.   Remember that the shell
is picky about tcpdump expressions.  For example, you may need to
put "( something ) or (something)" in quotes for testing directly
with tcpdump.

If you get really bizarre errors ... try your expressions on tcpdump
to see if they work.  It is always possible that you may need to visit
http://www.tcpdump.org and get a new pcap lib and install it.

There is a tricky front-end back-end problem with the BPF filter-set
graph label (1st label in bpf and bpf-next lines) which is that
RRDTOOL is picky about the syntax of the graph label.  In general
it should be simple, with no blanks (hyphens work ok), and should
begin with an alpha, not a numeric.  In general if RRDTOOL is
unhappy, about your graph, and is failing to create pictures in the
web directory, examine the omupdate.pl/RRDTOOL logfile:
~mrourmon/rrddata/ourmon.log.  If you have cranky messages there,
and they are updated every 30 seconds, odds are you are not getting
a graph.  You will have to fix the ourmon.conf and reboot ourmon
until RRDTOOL is happy.  Also in general the bpf expression labels
(*not the expressions* although short expressions are faster at runtime) 
should be *short* and pithy.  "areweinbelgiumyetiwouldliketoknow" is too long.

The RRDTOOL log file is found at /home/mrourmon/rrddata/ourmon.log.  
omupdate.pl puts RRD error messages in this file.

Another possible source of errors messages occur when you change
the number of lines in a BPF graph.  For example if you go from
three expressions to four, you won't get the new graph to work UNTIL
you delete the previous rrd.  So say, you have a graph label called
"subnets", and you add a new subnet to the graph.  You should go
to the RRDTOOL libs directory, and delete the subnets rrd, which
would most likely have the name bpf-subnets.rrd.

5.2.2 ourmon (front-end) dies.  

If your ourmon front-end dies, it is possible that the
topn lists, which malloc space for flow structures, have run out
of space.  The ourmon front-end logs messages in /var/log/messages.
If malloc blows up, there should be a message.  If this is the case,
your computer ran out of memory for ourmon for some reason (not
enough memory, too many flows).  Of course, you could turn topn
facilities off, but that isn't very interesting.  Note: that in
theory, our code is robust ... but running out of memory is a problem
if you have 100000's of flows.  (Yes, ourmon might have a bug.  Tell
us if that is the case).

Another big reason for ourmon going away may be that you have a flaky
interface card or driver.  This has happened in the real world.
It usually isn't ourmon's fault, but it is very annoying.
A workaround is to make sure that ourmon is rebooted out of
/etc/crontab, if it goes away, once a minute or once every five
minutes.  A non-installed perl hack called scripts/runourmon.pl
exists.  You can put it somewhere executable, and then have crontab
run it periodically to make sure that ourmon doesn't go away.
E.g.,

*/5    *       *       *       *       root    /etc/runourmon.pl

non-vixie crontab:

*/5    *       *       *       *       /etc/runourmon.pl

This would make sure ourmon is running every 5 minutes, and restart it if
it is not.  The ourmon probe goes background, and so /etc/runourmon.pl should
exit if it has to start ourmon.  Be sure and read runourmon.pl, and make
sure the pathnames are correct, and that the path to perl is correct too.

5.3 testing the backend (sans logging)

Given that we have a mon.lite file, we can now start to test the omupdate
facility.  

5.3.1. test omupdate.pl as a perl script

This is just a static test to make sure the omupdate.pl can find its rrd library.

	# cd bin
	# perl -d omupdate.pl

chdir to the bin directory and simply invoke omupdate.pl as a perl script.
If you have problems, the most likely problem is that the perl path
for finding the perl RRDTOOL library code is wrong.  

You may have to have fix this statement in omupdate.pl.

use lib qw( /usr/local/rrdtool-1.0.13/lib/perl ../lib/perl );

Or reinstall the rrd package with:
	# make site-perl-install

5.4 running omupdate.sh and omupdate.pl

At this point you should be ready, to run the system.  You should
have a mon.lite file in tmp for omupdate.pl to use.  You may want
to first make sure that the /etc/crontab entry for running
bin/omupdate.sh is off, and test running omupdate.sh by hand in
order to test the back-end standalone.  In general, there should
be no output from omupdate.pl (unless debugging is turned on).  You
can always sanity check omupdate.sh, and make sure the parameters in
it are correct (mon.lite is found in the proper place, etc.).  Of
course, omupdate.sh should be run once per minute, but for
debugging purposes running it by hand to see if anything explodes
is fine.

Let's assume that the crontab entry for ./bin/omupdate.sh is installed.
That means omupdate.pl will run every 30 seconds, twice per minute.
Possible sources of output are:

1. your web directory (/usr/local/apache/htdocs/ourmon), which
will have canned .html files in it installed by the configure.pl
utility and dynamically created png files in it, created by RRDTOOL.  
(Logging will produce some report files that have a
.txt suffix).  

2. the RRDTOOL libs directory (rrddata/) will have .rrd files in
it, and the ourmon.log file that contains rrdtool lib errors if
any.   Note that omupdate.pl dynamically creates rrds if they have
not been referenced before.  Users do not need (or want) to create
rrds.  The rrd databases are updated here by omupdate.pl as well.
It can be useful to watch the ourmon.log file here.
You can zero it out (if it has old data in it).  In general,
it should *not* grow unless there is a system problem.

3. if logging is enabled, log files in ~mrourmon/logs will be updated
every thirty seconds.  

mon.lite files are placed in logs/mon.lite:

~mrourmon/logs/mon.lite/Mon ... etc/date timestamp.mon.lite

tcpworm.txt files are placed in a similar way in:

~mrourmon/logs/tworm/Mon ... etc/date timestamp.tcpworm.txt

Various topn log files are placed in logs/topn_daily, with one or more
files per front-end topn filter.  Note that the daily.pl script
rotates these files in such a way, that for example if today is Tuesday
Tue and topn_daily are linked, and yesterday's files are in the Mon directory.
We show the log files below as they are associated with their front-end
topn filter name in ourmon.conf.  Note that one front-end filter may create
more than one back-end logfile.

~mrourmon/logs/topn_today may include:

logfile		   ourmon.conf name if any	explanation
------------------------------------------------------------------------------------------
topn_ip.log        topn_ip			top IP flows
topn_tcp.log       topn_ip			top TCP flows 
topn_udp.log       topn_ip			top UDP flows
topn_icmp.log      topn_ip			top ICMP flows
alert.log:         topn_ip			flows > N bits

syn_list.log       topn_syn			top TCP syn sending IP srcs

icmperror_list.log topn_icmperror		top ICMP error causing IP srcs
udperror_list.log  topn_icmperror		top weighted UDP error IP srcs

ip_scan.log        topn_scans			top IP srcs sending N IP dst pkts

ip_portscan.log    topn_port_scans		top IP srcs sending N L4 dst port pkts
topn_portscan.log  topn_port_scans		top IP srcs sending N TCP dst port pkts
topn_portscan.log  topn_port_scans		top IP srcs sending N UDP dst port pkts

tcp_ports.log      topn_port 			top TCP ports in use
udp_ports.log      topn_port			top UDP ports in use

info.html should be seen for the internal details of log entries.
Essentially mon.lite output as produced by ourmon is put straight into
the log files.

5.5 files found in the web directory

The following (incomplete!) list of files shows files that are either installed
in the web directory, or dynamically created.  We try to show here what
kinds of files one can expect.  If they are installed,
you probably want to modify them (generally .html files).  If they
are dynamically created, it may be for debugging reasons to check that indeed
they are created, when you add or modify new filters, especially BPF
filters.  Of course, you can make editing mistakes with these files,
or have parts of the system that you do not need.

Created files may have either .png or .txt suffixes.  (.txt
are perl reports derived from the topn logging mechanism).  

index.html - the basic index.html file.  You must edit it.  It will
	contain pictures (current pictures) for RRDTOOL and topn output.
	It will also cite sub-web-pages for RRDTOOL at least, to give one
	year's worth of data.  It also contains links for the top of top
	reports.  INSTALLED.  If you add a new bpf graph, change this file
	so that it can find the new .png pictures you are making.

indexstatic.html - any time you change index.html, copy it to this file,
	and remove the http cache update timings, so that this page is 
	NOT updated every thirty seconds.  (you don't have to do this of 
	course).  INSTALLED.

info.html - documentation on how ourmon and its filters works, 
	with filter explanations.  INSTALLED.

time.log - created dynamically by omupdate.pl.  This is the last time
	omupdate.pl ran.  CREATED.
		
mon.lite - copied into the web directory by omupdate.pl.  This is your
	"current" mon.lite file. CREATED.

pkts.html - this is the RRDTOOL sub-html file, referenced from index.html.
	It displays the following 4 graphs of daily/monthly/weekly/yearly
	information derived from the pkts filter rrdtool database.
	INSTALLED.

	As an example of pictures created by omupdate.pl with RRDTOOL
	for pkts.html, we should have the following 4 pics made
	at runtime every thirty seconds:

	pkts.daily-png - dynamically created by omupdate.pl/rrd code.
	pkts.monthly-png - dynamically created by omupdate.pl/rrd code.
	pkts.weekly-png - dynamically created by omupdate.pl/rrd code.
	pkts.yearly-png - dynamically created by omupdate.pl/rrd code.

cast.html - this is the sub-html file, referenced from index.html
	for the L2 "packet type" filter, multicast, unicast, broadcast.
	It also displays the following 4 graphs, etc as with the pkts
	filter.

	cast.daily-png
	cast.monthly-png
	cast.weekly-png
	cast.yearly-png

(The other hardwired filters are not mentioned here).

bpf-ports.html - This is a sub-page, referenced by index.html that
	shows the supplied example filter using BPF graphs, for
	watching ssh, etc. traffic.   Since the user bpf filters
	use RRDTOOL, we have daily/monthly/weekly/yearly graphs.
	If you create a new bpf filter, you should create a file
	like this, and reference the 4 png filters that will be
	created by omupdate.pl based on the supplied bpf graph label.
	Put some sort of link to the new bpf file in index.html.

	bpf.ports.daily-png
	bpf.ports.monthly-png
	bpf.ports.weekly-png
	bpf.ports.yearly-png

	Other bpf-based graphs will have a similar setup.

Dynamic second level files for top talkers are created by omupdate.pl.
	We won't list their filenames here.

Certain anomaly reports including udpreport.txt and portreport.txt
are also stored in the web directory.  They are also backed
up in their own logs directories.

Web summarization creates a number of top of top reports
which in general can be found at the bottom of the main index.html
page (and also irc summarizations are found on the irc.html web page).
These files are created via the shell scripts bin/batchip.sh and bin/batchipall.sh.
See info.html for more information.

Near midnight, the daily reports are pushed back one day in the following
manner:

topn_today.txt -> topn.0.txt
topn.0.txt -> topn.1.txt

topn.6.txt -> topn.7.txt

At the end of the day, the daily script, run at roughly midnight,
pushes the summary for the current day back one day.  So for example,
topn_today.txt is for *today*, topn.0.txt is for yesterday, topn.1.txt
for the previous day, etc.  The same is done for the other top of top
reports.

Web browsers:  firefox is our web browser of choice.

------------------------------------------------------------
6. creating a new user/bpf graph/filter
------------------------------------------------------------

One of the more important ourmon facilities, is the ability to
create a new BPF filter set, made up of BPF expressions, with one BPF
expression graphed per-line in the RRDTOOL strip chart graphs.  So
let's walk through the steps needed to create such a filter.

1. design the ourmon.conf configuration.  

Assume we want to graph VPN or IP tunnel traffic, consisting of
IPSEC ESP (encrypted), or IPSEC AH (authenticated, but not encrypted),
and various kinds of tunnels.  This means we need four BPF expressions,
and we will not turn on the xtra flag, since we assume that in our
case, there is too much traffic not using VPNs (or tunnels), which
would flatten the lines in the graph and make our case study here less
interesting.

Because we are intelligent and well-informed network engineers, we
know that ip protocols 50 and 51 are used for ESP and AH respectively.
Just to make life interesting let us also throw in the graph lines
for IPIP encapsulation and Cisco GRE encapsulation.  The former
might be ip protocol 4 or 94, and the latter should be ip protocol
47, as this may well show us Microsoft VPN tunnels.  This gives us
4 possible BPF expressions.  This leads to the interesting problem,
which is how do we construct a BPF expression, to capture that
traffic?  In general, one should read the tcpdump man page, and try
to construct an expression with tcpdump that works, and that you
can test ...  Once you have the expression, you can put it in the
/etc/ourmon.conf filter config you are creating.

In this case,

# tcpdump -n -i em0 ip proto 50

will work for ESP, and ip proto 51, will work for AH.  

(See /etc/protocols or IANA pages or google for that matter).

Armed with the correct syntax for our BPF expressions, 
we can now create a graph configuration.

# vpn/tunnel graphs
bpf "vpntuns" "esp" "ip proto 50"
bpf-next "ah" "ip proto 51"
bpf-next "ipip" "ip proto 4 or ip proto 94"
bpf-next "CiscoGRE" "ip proto 47"
bpf-noxtra

The graph label (which will be used for various filenames at the
back-end, including the rrd database, and the png files in the web
directory) is vpntuns.  Line labels are esp, ah, ipip, and CiscoGRE.
The BPF expressions for the 4 lines are put in quotes so that ourmon
knows where the begin and end.  Note the "or" for the two kinds of IPIP.

2. test the new filter with ourmon (front-end)

Now we restart ourmon, and see if it either complains because the
syntax was wrong and if we can actually get output in our /tmp/mon.lite
file.

cd to the ourmon/bin directory, and kill (-9) ourmon, and then restart it:

# ./halt_ourmon.sh
# ./ourmon.sh

If there is no whining about the config, we are hopefully in business.

30 seconds later ... in our mon.lite file we find something like this:

bpf:vpntuns:4:esp:43422:ah:0:ipip:0:CiscoGRE:771100:xtra:0

In mon.lite lingo, this means we have a graph named "vpntuns", and
that we actually seem to have ESP and GRE traffic, but none of the
other two species at this point.

3. sanity check back-end omupdate.pl

Now the next question is whether or not we are getting .png files.
If we are getting .png files, our filter works (you can always
look at them with a web browser, or a X tool like xv).

So we cd to our web directory, and do 

# ls *png 

to look for png files.  (Probably at least 30 seconds have to pass
... a one minute break here might be a good idea!).

Low and behold we have:

561(mrserver)(/home/mrourmon/web.pages)  ls *vpntuns*
bpf.vpntuns.daily-png   bpf.vpntuns.monthly-png bpf.vpntuns.weekly-png  
bpf.vpntuns.yearly-png

Note how the bpf graph label "vpntuns" has become part of the filename.   
omupdate.pl appends the string "bpf" to the RRDTOOL png graph names to
let us know that these are bpf-based filter outputs.

This means we have a working filter (in the sense that omupdate/pl
RRDTOOL liked your config, of course, you might not actually have
any network data!, not our fault ...).

You can now cobble together a .html file to show all 4 graphs,
daily, weekly, etc, and hook it up to your index.html file.  You
can use the supplied bpf-ports.html file as a template and reference
bpf-ports.html in index.html.  It is a good idea to put the
daily png file into index.html somewhere appropriate.

Finally whenever you change index.html copy it to indexstatic.html
and get rid of the cache entries so that indexstatic.html when referenced
by a web browser will not longer automatically update itself every 30 seconds.

In other words, get rid of these three lines:

<META HTTP-EQUIV="Refresh" CONTENT="30" >
<META HTTP-EQUIV="Pragma" CONTENT="no-cache" >
<META HTTP-EQUIV="Expires" CONTENT="Fri, 19 May 1999 23:37:00 GMT">

4. no bpf.vpuntuns.daily-png file, now what?

Change directory to the rrdtool library file, say ~mrourmon/rrddata,
and look at the end of the ourmon.log file in there.  There are two
common sources of problems here, one is that RRDTOOL may not like
your label, so make it simpler.  Also if you change the number of
lines (number of BPF expressions), you should first wipe out the
existing rrd database file, and then start over.  In that case, I
would cd to the rrd directory, and simply do:

# rm vuntuns.rrd

You should have a rrd file of course.  omupdate.pl dynamically creates it
the first time it sees a BPF filter it has not seen before based
on whether or not the rrd exists in the rrddata directory.

Running

# tail -f ourmon.log

here can be of use.  No errors means RRDTOOL is happy.

There are two important hints here:

1. sanity check your BPF expression with tcpdump.

2. if you don't get pictures, check the RRDTOOL log file, or possibly
remove the particular rrd database, and try again.  

If you want to preserve your RRD data look into the facilities
available in the RRD tool set in the installed RRD directory.
You may be able to convert the existing RRD to xml, edit it,
and then recreate a new RRD with the ability to tolerate a new
data source (this is beyond us however).

-----------------------------------------------------------e
7. optimizing ourmon
------------------------------------------------------------

In this section, we will briefly talk about how one can optimize
the ourmon front-end to get more work done.   

The easiest thing to do is get a faster computer.
A dual-core, dual or hyper-threaded Intel computer is a good idea
because interrupts and the application then can have their own CPUs 
and not contend for one CPU (in the worst-case scenario interrupts
always win and nothing gets done by the application.  This is called
livelock).

7.1. distributing the load, separate front-end and back-end.

Possible approaches to distributing the load might include:

1. separate the front-end, and back-end; that is, put them on two separate
machines.  (this section)

2. separate the work amongst a number of front-ends. (next section)

3. multiple front-ends on different hosts and multiple back-end web 
directories on the same host. (next section)

4. create a threaded front-end.  Hah!.  we are trying to do so.
Get a dual probe -- this is cheap and relatively easy to do.

First one can use two computers, one for the front-end, and one for the back-end.

Our back-end is a heavily used web server that does more work than
just creating ourmon graphs,  and consequently it would not
be a good idea to run the front-end and the back-end on the same box.
configure.pl is designed to install the front-end and the back-end
separately.  Thus installing only the front-end on some other box
is easy.  It also may be a good idea to simply dedicate the front-end
to the ourmon probe, as any other CPU load during network attacks,
may cause ourmon to lose packets.  TCP syn scans/attacks (and any minimal
sized packet attack including the UDP blaster attack) are difficult
to deal with for modern hosts.  They simply cannot keep up with small packet
streams under the current monolithic processing model.

The only remaining question becomes how to get the mon.lite file
from the front-end box to the back-end graphics box.  Exactly how
to do that is left up to you, but some possible avenues of attack
include:

1. use a lightweight web server like the freebsd ports/net/micro-httpd,
and then use wget on the back-end box to fetch mon.lite every thirty
seconds driven by  a crontab job.  

Note that bin/omupdate.sh has code to do that but it is commented out
by default.  Use that code ... 

the basic idea is something like this (this is just for the mon.lite file
as an illustration):

WGET="/usr/local/bin/wget --tries=2 --connect-timeout=2 "

$WGET -q --output-document=/home/mrourmon/tcpworm.txt http://$CASPER/tcpworm.txt
$SUCCESS=$?

if [ $SUCCESS -ne 0 ]; then
        echo "failure to fetch probe-files" >> $WEBDIR/event_today.txt
        exit 1
fi

--tries means wget tries twice, and it has a connect timeout of 2 seconds.
This avoids a pile of crontab requests if for some reason communication
is interrupted.

wget also fails nicely, and you can use it to abort the entire
run by checking for failure.  We are putting the failure
message in the web-based event log.

If your probe box is in a DMZ and you want to nail down access to
it to just the back-end server, use ipfw (or iptables on linux),
and make a firewall rule that only allows the back-end box to fetch
files from the probe.  In general, from the security point-of-view,
it is a good idea to consider the probe a bastion host.

2. use ssh with no passphrase.

3. use anon ftp, wget can also get ftp files.

4. use samba/nfs,rsync, etc.  But some kind of ACL mechanism
here would be a really really good idea.

5. your idea here ...  

For security reasons, you can always improve your setup with ipfw 
or iptables on linux.

7.2. distributing the load, multiple front-ends, same back-end.

It is also possible to have multiple front-ends that work with the
same back-end server.  One goal might be to have separate probes,
where each probe does different filter work (different filter labels)
with all the work unified at one back-end web directory and one rrd
database directory.

Again use configure.pl to install front-ends.  For the sake of
argument, assume you have two front-end probes, called alice and
bob.  Simply give them different filter rules.  E.g., alice might
do topn work, and bob might do bpf work.  In each /etc/ourmon.conf
make sure that the first config line is the special config tag,
"sysname".

E.g., on alice:

#
sysname alice


on bob:

sysname bob

The problem with trying to separate the filters between two boxes,
is that both probes will still produce a pkts drop/counts filter,
since you cannot configure it out.  In fact, you want this information,
because you want to know if any probe is dropping packets.  Use of
the sysname config tag, causes omupdate.pl to name the pkts graphs
and rrd database with the sysname as a file prefix.  In the web
directory, pkts.daily-png becomes alice.pkts.daily-png and
bob.pkts-daily-png.  RRD files become alice.pkts.rrd and bob.pkts.rrd.

As a result, one can have two pkts outputs on the same index.html,
and simply divide up the remaining filter work on two different probe machines.
Note: it is not currently possible to have the same filter on two
probes, somehow appearing on the same back-end web page.  

7.3. distributing the load, multiple front-ends, multiple back-end
web directories.

Of course it is always possible to setup say two different front-ends
that go to two different ourmon directories on the same or different
web host.  We have not done this.  In theory, fiddling with the parameters
in omupdate.sh to omupdate.pl (and also the logging inputs to monbackup.pl)
should help make this possible.

7.4. minimizing lost packets

On FreeBSD, when ourmon boots, a message like the following is placed
in /var/log/messages by ourmon:

ourmon: bpf_bufsize 32768: snaplen 68, timeout 1000

ourmon is kindly informing you about the size of the kernel
bpf buffer.  We find that we can do a fairly good job of
capturing max-sized Ethernet packets on a P4 gigabit Ethernet system
if the BPF buffer size is actually about 8 megabytes. 
We don't really know how BIG the buffer should be.  You can always
try making it bigger to see if it does any good.  Bigger is better.
Smaller is not so good.  The kernel default size is much smaller.   

ourmon.sh as installed looks something like this:

#!/bin/sh
BSIZE=8388608
sysctl -w debug.bpf_bufsize=$BSIZE
sysctl -w debug.bpf_maxbufsize=$BSIZE
/usr/home/mrourmon/bin/ourmon -a 30 -s 256 -i em0 -f /etc/ourmon.conf -m /home/mrourmon/web.pages/mon.lite &

Something like the following message should appear in /var/log/messages:

system ourmon[45692]: bpf_bufsize 8388608: snaplen 68, timeout 1000 

bpf_bufsize here should match the sysctl -w BSIZE value (8 megabytes).
It should NOT appear as follows:

system ourmon[111]: bpf_bufsize 32768: snaplen 68, timeout 1000

Before ourmon is booted, it sets two kernel sysctl variables that
control the size of the default kernel BPF buffer.  As above, we
set it too 8 megabytes.  With an AMD 2000, on a motherboard that
has 64 bit PCI, and a syskonnect gigabit Ethernet card, we found
that we would not drop max MTU-sized packets with a reasonable
workload, if the BPF buffer was set to 8 megabytes.  Any smaller,
and packets were dropped.

One then needs to compare the /var/log/message produced by ourmon
with the sysctl variables.  If they match, good.  If they do NOT match,
you need a new pcap library.  Get it, and recompile ourmon using it.

A big kernel buffer is a good thing for *snort* too.  

On the other hand, handling min-sized packets is an open security 
research problem and while a bigger kernel buffer than the default 
is better, there may be no perfect solution.  

Also note that we know a bigger BUFFER can be better.  In general,
if you have drops, try a bigger buffer.  If at some point it
doesn't help with drops, then stop making it bigger.  At this
point in time (spring 2006) dual machines help (hyper-threading helps).
At some point RSN we need a threaded front-ent running on
a dual dual-core box with hyper-threading (cheap parallelism of
some sort).  

7.5. Linux front-end probe note

We don't have a lot of experience with Linux as a front-end probe BUT
that said, we do have a little experience.  Recent mods have been
made that hopefully make the count/drop filter actually work.  Also
we know how in theory to minimize dropped packets.

In order to minimize dropped packets, you need to set big socket
buffers (ironically just as with the FreeBSD BPF ... go figure).
To do this, before ourmon is started, the ourmon.sh script
should be as follows:

# sysctl -w net.core.rmem_default = 8388608
# sysctl -w net.core.rmem_max = 8388608

This is done currently by configure.pl.

We thus set the socket buffers to be 8 megabytes.  Otherwise
ourmon will drop packets very easily.

This is also true for *snort*.

7.6 ourmon threads.  The front-end ourmon probe has a new
experimental thread option that must be compiled in.  See
info.html for more information.  This is not useful unless
you actually have multiple hardware/CPU "threads", e.g.,
with a dual dual-core motherboard.  
7.6 one can teach the ourmon front-end to simply ignore some packets.
Refer to the tcpdump man page and note that you have "or", "and", and "not" operators.
You can then modify the ourmon.sh script and give an expression to the ourmon probe
something like this:

	"not host IP 10.0.0.1"

to ignore any packets from 10.0.0.1.   The pro for doing this is
that you are telling the kernel to not put packets from that expression
into the kernel buffer.  The con is that if there are any security incidents
involving that expressions's packets -- you will know nothing about it.

7.7 dual computers even with a single-threaded ourmon front-end are USEFUL.
Even a hyper-channel P4 is useful.  Apparently the kernel's interrupt side
gets one CPU, and the ourmon probe gets another CPU, with the kernel 
buffer in between.  This is really a classic consumer/producer architecture.
The upshot is that the interrupt-side no longer contends with the application
side for the CPU.  Of course, you must make sure your O.S. actually runs SMP.
(On FreeBSD you may have to turn this on with the kernel).  2 CPUS > 1 CPU.

----------------------------------------------------------
8. ourmon front-end parameters

If for some reason, you are testing the front-end,
note the following parameters:

# ourmon -a 30 -s 256 -i de0 -f /home/mrourmon/etc/ourmon.conf -D /tmp &       

-a 30, means run with 30 second snapshots.  This is REQUIRED.
	unless you want to change and test omupdate.pl for about 1 year.

-s 256, we assume you are using the IRC filter and the TCP port report
	plus similar reports that come from the topn_syn filter.  Both
	of these filters look into the L7 payload of a packet and do
	pattern-matching (in the case of the topn_syn filter this might
	include the new PCRE pattern matching feature too).  
	If you are NOT using IRC or the TCP port report apps field (and you don't care 
	about its contents) possibly because you want ourmon to run as fast as possible,
	and are willing to foregoe pattern matching, then take -s 256 out
	of the parameter list.  Ourmon will more or less not read L7 payloads
	any more.  -s sets the "snap length" same as with tcpdump.

-i de0, specifies the Ethernet interface to use for promiscous mode.
	de0 is an SMC 100BASE interface.  change it as appropriate.
	Use ifconfig -a to learn your Ethernet interfaces.
	configure.pl tries to figure this out, but it is stupid and can
	get it wrong.  

-f /etc/ourmon.conf - tells ourmon where to find its configuration file.

-D /tmp/ - tells ourmon to override any paths in ourmon.conf and put
	all output files in the supplied directory.

The above are the most basic parameters to ourmon. 

ourmon acts as a daemon unless it is invoked with the -d flag
for debugging.  Use -d if you have some very peculiar problem,
and if you get lucky, perhaps it will give you a clue as to what
the problem might be.

ourmon can also read tcpdump output files using the -r flag.  This
can be useful for development and testing.  If -r is used, do not
use -i.  ourmon is invoked as follows for batch testing:

# ourmon -r tcpdump.dmp -f /etc/ourmon.conf -D /tmp &

-a and -i are not used.

When the batch run concludes, output files for all the packets
in the batch run are dumped to the output files in the /tmp directory.
This can be combined with -d for debugging.

Use of two interfaces with ourmon should be regarded as experimental.
We do not recommend it although the code used to support it.
It should not be considered as supported at this point.

9. bugs, debugging and other tips for running ourmon

0. the ourmon probe can survive receiving 802.1q headers (vlan)
which it will strip.  This means ourmon can run on a VLAN base
interface on FreeBSD which is quite a useful trick.  (such
an interface does not have an IP address and does not strip 802.1q
headers in promiscous mode either).  

Thus ourmon can support running on a host that is a trunk port in VLAN terminology.
**********************************************************************************

1. the monitoring interface does not have to have an IP address anymore.

2. if you have problems compiling the supplied ourmon.conf

GET A NEW LIBPCAP from www.tcpdump.org !!!!

It works ok with libpcap 0.8.3 on both Linux and FreeBSD.

3. if you are dropping packets, see the optimization section.  Try
a bigger kernel buffer.  (Or a faster computer).  See section 7.4,
minimizing lost packets.

4. in general the ourmon front-end should not run on the same box running snort or anything
else for that matter (that might be sucking down packets or heck a router ...
or a news server ... or a database server or a busy web server.)
Use common sense here.  If you are getting 20000 packets per second, a P1
might not make a good choice for a front-end computer.

5. if the src/ourmon/ourmon front-end build failed, this is probably
due to some problem with libpcap.a.  Get a new one, and run configure.pl
again and/or change the LIBS path explicitly in the makefile appropriate
for your system.

6. if bin/omupdate.pl fails and complains about RRDTOOL libs, it probably
can't find them.  See Section 2.1. omupdate.pl needs RRDTool.

In the rrd work/build directory:

	# make site-perl-install
	# make install

make site-perl-install is your friend.
#########################################