Recently, I had to monitor network transfer for multiple servers across various geographical locations. There are numerous tools available which can provide a pretty comprehensive reporting for network and other parameters, but I wanted something fairly simple to install and operate. The simplest solution that I came across was MRTG/SNMP combination, which provides me with a fairly accurate idea about my network utilisation.

For the post below, I am assuming three servers, srv1 with ip 1.1.1.1, srv2 with ip 2.2.2.2 and srv3 with ip address 3.3.3.3 are to be monitored from a single location i.e. srv1.

First, either get the latest port files, or update it to latest version on all three servers

portsnap fetch extract

or

portsnap fetch update

Now on all three servers, we need to install the SNMP port

cd /usr/ports/net-mgmt/net-snmp
make all install clean

The following data needs to be put into the file “/usr/local/share/snmp/snmpd.conf” on all three server with relevant changes

###########################################################################
#
# /usr/local/share/snmp/snmpd.conf
#
###########################################################################
#
# snmpd.conf
#
#   - created by the snmpconf configuration program
#
###########################################################################
# SECTION: System Information Setup
#
#   This section defines some of the information reported in
#   the "system" mib group in the mibII tree.

# syslocation: The [typically physical] location of the system.
#   Note that setting this value here means that when trying to
#   perform an snmp SET operation to the sysLocation.0 variable will make
#   the agent return the "notWritable" error code.  IE, including
#   this token in the snmpd.conf file will disable write access to
#   the variable.
#   arguments:  location_string

syslocation  server_location 

# syscontact: The contact information for the administrator
#   Note that setting this value here means that when trying to
#   perform an snmp SET operation to the sysContact.0 variable will make
#   the agent return the "notWritable" error code.  IE, including
#   this token in the snmpd.conf file will disable write access to
#   the variable.
#   arguments:  contact_string

syscontact  Name - email@domain.com

###########################################################################
# SECTION: Access Control Setup
#
#   This section defines who is allowed to talk to your running
#   snmp agent.

# rocommunity: a SNMPv1/SNMPv2c read-only access community name
#   arguments:  community [default|hostname|network/bits] [oid]

rocommunity  community_name 1.1.1.1

Three parameters need to be modified. syslocation can be changed to the actual location of the server. syscontact should be changed to the contact address of the server administrator.

The rocommunity string is the most important part. First parameter community_name should be set to something that is not easily identifiable to outside parties. I am assuming srv1_community, srv2_community and srv3_community for the three servers respectively. The second parameter is the ip address of remote server which will query the SNMP daemon for network statistics. In our case, our graphs are to be run on srv1, so we use its ip address. I would suggest creating a separate community_name for each server for better security. In case someone wants to encrypt the traffic between the SNMP daemon and the polling server, one can use SNMP v3. The links to the pages given below have a pretty good description on using SNMP v3.

Modify and save the contents to “/usr/local/share/snmp/snmpd.conf”

On all three servers, we need to add the following to the “/etc/rc.conf” file to make sure that SNMP daemon starts at boot process.

snmpd_enable="YES"

Now, run the SNMP daemon on all three servers:

/usr/local/etc/rc.d/snmpd start

Make sure that SNMP daemon is running on all three servers.

ps auwx |grep snmp

The above line should return the details of the running snmp daemon. If not, check for any errors.

We can now proceed to install mrtg on srv1. The code now needs to be executed only on srv1

cd /usr/ports/net-mgmt/mrtg
make all install clean

Configure mrtg config files for all three servers:

cfgmaker --global 'WorkDir: /usr/local/www/apache22/data/mrtg' --global 'Options[_]: growright,unknaszero' --output /usr/local/www/apache22/data/mrtg/srv1.cfg srv1_community@1.1.1.1
cfgmaker --global 'WorkDir: /usr/local/www/apache22/data/mrtg' --global 'Options[_]: growright,unknaszero' --output /usr/local/www/apache22/data/mrtg/srv2.cfg srv2_community@2.2.2.2
cfgmaker --global 'WorkDir: /usr/local/www/apache22/data/mrtg' --global 'Options[_]: growright,unknaszero' --output /usr/local/www/apache22/data/mrtg/srv3.cfg srv3_community@3.3.3.3

where “/usr/local/www/apache22/data/mrtg” is the location you want at your webserver for mrtg to display graphs
and “/usr/local/www/apache22/data/mrtg/srv1.cfg” etc. are the names of config files for each server.

Now generate the html files and other goodies through mrtg, thrice for each server.

/usr/local/bin/mrtg /usr/local/www/apache22/data/mrtg/srv1.cfg
/usr/local/bin/mrtg /usr/local/www/apache22/data/mrtg/srv2.cfg
/usr/local/bin/mrtg /usr/local/www/apache22/data/mrtg/srv3.cfg

Make sure that you repeat each line thrice. Although not necessary, this will remove any warning messages being displayed by mrtg.

Each server will now have it’s own graph page. If you are like me, you would rather have all of them on a single page. For that to happen, you need to run the indexmaker command with all the mrtg config files

indexmaker --output=/usr/local/www/apache22/data/mrtg/index.html /usr/local/www/apache22/data/mrtg/srv1.cfg /usr/local/www/apache22/data/mrtg/srv2.cfg /usr/local/www/apache22/data/mrtg/srv3.cfg

Pointing your browser to http://your_server_name/mrtg should now bring up a single page with 5 minute interval graphs for all your servers. Clicking on those graphs will take you to the individual pages for those servers with further data and graphs.

We now need to add polling every 5 minutes for all the servers. Add the following line to “/etc/crontab” file:

*/5     *       *       *       *       root    /usr/bin/env LANG=C /usr/local/bin/mrtg /usr/local/www/apache22/data/mrtg/srv1.cfg --logging /usr/local/www/apache22/data/mrtg/srv1.log
*/5     *       *       *       *       root    /usr/bin/env LANG=C /usr/local/bin/mrtg /usr/local/www/apache22/data/mrtg/srv2.cfg --logging /usr/local/www/apache22/data/mrtg/srv2.log
*/5     *       *       *       *       root    /usr/bin/env LANG=C /usr/local/bin/mrtg /usr/local/www/apache22/data/mrtg/srv3.cfg --logging /usr/local/www/apache22/data/mrtg/srv3.log

You can scale this for as many servers you like as long as you generate individual config files for each server, and create the mrtg and index files accordingly.

Amitabh

While writing the post above, I have taken extensive help from the following two pages:

http://forums.freebsd.org/showthread.php?t=248

and

http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch22_:_Monitoring_Server_Performance

The data center on which my servers are hosted had a sudden electrical failure which let to a downtime of around 5 hours for my servers (I know it sucks, but then it happens).  The machine was later rebooted manually by the technician, and it went into a background fsck mode as it happens in BSD’s.

Now for some unknown reason, the lighttpd server would not start. It would not give any errors. I checked both the lighttpd logs and the syslog files for any erros being thrown, but the server would simply not give out any errors, but still would not run. I tried updating the ports, manually de-installing and reinstalling the lighttpd port, but I would still be getting the same results.

Then it hit upon me to reset the permissions of my webroot folders. I simply ran a recursive chown command ( chown -R www:www /path/to/webroot/directory ) .  This solved the problem immediately. Now I know this is a very simple problem, I was amazed by a simple fact that it would throw any errors and simply exit. It wasted my 5 hours, so I hope this can be of help to somebody.