Snert . Com mod_watch/3.18
      from   Snert . Com

Apache
MRTG

Monitoring Interface for Apache 1.3 & MRTG

Virtual host www.snert.com daily activity
Daily Average Graph
www.snert.com

- apache2mrtg.pl - mod_watch - mod_watch.pl -
- Welcome - Software -

Description

This module will watch and collect the bytes, requests, and documents in & out per virtual host, file owner, remote-ip address, directory or location, and the web server as a whole. This module was designed for use with MRTG, which will make nice graphical representations of the data, but is general enough that it can be applied to other purposes, as the raw data is accessed by a URL. This module supports mod_vhost_alias and mod_gzip.

Configuration

The directives below can be added to the general Apache configuration file, httpd.conf.

SetHandler watch-flush
Context: global, <Directory>, <Location>, <VirtualHost>

When WatchStorage uses the hash table, this handler allows a remote script to request that the shared hash table be flushed and saved to disk for the purpose of backing up the data, without having to restart Apache. This handler is not required for normal operation. It should only be made available when required and protected by Apache access directives to allow only the localhost to make this request.

    <Location /watch-flush>
    SetHandler watch-flush
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1
    </Location>
    
SetHandler watch-info
Context: global, <Directory>, <Location>, <VirtualHost>

This handler is used to display a line of plain text containing the following space separated fields:

    ifName ifUptime ifInOctets ifOutOctets ifRequests ifDocuments ifActive ifOutRate

The ifName is the target name of the request made in a URL. The remaining fields are all numerical. ifUptime is the web server's uptime in seconds. The following is required in the global context of the httpd.conf file in order for mod_watch.pl to function:

    <Location /watch-info>
    SetHandler watch-info
    </Location>
    
    <Location /~*/watch-info>
    SetHandler watch-info
    </Location>
    

This will permit the following URL forms to get the virtual-host, file-owner, server, and spool-file, data respectively:

The token between the tilde (~) and the following slash (/) can be the name of any key that appears in the watch-table list and is the only way to obtain an entry when watching by-port.

SetHandler watch-list
Context: global, <Directory>, <Location>, <VirtualHost>

This handler is similar to watch-info, but presents a plain text list of all the entries that would appear in watch-table.

    <Location /watch-list>
    SetHandler watch-list
    </Location>
    
SetHandler watch-copyright
Context: global, <Directory>, <Location>, <VirtualHost>

This handler permits me to verify the version and copyright of an installation (see License part 2). This is only required to meet the License requirements for commercial redistribution:

    <Location /watch-copyright>
    SetHandler watch-copyright
    Order allow,deny
    Allow from snert.biz snert.com snert.info snert.net snert.org
    </Location>
    
SetHandler watch-table
Context: global, <Directory>, <Location>, <VirtualHost>

This handler is intended for debugging and analysis of the shared memory hash table performance and weenie files:

    <Location /watch-table>
    SetHandler watch-table
    </Location>
    
WatchDocuments mime-type ...
Context: global

A list of MIME types considered to be documents, for the purpose of counting. For example "text/html" would be considered a document, which loads page elements such as images, style sheets, audio files, etc. which are not documents, but elements of one. An asterisk in the mime-type string will act as a wildcard for zero or more of anything. This directive will override the default list:

    application/*word     application/*excel     application/pdf
    application/postscript application/rtf text/enriched
    text/html text/plain text/richtext
    text/xml
WatchHashTableSize size
Context: global

A positive number specifying the size of the shared memory hash table used for the WatchStorage hash policy. The default is 991 (a prime number, but not necessary to be so).

WatchNotLogged handler ...
Context: global

The combination of MRTG and mod_watch.pl can result in a significant number of uninteresting entries concerning watch-info to appear in the access log. This directive provides a means by which to suppress those entries. By default everything is logged. Specify one or more of the following handler names for which logging should be suppressed:

    watch-copyright, watch-info, watch-list, watch-table, all

NOTE this directive could affect the correct operation of other modules that follow after mod_watch in the execution sequence, and should be used with some care.

A better way to achive the same behaviour would be to alter the httpd.conf file with something similar to the following:

    SetEnvIf Request_URI "/watch-info$" IGNORE
    CustomLog /dev/null common env=IGNORE
    CustomLog logs/access_log common env=!IGNORE
    
WatchNetwork exclude ip/mask ...
WatchNetwork include ip/mask ...
Context: global

Add one or more network IP/mask pairs to either the exclusion or inclusion list. Requests coming from a network or host that is a member of the exclusion list and is not a member of the inclusion list are not counted. Otherwise the requests will be counted as outlined in WatchStorage.

The ip/mask can be specified in one of several ways and is the same as for Apache's Allow or Deny directives (domain or host name strings are not supported):

    Form     Example     Equivalent
    a1010.0.0.0/8
    a.b169.254169.254.0.0/16
    a.b.c192.168.1192.168.1.0/24
    a.b.c.d127.0.0.1127.0.0.1/32
    a.b.c.d/n192.168.1.0/24192.168.1
    a.b.c.d/A.B.C.D192.168.1.0/255.255.255.0192.168.1.0/24

WatchStorage files
WatchStorage hash
Context: global

The WatchStorage directive specifies the perfered storage policy for maintaining the counters.

The files and hash storage policies both use the WatchSpoolDirectory to store "weenie" text files. The filename of each text file is either the virtual host name, file owner (login ID), the remote-IP address, or the string "SERVER". The contents of each text file is a string of ASCII text containing several space separated numerical fields given below. The hash storage policy is the default storage policy.

    Name     Type     Description
    ifInOctets32-bit Counterbytes received from requests (except chunked input)
    ifOutOctets32-bit Counterbytes sent in response to requests
    ifRequests32-bit Counternumber of requests (raw hits) received
    ifDocuments32-bit Counternumber of documents sent
    ifActive16-bit Signednumber of active (concurrent) connections
    ifOutRateFloatapprox. 5 minute average of bytes sent per second
    periodOctets32-bit Unsignedbytes sent during the current 5 minute period
    periodMarker32-bit Unsignedstart time() of the current 5 minute period

When updating a set of counters, they are first read from the specified storage medium. The number of octets in and out for a request are added to the current counters, and the number of requests is incremented by one. The number of documents is incremented by one, if the response MIME type matches one of the MIME types specified for the WatchDocuments directive. The number of active connections is incremented at tne start of a request and decremented at the end of the same request. The counters are then returned from whence they came. The counters, ifInOctets, ifOutOctets, ifRequest, ifDocuments, are allowed to roll over.

The ifOutRate, periodOctets, and periodMarker are experimental fields and may change in future versions. The ifOutRate value is computed on the first request after a minimum 5 minute wait period. The periodOctets counts the number of octets sent for each 5 minute period. The periodMarker is the the value returned from time() and marks the start of current 5 minute period.

The difference between files and hash storage policies is that the files policy will read/write to disk several sets of counters (see WatchWhat) for each request. The hash storage policy attempts to improve performance by reducing disk access through the use of a shared memory hash table, occasionally swapping one or more sets of counters out to disk.

WatchSpoolFile filename
Context: <Directory>, <Location>

Used to watch by directory or location. Each filename must be unique amongst all the virtual hosts, file owners, IP addresses, and other WatchSpoolFile directives, otherwise the behaviour is unspecified. The filename is used to store runtime data about a location within the WatchSpoolDirectory and is used to extract this data using a URL of the form:

    http://www.snert.com/~filename/watch-info

The filename will also appear as an entry in the watch-table list.

WatchSpoolDirectory directory
Context: global

An absolute or server-root relative path to a directory where any support and runtime files will be kept. This directory must be readable and writeable by the Apache child processes. The default is /var/spool/mod_watch.

WatchWhat item ...
Context: global

mod_watch, by default, will collect information for the virtual host, file owner, and the server as a whole for each request made. For performance reasons, it might not be necessary to watch all these things. Specify one or more of the following to select which items to watch:

    all        nothing
    virtual-host virtual-host-by-port
    file-owner file-owner-by-port
    remote-ip remote-ip-by-port
    server server-by-port
    headers-in headers-out
    content-length  

The by-port variants collect and separate the information according to the port number, instead of grouping all requests regardless of port. For example, virtual-host-by-port would consider requests for http://www.snert.com/ different from https://www.snert.com/ and so would separate the information into "www.snert.com:80" and "www.snert.com:443". This also holds true if Apache is listening on other ports. Specifying both variants is allowed and would provide both a break down by port and the sum on all ports.

Specifying headers-in or headers-out will add the length of the headers of the request and response to the ifInOctets and ifOutOctets counters repectively. These are not on by default, because Apache does not count them in the acceess log and may skew comparisions made between web log analysis tools and the MRTG data.

Specifying content-length will count the value of the Content-Length header on POST requests. Note however that this value can be falsified by malicious people in order to deliberately skew the graphs and so its value is suspect. There is no way to guarantee that a CGI or module actually read the input data.

Specifying all, remote-ip, or remote-ip-by-port should be used with care as monitoring remote-IP addresses can generate large volumes of data and impact web server performance. Note that the values of ifInOctets and ifOutOctets reflect data received from and sent to the remote-IP address.

Download

Installation

  1. Unpack and install MRTG prior to installing this module.

  2. Unpack this module into its distribution directory.

  3. Before building the mod_watch binary, please review and alter the configuration settings found at the top of the Makefile. Note that Apache's "apxs" utility is required to have already been installed prior to building either static or dynamic versions of mod_watch.

  4. The fastest and simplest way to install mod_watch, is as an Apache DSO library. This presumes that Apache has been compiled and built with mod_so (see Apache documentation):

    cd (path to)/mod_watch-3.18
    make install-dynamic
    

    Some system administrators prefer to statically link their Apache modules into Apache itself. This requires more effort on the part of the system administrator. To start the process do the following and follow any additional instructions supplied by the Makefile:

    cd (path to)/mod_watch-3.18
    make install-static
    

    In both dynamic and static installations, the Makefile will install the following scripts into /usr/local/sbin (see SCRIPTDIR in the Makefile):

    apache2mrtg.pl, mod_watch.pl

    Also the directory /var/spool/mod_watch (see SPOOLDIR in the Makefile) will be created to hold the weenie files managed by mod_watch.

  5. The following must be added to the global server configuration within httpd.conf in order for mod_watch.pl to work (without modification):

    <IfModule mod_watch.c>
      # Allows the URL used to query virtual host data:
      #
      #	http://www.snert.com/watch-info
      #
      <Location /watch-info>
      SetHandler watch-info
      </Location>
    
      # Allows the URL used to query file owner and web
      # server data:
      #
      #	http://www.snert.com/~achowe/watch-info
      #	http://www.snert.com/~SERVER/watch-info
      #
      <Location /~*/watch-info>
      SetHandler watch-info
      </Location>
    
      # Intended for debugging and analysis of shared memory
      # hash table and weenie files:
      #
      #	http://www.snert.com/watch-table
      #
      <Location /watch-table>
      SetHandler watch-table
      </Location>
    
      # The following allows me to verify the version
      # and copyright for the purposes of support:
      #
      <Location /watch-copyright>
      SetHandler watch-copyright
      Order allow,deny
      Allow from snert.biz snert.com snert.info snert.net snert.org
      </Location>
    </IfModule>
    
  6. Use apache2mrtg.pl to parse the httpd.conf file to create an MRTG configuration script for all the defined virtual hosts. The output can be appended to or included in an existing mrtg.cfg file. There currently is no support to generate MRTG configurations for virtual hosts derived from mod_vhosts_alias or local user names.

    perl apache2mrtg.pl (path to)/httpd.conf >web.cfg
    

    Use this generated configuration with MRTG for a cron job that executes regularly to generate the graphs (see MRTG installation documentation). This configuration will periodically make calls to mod_watch.pl to obtain the MRTG target name, uptime, bytes in & out, requests in, and documents out.

  7. Stop/start the web server:

    apachectl stop; apachectl start
    

    Please note that "apachectl restart" does not always appear to load correctly a DSO module that uses shared memory and semaphores. For this reason it is recommended that you stop/start the web server instead.

  8. Once the Apache web server starts receiving requests, all the bytes received in requests, where a Content-Length header is defined and all the bytes sent in responses are added to the current 32-bit counters and stored.

    If the MRTG output is properly configured, you should be able to use a URL to see assorted graphs concerning data transfer rates of virtual hosts, file owners, and the web server.

Notes

  • Sample MRTG output & configurations for:

    Octets In & Out
    The follow MRTG configuration is produced by apache2mrtg.pl:

    Title[www.snert.com]: www.snert.com Data Traffic
    Target[www.snert.com]: `/usr/local/sbin/mod_watch.pl http://www.snert.com/watch-info`
    MaxBytes[www.snert.com]: 1250000
    PageTop[www.snert.com]: <h2>www.snert.com Data Traffic</h2>

    Requests & Documents
    The following MRTG configuration was done by hand to produce Request & Document traffic:

    Options[www.snert.com,req]: growright, perhour
    Title[www.snert.com,req]: www.snert.com Request &amp; Document Traffic
    Target[www.snert.com,req]: `/usr/local/sbin/mod_watch.pl -f ifRequests,ifDocuments http://www.snert.com/watch-info`
    MaxBytes[www.snert.com,req]: 1250000
    PageTop[www.snert.com,req]: <h2>www.snert.com Request &amp; Document Traffic</h2>
    YLegend[www.snert.com,req]: Hits per Hour
    ShortLegend[www.snert.com,req]: H/h
    Legend1[www.snert.com,req]: Requests (raw hits) received per Hour
    Legend2[www.snert.com,req]: Documents sent per Hour
    Legend3[www.snert.com,req]: Maximal 5 Minute Incoming Request Traffic
    Legend4[www.snert.com,req]: Maximal 5 Minute Outgoing Document Traffic
    LegendI[www.snert.com,req]: &nbsp;Hits:
    LegendO[www.snert.com,req]: &nbsp;Docs:

  • mod_watch has been built and/or tested on the following platforms (please email me your platform if you've made it work on something different):

    Cobalt Qube 1, RedHat Linux 5.1, Apache 1.3.22
    Use the default (System V) configuration.

    FreeBSD 4.2, 4.4, Apache 1.3.22
    Use the default (System V) configuration. Not yet tested with POSIX API.

    From <benoitdm at citeglobe dot com>: "If Apache is running in a jail on FreeBSD 4.4, you must setup jail.sysvipc_allowed=1 to allow use of shared memory in the restricted environment. This OID is a kernel configuration option, through you can set it up in /etc/rc.conf or on the fly with sysctl."

    HP-UX, Apache 1.3.22
    Use the default (System V) configuration.

    Max OS X 10.1, Apache 1.3.24
    Specify the following macros in the Makefile:
    DEF=-DSERIALIZATION_API=FCNTL_API -DSHARED_MEMORY_API=MMAP_ANON
    RANLIB=ranlib
    
    SunOS 5.7 (or better), Apache 1.3.22
    mod_watch/3.0: To build as a DSO module, Apache must be built with --enable-rule=SHARED_CORE. Modify the top of the Makefile to enable the following C defines and link with the correct libraries:
    DEF=-DSERIALIZATION_API=POSIX_API -DSHARED_MEMORY_API=POSIX_API
    LIB=-lrt
    
    Solaris 5.6, Apache 1.3.22
    mod_watch/3.0: To build as a DSO module, Apache must be built with --enable-rule=SHARED_CORE. Modify the top of the Makefile to enable the following C defines and link with the correct libraries:
    DEF=-DSERIALIZATION_API=POSIX_API -DSHARED_MEMORY_API=POSIX_API
    LIB=-lposix4
    
  • Help in French & English for MRTG and mod_watch.

  • Thank you to the folks at Security Space for including mod_watch in their monthly Apache Module Report.

  • ifOutRate introduced in 3.14 is experimental and subject to change.


- apache2mrtg.pl - mod_watch - mod_watch.pl -
- Welcome - Software -

light bulbs since 6 December 2001