Evaluation of Rulesets - Process

The first part of this article contained several thoughts about measuring the qualities of a given ModSecurity ruleset. This second part describes a setup that can be used to measure the qualities of your ruleset. First I will give an overview of whole measuring process and the testing environment that is used to evaluate a specific ruleset. Part 3 will contain results of testing several rulesets as the core-rules and the basic gotroot-rules.

Evalution Process

The ruleset evaluation is based on a set of requests, in the following referred to as a test-scenario. These scenarios are created from ModSecurity audit-logs. The ruleset to be evaluated is employed in a reverse-proxy server that consists of an apache webserver with the ModSecurity and mod_proxy modules integrated. This is a quite common setup for a generic protection of web applications.

For evaluation, the requests of the test-scenario are re-injected into the proxy-server. This will result in a set of audit-events (alerts) in the audit-log of the proxy server. The alerts then need to be divided into real attacks, false-positives. Since this task can hardly be automated it needs to be done manually. The results can then be used to calculate the detection factor as described in the first part of this article. The following graphic depicts the process of evaluation.

The apache servers that are shown on the right side of that graphic might be two distinct real servers or virtual hosts included in one apache instance. The basic components of the environment are the Test-Client which injects the requests of the test-scenario into the reverse-proxy. This proxy includes the ruleset that is to be evaluated and sends all requests to the web-server on the right side of the picture. The web-server maps all incoming requests to a dummy url which is delivered back to the client. Since a ruleset should fire in ModSecurity phases 1 or 2 to prevent attacks against an application, there should be no need to evaluate the server response. This approach might be arguable but it eases the setup of the test environment as there is no need to setup a specific target application.

Test-Client

The clients task is to read a set of requests, e.g. from an audit log, and inject it into the proxy server. This can be achieved by the AuditViewer for single requests or by the use of my TestClient application. The TestClient supports several transformations that can be applied to requests before their injection. Additionally, REMO provides a command-line tool for re-injecting requests.

The important fact of the test-client component is, that all its scenarion-injections need to be easily repeatable to provide an equal scenario for each of the rulesets that are to be benchmarked.

Reverse Proxy

The ruleset is employed within the reverse proxy component. This proxy is configured to run each request through the ruleset and then proxy the request to the backend server if the rules did not block the request. Each request that is found to be an attack needs to be logged into the audit-log of the proxy server. These audit-log contains the requests that are subsequent used to calculate the detection factors of the ruleset. Therefore the audit-log needs to be clean before each run of a test scenario. After the run, the audit-log has to be saved together with the test scenario for later analysis.

    <VirtualHost *:80>
        ServerName proxy.jwall.org
        ServerAdmin chris@jwall.org
        
        DocumentRoot /path/to/root

        <Directory /path/to/root>

        </Directory>

        Include /path/to/modsecurity-default.conf
        Include /path/to/ruleset.conf

        <Proxy *>
            Order allow,deny
            Allow from all
        </Proxy>

        ProxyPass / http://backend.jwall.org/
        ProxyPassReverse / http://backend.jwall.org/

        ErrorLog /var/log/apache2/proxy.jwall.org-error.log
        AccessLog /var/log/apache2/proxy.jwall.org-access.log
    </VirtualHost>

In the above example setup, the file modsecurity-default.conf contains the basic settings for ModSecurity that are not specific to a ruleset, such as the location of the log-files, temporary data, etc. The ruleset that is to be evaluated is assumed to be found in /path/to/ruleset.conf.

Backend Web Server

The backend server is used to provide a target for the proxy-module. Its task is to respond with a dummy page to each incoming request. This is done by the use of mod_rewrite as done in the following example configuration:

    <VirtualHost *:80>
        ServerName backend.jwall.org
        ServerAdmin chris@jwall.org
        
        DocumentRoot /path/to/root

        <Directory /path/to/root>

        </Directory>

        RewriteEngine On
        RewriteRule !^/index.html$ /index.html [L]

        ErrorLog /var/log/apache2/backend.jwall.org-error.log
        AccessLog /var/log/apache2/backend.jwall.org-access.log
    </VirtualHost>