{CWoC} Details of the code behind the Altiris IIS Log Analyzer versions 2 (aila2)

Table of Content

Introduction

In Part I [1] we have started looking at the code behind aila2 as it was in version 2, and modified it to have a single source file for quick and easy compilation.

In Part II here, we will take the results from Part I and do some code clean up whilst further review what makes up the codebase and and how the code is modeled around simple data structures.

Note that we start with the attached file aila2.cs_.txt and the result of today's work is attached as aila2-v2.1.cs_.txt.

Top

Cleaning up the constants class

The constant class was inherited back from the version 1 of aila (written in plain C) and was changed over time to support json output instead of plain text output. As such the class as grown overtime and was not really well maintained, so before we can look at what the class contains, let's make sure that there are no unused members.

To do so we scan through the class and for each member we search and count the occurrences of the member name. If the count returns 1 the member is unused and due for deletion. I went through this process and we're not going to document what was removed (it's not interesting) but rather the lines to remove, in order to get a clean class:

delete lines 780 to 822
delete lines 829 to 824
delete lines 858 to 869
delete lines 893 to 902
delete lines 909 to 926
delete lines 944 to 950
delete lines 951 to 955

This clean-up shrinks the aila2.cs file from 1,062 lines / 42,691 bytes down to 952 lines / 40,144 bytes.

Top

Overview the constants class

Here is an outline of the class. You will notice that it only contains read-only members (of type string[] or enum) hence the class name ;-).

class constants {
    public enum HOURLY_TABLE;
    public enum ATRS_IIS_VDIR;
    public static readonly string [] atrs_iis_vdir;
    public enum ATRS_IRM_PARAMS;
    public static readonly string[] atrs_irm_params;
    public static readonly string[] json_irm_params;
    public static readonly string[] atrs_task_req;
    public static readonly string[] json_task_req;
    public enum ATRS_AGENT_REQ;
    public static readonly string [] atrs_agent_req;
    public static readonly string [] http_mime_type;
    public enum IIS_STATUS_CODES;
    public static readonly string [] iis_status_code;
    public enum IIS_WIN32_STATUS;
}

Top

Going deep: looking at the mime-type handling code

Now, in order to really understand how things work let's take a specific accounting type and review how it is handled in the code. Below is an subset of the file that only retains the higher level code structure and anything related to mime-type accounting.

namespace Symantec.CWoC {
    class aila2 {
        class ResultSet {
            public int[] MIME_TYPE_hit_counter;

            public ResultSet() {
                MIME_TYPE_hit_counter = new int[constants.http_mime_type.Length];
            }
        }

        class LogAnalyzer {
            private void AnalyzeLine(ref string line) {
                // Analyse mime types
                Logger.log_evt(log_levels.debugging, "Running analysis - part II (mime type) ...");
                Analyze_MimeTypes(ref current_line[(int)FieldPositions.uristem]);
            }

            private int Analyze_MimeTypes(ref string uri) {
                int i = 0;
                foreach (string type in constants.http_mime_type) {
                    Logger.log_evt(log_levels.debugging, string.Format("Checking mime-types {0}", type));
                    if (uri.EndsWith(type)) {
                        // Increment mime-type counter
                        results.MIME_TYPE_hit_counter[i]++;
                        Logger.log_evt(log_levels.debugging, string.Format("Current request mime type is {0}.", type));
                        return i;
                    }
                    i++;
                }
                results.MIME_TYPE_hit_counter[i - 1]++;
                return i - 1;
            }

            public void DumpResults() {
                // MIME TYPE STATS
                output.AppendFormat("\t\t\"mime_type\" : [\n");
                output.AppendFormat("\t\t\t[\"Mime type\", \"Hit #\"],\n");
                for (int j = 0; j < results.MIME_TYPE_hit_counter.Length; j++) {
                    output.AppendFormat("\t\t\t[\"{0}\", {1}],\n", constants.http_mime_type[j], results.MIME_TYPE_hit_counter[j].ToString());
                }
            }
        }
    }

    class constants {
        /******************************************************************************
         * Http Mime types enumeration
         * Standard html mime types found in a Notificatoin Server
         ******************************************************************************/
        public static readonly string [] http_mime_type = new string []{
            "htm","html","asp","aspx","asmx","ascx","axd","ashx","xml","css","js","gif","png","jpg","Other"
        };
    }
}

This makes understanding the code a whole lot simpler now. In the aila2.ResultSet embedded class we define and initialise the MIME_TYPE_hit_counter string array to hold "constants.http_mime_type.Length" entries. In this manner we can add any mime types (as we will do later) without having to modify code in too many locations.

In the LogAnalyzer class we define a method to check for mime type details in a given log line (represented as a string array). This method is called from the AnalyzeLine method, and checks if the current URI request ends is defined in the "constant.http_mime_type". If it is we increment the relevant hit counter ("results.MIME_TYPE_hit_count[i]++"). If not we update the last counter which is the "other" bucket. Finally we find the mime type variables used in the DumpResults method. This simply prints out the JSON formatted string section that pertains to mime type.

Top

Mime type JSON output sample

"mime_type" : [
            ["Mime type", "Hit #"],
            ["htm", 1],
            ["html", 0],
            ["asp", 1469],
            ["aspx", 43307],
            ["asmx", 0],
            ["ascx", 0],
            ["axd", 0],
            ["ashx", 0],
            ["xml", 17],
            ["css", 0],
            ["js", 0],
            ["gif", 0],
            ["png", 0],
            ["jpg", 0],
            ["Other", 1114]
        ],

Top

Adding new mime types support

As briefly mentioned above we can easily add new mime types for support. For example I have a new customer that works with a lot of Mac computers. So we'll add the DMG type. Also from a Package Server standpoint I know we can have a lot of exe and msi downloads, so these can be added too along with scripting types.

And here is how you add support for new mime types: paste the following 30 lines on-top of lines 904 to 924.

        /******************************************************************************
         * Http Mime types enumeration
         * Standard html mime types found in a Notificatoin Server
         ******************************************************************************/
        public static readonly string [] http_mime_type = new string []{
            "htm",
            "html",
            "asp",
            "aspx",
            "asmx",
            "ascx",
            "axd",
            "ashx",
            "xml",
            "css",
            "js",
            "gif",
            "png",
            "jpg",
"dmg",
            "msi",
            "msp",
            "exe",
            "sh",
            "pl",
            "vbs",
            "cmd","Other"
        };

Top

Correcting sub-optimal processing

I have no doubt that you, sharp as you are dear reader, spotted a sub-optimal piece of code in the extracts above. Here it is again:

                foreach (string type in constants.http_mime_type) {
                    Logger.log_evt(log_levels.debugging, string.Format("Checking mime-types {0}", type));
                    if (uri.EndsWith(type)) {

As you can see from above the mime types are not defined with the seperating dot. We'ere not defining .htm, .html etc. And when we check the URI do match the end of the URI with the mime type constant, without a dot. This means we may account for uri queries that end with xml, which could include "myfile.xml" as well as "folderwithxml". So we will fix this right away with a simple change going line 507:

                    if (uri.EndsWith("." + type)) {

Top

Simplifying the output

As you can see from the modifications we have made today the mime type output could contain a lot of entries where the hit count is 0. So we'll also modify the DumpResults code that pertains to mime type to not output data when the hit count is 0. Here it is:

                // MIME TYPE STATS
                output.AppendFormat("\t\t\"mime_type\" : [\n");
                output.AppendFormat("\t\t\t[\"Mime type\", \"Hit #\"],\n");
                for (int j = 0; j < results.MIME_TYPE_hit_counter.Length; j++) {
                    if (results.MIME_TYPE_hit_counter[j] > 0)
                        output.AppendFormat("\t\t\t[\"{0}\", {1}],\n", constants.http_mime_type[j], results.MIME_TYPE_hit_counter[j].ToString());
                }
                output.Length = output.Length - 2; // Remove the last ",\n"
                output.AppendLine("\n\t\t],");

Top

Wrapping up

We can now compile the attached file (as aila2-v2.1.cs_.txt) and run the program to check the output. Using the same log file as for the sample results above we now have:

"mime_type" : [
            ["Mime type", "Hit #"],
            ["htm", 1],
            ["asp", 1469],
            ["aspx", 43307],
            ["xml", 17],
            ["dmg", 12],
            ["msi", 17],
            ["exe", 120],
            ["sh", 46],
            ["pl", 5],
            ["vbs", 46],
            ["cmd", 4],
            ["Other", 864]
        ],

So we have a more precise output (both in accuracy and accounted mime type) that is clear of irrelevant (not requested) mime types.

Top

References

[1] {CWoC} Details of the code behind the Altiris IIS Log Analyzer versions 2 (aila2) - Part I