Event streams in client software - Exploring anomalies

The input data

The source

I chose to listen to ETW events on Windows. I chose Microsoft-IEFRAME, which corresponds to events from Internet Explorer UI.

Encoding it

I give the data to NuPIC as three columns: timestamp, thread id, event. I treat the thread id and event as strings, which NuPIC just treats as a "category".


20:35:34.908, Thread 2408, (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 113 Start)
20:35:49.830, Thread 5336, (Microsoft-IEFRAME LegacyHistoryQuery 52 Start)
20:35:49.830, Thread 5336, (Microsoft-IEFRAME LegacyHistoryQuery 53 Stop)
20:36:00.899, Thread 3652, (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)
20:36:00.912, Thread 3652, (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 113 Start)
20:36:00.003, Thread 3652, (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)

Configuring the model

The swarm

My input:


SWARM_CONFIG = {
    "includedFields": [
        {
            "fieldName": "timestamp",
            "fieldType": "datetime",
        },
        {
            "fieldName": "thread_id",
            "fieldType": "string",
        },
        {
            "fieldName": "event",
            "fieldType": "string",
        },
    ],
    "streamDef": {
        "info": "event",
        "version": 1,
        "streams": [
            {
                "info": "TwentyGithubTabs1967",
                "source": "file://data/TwentyGithubTabs1967.csv",
                "columns": [
                    "*"
                ]
            }
        ]
    },
    "inferenceType": "TemporalAnomaly",
    "inferenceArgs": {
        "predictionSteps": [
            1
        ],
        "predictedField": "event"
    },
    "swarmSize": "medium"
}

Its output:

It ignores most of my input -- it finds it can do better by just focusing on the event sequence. Some day I'll prove it wrong.


MODEL_PARAMS = {'aggregationInfo': {'days': 0,
                     'fields': [],
                     'hours': 0,
                     'microseconds': 0,
                     'milliseconds': 0,
                     'minutes': 0,
                     'months': 0,
                     'seconds': 0,
                     'weeks': 0,
                     'years': 0},
 'model': 'CLA',
 'modelParams': {'anomalyParams': {u'anomalyCacheRecords': None,
                                   u'autoDetectThreshold': None,
                                   u'autoDetectWaitRecords': None},
                 'clParams': {'alpha': 0.050050000000000004,
                              'clVerbosity': 0,
                              'regionName': 'CLAClassifierRegion',
                              'steps': '1'},
                 'inferenceType': 'TemporalAnomaly',
                 'sensorParams': {'encoders': {u'event': {'fieldname': 'event',
                                                          'n': 121,
                                                          'name': 'event',
                                                          'type': 'SDRCategoryEncoder',
                                                          'w': 21},
                                               u'thread_id': None,
                                               u'timestamp_dayOfWeek': None,
                                               u'timestamp_timeOfDay': None,
                                               u'timestamp_weekend': None},
                                  'sensorAutoReset': None,
                                  'verbosity': 0},
                 'spEnable': True,
                 'spParams': {'columnCount': 2048,
                              'globalInhibition': 1,
                              'inputWidth': 0,
                              'maxBoost': 2.0,
                              'numActiveColumnsPerInhArea': 40,
                              'potentialPct': 0.8,
                              'seed': 1956,
                              'spVerbosity': 0,
                              'spatialImp': 'cpp',
                              'synPermActiveInc': 0.05,
                              'synPermConnected': 0.1,
                              'synPermInactiveDec': 0.05015},
                 'tpEnable': True,
                 'tpParams': {'activationThreshold': 14,
                              'cellsPerColumn': 32,
                              'columnCount': 2048,
                              'globalDecay': 0.0,
                              'initialPerm': 0.21,
                              'inputWidth': 2048,
                              'maxAge': 0,
                              'maxSegmentsPerCell': 128,
                              'maxSynapsesPerSegment': 32,
                              'minThreshold': 11,
                              'newSynapseCount': 20,
                              'outputType': 'normal',
                              'pamLength': 3,
                              'permanenceDec': 0.1,
                              'permanenceInc': 0.1,
                              'seed': 1960,
                              'temporalImp': 'cpp',
                              'verbosity': 0},
                 'trainSPNetOnlyIfRequested': False},
 'predictAheadTime': None,
 'version': 1}

Making the output glance-able

I started with raw output that looked something like:


timestamp  thread_id   event   anomalyScore    prediction1
20:35:34.908   Thread 2408 (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 113 Start) 0   (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)
20:35:49.830    Thread 5336 (Microsoft-IEFRAME LegacyHistoryQuery 52 Start) 1   (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)
20:35:49.830    Thread 5336 (Microsoft-IEFRAME LegacyHistoryQuery 53 Stop)  1   (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)
20:36:00.899    Thread 3652 (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)  1   (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)
20:36:00.912    Thread 3652 (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 113 Start) 0   (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 113 Start)
20:36:00.003    Thread 3652 (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 114 Stop)  0   (Microsoft-IEFRAME Browseui_Tabs_WaitMessage 113 Start)

I changed it to this:

I color-coded the most common events.

Noteworthy moments in the output

Foreword

I used the generic UI thread loop events as "training wheels" for validating my model.

This weekend I never took the training wheels off.

The beginning

Overlapping start/stop events from multiple tabs

Even after hundreds of thousands of events, this still causes a burst of anomalies.

This is why I tried adding Thread IDs, but swarming still determined that it was better off without them. But maybe it would change its mind with a different dataset.

The traumatized model

These unexplainable anomalies keep happening. Sometimes they even happen when the event was correctly predicted with 100% likelihood!

The case of the traumatized model

As seen above, I see unexplainable bursts of anomaly scores even when the prediction was 100% correct.

I had theories about why this happened. It involves the classifier. The next step is to observe more than is currently available in the OPF prediction_result.

Say hello to predictive_bits:

Let's explore two predictions.

The good:

The bad:

What were their bits?