Identifying persistent activites

Most computers have some capability to keep track of their activities over time. This is sometimes built into the system. In addition, open source developers have created tools that provide detailed data about the performance of the processor, disks, memory, network etc. The sysstat utilities are excellent examples of such tools. The sysstat tools collect a wealth of information about the performance of a linux computer. It is usually set up to run at regular intervals, usually every ten minutes. Each day's information is usually collected into a file; normally 30 days of history are kept in this way.

The summary file prepared each day is in a readable form, but it contains a lot of information in a number of different tables. This data can be interpreted to understand what is happening on your computer. For example, if the processor is very active, that means programs are doing something intense, such as generating graphics or analyzing large datasets. If the input/output activity is high, that usually means there is heavy disk use, perhaps from data operations. That data operation could be expected activity from some data fetching and writing. It could also be because someone has made your computer into a zombie and is using it to send spam. In the latter case, you will also see higher network activity along with the disk activity.

With some processing (a part of our software) we can convert the data from sysstat into a stream of data containing about 100 variables. (There are more variables tracked by sysstat, but usually only about a 100 have relevant values, depending on the main function of the system.) We can further convert this information into a gray scale picture.

This picture shows the activity level of several variables over a few days. As it turns out, each day contains 143 observations (it is 24 hours, one observation about 10 minutes, but there is no observation at the starting time of 12 midnight/am, thus we have 144-1=143 observations.) The picture is therefore marked in units of 143. The darker regions indicate higher activity. In the interactive version of this, you can click on areas of the picture to find the time of each observation (and a few around it) as well as the variables that are involved.

The gray scale picture can be colored with false colors, but it may be more useful to select a few variables to display as functions.

The functions are drawn over the gray scale distribution. Each color corresponds to a location indicated on the left, thus for example the blue and violet graphs correspond to some variables tracked at the bottom of the picture while greener colors are towards the middle. In this case, the middle corresponds to CPU activity and the bottom part relates to memory usage.

These pictures are relatively easy to understand compared to looking at the data tables. You can also scroll through thirty days pretty easily to identify unusual activity visually. However it is also possible to identify persistent activity analytically.

In the second picture above, you can see some of the functions being higher at some times. Many unusual activity detectors would flag these, but mostly these are false negatives. While users have established ways of working, they do not always do the same things. In this case, the blue lines that are high correspond to memory usage. These are usually higher around 9 am and 3 pm due to the way this user works: around 9 he starts by testing some things done previously, and around 3 he is ready to run initial tests on the things done today. Some tests are more intense than others, but there is nothing unusual in that.

A crook's strategy is different from this. He may for example use the CPU to do a search, but his strategy is essentially one of experimentation. This experimentation is generally done over a long period. To avoid detection, most "sophisticated" crooks (you are still a crook though) will try not to raise flags that monitor high activity. But they will have persistent unusual combinations of activity, for example disk activity correlated with network activity (while stealing files.)

This strategy can be detected by our analysis. In this case, our strategy analysis looks for such data stealing activities in terms of combinations of variables from the sysstat collection.