Just another ComMetrics – social media monitoring, best metrics, marketing metrics weblog

Security metrics – does methodological preciseness help?

December 13th, 2006 · No Comments ·

Previously we have addressed various issues regarding security metrics that must be considered in order to develop measures that make sense, such as:

Security metrics – what affects business continuance?

Security metrics – do you know what your boss wants?

Security metrics – spreadsheets are full of errors as we know

CyTRAP Labs – guide – developing IT security metrics that work for you

Today we will discuss a bit the preference of some people to invent new terminology that might result in more confusion than we need.

Andrew Jaquith recently shared some examples he uses in his latest Security Metrics: Replacing Fear, Uncertainty and Doubt from Addison-Wesley and Symantec Press (2007) to illustrate what it takes to use good security metrics. He shared one example which is as follows:

… It involved comparing users’ training class latency per facility (i.e., elapsed days since last security training class) with whether or not users in that facilities swiped their cards an even or odd number of times each day. (This is a facility that requires card-swipes to get in AND to get out. Odd swipes suggests tailgating…) Training latency is a leading measure; tailgating prevalence is the lagging measure. In the case of this enterprise, the numbers correlated very highly. Brilliant, simple, powerful.’

What confused me was the term leading measure, most of us would probably just call it the independent variable. Also, what Andrew Jaquith is calling the tailgating prevalence the lagging measure would most call the dependent variable in most other people’s statistic or math books.

Besides this definitional confusion this created for me, there as some further methodological issues that warrant some discussion. For instance, it could be that training has a major effect (e.g., causal) on tailgating prevalence regarding card-swipes for getting in and out of a building or a facility. On the other hand, it could also be quite likely that other factors, such as socio-demographic variables (e.g., age, income, education levels) all have effects on this dependent variable called tailgating prevalence. In fact some of these variables’ effect may be far greater than training class latency. Moreover, after having controlled for these effects, training may no longer have a significant correlation or causal effect on the dependent variable called tailgating prevalence in this example.

So before we can say it was a ‘…brilliant, simple, powerful’ metric that confirmed the importance of awareness raising training in the security domain, these questions and methodological issues do warrant careful consideration. Until then, the training class latency per facility metric alone cannot take credit to have the highest correlation or effect upon the dependent variable called tailgating prevalence.

Also, it seems a bit unfortunate to use terms such as leading measure and lagging measure when scientists and statisticians have used labels that are far more established and more widely used and understood. Until then I continue calling a leading measure an independent variable and the lagging measure the dependent variable. As well, other things being equal, various additional independent variables’ possible effect or correlation with the dependent variable must be controlled for, before we should go ahead and just decide that we have a great metric = A causes B.

You be the judge …


Incidentally, there are nice ways to prevent tailgating through doors that require card-swiping. Moreover, what are the consequences for the firm and the individual when he or she has been tailgating. Maybe these employees simple do not understand or the firm has not established any consequences. Moreover, a system that allow this may have limited value anyway.


→ No CommentsTags: dependent · effect · facility · lagging · prevalence · tailgating · training · variable