Just another ComMetrics – social media monitoring, best metrics, marketing metrics weblog

3 ENISA – awareness raising study – better prevention thanks to data crunching

September 12th, 2007 · No Comments ·

Recently we reported:

1 ENISA – awareness raising study – what it can tell us

2 ENISA – awareness raising study – what it does not explain

Today we provide you with some answers we received when we asked ENISA’s senior expert regarding awareness raising for some additional information regarding the study:


Under Working Program 2.1.2 — ENISA budgeted Euro 23,000 for an Awareness Study
Who did the study: PricewaterhouseCoopers LLP (PwC)
Who partcipated in the study: some firms from 9 European countries
What new ideas and facts does the study reveal: interesting ones indeed – see below.

We have pointed out in various postings how important awareness raising is. ENISA’s recent release must be applauded. As important is that ENISA’s report:

Information security awareness initiatives: Current practice and the measurement of success (July 2007). Heraklion, Crete: European Network and Information Security Agency (ENISA)

also illustrates 2 things very nicely:

1 Why we need

    a)data that can be used to generalize from, and
    b) enough information about the methodology used to collect data so we can repeat the study, and

2 how data crunching can help us to improve our understanding of the issue investigated in order to design better programs for raising information security awareness.

In fact, data can support our efforts to create a more secure IT environment by allowing the extraction of insight from data gathered during the operation of awareness training, or implementing new security policies in the workplace.


Better information security awareness training thanks to data crunching (Please click on the link, Login as guest – click on this link again and voila free access)
Challenge How research may address it
Could the effect differ across industries – yes if we control for the effect (e.g., financial industry versus others, men vs. women) control variable
Could one say that having anti-virus software on one’s PC (M) mediates the causal effect of awareness training (X) on malware infection rates on user’s PCs (Y) – test using a group of users with and one without anti-virus software on their PC – both groups attending an awareness raising workshop and so on mediating variable
How can one describe the nature and process by which the independent variable affects the dependent one such as privacy policy violations or security incidents – test effect with help of moderating variable (latter could be awareness training) moderating variable
What is your attitude toward information security awareness programs – ask participants some questions direct measure for metrics
What is your attitude regarding information security – how often does the person violate security policy indirect measure for metrics
Can the results be repeated by another person – yes if we use a metric or yard stick and outline what research methodology we used…. so somebody else can repeat the study reliability
Does it measure what it is supposed to measure – yes but only if we do use a metric stick if this is our standard we follow – using a yard stick results in invalid data if the measure we agreed to use is based on meters and centimeters validity

Regression can help to uncover connections between, say, the chances of people violating a privacy policy or being responsible for a data security breach and factors influencing that risk, such as age, income and type of position held in the organization. Regression (a statistical method) helps uncover these connections.

To reveal the influence of a single factor – say, attending an information security awareness workshop – in the presence of lots of extraneous effects, randomisation can help. In this, people are randomly allocated to two groups:

a) those who will be exposed to the workshop, and
b) those who will not be explosed to this workshop.

In psychology one also calls this using control groups, namely one that is exposed to the training and the other one that is not.

Differences between the two groups can than be put down to the effect of the training, as all the other factors have been evenly distributed between them.

Using control variables (Please click on the link, Login as guest – click on this link again and voila free access) or a mediating variable can further help in discovering if training helps certain groups of employees more than others (e.g., using age or gender).

Medical scientists have used the technqiue for decades to identify life-saving new therapies, via patients recruited into randomised controlled trials (RCTs). So have industrial and social psychologists for testing leadership and other motivational theories and techniques in various settings.

We hope that during ENISA’s upcoming workshop in Lisboa on 2007-09-18, these issues will be discussed. Only then if we are clear about the methodology that will be used, addressing both randomization as well as control and mediating variables will we be able to find effective ways in measuring awareness raising compaigns. Moreover, such findings will be helpful to share and disseminate all to facilitate an improvement in information security for SMEs and home-users.

ENISA -2007-09-18 Lisboa – 3rd awareness raising dissemination workshop

This we should all keep in mind
correlation is not causation (wider gun ownership correlates with lower crime rates in some States in the U.S. – HOWEVER does not mean the former caused the latter!)
‘garbage in’ still means ‘garbage out’ (i.e. collecting ‘bad’ data does not result in valid information that can be used to make important and possibly expensive decisions)
data crunching may be the worst possible basis for making a big decision about awareness raising policies – apart from all others :-)

PS.In a preliminary work program for 2008 (file is named 2007-08-24_Preliminary WP2008.doc) amongst many laudable goals the program entitled:MTP 2: Developing and maintaining cooperation between Member States

is one of many activities that ENISA intends to pursue during 2008. However, as most others that are proposed in this document, it could benefit from a methodology that is clearly defined and Key Performance Indicators (KPIs) that are specific and defined. Hence, KPIs must go beyond general statements but, intead, be based on metrics that allow one to evaluate a program. In particular, what makes an activity a success and what does not?

If metrics or performance measures are not agreed upon and clearly defined before the work program is approved, we will take the risk of getting reports and activities whose impact upon the buttom line cannot be demonstrated. Hence, at its best the evaluation process ends up in a useless exercise and at its worst in a farce. (PS. other issues are the allocation between Work Packages or WPKs that is difficult to understand)

Under MTP 3: identifying emerging risks to create trust and confidence

Here it seems that it would be most fruitful to do a work process assessment whereby the objective should be to develop practical tools that help policy-makers in managing emerging risk applications and disaster recovery more effectively and efficient than it currently seems possible.

Success for MTP1, MTP2, MTP3 requires a clear methodology and KPIs that are the foundation for useful data crunching.  In turn, this allows the exctraction of insight from data gathered during the operation of a campaign to raise information security awareness and/or reducing security incidents due to human error.

Regression and randomization are important tools but, as pointed out above data collection has to assure that the information obtained can be used to account for many effects. Failing to measure these variables once again in the Work Packages for MTP1, MTP2, and MTP3 for ENISA’s program for 2008 will not all to make sure that they or other effects might not account for the outcome. In turn, decisions could be based on numbers or data that are simply not the right ones….. the wrong decision, however, is costly at best and a disaster for improving information security at worst.

Key Performance Indicators (KPIs) (Please click on the link, Login as guest – click on this link again and voila free access) assessing the effectiveness of awareness raising efforts have to be both, reliable and valid to allow us to make sensible conclusions from those data collected. And yes, identifying KPIs that can be measured and allow others to repeat one’s study is a good thing….. of course, if you believe in the merit of scientific research that is – you should considering your health.

To make it more convenient for you to take advantage of CyTRAP Labs’ offerings, just provide us with your e-mail address below. You can personalize your subscription to make it suit your needs.


→ No CommentsTags: awareness · effectiveness · enisa · explain · measurement · raising · study