Just another ComMetrics – social media monitoring, best metrics, marketing metrics weblog

2 ENISA – awareness raising study – what it does not explain

September 7th, 2007 · No Comments ·

Recently we reported:

1 ENISA – awareness raising study – what it can tell us

Press release (2007-08-22). How to measure success? ENISA presents the 1st report on current EU practices and assessing the success of information security awareness raising activities.

Information security awareness initiatives: Current practice and the measurement of success (July 2007). Heraklion, Crete: European Network and Information Security Agency (ENISA).

You should definitely have a look at this report. It provides an important contribution to the current discussions about how we could improve measures used to assess the effectiveness of awareness programs in the information security domain.


Under Working Program 2.1.2 — ENISA budgeted Euro 23,000 for an Awareness Study
Who did the study: PricewaterhouseCoopers LLP (PwC)
Who partcipated in the study: some firms from 9 European countries
What new ideas and facts does the study reveal: interesting ones indeed – see below.

We have pointed out in various postings how important awareness raising is.Now ENISA has released a study that deals with this issue.Today we provide you with some answers we received when we asked ENISA’s senior expert regarding awareness raising for some additional information regarding the study.ADDENDUM – SOME RESPONSES WE GOT FROM ENISA REGARDING OUR POSTING

1 ENISA – awareness raising study – what it can tell us

1) The survey was on a self-select basis, based on distribution to a large (but not measured) number of companies. The response rate was not measured. The results in the study should not be extrapolated as statistically valid for the actual population of European organisations, since

(a) the sample was not statistically drawn and is not free from bias (indeed it is heavily biased towards large (>500 employee) organisations),

(b) the response rate is low. The results are included in the study as indicative of what some of the leading organizations in this area do.

2) All questions were asked in a balanced way, so response bias should be minimal compared to the bias inherent in the self-select nature of the survey.

3) The respondents are predominately from large (>500 employee) firms. Of the respondents that answered the questions, the following is the breakdown:

What size of firms did participate in the awareness survey
number of employees number of
none 1
Less than 10 2
10-49 3
50 to 249 6
250-499 7
500-9,999 26

4) Respondents came from these countries: UK, Germany, Malta, Sweden, Luxembourg, Netherlands, Norway, Belgium and Switzerland.5) No data were collected regardging age or position held by respondents nor their education (e.g., what type of final university degree, computing science, sociology, etc.)HOW A BYSTANDER WOULD RESPOND THE THE COMMENTS AND EXPLANATIONS PROVIDED BY ENISA (see above)1) This clarifies the issues. However, the report (printed and electrnoic) does not point out this problem succinctly under a section such as limitations of this study or in the preface.

2) Asking questions in a balanced way does not mean they are formulated well as Sudman and Bradburn have been trying to tell us for years (e.g., Asking Questions: A Practical Guide to Questionnaire Design by Seymour Sudman and Norman M. Bradburn (Hardcover – Oct 29 1982)3) This information is interesting and shows that the researchers were using what is called an ordinal variable – a special type of categorical variable for which the levels can be naturally ordered (i.e. number of employees – from low to high). As ENISA stated large firms dominate.

Unfortunately, this does still not tell us how the size of the firm could have influenced the responses gotten for the survey.

4) Interesting is also that the report (p. 1) says … within the European Union (EU) but neither Norway nor Switzerland are members of the European Union. Maybe this mistake was not caught during final editing?

Especially interesting is if in such countries as Belgium and Switzerland several language versions of the survey were available (e.g., both Flemish and French are official languages in Belgium). Our guess is that possibly in Belgium, Switzerland and Luxembourg the French version was used to stay within the small budget. In Sitzerland a German version could also have been used if a German version was prepared for Germany.

Important here is that as cross-national studies have shown and the current political situation in Belgium might suggest, people’s attitudes and beliefs appear to differ across regions with different languages.

Also translations are time-consuming (i.e. cost a lot of money – see small budget for study) and require much work in order to minimize methodological pitfalls. The latter affect the validity of the data collected.

5) Not having collected such information does not provide one the opportunity to control to effects that could be due to size of firm or industry. It seems sensible to think that maybe the financial industry, in particular a large bank might use a different approach than a small manufacturing firm when it comes to a campaign to raise information security awareness amongst staff.

Without accounting for differences using some of the variables suggested here it is difficult to explain why certain firms might use information security awareness campaigns and why others might not.


This study was done on a very small budget (Euro 23,000) and very quickly (within just about three months). Unfortunately, if we consider that doing cross-national studies means spending about Euro 200 – 400 per collected survey (67 collected = already more than half of the budget gone) and at least Euro 400 or more for each interview conducted. This means there is not much money left for:

– designing the study and making sure it builds on past work in the area AND

– analysing data collected with the survey and those received via the interviews.

P. 20 is a mystery to all of us at CyTRAP Labs. We read the study several times, discussed it with various people including asking ENISA experts for clarifications did help pus a lot. What issues remain are:

a) how the findings presented in the report published allow one to arrive at the KPIs offered on p. 20 (also question about metrics on p. 19 does not help to understand better how and why the authors of the study arrived at these general KPI categories) and, most importantly,

b) how these KPIs will allow one to measure any kind of improvement regarding better security and prevention of incidents, virus infections or whatever.

Even if the KPIs suggested on p. 20 are exploratory, the methodological weaknesses in the study make their subsequent use a risky undertaking.

From experience it seems that a consultantcy such as PwC most certainly knows that the budget they got for this study is small. In fact, it makes it very difficult to provide a high quality study if not outright an impossible undertaking. So you may ask why take it on and risk being called for it as we do here? We do not know. Nevertheless, some people in the consulting business have pointed out to us that a consultant might do this to secure more interesting work down the line from the client. But the risk one takes is that the final report promises more than it can keep …. If this was the strategy followed by PwC it did ENISA a disservice.

Besides all the above, ENISA should be commended for having started to tackle this important issue regarding information security awareness initiatives and how their success can be measured. We do hope that many more reports will follow that try to shed more light on this critical issue.

Finally, we wholeheartedly agree with the authors of the study and ENISA that awareness is a first step, while prevention is the key:

CyTRAP Labs’ FAQ – best practices for protecting your organization’s systems against malware


4 Tips for building an effective Early Warning System – organizational and human resource issues

Common Malware Enumeration – CME – where is it going?


We would also like to thank ENISA’s senior expert – awareness raising for the the great help we got for getting thoughtful and detailed answers to our questions.


NExt week we have a follow-up to this story

– 3 enisa – awareness raising study – Lisboa – 3rd awareness raising dissemination workshop

To make it more convenient for you to take advantage of CyTRAP Labs’ offerings, just provide us with your e-mail address below. You can personalize your subscription to make it suit your needs.


→ No CommentsTags: respondents