How to Measure Security Awareness Training Effectiveness

Any successful enterprise cybersecurity plan will include security awareness training. But taking a course once in a while or only during onboarding doesn’t mean workers will make safer choices on the job.

You have to look at what the users do in real-life situations and not what they have learned in the past to figure out if the training you’ve provided is actually working.

Measuring the effectiveness of security awareness training comes down to measuring behavioural changes, identifying the remaining risks and seeing if that change in behaviour actually leads to an overall noticeable change in security.

Why Simply Completing Training Isn’t Enough

Training completion rates will indicate the number of your people who have actually completed the training. Additional assessments can be used to find out if they have really understood the content they just completed (or not). Neither of these measures proves that the employees will make use of what they have learned when push comes to shove.

An employee might pass a phishing assessment but still click on a suspicious link when distracted or under pressure. Another might identify a security threat but not report it simply because the reporting process is unclear.

Security teams should look at three things in order:

1. Participation: Did the staff get and complete the training?
2. Knowledge: Do they know the signs of threats and do they know how to appropriately react to those signs?
3. Behaviour: Do they do the right thing in a real-life workplace situation?

The third one is especially important when it comes to measuring security awareness training effectiveness. This measurement helps organisations understand if training is indeed changing the way employees respond to security risks.

What Security Metrics Should Be Tracked By Teams?

There isn't one single metric that will tell you the effectiveness of security awareness training. What is most useful to do will depend on the risks faced by an organisation regularly, and which behaviours need to be improved the most.

There are a number of indicators security teams can use to understand whether employees are implementing what they’ve been taught, including:

1. Phishing simulation results

Phishing simulation tests are exercises in a controlled environment that send fake messages to employees, mimicking real phishing messages. Their purpose is to show how employees react to potentially suspicious messages in a non-training environment.

Here security teams can see the percentage of employees who clicked on fake malicious links, entered sensitive information or reported the message through the proper channels.

The results must be compared over time, taking into account the difficulty of each test. Employees, for example, may be less likely to click on a blatant phishing attack than on a plausible email that appears to come from a supplier. Click rates alone therefore don't provide a complete picture.

It also matters what employees do after clicking a link. Someone who quickly reports their mistake responds differently from someone who takes no further action.

The goal is to identify in which areas employees need support, rather than simply taking a count of how many people fail a test.

2. Reporting rate and response time

The sooner employees report suspicious activity, the quicker security teams can investigate and remedy security threats.

Useful metrics to look out for include:

• The percentage of simulated phishing messages that are reported.
• The number of relevant suspicious activity reports received.
• Time taken to report an untrustworthy message.
• The percentage of reports that contain useful information.

Compare these figures over time and across similar exercises, taking account of the types and number of suspicious messages. Look at the proportion of relevant threats reported, how quickly employees report them and whether reports contain useful information.

Higher reporting rates can indicate better awareness, but the number of reports alone doesn’t prove this. It may also reflect an increase in phishing activity.

Rather than expecting reporting rates to increase indefinitely, look for consistent reporting behaviour, timely responses and accurate reports across comparable exercises.

3. Repeated risky behaviour

Another useful measure is whether employees continue to make the same mistakes even after they have been trained to correct them.

Depending on the organisation, this could include sharing sensitive information multiple times over unapproved channels, not validating unusual requests, or ignoring established security processes.

Problems that are recurring may be an indication that workers need more direction in a way that suits their learning style or further practical training. It may also be that the reporting procedures are making it difficult to engage in safe behaviour.

If staff continue to approve strange payment requests without checking them themselves, then the organisation needs to consider whether its verification processes are clear and easy to follow. The focus is on finding and fixing recurring risks, not on blaming the employees.

But, Before You Start Tracking Progress

Setting a baseline is crucial before you start measuring any security awareness training progress.

Determine where employees’ performance currently is and only then should you start a new training program or implement intervention. That way you can have something to compare the results to.

A baseline might be:

• The percentage of employees who report receiving fake phishing messages.
• The percentage of people who click on simulated malicious links.
• The average time to report suspicious activity.
• The frequency of selected policy violations.

Security teams can then repeat these measurements after training has been completed and compare the before-and-after results.

For meaningful comparisons, keep tests as consistent as possible. A difficult phishing simulation shouldn’t be compared directly with an obvious one without taking the difference into account. Where appropriate, compare results across similar employee groups or roles.

Organisations should also re-measure performance after a period of time. An improvement immediately after training doesn’t mean that employees will maintain the same behaviour months later.

Leading Versus Lagging Indicators

Security teams should measure training outcomes using both leading and lagging indicators:

• Leading indicators help teams assess whether behaviours and controls are moving forward in the desired direction. Examples of leading indicators include faster reporting of suspicious messages and following verification procedures.
• Lagging indicators are all about the past. They show you what has already happened or gone wrong, such as successful phishing attacks, confirmed security incidents and losses from social engineering.

Both are useful indicators, but should not be used in isolation to get the complete picture.

More reports of suspicious emails, for instance, could suggest that employees are becoming better at identifying threats. A decline in successful phishing attacks may also be heartening, but may instead be a function of improvements in email filtering or other security controls.

Using a mix of both types of indicators enables organisations to have a clearer picture of true progress.

Link Training Results to Business Risks

Training metrics are more useful if they relate to the real risks the organisation experiences and wishes to reduce.

Security teams should move beyond measuring awareness in general and focus on more specific threats, the behaviours that could help prevent them and the outcomes that need to be tracked.

For example, a fraudulent payment request may be of concern to an organisation. The process might look like this:

• Security risk: Employees approve fraudulent payments or changes to suppliers' bank details without verifying them first.
• Target behaviour: Employees verify unusual requests through a validated channel before taking action.
• Metrics: Employee responses to relevant phishing simulations, reports of suspicious payment requests and compliance with verification procedures.

This gives security teams better visibility into whether the training is truly closing the gap when it comes to a relevant risk.

It also helps to see when training alone isn’t enough to inspire change. If employees understand the rules, but they can’t easily comply with them, the organisation may need to improve its approval process or add additional guardrails.

How to Use Measurement for Improvement

Collecting metrics is only useful if the results inform decisions. Security teams can follow five steps to evaluate and improve their training programmes:

1. Identify the behaviour you want to change

Start with one specific action employees should take, such as reporting suspicious emails or confirming unexpected payment requests before taking action.

2. Establish a baseline

Measure current behaviour of employees before introducing training or changes. This gives you a baseline to see how you're doing.

3. Conduct specific training

Concentrate on the most important risks and behaviours to your organisation. Use real-world examples to help employees understand how to respond.

4. Measure the outcome

Do the assessments again and compare the results with the baseline. Consider the possibility that other changes may have affected the results.

5. Make the program better

Use the findings to identify gaps, improve training, and address issues with existing processes. Keep measuring performance to see if improvements stick.

This approach makes measurement an ongoing process rather than a once-a-year reporting exercise.

Measuring Security Awareness Training Effectiveness Goes Beyond a Score

To determine the effectiveness of security awareness training, organisations need to look beyond course completions and assessment scores to what employees actually do.

Useful evidence includes phishing simulation results, reporting rates, repeat risky behaviours and security incidents. Measuring these over time allows organisations to understand where they are improving and where they need to improve more.

The findings should guide the next steps. Some gaps may need better training, some clearer procedures and some stronger security controls.

Ultimately, success depends on how well employees identify threats, react appropriately and help to reduce security risks.

By connecting training outcomes to observable behaviour and business risks, organisations will be better positioned to make decisions around their security awareness programs and where to focus their efforts.

For more on the latest security-related developments, threats and best practices, check out the EM360Tech Security feed.