Imagine your organization’s risk management approach can only deal with qualitative information (such as policy papers, incident reports, or assessments) that describe risk in terms of high, medium and low. Such an approach would miss the patterns and trends that could be spotted by including quantitative information (such as network flows, or numbers of security incidents). Drawing on a variety of information sources may reveal risks that would otherwise be missed.

It is rare for organizations to explicitly exclude certain types of information, but they often have an unspoken bias towards a given type. Security is sometimes claimed to be ‘unquantifiable’, or qualitative information is discounted because it’s one person’s (subjective) opinion. Again, these biases can cause organizations to overlook valuable information when conducting cyber risk assessments.

You’re more likely to fall into this trap if your organization adopts a single, standardized approach for every kind of cyber risk assessment. This is more likely to occur when organizations focus on completing the process of risk management, rather than on the risk reduction activities which should flow from it. When organizations get into this ‘defensive’ pattern of risk management behavior, this closing down of what counts as ‘legitimate’ risk information can be exacerbated.

How can you know if you’re considering enough information sources?

This is more of an art than a science, and the technique we’re suggesting below uses a matrix that classifies information as qualitative or quantitative, and objective or subjective:

• qualitative information is about describing something in human language, such as written information presented in documents.

• quantitative information is about things that can be measured in numbers.

• objective information is verifiable and not subject to opinion (such as the number of laptops that your organization holds, or the amount of money it would cost you to purchase a particular antivirus solution).

• subjective information is a matter of opinion (such as the judgement that a particular organization is more at risk of DDoS attacks than of ransomware attacks).

By assigning each information type to the corresponding location on the grid, you’ll quickly be able to identify if there are any potential blind spots, as any empty quadrants will be immediately apparent. The grid below provides some examples of each type of risk information.

The purpose of this grid is not to categorize individual pieces of information. Neither are we suggesting that information from any of the four quadrants is ‘better’ than any other type. It is about looking at the spread of information sources that you use in your risk analysis, and spotting any blind spots.

---

So how might you go about doing this?

---

One. Start by going go through all the various information sources that feed into your organization’s risk assessment process.

Two. Place them into the grid above. If you’re not sure where to start, go back to a decision that related to cyber security in your organization. What information was used to inform that decision? If nothing was written down, go back and speak to the person who made the decision and ask them what they used to decide on that security issue.

Three. Examine the grid. What does it look like? Are you weighted towards one quadrant, or perhaps one half?

Four. What other information could you have gathered to fill the gaps? How might that have changed the decision? Why are the gaps where they are? What blind spots this might cause in your approach to risk analysis?

The goal of this exercise is to help you spot situations where your risk assessments might be missing some valuable information. It won’t tell you exactly what you’re missing, but it can shine a light on organizational biases towards a particular type of information.

This is by no means the only way of categorising risk information. There are other properties of risk information which may be just as useful. For example, it is also worth considering whether you are using a balance of information about the past, and information about how you anticipate the future will unfold, with some interpretation.

By using the qualitative/quantitative, objective/subjective technique, the BCSF have recognized a common bias in many organizations, where the grid is heavily populated in the top-left and bottom-right quadrants, and empty in the other two. In such organizations, when assessing cyber risk, the terms ‘objective’ and ‘quantitative’ were taken to mean the same thing.

Our findings demonstrate that in these organizations, information that was inconsistent with this flawed assumption was ignored. For example, experts’ subjective assessments of probability were discounted.