I am occasionally asked by our clients to measure how secure a thing is. That is perfectly reasonable to want to know. Is it secure enough? Do we need to spend more on security to make it secure enough? Are we getting better or worse? And so, managers are surprised, as well as disappointed, to learn that measuring security is nearly impossible.

Security is not visible. It is kind of like dark matter, in that you cannot see it directly, you can only detect it by its absence. The absence of security, i.e. vulnerability, can be detected and measured in a multitude of ways, such as vulnerability counting, defect density, or days of vulnerability exposure. You can measure how vulnerable a thing is, but that is a lower bound on how vulnerable it is, it could be much worse.

What do we know if we measure vulnerability? We know that the thing has at least “this” set of vulnerabilities. But we don’t know how many others there might be, and we especially don’t know how many there are in the places we haven’t looked, or worse, places we haven’t thought of. Thus, the question itself of “how secure is it?” is effectively a divide-by-zero error, the ratio of its attack surface divided by its vulnerabilities, and when we get to “secure” (no vulnerabilities) the result is defined as not-a-number.

A common way to try to measure product vulnerability is to count the number of vulnerability disclosures against a product, and that data is easily available. However, that data produces skewed results. The number of vulnerabilities disclosed in a year is as much a function of how much security attention the products received as it is the vulnerability of the product. Thus, counting vulnerabilities is considered a very bad security metric because it measures the environment more than it does the products.

You can measure how often intrusion attempts are caught by automation, but this is very similar to measuring vulnerability disclosure rates. What this is really measuring is how much pre-attack research the attacker invested in before attempting to breach. An attacker who is bulk rattling door knobs will very often trip automated alarms, and not care, because they just move on to find someone else. An attacker with a very focused interest in a specific target will learn what automation is in place before attempting anything, and thus is unlikely to trip automated alarms.

This is not to say that you can’t measure some aspects of security. For instance, you can measure how quickly a vendor responds to vulnerabilities found in their products, how quickly an IT department responds to security incidents, and the volume of attack incidents. But these are just instances of measuring vulnerability, none of them measure your security. As David LeBlanc says, there is no “secure”, there is only “insecure” and “good enough for now.”

So, if you ask how secure a thing is, and the answer is a vulnerability assessment, then that begs the question “did you check all the things?” Did whoever did the vulnerability assessment really check everything that could be a threat?

“Everything” is a long list. Borrowing a glib phrase from Andy Lowry, “Had the Sliver Surfer not befriended Galactus, Galactus would have consumed the Earth, causing most terrestrial computers to fail.” Thus, getting a usable “everything” requires us to construct a reasonable threat model that hypothesizes an attacker and their capabilities (what the attacker already has access to, such as your web site) and what the attacker’s goals are (e.g. steal your password list).

If a product or service is online, then the threat model must at least include “the Internet”, because all attackers have Internet access. If the service offers free accounts, then the threat model must assume that the attacker has numerous accounts on the system, giving them access to all the resources that normal users have, and the ability to collude multiple “users” to achieve effects that a single user can’t, e.g. artificially voting up the number of stars on a particular thing.

One can’t just “block the bad guys” because attackers rarely attack from their own computer, they attack some oblivious victim and use that other person’s computer to attack you. If one tries to investigate the attacker or “hack back” what we find is a wiped computer and a confused victim who doesn’t know what all the shouting is about.

If a software product is for sale or offered for download, then the threat model must assume that the attacker has obtained copies of the product, and tested their attacks against it, and possibly reverse-engineered the product with a de-compiler. That means that if they attempt to attack this product in the wild, they have already bypassed the product’s built-in defenses.

This list of real threats goes on for a while, the attackers have lots of opportunities to attack any product or service available to the general public, or even a moderately large set of selected people. To ensure that “everything” that a skilled attacker might exploit has been checked, a similarly skilled penetration tester must enumerate, inspect, and validate all possible attack surfaces. Which is why security consulting services such as the Leviathan Security Group exist, to provide that kind of penetration testing expertise on demand. Large companies often have internal penetration test teams that continuously attack their own systems, searching for vulnerabilities so that they can close them before attackers find them. Smaller firms lack the resources to justify maintaining their own penetration test teams, and so will contract with firms like Leviathan for outsourced penetration testing. Even large firms that do have internal penetration testing teams will often outsource external penetration testing to get a different perspective on “everything” to be checked. If you don’t know whether your product or service is secure enough, let us help.