[BreachExchange] How Reliance on Inadequate Data Can Lead to Faulty Conclusions

[Destry Winant]

https://www.riskbasedsecurity.com/2021/03/02/reliance-on-inadequate-data-leads-to-faulty-conclusions/

Recently, a company called Redscan released a report that analyzed
vulnerabilities in 2020 via the National Vulnerability Database (NVD)
data, which is based entirely on MITRE’s CVE. Redscan is a company
that performs penetration tests and offers a managed security
platform. Based on their web page, they do not run a vulnerability
database or aggregate disclosures themselves. Despite that, they made
quite a few bold conclusions including a “record breaking number of
vulnerabilities reported in 2020”. These are the softballs of
vulnerability statistics.

Looking at the actual report in more detail, we wanted to provide some
input and offer caveats and qualifications to their conclusions and
statistics, as well as contrast their findings to our own, based on a
much larger and more comprehensive dataset. For example, in their key
findings they say that, according to NVD, there was an average rate of
50 CVEs per day. Compare that to VulnDB, where we saw an average of 69
per day, and one can immediately see a coverage gap between the two.

One of the statistics that they presented stood out to several of us
at Risk Based Security, where they say that there were “more high and
critical severity vulnerabilities in 2020 than the total number of all
vulnerabilities recorded in 2010 (4,639 including low, medium, high,
and critical)”. Looking at VulnDB, we see there were at least 9,360
vulnerabilities disclosed that year, almost double what CVE
aggregated, rendering their statistic inaccurate. It also reminds us
that if one wants to create a shocking statement, one can choose an
arbitrary year that best suits your needs. That same comparison to
2015 probably doesn’t work, but if they compared it to 2000, they
could have had more “fun” with the even bigger gap.

This comparison between 2020 and 2010 also highlights another problem.
Analyzing data may seem like a straightforward endeavor, but you
simply can’t do it properly unless you understand how the data was
collected and the caveats that come with it. In 2010, CVE had
different leadership, higher standards, and different abstraction
rules. CVE in 2020 is a very different beast; trying to make a direct
comparison of data from those two years is more akin to comparing an
alpaca to a llama. They are both camelids but the similarities stop
shortly after that.

In a similar vein, on page six of the report they include a section
and chart about “Attack Complexity” or “AC” which goes back to 1988.
Unfortunately, this chart has no actual value due to a significant
change in CVSS. Under CVSSv2, the “AC” metric was formerly “Access
Complexity” which “measures the complexity of the attack required to
exploit the vulnerability once an attacker has gained access to the
target system.” When CVSSv3 was introduced, AC changed to “Attack
Complexity” which “describes the conditions beyond the attacker’s
control that must exist in order to exploit the vulnerability.” This
is a significant difference and means that at some point on this
chart, the definition of “AC” changed and a straight comparison cannot
be made.

When did that change take place? NVD started scoring CVSSv3 for the
first time in 2016 and did not choose to “backfill”, or retroactively
score, older vulnerabilities. That means the chart Redscan provided
can only be used to compare 1988 – 2015. Everything from 2016 – 2020
would have to be looked at separately as the data is completely
different.

RELATED: The Value of Backfilling

With any significant dataset, it is possible to pull out and focus on
smaller points of interest. For example, we have written about
Electronic Voting Machines which represent a tiny fraction of all
vulnerabilities disclosed. However, the context around that was the
severity of those vulnerabilities in the face of upcoming elections,
and the potential for just one vulnerability to have severe
consequences.

The flip side to that is focusing on a tiny subset of vulnerabilities
that has no apparent narrative other than “the numbers changed”.
Redscan’s report does this by focusing on physical vulnerabilities,
saying that they saw a “large spike” in 2020. In reality, there was
steady growth from 2018 – 2020. The significant spike occurred from
2017 to 2018 if anything, but we’re still talking about a gap of fewer
than 300 physical vulnerabilities in each of those years. Meanwhile,
the report also talks about ‘Adjacent’ vulnerabilities, which is
considerably more interesting as far as the numbers go, but they chose
not to focus on them.

Another issue that can manifest in conducting analysis is focusing on
the narrative that accompanies each point of examination, without
considering how these comments compare to each other. For example,
Redscan says that the “prevalence of low complexity vulnerabilities in
recent years means that sophisticated adversaries do not need to
‘burn’ their high complexity zero days on their targets and have the
luxury of saving them for future attacks instead.” This is their
conclusion after looking at the complexity of vulnerabilities based on
NVD’s CVSS scoring. Anyone well-versed in vulnerability intelligence
knows this is a really bad metric to use for any real analysis of
severity.

Later in the report they examine the privileges required to exploit
vulnerabilities in the same year, and come to this conclusion:

“It is also encouraging that the proportion of vulnerabilities
requiring high-level privileges has been on the increase since 2016.
This trend means that cybercriminals need to work harder to conduct
their attacks.”

On one hand, there is a prevalence of low complexity vulnerabilities,
and attackers don’t need to work hard. On the other hand, the
privileges required to exploit vulnerabilities is on the rise, so
attackers have to work harder to exploit them. While both may be
technically true, and some attackers may align with one statement or
the other, using such blanket statements doesn’t mix well.

We don’t think that Redscan is spreading misinformation or
purposefully manipulating the narrative to fit their needs. This
dissection just illustrates how nuanced and specific the vulnerability
disclosure landscape can be. When it comes to vulnerabilities, there
is so much more than what is aggregated in CVE/NVD. In fact, CVE is
missing over 80,000 vulnerabilities that you may not know about.

Caveats are important so we always make sure that our clients
understand how they can affect their Vulnerability Management
programs. For the latest details involving vulnerability trends (with
all the disclaimers), check out our 2020 Year End Vulnerability
QuickView Report. For organizations wanting a full picture of their
risk profile, we invite you to see for yourself the importance of
comprehensive, detailed and timely vulnerability intelligence.