[BreachExchange] A Scanning Solution is Only as Good as the Vulnerability Data That Drives it

[Destry Winant]

https://riskbasedsecurity.com/2019/06/26/a-scanning-solution-is-only-as-good-as-the-vulnerability-data-that-drives-it/

We had some great conversations at JFrog’s user conference, SwampUp
2019. Brian Martin, our Vice President of Vulnerability Intelligence,
took part in an all-star keynote of experts where he discussed how our
VulnDB® service helps secure JFrog Xray user pipelines.

The integration of VulnDB allows DevOps teams to discover, receive
notifications on, and help remediate vulnerabilities in third-party
libraries and dependencies early in the development cycle. As JFrog
puts it, “a security scanning solution is only as good as the database
of vulnerabilities that drives it.” Driven by Risk Based Security’s
comprehensive data, Xray with VulnDB is the best security intelligence
solution on the market for developers.

Why does data from VulnDB give you an edge?

VulnDB is the most comprehensive source of vulnerability data
available, with almost 69,000 vulnerabilities that are not found in
CVE or the National Vulnerability Database (NVD). As Brian shared in
his SwampUP keynote presentation, an average of about 70 new
vulnerabilities are disclosed every day. This is an alarming volume,
especially if your organization isn’t seeing the complete picture.
That’s why our rallying cry is #BetterDataMatters. VulnDB is so much
more advanced than any other database because we are looking for
vulnerabilities and we speak with the DevOps community to ensure we
are monitoring the libraries they are using. VulnDB includes more
vulnerabilities, and carries more metadata and research on entries.
This allows you to arm your organization with the most complete and
up-to-date information available so you can make data-driven decisions
to effectively manage and prioritize risk mitigation.

Taking this to mind, let’s look at some real-world applications.
Recently, Sophos put out a very thought provoking article. The article
made some very interesting points:

Most vulnerabilities aren’t exploited, and if they are, they tend to
have a high CVSS score.
There is apparently no relationship between the proof-of-concept (PoC)
exploit code being published online and the start of real-world
attacks.
In order to patch vulnerabilities, a “reference tagging” machine
learning model is the most efficient method.

Sophos based their conclusions on data provided in a whitepaper that
researchers from Cyntia, Virginia Tech, and the RAND Corporation
published. The findings were extremely engaging, however, the data
used to support these claims is lacking…comprehensiveness.

Looking further into the data provided, it’s apparent that the
researchers relied very heavily on security sensors based on CVE IDs,
meaning that only vulnerabilities within CVE were being considered.
This means that there are almost 69,000 vulns being missed in this
study. To make matters worse, security scanning devices tend to cover
half of the vulnerabilities in CVE, which makes the subset of data
even smaller.

In addition, Risk Based Security believes the machine learning models
used in the study may have mis-categorized focused attacks. In
situations where someone determines that a remote target is running
specific software, then tries a comprehensive list of attacks against
it, a detected attack would likely be labeled incorrectly. Since the
researchers were basing their findings off of CVE data, it is highly
likely that their sensors were not aware of specific vulnerabilities,
resulting in a label of “Generic XSS” for example. This could have
skewed results.

Last, these type of reports typically don’t share their full
methodology, let alone what they are capable of matching against. This
means that there is no way to reproduce or validate their findings.
Unfortunately, if the research is solely based on CVE data it means
that several vendors performing a similar study will also provide the
same rough figures. CVE is the industry “standard,” yet it is missing
a huge amount of vulnerabilities, with many of them possessing high
CVSS scores and affecting major vendors. As previously stated, “a
security scanning solution is only as good as the database of
vulnerabilities that drives it.”

#BetterDataMatters. We would be very interested if the findings
presented would be the same if more up-to-date data was used.