[BreachExchange] Why metrics are crucial to proving cybersecurity programs’ value

[Matthew Wheeler]

https://www.csoonline.com/article/3655096/why-metrics-are-crucial-to-proving-cybersecurity-programs-value.html

Methodologies to measure the effectiveness of cybersecurity efforts exist.
Tying them to the real world is the trick.

By Cynthia Brumfield

CSO | MAR 30, 2022 2:00 AM PDT

As solutions to managing cybersecurity threats increase, surprisingly few
metrics are available on how well these methods work to secure
organizational assets. The National Institute of Standards and Technology
(NIST) has pioneered information security performance measurement models
that can produce metrics. (Note: NIST’s work in this area is now being
updated.)

Aside from government agencies’ requirements to produce information
security performance measures, the measurement models NIST recommends can
also be used for internal overall IT improvement efforts. Either way, NIST
recommends considering four factors while developing and implementing an
information security measurement program:

Quantifiable measures

Readily available data that support the measures

Repeatable information security processes

Utility for tracking performance and directing resources

As is true of many NIST cybersecurity efforts, its information security
performance measurements lack real-world implementation guidance that could
assist technologists in measuring security performance, leaving the
industry struggling for pragmatic advice. Speaking at Shmoocon last week,
Robert Weiss, head of information security at OpenVPN, tried to fill the
void by providing security professionals with practical ideas on starting
their information security metrics programs.

The most critical measurement is risk

“No metrics presentation has ever been funny, and this one is no
exception,” Weiss said. All jokes aside, he stressed that metrics are
crucial to effective cybersecurity programs despite how rarely
organizations do a good job or make any effort to rely on them. “If our job
as information security professionals is to reduce information security
risk, at the end of the day if we can't demonstrate that we're
accomplishing this objective, resources will and should go elsewhere.”

The most important thing to measure is risk. “Our programs are designed to
reduce risk,” Weiss said. “The relationship of the program’s cost to the
amount of risk reduction is the business value being created.” But
measuring risk reduction isn’t the only goal of a security metrics program.
“We may often do other things like program performance or create
situational awareness,” he added.

“In a perfect world, you would have systems and processes for tracking
performance, situational awareness, and risk. You track metrics that
matter. You do not rely on surveys. You pull empirical data from your
systems and reason about your uncertainty and margin error.”

Ideally, “you can express risk in the probability that the annual loss
expectancy for a series of risks falls within a particular range. You
immerse yourself in the language of probability,” Weiss added. “Very few
organizations can do this. This actually represents a huge opportunity for
both practitioners like yourselves and your businesses.”

[ Learn how IT can harness the power and promise of 5G in this FREE CIO
Roadmap Report. Download now! ]

Two basic security metrics methodologies

Weiss emphasized two primary methodologies to help security professionals
establish metrics programs. The first is “just measure everything.”
Collecting everything “sends the message that you plan to build the culture
of measurement and make decisions on facts and analysis.”

There is a point of diminishing returns in this methodology. “If you have
no data, any new data will greatly expand your knowledge and reduce
uncertainty,” Weiss said. However, “there's an interesting corollary. If
you have a lot of data adding more isn't going to be very valuable.” You
want to spend just enough to collect data that will help make decisions,
but not more, he added.

If the data doesn’t exist, you can estimate it using secondary sources.
“Most of the time, you don't need a lot of data to make management
decisions. You can test a sample of servers. You can use secondary sources
like the Verizon breach report or others to get information about types,
incidents, and losses,” Weiss said.

The second methodology calls for collecting data and then applying
analytical techniques that help describe the information’s nature. Weiss
relied on the classification systems of psychologist Stanley Smith Stevens
who created the classic measurement scales of nominal, ordinal, interval,
and ratio in spelling out the merits of this approach.

“It is very common in information security to see a system or probability
impact plotted in some form of matrix because probability times impact
equals risk,” Weiss said. But, the dangers of the analytical approach come
into play, for example, “when you try to compare two ordinal scales [e.g.,
small, bigger, biggest] to each other. It is impossible to relate one
arbitrary step of probability to one arbitrary step of impact. Those things
cannot and should not be related without additional information. It’s like
multiplying by color.”

Don’t count only adversary incidents

Metrics programs should follow strategic goals and avoid certain traps that
undermine organizational security, Lesley Carhart, director of incident
response for North America at Dragos, tells CSO.  One of those traps is
when security metrics are based on adversary activity.

“You can’t predict reliably in the cybersecurity space when somebody is
going to attack or how often they’re going to attack,” Carhart says. “And
if they base their success on the number of incident responses they do or
the number of tickets that they handle based on adversary activity, what
happens if an adversary doesn’t attack that month? Or if they attack more
in one month than another?”

“It's non-sensical to base your measures of success on when a criminal does
something that’s completely unpredictable,” Carhart says. “You have to
really understand what you’re measuring. You don’t just do KPIs [key
performance indicators] for KPIs’ sake. It’s incredibly problematic. That’s
why we get unhealthy things like these phishing test scenario programs.”
Instead of, for example, clickbait rates, a better “measure is how often
people want to report things. Because just one campaign report could tip
you off and let you do your cybersecurity much faster.”

The phishing example highlights why “you don’t want to base any of your
metrics on whether a bad person attacks or not,” Carhart says. “Make sure
none of your measures are based on that. And think critically about what
you are actually trying to accomplish, your organization’s goals, and base
your metrics around that.”

Weiss agrees but tells CSO he wants all the numbers to start making
decisions as a CISO about which ones are the most important. The important
thing is to “make a commitment to data analytics,” Weiss stresses. “And you
don’t have to do everything perfectly.”
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.riskbasedsecurity.com/pipermail/breachexchange/attachments/20220330/6a00ac7e/attachment.html>