[BreachExchange] Will the CodeCov breach become the next big software supply chain hack?

[Destry Winant]

https://www.scmagazine.com/home/security-news/data-breach/will-the-codecov-breach-become-the-next-big-software-supply-chain-hack/

It’s always good to have your radar up on April Fool’s Day, constantly
on the lookout for potential pranks or tomfoolery. For one company,
what they discovered on April 1 was far from a joke.

Yesterday, software company Codecov, which sells a tool that lets
developers measure the testing coverage of their codebase, disclosed
that it suffered a breach. In particular, the attackers exploited a
bug in the company’s Docker image creation process to gain access to a
Bash Uploader script designed to map out development environments and
report back to the company. This small modification quietly called out
for user credentials that could have been used to access and
exfiltrate data from their users’ continuous integration environment.

In a note posted on the Codecov website, CEO Jerrod Engelberg said
that any credentials, authentication tokens or keys that were run
through an affected customer’s CI process were exposed, and with them
the attacker would have had access to any corresponding services,
datastores, application code and git repositories that could be
accessed by those credentials.

After discovering the breach on April 1, a follow up investigation
determined that the threat actor had been in their network since at
least January 31, going undetected for months. The vulnerability also
affected three other bash uploaders: Codecov CircleCI Orb,
Codecov-actions uploader for GitHub and the Codecov Bitrise Step.

“We strongly recommend affected users immediately re-roll all of their
credentials, tokens, or keys located in the environment variables in
their CI processes that used one of Codecov’s Bash Uploaders,”
Engelberg advised.

Codecov did not disclose how many of its clients were impacted, only
saying they had notified all affected parties in writing. The known
details of the intrusion, the nature of the company’s work and its
customer base has given rise to concerns that the breach could be just
the first shoe to drop in a larger software supply chain compromise
with potential for messy downstream effects. It lists a number of
high-profile customers on its website, including The Washington Post,
Atlassian, Mozilla, SweetGreen, GoDaddy and others.

Experts in software development and security reached by SC Media said
that the potential for downstream impact on Codecov’s users could be
high, but the scope of the damage will depend on a number of factors,
such as the identify and motivations of the actor, how Codecov
architects their network and what precautions, configurations and
access policies each individual user set up for their code
environment.

Knowing the identity of the group behind the attack would help shed
light on their possible goals, but several observers said the length
of time the attackers spent in Codecov’s network and the focus on
credentials indicate that they were more interested in getting access
to their customers’ code than the company itself.

Unlike SolarWinds and Microsoft, Codecov is not a publicly traded
company, has a few dozen employees on staff and measures its annual
revenue in the low millions of dollars per year. Despite the high
profile of some of their customers, they’ve only existed since 2014
and are not particularly well-known, indicating that the threat actor
may have done a fair bit of homework before selecting them as a
target.

“I would be leaning [towards espionage] just as a gut inclination.
Codecov is off the beaten path,” said John Bambenek, founder of
cybersecurity consulting firm Bambenek Labs. “Effectively the
compromise involved inserting one line of code and it’s giving
credentials. Now there are criminal networks that sell access to
organizations and credentials, so it’s not implausible that it’s a
fairly sophisticated financial actor that wants to sell them, but if I
had to bet, I’m putting my money on espionage.”

The type of credentials, and the access they provide, also matter.
Bambenek said if they only got their hands on testing credentials, the
impact would be far more limited than if the threat actor had access
to credentials for customers’ software production environment.

The extent of Codecov’s network segmentation could also determine in
part what customer information and data the group could have accessed.
John Zanni, CEO of Acronis, which focuses on data protection, cloud
and software security services, said his company has four separate
networks: one for work only devices, one for BYOD home devices,
another for guests and family members and one for their software
developers that not even the CEO can access.

They also don’t let their developers pull and use open-source code
straight from the internet. Before any software is updated, the
changes have to go through a code checking review and signing process
by another party, something that can guard against both unintentional
oversights and insider threats.

“It seems like every time I hire a new developer, that’s the first
thing they do with the code they right, so we have to put automated
checks in there so the moment somebody tries to do that, they get
caught and it stops,” said Zanni.

Robust code signing policies were cited as a best practice by others
as well. John Loucaides, vice president of research and development at
vulnerability research company Eclypsium said the breach represented a
“huge ROI for attackers to attack the supply chain” and that any
changes to software code have to be vetted by other parties before
approval.

Quinn Wilton, senior researcher at Synopsis Software Integrity, said
the breach demonstrates how “code signing is more important than ever,
and that transparency around the storage and disposal of those code
signing keys is going to be a vital step toward building trust in the
channels we all use to distribute software.”

While the attackers went undetected for months, Bambenek said that for
a small company with limited resources like Codecov finding,
investigating and disclosing a trivial change in their code within
three months is actually impressive. He compared it to the SolarWinds
breach, where the company itself and multiple customers and federal
agencies with larger budgets missed far more substantial code changes
in the Orion software build platform for at least a year, if not
longer.

“The foothold happened Jan. 31. For an early-stage company like that,
that’s solid work,” said Bambenek, who often advises smaller companies
on cybersecurity strategy and risk. “Yeah, we’d all like it to be
less, but startups are an easy target and so far, it looks like
they’re responding to it as well as they can. If they in fact have
[only a few dozen] employees, it would surprise me if they have more
than one security person.”