[BreachExchange] Click2Gov or Click2Breach?

[Inga Goddijn]

https://www.riskbasedsecurity.com/2018/06/click2gov-or-click2breach/

Here on the Cyber Risk Analytics <https://www.cyberriskanalytics.com/> research
team, we have more than our fair share of “glitch in the matrix moments” –
you know, that proverbial black cat walking across your screen that makes
you think: “Didn’t I just see this breach?” Usually it’s a case of similar
circumstances or simply two names that are a lot alike. Other times, it
might be something more.

While not yet verified, that “something more” appears to be the case with a
utility bill payment processing application known as Click2Gov
<https://www.superion.com/public-administration/click2gov/> from the company
 Superion <https://www.superion.com/>.

We have been tracking a handful of breaches taking place across the country
that on the surface look to be unique events with somewhat similar
descriptions. A local city or town discovers their online utility payment
portal has been attacked. The service goes dark while the city investigates
– along with their trusty vendor that may or may not run the portal – only
to learn that payment card details used to pay utility bills online have
been compromised. The city takes responsibility for the event and starts
posting notices to impacted persons. All and all, there was nothing
especially remarkable about the individual reports – until, that is, the
the name Click2Gov started popping up.
What We Know So Far

On May 25, 2018, the City of Oxnard, CA
<http://www.keyt.com/news/money-and-business/city-of-oxnard-data-breach-affects-online-utility-payers/748204303>
 was notified by a bank that their online utility bill payment service
appeared to have been breached, leading to a number of fraudulent
transactions. Transactions taking place between March 26, 2018 and May 29
(yes, 4 days after the city first learned of the issue – more on that
later) were exposed. The city identified  Click2Gov as their payment
processing application
<https://www.facebook.com/CityofOxnard/posts/10155301666717484>.

On June 6, 2018, the Village of Wellington, FL
<http://www.wellingtonfl.gov/how-do-i/apply-pay-register/potential-data-breach-information>
 was notified by Superion that certain vulnerabilities in Click2Gov might
have lead to a possible breach of their online utility payment
installation. Once again, Wellington officials in conjunction with Superion
shut down the system to investigate. While a breach has yet to be
confirmed, there was sufficient information for the Village to state that
payment card data used for online bill payments between July 2017 and
February 2018 is considered to be ‘at risk’.

Two events in a row referencing the same application got our attention and
sparked our curiosity. Especially so since the City of Oxnard event began
one short month after the Village of Wellington event seemingly ended. Our
immediate thoughts went to questions like: “Are there more breaches
involving Click2Gov? Could it be the same attackers jumping from one
vulnerable installation to the next? Is it possible that the source of the
issue is attackers inside Superion, picking off data from various clients?”
Definitive answers are not yet apparent, but it is clear that the issue is
larger than just two breaches.

Looking back in our database, the City of Ormond Beach, FL experienced a
similar incident
<http://www.news-journalonline.com/news/20171012/ormond-breach-utility-customers-see-fraudulent-charges>with
their Click2Gov system in October 2017. Like Oxnard, it was a credit card
issuer that first traced the issue back to Ormond Beach utility payment
system, alerting them of the problem on October 11. This, despite the fact
that customers had been reporting fraudulent charges they believed to be
linked to the City since September 22nd. Ultimately, cards used for payment
between approximately mid-September 2017 and October 4, 2017, when the city
opted to shut down their system, may have been compromised.

Shortly after, the City of Port Orange, FL launched their own investigation
<https://www.port-orange.org/DocumentCenter/View/1670/City-of-Port-Orange-Takes-Necessary-Precautions-Against-Any-Data-Breach>
 into their Click2Gov system. Their system was down for 5 days but
ultimately, they could find no evidence
<https://www.port-orange.org/DocumentCenter/View/1696/City-of-Port-Orange-Resumes-Online-Payment-System-for-Its-Customers>
 of a breach. Curiously, their statement included a quote that their
Click2Gov system had no “*potential flaws that could leave the system
exposed to a data breach*.” One can only wonder if they are equally
confident of no flaws now that Superion has notified at least one customer,
the Village of Wellington, of “*certain vulnerabilities*” in the Click2Gov
system.

Our research identified more breaches at several other cities that fit the
profile of a Click2Gov issue. The vendor wasn’t named in official
statements, but in several instances is clear Click2Gov is source:

   - City of Goodyear, AZ – May 7, 2018 the City became aware of an issue
   <http://www.goodyearaz.gov/government/departments-divisions-a-z/finance/utilities-customer-service>
    with their unnamed online payment system. They worked with the vendor
   and determined transactions between June 13, 2017 and May 5, 2018 had been
   exposed. Although the city does not come out and name Click2Gov as the
   vendor, it’s clear from the payment landing page URL that Click2Gov is the
   service provider: https://click2gov.goodyearaz.gov/Click2GovCX/index.html
   - City of Thousand Oaks, CA – February 28, 2018, the city learned of
   unauthorized access  <https://oag.ca.gov/system/files/CA_U811_v02_0.pdf>to
   their online payment system “Click to Gov”, exposing payment card details
   for transactions between November 21, 2017 and February 26, 2018.
   - City of Fond du Lac, WI – Once again, on December 12, 2017, the city
   got word from a bank that a breach had been traced back to their water
   payment portal
   <https://www.fdl.wi.gov/cofuploads/Water_Security_Statement_121917_12192017132146.pdf>.
   Payments made between August 2017 and October 2017 were exposed. Yet again,
   Click2Gov was not named but is clearly they are the provider of payment
   services: https://click2gov.fdl.wi.gov/Click2GovCX/index.html
   - City of Beaumont, TX – On August 24, 2017, the city announced they had
   received complaints of unauthorized charges after using the online water
   bill payment system
   <https://beaumonttexas.gov/potential-data-breach-city-online-water-billing-payment-system/>.
   Payments made between August 1st and August 24, 2017 may have been
   “jeopardized”. Beaumont did not indicate a vendor was involved, but it’s
   clear who their service provider is as well:
   https://beau-egov.aspgov.com/Click2GovCX/index.html
   - City of Oceanside, CA – In near lock step with Beaumont, on August 14,
   2017 the city received complaints from customers that credit cards used
   between June 1, 2017 and August 15, 2017 on the now-defunct “utility bill
   payment” link had been compromised. The link is no longer available so it
   is unknown whether it was Click2Gov, but the city’s notification letter
   <https://oag.ca.gov/system/files/Notice%20of%20Data%20Breach%20%28Final%209-6-17%29_0.pdf>
    does state their forensic examiner found “malicious code had
   infiltrated this vendor supported online payment system.” Perhaps most
   telling, the letter goes on to state, “the City is exploring alternative
   online payment solutions that offer improved security processes and
   systems.” Clearly a wise decision on their part.

As you can imagine, we suspect there are others.
About Click2Gov

Unfortunately, we aren’t intimately familiar with how Click2Gov software
works exactly. From how the cities are reporting the events, it appears to
be a software package that is downloaded and run independently for each
city. After all, the cities seem to be taking responsibility for the
breach, hiring the forensic teams to investigate and making statements to
the effect of updating their software and making changes to servers in
response. But further digging seems to reveal that while it is a software
package, there may be some vendors that are hosting it on behalf of their
clients and the Click2Gov solution may also provide credit card processing
capabilities.

What makes this interesting is that, for each incident that has been
reported, the breach is presented as some sort of misconfiguration issue or
a problem at the city itself, but it seems that it might be something
larger.

Despite indications there were issues with the service dating back to
August of 2017, it wasn’t until May 30th of this year in the City of
Oxnards’ breach notification that we start to see clear evidence the
problem lies with Click2Gov – and it’s not encouraging. Oxnard officials posted
the following on their Facebook page
<https://www.facebook.com/CityofOxnard/posts/10155301666717484>:

“Upon discovery, the city immediately reported the issue to the Police
Department and the city’s vendor, which engaged a third-party forensic firm
to determine what happened and what information may have been affected. *The
city’s vendor alerted the city to a software vulnerability that had the
potential to allow an unauthorized individual to gain access to the
computer used to process credit card transactions.”*

Keep in mind the City of Oxnard first learned of a possible breach on May
25, 2018. They reached out to Superion, seeking help with the issue.
Additionally, Superion most likely knew of potential security problems
since the City of Oceanside stopped using their service back in the summer
of 2017, and certainly since Beaumont, Texas was breached at approximately
the same time. Both facts make this next paragraph from Oxnards’ breach
notice all the more concerning:

“Security patches were applied by the city’s vendor on a new server to
eliminate the vulnerability with the thought that the issue was resolved.
On May 29, 2018, the city’s *vendor informed the city of additional
security controls that were required* to secure the system. The city shut
down the system immediately so these security controls could be
implemented. Even though *the vendor’s investigation could not specifically
confirm or verify the exact method by which any credit card data could have
been compromised, *the city decided to notify customers as a precaution.”

Multiple clients are breached over the course of a year and still it takes
two tries to get a fix in place? And is the problem really corrected if
they cannot confirm or verify the exact method of compromise? Looking back
to the City of Fond du Lac’s breach notification
<https://www.fdl.wi.gov/cofuploads/Water_Security_Statement_121917_12192017132146.pdf>,
it seems this is not the first time they stumbled over incident response.

“The compromised credit cards each used the City’s online Water Payment
Portal at some point approximately between August and October 2017 to pay a
City of Fond du Lac water bill.”

“In October 2017, the City’s vendor third party payment engine identified a
known vulnerability with the Water Payment Portal. This vulnerability was
communicated to the City and patched by the vendor on the same day. The
City received no information or alert from the vendor third party payment
engine or any other vendor of suspicious activity or a possible security
breach until December 12, 2017.”

Unfortunately for the Village of Wellington
<http://www.wellingtonfl.gov/how-do-i/apply-pay-register/potential-data-breach-information>,
it seems they too are now caught up in Superion’s questionable patching and
incident response practices. But at least this time, it was Superion that
reached out to Wellington instead of waiting for a call from a bank fraud
department:

“On June 6, 2018, the Village received a call from our vendor, Superion,
notifying us of vulnerabilities in their software. The software problem was
with the Click2Gov online payments for utility bills. Credit card
information may have been taken during transactions.”

“The Village immediately shut down our payment connection to Superion and
began working with them to determine if our resident’s information was
compromised. The forensic analysis is continuing, security patches are
being installed and new hardware and software are being installed to
eliminate the breach. Even though Superion could not specifically confirm
that our customer credit card data has been compromised, the Village
decided to notify our customers as a precaution.”

Superion Security

There isn’t a lot publicly known about potential security issues with the
Click2Gov solution. In taking a more detailed look at Superion’s website
<https://www.superion.com/> for any updates, there were none to be found
for the Click2Gov software product. In fact, when looking on their website
we were unable to find any links to security notices and when trying to
find a dedicated security page (e.g. https://www.superion.com/security) we
found nothing existed.

We then decided to reach out to Superion directly and email them at
security at superion.com as well as call their general enquiry and sales
numbers. Unfortunately, both phone numbers gave the same automated message
and then offered us to leave a voicemail.

As for a security@ mailbox, sadly but not unexpected, it bounced.

We then forward the message to their Media Inquiries address (
media at superion.com) to hopefully get some more information on the
situation. If we receive a reply we will update this post.
What Comes Next?

The issue might affect quite a few more cities than initially expected.  As
we were conducting our investigation we attempted to determine how wide is
the installation base of Click2Gov. Our results varied widely but what we
found was that there* appears to be between 600 to 6,000 installations of
Click2Gov indexed *(and potential thousands more depending how you look at
it).  Without spending much time digging, we quickly saw what appeared to
be quite old versions of Click2Gov running.

Unfortunately, given what we have seen so far we anticipate seeing more
breach reports coming to light thanks to the Click2Gov system. Superion and
their clients are clearly struggling to wrap their hands around the problem
and lock it down once and for all. In the meantime, any organization that
is currently a Superion customer using Click2Gov should be on alert for
suspicious activity. They should also consider reaching out to Superion for
more information on the vulnerabilities that have been identified in
Click2Gov, so that they can investigate whether they are exposed to the
issue and implement patches or workarounds to mitigate the issue.

We suspect there will be more to this story and will update this post as we
learn more.  If you have any information please contact us!
<info at riskbasedsecurity.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.riskbasedsecurity.com/pipermail/breachexchange/attachments/20180614/37afe4bb/attachment.html>