Cybersecurity vendor CrowdStrike caused a global IT outage a week ago that had nothing to do with hackers or malware, and therefore nothing to do with security. And it wasn't some “cyber” thing, it was a real-world disruption that paralyzed hospitals, banks, airlines, and even broadcasters. It will take months, maybe years, to tally up all the pain it caused, with business closures, job losses, and worse.
Unfortunately, CFOs, whether they are organizations affected by an outage or have been bystanders to a major IT outage, can only afford so much time to reduce the likelihood of something like this happening again.
“The CrowdStrike outage caused business interruptions, so the results will go straight to the CFO's desk,” said John Winsett, CEO of IT procurement solutions provider NPI, which has seen impacts for large enterprise clients.
For companies whose Windows machines blue screen, it could mean lost revenue and brand integrity. Winsett says the incident is a stark reminder that even well-resourced and trusted software vendors can inject vulnerabilities into a company's tech stack.
The lost “bucket” of trust
But this operational risk will be “difficult to mitigate going forward,” Winsett said: “Seemingly harmless automatic updates occur on devices every day across the entire stack, from the operating system to the keyboard. Stopping these updates has equally dangerous consequences.”
CrowdStrike has promised to improve testing of software updates and to phase future update submissions (known as “canary deployments”), but customers would be unwise to believe it: “The trust we built bit by bit over years was quickly lost in a matter of hours,” Shawn Henry, CrowdStrike's chief security officer, wrote on LinkedIn.
In general, companies should reassess potential points of failure in their IT systems and consider the need for greater redundancy and quality control across their networks. Specific steps companies can take to prepare for an event like CrowdStrike include strengthening response and recovery plans, analyzing IT vendor concentration risks, and reviewing contracts with software vendors.
Be careful with software updatesTo be cautious, NPI's clients are focusing on a phased software migration, which involves “determining which technologies are subject to a 'walk and run' approach, where updates are not automated but are rolled out to select groups for preventative testing before full distribution,” Winsett says. “This is a bit tongue-in-cheek, but your service level agreements (SLAs) should state that you shouldn't push untested kernel-level software updates globally all at once, especially on a Friday,” says Craig Callé, a data security, GRC and vendor risk consultant.
Updates to the recovery planResponse and recovery procedures for non-breach scenarios should be reviewed and practiced, Winsett says. System recovery plans after IT outages may include “back-out” procedures specifically for software updates that don't go as planned, according to a July 19 recommendation from Forrester Research. The procedures return systems to a known good state. “CFOs should demand special focus on revenue-centric systems,” Winsett says.
Chasing vendorsSoftware contracts can be used as a risk mitigation tool. CrowdStrike offers insurance if customers suffer a security breach. Forrester says CFOs should consider “asking for business interruption indemnification clauses from vendors in case a software update goes awry.” “Maybe this tells us we need to put more emphasis on damages in software contracts,” Callé says, compensating customers for lost business. “Money matters, and if you hold vendors liable for actual damages, they'll be more cautious,” he says.
Revisiting third-party risksTechnology teams should map the company's third-party ecosystem to identify significant concentration risks among vendors, especially those that support critical systems and processes, Forrester said. Additionally, “incident response and business continuity are key parts of third-party risk management. [management] “Your TPRM program should be about more than just sending and receiving surveys,” says Callé.
Overspend Alerts
However, proceed with caution on the above items, as they could lead to significant unplanned IT spending, says Winsett. “Some vendors will smell blood after this debacle, and CFOs would be wise to consider the implications and put in place a strategy to prevent overspending.”
A version of this story appeared in StrategicCFO360's Finance and Accounting Technology Briefing.
