(Editor’s note: TVTech reached out to several station groups to determine the impact of the CrowdStrike outage on operations. Many, including NextStar Media Group, News-Press & Gazette, Allen Media Group and PBS, reported the glitch had no effect; a few declined to comment. Operations and risk management executives from E.W. Scripps however, did respond.)
The CrowdStrike cybersecurity glitch, which one security expert has dubbed “the largest IT outage in history,” brought Microsoft Windows systems at airlines, financial institutions, hospitals and some in media to a grinding halt, at least for a time July 19, with the Blue Screen of Death (BSoD).
E.W. Scripps, one media company to suffer outages, limited their detrimental effects with a strategy put in place over the past 10 years to muster technical resources quickly to identify and correct issues when they arise, says Mike Epstein, vice president of risk management at Scripps.
“Our incident command, control and communication processes worked well,” he says. “Scripps has spent a decade improving its incident management program, bringing engineering, IT, business operations and other groups together to guide assessment and response.”
At Scripps, disruptions began early July 19 when CrowdStrike updated its Falcon software. Scripps News was down from 1:11 a.m. to 1:21 a.m. EDT. By 6 a.m. EDT, it was able to broadcast live shots from London, Cleveland, Detroit and Atlanta.
Some of the group’s local stations also suffered disruptions, but none were off air completely, says Scripps spokesman Michael Perry.
“In the early morning hours, some could not get local news up, but by 8 a.m., nearly 90% were able to air local news, and before noon it was 100%,” he says.
One was KSHB-TV in Kansas City, Mo. The Associated Press reported it aired Scripps News rather than its local news until 5:35 a.m.
The group’s newsroom systems took the biggest hit although some software-based production equipment was also affected, says Mark Gray, senior vice president, network and stations operations, adding that most of the problems were confined to on-premise computers.
‘Unknown Tech Issues’
Scripps jumped on the problem as word came in about the BSoDs. “Our Help Desk and station monitoring teams became aware of what were originally ‘unknown tech issues’ and activated our enterprise incident management program. That team assembled and began solutioning as we identified and coordinated workarounds for issues as they continued to develop across the enterprise,” says Epstein.
Some of the news workflow disruptions were ameliorated by “not an insignificant number” of Apple Mac-based systems that remained unaffected by the software glitch. While individual workarounds varied by location and application, the Mac systems were in place as backups “and were used as such,” says Gray.
One of the more challenging tasks was to provide one-on-one tech support to Scripps employees working remotely, he adds.
While the disruptions had a limited impact on what Scripps aired, the problem with the CrowdStrike software update shines a light on a greater vulnerability. “This is a challenge as the issue was caused by a trusted partner/service that is used to protect not only our systems, but obviously those of innumerable companies around the globe,” says Epstein.
“We were already working on cross-support capabilities across our footprint and will continue that program to help mitigate future risk…. While we can’t predict the next incident, we can continue to be sure we have the right tools to flex, adapt and respond.”
