Kansas 911 Administrator Blames Three-Hour Outage On ‘Technological Anomaly’

By Sherman Smith, the Kansas Reflector

January 28, 2021

TOPEKA — Rep. John Carmichael questioned the reliability of the state’s 911 call handling system during a hearing Thursday on why the system went down for 3 hours and 20 minutes on Nov. 29.

The Wichita Democrat said the outage demonstrates the failure of a backup system that was flagged as a risk by state auditors in 2018.

“The bottom line is my constituent standing on the street with her husband dying as a result of a traffic accident — she couldn’t get through to anybody,” Carmichael said. “So the system we have right now doesn’t work, does it?”

Scott Ekberg, an administrator for the 911 system, briefed lawmakers in a House panel about the Nov. 29 outage, which lasted from 1:30 to 4 p.m. He blamed the problem on equipment and software provided by AT&T but for security reasons declined to provide a copy of the analysis report conducted by AT&T of the root cause.

The outage shut down 55 of the state’s 92 call answering systems. Staff at those sites in the southern parts of the state could see calls coming in but couldn’t answer them.

“It’s all technology based, and technology fails,” Ekberg said. “It’s why we strive as hard as we can to make sure that we’ve tried to account for any failures. There’s no one in the nation anywhere that can guarantee you, regardless of the system they are utilizing, there is no one that can guarantee you that you’re not going to have service interruptions.”

Ekberg said the problem was caused by an analog-to-digital conversion device that was installed on Sept. 30. The device was incorrectly set to transmit data to the wrong server. The errant transmissions eventually caused the system’s firewall to go down.

The firewall problem initially went undetected because it appeared to be functioning normally. Servers then began kicking the failed conversion back and forth in a loop that prevented the servers from handling any other activity.

In an email after the hearing, Ekberg said he doesn’t know why it took two months for the firewall to fail.

“The firewall soft failed, so it appeared to the network that it was working correctly,” he said. “Had it failed hard, then the traffic would have automatically rerouted to the redundant firewall and this event would not have occurred.”

Technicians resolved the Nov. 29 outage by finding the device with the errant setting. The next day, staff reviewed the settings of all devices to ensure they were correct. A software upgrade planned for later this year will prevent servers from getting caught in a loop.

The Kansas City Star reported a 2018 audit warned of potential problems with the 911 system. Outages in 2017 and 2018 had disrupted services for dozens of counties, and 62% of emergency dispatch centers reported they had experienced down time.

Ekberg told lawmakers the system “works quite well, other than when we have a technological anomaly.” He said the death referenced by Carmichael is the only one he was aware of during the Nov. 29 outage.

Rep. Joe Seiwert, a Republican from Pretty Prairie and chairman of the House Energy, Utilities and Telecommunications Committee, said he appreciated the safety concerns raised by Carmichael.

“One death is truly a tragedy,” Seiwert said. “I wish we took that same feeling when we talk about the 50-60 people who die from deer accidents in the state. We don’t seem to take that seriously, and they’re just as important. And that’s every year, and not just a one-time incident.”

The Kansas Reflector seeks to increase people's awareness of how decisions made by elected representatives and other public servants affect our day-to-day lives. We hope to empower and inspire greater participation in democracy throughout Kansas.