Two months after an IT failure brought Delta’s operations to a halt, CEO Ed Bastian explained what went wrong and how the airline plans to avoid another incident. But the August breakdown also reveals how a complicated hodgepodge of legacy systems running behinds the scenes poses a more enduring obstacle for the industry.
On August 8th, a critical piece of equipment failed at Delta’s main data center in Atlanta. When the part, which was essential to powering the whole facility broke, it shot out sparks so strong that they went backwards and killed the transformer for the whole power grid. The entire building went dark. Power was restored after a couple of hours, but as backups went online five percent of the servers were misconfigured and didn’t work. When they rebooted the 750 applications that Delta uses, some were overwhelmed by the volume of changes. 2,300 flights were canceled over the three days of recovery and the company lost in excess of $100 million in revenue.
“It’s not gonna happen again,” assured Bastian.
This year Delta will invest $400 million in technology, under recently hired CIO Rahul Samant. Bastian said they’ve now checked all their backups and are building an entirely new data center nearby.
It wouldn’t be such a blow if Delta didn’t have a reputation to maintain as the world’s most reliable giant airline. “Delta had more political capital built up,” said Seth Caplan, a journalist at Airline Weekly. “So it also has more to lose.”
Despite the debacle, observers tend to think Delta handled the aftermath exceptionally well. “We track our customer satisfaction and we’ve been doing really well. We’re at our highest levels ever in our history,” Bastian said.
So Delta may be in the clear, but the source of its troubles continue to plague the industry. American and United have also experienced recent outages. And that’s because airline IT is massively complex, especially for the largest companies. Remember the 750 applications Delta is running? Exactly.
“The system is so large you have many sub-modules, connected modules, and data feeds that come in and out of those systems,” said Ahmed Abdelghany, a professor at Embry-Riddle Aeronautical University. “Sometimes older and newer technologies are connected to each other by a third type of technology.”
He explained that when you have so many different systems all trying to talk to each other, problems will inevitably arise. “All of the mergers and backwards compatibility is adding to the complexity,” he said. Over the past few decades, companies like Delta have continued to use aging systems while grafting newer ones onto their convoluted networks. That they run 24/7 means there’s never a moment to stop and refresh.
On top of that, no one really wants to invest in legacy infrastructure since they always think they’re gonna get rid of it in a couple years explained Zubin Irani, founder and CEO of technology company cPrime. “It’s like trying to change the wheels on a car while moving a hundred miles an hour,” he said.
But Irani sees eventual changes coming, and Delta, having been burned might soon be embracing them. “I had conversations with people at Delta about products they’re interested in,” he said. “Cloud is something they actively look at when investing in new technology.”
Big companies are slow to adopt changes. It’s an incremental process that often happens more quickly with a new CIO and a mission, as at Delta. Bastian is quick to acknowledge that 750 is a lot of applications and hinted at a new, simpler and more efficient technology capabilities.
“We’re going to streamline,” he said.