10 Responses

  1. Tony Reece
    Tony Reece at |

    Recovery counts. At that, DL usually excels. I wish them well “from across the field”.

  2. mhenner
    mhenner at |

    i wonder if flights to and from EU countries will generate a lot of payments based on the EU compensation requirements.

  3. Sice
    Sice at |

    A few thoughts, and these are based on my experience working at Worldspan 16 years ago (we shared datacenter building with Delta Technologies). The power situation in that building was robust, even back then, so it’s clear that any power outage damaged critical systems. The diesel generator plant outside the building was enormous. Power entered the building at multiple points. DL and Worldspan handled so much data it was staggering but I’m really confused why, in this age of redundant systems/datacenters/data there wasn’t coverage for this.

    Something really bad happened, let’s hope DL fesses up about what the actual problem was.

    1. Chuck
      Chuck at |

      Yeah, a world class datacenter will have multiple power inputs as well as multiple fiber lines for data I/O. And also inline uninterruptible power supply batteries to hold things up while the generators spin up. Even a dot-com startup ought to have a robust enough production infrastructure to have a hot-standby failover DC in a not-at-all-nearby location and should be able to migrate traffic to the backup in a handful of minutes.

      Between this and the WN issues earlier, I’m contemplating offering myself up to airlines as a consultant 😉

  4. Golfingboy
    Golfingboy at |

    This seems to be a rather common issue in the airline industry. Happened to AS, DL, UA, AA, WN and many others with some getting hit multiple times whether it is a 30 minute blip or drags several hours. I am not an IT expert, but from an outsider perspective I think the most perplexing part for me is why they (or SABRE/Travelport) do not have a secondary standby DC in a different hub city.

    My company owns 4 DCs across the US and in each DC we have two full sets of PROD servers (A and B, also color coded as Green and Blue) with its own coolant system, dedicated generator, water tank, data lines, separate air locks, etc. Everything in each DC is mirrored (identical servers, identical set up, identical application, and so forth), so if there is an isolated incident in one of the PROD server rooms they can switch to the B servers in matter of minutes. If it is a geographical location specific issue or possible severe inclement weather at our primary DC then it is a relatively easy near automatic re-routing to one of the three DC sites to maintain network resiliency.

    Granted 4 DCs with 8 sets of fully equipped PROD servers would be overkill for the airlines, but having at least a full set of standby servers at a different location seems to be a sensible course of action.

  5. Maureen
    Maureen at |

    Seth,
    Thanks for the analysis of what happened with Delta today; really appreciate the last point re minors. Will be a good way to assess future IRROPS if that caution appears.

  6. Watch #PaxEx Minute: Dealing with airline disruption meltdowns - Runway GirlRunway Girl

    […] Three thoughts on today’s Delta systems outage […]