There’s another list out this week ranking the best airline loyalty programs. US News & World Reports has released their Airline Rewards list ranking 10 US-based programs. The results are interesting and I don’t necessarily think they are completely right or wrong. But I do have a couple objections to the way the results were calculated. I do believe the list is more realistic than the SwitchFly study which comes out a bit earlier in the year, but still not completely on target. Here’s the list of winners according to US News:
- JetBlue TrueBlue
- Southwest Rapid Rewards
- Alaska Airlines Mileage Plan
- United MileagePlus
- American Airlines AAdvantage
- Virgin America Elevate
- Delta SkyMiles
- HawaiianMiles
- Frontier EarlyReturns
- Free Spirit
The problem is not the order they ended up with; it is how they got there. US News screwed up badly.
The rankings are focused on redeeming for travel awards so that helps justify the order, but they also do some strange things in calculating the numbers. Here are the factors they use:
- Number of Daily Flights (10%)
- Geographic Coverage (10%)
- Free Flight Options (20%)
- Ease of Earning Free Round-Trip Flight (30%)
- Additional Benefits (10%)
- Airline Quality Rating (20%)
The OK bits
I’m not a huge fan of the AQR study and believe it biased against some realities passengers face and overemphasizes other factors. I also think the Additional Benefits category is too vague, at least as they’ve presented the information. But I can put those two concerns aside, particularly because they’re relatively small portions of the score. Similarly, an airline with more flights should generally be better for passengers, both in terms of offering earning and redemption options. I like the first two categories being included for that reason. But US News ignores partners which limits the value of those categories in their analysis. The best ranking an airline can get comes from offering service to 200+ destinations. That’s rather narrow, really, unless partners are included. Alaska Airlines has a ton of partner destinations while JetBlue has almost none. Those sort of factors are harder to quantify but skew the results a bit. But only a little bit.
Free Flight Options
The Free Flight Options is where things start to go badly. Here’s how they define that rating:
The Free Flight Options score represents the number and variety of flights that can be purchased using miles or points. In order to calculate this score, U.S. News referred to the airline and Bureau of Transportation Statistics to determine each carrier’s five most highly trafficked routes. Each program is assigned between 1 and 5 points based on the average number of total and direct daily flights along these routes available for booking with rewards points. For example, airlines that offer an average of 10 total flights per day on its most popular routes receive 3 points; airlines offering more than 20 total flights earn 5 points.
So they’re taking the top 5 routes and then looking at how many total flights on those routes are available to redeem for awards. The revenue-based programs are likely to do much better on this sort of measurement, and they mostly do. Beyond that, however, is the oddity of only caring about the top 5 routes (by passenger count) each airline flies. If the measure is “Which program is better for people already flying that airline?” then maybe that makes sense, but even then I’m not so convinced.
Ease of Earning Free Round-Trip Flight
But the worst calculation factor is also the one most highly valued in scale: Ease of Earning Free Round-Trip Flight. The analysis completely ignores the differences in earning rules between the programs. They still only account for the 5 most trafficked routes, but then they go a step further, screwing up the means to figuring out how to earn enough points to redeem for that flight:
The average of minimum price in points for each route is then divided by the average length in miles (round-trip) of all five routes to determine the average number of round-trip flights program participants must take to earn a free round-trip flight.
That mostly makes sense for distance-based earning and redemption programs, presuming that earning only comes from flying. But it completely misses for revenue-based programs. JetBlue looks great on redemption rates if you believe that earning is distance-based. The 14,200 point average redemption cost for New York – Orlando seems spectacular based on 1888 miles round-trip (only 7.5 trips to get a freebie!), but that’s not how JetBlue earning happens. To earn enough points to redeem for that ~$225 round-trip flight a passenger would need to buy nearly 12 trips at a comparable price, not the 7.5 the US News calculations suggest.
Map generated by the Great Circle Mapper - copyright © Karl L. Swartz.
Hopefully better next time
So, yes, this is a ranking done by a reasonably reputable organization and it has a published methodology which is well defined. But it misses on two major marks: Assuming that people only earn points for flying and misunderstanding how revenue-based programs really work. Hopefully they get that fixed for next year when at least six of the ten programs surveyed will have that as their primary earning method.
Never miss another post: Sign up for email alerts and get only the content you want direct to your inbox.
Seth,
I applaud breaking down the (hazy/somewhat lazy) methodology in this ranking… and heck, pretty much every ranking widely publicized.
The question, of course, is why not do a better one?
I would think that with your expertise, and the interested/educated readership you have, there’d be a great opportunity to come up with a more sound methodology (or two) for ranking/rating the various airline (and hotel) programs.
Step 2… err… publicize? Step 3: Profit!
Resources. I haven’t managed to become independently wealthy so I cannot really afford the time it takes to do it right. I keep toying with it, though. One of these days….