US GAO says TSA BDO program is BS

When it comes to the science behind behavioral profiling the TSA rarely delivers much in the way of compelling evidence. The Government Accountability Office’s latest report (PDF) probably won’t help change that impression. Of the 36 behavioral indicators the TSA considers it lacks sufficient original research to support 28. As part of the audit the TSA provided 178 forms of evidence to the GAO to support the practices. The agency dismissed 175 of those as insufficiently scientifically rigorous. Put simply, the Behavioral Detection Officers operated from the get-go without sufficient scientific justification for their program.

From the report summary:

Specifically,

• Seventy-seven percent of the sources TSA cited (137 of 178) are news articles, opinion pieces, presentations created by law enforcement entities and industry groups, and screen shots of online medical websites that do not meet GAO’s definition of valid evidence.

• Twelve percent of the sources TSA cited (21 of 178) are journal articles, books reviewing existing literature, and other publications that may reference original research in the text, but do not themselves present original analysis, methods, or data whose reliability and validity can be assessed.

• Eleven percent of the sources TSA cited (20 of 178) are original research sources reporting original data and methods. However, 5 of these sources do not meet generally accepted research standards. Of the 15 sources that meet generally accepted research standards, 12 do not present information and conclusions that are applicable to the specific behavioral indicators TSA cited these sources as supporting.

In total, GAO found that 3 of the 178 total sources cited could be used as valid evidence to support 8 of the 36 behavioral indicators in TSA’s revised list. More specifically, TSA has one source of valid evidence to support each of 7 indicators, 2 sources of valid evidence to support 1 indicator, and does not have valid evidence to support 28 behavioral indicators. GAO makes no new recommendations in this report.

In 2013 the GAO performed a similar review and the results were not good for the TSA.

TSA has since reduced funding for its behavior detection activities. Although the Department of Homeland Security (DHS) did not concur with our November 2013 recommendation, TSA officials have stated that our recommendation was one of several factors in DHS’s decision to support a reduction in the number of full-time equivalent (FTE) BDOs. TSA reduced the number of BDOs from the 3,131 BDO FTEs that DHS requested and funded in fiscal years 2013 through 2015 to 2,660 BDO FTEs requested in fiscal year 2016. This reduced number of BDO FTEs in fiscal year 2016 represented a reduction in annual operating costs of about $35.4 million. In fiscal year 2016, TSA officials reported that they employed 2,393 BDO FTEs at 87 airports and spent approximately $186 million on the behavior detection and analysis program.

Those numbers suggest an average expense per BDO of $75,000, plus or minus a bit.

The 36 behavioral indicators is a drop from 94 that were in use in 2013. That may seem like good progress but the TSA also says that “most of the 94 behavioral indicators were combined, condensed, or updated for incorporation into a revised list and a small subset were eliminated.” So, same metrics just reclassified. Not really changing the process too much there.

The full list of indicators remains protected by the TSA but a couple of the examples are detailed in the report, mostly in how the GAO explains there is no evidence to support them:

TSA cited 105 sources to support the use of indicator number 11, which involves BDOs identifying individuals who seem to be attempting to conceal their normal appearance. However, we found that none of the 105 sources present original research that meets generally accepted research standards. In another example, TSA cited 63 sources to support the use of indicator number 5, which involves BDOs identifying individuals who seem to be sweating heavily. While we found that one of the 63 sources cited is original research that meets generally accepted research standards, we also found that this source does not present evidence that is applicable as support for this indicator.

Perhaps it is good news that the BDOs are now all back among the ranks of the regular screeners. There is no longer a standalone BDO program, though the “qualified” staff members “are required to spend at least a certain number of hours per pay period conducting behavior detection activities to ensure that the capability is regularly practiced and sustained.” Yes, the TSA continues to sustain a program it cannot demonstrate adds any safety value to aviation.

The TSA also continues to cling to the idea that stopping simple criminal activity is a stepping stone to protecting the aviation infrastructure, its primary mission:

Although DHS did not cite any examples of how its behavior detection activities have thwarted threats to the nation’s civil aviation system, it cited an instance where TSA’s behavior detection activities led to the identification of an individual attempting to conceal illegal drugs at an airport screening checkpoint. DHS asserted that the techniques used by individuals attempting to conceal illegal drugs may also be used by terrorists to conceal explosives, thus suggesting that behavior detection may be useful in identifying would-be terrorists.

The GAO takes the agency to task on this view, citing a study indicating “that success in identifying deception and intent in some studies incorrectly equates success in identifying terrorists with the identification of drug smugglers, warrant violators, and others.” Oops.

So, yeah, the BDO program lacks a basis in reality. The TSA is scaling it back but continues to push for at least parts to remain alive, despite the GAO recommending that it be shuttered unless real scientific research can be unearthed to support the cause.