<< Previous

People think SFSTs are scientific because NHTSA says they are. Over and over "scientific" research has reported SFSTs are extremely accurate. But these "scientific" studies were non-peer reviewed in-house jobs, arranged by NHTSA, paid for by NHTSA, done by NHTSA contractors, never accepted by any scientific journal but just printed up by NHTSA itself. That's like doing away with the EPA and having coal companies validate strip mines.

In important ways, NHTSA's SFST "science" isn't like real science.

All in-house

None of the studies that officially validate the SFST are taken from independent scientific literature. Every official study was arranged, paid for and printed by NHTSA.

More than validation was in-house. SFST development was also done entirely by the police bureaucracy. There is a small literature on sobriety testing in general, but the fact that the current standardized HGN, OLS and WAT tests supposedly measure sobriety, how each test should be administered and scored, how scores should be interpreted, and how accurate those interpretations are—all that was invented by NHTSA contractors and validated only by NHTSA contractors.

SFSTs are entirely the product of the police bureaucracy.

Never any peer review

None of the studies that officially validate the SFST was reviewed by independent non-NHTSA scientists.

The point of peer review is not so much to prevent cheating as it is to find and prevent mistakes. Thinking the Oktoberfest "accuracy" statistic validates a diagnostic test is an amateur's mistake competent peer review would have found and prevented.

All six NHTSA SFST validation studies involve a single contractor

NHTSA paid the contractor to invent the SFST, then paid the same contractor to validate, and re-validate, and re-re-validate the test the contractor had been paid to invent.

NHTSA validation started with the answer NHTSA wanted, and worked backward

Suppose the SFST really could, as police claim, identify a specific blood alcohol concentration (BAC). You'd think what scientific research would do is administer the SFST a bunch of times and find out what that BAC is. But that's not what NHTSA did. NHTSA started with the BAC that defined per se DUI, 0.10%, and told it's contractor to prove the SFST identified that alcohol level. Fail the SFST, your BAC is above 0.10%. Pass the SFST, you BAC is below 0.10%. That's what NHTSA's contractor did.

But the first SFST validation study was done in Colorado, which has a second per se BAC for commercial drivers, 0.05%. NHTSA's contractor discovered the SFST also works at that alcohol level. So SFST identifies alcohol levels as being above or below different levels, depending on what sort of license the driver has.

Then the legal limit changed. It went down from 0.10 to 0.08%. So NHTSA paid its contractor to re-validate the SFST, this time starting with the answer 0.08%. And that's what the contractor did. This time the contractor found the SFST—unchanged in any way— identified BACs above and below 0.08%.

There's even a fourth per se BAC limit, 0.04%. NHTSA contractors have also discovered the SFST identifies BACs above and below that limit.

0.10, 0.08, 0.05, 0.04—fail the SFST, your BAC is above the legal limit; pass the SFST, you BAC is below the legal limit, whatever that limit happens to be.

Bad science, good people
Please please understand, I'm not saying that when NHTSA's contractors claim to discover the SFST works at whatever BAC NTHSA wants it to work at, they are cheating. I don't think they are. My guess in NHTSA and its contractors haven't thought rigorously about how diagnostic testing works. I don't think they are liars. I think their science isn't very good.

Nothing here at SFST.US suggests police are liars or NHTSA's SFST validation contractors are cheats. Police are honest, honorable people. For goodness sake, they go to work in bullet-proof vests! They're better men than me. The lady police are better men than me. But they are not scientists.

The fine folks at NHTSA are good people doing their best. But mostly they are not scientists.

Science is harder than it looks. Validating diagnostic tests is harder than it looks. Validation methods based on common sense lead immediately to immense over estimations of test accuracy in a way that is entirely invisible to non-specialists. SFST validation science looks like real science. But it's not. SFSTs are not junk science because somebody is cheating. SFSTs are junk science because they were invented and tested by wonderful people who lacked the specialized training necessary to know the subtle tricks required to validate a diagnostic test in a scientifically meaningful way.

SFST.US is not about these good people, it's about that bad science.