Banged to rights.

Can you trust the data?

Channel 4 have just launched a new series ‘Banged Up – Teens behind bars’.
The premise is: “Difficult Brits do time in a full-on Florida jail” and as a result will be scared into seeing the error of their ways and adjust their current behaviour and attitude to crime before it becomes too late.

In his book ‘Black Box Thinking’, Matthew Syed explains how the same experiment took place in the late 1970s in New York. In a programme titled ‘Scared Straight’, the idea was exactly the same. Take a bunch of delinquent (or near-delinquent) teens and introduce them to real inmates. These would both be intimidating and terrifying in delivering what real life is like for hardened criminals in one of the world’s most notorious prisons.

It makes for great television. Parents love it as a tool to hopefully cut through all the nagging. Kids love it as they watch and think they’re too cool or ‘hard’ to be intimidated by it. The exact same emotions of those families participating. However, the bravado soon turns to tears and on the face of it, they commit to change their ways.

Oscar winner

The 1979 film won an Oscar for best documentary feature and the programme was rolled out across the world. It was massively effective and backed by judges, prison warders and a whole host of experts. The data was amazing and Judge George Nicola said “When you view the programme and review the statistics that have been collected, there is no doubt in my mind… that the juvenile awareness project… is perhaps the most effective, inexpensive deterrent in the entire correctional process.”

20 years after the original broadcast, in 1999, the documentary crew revisited the original seventeen teenagers. The results did indeed appear to be as brilliant and solid as had been stated. When interviewed they talked about their new lives and how their visit to the prison had put them on the right track. Their stories were inspirational to all who watched, from stealing cars to becoming a preacher. From drug dealing to a book-keeper and mother. One by one they credited the programme with saving them from a life of crime.
The stats backed it all up too. Evidence presented spoke of an 80-90% of attendees going straight and was hailed as “An amazing success story… unequalled by traditional rehabilitation methods.”

Hold on a minute

Then something happened. A professor of Criminal Justice, named James Finckenauer decided to look a little deeper into the data and statistics. He wanted to know how the 80-90% success figure had been arrived at. It was through responses to a four question questionnaire sent to parents of the teenagers who had taken part in the prison visits.
He didn’t believe this method was particularly robust or reliable.
The organisations and agencies responsible for running the programme were asked to write letters of commendation. In essence, asked to mark their own homework and determine future investment.
The professor then looked into the background of the thousands of teenage participants before the visit. They found many of them were not delinquent before, so how solid was the assertion that they had changed their behaviour?
Finally, the questionnaires to parents were often sent out within a few weeks of the prison visit and only the ones who responded were included in the data. So it was entirely plausible that only the parents of kids who had shown a positive response to the visit actually bothered to fill them in.
Add to this, the data never looked to any other mitigating factors. What was happening at home? Was anything being done different at school? Even, how was the economy performing? All possible contributing elements that were ignored.

Evidence ignored

Finckenauer conducted his own research by holding treatment groups vs control groups. The evidence now was overwhelming that the programme DID NOT WORK. More than that, the teenagers who went through the programme were more likely to get caught up in a life of crime. The damage it did to them psychologically tipped the vulnerable over the edge. In some instances, crime amongst participant groups increased as much as 28%.

With rigorous research proving overwhelmingly that ‘Scared Straight’ didn’t work and even worse had the opposite effect to that desired, how did the authorities react? You’d expect them to instantly drop it but here’s the interesting thing. Cognitive dissonance kicked in. The top brass were so invested in it as they’d been doing it for years. They believed the data told them it was the right thing to do. They couldn’t admit it had been a waste of time and money and worse still, was even having an adverse effect. So much of them and their reputation was invested in the programme. There was no way they were going to allow themselves to look like they had got it wrong. That they had in fact been fooled by dodgy data and measuring techniques.
40 years later and here we go again. Sending British kids into a situation that is proven to be dangerous for their future. Having learned nothing because the data being used is selective and the emotion (and TV viewability) attached to it means we want to believe it works.

Data junkies

Sad really and of course we’d never blindly follow false and misleading data in our own world, would we?
We wouldn’t ignore the stats about digital ad-fraud because 10+ years ago we were promised a new era of advertising accountability.
Collectively we’d scoff at anyone who suggested last click data wasn’t the most important metric and anything before that moment was vitally important.
We’d de-friend anyone who exposed the latest social fad, showing it was misusing our personal information to hound us with awful information and change our voting habits.
We’d close our ears to the merest implication that voice activated devices were gathering information on what you want for dinner tonight.
And of course we would all take with a pinch of salt the amazing results generated in agency case studies. Presented on their spangly websites and whizz bang creds documents.
Agencies and clients up and down the land are so invested in the shiny new toys that we’re terrified that the uprising of real information will show us all to have been hoodwinked. The digital revolution made everything accountable but never more so has the old saying of ‘lies, damn lies and statistics’ been more relevant.

The Emporor has no clothes

How would we ever explain to the boardrooms holding the purse strings that we got it all wrong? We can’t, we daren’t. So we keep on doing it. Cognitive dissonance insists we pretend everything is okay. That if we don’t trust in the data, someone else will and in turn win the battle for expenditure.

Maybe it’s time to be fierce and get back to doing what we agencies have always done best. Creating work that shifts perceptions and product and concentrating on one metric. Sales.

You can and should get hold of a copy of Black Box Thinking here.

Can you trust the data?

If you’d like an honest appraisal of what metrics you should really be looking at, drop us a note saying:. ‘I just want to sell more stuff’ here.