Social Scientists Can’t Predict Better Than Simple Models, Study Finds
I previously covered a study which found social scientists can’t predict better than laymen. As you may recall, ‘experts’ and laymen were asked to predict the size and direction of social change in the U.S. between April and October of 2020. Overall, the two groups were equally inaccurate – there was no overall advantage for the ‘experts’.
A new study finds that social scientists can’t predict better than simple models either. (The study is a pre-print, so hasn’t yet been peer reviewed.)
Dillon Bowen collated data from several previous studies in which ‘experts’ had been asked to predict the impact of various interventions on people’s behaviour. His analysis covered five previous studies, but I’ll focus on the first three since the other two are rather complicated.
In the first study, 90 ‘experts’ were asked to predict how effective 53 different nudges would be at encouraging gym users to exercise more often. These nudges included things like administering a free audiobook and a bonus for returning after missed workouts.
In the second study, 24 ‘experts’ were asked to predict how effective 22 text-message interventions would be at encouraging Walmart customers to get a flu vaccine. These interventions included things like a simple reminder and a joke about the flu.
In the third study, 237 ‘experts’ were asked to predict how effective interventions carried out by two government nudge units would be at changing citizens’ behaviour. These interventions encompassed domains like healthcare, state benefits and community engagement.
For all three studies, Bowen compared the ‘expert’ predictions to a null model that said none of the interventions would work (i.e., none of them would change subjects’ behaviour by a statistically significant amount).
Remarkably, he found that in all three cases the null model performed better than the ‘experts’.
In other words, the ‘experts’ overestimated the interventions’ effects by such an extent that they would have been more accurate by simply guessing ‘no effect’ for every single one. Or put in other way, the effects of the interventions were generally closer to zero than they were to the ‘experts’ predictions.
Behavioural interventions are widely used by governments and other interventions to tweak people’s behaviour without coercing them. Yet practitioners consistently overestimate how well these interventions work. As it turns out, ‘they don’t work at all’ is a safer assumption than whatever the ‘experts’ are saying.
To join in with the discussion please make a donation to The Daily Sceptic.
Profanity and abuse will be removed and may lead to a permanent ban.
Governments use of coercion has gone far beyond encouragement or tweaking. It is in the realm of propaganda and should be stopped.
As I recall it was the Liberal Democrat David Cameron who introduced the “nudge unit” into Downing Street.
Nudge: if persuasion doesn’t work, use force.
A huge problem here is that the ‘social scientists’ haven’t included an understanding of how over-use of their ‘nudges’ could result in widescale mistrust of any future ‘nudges’.
I suggest that the extreme propaganda and ‘psychological meddling’ that went on in 2020 and 2021 will in time result in extreme social unrest that could well have a huge impact.
Are you aware of this publication from Amnesty in 1973? The coercions utilised relentlessly are recognised as forms of torture. Slide 26 p49
https://www.amnesty.org/en/documents/ACT40/001/1973/en/
There’s only one reliable predictor of human behaviour at our disposal and that’s the stockmarket. So powerful that the US military seriously considered using a market to bet on the likelihood of future terrorist attacks. Yes, seeing where the money would go would be used to try and get one step ahead of terrorists across the globe. Obviously it was quickly shelved as being unethical and never used. One curious aspect of Covid was that on March 23rd the stockmarket, in the midst of a new apparently deadly virus outbreak, started one of the biggest bull runs in history. At its peak in November 2021 it had run up a 149%. Put £1 per point (Margin required about £500) on March 23rd 2020 in a spread betting account and you’d have had £10,000 by November 2021 or a staggering annual return of 25,000%! Why, when the world was facing devastation from a new deadly virus that could decimate the labour force, did the stockmarket skyrocket? Part may have been the prospect of stimulus but that still wouldn’t have helped if the workforce was going to be dead or seriously ill. The other explanation is that if you, as a massive… Read more »
Well, they seem to have manipulated the behaviour of the Chancellor – according to the propaganda world service this morning, he has to come back from America a day early; money talks.
“the ‘smart’ money did their own research on Covid”
Or, more plausibly the ‘smart’ money was actually backing / running the scam.
One of the initial aims of the Scamdemic was to siphon money from ordinary people and in to the pockets of the “elites.” A plan which appears to have worked quite well so far.
Ah yes, the often overlooked Null Hypothesis. No money in advising politicians to do nothing.
That’s true because in our ‘Democratic’ system the profit from buying a slice of political power and influence is privatised, concentrated among a relative few, whereas the cost is socialised spread across the many.
The politicians are middle-men and take their commission
So governments take note, this implies the people, not reacting, recognise propaganda when they see it, and the net result will be an increase in cynicism in government. And isn’t that precisely what we now have?
It is perhaps, but it doesn’t seem to stop people from still obeying without question.
Society isn’t a science. It is the natural emergence of spontaneous order brought about by shared language, heritage, family, culture, customs, traditions, manners and economic interdependence.
It did this very well for tens of thousands of years prior to social scientists and politicians.
All social problems are the result of interventions to shape society according to whatever fashionable ideology, crazy, half-wit notion, and the political agenda du jour.
I’ve got a problem with the term “social scientist”. Are they into psychology (https://dictionary.cambridge.org/dictionary/english/psychology ), or is it just a made up term to justify their activities, or make them look more valuable (financially, perhaps)? As we are social animals, normal “psychology” would fit the bill, maybe with an element of speciality on group behaviour.
Behaviour manipulation goes on all the time; that’s politics, managing a business etc – a list as long as your arm.
“Behavioural interventions are widely used by governments and other interventions to tweak people’s behaviour without coercing them” Thank god it doesn’t work.. always.. It could be argued that these sorts of behavioural interventions, or ‘nudge’ represent the most insidious, damaging and evil form of coercion. Bob holds a gun to his neighbour Gary’s head and tells him to give up his garden, as he needs the space for his children to play. The dynamics of this interaction are clear to both parties – the incentive for Gary to comply is fear of being shot. Bob wants the garden and has the leverage to get it – his actions in using the threat of force make it harder to hide from his own motivations. If he shoots Gary, he’ll have to live with the consequences – in doing so being forced to examine whether the threat or his course of action was proportionate and fair. If Bob instead uses tools of psychological manipulation to persuade Gary that it’s in his best interests to surrender his garden, convincing him he’ll be happier without the responsibility of caring for it; that he’s selfish and bad for having such a big garden when his… Read more »
Where does the quote from Lewis come from? I’m curious as to who he was addressing his remarks to at the time.
It’s from God in the Dock: Essays on Theology (Making of Modern Theology).
Maybe they are no good at prediction because social science is not a science.
Excellent article, especially enjoyed ‘Remarkably, he found that in all three cases the null model performed better than the ‘experts’.’
The reality is that genuine science deals with the observation, understanding and manipulation of purely physical substances and processes (atoms, molecules, sound waves, gravity etc) whose behaviour is fixed and therefore predictable / repeatable.
Above and beyond our physical bodies (which certainly can be approached scientifically via eg medical pathology) human beings possess free will (via a non-material and eternal soul) therefore our thoughts and behaviour are both non pre-determined and non predictable.
Which is a very long winded way of saying that all ‘social science’ is pseudo science (as a For a fist full of roubles already pointed out);
And because of its fundamentally deceptive, statistical and dehumanising approach a very dangerous type at that (as a glance at its history in practice will immediately reveal).
“Social Scientist” – the ultimate oxymoron.