Thursday 10 March 2016

Threatened stereotype threat

 

the dark road

I’m in a dark place. I feel like the ground is moving from underneath me and I no longer know what is real and what is not.

Thus laments Professor Michael Inzlicht of Toronto, a man with honest doubts. Steve Hsu has described Inzlicht’s predicament here:

http://infoproc.blogspot.co.uk/2016/03/one-funeral-at-time.html

Inzlicht is very open about the personal advantages his work on ego depletion has brought him, and now his ego is depleted. He was also a strong believer in stereotype threat, and that is under threat. What has caused the ground to move beneath his feet?

Inzlicht has found that his frequently replicated effect has not survived the more intense scrutiny of a large sample, pre-registered study. You know very well that I keep repeating the complaint that so much of psychology uses unrepresentative, small samples that it produces weak results. Social psychology seems to be the most florid example of this lamentable practice. Now a strong method has superseded oft-repeated weak methods, as it should, and we are closer to the truth.

He adds, with commendable frankness: I edited an entire book on stereotype threat, I have signed my name to an amicus brief to the Supreme Court of the United States citing stereotype threat, yet now I am not as certain as I once was about the  robustness of the effect. I feel like a traitor for having just written that; like, I’ve disrespected my parents, a no no according to Commandment number 5. But, a meta-analysis published just last year suggests that stereotype threat, at least for some populations and under some conditions, might not be so robust after all. P-curving some of the original papers is also not comforting. Now, stereotype threat is a politically charged topic and I really really want it to be real. I think a lot more pain-staking work needs to be done before I stop believing (and rumour has it that another RRR of stereotype threat is in the works), but I would be lying if I said that doubts have not crept in. ...

I digress, but I had always thought that stereotype threat was monumentally silly, particularly when it was advanced to account for racial differences in ability. What this body of work asserted was that African Americans, far from being lower in ability than Europeans, were in fact so sensitive about being thought to be of lower ability that they performed as if they were lower in ability. Anything which triggered a sense of threat (black people not as bright as white people) immediately made them go into a tizzy of incompetence. Frankly, this seemed to be an excellent excuse not to employ them in any occupation which required a modicum of thought. It was a case of the remedy being worse than the illness, and of treating every African American as a delicate flower to be shielded from so much as a scintilla of intellectual challenge. This was nonsense, from which all should be spared, particularly African Americans, but also women, who apparently suffered from their own stereotype threat when asked to do sums.

By the way, I do not doubt that you can temporarily affect a person’s motivation by demoralising them in some way. Decades ago Seligman reported on the bad effects on problem-solving of inducing a sense of helplessness. It is a temporary manipulation, and part of normal fluctuation in the extent to which people will persist at tasks, though those most affected by such discouragement need to avoid challenging work, if they can.

But, my pleasure in seeing this research go down the plughole is tempered by two considerations:

1 How did psychology make such a damn fool mistake and repeat it so often?

2 What findings that I cherish are about to go down that same plughole?

Here, I go so far as to quote myself:

We must judge our cherished ideas by the same harsh standards we apply to the ideas we find most repellent.

Why are we so resistant to disconfirmation, so wedded to our opinions? Many fanciful ideas have been proposed: deep personal failings, deep political biases, a shallow love of peer approval and social status, and shallow, sanctimonious, posturing mendacity. I have a more modest proposal: effort justification. I think everyone is prone to being proud that they have actually read something, particularly a long book. At the end of that, having been in the company of one mind, they tend to see the author as their friend, a trusted companion. Having read the damn thing, the last thing they want to hear is that their efforts have been wasted. The situation is compounded if they have taken some notes and repeated the main arguments to friends. Any objections from critics can be countered by saying: Read the book yourself. The identification with the thesis is considerably strengthened if you give a talk about the work, showing slides covering the main points, and receive the approbation of an audience, particularly when audience members provide examples which substantiate the thesis. I call this the PowerPoint Effect: if it looks pretty and gets approbation, it is probably true. Lecturers are particularly prone to it. Once again, effort justification come into play.

Multiply all this tenfold if you had the original research idea, ran a pilot which gave positive results on a small sample, applied for a grant, appointed a research assistant, struggled to get experimental subjects, and finally got a result which seemed good, just like the pilot study did, and then wrote a book about the further studies which seemed to provide confirmations of your thesis.

The problem of trying to save a cherished idea from going down flames is not restricted to the unhappy Prof Inzlicht, nor to social psychologists with political views, nor to psychologists generally. It has general application, a potency of territorial proportions, and the seductive quality of a warm bed. Of course, intelligence research is different in that it occupies a lofty sanctuary of the mind, well protected by the teachings of the wise, but who would lose, full of pain, the Collects that give comfort and sustain faith, and suffer all of that travail just because of an awkward fact, a spoke in the spinning wheel of belief?

As the Alexandrian poet says in his magisterial The God Abandons Antony

Above all, don’t fool yourself, don’t say
it was a dream, your ears deceived you:
don’t degrade yourself with empty hopes like these.

As one long prepared, and graced with courage,
as is right for you who proved worthy of this kind of city,
go firmly to the window
and listen with deep emotion, but not
with the whining, the pleas of a coward;
listen—your final delectation—to the voices,
to the exquisite music of that strange procession,
and say goodbye to her, to the Alexandria you are losing.

There are times when we must kiss ideas goodbye.

Postscript: By the way, I don’t know if Cognitive Dissonance, of which Effort Justification is a variant, still holds up. Has it been subjected to a large sample, pre-registered replication? I wonder. I have lectured about cognitive dissonance often. Too often, perhaps. Another psychological fact going down in flames?  Say it isn’t true.

16 comments:

  1. It's a dangerous situation when a scientist thinks "I really really want it to be real" as Inzlicht admits.

    ReplyDelete
    Replies
    1. "stereotype threat is a politically charged topic and I really really want it to be real": heavens, he's essentially admitted to deep political bias.

      I suspect that the good doc's "effort justification" is usually too kind, and that he's nearer the mark with "deep personal failings, deep political biases, a shallow love of peer approval and social status, and shallow, sanctimonious, posturing mendacity."

      Though he omitted to mention deep love of being awarded research grants, tenure, and promotion, and other worldly concerns.

      Delete
  2. Razib has a post on this, with an amusingly pithy excerpt:

    "Emotions are running high. Two groups of very smart people are looking at the exact same data and coming to wildly different conclusions. Science hates that. This is how beleaguered Gilbert feels: When I asked if he thought his defensiveness might have colored his interpretation of this data, he hung up on me."

    ReplyDelete
    Replies
    1. Yes, saw it, and it made me laugh. My version: "Has anyone ever accused you of being arrogant?" Answer: "Only people of no significance".

      Delete
  3. The reason that stereotype threat has always seemed implausible to me is that the typical research design – comparing a treatment group that is primed with some stereotype threat and a control group that is not, and finding a depression in those who are primed – can’t explain group differences in test performance that are observed in real life. The “real life” test takers are equivalent to the control group – that is, they are not specifically primed with stereotype threat before taking the test. If you say that stereotype threat is “in the air we breathe” and so it affects real-life test-takers, then why is it not in the air being breathed by the control group in stereotype-threat research?

    ReplyDelete
  4. Proponents would argue that the negative stereotypes were only applied to Africans (and women), so they were under pressure and the white control group were in the relaxed majority, able to give off their best. Increasing the stereotype threat by an experimental manipulation merely augments it so that it can be measured. So the argument went.

    ReplyDelete
  5. What about stereotype challenge? You know, "you're black so you must be good at basketball, so throw this ball through that hoop". I'm sure there's a nice little earner in that notion.

    ReplyDelete
    Replies
    1. Ah, stereotype demand! Which leads to performance anxiety, unless skillfully deflected by a special training program. You will make millions, no doubt about it.

      Delete
  6. "an awkward fact, a spoke in the spinning wheel of belief"

    Just this. Thank you. That's all we're ever looking for.

    ReplyDelete
  7. It is almost like no one actually read the original paper where the author noted that the scores were equal only after the gap in SAT scores between black and whites were corrected for. Even if sterotype threat had a noticeable effect size, it is after the author corrected for an average gap between racial disparities in SAT scores.

    ReplyDelete
  8. And much of my blog is taken up with looking at all the assumptions hidden away in making these "corrections" or, as the English say, "fiddles".

    ReplyDelete
  9. The original stereotype threat study at Stanford didn't show any ability to increase test scores on a low stakes in-class test over what the students scored on the high stakes SAT, it just showed it was possible to depress black students scores on the low stakes test by telling black students they weren't expected to do well on it. One possibility is that all that was happening was that black students took that to mean they shouldn't bother working hard on this low stakes test.

    Scoring high on tests takes some mental effort, so it's not surprising if it were possible to prime students not to put in much effort on a test that is meaningless to them.

    ReplyDelete
    Replies
    1. Or maybe it's just publication bias and the like.

      I have this discussion with Andrew Gelman every so often. When some study fails to replicate, I tend to concoct some excuse for the original researchers, but he usually says, Nah, there was probably nothing ever there.

      He usually wins these debates with me.

      Delete
    2. Or maybe it's just publication bias and the like.

      I have this discussion with Andrew Gelman every so often. When some study fails to replicate, I tend to concoct some excuse for the original researchers, but he usually says, Nah, there was probably nothing ever there.

      He usually wins these debates with me.

      Delete
    3. Agree that you can get temporary effects by letting people know that a task is not important, or does not suit their skills. It is part of a general issue, which is that lab psychology does not always generalize to real world situations, particularly where a social manipulation is involved. Studies fail to replicate because sample sizes are way too small, far too unrepresentative (mostly psychology students) and the raw results are subject to too many "corrections" and adjustments.

      Delete