
So, a while ago I mentioned that I would write about why I'm so discouraged about neuroimaging approaches to understanding neurocognitive deficits in psychopathology. It's a fancy way of describing the use of techniques that pick up on brain activity (like electroencephalography or functional magnetic resonance imaging) to understand why people with certain psychological disorders (like schizophrenia) have problems with basic cognitive abilities like attention and memory. This is what my dissertation is on, so it's a bit unfortunate that I'm so discouraged about it. It's a complicated enough subject that I think it's going to take two posts to do it justice: part one, on falsification and why I feel it is so important, and part two, on how I feel we've moved away from falsification in my field.
To start out, it would help to back up a bit, to my senior year of high school. I went to an all-boys school that required us to take about 2 hours of sports every day. That's a crazy amount of athletics for most people, and although initially I tried to tough it out and take "hard" sports like wrestling and track, by senior year I was just sick of it. My friend Pat and I discovered that our sister school had bullshit "non-team" sports that through some oversight we were actually allowed to take. One of these "sports" was Aikido. It was me, Pat, one other guy, and 3 girls. We were taught by a lady who, in retrospect, looked a lot like Aileen Wuornos, but did not hate dudes with a murderous passion. She was, however, very enthusiastic about Aikido, and told us all sorts of stories about the charming older Japanese gentleman who had founded Aikido, whom she called
Osensei. She told stories of his demonstrations of how to handle being attacked from multiple angles, which involved him throwing brawny young men about every which way all at once. She mentioned that there were whispered stories of him having such fast reactions that he was able to dodge bullets in the war.
This last struck me as bullshit, but I was intriqued by the basic moral of these stories: if you spent years and years learning Aikido, eventually you would totally be able to kick anyone's ass. It seemed a little implausible, given that Aikido training mostly consisted of having people come at you very slowly with their arm out as if they were going to chop you in the forehead. Given that all-boys schools are a fair approximation of Lord of the Flies, I had seen my fair share of fights, and I'd never seen anybody chop anybody else in the forehead. Mostly dudes punched or tackled each other (my preferred move was the headlock, left over from my wrestling days), and with a rapidity that would seem to render ineffective most of the rather complicated maneuvers that we were learning. Still, I was an adolescent, and I was so open to new ideas that I was still reading Ayn Rand, so I wasn't about to call bullshit on this nice lady. But, as soon as the season ended, so did my interest in Aikido.
Flash forward to several years later when I read an article about this new-fangled thing called the Ultimate Fighting Championship; I believe the article was in Spin (ha! the nineties!). As I recall, the article was mostly about the dominance of the Gracie family in those early years, but it mentioned that the first few years of UFC had been more of a "death-match" atmosphere, where all different styles of martial arts went head to head against each other, with the winner advancing to the next round. You had Sumo wrestlers squaring off against boxers, kung-fu masters against tae-kwon-doe champions, even a few masters of the dark arts of nin-jitsu showed up (sans throwing stars). There was even, if memory serves, a hulking young man who was a black belt in Aikido. If you don't follow the UFC, the story ends badly for most of these styles of fighting. People with backgrounds in wrestling, kick-boxing, and most importantly Brazilian Jiu-Jitsu basically embarrassed all the black belts in the other domains. The poor Aikido guy got his ass handed to him. Turns out that slow-motion forehead chops did not figure heavily in the match, much to his chagrin.
So here's my thesis: pre-UFC, martial arts were akin to religion: each sect was able to claim whatever powers it wished, to assert its superiority over all other sects, to assuage the doubts of initiates by assuring them that many more years of study would answer all their questions, to schism endlessly over matters that might seem trivial to outsiders. Eighty-year-old dudes who weighed 120 pounds were allowed to claim that they could defeat virtually anyone in hand-to-hand combat. Why? Because nobody ever called bullshit. Nobody ever said: if what you are claiming is true, then it follows that you should be able to get in the ring with that enormous 22-year-old guy over there who is not one of your students, and you should be able to beat the shit out of him. No one ever attempted to falsify these claims that were made. Post-UFC, there was suddenly a very visible way to test any claim someone wanted to make about their particular brand of martial art. If Crane-style Kung Fu is so great, then go kick that guy's ass. Suddenly everyone was trying to falsify each other's claims.
Falsification. Looking not for confirmatory evidence, but for disconfirmatory evidence. As I mentioned previously, Karl Popper famously described falsification as the cornerstone of scientific progress. If you are interested in finding out the truth about something, it's what you have to do. You can see the dangers of what happens when you look only for confirmatory evidence in pseudo-sciences like phrenology - well-intentioned scholars who swore up and down that personality characteristics like conscientiousness were actually detectable in bumps on the skull. How did they come to believe something so crazy? Well, probably they started out by feeling the skull of some really conscientious fellow, found a bumpy part, and then looked for that part to be bumpy in any new person they met who was conscientious. If someone was missing that bump, probably they weren't all that conscientious, even if they said that they were. Nobody was ever a jerk to the phrenologists, nobody ever said "If what you're claiming is true, you should be able to take 100 men, 50 of whom are generally agreed to be conscientious and 50 of whom are total slackers, and you should be able to sort them correctly into two groups by feeling their heads. If you can't do that, then you're probably full of shit."
You can also see the perils of looking only for confirmatory evidence in the history of psychology. Irving Bieber is one figure that leaps immediately to my mind: he was a psychiatrist who wrote a very influential early study of homosexuality and its origins. He performed psychoanalysis with hundreds of gay men in the 40's and 50's, seeking to piece together the common thread that could lead a man to become "a person whose heterosexual function is crippled, like the legs of a polio victim." He found that homosexual men were created when their fathers were excessively cold or distant and did not protect them from the subtle seductive or "close-binding" attempts of their mother. Now, many people reading this in 2008 will immediately be able to spot the flaws in this chain of reasoning - first of all, of course gay men in the 40's and 50's had weird relationships with their dads. That proves fuck-all. Second of all, you're only looking at gay men who are in traditional 3x/week psychoanalysis, so that's not exactly a representative sample. Those guys probably had even weirder relationships with their dads than most gay men of the era. Finally, when he happened upon gay men who claimed to have good relationships with their dads, he would question them and undermine them and work on their "defenses" until he was satisfied that in fact the relationship had been terrible all along. It never occurred to Dr. Bieber, but he would never see anything except for confirmatory evidence of his theories, because that's all he ever looked for. Even in the 70's when people started calling bullshit, he stuck to his guns and never changed his mind.
Popper famously criticized Freudian psychoanalysis because so many of Freud's claims were inherently unfalsifiable - e.g. the claim that the human sexual system is structured around a scaffolding of an inherent desire to have sex with your parents (ick, right?). If you've ever had thoughts or dreams about having sex with your parents, that's just proof of what Freud was saying. If you have never had such thoughts or dreams, that's also proof of what Freud was saying, because you suppresssed your desires because they were so powerful and dangerous. Thus, both the presence and the absence of some phenomenon are taken as proof of the theorem, and as such it cannot be falsified. Now, I know that many of my peeps who are into psychodynamic therapy will have problems with this characterization, and I admit there is room for argument. I think that some (not all) psychoanalytic ideas are entirely falsifiable (I also would be willing to bet that a even a cursory effort at experimentation would, in fact, falsify them; but that's another story).
One point that is undeniable is that the history of psychoanalysis has been riddled with all the features that we discussed earlier: endless claims of amazing powers and superiority over other strains of therapy, terrible schisms among sects over seemingly trivial issues, assurances that even basic proficiency in psychoanalysis could come only after many years of dedicated study, etc. In contrast, academic psychology has never had a permanent schism. It has had fads, it has had crazy ideas that held sway for too long, but eventually everyone gets welcomed back into the fold or else they die of old age. People seldom get too big for their britches for long before some young whippersnapper brings them down a peg or two.
So how does academic psychology do it? Do we just have really great personalities and thinking skills? Absolutely not. We do it the same way that all sciences manage it: we have an agreed-upon method of settling arguments. When people disagree, we perform experiments. If you think someone else is full of shit, you design an experiment to prove it. If somebody thinks that something is true, they design an experiment in such a way that they are essentially trying to prove themselves wrong. That way, when that one jerk stands up in the back during your presentation and tells you that you're full of shit because you didn't consider such and such hypothesis, you can say "Actually, we did consider that, and we tested it, and even so we didn't manage to prove our hypothesis wrong". All experiments are supposed to be designed to satisfy that jerk who thinks that you're actually just full of shit. When two scientists disagree on whether a theory is true, they should be able to come up with an experiment that they can both agree beforehand is a good way of settling their disagreement. Just as two martial artists can agree that getting into a cage and trying to kill each other is probably a good way of figuring out whose method is better for trying to kill people.
For instance, if I was talking to Dr. Bieber back in the day, I'd present my concerns with the evidence that he had gathered for his theory. Then I'd say, well, if having a weird relationship with your dad makes you gay, then maybe we can find a group of people who are more likely to have weird relationships with their dads for unrelated reasons and see if more of them are gay. Like, we could look up men who were raised by their stepdads and not their real dads. And we could limit our sample to men whose stepdads were convicted of violent offenses prior to their birth, just to get a nastier group of stepdads. And we'll compare them to men who grew up with their real dads, dads who haven't ever been convicted of a crime. We'll double check to make sure our two groups actually differ in terms of how weird their relationship with their dad is, and then if you're right, there should be a few more gay dudes in the stepdad group. If there's no difference between the two groups, then you have to agree to stop with this stupid theory.
Does it actually work? Not always, some guys are just stubborn as hell, but eventually those guys die of old age and everyone else can see you did everything you could to take their criticism seriously. The ideas that stand up despite all your best efforts to falsify them are the ones that get transmitted to the next generation of scientists.