Malcolm Gladwell's Mismatch Problem

Malcom Gladwell has a thought-provoking video up, this time on something he calls the mismatch problem. In a nutshell, we think we know how to measure talent. The problem is that we have no idea what we're doing. Our measurements appear to bear no relation to future performance. In Gladwell's words, it's a misjudgment that occurs when the criteria we use to assess someone's ability to do a job is radically out of step with the demands of the job itself.

Since S. recently heard about a consulting firm that gives a battery of tests to help you find a career that matches your strengths and weaknesses, and since I've taken an interest in something called the Signaling Theory of Education, it's worth rehearsing some of Gladwell's points.

Every year, for a week or weekend, before professional sports teams draft the incoming rookie class, they hold what are called combines, a sort of mini-camp where prospective draftees go through a dog and pony show for all the scouts. The combine's purpose is let scouts collect objective data about who these kids are and what they can do. Millions and millions of dollars are at stake after all, so naturally teams want some reliable way to predict how good of an investment they're making in whomever they draft. Will this guy help us win? Is he worth it? In the NBA combine, these kids jump, they run, they lift weights, they take IQ tests, run drills--anything and everything to predict how well these kids will play.

But do you know who D.J Strawberry is? I certainly don't. But last year his scores at the NBA combine were the highest of any incoming rookie. And yet he was awful. Strawberry averaged just two points a game during the entire year for the Phoenix Suns. All the top scorers at the combine turned out to be awful. Some didn't even play in the NBA. But what about Kevin Durant, who won the rookie of the year award? He ranked 78th at the combine. In fact, the top five draft picks all performed poorly at the combine. It's a remarkable waste of time--the combine and all of its tests are poor predictors for how well anyone plays basketball.

Or consider the Wonderlic test, an intelligence test given to NFL rookies in their combine. Being a quarterback in the NFL requires great cognitive demands and sophisticated decision making. Thousands of plays have to be memorized. Quarterbacks watch 100 hours of video every week to study the offense they plan to run and how the defense of the opposing team will react. So again, you might think that measuring intelligence with the Wonderlic might be important. But it turns out that among the 7 worst Wonderlic scorers in history are two of the best to ever play the game--Terry Bradshaw and Dan Marino. On the other hand, none of the seven best come close--Drew Henson, Eli Manning, Tony Romo...the list is laughable. So if in an industry where millions of dollars are at stake, where scouts are paid hundreds of thousands to measure talent in any quantifiable way, if in these multi-billion dollar industries our methods to predict productivity fail, then how about in other areas of work where measurements of productivity are even more vague and elusive?

When everything we use to assess talent has no correlation, or worse, a negative correlation with our actual performance on the job, it's time we start rethinking some of these institutions. Consider more from Gladwell:

How do we ensure which teachers we hire are the best? Currenlty teachers are required to have a BA, teaching training, license from the state, academic work associated with your specialty. Does this ensure a high quality teacher? Not at all. Do any of these requirements correlate with increasing student performance? Not at all. So it's the Combine all over again.

What about lawyers? Surely lawyers would know what correlates with what makes a good lawyer. But of course they don't. The University of Michigan has an extremely generous affirmative action program, meaning minority applicants with lower GPAs, test scores and so on are accepted more frequently than white candidates with those same scores. This then set up a natural experiment: measure the success across time of all UM Law graduates--30 years worth--and compare the success of those graduates who tested well with those minority graduates who did not. As it happens, on any measure of success, Gladwell claims there was no correlation (according to UM's research). So, yet again, we have a mismatch problem. The criteria, which we think are associated with being a good lawyer--high LSATs, high GPA--have no bearing on how good of a lawyer you end up becoming.

So why do mismatch problems occur? Gladwell says it's a madness for imposing certainty. A deep-seated need for clear and reliable statics, a hard-wired impulse for rational plan making. Also, he says, the complexity of jobs has increased. The cognitive demands required to succeed have multiplied and we can't track them. So the world has changed, but the way we hire people hasn't.

But there's only one reliable way to measure productivity: to wait until someone's on the job.