Lately I saw this video - by accident - because the title was a completely different one and in the video the speaker - Malcolm Gladwell - mentions that the original topic was planned differently.
He talks about something that he calls the "mismatch problem".
Basically in a nutshell he explains that the way we measure things and the parameters we think to be important for a particular job often do have nothing to do with what really matters. The way we measure performance of people through tests, graduations, certificates often does not bring the expected results. Instead the number of people that would do a good job is reduced by filtering all those out with bad marks and such who are missing particular graduations. Malcolm shows a few examples where even the opposite can be seen: That those people who have the worst testing results and graduations are the best performers.
He recommends not to rely on standardized testing routines and instead prefer looking on how people are making it on the job in the real world.
I already had similar thoughts many times. In particular I remember support calls where people emphasized their academic titles when calling and had the worst understanding of how the software works. My experience is that an academic title is no guarantee that the person behind is an intelligent and productive person. On the other hand there are persons without an academic title who are really smart, creative and good performers. Don't get me wrong, there are very good academics also. I only want to point out, that a title or a certificate is no guarantee for quality.
I think that in many, many cases our analysis, testing and labeling of things, people or events happening are affected by the mismatch problem in the way that the interpretation we are attaching to our impressions is hardly ever fitting the reality. Even history of science is full of mistakes and misinterpretation of results.
Sometimes it is better to rely on the gut feeling instead of doing scientific tests, polls and statistics.
Related posts: In the present moment, Detect the weak point, Crappy guesswork.