[The following is a guest post by Ganzir and does not necessarily reflect the views of Pumpkin Person]

Let us make the following assumptions to simplify an illustrative example, although they are unnecessary for such an effect to arise.

  • Intelligence is normally distributed.
  • Our model intelligence test item has an item characteristic curve, i.e., graph with intelligence on the x-axis and probability of solving the item on the y-axis, with a break-even point, i.e., the IQ at which 50% of people solve it, of 130.
  • This item’s characteristic curve is symmetrical in that going X IQ points above 130 will increase your probability of solving the item by the same amount that going X IQ points below 130 will decrease your probability of solving the item. For example, a person with an IQ of 120 has a 35% chance of solving this item, which means that a person with an IQ of 140 has a 65% chance of solving it.
  • Although people above IQ 130 might miss this item and people below 130 might solve it, the only determinant of whether or not a person solves the item is the probability on the item characteristic curve corresponding to their IQ.

If I am told that a randomly selected person solved this item, what is my best estimate of their IQ?

130?

Wrong!

If you said that, you committed the base-rate fallacy because you forgot that there are more people below 130 than above 130.

To calculate the answer, apply Bayes’ Theorem. Look that up if you do not know what it is (that is a good habit to learn; look something up yourself before asking someone else). In this problem, it will tell you that our general formula is to calculate A = B * C / D for IQ X, where

A = The probability that someone has an IQ of X (in this case, X = 130), given that they solved the item

B = The probability that someone will solve the item, given that their IQ is X

C = The proportion of people who have an IQ of X

D = The total proportion of people who solved the item

D is the sum of, for each IQ, the proportion of people who will solve that item at that IQ multiplied by the proportion of all people with that IQ. In other words, D is the sum of B * C calculated for every possible value of X. Technically this would be an integral, but we could treat it as a sum by doing the calculation for each IQ point, or each 5-point IQ range, or whatever division of the intelligence spectrum.

In a sentence, the formula A = B * C / D means that the probability that someone who solves the item has an IQ of X is equal to the proportion of people with an IQ of X who solved the item divided by the total proportion of people who solved the item. If that does not make sense, please contemplate it until it does. Venn diagrams might help.

Once you have calculated B * C / D, find the maximum value of A. That is your best guess of the person’s IQ.

Why is this important? Because if you have an item for which the break-even point is far from the average and for which the item characteristic curve is (informally speaking) not close to flat around the break-even point, then the maximum value of A will be closer to the center of the IQ distribution than the break-even point is. If that sentence was a bit much to digest, someone who scores really high on a test, or solves a really hard item, probably has a lower IQ than almost everyone would think!

Allow me to demonstrate on our example item, with the help of <a href=”https://www.iqcomparisonsite.com/IQtable.aspx“>this chart</a>. A very, very rough estimate of D is the sum of the following:

  • Proportion of people with an IQ of 120 * Proportion of people with IQ 120 who solve the item
  • Proportion of people with an IQ of 130 * Proportion of people with IQ 130 who solve the item
  • Proportion of people with an IQ of 140 * Proportion of people with IQ 140 who solve the item

I calculated the proportion of people at each IQ level by, with numbers from the table linked above, subtracting the proportion of people at IQ X from the proportion at IQ X+1. Now we plug in:

  • 0.0105 * 0.35 = 0.003675
  • 0.0034 * 0.50 = 0.0017
  • 0.0008 * 0.65 = 0.00052

(Please do not bitch to me about significant figures. There is no need to bother with them here, and to how many decimal points I calculated the proportion of people at each IQ is arbitrary and irrelevant.)

What do we learn from this? If everybody with an IQ of 120, 130, or 140 attempted this item, and one of them were randomly selected, the probability that they have an IQ of 120 is 0.003675/(0.003675+0.0017+0.00052), which is about 62%. Even though the item “nominally” discriminates at 130, the majority of people who pass it have an IQ of 120!

This is a contrived example calculated with very generous assumptions, but my point, which stands even in the real world of non-spherical cows, is that a person who does one thing that seems to indicate exceptional smartness is probably not so exceptionally smart. The “Ganzir Effect” applies to reality as well as intelligence tests, which should be understood in the context that an intelligence test is only worthwhile insofar as it predicts real-life performance.

A concrete example of the Ganzir effect

I can give you a concrete example of this effect from the Mega Test norms: 

The fourth column gives the percentage of testees in each six-point range who solved problem 36 (three interpenetrating cubes), the hardest problem on the Mega Test. Only the range 43-48, where 8 out of 13 candidates solved this item, exceeds its break-even point. However, from the third column, we find that 87 people solved it. Even though candidates in the ranges 37-42 and 31-36 were much less likely to solve the problem, 25 people in 37-42 did and 23 people in 31-36 did. If all I know is that someone solved the three cubes problem, it is almost three times as likely that they scored in the 31-36 range, i.e., high Triple Nine to low Prometheus, than in the 43-48 range.