A Crash Course in Probability: Primer for Discrimination Discussion

I’ve got some ideas for a few articles I want to write, mostly about the topic of discrimination.  But I’ve realized that in order to understand what I’m trying to say about these topics, you’re going to need to know at least a little bit of probability and statistics.  This is because a lot of discriminatory ideas rely on very flawed thinking about the nature of statistics.  So in order to address these flaws, I am going to have to talk about the statistics behind the flaws, and that will require you guys to know enough statistics to get by.

So let me start by saying that the significant majority of people are absolutely terrible at working with statistics.  Part of the problem is that people just don’t like math, and part of the problem is the cognitive biases that enter into actual implementations in real-world situations, but another part of the problem is that the education system just doesn’t teach you what you need to know.

For example, most intro-level probability courses (which are all your average person ever takes) will teach you about the mean, median, and mode.  The mean of a distribution is what we more commonly refer to as the average.  The mode is the most frequently-occurring result, and the median is the result that lies right in the middle of the distribution.  But if I add 14 plus the rolls of two 6-sided dice (abbreviated as 2d6 + 14) and compare this to adding the rolls of two 20-sided dice (abbreviated as 2d20), I will find that the two distributions have the same mean, median, and mode (all of which are 21).  So your average person has no idea how to mathematically describe the difference between 2d6 + 14 and 2d20 (though the average Dungeons and Dragons player might).  The statistical concept required to differentiate the two actions just isn’t taught at an introductory level.

Furthermore, when you get into real statistics the median and the mode are mostly ignored.  The mean (which, despite the terminology introduced in your high-school classes is actually referred to as the average by everyone I’ve ever worked with) is used all the time, but nobody cares about the median and the mode.  (Though now that I think about it, this may be because with a Gaussian or Normal distribution, mean = median = mode, and there’s no use giving someone the same information three times.)

What people do care about, in addition to the mean, is a number called the standard deviation.  This is a number that represents the spread of outcomes in a distribution.  While 2d6 + 14 and 2d20 have the same mean, median, and mode, they have very different standard deviations.  This is because the possible outcomes of 2d20 range from 2 to 40, while the possible outcomes for 2d6 + 14 range only from 16 to 26.  So while the two distributions have the same average, the same most likely outcome, and the same middle outcome, the possible outcomes in 2d6 + 14 are grouped much more closely together, while the possible outcomes in 2d20 are more spread out.

The standard deviation is more difficult to calculate than the mean, median, and mode (which is probably why it’s left out of introductory courses).  Mathematically, it’s the difference between the average of “outcome squared” and the square of “average outcome.”  But you don’t actually need to be able to calculate standard deviations to understand what I want to say about discrimination.  All you need to know is what a standard deviation represents, and that’s the spread of possibilities.  The larger the standard deviation, the larger the spread.

This is easier to see graphically, so here you go: Two graphs representing the distribution of 2d6 + 14 versus 2d20.

Like I said, the mean, median, and mode of these two distributions is exactly the same (all 21).  Yet we can plainly see that the distribution for 2d20 is much more “spread out.”  The standard deviation is the mathematical concept that quantifies this “spread-out-ness.” 2d20 has a much larger standard deviation than 2d6 + 14 because its possible values are not nearly as clumped together.

Now I’m going to show you something that is absolutely crucial to my thoughts on discrimination.  Here’s a picture of what happens when I look not at 2d6 + 14, but at 2d6 + 18.

As we can see here, the average of 2d6 + 18 is much higher than the average of 2d20.  So if high dice rolls are good, then we can expect your average 2d6 + 18 guy to outperform your average 2d20 guy.  And yet, at the same time, we can also expect that there will be plenty 2d20 guys at the high end of the spectrum.  In fact, the people who perform the best are all 2d20 guys, even though the 2d20 guys have a lower average.  This is because they have a much larger spread.

What does this have to do with discrimination?  EVERYTHING

Because people never learn about standard deviation, and never come up with the idea on their own, reporting of statistics that reach the layman are handled in one of two ways.  Either the average, and only the average, is reported (like I said, median and mode rarely come up in research), or a standard deviation is reported but ignored (because the layman has no idea what it is).  But then the layman uses the only piece of information he has/understands and performs a false extrapolation by imagining two distributions with the same spread but different means.

In other words, when a study shows that 2d6 + 18 guys outperform 2d20 guys on average, the layman has the following picture in his head: 

Your average person doesn’t consider the spread of the data.  They think of the average, the only information they get and possibly the only information they understand, and they try to use it to extrapolate the entire distribution.  And because the average is all they’ve got, their extrapolation is almost always tightly grouped around that average.  And they end up with a mental picture like the one above, which in turn leads them to believe that virtually every 2d20 guy is going to do significantly worse that the virtually every 2d6 + 18 guy.

To bring this back to the everyday world, let’s consider an actual case of discrimination.  I’ll take as my example the topic of women in mathematics.  There are basically two positions that people hold with regards to this topic.

People in the first position proceed as follows: First, they believe, for whatever reason, that males are on average better than females when it comes to mathematics.  Then they form a mental picture about the distribution of mathematical capability in males and females much like the erroneous picture for the 2d20 guys and the 2d6 + 18 guys.  From this erroneous picture, they conclude that nearly all women are virtually incapable of competing with men in terms of mathematical capability.  Thus, we shouldn’t bother trying to push women into mathematics (and perhaps we shouldn’t even let them try).

The second route tends to be a response to the first.  People see the conclusion “women can’t be mathematicians” and are uncomfortable with this conclusion.  In order to reject it, the second route people challenge the premise.  They reject the idea that men are, on average, better than women when it comes to mathematics.  And for the most part, they seem to reject this because they don’t believe the conclusion that we shouldn’t let/expect women to be mathematicians.

Even though members of the second group are very interested in arguing against the conclusion that women don’t make good mathematicians, they never seem to stumble on the simple fact that “on average, men are better at math” does not lead us inevitably to the conclusion that women can’t compete at the top tier.  They seem to agree that if the average for women is lower, then the distributions look like the layman’s picture, and thus the women stand no chance at actually becoming competent mathematicians.  And since the pro-equality people never notice the flaw in the argument, they’re forced to denounce the premise.

This leads us to people who insist that there’s no biological factor that makes men better at math.  If you do a study and find that men perform better at math then women, they will insist that this is caused by social pressure, not biological factors.  So while it’s obvious that there are many physical differences between men and women, including chemicals that definitely make contact with the brain, we have many people insisting that there is no biological factor involved in any of our perceived mental differences.  They will insist that all the data we have is tainted by social influences, and then go on to insist that even though we have no untainted data, men and women are definitely equally capable.  And they do this because they think it’s the only way to avoid conclusions like “women can’t be mathematicians.”

And this leaves us with the problem of a false dichotomy.  Everyone believes that there are only two possible ways to resolve the issue.  Either men are better than women on average, and thus we shouldn’t bother letting women try, or men and women are equal on average, and thus we should do what we can to make sure we have as many female mathematicians as male (which would indicate that we have ended the social discrimination).  And nobody ever seems to consider the other logically consistent options like men are better than women on average and yet there are still plenty of capable women, or even men are better than women on average and yet most of the best are women.

This is the first step in understanding my thoughts about discrimination and equality.  A simple report of average capability is not enough information.  We can’t use that data to conclude decisively that one or more groups are doomed to fail.  In addition to the averages of the distributions, we also need to know their spreads.  Only when we know both the average innate mathematical capability of men and women and the spreads of their respective distributions can we actually make a reasonable guess about the expected gender distribution of mathematicians if we end discrimination.