What is Simpsons Paradox?
Simpsons Paradox is a statistical phenomenon that occurs when you combine subgroups into one group. The process of aggregating data can cause the apparent direction and strength of the relationship between two variables to change.
Welcome to the intriguing world of Simpson’s Paradox, a statistical phenomenon highlighting how misleading aggregated data can be without a deeper examination of the underlying segments. Imagine examining a set of data that tells one story when viewed as a whole, but a completely different narrative emerges when you break it down into subgroups. This paradox challenges our intuition and underscores the critical importance of a nuanced approach to data analysis.
In this post, we will explore the concept of Simpson’s Paradox through real-life examples, including the famous UC Berkeley gender admissions case and recent COVID-19 vaccination data, demonstrating how essential it is to consider all variables before drawing conclusions.
Learn more about Data Aggregation: Strengths & Weaknesses.
Why Does Simpson’s Paradox Occur?
Simpson’s Paradox occurs because a third variable can affect the relationship between a pair of variables. Statisticians refer to this type of third variable as a confounder or confounding variable. To understand the correct relationship between two variables, you must factor in the influence of confounders.
Simpson’s Paradox occurs when the process of aggregating data excludes confounding variables. Usually, this happens unintentionally. The researchers might not realize the consequences of their actions.
Simpson’s Paradox is essentially the same concept as omitted variable bias in regression analysis, except that it is specific to cases where you combine data and ignore subgroup information. Learn more about Confounding Variables and Omitted Variable Bias.
Graphical Example
Let’s look at Simpson’s paradox graphically before returning to the admissions example. The data below show one group that seems to have a negative correlation between the X and Y variable. As X increases, Y tends to decrease.
Now we’ll factor in the subgroups. Below, it’s easy to see how there is actually a positive relationship between X and Y after including the subgroups. Combining the data and ignoring the subgroups obscured that relationship. That’s how Simpson’s paradox distorts the results!
In the context of Simpson’s Paradox, the subgroups capture the confounding variable. By aggregating the data, you are effectively removing the confounder from the analysis, and it distorts the results.
Explaining the Admissions Example
If you want to compare the admissions rates for men and women at UC Berkeley, it seems logical that you can just look at the overall rates. I show the actual acceptance rates below:
| Men | Women |
| 45% | 30% |
It sure appears that Berkeley prefers men and disadvantages women. However, there is more to the story thanks to Simpson’s Paradox!
Unfortunately, aggregating the data from all departments removes departmental differences from the analysis. It turns out that some departments have much lower acceptance rates than others. They’re more selective. The following two factors create the misleading, unbalanced acceptance rates in the previous table:
- Women tended to apply for the harder departments, lowering their overall acceptance rate.
- Men were inclined to apply for the easier departments, boosting their rates.
To determine whether the selection process favors men, we need to assess the data at the departmental level and compare acceptance rates within each department. This method counters Simpson’s paradox by accounting for each department’s acceptance rate, allowing for valid comparisons.
Let’s look at the data! There are 85 departments. The table shows the largest six.
Comparing the rates within departments paints a different picture. Women have a slight advantage over men in most departments.
The subgroup analysis accounts for the confounding variable of the varying admission rates.
Simpson’s Paradox Example
Simpson’s paradox occurs in numerous contexts. More recently, analysts observed it in media reports of COVID deaths among the vaccinated than the unvaccinated. In September 2022, 12,593 COVID deaths occurred in the United States. Of those, 39% were unvaccinated, while 61% were vaccinated. What?!
It turns out that the relationship between being vaccinated and having a higher percentage of deaths is a fiction created by aggregating data and tossing out relevant information—Simpson’s Paradox.
In the United States, the COVID vaccinated population tends to be older and has more risk factors. This group naturally tends to have worse COVID outcomes. However, when you adjust for age and other risk factors, the CDC finds that COVID vaccinated and boosted individuals have an 18.6 times lower risk of dying from COVID. The vaccines are working!
To wrap up, Simpson’s Paradox occurs when you fail to account for relevant information when analyzing data. This paradox occurs when you aggregate data and lose essential details in the process. With the enrollment example, you get opposite results when you look at the overall acceptance rates by gender but don’t consider the varying departmental acceptance rates. For the COVID example, you get confusing results when you assess the overall death COVID percentages by vaccination status without accounting for underlying risk factors.
A Caution about Simpson’s Paradox
It shouldn’t be surprising that discounting relevant factors will distort your results. But it is shocking how easily it can happen if you don’t watch for it!
Simpson’s Paradox is a powerful reminder of the complexities inherent in data analysis. As we’ve seen through examples from university admissions and public health, failing to account for subgroup variations can lead to conclusions that are not only incorrect but potentially misleading. This paradox teaches us the importance of vigilance and precision in statistical analysis, urging researchers to delve deeper into the data rather than accepting surface-level insights.
By understanding and acknowledging the impact of confounding variables and the dangers of data aggregation, we can prevent misinterpretations and ensure that our analyses truly reflect the reality they intend to capture. Be sure to do the following:
- Always question the data.
- Look beyond the aggregates.
- Strive for clarity and accuracy in every dataset you encounter.
By doing this, you can ensure that your study results accurately reflect the underlying trends and patterns in the data.
References
Sex Bias in Graduate Admissions: Data from Berkeley
Why Do Vaccinated People Represent Most COVID-19 Deaths Right Now?
CDC COVID Data Tracker: Rates of COVID-19 Cases and Deaths by Vaccination Status
Just a note regarding the Berkeley Admissions example from (Wagner, 1982) -> “This was not a complete instance of Simpson’s paradox because, when the data were disaggregated, the overall tendency toward a higher acceptance rate for male applicants was not reversed in each academic department.” https://www.jstor.org/stable/2684093
Hi Clari,
The definition of Simpson’s paradox is probably a bit imprecise about how completely the original effect needs to be reversed. All I know is that the Berkeley case is considered a classic example of Simpson’s Paradox. By breaking down the data, you get a very different perspective, changing from strongly favoring males to more balanced.