With the arrival of Fall in the Northern hemisphere, it’s flu season again.
Do you debate getting a flu shot every year? I do get flu shots every year. I realize that they’re not perfect, but I figure they’re a low-cost way to reduce my chances of a crummy week suffering from the flu.
The media report that flu shots have an effectiveness of approximately 68%. But what does that mean exactly? What is the absolute reduction in risk? Are there long-term benefits?
In this blog post, I explore the effectiveness of flu shots from a statistical viewpoint. We’ll statistically analyze the data ourselves to go beyond the simplified accounts that the media presents. I’ll also model the long-term outcomes you can expect with regular flu vaccinations. By the time you finish this post, you’ll have a crystal clear picture of flu shot effectiveness. Some of the results surprised me!
We’ll Assess Randomized Controlled Trials (RCTs)
My background is in scientific research. And, I love numbers—they’re how I understand the world.
Whenever I want to understand a topic, I combine these two things—numbers and scientific research.
To understand flu shot effectiveness, we’ll look at scientific, peer-reviewed articles. By assessing their methodology, seeing the actual data, and how they draw their conclusions, we’ll be able to comprehend flu shot effectiveness at a much deeper level.
We’re going to evaluate only double-blind, randomized controlled trials (RCTs), the gold standard. RCTs are more expensive to conduct than observational studies, but they provide the tremendous benefit of identifying causal relationships rather than mere correlation. After all, we need to determine whether flu shots cause a reduction in the risk of contracting the flu.
I’ve found two influenza vaccination RCTs for us to analyze. The CDC list these studies on their website in a reference section for healthcare professionals. We’ll take a close look at these studies and analyze their data ourselves!
Defining the Effectiveness of Flu Shots
The definition of influenza vaccine effectiveness is very precise in studies. It’s essential to understand the meaning and its context before we continue.
Flu shots contain vaccines for three or four strains of the influenza virus that scientists predict will be the most common strains in a flu season. However, there are many other viruses (other strains of flu and non-flu) that can make you sick. Some of these are flu-like illnesses that are not the flu but can make you feel like you have the flu.
Consequently, the best flu vaccination studies use a lab to identify the specific virus that makes their subjects sick. These studies count participants as being infected with the flu only when he or she catches one of the influenza strains in the vaccine. Flu shot effectiveness is the reduction in cases involving these particular strains among those who were vaccinated compared to those who were not.
Two Flu Vaccination Studies
Let’s get to the data! This is the exciting part where the rubber meets the road. The media discusses the effectiveness, but it all stems from these counts of sick people in experimental groups. These deceptively simple looking tables contain the answers to our questions. They’re deceptively simple because these studies use meticulous experimental methodologies, time consuming data collection and validation procedures, and extensive subject selection protocols to collect these data.
The Beran Study
Beran et al. study1 the 2006/2007 flu season and follows its participants from September to May. Participants in this study range from 18-64 years old. The table below shows the number of lab-confirmed flu cases for the treatment and control groups.
| Treatment | Flu count | Group size |
| Shot | 49 | 5103 |
| Placebo | 74 | 2549 |
This table represents the observations from a random sample of the population. To determine whether the observed difference between the group proportions represents an effect that exists in the population, we need to perform a hypothesis test. In the case, we’ll use the 2 Proportions test.
This test answers the question: Is the difference between the proportion of sick people in each group statistically significant?
Related post: How Hypothesis Tests Work: P-values and Significance Levels
The p-value of 0.000 indicates that the difference between the two groups is statistically significant. The Sample p column displays the proportion of flu cases in each group, which I’ll convert to percentages (0.96% vs. 2.9%). The estimate for the difference shows that the vaccinated group has 1.9% fewer cases than the placebo group. The confidence interval indicates that we can be 95% confident that the population difference is between 2.64% and 1.23%.
Related post: How to Interpret P-values Correctly
Because this study is an RCT, it’s reasonable to assume that the vaccinations cause the reduction.
The difference between the groups (1.9%) is not what we hear about in the media. They report the vaccine effectiveness, which is the relative reduction in the risk for the vaccinated group. The formula for effectiveness is:
This study reports a 66.9% vaccination effectiveness for the flu shot compared to the control group.
The Monto Study
The Monto et al. study2 evaluates the 2007-2008 flu season and follows its participants from January to April. Participants are 18-49 years old.
| Treatment | Flu count | Group size |
| Shot | 28 | 813 |
| Placebo | 35 | 325 |
Let’s perform the 2 Proportions test on these data.
Like before, the small p-value indicates that the difference between the flu shot group and the placebo group is statistically significant. For this study, the vaccinated group has 7.3% fewer cases of the flu.
Let’s calculate the effectiveness:
This study finds a 68.0% vaccine effectiveness for the flu shot.
Conclusions Up to This Point
First, isn’t it cool how we can analyze their data and draw our own conclusions? Our findings are consistent with those reported by the news media. The flu vaccine produces a statistically significant reduction in the number of influenza infections. Further, the flu shots are about 68% effective.
However, seeing the data for yourself, did you gain some new insights? I know I did. For me, I think the percentage point reduction in flu cases sounds less impressive than the 68% effectiveness. The former measures absolute risk while the later measures relative risk. Same data but different ways of presenting the results. Let’s investigate this.
For an interesting comparison, read my post about the Effectiveness of a COVID-19 Vaccination.
Relative Risk versus Absolute Risk
I don’t think the 68% effectiveness statistic is helpful because it’s the relative risk. Here’s an example to illustrate the difference between relative and absolute risk.
- Relative assessment: Your car travels half as fast as another car, but you don’t know the actual velocity of either car.
- Absolute assessment: Your car travels at 30 MPH while the other car travels at 60 MPH.
Obviously, the absolute assessment is more helpful. The same idea applies to knowing the absolute risk of catching the flu after vaccination compared to no vaccination.
Vaccine effectiveness is a relative risk
Vaccine effectiveness doesn’t inform you about the absolute risk of catching the flu for either group. Additionally, it is a particularly confusing measure of relative risk. Recall that effectiveness is calculated as the inverse of the relative risk. That’s much harder to interpret. A 67% effectiveness is equivalent to having one-third the risk of acquiring the flu if you get a flu shot (1 – 0.67 = 0.33).
I find that vaccine effectiveness is a confusing statistic. It doesn’t tell you about the absolute risk for anyone, and it’s the inverse of what you want to know!
Related post: Relative Risks: Definition, Formula and Interpretation
Group proportions of flu cases estimate the absolute risks
In the statistical output, we can use the proportion for each group to estimate the absolute risk of acquiring the flu. Then, we can subtract the two proportions to calculate the flu vaccine’s absolute reduction in risk. In the table below, I summarize this information as percentages and add in two more flu seasons from another study (Bridges et al.3):
| Flu season | Placebo | Flu Shot | Risk Reduction |
| 1997/98 | 4.4 | 2.2 | 2.2 |
| 1998/99 | 10.0 | 1.0 | 9.0 |
| 2006/07 | 2.9 | 1.0 | 1.9 |
| 2007/08 | 10.8 | 3.4 | 7.4 |
| Average | 7.0 | 1.9 | 5.1 |
The risk of catching the flu varies by season. The differences aren’t surprising because the studies assess different flu seasons, which have different viruses. Some flu seasons are worse than others because the strains are more virulent. Sometimes the strains in the vaccinations aren’t a good match for the most prevalent influenza strains.
The last row in the table averages the values across the four flu seasons. When you do not receive the flu vaccination, you have a 7.0% chance of catching the flu on average. However, when you get the shot, your risk is 1.9%. Consequently, the average reduction is 5.1%.
I don’t know about you, but the 5.1% reduction doesn’t sound nearly as remarkable as 67% effectiveness! Both statistics are calculated from the same data but provide different types of information.
Knowing that the annual average risk reduction is 5.1% is helpful. While this absolute reduction might sound small, flu shots aren’t supposed to be a one-time event. Let’s look at the long-term benefits of getting the flu shots annually. I want to see if my impression that annual flu shots will eventually spare you from a week of misery is correct!
Modelling Flu Outcomes Over Decades
Flu shots benefits are like investing year-after-year. The cumulative effect magnifies the differences over time. I’m going to model flu outcomes over decades in the same manner that financial planners illustrate the differences between different courses of action.
Investment return rates fluctuate over the years just like the influenza rates. Financial planners use reasonable long-term averages to provide an estimate of different outcomes. Similarly, I’ll use the average infection rate for those who are vaccinated and those who are not. This approach produces comparative results for annual flu vaccinations versus no flu vaccinations over decades.
To model this statistically, I’ll use probability distribution plots. Because the data are binary (infected or not infected), I’ll use discrete probability distributions that are designed for binary data—the binomial and geometric distributions. For the graphs, the assumptions are that the average infection rate for the:
- Unvaccinated is 7.0% annually.
- Vaccinated is 1.9% annually.
We’ll learn what effect that 5.1% difference has over decades. I’ll compare two different scenarios—getting the flu shot every year versus never getting the influenza vaccination. Using our estimates, I’ll answer two questions:
- How long until my first case of the flu on average?
- How many times will I get the flu?
In both plots, the left panel displays the no flu shot scenario while the right panel shows the annual influenza vaccine scenario.
How long until my first case of the flu on average?
I’ll use the geometric distribution to model the probability of first catching the flu. Each bar in the graph indicates the likelihood of first getting the flu in a specific year. The chances for any given year is small. More importantly, I’ve shaded the graphs to indicate the number of years until the cumulative probability reaches 50%. The graph stops at 65 years because flu shot effectiveness decreases around that age.
In general, the chart shows how your probability of first contracting the flu is much higher early on when you aren’t vaccinated (left) than when you are regularly vaccinated.
The left panel indicates that without flu shots, you have a 50/50 chance of getting the flu in 11 years. On the other hand, the right panel displays the probabilities when you are vaccinated annually. You don’t reach a 50% cumulative probability until 38 years. Furthermore, when you never get the flu shot, you have only a 6.8% chance of not catching the flu in 38 years!
Learn more about the Geometric Distribution.
How often will I catch the flu?
Let’s statistically model the number of times you can expect to catch the flu in 20 years. For this graph, I’ll use the binomial distribution. Each bar indicates the likelihood of catching the flu the specified number of times. I’ve shaded the bars to represent the cumulative probability of catching the flu at least twice in 20 years.
There’s a big difference that jumps out at you! The largest bar on the graph is the one that represents zero cases of the flu in 20 years when you get flu shots. When you’re vaccinated annually, you have a 68% chance of not catching the flu within 20 years! Conversely, if you don’t get flu shots, you have only a 23% of escaping the flu entirely.
In the left panel, the distribution spreads out much further than in the right panel. Without vaccinations, you have a 41% chance of getting the flu at least twice in 20 years compared to 5% with annual vaccinations. Some unlucky unvaccinated folks will get the flu four or five times in that time span!
Learn more about the Binomial Distribution.
Closing Thoughts about Flu Shots
By analyzing the data ourselves, we found that flu shots reduce the risk of catching the flu. However, the reduction in absolute terms is small. In fact, without flu shots, you still have a low probability of getting the flu in any one flu season. This low risk explains why I hear from people who don’t get flu shots and they haven’t had the flu for a while.
However, when you model the long-term outcomes of being vaccinated against the flu regularly, it supports my hypothesis that flu vaccinations will save you from at least one week of misery, and probably more! Regular flu shots both lengthen your expected time until first catching the flu and reduce the number of times you can expect the flu virus make you sick within a 20-year timeframe.
There are some additional complications when it comes to the flu. Recall that other viruses are in circulation. These can be other strains of flu and non-flu viruses that cause flu-like symptoms. In this post, the probabilities of infection only refer to catching a strain of flu that is in that year’s flu vaccinations. It’s possible to be vaccinated annually and never catch a flu strain that you were vaccinated against but yet experience flu-like symptoms from some other virus or flu strain. Nevertheless, you are still receiving the benefits discussed above.
So, what am I going to do about shots?
Of course, I’m not a medical doctor, and you should talk to yours for specific advice. Even though the benefits might seem small for a single year, I am going to continue to get my flu shots annually. I’m sure that they will prevent misery down the road! Additionally, it reduces the risk of me passing the flu to my 99-year-old grandmother (you can read about her in this blog post), which would be very dangerous for her.
References
1. Beran J, Vesikari T, Wertzova V, Karvonen A, Honegr K, Lindblad N, Van Belle P, Peeters M, Innis BL, Devaster JM. Efficacy of inactivated split-virus influenza vaccine against culture-confirmed influenza in healthy adults: a prospective, randomized, placebo-controlled trial. J Infect Dis 2009;200(12):1861-9.
2. Monto AS, Ohmit SE, Petrie JG, Johnson E, Truscon R, Teich E, Rotthoff J, Boulton M, Victor JC. Comparative efficacy of inactivated and live attenuated influenza vaccines. N Engl J Med. 2009;361(13):1260-7.
3. Bridges CB, Thompson WW, Meltzer MI, Reeve GR, Talamonti WJ, Cox NJ, Lilac HA, Hall H, Klimov A, Fukuda K. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial. JAMA. 2000;284(13):1655-63.
Thanks for the reply Jim. Since you mentioned Grandma, it is worth pointing out that flu shot protection in the elderly is even significantly less than the average 5.1%. This is documented both in vaccine information and on the CDC website. So the big question comes back: Do flu shots help the overall immune health of our elderly or worsen it, and what is the risk-benefit ratio?
You mentioned that you have had no adverse reactions to the shot. Those are not the kinds that worry me as much as the more subtle ones that might develop over the years of our lives, such as the onset of Alzheimer’s, asthma, Crohns, Colitis, MS, and other auto-immune diseases, all of which are on the rise in America along with the increased frequency in vaccinations. While a correlation doesn’t equal a cause, it should at least be cause for some concern. The only side effects we know about for sure our the ones that are reported and immediately connected to the shot. The uncertainty of this alone, far outweighs any perceived 5.1% reduction in risk of getting a particular strain of flu.
Lastly, we can be certain that even the 5.1% absolute risk reduction is exaggerated due to the fact that those who have had a flu shot are far less likely to have it diagnosed by a doctor due to the fact they believed the vaccine protected them.
Like you said, do what you think is best for you. I think my immune system and overall health are much better off without them, and Grandma is better off too.
Hi Rob,
Just a clarification. I meant that if I get a flu shot and it lowers the risk that I’ll be infected, it thereby lowers the risk that I’ll pass it on to my Grandmother.
The most common side effects from the influenza shot are soreness, redness, and tenderness or swelling where the shot was given. Low-grade fever, headache and muscle aches might also occur. The most severe proven problem is Guillain-Barré syndrome, which affects no more than 1 or 2 cases per million people vaccinated. This is much lower than the risk of severe complications from flu, which can be prevented by flu vaccine. And, getting the flu itself is a risk factor for GBS.
As for the 5.1%, in these medical studies, everything is verified in a lab with tests. It’s not a self-reporting situation.
There are risks associated with getting the flu and there are risks associated with getting the flu shot. Not all of those risks are known. Such is life. My hope with this blog post is that by presenting information clearly, people can make a decision that is better for themselves. Ultimately we each have to decide which is best for us and our loved ones.
Jim,
I love the fact you illustrate absolute statistics versus relative statistics. Neither the vaccine inserts nor the CDC tell us this. However, after all this research, good information, and understanding of the facts, I am baffled at your decision to still want flu shots. That 68% reduction is cumulative only from a statistical perspective. If you have your arm injected with a flu vaccine annually, you have reduced your odds of getting the flu just one time over a 20 year span by 68%. You will still continue on average to have only an absolute risk reduction of 5.1% each year. And you will still be just as likely to get other types of flu which there are no vaccines for. That doesn’t get me excited to have my intramuscular system invaded with a foreign substance every year.
On top of this, you have not considered the risk – benefit ratio. Flu shots have very real risks which are documented in the vaccine inserts. These risks are not only cumulative statistically, but perhaps physically as well, in that we are repeatedly adding adjuvants like aluminum and mercury into our body through our muscular system. This is not the same as ingesting foreign substances in our food. Remember, Vaccine makers can only publish the documented risks. No study to date has ever studied these vaccines for long-term safety. Who would want to fund such a study? There is no money in it for anybody. There is no telling what kind of neurological and auto-immune problems we could be giving ourselves by regular vaccination – especially over a 20-year period.
Lastly, why would you trust getting a shot from a medical industry that doesn’t provide the full truth presented in the valuable, absolute statistical efficacy that you provided?
Hi Rob, thanks for taking the time write the thoughtful comment.
I certainly agree that flu shots are not perfect. And, I do have concerns about some for-profit aspects of the medical industry.
Here’s why I personally think flu shots are still worthwhile getting for myself. Of course, your mileage may vary!
The absolute risk reduction is 5.1% a year. However, as I illustrate with the graphs later in the post, this translates into noticeable gains over decades. For example, if you get flu shots regularly, you’ll typically get the first case of the flu in 38 years rather than 11 years. And, with annual flu shots, you have about a 5% chance of catching the flu at least twice in 20 years whereas, with no flu shots, you have a 41% chance of catching the flu at least twice. Those benefits are enough for me. Then, throw in the fact that I regularly visit my nearly 100 year old Grandmother. Catching the flu could be disastrous for her. I’m happy to get the flu shot to help reduce the risk of me passing the flu to her.
As for the risks. I’ve personally never have had a negative reaction to the many flu shots I’ve received. However, some people are allergic to the specific ingredients and do need to be careful. The known risks associated with long-term flu vaccinations are rare–and flu shots have been around since 1938!
Conversely, there are known risks and complications associated with catching the flu– such as pneumonia, bronchitis, inflammation of the heart (myocarditis), brain (encephalitis) or muscle (myositis, rhabdomyolysis) tissues, and multi-organ failure (for example, respiratory and kidney failure), worsening of chronic heart disease, etc.
Everyone should evaluate the benefits and risks for themselves. Personally, I feel that the benefits outweigh the risks–particularly when I think about my Grandma!
The cause(s) of GBS has not been definitely proven or identified. Furthermore, you know what else has been linked to GBS? The Flu itself. Nothing is risk free, we can only approach situations like these through a lens of cost/benefit. The chances of you (or anyone) dying from the flu or complications as a result thereof, are significantly higher than getting GBS. That might mean nothing to you, and that’s your business but from a cost/benefit perspective getting the shot offers the greatest benefit with the least amount of associated risk.
but flu infection also can cause severe complication like pneumonia
Love the statistics showing absolute risk. However, I disagree with your decision to continue to get an annual flu shot. You are neglecting to consider risk to benefit ratio. The flu shot is NOT risk free and is the number one vaccine associated with Guillain-Barré syndrome (GBS). The National Childhood Vaccine Injury Act (NCVIA) of 1986 Prevents lawsuits/claims against vaccine injuries. The BILLION dollar pharmaceutical industry has a cash cow with vaccines because they bear no culpability or financial liability for any adverse reactions. …from a concerned pharmacist.
This post is great as usual!! If you have time, maybe you could do a post about geometric distribution and link to it here:) . It makes so much more sense when i learned that geometric distributions are often used when the variable of interest is the number of trials required to obtain the first “success,” etc.
Thanks, Jessica! And, that’s a great idea. I’ll write about that distribution and others in the upcoming month.
Hi Jessica,
I wanted to let you know that I’ve written a post about making the most out of your binary data, which includes the various probability distributions such as the geometric distribution (amongst several others).
This was really great information. Thanks for plowing through the math and writing it down in a logical form. Thanks!
Thanks Andrew!
I don’t understand how you can calculate your chances of getting the flu if you get a flu shot every year for 20 years.
It seems you have only calculated the likelihood you will contract a flu strain contained within a vaccine.
But we do not know what percentage of total flu strains are included in each year’s vaccine, nor the prevalence of each of those strains excluded.
Hi Matt,
What I’ve done is to use the estimates from the studies. You can estimate the probability of catching the flu when you have had the vaccination and when you haven’t. These studies track what virus is making someone sick. So, they count only those who get sick for one of the strains included in the shot. As I mentioned, these probabilities are for only the specific 3 or 4 strains in the vaccines. It wouldn’t make sense to see how effective the vaccines are for strains that are not in the vaccine. We know the average absolute risk reduction and use that to project the effects over 20 years. As I mention, you might very well get sick with other flu strains or entirely different types of viruses. In the end, we are estimating the benefits associated with the shots for the specific strains that they include.
I hope this helps clarifies matters,
Jim