With the arrival of Fall in the Northern hemisphere, it’s flu season again.

Do you debate getting a flu shot every year? I do get flu shots every year. I realize that they’re not perfect, but I figure they’re a low-cost way to reduce my chances of a crummy week suffering from the flu.

The media report that flu shots have an effectiveness of approximately 68%. But, what does that mean exactly? What is the absolute reduction in risk? Are there long-term benefits?

In this blog post, I explore the effectiveness of flu shots from a statistical viewpoint. We’ll statistically analyze the data ourselves so we can go beyond the simplified accounts that the media presents. I’ll also model the long-term outcomes you can expect with regular flu vaccinations. By the time you finish this post, you’ll have a crystal clear picture of flu shot effectiveness. Some of the results surprised me!

## We’ll Assess Randomized Controlled Trials (RCTs)

My background is in scientific research. And, I love numbers—they’re how I understand the world.

Whenever I want to understand a topic, I combine these two things—numbers and scientific research.

To understand flu shot effectiveness, we’ll look at scientific, peer-reviewed articles. By assessing their methodology, seeing the actual data, and how they draw their conclusions, we’ll be able to comprehend flu shot effectiveness at a much deeper level.

We’re going to evaluate only double-blind, randomized controlled trials (RCTs), the gold standard. RCTs are more expensive to conduct than observational studies, but they provide the tremendous benefit of identifying causal relationships rather than merely correlation. After all, we need to determine whether flu shots *cause* a reduction in the risk of contracting the flu.

I’ve found two influenza vaccination RCTs for us to analyze. The CDC list these studies on their website in a reference section for healthcare professionals. We’ll take a close look at these studies and analyze their data ourselves!

## Defining the Effectiveness of Flu Shots

The definition of influenza vaccine effectiveness is very specific in studies. It’s important to understand the meaning and its context before we continue.

Flu shots contain vaccine for three or four strains of the influenza virus that scientists predict will be the most common strains in a flu season. However, there are many other viruses (other strains of flu and non-flu) that can make you sick. Some of these are flu-*like* illnesses that are not the flu but can make you feel like you have the flu.

Consequently, the best flu vaccination studies use a lab to identify the specific virus that makes their subjects sick. These studies count participants as being sick with the flu only when he or she catches one of the influenza strains in the vaccine. Flu shot effectiveness is the reduction in cases involving these particular strains among those who were vaccinated compared to those who were not.

## Two Flu Vaccination Studies

Let’s get to the data! This is the exciting part where the rubber meets the road. The media discusses the effectiveness, but it all stems from these counts of sick people in experimental groups. These deceptively simple looking tables contain the answers to our questions. They’re deceptively simple because these studies use very careful experimental methodologies, time consuming data collection and validation procedures, and extensive subject selection protocols to collect these data.

### The Beran Study

The Beran et al. study^{1} analyzes the 2006/2007 flu season and follows its participants from September to May. Participants in this study range from 18-64 years old. The table below shows the number of lab-confirmed flu cases for the treatment and control groups.

Treatment | Flu count | Group size |

Shot | 49 | 5103 |

Placebo | 74 | 2549 |

This table represents the observations from a random sample of the population. To determine whether the observed difference between the group proportions represents an effect that exists in the population, we need to perform a hypothesis test. In the case, we’ll use the 2 Proportions test.

This test answers the question: Is the difference between the proportion of sick people in each group statistically significant?

**Related post**: How Hypothesis Tests Work: P-values and Significance Levels

The p-value of 0.000 indicates that the difference between the two groups is statistically significant. The Sample p column displays the proportion of flu cases in each group, which I’ll convert to percentages (0.96% vs. 2.9%). The estimate for the difference shows that the vaccinated group has 1.9% fewer cases than the placebo group. The confidence interval indicates that we can be 95% confident that the population difference is between 2.64% and 1.23%.

**Related post**: How to Interpret P-values Correctly

Because this study is an RCT, it’s reasonable to assume that the vaccinations cause the reduction.

The difference between the groups (1.9%) is not what we hear about in the media. They report the vaccine effectiveness, which is the relative reduction in the risk for the vaccinated group. The formula for the effectiveness is:

This study reports a 66.9% vaccination effectiveness for the flu shot compared to the control group.

### The Monto Study

The Monto et al. study^{2} evaluates the 2007-2008 flu season and follows its participants from January to April. Participants are 18-49 years old.

Treatment | Flu count | Group size |

Shot | 28 | 813 |

Placebo | 35 | 325 |

Let’s perform the 2 Proportions test on these data.

Like before, the small p-value indicates that the difference between the flu shot group and the placebo group is statistically significant. For this study, the vaccinated group has 7.3% fewer cases of the flu.

Let’s calculate the effectiveness:

This study finds a 68.0% vaccine effectiveness for the flu shot.

## Conclusions Up to This Point

First, isn’t it cool how we can analyze their data and draw our own conclusions? Our findings are consistent with those reported by the news media. The flu vaccine produces a statistically significant reduction in the number of influenza infections. Further, the flu shots are about 68% effective.

However, seeing the data for yourself, did you gain some new insights? I know I did. For me, I think the percentage point reduction in flu cases sounds less impressive than the 68% effectiveness. The former measures absolute risk while the later measures relative risk. Same data but different ways of presenting the results. Let’s investigate this.

## Relative Risk versus Absolute Risk

I don’t think the 68% effectiveness statistic is helpful because it’s the relative risk. Here’s an example to illustrate the difference between relative and absolute risk.

**Relative assessment:**Your car travels half as fast as another car, but you don’t know the actual velocity of either car.**Absolute assessment:**Your car travels at 30 MPH while the other car travels at 60 MPH.

Obviously, the absolute assessment is more helpful. The same idea applies to knowing the absolute risk of catching the flu after vaccination compared to no vaccination.

### Vaccine effectiveness is a relative risk

Vaccine effectiveness doesn’t inform you about the absolute risk of catching the flu for either group. Additionally, it is a particularly confusing measure of relative risk. Recall that effectiveness is calculated as the inverse of the relative risk. That’s much harder to interpret. A 67% effectiveness is equivalent to having one-third the risk of acquiring the flu if you get a flu shot (1 – 0.67 = 0.33).

I find that vaccine effectiveness is a confusing statistic. It doesn’t tell you the absolute risk for anyone, and it’s the inverse of what you want to know!

### Group proportions of flu cases estimate the absolute risks

In the statistical output, we can use the proportion for each group to estimate the absolute risk of acquiring the flu. Then, we can subtract the two proportions to calculate the flu vaccine’s absolute reduction in risk. In the table below, I summarize this information as percentages and add in two more flu seasons from another study (Bridges et al.^{3}):

Flu season | Placebo | Flu Shot | Risk Reduction |

1997/98 | 4.4 | 2.2 | 2.2 |

1998/99 | 10.0 | 1.0 | 9.0 |

2006/07 | 2.9 | 1.0 | 1.9 |

2007/08 | 10.8 | 3.4 | 7.4 |

Average | 7.0 | 1.9 | 5.1 |

The risk of catching the flu varies by season. The differences aren’t surprising because the studies assess different flu seasons, which have different viruses. Some flu seasons are worse than others because the strains are more virulent. Sometimes the strains in the vaccinations aren’t a good match for the most prevalent influenza strains.

The last row in the table averages the values across the four flu seasons. When you do not receive the flu vaccination, you have a 7.0% chance of catching the flu on average. However, when you get the shot, your risk is 1.9%. Consequently, the average reduction is 5.1%.

I don’t know about you, but the 5.1% reduction doesn’t sound nearly as remarkable as 67% effectiveness! Both statistics are calculated from the same data but provide different types of information.

Knowing that the annual average risk reduction is 5.1% is helpful. While this absolute reduction might sound small, flu shots aren’t supposed to be a one-time event. Let’s look at the long-term benefits of getting the flu shots annually. I want to see if my impression that annual flu shots will eventually spare you from a week of misery is correct!

## Modelling Flu Outcomes Over Decades

Flu shots benefits are like investing year-after-year. The cumulative effect magnifies the differences over time. I’m going to model flu outcomes over decades in the same manner that financial planners illustrate the differences between different courses of action.

Investment return rates fluctuate over the years just like the influenza rates. Financial planners use reasonable long-term averages to provide an estimate of different outcomes. Similarly, I’ll use the average infection rate for those who are vaccinated and those who are not. This approach produces comparative results for annual flu vaccinations versus no flu vaccinations over decades.

To model this statistically, I’ll use probability distribution plots. Because the data are binary (infected or not infected), I’ll use distributions that are designed for binary data—the binomial and geometric distributions. For the graphs, the assumptions are that the average infection rate for the:

- Unvaccinated is 7.0% annually.
- Vaccinated is 1.9% annually.

We’ll learn what effect that 5.1% difference has over decades. I’ll compare two different scenarios—getting the flu shot every year versus never getting the influenza vaccination. Using our estimates, I’ll answer two questions:

- How long until my first case of the flu on average?
- How many times will I get the flu?

In both plots, the left panel displays the no flu shot scenario while the right panel shows the annual influenza vaccine scenario.

## How long until my first case of the flu on average?

I’ll use the geometric distribution to model the probability of first catching the flu. Each bar in the graph indicates the likelihood of first getting the flu in a specific year. The chances for any given year is small. More importantly, I’ve shaded the graphs to indicate the number of years until the cumulative probability reaches 50%. The graph stops at 65 years because flu shot effectiveness decreases around that age.

In general, the chart shows how your probability of first contracting the flu is much higher early on when you aren’t vaccinated (left) than when you are regularly vaccinated.

The left panel indicates that without flu shots, you have a 50/50 chance of getting the flu in 11 years. On the other hand, the right panel displays the probabilities when you are vaccinated annually. You don’t reach a 50% cumulative probability until 38 years. Furthermore, when you never get the flu shot, you have only a 6.8% chance of not catching the flu in 38 years!

## How often will I catch the flu?

Let’s statistically model the number of times you can expect to catch the flu in 20 years. For this graph, I’ll use the binomial distribution. Each bar indicates the likelihood of catching the flu the specified number of times. I’ve shaded the bars to represent the cumulative probability of catching the flu at least twice in 20 years.

There’s a big difference that jumps out at you! The largest bar on the graph is the one that represents zero cases of the flu in 20 years when you get flu shots. When you’re vaccinated annually, you have a 68% chance of not catching the flu within 20 years! Conversely, if you don’t get flu shots, you have only a 23% of escaping the flu entirely.

In the left panel, the distribution spreads out much further than in the right panel. Without vaccinations, you have a 41% chance of getting the flu at least twice in 20 years compared to 5% with annual vaccinations. Some unlucky unvaccinated folks will get the flu four or five times in that time span!

## Closing Thoughts about Flu Shots

By analyzing the data ourselves, we found that flu shots reduce the risk of catching the flu. However, the reduction in absolute terms is small. In fact, without flu shots, you still have a low probability of getting the flu in any one flu season. This low risk explains why I hear from people who don’t get flu shots and they haven’t had the flu for a while.

However, when you model the long-term outcomes of being vaccinated against the flu regularly, it supports my hypothesis that flu vaccinations will save you from at least one week of misery, and probably more! Regular flu shots both lengthen your expected time until first catching the flu and reduce the number of times you can expect the flu virus make you sick within a 20-year timeframe.

There are some additional complications when it comes to the flu. Recall that other viruses are in circulation. These can be other strains of flu and non-flu viruses that cause flu-like symptoms. In this post, the probabilities of infection only refer to catching a strain of flu that is in that year’s flu vaccinations. It’s possible to be vaccinated annually and never catch a flu strain that you were vaccinated against but yet experience flu-like symptoms from some other virus or flu strain. Nevertheless, you are still receiving the benefits discussed above.

So, what am I going to do about shots?

Of course, I’m not a medical doctor, and you should talk to yours for specific advice. Even though the benefits might seem small for a single year, I am going to continue to get my flu shots annually. I’m sure that they will prevent misery down the road! Additionally, it reduces the risk of me passing the flu to my 99-year-old grandmother (you can read about her in this blog post), which would be very dangerous for her.

### References

1. Beran J, Vesikari T, Wertzova V, Karvonen A, Honegr K, Lindblad N, Van Belle P, Peeters M, Innis BL, Devaster JM. Efficacy of inactivated split-virus influenza vaccine against culture-confirmed influenza in healthy adults: a prospective, randomized, placebo-controlled trial. J Infect Dis 2009;200(12):1861-9.

2. Monto AS, Ohmit SE, Petrie JG, Johnson E, Truscon R, Teich E, Rotthoff J, Boulton M, Victor JC. Comparative efficacy of inactivated and live attenuated influenza vaccines. N Engl J Med. 2009;361(13):1260-7.

3. Bridges CB, Thompson WW, Meltzer MI, Reeve GR, Talamonti WJ, Cox NJ, Lilac HA, Hall H, Klimov A, Fukuda K. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: A randomized controlled trial. JAMA. 2000;284(13):1655-63.

Matt Dubuque says

I don’t understand how you can calculate your chances of getting the flu if you get a flu shot every year for 20 years.

It seems you have only calculated the likelihood you will contract a flu strain contained within a vaccine.

But we do not know what percentage of total flu strains are included in each year’s vaccine, nor the prevalence of each of those strains excluded.

Jim Frost says

Hi Matt,

What I’ve done is to use the estimates from the studies. You can estimate the probability of catching the flu when you have had the vaccination and when you haven’t. These studies track what virus is making someone sick. So, they count only those who get sick for one of the strains included in the shot. As I mentioned, these probabilities

arefor only the specific 3 or 4 strains in the vaccines. It wouldn’t make sense to see how effective the vaccines are for strains that are not in the vaccine. We know the average absolute risk reduction and use that to project the effects over 20 years. As I mention, you might very well get sick with other flu strains or entirely different types of viruses. In the end, we are estimating the benefits associated with the shots for the specific strains that they include.I hope this helps clarifies matters,

Jim

Andrew Stone says

This was really great information. Thanks for plowing through the math and writing it down in a logical form. Thanks!

Jim Frost says

Thanks Andrew!