P values determine whether your hypothesis test results are statistically significant. Statistics use them all over the place. You’ll find P values in t-tests, distribution tests, ANOVA, and regression analysis. P values have become so important that they’ve taken on a life of their own. They can determine which studies are published, which projects receive funding, and which university faculty members become tenured!

Ironically, despite being so influential, P values are misinterpreted very frequently. What *is* the correct interpretation of P values? What do P values *really* mean? That’s the topic of this post!

P values are a slippery concept. Don’t worry. I’ll explain p-values using an intuitive, concept-based approach so you can avoid making a widespread misinterpretation that can cause serious problems.

## What Is the Null Hypothesis?

P values are directly connected to the null hypothesis. So, we need to cover that first!

In all hypothesis tests, the researchers are testing an effect of some sort. The effect can be the effectiveness of a new vaccination, the durability of a new product, and so on. There is some benefit or difference that the researchers hope to identify.

However, it’s possible that there actually is no effect or no difference between the experimental groups. In statistics, we call this lack of an effect the null hypothesis. When you assess the results of a hypothesis test, you can think of the null hypothesis as the devil’s advocate position, or the position you take for the sake of argument.

To understand this idea, imagine a hypothetical study for medication that we know is entirely useless. In other words, the null hypothesis is true. There is no difference at the population level between subjects who take the medication and subjects who don’t.

Despite the null being accurate, you will likely observe an effect in the sample data due to random sampling error. It is improbable that samples will ever exactly equal the null hypothesis value. Therefore, the position you take for the sake of argument (devil’s advocate) is that random sample error produces the observed sample effect rather than it being an actual effect.

## What Are P values?

P-values indicate the believability of the devil’s advocate case that the null hypothesis is correct given the sample data. They gauge how consistent your sample statistics are with the null hypothesis. Specifically, if the null hypothesis is right, what is the probability of obtaining an effect at least as large as the one in your sample?

- High P-values: Your sample results are consistent with a true null hypothesis.
- Low P-values: Your sample results are not consistent with a null hypothesis.

If your P value is small enough, you can conclude that your sample is so incompatible with the null hypothesis that you can reject the null for the entire population. P-values are an integral part of inferential statistics because they help you use your sample to draw conclusions about a population.

**Background information**: Difference between Descriptive and Inferential Statistics and Populations, Parameters, and Samples in Inferential Statistics

## How Do You Interpret P values?

Here is the technical definition of P values:

P values are the probability of observing a sample statistic that is at least as extreme as your sample statistic when you assume that the null hypothesis is true.

Let’s go back to our hypothetical medication study. Suppose the hypothesis test generates a P value of 0.03. You’d interpret this P-value as follows:

If the medicine has no effect in the population as a whole, 3% of studies will obtain the effect observed in your sample, or larger, because of random sample error.

How probable are your sample data if the null hypothesis is correct? That’s the only question that P values answer. This restriction segues to a very persistent and problematic misinterpretation.

**Related posts**: Understanding P values can be easier using a graphical approach: How Hypothesis Tests Work: Significance Levels and P-values and learn about significance levels from a conceptual standpoint.

## P values Are *NOT *an Error Rate

Unfortunately, P values are frequently misinterpreted. A common mistake is that they represent the likelihood of rejecting a null hypothesis that is actually true (Type I error). The idea that P values are the probability of making a mistake is WRONG! You can read a blog post I wrote to learn *why* P values are misinterpreted so frequently.

You can’t use P values to directly calculate the error rate for several reasons.

First, P value calculations assume that the null hypothesis is correct. Thus, from the P value’s point of view, the null hypothesis is 100% true. Remember, P values assume that the null is true, and sampling error caused the observed sample effect.

Second, P values tell you how consistent your sample data are with a true null hypothesis. However, when your data are very inconsistent with the null hypothesis, P values can’t determine which of the following two possibilities is more probable:

- The null hypothesis is true, but your sample is unusual due to random sampling error.
- The null hypothesis is false.

To figure out which option is right, you must apply expert knowledge of the study area and, very importantly, assess the results of similar studies.

Going back to our medication study, let’s highlight the correct and incorrect way to interpret the P value of 0.03:

**Correct:**Assuming the medication has zero effect in the population, you’d obtain the sample effect, or larger, in 3% of studies because of random sample error.**Incorrect:**There’s a 3% chance of making a mistake by rejecting the null hypothesis.

Yes, I realize that the incorrect definition seems more straightforward, and that’s why it is so common. Unfortunately, using this definition gives you a false sense of security, as I’ll show you next.

**Related posts**: See a graphical illustration of how t-tests and the F-test in ANOVA produce P values.

## What Is the True Error Rate?

The difference between the correct and incorrect interpretation is not just a matter of wording. There is a fundamental difference in the amount of evidence against the null hypothesis that each definition implies.

The P value for our medication study is 0.03. If you interpret that P value as a 3% chance of making a mistake by rejecting the null hypothesis, you’d feel like you’re on pretty safe ground. However, after reading this post, you should realize that P values are not an error rate, and you can’t interpret them this way.

If the P value is not the error rate for our study, what is the error rate? Hint: It’s higher!

As I explained earlier, you can’t directly calculate an error rate based on a P value, at least not using the frequentist approach that produces P values. However, you can estimate error rates associated with P values by using the Bayesian approach and simulation studies.

Sellke et al.* have done this. While the exact error rate varies based on different assumptions, the values below use run-of-the-mill assumptions.

P value | Probability of rejecting a true null hypothesis |

0.05 | At least 23% (and typically close to 50%) |

0.01 | At least 7% (and typically close to 15%) |

These higher error rates probably surprise you! Regrettably, the common misconception that P values are the error rate produces the false impression of considerably more evidence against the null hypothesis than is warranted. A single study with a P value around 0.05 does not provide substantial evidence that the sample effect exists in the population.

These estimated error rates emphasize the need to have lower P values and replicate studies that confirm the initial results before you can safely conclude that an effect exists at the population level. Additionally, studies with smaller P values have higher reproducibility rates in follow-up studies. Learn about the Types of Errors in Hypothesis Testing.

Now that you know how to interpret P values correctly, check out my Five P Value Tips to Avoid Being Fooled by False Positives and Other Misleading Results!

Typically, you’re hoping for low p-values, but even high p-values have benefits!

### Reference

*Thomas SELLKE, M. J. BAYARRI, and James O. BERGER, Calibration of p-values for Testing Precise Null Hypotheses, The American Statistician, February 2001, Vol. 55, No. 1

Aakash Sharma says

My p value of one way ANOVA is 1.09377645180021E-12. What does it mean.. Is it significant ?

Jim Frost says

Hi Aakash, your p-value is written using scientific notation. Scientific notation is a convenient way to represent very large and very small numbers. In your case, it represents a very small p-values. Yes, it significant!

The minus 12 indicates that you need to move the decimal point 12 places to the left. Your p-value is much smaller than any reasonable significance level and, therefore, represent statistically significant results. You can reject the null hypothesis for your ANOVA.

I hope this helps!

Zaki Khadija says

Hello, Thank you very much for your explanations, I have studied the significance of the correlation between several quantitative variables at the base of a software, but practically I want to know how to calculate p-value manually? in order to understand its principle.

On the other hand, concerning the p-value, what does it mean technically, because I find it difficult to define this parameter practically in my field of environmental chemistry?

Cordially

Lizette Franks says

Hi! thanks so much! this clarifies the difference very much. I’m analyzing and writing reports about Nutrition related literature. Two of the studies are prospective cohort studies, with several covariates. The topic is the egg/dietary cholesterol relationship with cardiovascular disease. You probably know that nutrition research is like a roller coaster 🙂 So I encountered new terms for statistics analysis used on these type of studies that explore non linear associations. The Rao-Scott chi-square test, the Cox proportional hazard models, restricted cubic splines are terms that I’ve learned recently. I love your blog, it’s helping me A LOT to understand, clarify basic and more advanced statistical concepts. I have bookmarked it and will be using it a lot!

Lizette

Jim Frost says

Hi Lizette, I often describe statistics as an adventure because it’s a process that leads to discoveries but it is filled with trials and tribulations! It sounds like you’re having an adventure! And, of course, we like having our “cool” terms in statistics! I don’t have blog posts on the procedures you mention, at least not yet.

I’m so glad my blog has been helpful in your journey! Thanks for taking the time to write. I really appreciate it!! 🙂

Lizette says

Hi, I’m trying to understand what “p linear” and “p non linear trend” mean. I have only taken basic statistics and I’m working on reviewing nutrition related research articles. thanks so much!

Jim Frost says

Hi Lizette,

The context matters and I’m not sure what kind of analysis this is from? I’ve heard of those p-values in the context of time series analysis. In that scenario, these p-values help you determine whether the time series has a constant rate of change over time (p linear) or a variable rate of change over time (nonlinear). The meaning of linear trend is easy to understand because it represents a constant rate of change. Nonlinear trends are more nuanced because you might have a greater rate of change earlier, later, or in the middle. It’s not consistent throughout. You can also learn more from the combinations of the two p-values.

If the linear p-value is significant but nonlinear is not significant, you have a nice consistent rate of change (increase or decrease) over time.

If both p-values are significant, it would suggest a variable rate of change but one that has a consistent direction over time.

If neither p-value is significant, it suggests that the variable does not systematically tend to increase or decrease over time.

If the nonlinear p-value is significant but not the linear p-value, it suggests you have variable rates of change in the short term but in the long run there is no systematic increase or decrease in the variable.

I hope that helps!

Natalie says

How to you interpret a p-value that is displayed P=1.5 X 10 -19?

Jim Frost says

Hi Natalie,

That p-value is written using scientific notation. Scientific notation is a convenient way to represent very large and very small numbers. In your case, it represents a very small p-values. Yes, it significant!

The minus 19 indicates that you need to move the decimal point 19 places to the left.

Your p-value is much smaller than any reasonable significance level and, therefore, represent statistically significant results. You can reject the null hypothesis for whichever hypothesis test you are performing.

I hope this helps!

Shaheen Majeed Wani says

I am getting a P value of 0.351. Can you please explain it.

Amrita bhat says

Mine p value is 6.18694E-23 what does it mean.. Is it significant

Jim Frost says

Hi Amrita,

That p-value is written using scientific notation. Scientific notation is a convenient way to represent very large and very small numbers. In your case, these represent very small p-values. Yes, it significant!

The number after the E specifies the direction and number of places to move the decimal point. For example, the negative 23 value in “E-23” indicates you need to move the decimal point 23 places to the left. On the other hand, positive values indicate that you need to shift the decimal point to the right.

Your p-value is much smaller than any reasonable significance level and, therefore, represent statistically significant results. You can reject the null hypothesis for whichever hypothesis test you are performing.

I hope this helps!

Alfonso says

Thanks so much for your answer Jim!

Indeed I think we want to reach the same conclusion, but I’d like to see the results of the wrong approach to further cement my understanding since I’m not an expert in statistics (“seeing is believing!”)

In other words I entirely agree that it’s WRONG to keep testing the pvalue as the experiment runs. But how can I prove it empirically? My idea was to show to me and others that all tests that should fail to reject the null hypothesis (my list of A/A tests described above), can reject it if left to run long enough (in other words, every A/A test will have p < 0.05 if it runs long enough).

Is this statement correct? if not, why not?

Thank you!

alfonso says

Jim,

after reading again, I have one more question:

“If the medicine has no effect in the population as a whole, 3% of studies will obtain the effect observed in your sample, or larger, because of random sample error.”

– in online testing, is it correct to say that we cannot have sampling error? Since we always compare (for a limited time) the entire population in A and the entire population in B? If yes, how does that affect the interpretation of pvalue?

Jim Frost says

Hi again Alfonso,

It’s still a sample. While you might have test the entire population for a limited time, you are presumably applying the results outside of that time span. That’s also why you need to be careful about the time frame you choose. Is it representative? You can only apply the results beyond the sample (i.e., to other times) if your sample is representative of those other times. If you collect data all day Sunday, you might be capturing different users than you’d get during the week. If that’s true, you wouldn’t be able to generalize your Sunday sample to other days. Same if you were to collect day during only a specific time during the day.

In your context, you’re still collecting a sample that you want to generalize beyond the sample. So, you’d need to use hypothesis testing for that reason. You also ensure that your sampling method collects a representative sample.

I hope this helps! I’m glad that you’re hooked and reading!!

alfonso says

Dear Jim,

A colleague just shared your blog with me and after 2 posts I’m hooked. I will read more today.

I use ttest and pvalues in the domain of web and app A/B testing and I’ve read everything I could find online but I still wasn’t sure I understood.

I built an A/A simulator in python and I got a lot more statistically significant results than 5% so I’m confused.

Just for clarity I call an A/A test a randomise observation where both series use the same success rate in %.

Even after reading your article alpha and pvalue are still somehow overlapping for me. I’ll keep reading your article to further clarify.

I have 3 questions that I hope you can answer:

– what would the graph look like of plotting the pvalue of 20 A/A tests over time? I would expect the pvalue to swing widely in the beginning and then stay firmly above 0.05 and every so often a test would go to statistical significance for a while and then come back above 0.05. I would expect 1 or max 2 statistically significant experiments *at any given point in time* (this is crucial in my understanding) after a sample size big enough has been reached

– is it true that if I keep collecting samples every single A/A test will eventually turn statistically significant even if just briefly?

– given that I will run hundreds or thousands of tests, is there an accepted standard of build my analysis framework to guarantee a 5% false positives rate? I was thinking all I needed was to set the sample size at the start to avoid falling into the trap I ask about in the previous question but now I’m not so sure anymore. (I use a well known tool online to calculate my sample size based on base conversion rate and observable absolute or relative difference)

I will keep reading but if you talk about any of this in details in any other article I would be grateful if you could share the link and if you haven’t covered these topics I hope you might do so in the future.

Jim Frost says

Hi Alfonso,

I know enough about the context of A/B test online to know that it is often fairly different than how we’d teach using hypothesis tests in statistics.

For statistical studies, you’d select a sample size, collect your sample, perform the experiment, and analyze the results just once. You wouldn’t keep collecting data and performing the hypothesis test with each new observation. The risk with continuing to perform the analysis as you collect the data is that, yes, you are very likely to get an anomalous significant result at some point. I don’t recommend the process you describe of plot p-values over time. Pick a sample size and stick with it and calculate the results only at the end.

Also, be aware that different types of users might be online at different times and days of week due things like age, employment status, and time zone. Use a sampling plan that gets a good representative sample. Otherwise your results might apply only to a subset.

If you follow the standard rule of collecting the sample of a set size and analyzing the data once at the end, then your false positives should equal your significance level. If you’re checking the p-values repeatedly or keep testing until you get a significant p-value, that will dramatically increase your false positive rate.

Finally, I’ve heard that some A/B testing uses one tailed testing. Don’t do that! For more details, read my post about when you should use one-tailed testing.

Damodar Suar says

I have read the comments. I am not a specialist in statistics but i use stastistics in my research.Let us come to application of p at least to t and r. In each case a study is conducted and results are significant at 5 % level (p <=.05). t test access the mean difference on wage rate of females in two locations X, and Y, the mean indicate Y has a higher value), r indicates the relation between depression and low exam marks among students, in each case the sample-size is 100. It may be understood that in research, we test directional alternate hypothesis , not null (which is obviously the no difference or no relation in null and opposite of alternate hypothesis). Taking the 'p' into account, how will we give a convincing interpretation or linguistic expression so that another, non-expert can understand it? Liking it to false positive, error may not be understood by a common man. Please reply. Does it mean the following in context of t and r respectively?

t- There are 95% chances that the wages in location Y are higher than location X and 5 %chances that difference will not be there.

r- The relations between anxiety and low exam marks hold good in 95% of cases and do not hold good in 5% of cases.

Jim Frost says

Hi Damodar,

Many of the answers to your questions are right here in this post. So, you might want to reread it.

P-values represent the strength of the evidence that your sample provides against the null hypothesis. That’s why you use the p-value in conjunction with the significance level to determine whether you can reject the null. Hypothesis testing is all about the null hypothesis and whether you can reject it or fail to reject it.

Coming up with an easy to understand definition of p-values is difficult if not impossible. That’s unfortunate because it makes it difficult to interpret correctly. Read my post, why are p-values misinterpreted so frequently for more on that.

As for your interpretation, those are the common misconceptions that I refer to in this post. So, please reread the sections where I talk about the common misconceptions! P-values are NOT the probability that either of the hypotheses are correct!

P-values are the probability of obtaining the observed results, or more extreme, if the null hypothesis is correct.

Himani Narula says

Hi Jim,

Thanks for a prompt reply. I have a fair understanding now. Please tell me if I am wrong if I say that for a statistically significant result, we say that if my null hypothesis is true, i would expect measure under consideration to be at least as large as the one observed in my study.

I came to this conclusion, by comparing my p value with alpha. if p value lies in critical region, we reject the null hypothesis and vice versa. Now that you stated that for a single study, we can’t state that error rate of false positive is alpha, how are we comparing alpha and p value for conclusions?

Jim Frost says

Hi again Himani!

If you have read it already, read my post about p-values and significance levels. I think that will answer many of your questions.

A statistically results indicates that IF the null hypothesis is true, you’d be unlikely to obtain the results that your study actually obtained. Statistical significance and p-values relate to the probability of obtaining your observed data IF the null is true. Always note that the probability is based on the assumption that the null is true.

You can think of the significance level as an evidentiary standard. It describes how strong the evidence must be for you to be able to conclude using your sample that an effect exists in the population. The strength of the evidence is defined in terms of how probable is your observed data if the null is true.

The p-value represents the strength of your sample evidence against the null. Lower p-values represent stronger evidence. Like the significance level, the p-value is stated in terms of the likelihood of your sample evidence if the null is true. For example, a p-value of 0.03 indicates that the sample effect you observe, or more extreme, had a 3% chance of occurring if the null is true.

So, the significance level indicates how strong the evidence must be while the p-value indicates how strong your sample evidence actually is. If your sample evidence is stronger than the evidentiary standard, you can conclude that the effect exists in the population. In other words, when the p-value is less than or equal to the significance level, you have statistically significant results, you can reject the null, and conclude that the effect exists in the population.

Please do read the other post if you haven’t already because I show how this works graphically and I think it’s easier to understand in that format!

Himani Narula says

Hi Jim,

Your blog has been of great help. It would be great if you could explain a bit further about how alpha (false positive) is different from the false positive rate (0.23) mentioned by you in the post and role of simulation in this case.

Big help!

Thank you

Jim Frost says

Hi Himani,

Thanks for writing with the excellent question. I can see how these two errors sound kind of similar, but they’re actually very different!

The Type I error rate is the probability of rejecting null hypothesis when it is actually true. It is a probability that applies to a class of studies. For an alpha of 0.05, it applies to studies with that alpha level and to studies where the null is true. You can say that 5% of all studies that have a true null will have statistically significant results when alpha = 0.05. However, you cannot apply to probability to a single study. For example, for a statistically significant study at the alpha = 0.05 level, you CANNOT state that there is a 5% chance that the null is true. You cannot obtain the probability for a single study using alpha, p-values, etc with Frequentist methodologies. The reason you can’t apply it to a single study is because you don’t know whether the null is true or false and the Type I error rate only applies when the null is true.

The error rates based on the simulation studies and Bayesian methodology can be applied to individual studies, at least in theory. However, to get precise probabilities you’ll need information that you often won’t have. Using these methodologies, you can take the p-value of an individual study and estimate the probability that the particular study is a false positive. However, I don’t want you to get too wrapped in mapping p-values to false positive rates. You’ll need to know the prior probability, which is often unknown. However, the gist is that the common misinterpretation of p-values underestimates the chance of a false positive. Also, a p-value near 0.05 often represents weak evidence even though it is statistically significant.

I hope this clarifies matters!

Edward says

Hi Jim,

Thanks for helpful posts. I have been browsing your blog for some time now and I gained a lot.

One quick question:

What happens if the null hypothesis is rejected based on t-test but we can’t do so by looking at p-value.

I know one is derived from the other statistic. But which one we should look at first in order to be able to say something about the null hypothesis: t-statistics or p-value in the t-test?

The same applies to ANOVA as well.

Which one do we look at first? Whether if Significance F is less than F statistics or the P-value alone?

Jim Frost says

Hi Edward,

You can either reject the null hypothesis by determining whether the test statistic (t, F, chi-square, etc.) falls into the critical or by comparing the p-value to the significance level. These two methods will always agree. If the test statistic falls within the critical region, then the p-value is less than or equal to the significance level.

Because the two methods are 100% consistent, you can use either one to evaluate statistical significance. You don’t need to use both methods, except maybe when you’re learning about how it all works. Personally, I find it easiest just to look at the p-value.

To see how both methods work, read my posts about how hypothesis tests work, how t-tests work, and how the F-test works in one-way ANOVA.

I hope this helps!

Yash Guleria says

Hi Jim,

I have 3 p values .. 0, 2E-12 and 3.2E-316.

I dont know what is wrong but how do i interpret these values?

Jim Frost says

Hi Yash,

Those p-values are written using scientific notation. Scientific notation is a convenient way to represent very large and very small numbers. In your case, these represent very small p-values.

The number after the E specifies the direction and number of places to move the decimal point. For example, the negative 13 value in “E-12” indicates you need to move the decimal point 12 places to the left. On the other hand, positive values indicate that you need to shift the decimal point to the right.

These values are smaller than any reasonable significance level and, therefore, represent statistically significant results. You can reject the null hypothesis for whichever hypothesis test you are performing.

I hope this helps!

adamson okunmwendia says

you are good jim. you are the best

Trent says

Jim, thank you so, so much for your patience and help over the past week. I think I can finally say that I get it. Not easy to keep everything straight, but your simplistic breakdown in your most recent post really helped to clear everything up. Even though I previously read about p-values and type I errors from your other blog posts, I guess I needed to re-hear/re-think those tricky concepts in a variety of different ways to finally absorb them. I finally feel comfortable enough to share these cool insights with my research peers, and I’ll point them to your blog for extra stats goodies!

Thank you so much, again. I’m slowly making my way through your blog (trying to balance grad school at the same time); I look forward to your other posts!

aloha

trent

P.S. Please do email me about the notification issue, I don’t believe I received an email from you yet. Your blog has really helped me get a better grasp of stats (I found your blog from your chocolate vs mustard analogy for interaction analyses, that was brilliant!), and so I’d be more than happy to help with the notification issue in any way I can.

Jim Frost says

Hi Trent,

You’re very welcome! P-values are a very confusing concept. Somewhere in one of my posts, I have a link to an external article that shows how even scientists have a hard time describing what they are! They’re not intuitive. And, when you conduct a study, p-values really aren’t exactly what you want them to be. You

wantthem to be the probability that the null is true. That would be the most helpful. Unfortunately, they’re not that–and they can’t be that. I’m not sure if you read it, but I’ve written a post about why p-values are so easy to misunderstand.Despite these difficulties, p-values provide valuable information. In fact, as I write in an article, there’s a relationship between p-values and the reproducibility of studies.

Just a couple more p-value posts to read if you’re so interested! If you haven’t already.

Best of luck with grad school! I’m sure you’ll do great!

By the way, I did email you. If you haven’t received it, that’s odd! I will try again from a different email address over the weekend.

A says

Jim… I cannot explain how many videos I have watched and articles I have read to try and understand this and you just cleared it all up with this. Saved my life. Thank you, thank you, thank you.

Jim Frost says

You’re very welcome! Presenting statistics in a clear manner is my goal for the website. So, it makes my day when I hear that my articles have actually helped people! Thanks for writing!

Trent says

Hi Jim, thank you so much for your reply! I’m sorry I wasn’t able to check back in until now. It seems that I still haven’t been able to connect the final pieces of the puzzle, based on your response to: “Thus, for a sample statistic assessed by a large group of similar studies, a P<0.05 would translate to a Type I error rate of <5%."

This is where I'm getting stuck:

Prior to a study, researchers typically set their significance level (alpha level) to 0.05. Researchers will then compare their p-value to the alpha level of 0.05 to determine if their results are statistically significant. If P<0.05, then the results are statistically significant at an alpha level of 0.05, which by extension means that the results have a 5% or lower probability of being a false positive (since the alpha level was set to 0.05, and alpha level = probability of a false positive), right? If this is all true, then a P<0.05 for a study with a significance level of 0.05 does not have a false positive probability of 23% (and typically close to 50%)… it has a 5% or lower probability of being a false positive.

That said, based on your article, I know I'm messing up my logic somewhere, but I can't figure out where…

aloha

trent

P.S. I double checked my gmail spam & trash folders and there were no notification emails of any of your replies.

Jim Frost says

Hi Trent,

I’m going to send you an email soon about the notification issue. So, be on the lookout for that.

I

thinkpart of the confusion is over the issue of single studies versus a group of studies. Or, relatedly, a single p-value versus a range of p-values. Alpha is a range of p-values and applies to a group of studies. All studies (the group) that have p-values less than or equal to 0.05 (range of p-values) have a Type I error rate of 0.05. That error rate applies to the groups of studies. You can’t apply it to a single study (i.e., a single p-value).A single p-value for a single study is not that type of error rate at all. It represents the probability of obtaining your sample if the null is true. In other words, the p-value calculations begin with the assumption that the null is true. Therefore, you can not use the p-value to determine the probability that the null (or alternative) hypothesis is true. In other words, you can’t map p-values to the false positive rate.

So, when you say “If P<0.05, then the results are statistically significant at an alpha level of 0.05, which by extension means that the results have a 5% or lower probability of being a false positive (since the alpha level was set to 0.05, and alpha level = probability of a false positive)," that's not true. For one thing, the p-value assumes the null *is* true. For another, the group of studies as a whole has an error rate of 0.05, but you don't know the error rate for an individual study. Additionally, you just don't know whether the null is true or false. The error rate only applies to studies where the null is true. And, the p-value calculations assume the null is true. But, you don't know for sure whether it is true or not for any given study.

Let's go back to what I said about the p-values being the "devil's advocate" argument. For any treatment effect that you observe in sample data, you can make the argument that the effect is simply random sampling error rather than a true effect. The p-value essentially says, "OK, lets assume the null is true. How likely was it for us to observe these results in that case." If the probability is low, you were unlikely to obtain that sample if the null is true. It pokes a hole in the devil's advocate argument. It's important to remember that p-values are a probability related to obtaining your data assuming the null is true and *not* a probability that the null is true. You're trying to equate p-values to the probability of the null being true--which is not possible with the Frequentist approach.

Trent says

Hi Jim,

Thank you for your reply. The two other articles you linked were really helpful. I think I’m almost there with understanding the whole picture. May I clarify my current understanding with you?

Alpha applies to a group of similar studies, thus we can’t directly translate the p-value of a single study to the Type I error rate for a given hypothesis. However, using simulation studies or Bayesian methods, we can estimate the Type I error rate–from a single study–for a P=0.05 sample statistic to 23% (and typically close to 50%).

That said, in order to estimate the Type I error rate directly using alpha (and P-values), we need to see the results from a group of similar studies (ie meta-analysis). Thus, for a sample statistic assessed by a large group of similar studies, a P<0.05 would translate to a Type I error rate of <5%.

How did I do?

aloha

trent

P.S. I'm unsure how the "Notify me of new comments via email" function is supposed to work on your blog, but it didn't notify me via email of your reply. So I had no idea that you replied to my comment until I checked back on this post.

Jim Frost says

Hi Trent,

I’m glad the other articles were helpful! There’s actually quite a bit to take into understand p-values. It’s possible to come up with a brief definition, but implies a thorough knowledge of underlying concepts! I will look into the Notify function. It should email you. I’ll hunt around in the settings to be sure, but I believe it is set up to send emails. Is there a chance it went to your junk folder?

Yes! That’s

veryclose! Just a couple of minor quibbles and clarifications. I wouldn’t say that you use simulation and Bayesian methods to estimate the Type I error rate. That’s specific to the hypothesis testing framework. And, it applies to group of similar studies. Alpha = the Type I error rate. And both apply to a group of studies.Simulation studies and Bayesian methods can help you take a P-value from an individual study and estimate the probability of a false discovery (or false positive). P-values relate to individual studies and the probability of a false positive applies to that individual study. So, we’ve moved from probabilities for a group of studies (Alpha/Type I error) to probabilities of false positive for an individual study. To make that shift from a group to an individual study, we must switch methodologies because the Frequentist method cannot calculate the false discovery rate for a single study.

An important note, for simulation studies or Bayesian methodology to estimate the false discovery rate, you need additional information beyond just the sample data. You need an estimate of the probability that the alternative hypothesis is true at the beginning of the study. This is known as the prior probability in Bayesian circles. To develop this probably, you already need to know and incorporate external information into the calculations. This information can come from a group of similar studies as you mention. This probability along with the P-value affects the false discovery rate. That’s why there is a range of values for any given P-value. There is no direct, fixed mapping of p-values to the false discovery rate. A criticism of the prior probability is that it is being estimated. Presumably, the researchers are performing a study because they’re not sure if the alternative is true or not.

It’s not clear to me what you mean in your sentence, “Thus, for a sample statistic assessed by a large group of similar studies, a P<0.05 would translate to a Type I error rate of <5%." I'll assume you're referring to a p-value from a meta analysis. In that case, it still depends on the prior probability. If the prior probability is very high, the false discovery rate will be low. Conversely, if the prior probability is low, the false discovery rate will be higher. You can't state a general rule like the one in your sentence.

Thanks for writing with the interesting questions!

Trent says

Hi Jim, wonderful post! A lot to chew on. May I clarify a point of confusion?

I’ve been taught that alpha is the probability of committing a Type I error. In addition, studies typically set alpha to 0.05, and beta to 0.20 (giving a power of 0.8).

Based on your article, this must be false. A true statement should read:

“Studies typically set the P-VALUE cut-off to 0.05, and beta to 0.20 (giving a power of 0.8).”

Logically following, this means that alpha is generally not set to anything. And for a study with a p-value cut-off of 0.05, the alpha would actually be about 0.23 (and typically close to 0.50).

Is my understanding, correct?

aloha

trent

Jim Frost says

Hi Trent,

It’s correct that alpha (aka the significance level) represents the probability of a type I error. Hypothesis tests are designed so that the researchers can set that value. However, it’s not possible to set beta. You can estimate beta using a power analysis. Power is just 1-beta. However, power analyses are estimates and not something your technically setting like you do with alpha. I write more about this in my post about Type I and Type II errors.

I definitely understand your confusion regarding p-values and alpha. The important thing to keep in mind is that alpha really applies to a class of studies. Of all studies that use an alpha of 0.05 and the null is true, you’d expect to obtain significant results (i.e., a false positive) in 5% of those cases.

P-values represent the strength of the evidence against the null for an individual study. You can state it as being the probability of obtaining the observed outcome, or more extreme, if the null is true. However, you can’t state that it is the probability of the null being true. It’s the probability of the outcome if you assume the null is true (which you don’t really know for sure). Not the probability of whether the null is true.

I think based on what you write, you might be confusing that issue (re: alpha actually being 0.23). Both P-values and alpha relate to cases where the null is true–which you don’t know. The false positive error rates which I think you’re getting at, and I write about at the end, are dealing with the probability of the null being true. In the former, you’re assuming the null is true while in the latter you’re calculating the probability of whether it is true. Using the Frequentist approach (p-values, alpha) you cannot calculate the probability of the null being true. However, you can do that using simulation studies and sometimes using Bayesian methods.

I always think this is a bit easier to understand using graphs and so highly recommend reading my post about p-values and the significance level, which primarily uses graphs.

I hope this helps!

YIHENEW says

Thank you. You give me good insight

David says

Awesome read! How would sample size affect the True Error rate? I would assume since p-values tend to become smaller as sample size increases, that would also effectively reduce the True Error rate since you are more confident about the population (assuming True Error means type I and type II errors).

Jim Frost says

Hi David, Thanks, and I’m glad you enjoyed the article!

There are two types of errors in hypothesis testing. So, let’s see how changing the sample size affects them. You might want to read my article about Type I and Type II Errors in Hypothesis Testing.

There’s three basic components for calculating p-values. The effect size, variability in the data, and the sample size. For the sake of discussion, let’s hold the effect size and the variability constant and just increase the sample size. In that case, you would expect that the p-values would decrease. Frequentists will cringe at this, but lower p-values are associated with lower false discovery rates (Type I errors). Additionally, increasing the sample size while holding the other two factors constant will increase the power of your test. Power is just (1 – Type II error rate). So, you’d expect the Type II errors (false negatives) to decrease. Increasing the sample size is good all around because it lowers both types of error

for a single study! I explain the italicized text later!However, a couple of important caveats for the above. Of course, as I point out in this article, you can’t calculate any error rates from the p-value using the frequentist approach. There’s no direct mapping from p-values to an error rate. You can use simulation studies and the Bayesian approach to estimate the false positive rate from the p-value. However, this requires an estimate of the a priori probability that the alternative hypothesis is correct. That information might be hard to obtain. After all, you’re conducting the study because you don’t know. Additionally, it’s always difficult to calculate the type II error rate. So, while you can say that increasing the sample should reduce both type I and type II errors, you don’t really know what they are! By the way, in a related vein, you might want to read how P-values correlate with the reproducibility of scientific studies.

Let’s return to Frequentist approach because there’s another side of things that isn’t obvious. In contrast with the earlier example for an individual study, the Frequentist approach talks about the Type I errors not for an individual study but for a class of studies that use the same significance level. A result is statistically significant when the p-value is less than the significance level. The significance level equals the Type I Error for all studies that use a particular significance level. For example, 5% of all studies that use a significance level of 0.05 should be false positives. Of course, when you see significant test results, you don’t know for sure which ones are real effects and which ones are false discoveries.

Let’s now hold the other two factors constant but

reducesample size. Let’s reduce it enough so that you have low power for detecting an effect. As your statistical power decreases, your test is less likely to detect real effects when they exist (the Type II error rate increases). However, the hypothesis test controls or holds constant the Type I error rate at your significance level. That’s built into the test. If you have a low power hypothesis test, the test’s ability to detect a real effect is low but it’s false positive rate remains the same. Consequently, when you obtain statistically significant results for a test with low power, you need to be wary because it’s relatively likely to be false positive and less likely to represent a real effect.That’s probably more than what you wanted, but it’s a fascinating topic!

Tetyana says

Dear Jim, thank you very much for you posts!

Does it mean that after I have obtained some small p-value, I have to do some other tests?

Jim Frost says

Hi Tetyana,

After you obtain a small p-value, you can reject the null hypothesis. You don’t necessarily need to perform other tests. I just want analysts to avoid a common misinterpretation. Obtaining a statistically significant result is still a good thing, but you have to keep in mind what it really represents.

Ahmad Allam says

Thank you.

Ahmad Allam says

Thank you very much. You made me reassuring . Appreciated.

How could I record this result in a scientific manuscript?

Jim Frost says

Hi Ahmad,

I think it’s perfectly acceptable to report such a small p-value using the scientific notation that is in your output. The other option would be to report it as a regular value by moving the decimal point 16 places to the left, but that takes up so much more room. So, I’d use scientific notation. It’s there to save space for extremely small and large values depending on the context.

Ahmad Allam says

Hi Jim. Thanks for this value post. But if you can help me on that, I got this result (6.79974E-16) ??? What that mean?

Appreciated.

Jim Frost says

Hi Ahmad,

That is called scientific notation. The E-16 in it indicates that you need to move the decimal point 16 digits to the left. That’s a very small value. Therefore, you have a very significant p-value!

Pamela Marcum says

What an awesome post! Should be required reading for all STEM students.

Jim Frost says

Thanks, Pamela. That means a lot to me!

Amit Kumar Sahoo says

Thanks Jim for your response. i think i got it..

Amit Kumar Sahoo says

Hi Jim,

Thanks for the post. Am little confused with the statement below

“If the medicine has no effect in the population as a whole, 3% of studies will obtain the effect observed in your sample, or larger, because of random sample error.”

Now as per defination

“P-values indicate the believability of the devil’s advocate case that the null hypothesis is true given the sample data. ”

So doesn’t that mean higher P value will accept the alternate hypothesis since higher the probability of alternate happening when null is true. Am not able to get my head wrapped around this concept..

Amit

Jim Frost says

Hi Amit,

Great question! So, the first thing to realize is that the null and alternative hypotheses are mutually exclusive. If the probability of the alternative being true is higher, then the probability of the null must be lower.

However, the p-value doesn’t indicate the probability of either hypothesis being true. This is a very common misconception. Anytime you start linking p-values to the probability that a hypothesis is true, you know you’re going in the wrong direction!

P-values represent the probability of obtaining the effect observed in your sample, or more extreme, if the null hypothesis is true. It’s a probability of obtaining your data assuming the null is true. Consequently, a low p-value indicates that you were unlikely to obtain the sample data that was collected if the null is true. In this manner, lower p-values represent stronger evidence against the null hypothesis. Lower p-values indicate that your data are less compatible with the null hypothesis.

I think this is easier to understand graphically. I have a link in this post to another post How Hypothesis Tests Work: Significance Levels and P-values. This post shows how it works with graphs. I’d recommend taking a look at it.

I hope this helps!

Khursheed statistics says

Hello sir …..hope u r f9

I hve no words that u hve cleared me a lot of concepts of stats ….nd I am really hppy

……nd Wht evr u r uploading

Owsme

Jim Frost says

Hi Khursheed, I’m so happy to hear that you found this post to be helpful. Thanks for the encouraging words. They mean a lot to me!

naseer says

What should be the nature of the relationship of p values (especially Bonferroni corrected) with the Cohen’s d values for the same set of data?

Sean Saunders says

Jim, thanks for this post, but perhaps you could clarify something for me: assuming that H0 is true, if we set an alpha=0.05 level of significance and get a p-value less than that as the result of our sample data, wouldn’t that indicate, since less than 5% of samples would have such an effect due to random sample error, that there is only a 5% chance of getting such a sample, and thus, a 5% chance of rejecting the null hypothesis incorrectly? What am I missing here? Almost every stats book I’ve ever read has presented the concept this way (a type 1 error is even called an alpha-error!) Thanks for your feedback!

Jim Frost says

Hi Sean, thanks for your comment. Yes, you’re absolutely correct. The significance level (alpha) is the type I error rate. It’s the probability that you will reject the null hypothesis when it is true. However, the p-value is not an error rate. It’s a bit confusing because you compare one to the other.

In the post above, I provide a link to a post where I explain significance levels and p-values using graphs. I think it’s much easier to understand that way. I’ll explain below, but check that post out too.

Both alpha and p-values refer to regions on a probability distribution plot. You need an area under the curve to calculate probabilities. You can calculate probabilities for regions, but not a specific value.

That works fine for alpha. If the null is true, you expect sample values to fall in the critical regions X% of times based on the significance level that you specify. For p-values, the problems occur when you want to know the error rate for your specific study. You can’t do that for a single value from an individual study because you need an area under the curve.

The best you can say for p-values is: if the null is true, then you’d expect X% of studies to have an effect at least as large as the one in your study. X = your P-value. Notice the “at least as large.” That’s needed to produce the range of values for an area under the curve. It’s also means you can’t apply the percentage to your specific study. You can apply it only to the entire range of theoretical studies that have an effect at least as large as yours. That range collectively has an error rate that equals the p-value, but not your study alone.

Another thing to consider is that, within the range defined by the p-value, your study provides the weakest results because it defines the point closest to the null. So, the overall error rate for the range is largely based on theoretical studies that provide stronger evidence than your actual study!

In a similar fashion, if you reject the null for your study using an alpha = 0.05, you know that all studies in the critical region have a Type I error rate = 0.05. Again, this applies to the entire range of studies and not yours alone.

I hope this all makes sense. Again, read the other post and it’s easier to see with graphs.