The F-test of overall significance indicates whether your linear regression model provides a better fit to the data than a model that contains no independent variables. In this post, I look at how the F-test of overall significance fits in with other regression statistics, such as R-squared. R-squared tells you how well your model fits the data, and the F-test is related to it.
An F-test is a type of statistical test that is very flexible. You can use them in a wide variety of settings. F-tests can evaluate multiple model terms simultaneously, which allows them to compare the fits of different linear models. In contrast, t-tests can evaluate just one term at a time.
Read my blog post about how F-tests work in ANOVA.
To calculate the F-test of overall significance, your statistical software just needs to include the proper terms in the two models that it compares. The overall F-test compares the model that you specify to the model with no independent variables. This type of model is also known as an intercept-only model.
The F-test for overall significance has the following two hypotheses:
- The null hypothesis states that the model with no independent variables fits the data as well as your model.
- The alternative hypothesis says that your model fits the data better than the intercept-only model.
In statistical output, you can find the overall F-test in the ANOVA table. An example is below.
Related Post: What are Independent and Dependent Variables?
Interpreting the Overall F-test of Significance
Compare the p-value for the F-test to your significance level. If the p-value is less than the significance level, your sample data provide sufficient evidence to conclude that your regression model fits the data better than the model with no independent variables.
This finding is good news because it means that the independent variables in your model improve the fit!
Generally speaking, if none of your independent variables are statistically significant, the overall F-test is also not statistically significant. Occasionally, the tests can produce conflicting results. This disagreement can occur because the F-test of overall significance assesses all of the coefficients jointly whereas the t-test for each coefficient examines them individually. For example, the overall F-test can find that the coefficients are significant jointly while the t-tests can fail to find significance individually.
These conflicting test results can be hard to understand, but think about it this way. The F-test sums the predictive power of all independent variables and determines that it is unlikely that all of the coefficients equal zero. However, it’s possible that each variable isn’t predictive enough on its own to be statistically significant. In other words, your sample provides sufficient evidence to conclude that your model is significant, but not enough to conclude that any individual variable is significant.
Related post: How to Interpret Regression Coefficients and their P-values.
Additional Ways to Interpret the F-test of Overall Significance
If you have a statistically significant overall F-test, you can draw several other conclusions.
For the model with no independent variables, the intercept-only model, all of the model’s predictions equal the mean of the dependent variable. Consequently, if the overall F-test is statistically significant, your model’s predictions are an improvement over using the mean.
R-squared measures the strength of the relationship between your model and the dependent variable. However, it is not a formal test for the relationship. The F-test of overall significance is the hypothesis test for this relationship. If the overall F-test is significant, you can conclude that R-squared does not equal zero, and the correlation between the model and dependent variable is statistically significant.
It’s fabulous if your regression model is statistically significant! However, check your residual plots to determine whether the results are trustworthy! And, learn how to choose the correct regression model!
If you’re learning regression and like the approach I use in my blog, check out my Intuitive Guide to Regression Analysis book! You can find it on Amazon and other retailers.
Note: I wrote a different version of this post that appeared elsewhere. I’ve completely rewritten and updated it for my blog site.
Share this:
Reader Interactions
Comments
Comments and Questions
Hi Dr. Jim.
What happen when the omibus and F test are significant (p < 0.05) but some model assumptions (normality, autocorrelation etc) are not meet?
Thank for your reply!
Thanks Jim, was searching for this explanation the whole day!
Hi Jim,
Thank you very much for your detailed posts, they have been most helpful.
I too am wondering about contradictions between overall model fit, F, and independent variable significance. I have a moderation regression model that shows nonsignificant F but significant interaction. My instinct tells me that I cannot interpret the interaction without a significant overall model, as F indicates that the model with the interactions does not significantly explain any more variance than a model with no independent variables. This would be like running post hoc tests after a nonsignificant ANOVA. However, I have also found papers which suggest the contrary, and that in the case of moderation, if the interaction is significant and was planned or predicted that this can be interpreted.
I was wondering if you could point me in the right direction to understand this better.
Many thanks,
Tom.
Hi Jim,
first of all thank you very much for all your videos and this website in general! You carry the analysis and interpretation part of my bachelor thesis!
Nevertheless, I’ve got one question.
You say here
“This disagreement can occur because the F-test of overall significance assesses all of the coefficients jointly whereas the t-test for each coefficient examines them individually. For example, the overall F-test can find that the coefficients are significant jointly while the t-tests can fail to find significance individually.”
In my paper I have only one independent variable, not multiple, hence a bivariate regression model. If I understand your statement above correctly, the F-test is important to test overall significance when there are multiple IVs. Therefore, the correlation coefficient R, the coefficient of determinantion R-squared, the regression coefficient beta and the corresponding p-value of an conducted t-test should be sufficient to interpret the results of my study, right?
Thank you very much in advance for your answer!
Greetings from Vienna,
David
Hi David,
Yes, you’re correct on all counts! In fact, the p-value for the coefficient’s t-test and the overall F-test should be the same when you have one IV (aka simple regression).
Hi Jim,
I created a regression model in Excel with the objective of determining the probability of a customer defaulting on payment. I am not clear on how to use the regression output to assess that probability. In Excel column, I have the number of invoices per month and in a second column, I have the number of default payments per month. When I ran the regression, I used the NSF volumes as the Y variable and the invoice volumes as the X variable. I set the alpha to 5% and the p-values for both the volume of invoices and the intercept is less than 5%. I don’t know if this is sufficient to determine the probability of default. Do I need to do something else? I was thinking of using binary; for example, 0 for no default and 1 for default. Could you please provide some guidance on how I can use the regression to determine probability of default? Much appreciated.
Thanks,
Tony
Hi Tony,
To be able to calculate the probability of defaulting, you’d need a binary dependent variable that indicates whether the customer defaulted or not. You can have a mix of continuous and categorical predictors. Then use binary logistic regression. Using that type of model, you can determine the probability of a customer defaulting based on the values of the predictors. Unfortunately, you can’t use Excel’s built-in regression function to perform that type of analysis. it’s not valid for the binary dependent variable.
It’s harder to perform binary logistic regression in Excel. I think it can be done using its analyzer function, but that involves some set up. Down the road I might right a post about that. But any other statistical package can do logistic regression.
Hi, how do you comment on the appropriateness of the overall significance of the model, thanks
Laureen
Thank you so much Jim.
I am agog with anticipation.
Hi Jim. I LOVE your site so much. It’s so well written, I don’t think I’ve ever NOT understood one of your posts. Which is why I’m really hoping you’ve either written, or could be persuaded to write a post explaining what I know to be true but have never really got my head around – ANOVAs and regressions are the same. Waiting in hope of enlightenment!
Hi Chris,
Thanks so much! I really appreciate your kind words!
I have not written such a topic yet, but I’ll add it to my list! You’re correct, linear regression and ANOVA are two sides of the same coin. Stay tuned!
Hi Jim,
Thanks for the confirmation!
I really appreciate it.
Thanks again,
Josh
Hello Jim!
Is my understanding correct that in a multiple linear regression, Y = B_0 + B_1*X_1 + B_2*X_2 + B_3*X3 + e, wherein B_0 = constant term, B_i = coefficient of ind. var., X_i = ind. vari., e =error term.
Am I right to assume that I can use the f-test to determine if there is a significant difference between the variables by stating my null hypothesis as B_1 = B_2 = B_3 = 0 ?
Thank you
Josh
Hi Josh,
Yes, that’s correct. The overall F-test assesses whether all coefficients jointly equal zero (the null hypothesis). When you reject the null, you favor the alternative hypothesis, which states that they don’t all equal zero.
Hello Jim,
I enjoyed looking over your post! Statistics is my weakness but your posts always help clear topics up for me. I was hoping you could help me interpreting a finding from a study I am reading. In the study the authors evaluated the drug effect, time effect, and drug x time effect for improvement in specific symptoms of a disease state. It looks like they found a statistically significant drug x time effect (p = 0.02), but they also provide a F value which is lower than the F critical value. I am wondering if this is truly statistically significant or not. Please let me know what you think!
Thanks,
Doug
Hi Doug,
I’m not sure what is going on there! Be sure that you’re comparing the right p-value to the right F-test. With categorical factors, you can have F-tests for specific terms in the model that are not the overall F-test. It’s also possible to have significant terms in the model but have the overall F-test not be significant. I discuss some reasons that can happen in this post.
I’m not sure what is happening in that study, but be aware that one model can have multiple F-tests when dealing with categorical factors and they can have differing degrees of freedom, which changes the critical value.
Dear Jim,
I am a bit confused about the degree of freedom. How is the degree of freedom assigned to the terms in regression analysis. For example, in the table shown above, North, South and East are provided with a DOF of 1. Why 1?
Hi Gururaj,
Continuous independent variables each use 1 DF because you’re estimating one parameter, the coefficient. Conversely, categorical IVs can use more than one DF, depending on the number of levels the categorical variable has.
For this example, North, South, and East or continuous IVs, so they each use 1 DF.
Hi Jim,
I have used a package to determine the best 6 of 30 independent variables for 11 cases for 1 dependent variable. The quadratic model, going out to the J’th term (most for 11 cases) generates a regression MS of 683 and residual SQ of 0.0015158, therefore the p value is very very small.
Is this enough info to produce alpha = 0.05 confidence intervals around the estimates within the range of the dependent varialbe?
Hi Kerry,
Using this automated method for selecting a model isn’t recommended. Please read my post about specifying the correct model. After reading about your approach, I’m worried that you might be overfitting your model or datamining. I talk about those problems and have links to them in my model specification article.
I hope that helps!
how can we find F-values using full model
Dear Sir,
I have two different models with different f-statistic. Model 1 has only two independent variables and they’re all statistically significant whiles model 2 has five independent variables but only two are statistically significant. Model 1 has f-statistic of 132.198 and that of model 2 is 54.125. Please with respect to the f-statistic, which of the models is better?
So if the p value for the f-statistic is larger than the significant level, that means my sample data does not fit my regression model better?
Hi Sarah,
That’s correct. You want a low p-value, less than your significance level (e.g., 0.05), to indicate a statistically significant model. However, check the residual plots and the p-values for your independent variables. If your residual plots have patterns, you might be able to improve your model (and make it significant) by adjusting it, such as fitting curvature. Additionally, if you have at least one IV that is significant (p < 0.05), then that variable is still significant and you have some findings. If you have a significant variable but some are not significant, you might consider removing the non-significant variables. While you'd hope for a significant p-value for the overall F-test, there are other things to look for and try.
Hi Jim,
Thanks for the confirmation/feedback and the additional information.
It makes sense that if the F-statistic is larger than 1 and if the model is statistically significant, that the prediction is better than the sample mean (i.e, better than the intercept-only model).
However, in terms of the accuracy of the prediction of the model (assuming for simplicity only one independent variable), the prediction will be limited by the Residual Standard Error (RSE) of the regression model, right?
That is, the prediction of the linear regression model will be better than the sample mean, but the accuracy, on average, will be bounded by the RSE of the regression model, for any value of the predictor variable? Is this the correct interpretation?
Thanks again for your valuable feedback!
H-
Hi Humberto,
Yes, that’s correct. Just bear in mind that you need to make an allowance for random error. You can have an F-value greater than one due to random error. Hence, the need for the hypothesis test rather than just going straight to looking for F-values over 1.
Also, read more about this topic in my post about using the standard error of the regression, which is closely related to the RSE. It measures the precision of the predictions.
Hi Jim,
Since the F-statistic = (MSR)/(MSE),
MSR = Mean Square Error of Regression (i.e, for the model)
MSE = Mean Square Error for the Residuals
Does that imply that for any regression model with an F-statistic value > 1, the prediction is better than the sample mean?
Thanks in advance for the clarification.
Regards,
Humberto-
Hi Humberto,
Yes, that’s basically it.
One additional point to keep in mind is that the test also accounts for random error. Let’s assume you have model that is NOT better than using the mean to predict the response for the population. Random sample error might cause you to observe an F-statistic that is greater than 1 in your sample. The test also factors that in random error. For more details about this aspect, read my post about the F-test in One-Way ANOVA. While that is a different use for the F-test, the ideas are similar, you’re just using different values in the F ratio. In the later parts of that post, you’ll see how F-tests in general incorporate random error.
The point being, you can have an F-value greater than 1 that is still not statistically significant.
Thank you.
Hi Jim,
This was very helpful, thanks! However, I did have a doubt and would love your opinion on it:
My model has seven independent variable with one dependent variable.
My regression results show that the p-value>alpha of 0.05 for three variables. Hence, they cannot be included in the model. However, the ANOVA test shows a significant f-test result and the p-value here is less than this f-test result.
My question now would be, how do I interpret this?
My confusion arises due to the fact that in such a case, the null hypothesis [The seven independent variables (Work Environment, Rewards (Monetary), Rewards (Non-Monetary), Learning and Development, Work Benefits, Relationship with Peers & Promotion have a positive relationship with the dependent variable (Job Performance)] is rejected. Unfortunately, I cannot reject it because three variables cannot be included.
Nazia
Hi Jim, I’m looking at a simple robust regression model with only one predictor (using fitlm in Matlab). I have an significant t-test but significant f-test. How should I interpret this? Thanks
Linear regression model (robust fit):
y ~ 1 + x1
Estimated Coefficients:
Estimate SE tStat pValue
__________ ________ _______ __________
(Intercept) 0.23946 0.03467 6.9067 1.9063e-11
x1 -0.0013939 0.001356 -1.0279 0.3046
Number of observations: 411, Error degrees of freedom: 409
Root Mean Squared Error: 0.146
R-squared: 0.0157, Adjusted R-Squared: 0.0132
F-statistic vs. constant model: 6.5, p-value = 0.0111
Hi,
Ah, it’s a robust regression model. I’m not super familiar with this form of regression. The most common reasons I know of for using robust regression are when you have outliers that you can’t remove or heteroscedasticity. If those don’t apply to your data, you should use ordinary least squares regression. When you have one predictor in least squares regression, the overall f-test and the t-test for the predictor are equal. Apparently, that’s not the case with robust regression.
I’m not sure how that works out that you can get a significant f-test but a non-significant predictor when you have only one predictor. However, in general, you have a weak model. It doesn’t explain much of the variance (less than 2%) and the predictor isn’t even close to significance p = 0.30). Even though the f-test is significant, your model sure doesn’t look significant to me.
Hi Jim, your website is very helpful to understand statistics. I visit this website everyday. Thanks a lot!! I have a simple questions.
1) Slope β1 is significant, but F value is not significant – How can I interpret this?
2) The other way around, slope β1 is not significant, but F value is significant – How can I interpret this?
Many thanks!!!
Hi JK,
For your first question, I can infer that your model must have more than one IV and that at least some of them are not significant. If you have one IV (simple regression) that is significant, the overall F-test will also be significant. However, if you add IVs that are not significant, they can “dilute” the significance of the entire model. Additionally, including IVs that are not significant can reduce the precision of your model. You might consider removing those variables. However, if you have theoretical reasons to include them, or the goal of your study is to test those variables specifically, it is OK to leave the insignificant variables in the model.
For the second question, you can think of it in terms of having enough evidence to conclude that your model as a whole is an improvement over using the mean of the DV to explain the variability of the DV. However, you don’t have enough evidence to determine the specific variables that are statistically significant. For a more technical explanation of how this happens, focus on the middle portion of the post where I explain different between the joint versus individual tests of significance in this context.
I hope this helps!
Dear Jim,
I think I have similar problem as Laura. Should I do the same?
I have multiple regression with five independent variables. Four of them are insignificant, but one is significant (sig. 0,007). However, ANOVA F test sig. is 0,062, which means (if I am not wrong) that none of the independent variables are significant. Adjusted R-squared is 0,335. VIFs are all ok.
I have 19 companies in the sample.
If Ho is “regression is not statistically significant”, while H1 is “Explanatory variables have a statistically significant effect on the movement of the dependent variable (at least one of the parameters is significantly different from zero)”.
ANOVA sig. suggests that I need to accept H0, but is that correct if I have one explanatory variable which is significant.
Many thanks in advance.
Hi Boba,
Typically, you’d consider removing the insignificant variables from the model. That should help make the overall F-test significant. The entire model becomes stronger without unnecessary variables that are not significant. Include insignificant variables only when there are strong theoretical reasons for doing so or the purpose of your study is to test their significance specifically.
Also, recall that you do not accept the null hypothesis!
Unstandardized Coefficients Standardized Coefficients 95% Confidence Interval for B
B Std. Error Beta t Sig. Lower Bound Upper Bound
Constant 6.67 3.07 0 2.17 0.968 0.61 12.73
Var0003 0.15 0.03 0.41 5.7 0 0.1 0.2
How can I interpret the data when my Sig is 0.968 for the constant
Hi Nancy,
Please read my article about how to interpret the constant. If you have any additional questions after reading that, please post in the comments section for that article. Thanks!
Hi i hope my comment will finds you well.. Sir i am M Phil scholar currently my research is on stone crusging industry impacts on human health expenditures and environment.. so basically i am calculating environmental pollution based health cost of individuals..The model is given bellow
Y= a+bixi +b2zi +b3wi + b4cri +b5ri +d1+d2+ui
so the independent variables Xi is health condition while zi is socio-demographic characteristics.. Wi and cri are variables of water and crops respectively..similarly ri is radius which means that how much (distance) a respondent is living away from stone crushing industry..d1 is dummy variable for uncontrolled group while d2 is dummy variable for controlled group..
the model i am using for it is propensity score matching.. but i dont know to use the propensity score matching in spss..This is what the problem actually i am facing now.. So kindly help me out..that i could procees my work..
Thank you
Hi Syed, I have not used propensity score matching in SPSS so I wouldn’t be much help there. Sorry.
Hi Jim.
I’m identifying Environmental potential determinants of dengue illness. I used logit model 1st but all the coefficient of variables are negatively correlated and statistically significant and also I used various test and techniques such variables dropping one by one as well data censoring from the above, below and from middle but everything were still same.
My DV in binary form (1,0) 1 = dengue affected people and 0 = normal people.
IVs some are continuous and categorical.
Now I used the LPM model all variables coefficient are corrected but the coefficient of constant is negative ant t-value also negative, R squared is 0.65.
Most of my friends suggest me logit model instead of LPM because they tell me the results of LPM is not good.
Thanks!
Hi Khan,
I have not used the Linear Probability Model (LPM) myself. However, analysts will typically use logit and probit models more frequently. The LPM model is relatively simple and inefficient model. It can also produce probabilities for values that fall outside of 0 and 1.
Unless you have a very strong reason for using LPM, I’d used a logit model instead. Both models can tell you similar types of information. However, for the reasons I mention, logit models are typically the better choice.
I hope this helps!
Hi Jim,
I have a different scenario, my ANOVA reveals that F(3,1) = 31.88 and p>0.05. Additionally, my all the explanatory variables (3 variables) are insignificant.
1. Can I still use the coefficients for predictions?
2. If so or if not, why?
My adjusted R Square is 0.959 and my data set consist of 30 observations.
PS: This is for a social science thesis.
Thanks in advance.
Hello Jim
Thanks a lot for the reply.. It did help.
Hi Jim,
Please help.
I have calculated ANOVA (one way) using ORIGIN 8 software.
The degree of freedom for Model is 2 and error is 33 and corresponding mean square is 1.20 and 223.9, which gives the F Value as 0.054, with prob>F is 0.99.
I was wondering if I can get so low F value, when the corresponding critical value will be 3.2
Hi Rashmi,
If a p-value equals one, then you know that the sample data exactly equal the null hypothesis condition. Your p-value is 0.99, so you know that the sample data almost exactly equals the null hypothesis value. For one-way ANOVA, the null hypothesis states that all group means are equal. So, your sample means must be almost exactly equal. You should be able to have your software calculate the mean for each groups and you should notice how that set of group means almost have the same value.
I hope that helps!
Hi Jim,
Thank you for getting back to me. In the t-test that I conducted, I only compared two IVs, and they were statistically significant from each other with a p-value of 0.006. Confidence interval was set at p<0.01.
The R-squared value was 0.1837.
I'll am using a program called prism, I have never used it before. I only have experiences with SPSS. I'lll see if I can check the residual plots.
Hi Jim,
I am currently having the opposite issue. With the data that I have collected, the t-test came out to be significant. However the F-test demonstrated that there was no group level significance.
I am still trying to figure out what my results mean in that the groups that I measured individually were different, but that the whole entire data groups were not different? Could use some assistance here.
Hi Pablo,
Do you have any independent variables in your model that are not significant? If you have a mix of significant and non-significant IVs, I could see the overall F-test potentially not being significant. However, if all of your IVs are significant, I’m not sure why your F-test wouldn’t be. What is your R-squared? How many observations and how many variables and other terms? Did you check the residual plots to be sure you’re not missing curvature or other patterns that you can model better?
Thanks! Really helpful!
Hello
I have a multilinear regression model with 2 independant variables. The overall model is signficiant (p<0.05) but the individual independant variables are not. HOWEVER, if i perform a simple linear regression with each indpendant variable individually (or indeed a pearson's correlation), each variable does significantly correlate with my dependant. I don't understant how they can be sigificant on their own, and significant together in terms of the model but not significant in the coefficients table?! I have a sample size of 35 btw.
Thanks in advance!
Charlotte
Hi Charlotte,
This condition definitely happens! Chances are that the two independent variables are correlated. You should check their pairwise correlation. If it’s too high, you might have problematic levels of multicollinearity. Read my post about multicollinearity to learn about it and how to check it using VIFs. If there is a high correlation between IVs, it can weaken the power of the statistical tests by causing higher p-values.
Even if the multicollinearity isn’t at problematic levels (check those VIFs), your two independent variables are probably still correlated to some degree. Because they’re correlated, they’re both supplying some of the same information. Further, when they’re both in the model, each IV is estimated while controlling for the other variable. Neither variable gets “credit” for the shared portion of the variance that they explain with the other variable. Given your results, the unique portion of each IV might not be strong enough to be significant when they’re in the model together. However, without the other variable in the model, it can stand by itself.
If multicollinearity is a problem, there are methods that can help there, such as LASSO and Ridge regression. If the IVs are correlated but not high enough to be problematic, you might just need a large sample size to detect significant effects with both in the model.
I hope this helps!
Hi Jim,
I am working on panel data and doing the regressions with R-studio. The interest of the study is to see whether the variable x has a negative/positive effect on y from all the independent variables. The question is whether the two-tailed test is valid to see this relationship, and can conclusions be made if the coefficient of the x is negative or positive (while having a p-value<0,05)? Furthermore, the F-statistic is significant (having a p-value of 0,0002) while the adjusted R-squared is negative(-0,1), and it leaves me to wonder if it is caused by the model used or something else. The "within" model for fixed effects is used and does not provide an intercept.
Best regards,
Niklas
Hi Jim,
i really need your help as i didn’t find any information after several days on this topic.
For my master thesis, i’ve done several multiple regressions, where two models got not significant (p=0.127 and p=0.194), BUT some of the indivdual predictors (p<0.05).
How can i interpret this?
My R-sqaured is 0.6 which seems to be quite good (according to Cohen).
Could you please help me to unterstand this ?
Best regards,
Laura
Hi Laura,
If not all of the predictors are significant, that might be bringing the p-value for the F-test down. The F-test is for the overall model. Given the good R-squared but the insignificant overall F-test, I’m guessing that your model has more predictors that are NOT significant than are significant and/or you have a small sample size. In either case, consider removing the non-significant predictors. Start with the one that has the highest p-value and remove it. Refit the model and proceed to remove the next. Repeat. This process is called reducing the model. However, don’t remove an insignificant predictor if you’re specifically testing it or if theory or subject-area standards say it’s important to include.
When you include too many predictors that are not significant, it can reduce the precision of your model. I suspect that with mostly significant predictors, the overall F-test will also be significant.
Best of luck with your analysis!
I’m struggling to understand my f statement. F (1,18) =.381, p=.001, R2=.021
I understand that the p value is showing that my results are significant, and that r squared is showing that the effect size is low, but what is the f value showing?
Hi Jim,
My regression output shows me the following:
R-squared: 0.1130
Adjusted R-squared: 0.038
P-value (F) = 0.047
The P-value is less than 0.05, which tells me that my explanatory variables together provide significant explanatory power (so R-squared is significantly different from zero).
My question now is: “Does the P-value of the F-statistic also tell me that the adjusted R-squared is significantly different from zero or does it only relate to the normal R-squared?”
Thanks in advance!
Best Regards,
Simon
Hi Simon, the overall F-test applies only to the regular R-squared. However, the R-squared is a biased sample estimate of the population value while the adjusted R-squared is an unbiased estimate of the population value. So, the true population value is likely to be non-zero but probably closer to the adjusted R-squared value than the R-squared value.
Hi Jim
I have run a linear regression analysis in SPSs. My adjusted R squared is 92.1. Sample size is 334.
I get a F value of 237 significant at .000.
What does a very high value of F suggest and how do I interpret it. Please help.
If my p value is larger than alpha, what does my f stat value have to be to also suggest failing to reject the null? My p value is larger than my alpha but my professor said what’s the f value. I have the f values from the ANOVA but I don’t understand enough to interpret the values.
Hi Aujah,
The p-value is based on the F-value. Typically, you don’t interpret the F-value directly. Your statistical software takes the F-value, factors in the degrees of freedom, and then uses that information to calculate the probability, which it presents as the p-value.
So, I am a little puzzled as to why your professor wants the F-value. By itself, it is fairly meaningless. It’s only after further calculations to produce the p-value does can you interpret it. I write about how the F-test works in ANOVA. It’s the same test but in a different context. Reading my other article will help you understand the relationship between the F-value and p-value. And, show you what I mean by incorporating degrees of freedom and calculating the probability.
At any rate, just focus on the p-value for the F-test.
Hi Jim,
I am struggling to understand the relationship between the factors that can influence the F-Statistic. For example, if I were to double a sample size from 1000 to 2000, how would this influence the F-Statistic and why?
Hi Ali,
To understand what to expect, you have to know whether the relationships that your model describe actually exist in the population. If those relationships truly exist, then as you increase sample size, you’d expect the F-value to increase and its p-value to become smaller (more significant). The reason why is that because these relationships exist, the increased sample size gives the hypothesis test more power to detect them.
However, if those relationships do not exist, then F-values should decrease and the p-values increase (less significant). The reason why for this case is because those relationships do not exist, increase the power to detect them does not help. There’s nothing to detect.
Hi Jim,
I have a p value less than 0.05 and an F value of 84.42. Does this mean that I accept the null hypothesis or the alternative hypothesis?
Thanks, Tash
Hi Tash,
In a hypothesis test, you have two choices:
1) Fail to reject the null
2) Reject the null
Assuming your significance level is 0.05, you can reject the null because your p-value is less than it. Your model explains more of the variance around the dependent variable than just using the mean of the dependent variable. That’s a good sign!
And, a quick clarification, you never “accept” a hypothesis. I know that sounds strange, but read my post about failing to reject the null hypothesis for more information.
Good evening Jim, I am working on an RD design and am having problems interpreting the data. I was wondering if you could help? One question that I have is how small can an R squared value be and still be ok? For example:
Residual standard error: 0.01852 on 3968 degrees of freedom
Multiple R-squared: 0.0122, Adjusted R-squared: 0.0112
F-statistic: 12.25 on 4 and 3968 DF, p-value: 6.702e-10
The p-value on my F-statistic shows that it is significant but the R-squared value is only .0122? Is that telling me that my model does not fit the data well?
Hi Russ,
Technically, your F-test of overall significance indicates that your R-squared is significantly different from zero. However, your model barely explains any of the dependent variable’s variance. Were any of your IVs significant?
Hi Jim, thank you for this post. I’ve run into a problem that doesn’t seem easily searchable. I have two regression models with the same dependent variable. They each have 3 independent variables, 1 of which differs between the models while the other 2 are the same. The 1 variable that differs between the two models is a continuous variable. I am trying to determine if either model is significantly more predictive of the dependent variable than the other. Do you know how I might to do that?
Hi Steven, because they have the same dependent variable, you can use the standard error of the regression (S) to compare how well each model fits using the units of the DV. For more information, read my post about the standard error of the regression.
Hi, I need some help, please. I have got F value in ANOVA (I did spss regression) = 67,662. How can I interpretate this result ? Because I got Sig in Anova = .000, my R square for multiple regression is .936. Altough in table coefficients on variable has Sig .0001, beta .5, and I got one more with Sig .000 and beta=.6.
Thank You.
Best regards,
MR
Hi Marko,
Typically, you don’t interpret the F-value directly, but instead the p-value associated with it. For the F-test, your p-value of 0.000 indicates the model as a whole is statistically significant. Additionally, it looks like your independent variables are also significant. The R-squared is also high. It looks like good results overall. However, check the residual plots to be sure there aren’t any problems.
Hi Jim,
Thank you so much for your valuable post, is this F -test only suitable for linear regression, or it also can be suitable for nonlinear regression? I use Tablecurve 2.0 to fit the data, it also give a output like this for nonlinear model?
Thank you very much.
Grace
Hi Jim,
Please how do I interprete a multiple regression model whose result yielded: multicollinearity absence, positive autocorrelation and heteroscedasticity presence.
Here’s the model:Y =76.891X1+ 0.006X2- 521308.877
Hi Oladayo,
I recommend you read my post about OLS assumptions to see how the autocorrelation and heteroscedasticity affect your results.
For help about interpreting the equation, read my post about how to interpret regression coefficients. Although, beware of the effects of violating the assumptions.
Hi Jim,
thank you very much for your excellent posts. I have a question. I am doing some multiple regression analysis for my thesis to investigate the impact of CSR on ROE with some other control variables. If the F test is significant (p less than 0.05) but the coefficient of CSR is not significant (p larger than 0.05) can I conclude that the null hypothesis for this predictor is accepted ? I mean CSR doesn’t impact ROE.
Many thanks for your attention.
Best regards
Dam
Hi Jim,
How can I determine if the difference between two independent sample regression model from the same population are significant when I create two subsamples male and female.
Hi Fieze,
Read this post about comparing regression models and use gender as the indicator variable as I describe. Then, check the significance of the gender variable.
Best of luck with your analysis!
Hi Jim! Thank you for the above answer. I have a little different question. Since F-statistic is obtained from F-distribution which in turn is ratio of two ch-squared variable with the respective degree of freedoms.
Some nomenclature:
TSS = Total sum of squares = variances of y from its mean
RSS = Residual sum of squares = summation of residuals
p= number of predictors
n= number of observations
F =((TSS-RSS)/n)/(RSS/(n-p-1))
E[RSS/(n-p-1)] = sigma^2 (i.e variance of error term). [This i know]
How to obtain that under the null hypothesis i.e when all regression coefficients are zero
we get E[(TSS-RSS)/n] = sigma^2 ?
Thank you!
Thank you for your clear and helpful answer. It helped me a lot !
Thanks so much for the article.
I wonder what’s the difference between F-test for variance and F-test in regression.
F-test in variance: https://www.youtube.com/watch?v=g9pGHRs-cxc
Is that they’re the same kind (F-test), but a bit modified for each use-case? Thanks
Hi Chau,
The F-test is a very flexible type of hypothesis test. This test assesses whether the variances in the numerator and denominator are equal. By changing the variances that you include in the numerator and denominator, you change the purpose of the test. As the video shows, you can simply test to determine whether the variances of two populations are different. You put your variance estimate for one population in the numerator and the other variance estimate in the denominator.
However, if you change the numerator and denominator, you’re assessing different things. The F-test of overall significance in regression analysis uses the following variances in the numerator/denominator:
F = Variance the model explains / Error (Unexplained) variance
So, it’s the same test but you change the variances that you use. In turn, that changes the meaning of the results. I discuss how to interpret the F-test of overall significance in regression in the post so I won’t restate it here.
For yet another application of the F-test, see how ANOVA uses the F-test to determine whether the mean of three or more groups are different. It’s still an F-test but is uses different variances in the numerator and denominator.
Thanks for the great question!
Hey Sir, Appreciate your effort .I have one question that I m analyzing cross sectional data related to determinants of child nutrition.By using OLS technique, it will not give consistent estimate because of endogenous variable.So I have to use 2SLS .But when I first stage regression by regressing endogenous variable on all other independent variable and my instrument ..The F value increased as 4540.17 with n=12834 ?Can you explain what is the problem?
HI Mani,
You might not have a problem. You have an extremely large sample size which gives the F-test a great deal of statistical power and explain the large F-value.
Hi Jim,
Thank you for the post. It is very clear and helpful! Quick question, if the F-test indicates no significance of the model, does it mean the model is completely useless? Can I still look at the Intercept or residuals to say something about the data?
Hi, thanks! I’m glad you found it to be helpful!
Unfortunately, there’s not usually much you can say when the overall model is not significant. You can assess the residuals to double-check that you’re not missing curvature that might be present in your data. If that’s the case, you can fit the curvature and possibly obtain a significant model.
Occasionally, the overall F-test can be insignificant but an IV in the model can still be significant. That disagreement doesn’t happen often, but when it does, you can still say that the IV is statistically significant and draw conclusions about that relationship.
While I’m sure that you were hoping for a significant model, insignificant results can be helpful too. It helps stop you from jumping to conclusions. I write about this in my most recent post about how high p-values can be meaningful. I write about it in the hypothesis testing context (e.g., 2-sample t-tests and the like) but it also applies to other tests, such as the F-test of overall significance for linear models.
Hey Jim, Have you published your book on regression analysis? if yes please provide me the link.
Hi Balbir,
Not quite yet but it’s oh so close! Just a week or two from now! I have one small hitch at the end.
Hey Jim, I am new in using regression equation. can I test 1% to 5% significance level using this “F test”
Hi Fokhrul,
Yes, you can use either of those standard significance levels. 5% is more common!
Hi, this is helpful but I still struggle in interpreting the results using STATA and how to implement it correctly, any advice?
Hi Rema, if you’re refering to difficulties with regression analysis in general, you’re in luck, I’ll be publishing my ebook on regression analysis in March 2019. I think that’ll be very helpful for you. I cover all aspects of it in a very intuitive manner.
Hi Jim,
Thanks so much for the great posts, really helping me study for my stats exam!
I’m having a hard time grasping what exactly the F-test is testing. I read that the null hypothesis is that all group means are equal when using ANOVA, and here it says that it is that the model with no independent variables fits the data better than one with independent variables. Are these two things equivalent?
Both interpretations were used in my lecture simultaneously and not really differentiated so I am a bit confused. Would appreciate some clarification, thanks a lot!
Hi Anne-Lise,
You’re very welcome! 🙂
The F-test is a very flexible test. In its most general sense, the F-test takes a ratio of two variances and tests whether the ratio equals 1. A ratio of 1 indicates that the two sets of variances are equal. A ratio greater than one suggests that the numerator is greater than the denominator. Statistical significance indicates that you can reject the null hypothesis that the ratio equals 1.
By changing the variances that are in the numerator and the denominator, you change what an F-test assesses. And, that’s leading to your confusing because there are different possible uses for F-tests depending on how it’s set up.
For an overall F-test, the ratio contains the variance accounted for by your model in the numerator and the variance accounted for by a model with no predictors in the denominator. If the ratio is significantly greater than one, you know that the numerator is significantly greater than the denominator. Hence, based on what is included in the numerator and denominator in the overall F-test, you know that your model (numerator) provides a fit that is significantly better than one without any predictors (denominator).
However, the F-test is set up differently in the ANOVA context by including different variances in the numerator and denominator, which changes your interpretation of the results. I’ve written about how the F-test works in ANOVA. As you read through that post, notice how the numerator/denominator are different than what I describe above for the overall F-test.
To answer you question, are these two things equivalent? They both use the same underlying test, but the variances in the numerator/denominator are different. In one sense, the interpretation is the same–statistical significance indicates that the ratio is greater than one, which, in turn, indicates that the numerator is greater than the denominator. However, interpreting numerator > denominator understandably depends on the variances that you include in them.
I hope this helps!
Thank You so much completely reply!!
Hi, Sir
Thank you so much for your reply!!
The overall F-test is compared with their critical value ( critical F-statistic) al alpha value 0.01, 0.05 and 0.1 level, not use p-value. When I test the model by using 0.05 and 0.1 level, it is significant. But, when using the alpha value 0.01 level, the overall model is not significant ( all model is rejected ) . So, how should I interpret for 0.01 level according to theorical terms. I ever see the lover significance level, the more the data must diverge from the null hypothesis to be significant.
Thank you..
Hi, whether you compare the F-value to the critical value or the p-value to the significance level, the results are the same. It’s a different way of doing the same thing. An F-value that equals the critical value is equivalent to the p-value equaling the significance level.
You pick one significance level before performing the analysis and then determine significance. If you compare your results to lower and lower significance levels, you’ll always find that it is not significant at some significance level. You never truly obtain a p-value of 0 (software might round to zero, it’s not actually zero.)
So, pick one level, and then determine significance using only that level, and that’s that. Don’t use more than one significance level.
For your last question, yes, lower significance levels require stronger evidence that the null hypothesis is not true. For statistical significance, you’ll need a larger divergence between the observed value and the null hypothesis value. You can read more about it in my post about the significance level and p-values.
Hi Jim,
If the overall regression model is not significant at alpha level 0.01, how should I interpret about this?
Thank you so much!!
Hi,
If you set your significance level at 0.01 and your p-value for the overall F-test is greater than that value, you fail to conclude that your model fits the data better than a model with no predictors. However, I should point out that a significance level of 0.05 is more common. Although, there can be good reasons for using a lower significance level like you are. A lower significance level requires more evidence that your model is a good fit.
I hope this helps!
Jim,
Thank you so much for your reply! You are great, and thanks to your blog I might have a not only a chance to understand a little more statistics, but also to get better marks for my module!
Have a lovely Christmas!
Regards,
Lore
Hi Lore,
I’m so happy to hear that it has been helpful! 🙂
Merry Christmas to you as well!
Hi Jim,
Thank you so much for replying to my message. I would like to apologise for the quality of the output, and I must say that you’ve been so good in trying to make sense of that mess. I have no idea what sort of software was used, I would say R or STATA, however the bits I’ve posted here were copied from a regression table- hence the poor formatting.
Hi Lore,
I had to edit the extensive output out of your comment because it was so long. But, yes, the overall F-test indicates whether your model is better than a model with no predictors. And the t-tests for the individual variables indicate whether specific variables are significant. For more information about that aspect, read my post about regression coefficients and p-values.
Hi Jim,
Thank you for your post/page and I cannot tell you how much I appreciate that someone like you is trying to make it easier for those like me!
I find myself thrown into the deep end, as I started a phd in political science with a background in law. I’ve never studied statistics and we’ve been literally bombarded with projects and term papers.
My lecturer isn’t the best at explaining, and lots of us are dreading the deadlines. On top of
that we have to score an overall B to pass, and here I am asking basic questions. Oh dear!
Please spare a minute to look at this:
R2 . 033 . 049
Adjusted R2 .031 .043
Residual Std. Error 2.266 (df = 489) 2.253 (df = 487)
F Statistic 16.866*** (df = 1; 489) 8.297*** (df = 3; 487)
*P<.05
**P<.01
***P<.001
Am I correct in saying that the results indicated the the F test shows that the independent variable are significant in both models?
R2 .008 .105 .134
Adjusted R2 .006 .099 .121
Residual Std. Error .928 (df = 497) .884 (df = 490) .873 (df = 486)
F Statistic 3.930* (df = 1; 497) 19.096*** (df = 3; 490) 10.698*** (df = 7; 486)
*P<.05
**P<.01
***P<.001
Would the same apply in this case?
Thank you so much for this.
Regards.
Hi Lore,
That does not sound like a fun situation to be stuck in! Sorry about that!
It looks like those are overall F-tests that are possibly looking at models possibly in a stepwise regression, or at least multiple models. It seems to be adding variables. Although, it’s not completely clear from the output. However, assuming these are overall F-tests, it does not refer to the significance of individual independent variables. Instead, it’s testing whether the model as a whole is better than a model with no independent variables. It’s also testing whether you can reject the null hypothesis that R-squared equals zero. This post talks in more depth about how to interpret these results.
It appears like all the models are significant. Although, again, I’m not entirely clear what is going on. The top set of output appears to assess two models, one with 1 IV and another with 3 IVs. The bottom set appears to assess three models: 1 IV, 3 IVs, and 7 IVs. For the models with 1 IV, the overall F-test does indicate that the single IV is significant. But, you can’t tell about individual IVs with models that have more than 1 IV.
What software produced this output? I don’t like how it doesn’t calculate the exact p-value! That’s poor practice in statistics. (That’s not directed at you, I’m just surprised that the software would not provide the exact p-values!)
I hope this helps!
I would like to express my sincere gratitude for your efforts in making statistics more easily
I have a multiple regression model, and I have values of F test for 6 models and they are range between 17.85 and 20.90 and the Prob > F for all of them is zero, and have 5 independent variables have statistical significant effects on Dependent variable, but the last independent variable is insignificant. and the adjusted R square range between 0.48 to 0.52 . could you please help in interpreting my findings ? and clear my doubt about the insignificant independent variable ( the last one ), whereas F test is significant .
Hi Mohammad,
The overall F-test of significance just tells you whether your model predicts the outcome variable better than using the mean. Your models all do this. In other words, each of your models is a significant improvement over just using the mean value of the outcome variable. The overall F-test isn’t meant to tell you anything more. It doesn’t help you determine which independent variables are significant and should be included. So, it’s not surprising to have a significant overall F-test but an insignificant variable (or even more than one).
Regarding the model with the insignificant independent variable, you’ll have to use a mix of statistics and theory to determine whether to leave that variable in the model. Because its p-value is not significant, you’d often remove that variable from the model. Including insignificant variables in the model can reduce the precision of the model. However, if theory and subject-area knowledge suggest that it’s an important variable that should not be left out, it’s ok to keep it, just indicate the reasoning behind that in any report/paper.
As for adjusted R-squared, I’ll refer you to my post about it: adjusted R-squared.
I hope this helps!
Hi Jim,
I have values for F-statistics ranging from 39.39 to 69.81 for 6 different models with their respective p-values all <0.0001. What would you make of such information?
Hi Jackson,
Typically, you don’t interpret the F-values directly. Instead, you can use the p-values. Because your p-values are less than all common levels of significance, your models are statistically significant. This post tells you what a statistically significant model means.
Hi Jim,
When running the linear model in RStudio I get this results.
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 11063.069 7305.303 1.514 0.1353
mpg -114.743 79.172 -1.449 0.1526
rep78 710.879 322.117 2.207 0.0312 *
headroom -725.636 416.660 -1.742 0.0868 .
trunk 70.113 103.633 0.677 0.5013
weight 4.034 1.532 2.634 0.0108 *
length -84.390 43.982 -1.919 0.0599 .
turn -207.480 131.497 -1.578 0.1200
displacement 16.630 8.995 1.849 0.0695 .
gear_ratio 1642.587 1061.150 1.548 0.1270
—
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 2178 on 59 degrees of freedom
(5 observations deleted due to missingness)
Multiple R-squared: 0.5149, Adjusted R-squared: 0.4409
F-statistic: 6.958 on 9 and 59 DF, p-value: 9.131e-07
p-value: 9.131e-07 is not in the threshold right? but my R-Square and F-Stats have a good high result. I am having a hard time interpretating this result. Could you help me?
Hi Yamila,
Your p-value is written in scientific notation. You need to move the decimal point 7 places to the left. Your p-value is actually 0.0000009131. That’s extremely low and your model is statistically significant.
When i run the regression i took 1 dependent and 2 dependent variable.. After run the regression my results are F =8.385337 and F Significance=0.106549 and Rsquare=0.893450 and p value=0.0027062 so plz tell me according to this results what is the interpretation of R-square and model significance as per probability of F test …
I mean what is the actual conclusion of the interpretation plz guide me…..
Hi Hifza,
Usually you don’t need to interpret the F-value itself. Your statistical software uses the F-value to calculate the p-value, which is what you should focus on. (I don’t know what the F significance value refers to.)
Because your p-value for the overall F-test (0.002) is less than the typical significance level of 0.05, we can conclude that your model explains the variability of the dependent variable around its mean better than using the mean itself. As for how it relates to the R-squared, you have sufficient evidence to reject the null hypothesis that your model’s R-squared equals zero. By most standards, you have a nice and high R-squared value. Fortunately, it all suggests that you have a good model–at least according to the statistics. However, be sure to check the residual plots as well!
Usually, you also look at the p-values for the specific independent variables to see which ones are significant on their own.
I hope this helps. Best of luck with your analysis!
Hello sir,my regression model has the r-square value is 0.392 and p value is 0.00029 ,which is good,but none of my independent variables have good coefficient or are not significant.So , what should i conclude from this.Help me sir.
Hi,
If that p-value is for the overall F-test, it suggests that your model is statistically significant. It is unusual to see such a low p-value for the overall F-test and not have any of the independent variables also be significant. It’s hard to know for sure, but it’s possible that your model has multicollinearity (correlated independent variables). This problem can make significant variables appear to be insignificant. However, multicollinearity does not affect R-squared and the overall F-test–which might explain what that p-value is still significant while the others are not.
To learn if this problem affects your model, read my post about multicollinearity.
I hope this helps!
Thank you for this! I have a question regarding this. Can I use the F test to pick which model fits my data the best? I have tried several different models, and the F test for some of them is significant. Since I have multiple significant models, how do I choose which one to use? Do I choose the one with the lowest significant F test? The p-values for the individuals vary between the models, so picking which model to use really affects the results of my analysis.
Hi Emily,
Determining the best model for your data can be complicated. It will involving looking at more than one statistics–such as the F-test. I’ve written a blog post about how to choose the best model, which goes over a variety of things you should check. That post should answer a lot of your questions. But, don’t hesitate to ask if you have more!
Also, if the p-values and coefficients change dramatically depending on the variables that you include in the model, your model might have multicollinearity (correlated independent variables). This issue can make identifying the best model more difficult. You might want to read about it in my post about multicollinearity.
I hope this helps!
Merci beaucoup, tres concit, a real pleasure…to follow your teaching
Thank you, Emmanuel!
pls give a lecture on the Wald Chi square compared to F-value or f-test
Hello Jim,
I enjoy every bit of your lectures here. Please does it imply, when F-test is statistically insignificant?
Hi Cordelia,
I’m glad that you find these to be helpful!
If your F-test of overall significance is NOT significant, then you fail to reject the null hypothesis. For this test, you can interpret it in several equivalent ways.
You can say that you have insufficient evidence to conclude that your model explains the variation in the dependent variable any better than just using the mean of the dependent variable.
Or, you can say that you have insufficient evidence to conclude that the R-squared is significantly greater than zero.
In short, your model is not explaining the variability in dependent variable to a statistically significant degree.
I hope this helps!
Jim,
Thank you for a great post! I have a question. If my F-value was found to not be significant (p=.069), do I still interpret the individual t-values in the coefficients table? If I do, then there are two variables that are significant (p<.05). I have been searching and having difficulty finding an answer to this!
Thank You
Dawn
Hi Dawn, yes, the results of the different tests can disagree. Despite the insignificant F-test, you can still conclude that your two variables are statistically significant. I’d guess that either you’re leaving insignificant variables in the model and/or those two variables are close to the significance level.
Hi Jim,
if the overall regression model and none of predictor is not significant, how should I interpret the F-value and R-squared?
Hi Nara, unfortunately, when the overall F-test is not significant and none of the predictors are significant, you really have no evidence of any relationships between your model and the response variable.
In terms of how to interpret the F-value, that’s the test statistic for F-tests. The test uses this statistic to calculate the p-value. The F-value is the ratio of two variances. For this type of test, the ratio is: Variance explained by your model / Variance explained by the intercept-only model. As the F-value increases for this test, it indicates that your model is doing better compared to the intercept-only model. When the F-value reaches a critical value, you can reject the null hypothesis. I’ve written about how the F-test works in one-way ANOVA. That post shows how F-values are converted to P-values. That’s a different use of the F-test, but the ideas are very much the same. You just change the variances that are included in the ratio.
As for the R-squared. Because your F-test of overall significance is not statistically significant, it means you have insufficient evidence to conclude that your R-squared is greater than zero. The R-squared value in your analysis might not equal zero, but that’s probably just due to chance correlations rather than a true explanation of the population variance.
Hi Jim,
I’m trying to interpret the results of a general linear model I have run. I have two factors – treatment and date (where the same experiment was repeated on different dates). Both give a significant p value but one has a much higher F value (136 compared to 8). Does that mean the factor with the higher F value is having a greater effect?
Hi Kate, that’s a great question. In a nutshell, no, the higher F-value doesn’t indicate a greater effect. I write about how to identify the most important variables in your model. I talk about it in the regression context, but you can apply some of the principles to ANOVA as well. I think that post will help you with this issue.
Hi Jim. i want to know in ANOVA table of regression analysis if p value is significant, still is there any limit for F value? what if it comes as big as 300 0r 450 etc
Hi Aasia, an F-value is the ratio of two variances. Theoretically, there is no limit to the F value. In terms of the explained variance, the better your model is compared to the intercept only model, the higher the F-value. However, for a specific model with a given number of degrees of freedom in the numerator and denominator, higher F-values occur less frequently.
I hope this helps! Thanks for the great question!
Hi Jim,
Thank you so much for your interesting and easily understandable post. However, I have a question when we have the conflict between overall F-test and significant t-test for each predictor. What should we do if the t-test for some of prediction is non-significant? Should we remove this predictors and fit the model again?
Many thanks,
Duc Anh
Hi Duc-Anh,
Frequently you do remove an independent variable from a model if it is not statistically significant. There are some exceptions to this rule. If you believe that theoretical considerations suggest that the variable should be in the model despite an insignificant p-value, you could consider leaving it in. Additionally, if it is a variable that you are specifically testing in an experiment, you would leave it in to demonstrate the test results.
But, yes, frequently you would consider removing the predictor from the model if it is not statistically significant. Your dataset provides insufficient evidence to conclude that there is a relationship between that predictor and the response.
One more point, be sure to check the residual plots. There might be a curvilinear relationship.
I hope this helps,
Jim
Hi Jim,
Thank you so much for your reply. In case we keep one or more predictors that are not statistically significant based on some except rule you mentioned in the previous comment, how can we interpret the results now?
Best regards,
Duc Anh
Hi again Duc Anh,
It depends on why you leave the predictor in the model. If you’re leaving it in the model because it’s the specific term you are testing for your experiment, then you state that you have insufficient evidence to conclude that there is a relationship between this variable and the response.
However, if you’re leaving the variable in for theoretical reasons, that’s what you should state. The variable wasn’t statistically significant but theory/other studies suggest it belongs in the model. You might even investigate possible reasons for why it is not significant, such as a small sample size, noisy data, a fluky sample, etc. Even though you suspect the variable belongs in the model, your sample still provides insufficient evidence to conclude that the relationship exists. You really have to make sure you have a good strong reason for this approach and state clearly why you are doing so.
I hope this helps,
Jim
Hi Jim,
Thank you so much for your very specific response. I think that it is very true when we interpret the model parameter. How’s about when we use the model with one or more non-statistically significant variables to make prediction? Sorry for my stupid questions!
Best regards,
Duc Anh
Hi Duc Anh,
I was referring to the case where you leave a predictor in the model when it is not significant. If you’re using the model to make predictions, you have the additional consideration of the precision of the predictions. Leaving an insignificant predictor in the model might reduce the precision.
What you want to do is to compare the predicted R-squared and width of the prediction intervals between the model with the insignificant predictors and the model with only significant predictors. Read my post about using regression to make predictions for more information!
And, there really is no such thing as a stupid question! 🙂
Jim