How do you compare regression lines statistically? Imagine you are studying the relationship between height and weight and want to determine whether this relationship differs between basketball players and non-basketball players. You can graph the two regression lines to see if they look different. However, you should perform hypothesis tests to determine whether the visible differences are statistically significant. In this blog post, I show you how to determine whether the differences between coefficients and constants in different regression models are statistically significant.
Suppose we estimate the relationship between X and Y under two different conditions, processes, contexts, or other qualitative change. We want to determine whether the difference affects the relationship between X and Y. Fortunately, these statistical tests are easy to perform.
For the regression examples in this post, I use an input variable and an output variable for a fictional process. Our goal is to determine whether the relationship between these two variables changes between two conditions. First, I’ll show you how to determine whether the constants are different. Then, we’ll assess whether the coefficients are different.
Related post: When Should I Use Regression Analysis?
Hypothesis Tests for Comparing Regression Constants
When the constant (y intercept) differs between regression equations, the regression lines are shifted up or down on the y-axis. The scatterplot below shows how the output for Condition B is consistently higher than Condition A for any given Input. These two models have different constants. We’ll use a hypothesis test to determine whether this vertical shift is statistically significant.
Related post: How Hypothesis Tests Work
To test the difference between the constants, we need to combine the two datasets into one. Then, create a categorical variable that identifies the condition for each observation. Our dataset contains the three variables of Input, Condition, and Output. All we need to do now is to fit the model!
I fit the model with Input and Condition as the independent variables and Output as the dependent variable. Here is the CSV data file for this example: TestConstants.
Interpreting the results
The regression equation table displays the two constants, which differ by 10 units. We will determine whether this difference is statistically significant.
Next, check the coefficients table in the statistical output.
For Input, the p-value for the coefficient is 0.000. This value indicates that the relationship between the two variables is statistically significant. The positive coefficient indicates that as Input increases, so does Output, which matches the scatterplot above.
To perform a hypothesis test on the difference between the constants, we need to assess the Condition variable. The Condition coefficient is 10, which is the vertical difference between the two models. The p-value for Condition is 0.000. This value indicates that the difference between the two constants is statistically significant. In other words, the sample evidence is strong enough to reject the null hypothesis that the population difference equals zero (i.e., no difference).
The hypothesis test supports the conclusion that the constants are different.
Related posts: How to Interpret Regression Coefficients and P values and How to Interpret the Constant
Hypothesis Tests for Comparing Regression Coefficients
Let’s move on to testing the difference between regression coefficients. When the coefficients are different, it indicates that the slopes are different on a graph. A one-unit change in an independent variable is related to varying changes in the mean of the dependent variable depending on the condition or characteristic.
The scatterplot below displays two Input/Output models. It appears that Condition B has a steeper line than Condition A. Our goal is to determine whether the difference between these slopes is statistically significant. In other words, does Condition affect the relationship between Input and Output?
Performing this hypothesis test might seem complex, but it is straightforward. To start, we’ll use the same approach for testing the constants. We need to combine both datasets into one and create a categorical Condition variable. Here is the CSV data file for this example: TestSlopes.
We need to determine whether the relationship between Input and Output depends on Condition. In statistics, when the relationship between two variables depends on another variable, it is called an interaction effect. Consequently, to perform a hypothesis test on the difference between regression coefficients, we just need to include the proper interaction term in the model! In this case, we’ll include the interaction term for Input*Condition.
Learn more about interaction effects!
I fit the regression model with Input (continuous independent variable), Condition (main effect), and Input *Condition (interaction effect). This model produces the following results.
Interpreting the results
The p-value for Input is 0.000, which indicates that the relationship between Input and Output is statistically significant.
Next, look at Condition. This term is the main effect that tests for the difference between the constants. The coefficient indicates that the difference between the constants is -2.36, but the p-value is only 0.093. The lack of statistical significance indicates that we can’t conclude the constants are different.
Now, let’s move on to the interaction term (Input*Condition). The coefficient of 0.469 represents the difference between the coefficient for Condition A and Condition B. The p-value of 0.000 indicates that this difference is statistically significant. We can reject the null hypothesis that the difference is zero. In other words, we can conclude that Condition affects the relationship between Input and Output.
The regression equation table below shows both models. Thanks to the hypothesis tests that we performed, we know that the constants are not significantly different, but the Input coefficients are significantly different.
By including a categorical variable in regression models, it’s simple to perform hypothesis tests to determine whether the differences between constants and coefficients are statistically significant. These tests are beneficial when you can see differences between models and you want to support your observations with p-values.
If you’re learning regression, check out my Regression Tutorial!
Hie Jim. I want to determine value relevance of negative earnings (losses) on EBIT and book value. My panel dataset has 30 companies and I did my regressions using System GMM estimators in Stata on this dataset. I then divided the dataset into profit-making and loss-making companies and ran the same regressions as before. Unfortunately the loss-making sample became too small to run meaningful regressions (observations were lost during conversion of EBIT to logs because of negative earnings).
Is it possible to determine the impact of the loss-making sample by comparing regression results of the full sample to those of the profit-making sample, say using an F-test? Unfortunately, the profit-sample is already part of the full sample so combining the two datasets as described in this post does not work. Is there another way of comparing the regressions run from two datasets, where one is a subset of the the other
Hi, I might not be completely understanding your goal. But, it sounds like you just need to add an indicator variable in your dataset that identifies whether the observation is from the profit sample or loss sample. You can use 1 to represent profit and 0 to represent loss. Then you can include the indicator variable as both a main effect and an interaction effects with the various predictors as I show in this post. This process will help you determine whether the differences between the intercepts and coefficients for the two groups are statistically significant.
Best of luck with your analysis!
Great job Jim. I’m impressed with your presentation.
I need some clarifications. I have an interaction between a continuous variable and categorical variable that has 6 categories. The interaction term is jointly significant and I read somewhere that I need to calculate the marginal effects, which I did. However, I’m stuck with how to interpret the marginal effects.
Any help is much appreciated. Thanks.
” … if summer is 1 and winter is 3, the analysis would assume that 3*summer = winter!” LOL, I think that one qualifies as the “awesome quote of the week” (for geeks, of course)!! Thanks again! And thanks for the kind words 🙂
That’s too funny. Geek humor is the best! BTW, I sent you an email.
Thanks for that explanation, Jim. I now understand what I was completely missing … I wasn’t thinking in terms of each “Fall”, “Winter”, “Spring” and “Summer” as essentially being individual “indicators” with their own columns of data … where the “bit” that is flipped to “1” indicates the applicable category for that data entry. So the “levels” that you were referring to earlier are these different columns of 0’s and 1’s (e.g, the “fall”, “winter”, and so on columns). When I asked the question, I naively thought of “seasons” as a single indicator (one column of data) that would take on a value of 0, 1, 2 or 3 for Fall, Winter, Spring, Summer, respectively (or however one wanted to assign those values) and assumed the magic of linear regression math would work out the coefficients to accommodate these arbitrary season value assignments. I now see that I was completely wrong in this thinking! Trying to extend a 2-category model to 3+ categories is where I went astray. (I am writing these details in case it helps other newbies like me who missed this initial critical concept!). Thanks again, Jim, for your delightful patience and tremendous assistance.
I don’t think it’s intuitive at all until you see it in action. You need to create a number of variables to describe one categorical variable! But, it all works together. And, you can’t use numbers to represent the seasons. For example, if summer is 1 and winter is 3, the analysis would assume that 3*summer = winter! It also suggests that the difference between summer and winter is twice the difference between summer and Fall! You can’t treat categorical variables as integers for those reasons. Categorical variables deal with the presence or absence of characteristics that you can’t (or just aren’t) measuring numerically.
Again, you don’t have to worry about this issue with most modern statistical software. You can just include a categorical variable in the model. But, back in the day, I did have to create indicator variables! Now, you know what is going on behind the scenes!
You’re very welcome, Pamela! By the way, I’ve read your blog, and you’re a very good writer! Take care!
Hey Jim, you say in the above reply to Patrik’s question: “For indicator variables, you must leave one level out of the model. … One level must be excluded because it’s redundant to have one indicator variable say that an observation is condition A while another indicator variable says that the same observation is not condition B.”
I’ve come across similar explanations on a few other sites. Clearly there is something that I am not understanding as well as I thought: if an observation can only have a condition of “0” versus “1”, how is that condition more than one level,and what does it mean to leave out one of the levels in the model? Let’s take the example of the binary condition regarding whether one is a pet-owner or not (“Yes” versus “No”, respectively). The “yes” gets turned into a ‘1″ and the “No” gets turned into a “0”. There is only a single number … a “0” or a “1”. How in this syntax is a level excluded ? Isn’t the interaction term dealing with “Condition” that can either have a value of “0” or “1” … and you need both of those values to properly evaluate the linear regression equation? It’s the “leave one level out of the model” phrase that is utterly confusing to me, because in my mind I am translating that phrase as “leave out all the data from the model that is associated with a Condition=0”, which obviously must not be the case.
On another related note, how does one assign value to conditions that are more than just binary … maybe an example would be what kind of pet you own (“N”=no pet, “C”=cat, “D”=dog, etc). Would you do something like: 0=no pet, 1=cat, 2=dog, etc? (The arbitrariness of the number assignment and possible unintended consequences of the larger valued conditions giving more prominent “weight” to terms in the regression equation makes me nervous).
Hey Pamela,
If you have just one binary column of data that defines the presence or absence of a condition, that’s fine. The rule comes into play for the second part of your question when you want have a categorical variable that is NOT binary.
So, let’s use an of the seasons of the year: Spring, Summer, Winter, and Fall. Those four values are the levels of the categorical variable “Seasons.” To include Seasons in the regression model, we need to create a binary indicator variable for each level of the categorical variable. So, we have a column for Spring, which would contain 1s and 0s to define whether each specific observation occurred in the spring. Same for the other three seasons. Now, we need to include them in the model, but this point is where we have to leave one of them out. Why? One of the assumptions for OLS regression is that there is no perfect correlation amongst the predictor variables. When there is perfect correlation, the OLS procedure won’t even run.
If you include all four indicator variables, there is perfect correlation. To illustrate this, assume that we’re looking at a Spring observation. That observation has a 1 in the Spring column. However, it also has 0s in all of the 3 remaining indicator variable columns. In other words, if you see 0s in Summer, Fall, and Winter, you know that it’s a Spring observation even if you don’t see the Spring value. You can use these other three columns to perfectly predict the fourth column. That’s perfect correlation. So, you have to take one column out. But, you’re not losing any of the information because that column was redundant to begin with. Say we take out the Spring column, we and the regression model will still know which observations are in the Spring because it’ll have the three 0s for the other seasons.
Statistically, it doesn’t matter which indicator variable you leave out. It won’t affect the statistical significance. It does affect the correlation coefficients but in a logical consistent manner. The coefficient for each indicator variable represents the mean difference between each level and the omitted level. For example, suppose the coefficient for Summer is 1.5 and we leave out Spring. This indicates that the Summer observations are an average of 1.5 units higher than Spring observations. If we left out Summer instead, Spring would have a coefficient of -1.5 because it is an average of 1.5 units less than Summer. So, it doesn’t really matter which one you leave out. If there is a natural baseline, comparison, or control level, the results are more intuitive to interpret if you leave that column out of the model.
A couple of notes. I was going to use your Cat and Dog example, but that’s not necessarily mutually exclusive so it doesn’t work. Someone can own both cats and dogs. The levels of categorical variables must be mutually exclusive. And, most modern statistical software will do all of this coding for you behind the scenes. Back in the day, you had to create these indicator variables yourself, but now you shouldn’t need to. The most you might need to do is to specify which level to use as the reference or baseline level–and the software will leave that level out of the model.
Very illustrative example you provided — thanks again!!
I just realized that any cond^2 term is unnecessary when “cond” is either a “0” or “1”, in which case “cond^2” is redundant with the “cond” term! My question still remains, though, if one simply sets the “f” coefficient in the above examples to zero!
Hi Jim,
I have a follow-up question to a reply you recently provided me on one of your other posts. In that reply, you suggested that I perform a regression fit with an interaction term. I’ve made quite a bit of progress, but now I am stymied by another question: Suppose in your example above, a quadratic fits the data better than a line for each of the 2 conditions, e.g., for Condition A, output = a + b*input + c*input^2 (and similarly for Condition B but with different coefficients, of course). My question is, how do we then handle the interaction term? (specifically, do we have to worry about what I could call the “cross terms”?) In other words, would we use:
out = a + b*in + c*in^2 + d*cond + e*cond*in
… or instead must we use ….
out = a + b*in + c*in^2 + d*cond + e*cond*in + f*cond^2
… or do we have to also worry about interaction with the in^2 term and use …
out = a + b*in + c*in^2 + d*cond + e*cond*in + f*cond^2 + g*cond^2*in^2
where “out” = output, “in” = input, “cond” = condition and a-g are coefficients.
My gut tells me that if one is dealing with a 2nd order polynomial, one must take into account all the “cross terms” (or whatever they are called), eg the terms associated with “f” and “g” coefficients in my example above. The question then is how to interpret the resulting p-values: if the p-values associated with, say, the cond*in are very small but large for the cond^2 term, then what would one say about the significance of that interaction effect? Thanks in advance for any guidance you can provide!
Hi again Pamela,
Yes, you can and probably should try an interaction for the quadratic term. Let’s start with this model:
Output = Constant + Input + Input^2 + Input*Condition + Input^2*Condition
Here’s how this works. The Input*Condition is the interaction effect for the slopes of the lines. The Input^2*Condition is the interaction effect for the shape of the curve. The interpretation depends on whether one or both of these interaction terms are significant.
If Input*Condition is not significant, the overall slopes of the lines are the same. Even though we’re talking about curved lines, their overall orientation on the graph would be the same. On the other hand, if this term is significant, then the overall slopes are different.
If Input^2*Condition is not significant, the overall shapes of the curves are the same. However, if this term is significant, the shapes of the curves are different to a statistically significant degree. Maybe one curve is tighter than the other.
And, you can combine the two interactions to fit various conditions. For example, the slopes and curves can both the same or both different. Or maybe the slopes are the same and the shapes of the curves are different. Or, vice versa! Just look at the p-values to make that determination.
Below is an example where we’re looking at interactions in a model with a quadratic term. In this case, the Input*Condition interaction term is significant but the Input^2*Condition term is not significant. The result is that the slopes are different but the shapes of the curve are the same. Basically, you take the same shape curve and just rotate it.
Interpreting the coefficients and regression equations themselves would be particularly difficult with both curvature and interactions. Graphs really bring the analysis to life! And then we can say that the patterns in the graph are statistically significant, or not, as the case may be.
Dear Jim. It is very nice to know your website. It is much more useful for my study and research. I found your web this afternoon as I am reviewing basic statistic and statistic tests for ecology. Could you help me to have your email address? Thanks.
Hi Nguyen, thanks so much for your kind words. They mean a lot to me!
If you have statistical questions, please find the relevant blog post and ask your question in the comments there. I like to have public questions and answers because they can help all readers. Thanks!
Ok thanks Jim, it seems clear now!
Best regards!
Thank you again Jim.
Yes now its much clear,
You meant the “input” is multiplied by condition (0 or 1), meaning that basically the variable Input*Condition will be 0 when Input is multiplied by 0 (Condition) and Input when is multiplied by 1, right?
Thank you in advance!
Patrik
That’s correct. The value for the interaction term for each observation is basically either zero or the input value depending on whether Condition is A or B. The regression procedure uses this to determine the interaction effect.
Hi again Jim,
This is a very interesting post. However, I am felling a little confuse. Related to the fist graph shown in this post, the regression model was running separately, right? Because you have to constant coefficient (One for Input versus Output for A condition and another for input versus Output for B condition)! Is the first equation refers to the results of this two models? Since that one model can not produce two coefficient for a single variable.
In relation to the second regression results (plotted), I see that Condition B is shown as part of independent variable…shouldn’t be Condition only (including A and B). Why is showing Condition B?
Another question is related to the interaction terms (I have read your post about it also), what you mean by Input*Condition (*) is not a multiplication since there are numerical and categorical variable, how is it incorporated in the model? There is some option in the software to incorporate it?
I hope you understand me, it seems confuse to me!
Waiting for your kind feedback,
Patrik Silva
Hi Patrik,
So, that first graph can represent two different possibilities. It can represent two separate regression models displayed on one graph. Maybe the analyst collected the data for the two conditions at different points in time? Or, it can represent a single regression model. A big point of this blog post is that sometimes analysts want to compare different models. Are the differences between models significant. So, let’s assume for the entire post that the analyst collected the data for each condition separately and originally fit separate models for each condition. At a later point, the analyst wants to determine whether the differences between the models are statistically significant. That sounds difficult. However, if you combine the two datasets into one and fit the model using an indicator variable and interaction terms as I describe, it’s very easy!
As for the two regression equations with different coefficients. Again, that can represent the two possibilities I describe above (separate models or models that use indicator variables and interaction terms). When you include an indicator variable (Condition in these examples), you’re adding a fixed amount of vertical height on the Y-axis to the fitted line. In this case, the effect of Condition B is 10 units. So, it shifts up the fitted regression line by 10 units on the Y Axis. You can represent this using a coefficient of 10 for Condition B or you can add those 10 units to the intercept for the Condition B model. They’re equivalent. The software I use automatically outputs both the separate models and the coefficients. However, it’s in the coefficients table where you can tell whether the effect of Condition is significant or not.
As for the difference in slope coefficients in the second example, that’s a similar situation but instead of depending on the indicator variable (Condition), it depends on the interaction term. An interaction indicates that the effect changes based on the value of another variable. This shows up in the graphs as different slopes, which corresponds to different slope coefficients in the output. Again, my software automatically outputs the equations for the separate models and the coefficients. That explains the different slope coefficients, but again, it is in the coefficients table where you can determine whether the difference between slopes is statistically significant.
The condition variable is categorical–A or B are the two levels. However, behind the scenes, statistical software have to represent it numerically and it does this by creating indicator variables. This type of variable is simply a column of 1s and 0s. A 1 indicates the presence of a characteristic while 0 indicates the lack of it. You used to have to create these variables manually but software does it automatically now. It’s all behind the scenes and you won’t see these indicator variables.
In this case, the software defined the indicator variable as the presence of condition B. For indicator variables, you must leave one level out of the model. The software left out condition A. One level must be excluded because it’s redundant to have one indicator variable say that an observation is condition A while another indicator variable says that the same observation is not condition B. That analysis won’t run because of the perfect correlation between those two independent variables. So, the software left out the indicator variable for condition A. However, it could’ve left out B instead and the results would’ve been the same. That’s why the output displays only Condition B. And, that’s how you multiply input*Condition. It’s really either the input value multiplied by 0 or 1 for each observation depending on whether the observation is of process A or B.
I hope that makes it clear!
Thank you Jim for such an intuitive and efficient description (what many many expensive econometric books lack). There is one thing where I’m not sure in my own regression. Maybe you could help me?
I’m doing a panel univariate (gls) regression with two growth rates y on x (+ time effects). And like you have described in this post I want to test for a significant difference in the beta for two subsamples/conditions. My two conditions (A, B) are two different time periods. “Normally” from the separate two condition A and B regression and the full regression with the interaction term we should have (like in your example):
(from the full regression) beta_input + beta_input*condition = beta_input_B (from the separate B regression).
So we have exact betas (for A and B) regardless of taking them only from the separate A and B regression or taking/calculating both betas from full regression with the interaction term.
But when I controll for heteroscedasticity in my panel regression this equation is not true anymore.
So my actual question is: are the betas from the separate A and B regressions still “right” and the p-value for beta_input*condition in the full regression with interaction term still decides for this two betas whether they are statistically different or is the p-value for beta_input*condition only valid for the two (in my case with controlling for heteroscedasticity now slightly different) betas: beta_input and beta_input + beta_input*condition?
Would you please suggest me how to compare two curve (Relative luminosity vs time).
Hi Durgesh, I’m not exactly sure what information you need, but I have written a blog post about how to compare different curves to determine which one best fits your data. Maybe this is what you need? Curve Fitting using Linear and Nonlinear Regression