Regression analysis mathematically describes the relationship between a set of independent variables and a dependent variable. There are numerous types of regression models that you can use. This choice often depends on the kind of data you have for the dependent variable and the type of model that provides the best fit. In this post, I cover the more common types of regression analyses and how to decide which one is right for your data.
I’ll provide an overview along with information to help you choose. I organize the types of regression by the different kinds of dependent variable. If you’re not sure which procedure to use, determine which type of dependent variable you have, and then focus on that section in this post. This process should help narrow the choices! I’ll cover regression models that are appropriate for dependent variables that measure continuous, categorical, and count data.
Related post: Guide to Data Types and How to Graph Them
Regression Analysis with Continuous Dependent Variables
Regression analysis with a continuous dependent variable is probably the first type that comes to mind. While this is the primary case, you still need to decide which one to use.
Continuous variables are a measurement on a continuous scale, such as weight, time, and length.
Linear regression
Linear regression, also known as ordinary least squares (OLS) and linear least squares, is the real workhorse of the regression world. Use linear regression to understand the mean change in a dependent variable given a one-unit change in each independent variable. You can also use polynomials to model curvature and include interaction effects. Despite the term “linear model,” this type can model curvature.
This analysis estimates parameters by minimizing the sum of the squared errors (SSE). Linear models are the most common and most straightforward to use. If you have a continuous dependent variable, linear regression is probably the first type you should consider.
There are some special options available for linear regression.
-
Linear model that uses a polynomial to model curvature Fitted line plots: If you have one independent variable and the dependent variable, use a fitted line plot to display the data along with the fitted regression line and essential regression output. These graphs make understanding the model more intuitive.
- Stepwise regression and Best subsets regression: These automated methods can help identify candidate variables early in the model specification process.
Advanced types of linear regression
Linear models are the oldest type of regression. It was designed so that statisticians can do the calculations by hand. However, OLS has several weaknesses, including a sensitivity to both outliers and multicollinearity, and it is prone to overfitting. To address these problems, statisticians have developed several advanced variants:
- Ridge regression allows you to analyze data even when severe multicollinearity is present and helps prevent overfitting. This type of model reduces the large, problematic variance that multicollinearity causes by introducing a slight bias in the estimates. The procedure trades away much of the variance in exchange for a little bias, which produces more useful coefficient estimates when multicollinearity is present.
- Lasso regression (least absolute shrinkage and selection operator) performs variable selection that aims to increase prediction accuracy by identifying a simpler model. It is similar to Ridge regression but with variable selection.
- Partial least squares (PLS) regression is useful when you have very few observations compared to the number of independent variables or when your independent variables are highly correlated. PLS decreases the independent variables down to a smaller number of uncorrelated components, similar to Principal Components Analysis. Then, the procedure performs linear regression on these components rather than the original data. PLS emphasizes developing predictive models and is not used for screening variables. Unlike OLS, you can include multiple continuous dependent variables. PLS uses the correlation structure to identify smaller effects and model multivariate patterns in the dependent variables.
Nonlinear regression
Nonlinear regression also requires a continuous dependent variable, but it provides a greater flexibility to fit curves than linear regression.
Like OLS, nonlinear regression estimates the parameters by minimizing the SSE. However, nonlinear models use an iterative algorithm rather than the linear approach of solving them directly with matrix equations. What this means for you is that you need to worry about which algorithm to use, specifying good starting values, and the possibility of either not converging on a solution or converging on a local minimum rather than a global minimum SSE. And, that’s in addition to specifying the correct functional form!
Most nonlinear models have one continuous independent variable, but it is possible to have more than one. When you have one independent variable, you can graph the results using a fitted line plot.
My advice is to fit a model using linear regression first and then determine whether the linear model provides an adequate fit by checking the residual plots. If you can’t obtain a good fit using linear regression, then try a nonlinear model because it can fit a wider variety of curves. I always recommend that you try OLS first because it is easier to perform and interpret.
I’ve written quite a bit about the differences between linear and nonlinear models. Read the following posts to learn the differences between these two types, how to choose which one is best for your data, and how to interpret the results.
- What is the Difference Between Linear and Nonlinear Models?
- How to Choose Between Linear and Nonlinear Regression?
- Curve Fitting with Linear and Nonlinear Regression
Regression Analysis with Categorical Dependent Variables
So far, we’ve looked at models that require a continuous dependent variable. Next, let’s move on to categorical independent variables. A categorical variable has values that you can put into a countable number of distinct groups based on a characteristic. Logistic regression transforms the dependent variable and then uses Maximum Likelihood Estimation, rather than least squares, to estimate the parameters.
Logistic regression describes the relationship between a set of independent variables and a categorical dependent variable. Choose the type of logistic model based on the type of categorical dependent variable you have.
Binary Logistic Regression
Use binary logistic regression to understand how changes in the independent variables are associated with changes in the probability of an event occurring. This type of model requires a binary dependent variable. A binary variable has only two possible values, such as pass and fail.
Example: Political scientists assess the odds of the incumbent U.S. President winning reelection based on stock market performance.
Read my post about a binary logistic model that estimates the probability of House Republicans belonging to the Freedom Caucus.
Ordinal Logistic Regression
Ordinal logistic regression models the relationship between a set of predictors and an ordinal response variable. An ordinal response has at least three groups which have a natural order, such as hot, medium, and cold.
Example: Market analysts want to determine which variables influence the decision to buy large, medium, or small popcorn at the movie theater.
Nominal Logistic Regression
Nominal logistic regression, also known as multinomial logistic regression, models the relationship between a set of independent variables and a nominal dependent variable. A nominal variable has at least three groups which do not have a natural order, such as scratch, dent, and tear.
Example: A quality analyst studies the variables that affect the odds of the type of product defects: scratches, dents, and tears.
Regression Analysis with Count Dependent Variables
If your dependent variable is a count of items, events, results, or activities, you might need to use a different type of regression model. Counts are nonnegative integers (0, 1, 2, etc.). Count data with higher means tend to be normally distributed and you can often use OLS. However, count data with smaller means can be skewed, and linear regression might have a hard time fitting these data. For these cases, there are several types of models you can use.
Poisson regression
Count data frequently follow the Poisson distribution, which makes Poisson Regression a good possibility. Poisson variables are a count of something over a constant amount of time, area, or another consistent length of observation. With a Poisson variable, you can calculate and assess a rate of occurrence. A classic example of a Poisson dataset is provided by Ladislaus Bortkiewicz, a Russian economist, who analyzed annual deaths caused by horse kicks in the Prussian Army from 1875-1984.
Use Poisson regression to model how changes in the independent variables are associated with changes in the counts. Poisson models are similar to logistic models because they use Maximum Likelihood Estimation and transform the dependent variable using the natural log. Poisson models can be suitable for rate data, where the rate is a count of events divided by a measure of that unit’s exposure (a consistent unit of observation). For example, homicides per month.
Example: An analyst uses Poisson regression to model the number of calls that a call center receives daily.
Alternatives to Poisson regression for count data
Not all count data follow the Poisson distribution because this distribution has some stringent restrictions. Fortunately, there are alternative analyses you can perform when you have count data.
Negative binomial regression: Poisson regression assumes that the variance equals the mean. When the variance is greater than the mean, your model has overdispersion. A negative binomial model, also known as NB2, can be more appropriate when overdispersion is present.
Zero-inflated models: Your count data might have too many zeros to follow the Poisson distribution. In other words, there are more zeros than Poisson regression predicts. Zero-inflated models assume that two separate processes work together to produce the excessive zeros. One process determines whether there are zero events or more than zero events. The other is the Poisson process that determines how many events occur, some of which can be zero. An example makes this clearer!
Suppose park rangers count the number of fish caught by each park visitor as they exit the park. A zero-inflated model might be appropriate for this scenario because there are two processes for catching zero fish:
- Some park visitors catch zero fish because they did not go fishing.
- Other visitors went fishing, and some of these people caught zero fish.
Whew! That’s many different types of regression analysis! If you’re trying to figure out which one to choose, I hope you will use this information to point yourself in the right direction!
If you’re learning regression and like the approach I use in my blog, check out my eBook!
Hello,
May i ask if it’s possible to conduct a moderation analysis with an independent, dependent and moderating variables that are all ordinal? Which regression would work best in that situation.
Thank you.
Hi Allan,
Yes, you can. However, there are some additional considerations. The ordinal dependent variable requires that you use ordinal logistic regression, which can handle moderation (interaction) effects. However, entering ordinal variables as independent variables will require you to make some choices. You’ll need to enter them either as continuous or categorical variables. The decision depends on the nature and amount of your data along with goals of your analysis. Unfortunately, I don’t have a blog post to point you towards about all those details, but I do write about it in my regression analysis book.
I hope that helps!
Hi Jim,
Many thanks for your informative blog.
I have a large data set (more than 25000 data) investigating the effect of one categorised variable( in this case is when the window is open and when it is closed) on the building temperature (which is my dependent variable). There are other independent variables which have effect on the building temperature including solar radiation, air temperature and illuminance (these are three numerical independent variables). Now I want to find out the effect of my categorised variable (window open and close) on the building temperature (while other three variables are effecting on that as well. It will be much appreciated if you can provid eme with your advice in this matter.
Hi Bab,
I’d start with least squares regression because temperature is a continuous variable. It can also handle your mix of categorical and continuous independent variables.
dear JIM my DV is saving and my independent variable are age,sex,family size,employment level,education level martial status and incom.SO what type of recommendations are gives to me from you?
Hi, because savings is a continuous variable, I’d start with linear regression and see if you can get a good fit with that. You have a mix of continuous and categorical IVs, which is just fine for linear regression.
Good morning Jim You have a remarkable talent as an educator and as a welcomer. I’m reminded of the multitude of scenarios ….statistical scenarios …that have emerged from the Titanic tragedy. PERHAPS YOU CAN REFRESH OUR MEMORY WITH RESPECT TO ONE DICHOTOMOUS INDEPENDENT AND ONE DICHOTOMOUS DEPENDENT VARIABLE USING GENDER AFFECTING SURVIVAL …ON THAT FATEFUL VOYAGE. WHEN YOU HAVE HUNDREDS OF MALE/FEMALE OPTIONS FOLLOWED BY HUNDREDS OF SURVIVAL AND NON-SURVIVAL. ARE ANY OF THE REGRESSION APPLICATIONS APPROPRIATE? THANKS, JIM
Hi James,
Thanks so much!! I appreciate that!
There are several ways to handle that type of analysis. One would be to use a chi-square test of independence using gender and survival. That analysis will tell you whether an association exists between the two variables. However, that’s not regression.
In terms of regression, I’d use binary logistic regression. I’d set up the analysis so that survival (yes/no) is the dependent variable. You can then include various independent variables, include gender, class, age, etc. to determine which variables played a role in survival and the nature of that role. Here’s a link to a post I wrote that uses binary logistic regression for a different scenario, in case you’d like an idea of how it works.
I hope that helps!
Hi Jim,
Very useful post. I am studying the effect of genetic diversity (independent variable) on lifetime breeding success (LBS, dependent variable) in a wild pig species. The dependant variables are all count data ranging from 6 to 45, whereas the independent variables are continuous data. Should I use poisson regression or negative binomial regression (assuming over-dispersion in the dependant variables) to test the correlations?
Thanks
Deepanwita
Jim,
I am trying to predict interval y with ordinal level x1, x2, and x3 as well as interval level x4, x5, etc. I suspect (with good reason) that the ordinal level IVs are not very reliable, yet they seem to be more predictive (higher Adj B squared) then the presumably reliable interval level IVs. Could the ordinal variables be overestimated in the model? If so, how can I correct the problem?
Troy
Hi Jim,
I am conducting research using two psychological instruments to assess the quality of life of 150 patients with chronic illness. the instruments are self-assessment Likert Scale with final result dependent variable calculated in scale from 0 to 100 measuring 4 different domains. the independent variables; gender, age, education level, marital status, income, residency, number of family members, and number of complications due to illness. what is the model type that is useful to compare the independent variables as predictors of outcome dependent instrument scale? what statistical test should be used to compare between these two instruments knowing that they are using completely different scaling technique ?
appreciate your kind help
best regards
Hi Jim!
I’m helping conduct research using a categorical binary independent variable and a discrete numerical dependent variable. The categorical binary independent variable is originally a nominal variable, but I can choose to transform the data into binary form as the research allows for it.
I was wondering what the best method of analysing would be. How can I go about understanding the correlation in this case in terms of statistical tests?
Hi Jim!
I am currently doing my Psychology Honours thesis and am investigating the relationship between parental separation and conduct disorder (cross-sectional in children).
My main IV in binary (parental separation/non-parental separation) and my DV is continuous (conduct disorder rating scale). I am also hoping to control for other variables / other IVs (economic status, peer-related problems, gender, ethnicity and social support).
I have 3 aims so far. Firstly, I want to explore the inter-correlations between these variables on the DV and examine their one by one relationship. Secondly, I think I want to do a multiple regression with conduct disorder as the outcome for each of these IVs. And lastly, I think I want to use a moderation analysis to examine which of these IVs dampen or strengthen the relationship between parental separation and conduct disorder.
Would you have any advice or feedback regarding if I am on the right track?
Aswell, I would really appreciate any advice on what statistical analyses I might be able to use to measure these aims or how I might go about re-wording these aims so it sounds better.
Thank you for your help!
Keturah
Hi Keturah,
Most rating scales are ordinal. If that’s the case, you’ll need to use ordinal logistic regression rather than linear regression. So, be sure you understand the nature of your DV. If it’s truly continuous, you can use multiple linear regression.
It sounds like regression, whether ordinal logistic or linear multiple regression, would be appropriate. And, with either type you can model moderation effects. I write a post about including and interpreting interaction effects in regression models. Interaction effects and moderation are different names for the same thing.
It sounds like you’re on the right track!
Hi Jim
I would like to know that if my response variable is development time, measured in days, then what family of GLM is suitable?
Thanks a lot for your help
Hi Rabia,
Generally speaking, time is considered a continuous variable and you can treat it as such. OLS would be a good place to start!
Hi Jim,
great post – thank you for that! But I coudn´t really find the right regression type for me (maybe you could add some examples). My DV is a variable with a value between 0 and 6 (a company gets a value of 6 if they have an environmental strength in each of the six areas). Do you have a recommendation on what regression type would be suitable? That would be highly appreciated 🙂
Cheers,
Louisa
Hi Louisa,
Is your DV and ordinal variable? It sounds like it but the info is sparse. If it is an ordinal variable, use ordinal logistic regression. Look at that section in this post for an example.
Hi Jim,
First, thank you for all your helps..This page provides a quite comprehensive framework for regression analyis..
Need a little bit help on my current research about the behaviour of households.
– I have a dependent variable (:attitude change) with 3 subvariables (let’s call them a,b and c) (all of them are categorial, 5-likert-scale).
– There are also various independent variables, which are mostly categorial (5-likert-scale), and a few nominal (gender, etc.).
– In addition, there are 3 independent groups of observations (households) (each group > 100 samples).
Ideally, I would like to explore the influence of the independent variables on each of the dependent subvariables (a,b,c) for each group, separately. But not sure if it is feasible.
There are two concerns here:
1- Is it better to merge the dependent subvariables together and investigate only the main DV, or it is better to model the subvariables separately? Both are possible due to the nature of the topic, but not sure if it is recommended for Regression analysis.
2- As there are 3 different groups, will there be 3 different models for each group separately, or all of them come in only one model?
I guess the right model to use is the ordinal logistic regression analysis. The initial idea was to use mann whitney u test & kruskal Wallis test, but I am lost on choosing the right method.
Thank you very much in advance..
Hello Jim, thank you for the very helpful and crisp, lucid article. I am working on a data where the dependent variable has 10 categories and the independent variables are all categorical. Could you help me understand whether multinomial logistic would be a good option? If not, what other kind of regression can I use?
Thank you
Hi Chaitra,
If the level for the DV have no natural order (i.e, they’re categorical aka nominal), then use nominal logistic regression. However, if they have a natural order, your DV is an ordinal variable and you should use ordinal logistic regression.
Hello Jim I need to know which regration is suitable for my title” the effect of HRM practice on employees’ retention? And also I have demographics and HRM variable so can I test by logit in one block or measure independently. Thank you for your response and with regards.
Well i am comparing the population with the co2 emission worldwide the population is in the x axis and the co2 emission is the y axis so may i know what regression to use?
Hi Miduna,
Least squares regression would be a good place to start because you have a continuous dependent variable (CO2 emission). However, you might have some added difficulties. I’m assuming you’re measuring both population and CO2 over time. If so, as the CO2 tends to increase, you might experience heteroscedasticity. Additionally, if you’re tracking over time, you’ll need to assess autocorrelation in the residuals and likely need to include time related independent variables to model it.
Hello sir, what if the type of dependent variable is interval? What regression model do I need to use?
Hi,
Interval scale variables are a form of continuous variables. Consequently, a good place to start would be least squares regression.
Hi jim , I am dealing with a regression problem predicting rejection rate of a product after ordered.
due to 95% of products were not returned my data is highly inflated towards 0 . Can you suggest on any approach or algorithm to handle it more efficiently than building a normal ols regression.
Hi Hari, if your dependent variable is binary, such as rejected/not rejected, you can use binary logistic regression.
Which data is most suitable for linear regression?
Hi, you’ll find that answer in this post. Look in the early sections.
Dear Jim,
Thank you for your great and valuable insight. I am still a bit confused on what model to use for my regression. I want to look at the relationship between financial ratios and company bankruptcy for the year 2020. I am thinking a binary logistic regression model would be the right approach but would like to confirm with you whether I am on the right track or if I should switch the model.
Thank you!
Hi Alina,
If your dependent variable is binary (bankruptcy, no bankruptcy) then binary logistic regression is a good choice!
Hi,
Sir Jim, I have One DV such as GDP and two IV such as External Debt and Debt Servicing. Would you please guide me, Which type of Regression analysis should I use?
Hi Jim,
I have 5 independent variables (continuous), 1 dependent variable (continuous), sample size is 63, i did normality test and i found some variables not normally distributed based on q-q plots.
1- Initially, I compute the correlation coefficient to see the correlation between variables
2- because i have a small sample size i entered (only) the correlated variables in the linear regression and i excluded the non correlated variables and the variables with a high correlation coefficient above or equal (.8)
3- I used enter method: the model is significant but each variable was not
4- then, i used a stepwise method: i got one varible significant.
My question is the previous process true? and which method could i choose in linear regression (enter or stepwise)?
Hi Jim,
I have 5 variables 1 DV, 1 IV and three presumed mediators, 4 out of the 5 variables are not normally distrubuted. Which regression is most fitted to test this relation?
Hi Jasper,
Least squares regression is a good place to start! The IVs do not need to be normally distributed. Be sure to check the residuals for normality. If they aren’t normally distributed, there are various corrective measures you can try.
Hi Jim
I need your help
I have a dependent variable which is continuous (between zero to twenty). it is about negative feelings and emotions.
I used multinomial regression analysis and generalized linear mixed effect models but I didn’t get a good result.
What is your recommendation for me? to use other types of regression modelling?
By the way, I have several dependent variables which are categorical or continuous.
Hi Farzaneh,
Unless your DV values are nicely in the center of that 0 to 20 range, you’ll probably need to use a two limit Tobit model. The problem is if your data are two close to one limit or the other that they’ll be highly skewed, making it hard to get normal residual. Also, the model will predict outside the possible range, giving nonsensical results. Those aren’t necessarily problems for least squares regression when your data are more in the center of the range. Check those residual plots.
However, it is a problem, I hear that a two limit Tobit model is a good option. Unfortunately, I don’t have any practical experience with that type of model to share with you. But it might point you in the right direction.
Hi Jim,
I have 4 independent numerical (integer) data and 1 dependent continuous (float) data . Which regression algorithm should I use for classification?
Hi Nur,
Because you have a continuous DV, a good place to start is least squares regression!
Thank, Jim!
Good day Jim, this was very useful, but I do not see multinomial logistic regression. I think that is the analysis I need to use. I want to know whether the two categorical variables influence what group a taxa falls into. This is my data, I have a dependent variable – Group (three categories) and then my independent variables is again categorical data – range (Single vs Multiple sites) and Life form (three categories). I have been running the model separately for the two an also doing a full model with both variables. I want to know whether this type of test is correct for my data? And are there any important things to be aware of when using this type of test, like assumption that need to be met. I really don’t understand the result that I’m getting.
Hi Dewidine,
I have that type of regression listed as nominal logistic regression. It’s an alternate name for multinomial logistic regression. And, yes, that is the correct type for your data!
As for assumptions, you want to check the residual plots as normal. Look for things such as homoscedasticity (constant variance) in the residuals. Generally speaking, it’s better to include both IVs in the model rather than fitting separate model. That way each variable is estimated while holding the other variable constant. It sounds like you’re on the right track!
Thanks a lot for the reply Jim, my DV ranges from VERY close to 0 and around 8%, mostly its extremly low.
Hi, in that case, I’d guess that least squares regression won’t work and you’ll need to use one of the other methods. You can always try OLS. Just be sure to check those residual plots! But, that’s awfully close to the lower bound, which is likely a problem.
Hi Jim,
If I were to use a continuous independent variable (e.g. emotion regulation) with two levels (e.g. cognitive reappraisal, expressive suppression), with one continuous dependent variable (e.g. relationship satisfaction), would it be possible to use simple linear regression? Thanks so much!
Hi Jane,
It sounds like your independent variable is actually a categorical variable with two levels. Continuous variables don’t have levels like that. You can perform simple regression with one categorical independent variable. However, for simplicity, you can perform a 2-sample t-test and assess the mean difference between your two groups.
Hi Jim. I am working with a continuous dependent variable (engagement rate of posts) and 2 categorical independent variables(type – 4 categories and type2 – 4 categories) which represent what kind of post it is. Is logistic regression the way to go for me?
Hi Arnthor,
When it comes to selecting the type of regression analysis, the the nature of the DV is front and center. In your case, you actually have a proportion or percentage, which isn’t quite a continuous variable. If it’s not too close to 0 or 100%, you might be able to model it using least squares regression. I’d try that first. Check the results and residual plots. If it works out for your data, that’s the best route to go.
However, if the percentages tend to be near 0 or 100, the model will predict outside possible range for values that don’t make sense. In that case, you might not be able to use least squares regression but instead fit a two limit Tobin model to account for the range of 0-100%.
Logistic regression is not a good choice for you because that requires categorical, ordinal, or binary DVs. You don’t have that. You do have categorical IVs, but that doesn’t mean you should use logistic regression. Again, it’s more about the type of DV.
Best of luck with your study!
Hi Jim,
Thanks for all your work explaining different models and answering questions. I am currently working on statistical analysis with a categorical dependent variable and two categorical independent variables (based on ethnicity and religion). The dependent variable has three categories (“Yes”, “No”, “I don’t know”). I have tried the multinomial logistic regression model with these variables but I am getting extremely high p-values to my surprise. I have considered converting the dependent variable into a binary variable (removing “I don’t know” category) to see if that can improve the results. Any advice on what I should do here? Thanks
Hi Uzair,
You can certainly try that. It is a bit of cherry picking the analysis because the first one didn’t give you results that you expected. Ideally, you’d state up front that you were going to process and analyze the data that way (just use two categories). In the write up, be upfront about that process (the why and how).
You don’t want to do excessive cherry picking because if you do that, you’ll always find something significant even if it’s just a chance correlation. However, if that’s the only post hoc change you’ve made to your plan, it doesn’t sound too bad.
Hi Jim!
Would you know why a logit regression model with interval variables would be favored over a logit regression model with categorical variables and vice versa?
Hi Jayden, logit regression models are designed to model the probability of an event for binary or categorical variables (more than two levels). Events are categorical outcomes such as pass or fail, or scratch, dent, and tear.
Hi jim…such a gifted teacher! Thank you.
Can you do a regression with one categorical independent variable predicting one categorical dependent variable?
EG. Gender predicting admission to MIT..
Thanks, Jim
Hi James,
Yes you can! In your case, you’re talking about a binary dependent variable because it has only two levels (presumably), admitted and not admitted. In that case, you’d use binary logistic regression and it’s fine to use a binary (or categorical) independent variable.
If your dependent variable had more than two levels, you’d use nominal logistic regression.
Hi. Thank you so much for this. I have a question. Is there a regression analysis that can be used for overlapping categories of dependent variable?
Example: I am using a sample of rural agricultural households where individuals are engaged in multiple paid (business or wage work) and unpaid activities
So my y variable is broadly classified into 3 categories: business, wage, unpaid
Can Maximum Likelihood models be used in this case?
Hi Jim
Thank you so much for your response. It has helped a lot.
I really appreciate your assistance.
You’re very welcome, Ntombi!
Hi Jim,
Thanks for the reply. I thought about using ANOVA but the data isn’t normally distributed and the variances are unequal so normal ANOVA and Kruskal-Wallis don’t work. I asked my supervisor about using Welch’s ANOVA and was told to use regression instead as it is easier to interpret and relatively robust to non normal data.
The dependent variable is the number of people who have a disease. So across 10,000 doctors, each doctor uses one OS, and records how many patients they have with e.g. diabetes. I’m trying to see if the choice of OS affects the recording of diabetes.
The data isn’t normally distributed, but with log(), it does normalise the data but I would also need to +0.1 (arbitrarily chosen constant, does it matter much the constant is?) to every data point as there would be log(0)s that prevent further analysis eg with qqnorm. Should I normalise the data and all do analysis (e.g. I guess at my supervisors suggestion, regression) on log()+0.1 data?
Thank you.
Hi Denton,
F-test ANOVA and regression are two sides of the same coin because they use the same math “under the hood.” It’s true that ANOVA can be robust against deviations from normality, but it depends on how many samples you have per group. Read my post about parametric vs. nonparametric analyses. That has pros and cons for both. But look particularly for a table with samples for different analyses. It’ll show you how large they need to be so that you generally don’t need to worry about the normality assumption.
However, regular ANOVA and regression are susceptible to violations of the assumption of equal variances. I talk about that in this post about Welch’s ANOVA. You can’t get out of that assumption by having a larger sample size. If your variability is substantially different, you really should be using Welch’s ANOVA and NOT the F-test ANOVA, regression, or nonparametric analyses.
You might not need to transform the data with large enough sample sizes unless your data are very skewed. If your smallest group is 50, you have a good enough sample to not worry about nonnormality unless it’s very skewed. I suppose the transformation could also solve your non-constant variance problem too. I could see going that way as you suggest. However, you can still consider Welch’s ANOVA and potentially analyze your data in the raw data units, which I like to do whenever possible. You’re probably fine either way.
Hi Jim
My DV is the the % exports for sales of volumes. Is this a continuous?
There are a lot of zeros for those who did not export for my time period of interest. I do not want to drop these as they are true zeros – as they represent firms that did not export anything for whatever reason. Can I just go ahead and run my OLS with this DV ?
What can I do with the ” I do not know” responses?
Hi Ntombi,
Percentages as a DV can be a bit tricky. If the percentages are close to zero or 100 percent, the distribution tends to be skewed. If so, you might need to transform it. I’d graph the percentages to see what they look like in a histogram. Are they very skewed? If not, OLS might be fine. Try fitting a model and see how the residuals look. If they look good, that’s a great sign you can use OLS. If not, try transforming the DV to get a more normal distribution.
I hope that helps!
Hi Jim,
I’m not sure how to go about analysing my data statistically.
I have 4 computer Operating Systems that doctors can choose from, that all record 5 different disease conditions.
These 4 OS’ have different amounts of data – one OS is only used by 50 doctors whereas another is used by 3000 doctors(unsure if this makes a difference).
I want to identify variation in the prevalence rates (how common the condition is) of the 5 different conditions and if this is due to the different OS’. i.e. Does the OS used cause a difference in whether or not a disease is recorded. The descriptive statistics are all pretty different.
Thanks!
Hi Denton,
It sounds like you need to use one-way ANOVA. You have one factor (OS) with four levels (the different OS). It sounds like you have a continuous dependent variable, although I wasn’t totally clear about that. If you want to compare the means between the groups and determine whether they are statistically significant, one-way ANOVA is what you should use. I write about using one-way ANOVA in Excel. Even if you’re not using Excel, it should give you an idea of what it can do.
I hope that helps!
Hi Jim! Thanks for this great post! Just wondering why you did not include the log-binomial model in this post. Could be a good fit right after the binary logistic regression point, perhaps?
Hello Jin,
Thank you very much for this article and the article on multicollinearity, they really helped me.
I am struggling with my Master Thesis. I am analyzing the impact of food product attributes on organic food buying tendency.
On one hand, (for independent variable) I have a constant sum scale for 5 food attributes and for the dependent variable I have the number of organic food people buy out of 10 grocery products (so 1 to 10 – higher the number of organic products higher the buying tendency)
I am confused as to which type of regression to use, given that my independent variables would be correlated.
Any help is appreciated!!
Thank you,
Aishwarya
Hi Aishwarya,
I’ve never used a constant sum scale before. However, I’d imagine you’d need to pick one or several but not all of the attributes and use them for the IV(s). I don’t really know if there is a recommended practice in terms of how many attributes to include. Maybe there are one or two that are particularly correlated with the DV? You can’t use all of the attributes because that produces perfect multicollinearity for a constant sum scale.
Your DV is a count variable if I understand it correctly. I’d try Poisson regression or possibly negative binomial. Look in the count DV section for more details about that type. I’d try those types first. Being a count variable, it’s likely to be skewed but those regression types can handle it. If the DV isn’t too skewed, you could conceivably treat the DV as continuous and use linear least squares regression. But, that depends on the properties of the data.
I hope that helps!
Yes! It really does 🙂 thank you so much!
Thanks for the excellent post. It seems that regression works for both continuous and binary target label. However, if in a situation, the target label is continuous but is between 0 and 1. What will happen if we transform the target to binary values (0 and 1) with an indicator and fit the regression model? How will the model parameter and standard error change compared to the fitted regression with the original continuous label?
Hi Jim,
Super useful page- thank you! I am struggling to find examples that describe what to do with a binary categorical dependent AND independent variable. I want to know if the mating status of a father (mated/unmated) predicts the mating status of his son (mated/unmated). Would this still be a binary logistic regression?
Any help greatly appreciated!
Thank you 🙂
Hi Claudia,
You can certainly use binary logistic regression with both a binary dependent variable and a binary independent variable.
However, if you have only those two variables, you can also/instead perform a 2-sample proportions test. That’s easier to interpret (for you and your readers). That will answer the question of does the proportion of mating sons differ by the mating status of the father?
If you want to see a 2-sample proportions test in action, read my post about Flu Shot Effectiveness, where I use a 2-sample proportions test on real data to determine whether the proportion of flu infections differ by vaccination status.
If you have additional IVs that you didn’t mention, then you’d need to use binary logistic regression.
I hope that helps!
Greetings Jim
Great post , I have a query.
I am working on a study where dependent variable is num of records(values start from 0 to a million) , around 6500 rows of data. which is the best mode to analyse this? please help.
Hi, I have question on my research preparation. I have two dependent variable which is categorical(yes/No). I used binary logistic regression. but my question is .how i can do the analysis? Is that separately or mixed? How? I thanks who can helping me!
Hi Jim
Very happy to have found you.
What is the best regression to determine asset allocation for a group of firms
Dependent variable- equity asset allocation
independent variables – leverage ratio, market cap, stdev of operating cashflow, book value of equity etc
Hi Rukky,
Typically, you don’t choose regression models based on the subject area. Instead, start by looking at the type of DV and using that to choose a type of regression model as I describe in this post. If equity asset allocation is a continuous variable, I’d focus on that section in this post.
Hi Jim.
I am currently doing my study on technology investment decision.
The DV is a binary (1= firm invest in technology; 0=firm do not invest in technology). There are about 6 IV with different types of data (i.e dummy, ordinal, continuous)
The time horizon is longitudinal (i.e panel data firm=n and year=i)
I am not sure which method is suitable. Is it panel data logistic regression? I also read some articles that use Generalized Method of Moment analysis.
Please help me. I need your opinion.
Thank you
Hi Jim,
thanks a lot for the great post!
I am struggling with finding the appropriate statistical method for my master thesis. The purpose of my study is to identify reasons why farmers invest in or refrain from investing in new agriculture technologies on farms in Portugal. I want to test different hypotheses to understand what drives the adoption of technologies among farmers in Portugal. This means that my dependent variable is binary (adoption yes/no).
Now, I have different types of independent variables:
– Binary variables (e.g. did you participated in any training program related to new technologies? yes/no)
– Ordinal variables (e.g. I find new technologies easy to use. likert-scale)
I will probably need two statistical models. For independent binary variables, I could use the logistic regression. For independent ordinal variables, I am not sure which regression method to use? Or should I use a correlation method like chi square test for independence or Spearman’s?
I would be grateful for your input!
Looking forward to hear from you soon.
Best regards,
Martin
Hi Martin,
As I mention in this post, you often choose the type of regression model based on the type of dependent variable. The types of independent variables don’t typically influence the choice in regression model.
Because you have a binary dependent variable, you’ll need to use binary logistic regression regardless of the types of independent variables. You’ll be able to predict the probability that a farmer will adopt technologies based on the values of the independent variables.
However, ordinal IVs present a challenge in regression analysis. You’ll need to enter each ordinal variable into your model as either as a categorical (nominal) variable or a continuous (numeric) variable. The decision depends on the nature and amount of your data combined with your research goals. I write about that in my book about regression analysis. In fact, because your thesis depends on regression analysis, I’d recommend it for that reason too.
By the way, I don’t think a chi-squared test or correlation is the way to go. Those analyses won’t give you the answers you need.
I hope that helps!
Hi Jim
Thanks for very useful post. I have a question that can we use square root rather than log link in poisson regression? Poisson regression with my data results in p values = 1 for all coeffecients when I use log link.
I want to ask another thing , I am confused how to treat my variables in regression analysis, either numeric or factor? my variables are Concentration(500,1000,2000,10000 particles), Exposuretime(1,2,3,24 hours), Size of particles (3 microns,10 microns) and particle type (aged, unaged, with additives, without additives). I feel that all should be entered as factors but not sure. When I treat variables having number as numeric, software (R) by default considers them continuous variable (got to know it when did post hoc test and observed that software is choosing any value between set levels (like 7 hours between 3 and 24 hours) for comparison rather than the ones tested).
Please guide me as I have wasted much time in getting answer of these questions. I did not get any of my question unanswered on this forum and that gives me a great hope. I shall be thankful for your time and valuable suggestions.
Hey there Jim,
I need your help with the regression analysis portion of my paper, which deals with effectiveness of debt relief programs in the sub-saharan african region. My dependent variables will be macroeconomic factors like-GNI, Inflation etc and the independent variable is the amount of relief received.
With the help of this blog, I think Lasso regression will suit my research best.
Can you suggest other regression methods or statistical tools that can be used?
Most times I’m not able to comprehend the statistical language/ formulae that most guides used, do you know of a step-by-step guide that provides a basic framework for regression/quant analysis.
Thanks a lot.
Hi JIm, greetings
my research title is: “Determinant factors for effective donor utilization by NGOs working in Ethiopia: the case of ORDA”
my Dependent variable is ” effective donor fund utilization”
and independent variables are: managerial factors, technical factors, operational environment, financial management and donor behavior. so, could you tell me the metric and non metric of both variables (dep. & Indpen.) ?
what type of regression to be used? multiple regression or continuous linear regression?
thank you for your fast and cheerful professional advice!
Hi Degarege,
It takes subject area expertise and research to determine the best metrics for your study. Unfortunately, I’m not an expert in donor fund utilization. So, I don’t know which metrics are best. You’ll probably need to do a bit of research to identify them. For more information about this process, please read my post about 5 Steps for Conducting Studies with Statistics. The earlier stages of the process explain how you need to operationalize your variables.
The type of regression largely depends on the type of metric you use for the dependent variables, which is unclear at this point. If it’s a continuous variable, then multiple regression is probably a good place to start. I believe what you describe as “continuous linear regression” is synonymous with multiple regression.
Thanks for writing!
Really clear and useful post, Jim. Thank you!
May I ask, if the dependent variable is slightly right skewed, is it still appropriate to use a linear regression? would I necessarily have to transform the data by taking natural log of the variable? If I do this, I lose around n=100 (total n=1004) participants.
I ask because I received a comment to create a histogram or illustration of the dependent variable distribution to demonstrate that it is acceptable to model it with linear regression. Are there other factors that can make the dependent variable more or less appropriate?
I think I should plot the independent and dependent variable with a scatterplot – but this is not so informative because my independent variable is a categorical item with five levels (gender identity).
Many thanks in advance for your help,
Hi Maria,
The real question to ask is: are the residuals are normally distributed? Regression assumptions center on the properties of the residuals. So, the question about the distribution of the dependent variable is largely irrelevant. I say “largely” because if you have a highly skewed DV, then it can be hard to obtain normally distributed residuals. For more information on this aspect, please read my post about OLS assumptions.
In that post, I indicate you should use a normal probability plot of the residuals to determine whether they follow a normal distribution. For more information about that type of plot, please read my post about histograms vs. normal probability plots for assess normality. In that post, I’m not writing about residuals specifically, but the information applies. One thing to note, with 1000 observations you have a fairly large dataset. You could theoretically use a normality test to assess normality. However, with so many data points, the test will be very powerful and could potentially detect inconsequential deviations from the normal distribution. More reason to use a normal probability plot!
Before trying anything else, I’d check the residuals. Determine whether they’re normally distributed. You might not even have a problem. If they’re not normally distributed, or you see other patterns, there are other steps you can take that depend on the nature of the problem.
Because your research involves regression analysis, you might be interested in my regression analysis book. I write about all of these issues in that book along with various ways to resolve problems.
If your IV with five levels is an ordinal scale, such as Likert scale, you might have some difficulties with that because ordinal data have a mix of properties of continuous and categorical variables. You’ll have to choose one or the other ways to enter that variable into the model. I don’t have a post for that but I do write about in my book.
I hope this helps!
Just to clarify further the above question:
Apologies for a long message again. I have been advised to do like this by my project supervisor:
I am going to do Multiple regression after taking mean (or average of my ordinal variables).
1. I have total 10 independent variables out of 3 constructs (for independent variable)
and 3 dependent variables (CA1, CA2, CA3) out of 1 construct (for dependent variable- for competitive advantage) and all are ordinal variables. So basically, I have total 13 variables, therefore will I have 13 columns of average if I need to calculate?
2. Then I need to take average of all 10 independent variables (because they are on ordinal scale) and take average of 1 dependent variable (e.g. CA1) at a time, and then run the multiple regression analysis. Because dependent variable can be only one at a time.
Then repeat the step for CA2 and CA3.
I will have 3 equations Y1 = m1X1+m2X2+m3X3+…+m10X10 +C, this is for CA1
Y2= m1X1+m2X2+m3X3+…+m10X10 +C, this is for CA2
Y3 = m1X1+m2X2+m3X3+…+m10X10 +C, this is for CA3
to interpret from output of SPSS. Output is in different tables.
And all these three equations should satisfy separately.
Am I doing this alright? Or shall I just keep only CA and delete other two?
Or shall I just run ordinal regression 3 times for 3 dependent variables as SPSS allows ordinal regression so no need to take average (Interpretation will be a bit difficult)?
I am going to try on paper the linear regression after taking average 3 times and then see if there is a relationship as mentioned in 3 equations.
Looking forward to hearing from you soon.
Regards,
Chandra
Hi Chandra,
Ah, in my reply to your other comment, I didn’t realize you were boiling down all your IVs to one average. When you can use a good number of Likert values to produce an average, it is good.
Are your Y variables going to be averages or the actual Likert values (ordinal)? If you use the ordinal values, you’ll need to use ordinal logistic regression. If the Y variables are an average of multiple Likert items, you might be able to treat it as continuous and use linear least squares.
So, I’m not totally sure what your final model will be? Will have the 10 separate IVs (Likert items) or are you averaging them down to one? And, will the DV be ordinal or continuous?
But, in general, yes, I’d fit a model for each DV as you describe. As I mentioned in my other reply, check those residual plots to be sure that you’re not missing curvature or otherwise misspecifying your model! That’s always important but it’ll be extra important in your case because you’re working with averaging Likert items, which can work but not always linearly.
Hello Jim,
Thanks for your great posts!!
Need a little bit of help if you could oblige for my MBA project. I went through your various posts already.
Here is the scenario of my project:
I have total 10 Independent variables all in Likert scale from 1-7 (ordinal), these variables are coming out of my 3 constructs.
And I have 3 dependent variables (part of one main dependent variable) again on Likert scale of 1-7(ordinal).
My questions:
1.Which regression method do I use?
2. Can I take all independent variables and only one dependent variable at a time? And then analyze?
3. Can I take average of all my independent variables(as they are on Likert) and average of my dependent variables (again on Likert)? and run the analysis? But I am not sure how do I do it.
4. How do I interpret the result?
5. My preference is to use Linear regression.
Just to briefly describe about my project, I am analyzing Impact of innovation on competitive advantage. so 10 IVs from Impact of innovations and 3 DVs(Competitive advantage).
I would be grateful for your inputs. This is a bit of struggle at the moment.
Looking forward to hear from you soon.
Regards,
Chandra
Hi Chandra,
You’ll need to use ordinal logistic regression because of the ordinal dependent variable. Yes, it’s fine to use separate models as you indicate.
Many analysts will average (or sum) your Likert items together and treat them as continuous variables. That can potentially avoid the problems I discuss below. Just be sure that averaging or summing is appropriate for your variables. You’ll need to be extra sure to check your residual plots to be sure your model has an adequate fit.
Ordinal independent variables can be problematic. I write more about ordinal IVs in my book about regression analysis. They have properties of both categorical and continuous variables. You’ll need to enter them one way or the other. The correct decision depends on a combination of the characteristics of your data and dataset, and the goals of your study. But, if you can sum or average scores, you might be able to avoid these difficulties.
You can’t use linear regression when you have an ordinal DV. Although, if you can average/sum the DV to produce a continuous variable, you might be able to use least squares linear regression. Again, be extra sure to check those residual plots!
Because your study depends on regression analysis, you might consider getting my regression book. You can get it as an ebook from my website (best price) or in print from Amazon and other sellers.
Hellow sir,
I have a problem with my research work..My dependent variable is ordinal type (5 point likert scale) and my all the independent variables are also only ordinal type (likert scale).what is the most suitable regression type for me? According to my knowledge i think that my all the independent variables are ordinal variables are ordinal regression can not be used.because it do not have any continues or nominal variable..according to the assumptions ..
Hi, because of your ordinal dependent variable, you’ll need to use ordinal logistic regression. However, as you note, the ordinal independent variables are potentially problematic. I write more about ordinal IVs in my book about regression analysis. They have properties of both categorical and continuous variables. You’ll need to enter them one way or the other. The correct decision depends on a combination of the characteristics of your data and dataset, and the goals of your study.
If it’s appropriate, you can average (or sum) your Likert items together and treat them as continuous variables.
Hi Sir,
I have more than one categorical dependent variable and one continues independent variable. Which regression I can use for it?
Hi Jim,
thank you very much for the article. I was wondering if you could help me find the fitting method for my thesis. I am doing a survey. My independent variable as well as my dependent variable are both binary (“yes” and “no”, which I coded in 1 and 0). Furthermore I have a lot of control variables which are either binary as well like for example sex or continuous (e.g. age) or categorical (1-4 or 1-5). Can I use a normal binomial logistic regression, even though not only my dependent but also my independent variable is binary/ a dummy variable? Or what else should I use? And how could I then interpret the results, just like with normal logistic regression?
I also want to test the mediation effect of two variables which are categorical (measured on a 4 or 5 point Likert Scale). How can I do that?
I would really appreciate if you could help me! Thank you so much in advance!
Hi Jenny,
Yes, that’s absolutely fine to use binary logistic regression in that context of also having binary variables as IVs along with the DV. What might actually cause some difficulty are the ordinal (Likert) IVs. You’ll have to model them as either continuous or categorical. The best method depends on which method gives you a good fit, the number of observations versus categorical levels (if you use that approach), and your research goals. I write about that in my book about regression analysis, although not in the binary logistic context, but the issues are the same.
I hope that helps!
Hi Jim, Thanks for helping us statistically challenged folk! I manually corrected my dependent variable by adjusting for time and plugged the data into some software that does Backward Stepwise OLS and got a final answer, but it failed Normality (P<0.001) and Constant Variance (P<0.001) test. When I took the square root of my dependent variable and adjusted for time, the result were entirely different, but they passed Normality (P=0.980) and Constant Variance (P=0.144) tests. I read one can normalize data simply by doing a log transformation or taking the square root. Is this practice legit? I feel like I had manipulated the data.
Hi Michael,
It is true that that you can transform your data as you describe to solve both those problems. I mention this in my post about heteroscedasticity (non-constant variance). It can also fix the normality problems associated with a very skewed dependent variable. Just be sure that the non-normality is not caused by a misspecified model, such as incorrectly modeling curvature.
While it is an acceptable solution to these problems, it should be your solution of last resort. Try solving the problems other ways first. I mention some in my post about heteroscedasticity. And, you can read about other reasons for non-normal residuals in my post about OLS assumptions. However, if you can’t fix those problems using other methods, then you can use the data transformation. Just be aware that you’re then describing the relationship between your IVs and the transformed DV, which makes the interpretation less intuitive than using the raw data. The goodness of fit statistics, such as R-squared, apply to the transformed DV as well. I write more about that in my book about regression analysis.
So, yes, it can be a good solution but try resolving the problems other ways first!
Hi Jim,
Thanks for the great overview! I’m wondering if you could explain a little about how orthogonal linear regression fits into the mix?
Thanks!
Hi Shannon,
Orthogonal regression is useful when you have two different measurement systems and you want to determine the relationship between the two. For example, you might have an old and new way to measure weight, blood pressure, etc. and you want to understand the relationship between the two systems. Most regression analysis assumes that measurement error exists only in the dependent variable. In orthogonal regression, the analysis can handle error in both the IV and DV. However, you do need to know the ratio of measurement error for the two systems to be able to use orthogonal regression.
I hope that helps!
Hi Jim! I’m struggling with the statistical method for my thesis in bioscience. I have only taken a basic course in statistics (we use R) and need to use another analysis than what I’ve done so far, for this project.
I have a number of sites where bat activity has been recorded. The data is in the form of “activity” or “no activity”, Now I want to compare the activity with qualities of the nature at the site. This will be measured in type of nature (alternatives like decidous forest, grassland, planted forest, edge zone), close to water (yes or no), slope (facing south, west, east,north or no slope) and maybe more factors. Someone suggested regression trees, but the information I have found is at a too difficult level for my knoledge, and I’m not that good at mathematical English… And I’m not sure how many sites I would need. to get relevant results from my partly nominal data. I have about 15-35 sites with “activity” and the total number of sites is 35, if I count “no activity”.
Do you have any idea of a good analysis or how to use my data? Please ask if I my explanation is unclear or you need more information. Thank you for doing this website!
Hi Caroline,
It sounds like you can use binary logistic regression, which I write about in this post. Your response variable is binary, activity or no activity. And, you can include all the variables you mention. Using the model, you can learn how each factor relates to the binary outcome. You can even predict the probability of an event (one of the outcomes) occurring for a set of predictor values.
Hi Jim,
Thanks very much for your explanations of the types of regressions. I was wondering if you could advise me on the type of regression that is suitable to the data I have collected.
I would like to know which of the supplements I have been taking has any influence on my Parkinson’s Disease symptoms.
The severity of my symptoms vary from day to day. I can measure the severity by doing a simple tennis ball rolling exercise to get “rotations per minute” or RPM. I record the time of day (T) I do the exercise and record the combination of supplements I took the night before.
So my dependent variable is RPM and my independent variables are T and the supplements I took the night before (1=yes, 0=no).
What statistical method would you recommend?
Thanks,
Mike
Hi Michael,
If your outcome data are continuous (it sounds like they are), then linear least squares (OLS) regression would be a great place to start. Because your model involves time (daily measurements), you’ll need to be extra careful looking for autocorrelation in the residuals. Read my post about OLS assumptions for more information. Look for the autocorrelation assumption.
Hi Jim, thank you for this site. well written, and i am sure it takes up much of your time. Your passion shines thru. I am hung up a bit on one of the assumptions for binary logistic regression. in particular, where Independent variables are linearly related to the log odds of the dependent variable. how would we go about checking that? thanks John
Hi Jim!
Thank you a lot for your posts, they are very useful and well-written.
I’m trying to find a way to analyse multiple hypotheses based on questionnaire data. Each hypothesis requires the analysis of multiple dependent and independent variables. The independent variables are of continuous, categorical and binary types, while the dependent variables are in this case all categorical (Likert scale). If possible, I would like to find a single model to analyse each hypothesis, but I’m having a hard time figuring out if there is any statistics available to fit my needs. I have been considering a multinomial logistic regression, but as I understand it only allows for one dependent variable. Is there an alternative where multiple dependent variables are available? If all of my different types of independent variables cannot be included in a single test, which types of separate tests would you recommend? I am thinking of a MANOVA for the categorical independent variables, but I don’t know of any suitable tests for the continuous and binary variables where I can include more than one dependent variable. Thank you a lot in advance!
Kinds regards,
Laura
Hi Laura,
Because of the ordinal DVs, I think you’ll need to use ordinal logistic regression and analyze each DV in a separate model. Those models can each contain all of your continuous and binary IVs. The regular MANOVA isn’t suitable for ordinal DVs. I’m not aware of a similar analysis that can handle multiple ordinal DVs.
hi Hi Jim, I’m interested to find out the prediction of drought of categorical independent variables on a continuous dependent variable.
for example my is data contain five variables one dependent and four other independent but the dependent variable again has 5 categories so which regression model i use please help me
Hi Girma,
If the DV variable’s categories have no natural ordering, then you’d use nominal logistic regression. However, if they do have a natural ordering (e.g., a five point Likert scale), you’d need to use ordinal logistic regression.
Hello Jim,
Thanks for your post.
I have been tasked to perform an OLS regression on wage of a certain population. We are asked to have a close look at the definition of the variable education (1:low and 5:high). What kind of conceptual problem might there be in using it in an OLS?
Thanks in advance!
Hi Tim,
One potential issue is that you have an ordinal independent variable. You’ll have to determine whether to include it as a categorical variable or a continuous variable. That depends on the goals of your analysis, the nature and amount of data, and the adequacy of the fit using either method. If you have my book about regression, I write more about the decision and challenges of both approaches.
Also, regarding the definition of the variable, that sounds fairly subjective/vague. You should have a clear definition of how you’ll measure education level. Ideally, come up with an operational definition. Then test it on sample data to be sure that it provides consistent, accurate results in representing education level. I talk about creating operational definitions of your variables in my post about including statistical analyses in scientific studies.
Hi Jim,
Hello I’m trying to build a regression model. I have 122 independent variable and I am trying to predict
legal proceeding amount .I also have many zeros in my dependent variable. These zeros show that he is not under legal prosecution.Most of these independent variables include the frequency of loan usage by years. I also have a variable in how much credit they use in total. But there are too many zeros in these variables. Because it was entered as zero for not using credit in any year. Therefore, I have different variables such as gender and age. Which regression model would be appropriate for me in this context ?
I have a table showing readings of a variable that depends on 4 different ‘Yes-No’ type of variables which are independent of each other. Besides, in the descriptive part, I am told that these “+ – ” type variables are themselves can vary in the sense that suppose there is a circular button. This button is on or off is given in the table. Besides if on, its range of rotation is also given that influences on part of question. What type of regression model should I use?
Hi ,,
thank you for the valuable information ..
I have a question about ordinal logistic regression.
I have a likert-scale questionnaire with several subscales. I have calculated the composite score for each subscale. First , I did compressions between males and females in my sample using mean scores of each . further I focused on one of the subscales and wish to do ordinal logistic regression for each item under that subscale separately to test its strength of association with other IV (gender, GPA, etc) .
would that be a correct approach ?OR is it contraindicated to treat items separately using ordinal logistic regression if they have already been combined as a composite score in the same study?
HI, I was wondering if my DV is numerical (ratio), what is de best regression analysis? The IVs are both ratio (numerical) and continuous.
Hi Hugo, the best place to start would be linear least squares!
Hi, great informative webpage. I was curious. If dependent variable is natural logged and key ind. var is not (semi-log form) can your controls be natural logged? my dep var is percent change in poverty, my key ind. var. is # of new churches(so must be measured by units) and my controls are all measured in percents (i.e. percent of the population thats black, percent of the population that earned a HS diploma, etc.) so do I need to have these controls natural logged to interpret my regression as “a 1% increase in a control variable leads to a x% change in poverty” or, since they are already measure in my data set as percentages, is that already how the regression interprets?
Thanks!
Hi Ian,
Yes, you can “mix and match” as needed with using natural logs or not! It really depends on the the nature of the data and theories about what is appropriate for it.
Thank you for replying me.! If I want to investigate the impact of happiness on the GPA, what would be suitable? Happiness would be collected as a average score it would be in between 0 to 6. What regression would be suitable?
Hi Sajeeka,
Is happiness an average of a set of scores that are something like Likert scale items? If so, taking the average is a method to use it as a continuous independent variable in your model. The coefficient for this variable indicates the average increase in GPA for a one-unit increase in the happiness score. That’s still least squares regression.
Hi Jim!
Thank you for your posts they are really useful. I have a problem with my scenario. My DV is grade point averages (GPA). It is a continuous value between 0 and 4. My IVs are happiness score, gender and academic level (it has 4 levels as 1000,2000,3000 and 4000 level). what type of regression is suitable here? What type of regression would you recommend for this??
Thank you so much!
Hi Sajeeka,
I’d start with linear least squares regression (OLS). You have a continuous DV. Gender and academic level are categorical independent variables, which OLS can handle. For academic level, the regression model will determine whether the mean GPA for each academical level is significantly different from one the baseline academic level (which is something you have to pick).
I hope that helps!
Hi Jim,
My dependent variable is a score 1 – 5, with 5 being the best rating, and 1 being the worst. I want to see if gender has an association with a higher rating. I have been looking into doing an ordinal logistic regression analysis, but not sure if this is correct. I know I could create a binary outcome of (1-3) vs (4,5) but not sure if I want to lose that data? What type of regression would you recommend for this analysis?
Thanks!
Hi Helen,
Your dependent variable is an ordinal variable. Consequently, you’ll need to use ordinal logistic regression. You won’t need collapse any values using that method. That method will tell you if gender relates to those ordinal values.
Hi Jim! Your posts are really helpful. My DV is an happiness index lying between 0 and 1(continuous) that has been constructed from various responses in categorical form. Hence DVis ordered. Should ordered logit or ordered probit would be ideal regression model to fit? Earlier, I have fitted OLS to the happiness scores lying between 10 to 45. R square is coming out to be very low. In ordered logit, chi square is highly significant, but, the pseudo R square is very low. Waiting for your valuable comments.
Thanks and regards,
Bidisha
Hi Bidisha,
It sounds like you have a continuous DV that is restricted in range between 0 and 1. Because it is continuous, you cannot use binary or ordinal logistic regression. However, you can use a logistic sigmoid function to model it. This process forces the model to respect the limits of the DV. I’ve never performed this type of analysis, so I can’t offer much help. But that should point you in the right direction!
Jim, I have one continuous predictor (IV) variable in my multiple regression and one ordinal predictor (IV). Can I do this? Can I incorporate this ordinal variable which comes from a Likert scale into the multiple regression?
Hi Louis,
I write about this in more detail in my regression book. Go to my webstore for more details about it.
Ordinal variables have a mix of attributes of categorical and continuous variables, which makes including them in a model a bit more complex. You’ll need to enter your ordinal variable as either a continuous variable or a categorical variable. The decision depends on the nature of your data, the goals of your research, and the quality of fit each approach provides.
Hey Jim
Thanks for the clarifications on the models…they really help.
Am conduction a study on Gender decomposition on the effect of education on household savings….but under this Savings is my dependent Variable while education, gender, sex, age, income, marital status and occupation are the independent variables…Help me on the best model to use since saving is categorical variable while in the explanatory variables their is mix of both categorical and continuous variables…
Looking forward from your Help
best regard
Hi Abel,
I would start with linear least squares regression (aka ordinary least squares). Your dependent variable is continuous (savings), which this type of regression can handle. Linear least squares can also handle a mix of continuous and categorical independent variables, such as you have.
Hi Jim,
Could you recommend any references?
Thank you in advance
Hi Jim,
I’m interested to find out the impacts of categorical independent variables on a continuous dependent variable.
For example, what are the effects of the type of news (financials, dividend payout) on the volatility of stocks.
Hi Joel, that’s possible in linear least squares regression along with other types of regression. Assessing the role of categorical IVs is a fairly common and basic usage for many types of regression models. If you have only categorical variables in your model, that’s often called ANOVA (analysis of variances). However, ANOVA uses the same mathematics “under the hood” as regression.
I have two dependent variables(aboveground biomass and carbon content) and three indepedent variables(wood density, diameter and height). Which model is best for anaysing my data? Waiting you under reply. Thank you
Thank you for your answer. What about the fixed and random effects is ok to use the mix of binary and continuous varibales?
Yes, just be sure to use a type of model where you can specify both fix and random effects, such as the MIXED procedure in SPSS. That aspect does affect the type of procedures you can use.
Hi Jim,
Your content has helped me a lot in my work, Thank you!
I’m conducting a regression analysis using panel data on a sample of 74 individuals on an 8 year period of time. I’ve been wondering if I can use a mix of binary and continuous independent variables to explain a continuous dependent variable. Would that be ok? how would it affect the type of regression I use?
Thanks
Meriem
Hi Meriem,
Yes, it’s entirely ok to use a mix of binary and continuous independent variables for ordinary least squares. You still need to satisfy the same set of OLS assumptions, but there are no additional requirements. Binary independent variables are also known as indicator variables and analyst frequently use them in linear regression. Typically, the 1s and 0s of an indicator variable represent the presence or absence of a characteristic. You just need to interpret their coefficients differently. The coefficient represents the mean difference between observations with and without the characteristic. The p-value indicates whether that difference is statistically significant.
I hope this helps!
Hello Jim.
When using logistic regression how critical is to have balanced treatment groups. I have a binary response variable evaluated from two treatments at n=40 and n=14? How does the need for balance (approximately equal n’s for each group) compares to OLS analysis? Is there any published information on the subject? I appreciate your comments and expertise. Many thanks.
Hi Erick,
I don’t have a reference specific to binary logistic regression. However, in general, it’s OK to have unbalanced groups like that. Having balanced groups helps you maximize power for any given number of subjects. But, it’s ok to have an unequal number. I’m not sure for logistic regression, but for most analysis I’d say that having 14 in the smallest group is fine.
When doing multiple regression analysis, as apposed to a simple OLS, where we have a number of independent variables, do you recommend to plot each independent variable against the dependent variable, one at a time to see how the plot of each variable on its own (without the other variables) against the dependent variable looks like. After analyzing each plot on its own go forward with the statistical analysis
Hi Jim,
I am hoping to do a regression analysis on social posts. The question I am usually trying to answer is does a certain variable (for example a photo vs no photo) play a role in engagement rate.
I’ve been going about this as coding the post: “Does this post have a photo or not?” Yes = 1, No=0 , but then I’m unsure what type of analysis would make the most sense?
(I would replicate this analysis on a bunch of different categories as well, like is the post light vs. dark, morning or night etc )
Any help/ thoughts would be much appreciated!!
Sarah
Dear Jim,
I would like to describe the correlation between 1 categorial factor (3 levels) and 2 continuous responses which are non-linear over time. I have a continuous factor (concentration) which I would like to add to the model. But I’m not sure if I should use the MANCOVA or a non-linear regression model. What would you suggest?
Thanks in advance!
Dear Jim,
I’ve been reading some of your blogposts and they are very helpful! However, I do have some remaining doubts about the regression I should run for my thesis. The doubt is mainly related to the setup of my dependent variable. I am constructing a measure of corporate social responsibility performance by summing strengths and concerns of several dimensions for each firm per year. There are about 50 dimensions in total and a firm is either given a 1 or a 0 in case the firm is either known to perform well/poorly(1) or 0 otherwise. Hence the dependent variable is constructed by summing these binary data observations. The minimum value of both the total strengths and the total concerns variable is 0. The maximum value of the strengths is equal to 44 and for the concerns this is equal to 36. I have thus only non-negative integer values. The mean for both strengths and concerns is around 3.0, however the standard deviation of the strengths is equal to 3.8 and for the concerns it equals 4.9. I’m hoping to be able to use OLS regression, however I was wondering whether there is enough variance in the DV and whether it can be considered a continuous variable. After reading this blogpost I also started wondering about the count data dependent variable and whether something like a poisson regression isn’t more suited. I was also considering the ordinal logistic regression. For a bit more clearance: I am trying to find the effect of several ownertypes on CSR performance. I hope you can help me out! In case you have some questions about my dependent variable, please let me know, then I will elaborate further!
Many many thanks in advance!
Hello Sir Jim,
What if there are 12 variables 2 Variants ( Variable 1 to 6 measures well being and Variable 7 to 12 positive thinking?
Should I run simple linear regression or multiple? How can I know if the variables are affecting each other.thank you
Hello Mr Jim Frost,
Thank you so much for the information you have provided on your website, as well as the answers to the questions above they are very helpful!
I am currently designing a research proposal that will assess medication adherence for rural vs urban groups in Canada. I plan on using an adherence scale that will give the participants a score (from 1-8) and based on the score, their adherence will be categorised. So >7 =high adherence. 5-8 =moderate adherence. <5 is low adherence.
I am planning the statistical analysis for the adherence.
Based on the responses I have read, would it be correct to conduct an ordinal logistic regression for the two groups? We’re also planning on conducting an independent samples t-test between the two groups and a paired samples t-test to compare the same group at the start and end of the study. Would these t-tests be possible if we transformed the data to continuous variables?
We also wanted to conduct a Chi-square test; is this possible if we don’t have estimated adherence scales.
How would you go about the statistical analysis for this study?
Thanks!
Jessica
Hi Jessica,
Yes, your plan sounds like a sound one. Ordinal logistic regression is a good choice. You can use t-tests for Likert scale data. Your data aren’t exactly Likert scale. It’s an 8 point ordinal scale. My guess (though I don’t a a reference to cite) is that you could not use a t-test based on the recategorized scale but probably on the full eight point scale. If you really want to use the recategorized scale, you might need to use a Mann-Whitney test instead. Transformation is probably not necessary. As an ordinal scale has more points, it become more like continuous data. Not perfectly so, but studies have shown that at least 5 point scales are close enough to use t-tests. Read my post about using t-tests for Likert data for more information.
About the chi-square test, I’m not sure what you mean by not having estimated adherence scales?
I think any of those approaches are valid. Ordinal logistic regression has the benefit of being designed for ordinal data. While studies have shown that t-tests and Mann Whitney tests can both work, it avoids a potential debate about the results if you just use a test designed for that data type! Consequently, I’d probably lean in that direction myself.
Hii Jim,
If one respondent answering same set of question for different brand, when i run the regression analysis, should I compared both answer and use dummy variable, or I should run the regression separetly? because one of my aim is looking how customer trust affect customer loyalty
Hi Geska,
Let me make sure I’m understanding your scenario correctly. You’ll have the respondents answer the same questions about two or more brands and you’ll some dependent/outcome variable that you’re measuring? And, you want to determine how the responses related to the brands?
In that case, you’ll need to use a mixed ANOVA model. You’ll need to include a subject ID as a random factor, survey responses as fixed factors or covariates depending on the type of response, and the brand itself as a fixed factor.
I hope that helps! Unfortunately, I haven’t written a post about mixed models yet. But, that’s the direction it sounds like you need to go!
Hi Jim
I am wondering whether regression analysis can be used to predict the strength of relationships between multiple income variables and bottom line expenditure. I have a conundrum in how I compare Business performance pre Covid ( Feb) to future performance in say ( October) and how i might model and estimate future income and expenditure. Can you enlighten?
Hi Jim, thank you for sharing this information. It is very helpful! I have a question regarding gender. There are only a few males in my sample, the female – male ratio is 14:1. Will this affect my analysis if I run a multiple linear regression? How about a simple mediation or moderation analysis (IV, DV, mediator/moderator). I plan to use gender as a control variable for these three types of analyses.
Thank you! I look forward to your reply.
Hi Fei, because there are so few males, it’ll be hard for the model estimate the effect of gender and the statistical test will have low statistical power. However, if the effect of gender is large, it might still be a valuable addition and you could possibly get significant results. Honestly, I would just try it and see how it works. Don’t expect too much from gender, but if the effect is large it might still be important to include it.
Hi Jim,
I hope you can help me, I am riding the clock a bit.
I hope you are well and safe with all the madness of COVID-19 in the world. I was hoping you could please help me. I am a student currently in need of a lot of help. Statistical analysis is not my forte but when needs must, we grin and bare it and look to the professionals for help.
I have just received my data back from my survey and I have found that my independent variables are all 1-5 Likert scales. I am looking at consumer attitude and intention, however, I believe if I am not mistaken that my dependent variable is my ‘Intention’ as that is what I am looking to find out overall. From my research, I think ordinal logistic regression is most appropriate to use. I ended up looking at youtube on how to run the test and what goes where but a lot of it is contradictory to the next.
My independent variables are;
Relative advantage/ Compatibility/ Environmental Impact / Psychological ownership.
Each of these are subcategories with between 4 to 8 questions under each – All 5 point Likert
I need to determine these expectancy values against attitude and intention but I have no idea how to.
So my questions are:
Firstly, am I using the right regression
Do the IV go into the ‘factors’ or the ‘Covariate’ box
Is it Test of Parallel lines where I am receiving my null hypothesis data?
Is there another test I should be running with it?
Also when analysing the output, if I run it through ‘factor’ box, in the parameter estimates, under the location information, is the significance data suggesting that my data is proving that – say for instance – Relative advantage significance is linked to consumer intention or does it mean that there is no link between RA and intention?
Thank you very much for your time. I hope you can help.
Hi Michaela,
Choosing the correct type of regression depends on the dependent variable, and I’m not sure what your DV is. If your DV is also Likert scale, then, yes, ordinal logistic regression is the correct type.
For the IVs, Likert scale items can be tricky if you’re using the individual item scores for your values. They’re not continuous but they’re not categorical. However, you’ll need to model them one of those ways. That gets to your question about whether they are “factors” (categorical) or covariates (continuous). There are pros and cons for both ways. And, the goals of your study also play a role. I don’t have an article to refer you to about that, but I do write about it in my regression ebook.
If you’re summing or averaging Likert items together and using those values in your model, you might be able to treat them as continuous variables (covariates).
The test of parallel lines is for ordinal (or ordered) logistic regression tests whether the coefficients are the same for all levels of your DV, which I’m assuming are the Likert scale values. This test really is about determining whether you have a good model or not. If this test is significant, it’s not good. It means that there is something wrong with your model, but it doesn’t tell you what exactly. You could just have a poor fitting model or you might be using the incorrect link function. You want this p-value to be greater than your significance level.
You should also look at the chi-squared goodness-of-fit tables. Again, you want p-values that are higher than your significance levels. Low p-values are problematic here because they indicate your model doesn’t fit the data well.
You should also look at the pseudo R-squared values. Like the regular R-squared in linear regression, they give you an idea of how well the model fits the data. Low values don’t necessarily mean the model is incorrect but they do mean that it won’t be good for prediction.
For the location parameter estimates, low p-values that are statistically significant indicate that the IV is statistically significant. Suggests that the effect of IV does not equal zero (i.e., there is an effect). It’s a good thing!
I hope that helps! Best of luck with your study!
Hi Jim
I have read your books and a lot of your blog and have found all incredibly useful. One area I am repeatedly confused by (perhaps as it is not in your books) is the interpretation of logistic regressions.
I am looking at various negative outcomes for children (child labour, begging, etc.) and comparing them with a range of households characteristics (child headed, large households, elderly headed households etc)
The results show some interesting findings. For instance, regression the binary dependent variable of ‘begging’ or not give me the odds ratio of 2.5 for child headed households. For the same regression, te margins commands shows probabilities of 0.57 for child headed households and 0.35 for adult headed households.
I am at a loss as the best way to interpret these findings in a manner which makes sense for the data…
Is it more intuitive to say child headed households are 2.5 times more likely to beg than adult headed households (or is this not correct interpretation), or is the better interpretation that child headed households are 1.6 times more likely to beg than adult headed households (0.57 v 0.35).
Can you assist?
Hi Jim
Thanks so much for your useful blog and posts. Its really helped me understand regressions and do some really interesting analysis. And inevitably led me to by your books on linear regressions and statistics… I only wish there was one on logistic regressions too 🙂
I have run a simple logistic regression using a child labour survey in an East African country.
I ran the number of households that reported begging against those households that were child headed.
The logistic regression showed that child headed households had an odds ratio of 2.51. The constant was 0.54.I interpret this as meaning that child headed households are 2.5 times more likely to beg that non-child headed households (although I am not sure if i need to subtract the constant?)
However the margins command shows the predicted probability of begging in a child-headed household is 0.57 while in a non-child headed household it is 0.35. The odds ratio and margins do not speak to each other as I understand. How can the odds of begging in a child headed household be 2.5 times greater using the odds ratio but the probability of begging in a child-headed household be far less at 1.6 times greater?
Apologies for the no doubt obvious question, but I am struggling to find any answers.
. logistic begging i.child_head
Logistic regression Number of obs = 1,288
LR chi2(1) = 3.95
Prob > chi2 = 0.0470
Log likelihood = -836.89941 Pseudo R2 = 0.0024
——————————————————————————
begging | Odds Ratio Std. Err. z P>|z| [95% Conf. Interval]
————-+—————————————————————-
1.child_head | 2.51981 1.180173 1.97 0.048 1.006238 6.310081
_cons | .545676 .032052 -10.31 0.000 .4863365 .6122557
——————————————————————————
Note: _cons estimates baseline odds.
.
end of do-file
. margin child_head
Adjusted predictions Number of obs = 1,288
Model VCE : OIM
Expression : Pr(begging), predict()
——————————————————————————
| Delta-method
| Margin Std. Err. z P>|z| [95% Conf. Interval]
————-+—————————————————————-
child_head |
0 | .3530339 .0134158 26.31 0.000 .3267393 .3793285
1 | .5789474 .113269 5.11 0.000 .3569443 .8009505
——————————————————————————
Jim,
Thanks for the valuable advice!
Hi Jim,
Maybe you can tell me if I’m on the right track or getting derailed.
I think an Ordinal Logistic Regression or Poisson Regression is right for me. I’m still obtaining my counts and learning R-Studio, but I’ve been reading your site and others to figure out what I’ll do with the data I collect.
I have over 1200 observations from content analysis (text mining)
My dependent variable is a count of words relating to one of 5 levels (a,b,c,d,e.)
My independent variable is a count of words relating to one of 4 typologies (y,z,w,x-order does not matter.)
I want to determine if any of the 4 typologies correlate to any of the 5 levels.
Hi Jim,
Thanks for you article, it is very helpful!
I have a question about my research: I want to examine if a correlation exists between clinical characteristics (amount of pain, degree of physical disability, etc.) and radiological scoring of an X-ray. Most clinical parameters are continuous variables, but one is binary (success/failure) and another is Likert scale with 7 categories (ordinal of course). The X-ray scoring is ordinal with 4 categories (from no degeneration to extensive degeneration).
I figure I could use Spearman’s rho for continuous variables; Kendall’s tau, Somer’s d or Goodman and Kruskal’s gamma for Likert; and rank-biserial correlation coefficient for binary variable (success/failure). This way I can correlate these variables to the ordinal X-ray scoring.
The next step is to do regression analysis to determine which independent variables account most for the X-ray scoring. Since my dependent variable is ordinal I was thinking of ordinal logistic regression. But is it possible to incorporate continuous, ordinal and binary independent variables into this model?
Kind regards,
Dan
Hi Dan,
Yes, ordinal logistic regression is the way to go for your data. It can handle continuous and binary data with no problem. It can handle categorical data with recoding as indicator variables, which most software should do automatically these days. Your ordinal independent variable is a bit problematic. You’ll need to include it either as a continuous variable or a categorical variable. There are pros and cons for each way. The correct answer depends upon a combination of the goals of your analysis and the nature of your data. I don’t have article to point you towards but I do write about it in my regression ebook. I write about in in the context of least squares regression, but the same principles apply to ordinal logistic regression.
Best of luck with your analysis!
Great post, Jim.
I am dealing with a challenge. I am not sure what regression analysis method to use to analyze a data set where the independent variables are in Likert scale (1 through 5 – completely disagree through completely agree) and the dependent variables are Likert items (1 through 5 – completely disagree through completely agree).
Your advice would be greatly appreciate.
Thanks in advance
Hi Varma, because your dependent variable is ordinal (Likert scale), you’ll need to use ordinal logistic regression. However, including ordinal independent variables is problematic. You’ll need to include them as either categorical or continuous variables. Each approach has strengths and weaknesses. The correct choice depends on the goals of your analysis and the characteristics of your data. I write about it in my ebook about regression analysis.
Best of luck with your analysis!
Hello Jim,
I want some help in determining a threshold poplation density.I have two data sets.One is the poulation density in many counties of US and another is number of cholera disease cases in those counties.I wat to figure out that population density at which there are 50% chances of getting infected with cholera.Can regression help me here?
Thank You so much
Hi Yusuf,
It sounds like you need to use binary logistic regression because that’ll allow you to calculate the probability of an event occurring (cholera infection). You’d need the number of events (infections) and opportunities (county population) along with the population density values. Although, I’m not sure if any US county ever got up to 50% cholera infections.
hello jim, so my doctor in college gave me the task to make a regression analysis on the impact of smoking, exercise level. and i dont know how to approach it.
Hi there! I am trying to find the right correlation test for my data. I have a continuous independent variable, and a categorical dependent variable (with 4 possible categories). Do you have any suggestions for this type of data?
Hi EJ,
I’m not sure that finding a correlation is what you’d want to do with that type of data. Instead, use a one-way ANOVA to determine whether the mean of your continuous variable is different between the four groups. Click the link to see an example of that analysis.
Hello Jim , i need your help with my masters project, i have a set of audio recordings from which i extracted the MFCC matrix and the corresponding image sequence from which the mouth landmarks were detected for each MFCC frame, my goal ultimately is to give an audio to a system that will predict the mouth landmarks for each MFCC vector but i literally have no idea what that system should be or how to proceed , can you give me any advice concerning that?
Hello Sir,
Is Hosmer-Lemeshow statistic really important in binary logistic regression? Does a p-value of less tan 0.05 of Hosmer-Lemeshow sufficient enough to discard the otherwise good binary logistic regression model?
Thank You
Hi Yusuf,
In binary logistic regression, the Hosmer-Lemeshow goodness-of-fit test compares the observed and expected frequencies of events to determine whether the model adequately fits the data. It evaluates whether the differences between the expected probabilities and the observed probabilities are statistically significant. Consequently, your low p-value indicates that your model does not fit the data well because these differences are significant.
In this case, try different link functions and/or change your model.
Another question, sorry but I am a beginner in statistics. Now I have to check the residual plot ( same as scatterplot right?) for positive correlation right? If there is a positive correlation does it mean the dependent variable is normally distributed? Or shall I do another test to check for it’s normal distribution? One more question, I have to check the residual and the normal distribution for the dependent variable “for the means?” or for the original responses?
Thanks alot for your response
Hi, I have an article about residual plots. I put a link in my other reply. That’ll answer your questions.
Dear Jim,
Yes there is a positive relationship, you mean by the residual plot the scatter plot right? Can I send a screenshot for the scatter plot as a message on your fb page just to make sure that all is ok?
Hi Haneen,
You use residual plots to check the OLS assumptions. Read those posts for more information.
Dear Jim,
I have a sample of 140 instructors and 21 tool in Moodle website and I want to show statistically that the the awareness of Moodle tools (in general) influences its usage.
I conducted a 5-items likert scale survey. Two questions for each tool, one asks about awareness of the tool and the other about usage of the tool. Then I calculated the overall usage of each tool (by finding the mean of the responses for each tool) and the overall awareness of the tool (by finding the mean of the responses for each tool). then I used a simple linear regression between the overall awareness and the overall usage (i.e. between the means) (resulting in 21 plot in the scatter plot)) is this right???? or shall I make the linear regression between between all responses for all tools?? (21*140 dot in the scatter plot)
Can you kindly reply a.s.a.p?
regards,
Hi Haneen,
I’m not quite clear about your model. For Tool A, are you looking at the mean awareness of tool A by all subjects and the mean usage of tool A for all subjects? And, you have 21 tools, so for your model, your 21 observations are basically 21 means for the DV and the IV? And each mean is calculated using 140 values. Is that right?
You should be able to fit the model that way. Typically, each observation is an individual but in your case each observation is a mean. But, it should tell you whether higher awareness leads to higher usage. You should find a positive relationship. Just be sure to check the residual plots. Using 140 values to calculate each average should allow you to treat the underlying ordinal variables as continuous data. However, because they are based on ordinal data with a constricted range and the difference between values might not be constant, you might have curved relationships and nonnormal distributions. Pay particular attention to the residual plots. If you satisfy the assumptions, you should be good. If the distribution of the DV is highly skewed, you might need to use a transformation. Or possibly fit curvature.
Hi Jim,
I really liked your article, I only have one question. My dependent variable is a categorical variable with three levels, and my independent variable is a discrete variable with 5 levels. Is ordinal logistic regression then the right way to analyse my data?
Thanks in advance!
Rose
Hi Jim,
Thank you so much for all your valuable posts! i was going over them in order to prepare for my own analysis.
I decided to both run an ordered logic regression (ordered categorical dependent variable) and a multinomial logic regression (unordered categorical dependent variable). The question i right now have is what is the best way to interpret my coefficients received from the logit regressions?
shall i use the odds ratio? the RRR? or simply use conditional or average marginal effects? is there anything i need to take into account?
Many thanks already very much for your reply!! I am so looking forward to it!
please, which type of regression is appropriate to quantify the influence of funding on academic success and why?
Hi Jamaldeen, you don’t pick the type of regression based on the subject area but by the types of variables you have, particularly the type of dependent variable. So, determine what your dependent variable is and then look through this post for the types of regression that can analyze that type of dependent variable.
Do you discuss use Information -Theortic Approach in your blog? In particular do you discuss use of AICc in comparing models? I recently purchased your Regression Analysis book but don’t see this discussed.
Thank you very much Jim! I really appreaciate your responsiveness.
I know it would be better to consider it as a nested dataset, and this makes the analysis more complex and difficult to understand for me. Since I am a marketing student and not a statistician, and my research question wether there is a relationship between the maturity levels (fixed values for each brand, that I want to use as independent continous variables) and the consumer mindset metrics (different for each respondents, who answered for one brand only, and that I want to consider as my dependent variable)… wouldn’t it be sufficient to run a correlation? It would come with limitations but I have to answer my research question with my competencies.
Thank you and all the best!
There are several considerations. If your DV is Likert, you can’t use the regular Pearson’s correlation, you’d need to use Spearman’s correlation.
However, there is a larger issue with the correlation approach. When you include multiple independent variables in your model, it automatically controls for all the variables in the model. Consequently, the effect of each variable is calculated while controlling for (i.e., holding constant) the other variables in the model. You lose that with correlation because you’re assessing IVs one at a time. You could end up with biased estimates–basically omitted variable bias, which might or might not be a problem with your data. That’s a risk to weigh and involves subject-area knowledge that I don’t have. Read here to see an example of this problem in action.
I’m rethinking the nested design as I think I’m understanding your design better. Nesting occurs on the IV side of things. And you don’t have that there. Respondents are answering for one brand, but their responses are on the DV side. So, disregard that. I’d perform ordinal logistic regression with no nesting.
You could perform Spearman’s correlation but with the caveat I mentioned earlier.
Thank you! My supervisor suggested me to think about CLUSTERED DATA and perform REAPEATED MEASURE MULTIVARIATE ANOVA. This should imply to transform my dataset? And, also, I do not understand if it is the proper way to analyze my data, or if a specific type of correlation (e.g. I tried to perform Spearman) can be enough!
Hi Gaia,
Unless I misunderstood your design, which is possible (see my other comment), it doesn’t sound like you’re using repeated measures. It does sound like you have a nested design because each respondent is answering about one brand.
My other comment has thoughts about the type of regression and other details about your model. It’s a rather complex mix of things, at least if I’m understanding correctly.
Hi Jim! Could you please reply to my question? It is important for my Master Thesis!
Hi Gaia,
So sorry about the delay. I’ll get to it soon. It unintentionally fell through the cracks!
Hi Jim,
Great article! I am still a bit confused. Would a type of regression still be best if the data included 1 categorical independent variable (two groups), 2 continuous dependent variables, and 1 categorical dependent variable?
Hi Lauren,
Typically, you have just one dependent variable per regression model. If you have multiple dependent variables, you’ll need to fit separate models.
You can use least squares models to fit models for your continuous dependent variables with the categorical independent variables. Although, if you have a continuous DV and just two groups, you can use a 2-sample t-test for that instead of regression.
For the categorical dependent variable, you’d need to use a binary logistic model if it has two levels or a nominal logistic model if it has more than two levels and assuming there is no natural order to the levels.
I hope that helps!
Hi Jim,
Great article! Thanks for the info.
I have a question regarding a doubt I have on which regression to use for the dataset I’m currently working in.
I’m trying to analyze the impact of HIV on MSK health among women aging with HIV. So, I assessed body composition measures (fat mass, lean mass, etc) a in a cohort of women HIV-positive and women HIV-negative to see if there are differences between them. Also, I did some physical test (grip, chair standing among others). So, I want to see if body composition measurements affects in the scoring of those physical tests, and how HIV influence that.
In this case, as body composition measurments are continues variables and the physical test scores are also continues variables should I use a linear regression model? But I’m not sure how to plot the HIV impact part on that model. Thanks!!
Best,
Diego
Hi Rahma!
If you want to investigate the level of attendance, you dependent variable will be Categorical, because it has a natural order (Low, Medium and High) to obey.
Considering this, the regression model I suggest you is the Ordinal Logistic Regression.
Kind regards,
Gabriel Villas.
Hi Gabriel,
A categorical variable with a natural order is an ordinal variable, and I agree with the recommended analysis. Even better would be to record the number of days attended and use either least squares regression or Poisson (or related) regression depending on how well the count approximates the normal distribution. You’d probably get more information that way.
Hello I have a question regarding a dataset I must analyze. I did two research steps:
1. Based on a managerial maturity model, I ranked 8 brands according to their level of usage of social media tools. For example how many different content formats they use form -3 to +5. They are a total of 7 components that, summed all together, give the total maturity score of each brand.
2. I made a survey among consumers to measure their level of brand awareness, consideration and purchase intent. Each respondent was randomly assigned to a brand for which he had to give his rankings, so I have 8 conditions in the dataset.
3. Now my hypothesis is that there is a relationship between the level of maturity and the consumer mindset metrics (namely higher levels of maturity should score higher levels of mindset metrics).
So the questions are:
– I am analyzing the full sample because each condition has only 50 respondents. Should I split the dataset per condition for some analysis?
– The variables I want to compare are: interval Likert scales 1-5 for the mindset metrics (dependent) and ratio (probably I will rescale them from 0-100% for example) that are fixed for each condition (independent). How to do correlation and regression analysis? Specifically, which regression is more appropriate?
Sorry for the long request but I tried to be as precise as possible.
Hi Gaia,
This is much deeper into the real deep details of study than I can usually go in blog comments. As you’ll see, your design is a complex mix of elements. From your description, I’m not sure that I have a completely clear understanding of your study, which makes it difficult to comment. However, I think I’ve got and here are some thoughts.
If your dependent variable consists of ordinal data (Likert scale), you’ll probably need to use something like ordinal logistic regression–unless the DV is a sum of multiple Likert items, which you can sometimes treat as continuous. If the DV is a sum of multiple Likert items, you can try using least squares regression but you’ll need to pay particular attention to the residual plots to ensure you’re getting an unbiased fit.
It sounds like each respondent is nested within one condition (brand). If that’s the case, you’ll need to use a nested design. You’ll probably need to include respondent as a random factor (as opposed to a fixed factor) in your design.
That’s a complex model to run and you might need to consult with a statistician at your institution to help you out.
I hope that helps! Best of luck with your analysis!
Hi
I have a dataset with 3 continuous variables in which the dependent variable includes negative values too. How can i decide wat type of regression to apply(other than linear)? by looking at the scatterplot or something?
Hi!! I have a question!
I have data on the number of classes that 86 student attended.
I have grouped those students into LOW attendance, MEDIUM attendance and HIGH attendance, based on the number of classes they attend.
And I have many Independent Variables (age, gender, BMI, smoking status, employment status, martial status), both categorical and continuous.
1) Is my Dependent Variable categorical or continuous?
2) Which regression do I use?
Waiting for your answer!!! Thank you
Hi, Jim.
I really need your help on the appropriate test for my undergraduate thesis. I controlled 3 categorical variables (A,B,C). Each have 3 levels. This is a within subject experiment where each subject was tested for 12 conditions (e.g. condition 1: A1,B1,C2; condition 2: A1,B2, C2…) Could you please suggest a test to analyse the contribution of each variable? I was thinking of the binary logistic test since the response is binary.
Thank you so much for your attention!!
Hi Maggie,
In general, that sounds like the right approach. One potential hiccup is that given the with-subjects nature of the study, you’ll need to include subject as a random factor. I discuss this a bit in my post about repeated measures designs. I’ve never seen a similar type analysis with a binary dependent variable and a random independent variable. So, I’m not sure if there’s software out there that can do that analysis. I’d consult with a statistician that can look into the special needs of your study and find the best solution. If it weren’t for the random factor, I’d say that binary logistic regression would meet your needs.
Hi Jim
I wanted to analyze visual acuity outcome, which is primarily continuous variable but I have made the range and categorised it into three groups as good outcome, borderline and poor outcome, can I use ordinal, multinomial or still I required to use linear regression?
My independent variables are, age, sex, type of cataract, commobidities, and level of education.
Thanks.
Hi Katuta,
You can use ordinal logistic regression. However, if you have the continuous data for acuity outcome, you might consider analyzing it using linear regression on the continuous data rather than converting it to an ordinal variable. But, if you do transform it as you describe, use ordinal logistic regression.
Hi Jim,
Your post is very helpful and too clear, thank you. In my observtional study, the outcome variable is categorical, dichotomous – yes/no and the independent or explanatory variables are also categorical example – age groups, gender, education, smoking etc. I understood that we need to use binary logistic regressiON. Is this right? and how would we interpret beta coefficient?
Hi Jim, I have a question on the type of regression analysis that would be the most appropriate for me to use . I’m investigating the impact factors such as ethnicity, geographical region and gender have on educational attainment at GCSE level
I have 3 categorical variables; ethnicity, geographical region and gender which im measuring against a continous dependent variable.
Thank you!
Hi JIm,
Thank you again for the prompt response.
My regression lacks the offset variable.
So, in the Parameter Estimates table (SPSS) of this regression there is an Exp b column and I wonder how to refer to its values as IRR or RR?
thanks,
ariel
Hey Jim,
You are right my dependent variable is not a count variable but it is not a pure rating variable either. It is difference/delta between the student rating pre and post a course and it has a poisson distribution. Is it wrong to use a poisson regression for this type of variable?
Thank you for the prompt answer,
Ariel
Hi Ariel,
Yes, you can use Poisson regression in your case. And, that does seem to be borne out. However, it might raise questions amongst reviewers because it is an unusual use for it. However. Poisson regression is designed for data the follow a Poisson distribution. Usually that means count data. However, if your DV follows a Poisson distribution, the procedure will work fine even when it’s not count data. Just be ready to explain that!
Hey Jim,
I have a dependent variable that is the variation between likert scale student’s ratings pre and post a course they took. I did a Kolmogorov-Smirnov test and found that this variable has a poisson distribution so I constructed a multivariate poisson regression (without an offset variable).
My first question is did I construct the right regression?
My second question is that the Parameter Estimates table of these regression has an Exp b column and I wonder if to refer to it as IRR or RR?
thanks,
ariel
Hi Ariel,
Typically, you use Poisson regression for count data. However, the student ratings don’t sound like count data. Ratings are ordinal data. Consequently, you should consider using ordinal logistic regression.
I need help finding the right regression model for a categorical, binary independent variable and 3 different continuous dependent variables. Please!
Sounds like you need to use a binary logistic model.
Jim,
Could you give a thumbs up if zero-inflated models would be appropriate given the following question:
Is there a relationship between the number of backpacking trips in a 3-year period among a purposive sample (n=50) of participants in a outdoor leadership class (dependent variable) and their age, sex, occupation, and race (independent variables)?
Note: there is some evidence that many of the participants did not go on a backpacking trip in the last 3 years.
Thank you in advance and stay safe!
Hi Erik,
From what you describe, it sounds like a zero-inflated model might be appropriate. At least as a something to consider. You need to consider whether there are two separate processes that you can describe. One would be the desire to go on the trips. Some people want to go while others don’t. That will explain some of the zeros. Then, of those who want to go, there’s an expected distribution of trips. That distribution will likely include zero because I’d imagine there are people who would want to go but for some reason or other weren’t able to. So, you’d have the two sources of zeros, those who just didn’t want to go and those who wanted to go but weren’t able to go.
However, you don’t have to go straight to a zero-inflated model. You can try a regular Poisson or negative binomial model to see if you get a good fit. Sometimes you can even get a good fit with least squares regression when the mean count is high enough.
So, it does sound like a good possible approach but you might give the simpler models a try first. See if you do in fact have an excessive amount of zero trips that the simpler models can’t explain.
Best of luck with your analysis!
Hi Jim, first thank you for making statistics and regression analysis so easy to understand for the masses.
I am currently working on an analysis to find the cost of an hypothetical zero load carrying gas distribution pipe. All of my variables are continuous, as the pipe size increases the cost of it as well. But I have two main types of material (Plastic and Metal). How can I do this analysis, doing a regression analysis for each material type and then average the results or using an ANOVA? Im lost.
Thank you
Valuable information and it explains the subject to its primary scratch!!!
I just want to know the difference between Regression analysis and ANOVA?
Hi Miro,
If we’re comparing ANOVA to least squares regression, they’re actually the same analysis but have different historic traditions and terminology. However, the math “under the hood” is the same.
Regression typically works with continuous predictors, although you can add categorical variables. For categorical variables, regression uses binary coding (1, 0) so that you compare the results for each categorical value to a baseline value.
ANOVA typically uses categorical factors. These factors will often use effects coding (1, 0, -1), which allows you to compare each factor level to the overall mean (rather than to a baseline group). You can include continuous predictors but they’re called covariates. ANOVA is more focused on explaining the differences between groups that are formed by categorical variables. Consequently, ANOVA is also linked to post hoc testing that carefully assesses the differences between multiple groups. However, you could perform the same post hoc tests on regression results.
Despite these differences, you will obtain completely consistent results using both of these methods. Although the differences between coding for categorical variables can make some of the coefficients appear to be different, but they’re telling the same story. If you can control the coding scheme in your statistical software, you can obtain completely identical results.
Hi Jim,
This post is very useful i just need to know that if we have continuous dependent variable and three independent variable one is continuous second nominal and third ordinal, in this situation how to calculate regression.
Thanks!!
Hi Jyoti,
Yes, you can use least squares regression for that type of model with one caveat. There’s no way to include an ordinal independent variable in the model. You’ll need to include it in the model as either a continuous variable or a categorical (nominal) variable. There are pros and cons either way. I don’t have a post that discusses that decision but I do write about it in my regression ebook.
Best of luck with your analysis!
Hi Jim, thanks for the detailed breakdown of the different types of regression analysis!
I would like to seek your advice on how I should run some of my own tests for my undergrad final year study, which is on the topic of how temporary depletion of level of executive functioning may affect honesty behaviour in young children.
– My main IV is executive functioning: either depleted or not depleted (categorical).
– My other variables are a mix of categorical and continuous variables, namely: gender, age (in years), Theory of Mind score, and English vocabulary score.
– My only DV is the child’s honesty behaviour: either truth-teller or lie-teller (categorical).
I am currently about to carry out my statistical tests on SPSS, and have dummy coded my categorical variables to numerical codes.
I haven’t learned the more advanced statistical tests in my undergrad courses, but based on what I have read in your post, I think the Binary Logistic Regression is most suitable for my study’s design. The question I have is, how do I proceed from there?
1) Specifically, how do I control for my other variables (i.e. gender, age, ToM score, vocab score) to ensure that my model is mostly looking at the effects of executive functioning on honesty behaviour? Is that already accounted for by the regression? If not, do I need to do additional tests even before I carry out my regression? What tests would these be to control for these other factors?
2) Conversely, how do I check if the other variables have main effects or interaction effects with my main IV, assuming that they may present? Can the regression reveal these? If not, what other tests should I run?
I am sorry if these questions are really basic, I am just a mess when it comes to math and statistics. I would appreciate any feedback I can get from you! Thank you and stay safe!
Hi Jim,
I hope you’re keeping well wherever you are during these uncertain times.
I’d really appreciate it if you could give me some help when you get the chance!
I’m writing up the results from my study which is looking at the effect of a single nucleotide polymorphism (random genetic variation) on postprandial glucose response.
I have identified no significant difference between any of the three genotype groups and any of my glucose response outcome variables (fasting, peak, 2-hour postprandial, and incremental area under the curve). I have 73 participants in my sample (27 female and 46 male), and identified a significant difference between sexes for 2-hour postprandial glucose concentration.
Using SPSS statistics, I would like to conduct some kind of multiple regression to assess the effects of both sex and genotype on the different measures of glucose response (primarily 2-hour postprandial to begin with). Therefore, my variables will be categorical.
Would I have to assign numbers to my categories?
I am unsure if my data has a linear relationship, and I don’t know how to check this in SPSS especially with categorical variables. Most of my outcome measures are also non-normally distributed.
Will a non-normal distribution and lack of linear relationship prevent me from doing a multiple regression analysis?
Thank you in advance!
Hi Jonny,
Can you more fully describe the nature of your dependent variable and how you’ll record it? That would help me answer. Thank you!
Thank you so much!
Easy explanation of the different types of regression
Hi Jim,
I would like to establish the correlation or lack there of between the cost of accidents and the experience level of the person who caused the accident. I have three levels of experience.
Thank you
thank you jim! its all making much more sense. i guess im getting confused because ive been seeing some examples where people use t tests or mann whitney u tests to compare things like age or amount of a drug (continuous variables) to a categorical outcome, but that would obviously be incorrect since they’re the independent variables not the dependent variables right?
appreciate your help!
Hi There! thank you very much for this. I was wondering what kind of analysis you would use to compare a continuous independent variable (i.e. age) with a binary categorical dependent variable (i.e. mortality)?
would that just be a logistic regression?
Hi Eva, yes, that sounds like a situation where you’d use a binary logistic regression model.
Hello Jim,
So nice of you !
Thank you so much for the guidance 🙂
Regards
Sujit
You’re very welcome! Again, sorry for the delay! 🙂
You Never replied my query 🙂
Regards
Sujit
Hi Sujit,
Apologies for the delay. Sometimes life gets in the way, especially during these days.
Because you’re doing a count of walk-ins, it sounds like you might need Poisson or Negative Binomial regression. Although it depends on the distribution of your count variable. If the count is high enough, the distribution can approximate the normal distribution and OLS might be okay to use. You’ll need to define a consistent period for the count. For more information, read the section in the post titled, “Regression Analysis with Count Dependent Variables”–it’s near the end.
Thanks a lot for the explanation!
I’m collecting data at 2 time points, and would like my variables at Time 1 to predict Time 2. So my DV is a variable at Time 2 and as predictors, I have 1 variable at Time 1 (not continuous) as well as a 1 categorical variable from Time 2 (condition = 1 or 2). I am assuming that I’ll need to use multiple linear regression?
Regression is new to me, so I’m not sure which model to use if I have 1 categorical predictor and another predictor that isnt continuous.
Hi,
What type of variable is Time 1 and what type is Time 2?
I want estimate farmers willingness to use renewable energy. So I want to find out if I can use switch regression to analyze willingness to use. Thanks and waiting your response
I want to test if science and technology has increased peak performance and age amongst different positions in the NFL. What type of regression do you think would be best for this type of test?
Hi James,
Choosing the type of regression analysis is not usually based on the subject area. (There are some exceptions to that rule.) In other words, there’s not a correct type for performance in the NFL. Instead, you need to determine which variable(s) are your dependent variables and the nature of the relationship they have with the independent variables. If your DVs are continuous, I’d recommend starting with least squares regression and determining whether you can obtain a satisfactory fit with it. You can always change analyses as need. Sometimes you’ll learn things along the way that require you to change the type of regression. Although, advance research will help you in that regard as well.
Best of luck with your analysis!
Hello! Hope this finds you well. I am running a multilevel multiple regression analysis for my dissertation. At the individual level, my IV is bullying experiences in schools and DVs are three different health outcomes for students (1 dichotomous, 2 continuous; all mental health). I also have a moderator (LGBTQ identity, dichotomous). This data has been identified all at the school district level for an entire state. At the group level, I have voting data for school districts (second IV; currently in percentages: % for Hillary, % for Trump, in 2016 election). My analysis is hoping to explain the relationship between school district voting records, bullying experiences, and health outcomes for LGBTQ students.
I am planning on using Mplus for analysis. I have a few questions after reading this article:
1) What would your recommended analysis plan be? After reading this article, I was unsure how to apply this to a multilevel, multiple regression analysis in Mplus.
2) What would be the best way to insert voting data into the data set for analysis, given that it is in percentage form? Currently, I was thinking of calculating the difference between voting (80% Clinton, 20% Trump = .60; 20% Clinton, 80% Trump = -.60 on a scale of -1 to 1 from most conservative to most liberal). That way, it could serve as a continuous IV at the group level (level 2) of the multilevel regression.
Lemme know what you think. Thanks for any help you can provide a novice statistician!
Thank you so much Jim . Have a great vacation
Regards
Sujit
Hello Jim,
Any suggestions on my request
Regards
Sujit
Hi Sujit, I’m on vacation now. Will reply when I return. Thanks.
I present my thesis proposal entitled Impacts of ICT on Tax Administration for the Degree Master in Development Economics, I have prepared a yes no questionnaires does ICT has an impact on cost of taxation, tax compliance and tax revenue collection, but here the challenge is which model is preferable, for the time being in the proposal I mention to use Multi-linear regression using OLS, so my question is that my model choice really good?
HI JIM ,
I am going to do master thesis on determinant/factors that affect graduate youth unemployment and which regression model you advise me to use for single dependent variable
Hi Bedilu,
I’d start with linear regression and see how that works. From there you, see how well the model satisfies the assumptions.
Best of luck with your thesis!
Hi everybody,
I’m using panel data to examine the effect of CEO age on tone of annual reports for 5 years. I have applied this model but I am not sure whether it is the best model or no. I am using Stata, year as dummies and random effect:
xtreg tone CEOage controls i.year, re robust
Any help would be hugely appreciated.
Yours Sincerely,
dan billy
Hi,
I want to use a regression model to find the number of charging stations in cities (which is a count data) with respect to different features of the cities (i.e., population, vehicle miles traveled, etc). I tried different models and linear regression gave me the best goodness of fit. Is it wrong to use linear regression for this purpose?
Hi Linda,
The Poisson distribution, which is based on count data, begins to model the normal distribution when the counts are large enough. So, it might be entirely appropriate to use linear regression. Be extra careful in checking those residual plots. If those look good, it’s probably OK to use linear regression. With smaller counts, it’s harder to satisfy the OLS assumptions.
Keep in mind, that you can only interpret goodness-of-fit statistics when the model satisfies the assumption, which you assess by residual plots. A good goodness-of-fit doesn’t necessarily indicate that the residuals are good. Read my post about R-squared for more about this issue. If the residuals aren’t good, you can’t trust the model regardless of what the goodness-of-fit statistics indicate (i.e., you can’t trust them either).
Hello Jim,
Need your guidance for an exercise where I need to establish Walkin to a store as a function of number of print inserts by a brand + print inserts by competition
Regards
Sujit
Hi Jim . If my dependent variable is “Number of job switches” , it is Discrete random variable. Can I still use MLR?
Jim, thanks for the great information. While describing PLS regression you mentioned this is a good technique for predictive models but not good for screening variables. My motivation for wanting to understand regression techniques is largely because I want to be able to make predictions about things. What about PLS regression makes it better suited for this? Can’t we make predictions from any regression technique that fits the data?
Hi Michael,
I guess I wouldn’t say that PLS is better at prediction than other methods in general. Rather, PLS is a good method to use when you face particular problems–few observations per predictor and multicollinearity. In this context, PLS is relatively good at predictions and not so good at screen variables. However, if you don’t have those problems, I wouldn’t consider using PLS and would probably start with OLS. If you have severe multicollinearity and are interesting in determining the significance of particular variables, consider Ridge or LASSO regression.
Hi Jim,
I am a junior doctor and I need to do some paper in Statistics about ,,Types of regression analysis used in biomedical sciences”. Can you help me where I can find that data and which are the types of regression analysis used in biomedical sciences? Everyone is mentioning just linear regression analysis.
Thank you in advance.
Hi,
I wouldn’t be surprised if linear regression is the most common. I’ve also heard that binary logistic regression is commonly used in things like surgical procedures where the researchers want to model the variables that affect the probability of survival. They use binary logistic regression because the outcome is binary (patient either survives or does not). I can also imagine them using binary logistic regression to model the success or failure of other treatments. I once helped a fertility doctor to model the probability of pregnancy by the treatment and other factors. Again, the outcome is binary (pregnant and not-pregnant).
Hospitals could use Poisson regression to model the availability of hospital beds. The outcome is a count of available beds.
I think the medical sciences could use many different types of regression just like other fields. It depends on the nature of the dependent variable and the characteristics of their data. I would guess that linear regression and binary logistic regression are two of the more common types.
I hope this helps!
Hi Jim,
I have a binary outcome variable (presence of disease = yes or no)
And would like to study the effect of different independent variables on the presence of disease
Examples of my independent variables are as follows:
geographic longitude
geographic latitude
age
all of which are on a continuous scale
Which regression model should I use?
Thank you
Hi Naz,
Given your binary outcome (dependent) variable, you’d need to use binary logistic regression.
Hi, Jim. I’d like to ask, I want to examine how self-compassion (iv) can predict depression (dv), self-compassion itself is formed by 6 dimensions, could you suggest what regression that would fit for my research? I already run simple linear regression for sc and depression, so if I use multiple regression for the 6 dimensions and depression, am I doing it right? or should I run simple linear regression for each dimension and depression?
an answer would be really appreciated, thank you! 🙂
Hi Elsa,
The best place to start with is least squares regression. Use multiple regression so you can include all your IVs in the model. You don’t want to fit separate models for each IV because it won’t be controlling for the other variables.
If you had to form the 6 dimensions because of need to reduce the number of dimensions, and now you want to perform a regression on the dimensions (which are not variables), you might need to use something like partial least squares (PLS) regression instead. That will find the dimensions and perform regression on them. I don’t have much experience with that type of regression. There might be other methodologies that I’m unfamiliar with that you can use. Generally, you’ll have a specific need for reducing the number of dimensions and working with those rather than just using the variables themselves.
Thank you your post,
i’m doing advance regression modeling for house price prediction, so there have very large number of variables.do you can give me a suggest a method for the dimension reduction?
Hi Randunu,
I’ve seen regression models for house prices that use least squares regression. Unless you’re using a small sample size, I’m not sure that you need to reduce the dimensions. I’d at least start your investigation using least squares regression and move away from that only if it’s really required.
Hi Jim,
what if I want to apply regression if I have variables of # of complications like: dengue, Malaria etc. and have variable of deaths due to these complications. which regression should be used in this data.
thank you!!
Hi Deepanshu,
One thing I hope to have conveyed and the readers understand about this post is that the type of regression you use large depends on the type of variables you have, particularly for the dependent variable (DV).
For your case, you don’t state how you’re recording deaths. What type of variable is the DV? Are data recorded by patient and you have a binary death/survival outcome for each patient? If so, binary logistic regression. Is it by hospital (or geographic region) and you have counts of deaths, you could use Poisson, Negative Binomial, or possibly least squares regression if the counts are high enough to approximate the normal distribution. In that case, you might need to convert it to a rate for comparison purposes.
Hi, I am a student, am still confused which type of regression should I use, the Dependent variable is binary (Benign or Malignant).
Independent variables are three
1- Age group: which are 6 groups (1,2,3,4,5,6)
2-Color: binary variable only two color option.
3- Third variable is also binary variable.
What should I choose?
thanks in advance Mr. Frost.
Hi Alaa, you’d use binary logistic regression.
Hi I have 11 continuous random variables X1 . . . X10,Y. I need to create a regression model for predicting values of Y based on the values of X1 . . . X10. Which regression should i chose.
Hi Jim, sorry for the confusion! Each observation is a single Likert Value, so I guess Ordinal logistic regression applies. Could stepwise regression still be used with this kind of data?
Jesse
Hi Jesse,
Yes, it sounds like you’ll need to ordinal logistic regression. Also, as I mentioned in an earlier comment, you’ll have some thinking to do about whether to treat the ordinal DVs as continuous or categorical. There are pros and cons both ways. More details in my ebook about regression analysis!
I mean to say discrete and continuous variables…
Hello
Thank you for your time in answering our questions.
I have a problem and I would like your help please.
I have a dependent variable that takes the value 0 and 1. This variable is explained by other independent binary and continuous variables.
Only in the survey the answers of my dependent variable are conditioned by the answer obtained to a previous question ( say Q1)
so that if Q1 = 0 “no” then Q2 (which is my outcome) is asked.
if Q1 = 1 “yes” then Q2 is not asked.
Q1 and Q2 are not exclusive and Q1 is not a selection issue. The individual interviewed can answer yes to Q1 and also answer yes to Q2. I don’t understand why the survey design excludes them and it really bothers me
My question is, apart from a multinomial model, do you see any other models that I could apply to get around this serious selection problem?
Hi Jane,
I’m not entirely clear about the scenario and what you want to analyze.
Q2 is your outcome. And, only particular respondents are presented Q2 based on responses to other questions. At first, it sounded like it depended entirely on their response to Q1, but that might not be the case. Maybe other answers as well? You’ll have to examine how the survey presents questions to respondents to understand that portion.
If you’re analyzing Q2 as your outcome variable, I’d suggest figuring out who exactly answered it. If you can identify a particular subpopulation that answers Q2, you can perform the analysis and understand that the results apply only to that specific subpopulation. If Q2 is binary, you could use binary logistic regression with Q2 as the DV.
I hope that helps!
Hi Jim, Thank you for the response! I’m not sure I understand your question, but I think they are the score of each item, the scores for each scale is not averaged or summed. I think because its a Likert style scale with an absolute zero (ratio data), we would treat it as a continuous variable. But would this be a stepwise regression method to see which variables stay in the equation (i.e., are still significant and have the highest coefficient)?
Thanks!
Hi Jesse,
I’m asking whether for each observation (i.e., row in your data sheet) the values for each variable are a single Likert value (1, 2, 3, 4, 5, 6, or 7). Or is each value an average or sum or multiple Likert scores?
It makes a difference. A straight Likert score item, as I described before, is an ordinal variable. It is not a continuous variable. However, sometimes you can treat sums or averages of multiple Likert scores as a continuous variable. In fact, if you’re using a single Likert score for each value of the DV, you’d need to use ordinal logistic regression rather than least squares regression.
Thank you for the response Jim! To clarify, we had customers fill out a survey, all likert (1-7) style questions. We had an overall satisfaction question (Are you satisfied with our company overall?) and multiple satisfaction questions based on dept (e.g. How satisfied are you with our customer service?, How satisfied are you with our webpage?).
We basically wanted to see which dept was the best predictor of Overall satisfaction… essentially, which area of our company should we focus on most based on satisfaction scores?
Thanks again!!
Thanks for the extra details! I don’t use SPSS so I wasn’t familiar with their Enter Method!
One more question. Are the values for the independent variables and outcome variable for each observation the score of one item (e.g. 1, 2, 3, 4, etc) or the sum or average of multiple items? The answer to that will affect the method you can use. Least squares regression doesn’t play well with ordinal outcome variables. However, if it’s a sum of multiple Likert items, it might work fine.
For the IVs, if you’re using the value of individual items, it’s an ordinal variable which has the properties of both continuous and categorical (nominal). You’d have to go with one or the other. (I talk about that in my ebook about regression analysis.) If you use a sum or average of several items, you might be able to treat them as continuous IVs.
In theory, you should be able to do what you want to do. However, ordinal variables (such as Likert items) can be a bit tricky and you need to work out those details and choose correct analyses.
I hope this helps!
Hi Jim,
This is a simple regression model but I’m getting a difference of opinions on the answers. I’m looking at overall customer satisfaction (likert scale), and I want to know which area of the company is contributing most to this.. each area of the company each has their own likert scale. Would I just use overall satisfaction as a DV and put the different areas of the company as IVs in a ‘Enter’ type regression?
Thank you Jim!
Hi Jesse,
I don’t know what you mean by an “enter” type regression.
Here’s what it sounds like to me that you want to do. Please let know if it’s incorrect. You want see if there’s an association between area of the company and the Likert score? If so, you can do that. The dependent variable is Likert score, which is an ordinal variable if you’re just using a single Likert item for the DV values. In this case, you’d need to use ordinal logistic regression. If you want to use area of company (e.g., department) as an independent variable, that would be one categorical variable where the different values are the various departments. You can fit that model and see if there is a statistically significant association between department and the Likert score.
If you are summing multiple Likert scores and using that summed value as the DV, rather than just using the score for an individual item, you might be able treat the DV as a continuous variable and use least squares regression or ANOVA to assess the differences by department.
I hope this helps!
And by “Enter Method’ I meant in SPSS the ‘Enter’ procedure in linear regression (as opposed to Stepwise, or backward elimination methods. I just put in all the different depts as predictors and total sastisfaction as an outcome variable, then looked at which predictors were significant, and their coefficients.
Hi Jim, thanks for all your great posts! Apologies if this has already been covered but perhaps you can offer some further clarification.
Let’s say I have two variables, a continuous variable (e.g. test performance) and a categorical grouping variable with three levels (e.g. group 1, group 2, and group 3).
My question relates to how to structure the regression analysis itself. We could model “test performance ~ group” (so test performance is the DV and group is the IV). Alternatively we could use a logistic model “group ~ test performance” (so group is the DV and test performance is the IV).
Are there rules of thumb here in deciding which way to structure the model? Are both valid or does it depend on the specifics of the experiment and the underlying data?
Any thoughts on this would be appreciated!
Hi James,
If you’re creating a model to explain the relationships between variables, you’ll use the explanatory variable(s) as the independent variable and the variable you want to explain as the dependent variable. Often you’ll use theory, subject-area knowledge, and literature to guide you about the direction of the relationship. In your example, if you think the group assignment might cause differences in the test scores, you’d use group as the independent variable and score as the dependent variable. You want to learn about how the different groups affect the test scores.
However, if the test scores measure some characteristic of the individuals that might affect the group that the individuals join, you could use test score as the independent variable and group as the dependent variable. You want to learn about the characteristic that the test measures affects the group that individuals join.
So, a large part is the direction of the relationship that you are assuming to be true based on theory and subject-area knowledge.
However, if you’re creating a model to make predictions, then you’ll always use the variable you want to predict as the dependent variable. Imagine you want to predict variable A, which is correlated to variable B. Suppose the value of A causes changes in the values of B. In an explanatory model, you’d use B as the dependent variable because that fits the known nature of the relationships. However, imagine that you want to predict the value of A. In this case, you’d use A as the dependent variable and B as the independent variable.
In the end, you can really do the analysis either way depending on whether you want the model to explain the actual relationships between the variables or you’re making predictions for a variable.
Hi Jim,
I am trying to determine the predictive power of a nominal independent variable with 8 groups (age cohorts) for a continuous count dependent variable (number of stress incidences). What test do you think would be most appropriate? My samples are also very small (<5) for some age cohorts.
sir i have this business problem-
Company collected data from 5000 customers. The objective of this case study is to understand what’s driving the total spend of credit card(Primary Card + Secondary card)
Prioritize the drivers based on the importance.
and i have to built linear regression model.
please help me out like how to deal with two dependent variables at a same time with a set of predictors
Hi Âákäsh,
You’re asking about learning how to use, perform, and interpret multiple regression analysis. That’s way more than I can write about in a comment!
However, you’re in luck, I have written an ebook all about regression analysis, with a focus on multiple regression. Please take a look at my intuitive guide to regression ebook!
Hi Jim,
Thanks for such an insightful article!
I am trying to predict the best time (date) to contact a customer for an offer by analysing purchase history. I want to use Regression for this but having a hard time to decide which one to use. I have considered using logistic Regression where I thought I will see if in a month it is best time or not. But on the other hand also think that date is a continuous variable so I might be better off using multiple linear regression.
Would be very helpful to hear your thoughts.
Many thanks,
Naty
Hi Naty,
In making this decision, a good place to start is to understand your dependent variable. What’s the outcome that you want to explain/predict. What type of variable is it? If it’s something binary, like Sale/No Sale, and you want to use independent variables such as date/time to predict that, you’d need to use binary logistic regression. With binary logistic regression, you can use independent variables such as time (continuous) and day of week (categorical/nominal) to predict the binary dependent variable of sales/no sales.
That sounds like the direction you might want to go, but it really depends on the nature of your dependent variable.
Dear Sir,
If both independent and dependent variables are categorical and dependent variable have natural order, which test variables do you suggest?
Hi, sounds like you should use ordinal logistic regression!
I don’t know what you’re studying. You’ll have to use your subject area knowledge to determine which test variables to include. Depends on what you’re testing!
i am trying to analyse car accident data set.
there is 1 dependent and 1 independent variable in my equation
dependent variable is categorical (4 category) and represents number of wounded person
1- (1 person wounded)
2- (2 person wounded)
3- (3 person wounded)
4- (4 or more person wounded)
independent variable is also categorical (6 category) and represents crash type
1- (rear crash)
2- (side crash)
etcetera……
which type regression should i use to analyse crash type effect on wounded person number?
Hi Michael,
There would seem to be a variety of problems with modeling this data set.
The DV is sort of a count variable except for the four or more category. For count data, you can often use Poisson or Negative Binomial regression as I indicate in this blog post. However, because of the four or more category, it’s not really count data. I suppose the best would nominal logistic regression and treat your DV as a categorical variable.
However, there would seem to be too much information left out of your model to get meaningful results Number of passengers in the car, speed, and seat belt usage just to name three quickly. You might have the worst possible accident yet only have 1 person injured if there is only one person in the vehicle. I suppose if you had a large enough sample size you might get some meaningful information.
Again, you can try nominal logistic regression. Or even perform a chi-square test of independence because you have two categorical variables.
Dear Jim,
I have scale (income) dependent variable and both scale and categorical variables. Which regressuin model is appropriate?
Dear Sir,
I have a continuous dependent variable and a categorical independent variable with three groups.
whereas my covariates are continuous and few are categorical as well. Which statistical test can i apply?
can i use linear regression or ancova?
Dear Sir,
If both Independent and dependent variables are catagorical variables, Which test you suggest to test variables except Chi-square sir?
Hi Ranjith, you can use nominal logistic regression. This type uses categorical dependent variables with categorical and/or continuous independent variables.
Hi Jim!
Thank you for excellent explanations. I hav plotted my data and my curve looks more like a logarithmic graph. How do I best calculate the correlation between my two variables?
Lore
Hi Lore,
It sounds like you might need a log-log relationship, which you can read about in my post about log-log plots. For details on a variety of other methods you can use, read my post about curve fitting.
Ah shoot, I noticed after replying that you’re asking about correlation rather than how to fit the curve! I need more coffee! Try using the Spearman Correlation which can assess monotonic relationships. That’s a fancy word that means that the variables tend to move in the same relative direction (either a positive or negative correlation) but the rate of change is not constant.
Hi Jim,
Thank you for posting several useful information regarding data analysis. I follow your website regularly for statistical matter.
I have one question. Can we natural log transform independent continuous variable. I have independent variable of different distance points (1 to 180 m with nineteen measured distance points with inconsistent data ranges).
My dependent variable is over-dispersed count data (and I am using negative binomial with log link function).
Please could it be possible for you to answer this question.
Thank you.
Hi Sunita,
Thanks for reading! I’m happy to hear that you find my site to be helpful!
Yes, it is possible to use the natural log to transform a continuous independent variable. Of course, there’s no way I’d know whether it’ll resolve whatever problem you’re experiencing. I’d recommend fitting the model without the transformation and checking the residual plots. Then, transform your variable and fit the model again and check the residual plots to see if it helped. Transforming the IVs is particularly helpful for fitting specific types of curvature, which you can assess using residual plots.
Using transformations is a bit of a trial by error process. There are some guidelines that can help, but it often comes down to trying them to see if they do the trick. I write about this in detail my regression ebook. Although, I don’t write about it in the context of negative binomial regression.
I hope this helps!
Thank you for your time, Jim. Looks like i have to crack my head for some time.
Elsa
Hi Jim, I’m completely lost when trying to choose the type of regression analysis for my problem. Is it nominal logistic regression, time series, linear (autoregressive model), nonlinear or other? Can you please give me some advise?
345
658
116
380
651
851
709
196
269
071
478
518
451
989
519
843
662
428
794
949
Suppose, everyday I’m given 20 data as above. They’re random numbers generated by three devices, namely Device A, Device B and Device C. Every device is capable to generate random numbers from 0 to 9.
For the above first data of “345”, “3” was generated by Device A, “4” was generated by Device B and “5” was generated by Device C. Similarly, for the last data of “949”, “9” was generated by Device A, “4” was generated by Device B and the last “9” was generated by Device C.
I want to be able to predict the next outcome variable after the last “949”.
I have a total of 300 historical data for 15 days, 20 data per day.
From my rough observation for the recent 300 data, every device tends to produce repeated sequences. For instance, Device A will repeat the sequence “8-6-4-7-9” (as in the last 5 data). That means this sequence appeared at least twice within the 300 historical data for Device A. Similarly, for Device B and Device C, each of them tends to produce repeated sequences. For another example, the repeatation of the sequence “5-1-8-5-5” (the 2nd to the 6th data) in Device B.
I’d like to predict the next outcome variable after the last “949”. Is it possible to model it with nominal logistic regression, or time series (if i assume that 20 data was taken at successive equally spaced points in time)?
Otherwise, which model is more appropriate?
Thank you in advance.
Elsa
Hi Elsa,
Based on the limited knowledge that I have, this problem doesn’t sound like one that regression analysis can solve. If it’s random as you say, then you can’t predict it. However, if there are patterns in the data, it might be more of a pattern finding problem.
If there was some underlying phenomenon that influenced the numbers that the devices display, you might be able measure the phenomenon and link it to outcomes and use regression analysis to model that process.
Sorry I couldn’t be of more help. But it sounds like you’ll need to use something other than regression.
Hi Jim,
Great website. I have a quick question, excuse me if you have covered it before and i did not notice. When you build a regression model, multiple linear regression model or just simple one, do you need to have all variables with the same units to conduct the regression? . For example, if i want to study the relationship between stock market returns which is in % terms and funds flow which is in dollar terms, do i need to transform both variables to the same unit of measure ie lets say percentage % to study the regression? Thank you.
Hi Jim,
Thank you so much for your reply, very helpful but I am just unsure as to how to test for a linear relationship between a categorical IV and continuous DV… Could you please advise?
When trying to do scatter plots you just get a straight line…
Thank you in advance for any help you can offer!
Hi Sophie,
When you have a categorical IV, the type of relationship you’re testing for differs than between two continuous variables. When you have a categorical IV and a continuous DV, the analysis tests for differences between group means. The categorical IV defines the groups in your data. It’s performing an ANOVA with your data to see if the DV mean varies between groups. And, you don’t use a scatterplot to graph that type of relationship but either a boxplot or individual value plot.
I don’t want to keep pushing my book, but I go through an example of a regression model with both categorical and continuous IVs in it and show how to interpret the results.
Hi Jim,
I love the way you explain the different types of regression so I was wondering if you could help me on a project I am doing…
My IV is bullying victimisation which is categorical (never, some of the time, all of the time) and my DV is internalising problems which is on a scale from 1-20. I also have covariates some continuous, some categorical. What statistical method do you suggest I do? I have quite a few outliers in the DV but as I am using the MCS these are hard to remove and due to their being quite a lot I don’t believe I have a logical reason to remove them. I am also not sure on how to test for a linear relationship with my IV being categorical.
Do you have any suggestions for the above?
Look forward to hearing your thoughts!
Thanks!
Sophie
Hi Sophie,
In your other message, you clarified that the DV is a continuous variable, although constrained to a range of 1-20. Consequently, I’d start by using a least squares linear model first. Try that first and see if you can get a good fit–check the residual plots. Specify a variety of models and possible fit curvature and interaction effects. The constricted range of the DV might be a problem if it causes the scores to be extremely skewed. If your DV is very skewed, you might need to use a data transformation. The residual plots will help you determine what you need to do.
As for the outliers, if they’re a regular part of the distribution of outcomes for your population, it’s often best to leave them in as they represent variability that is inherent in the process. However, if they represent scores from people outside your defined population, or some other fluke during measuring, testing, etc., that doesn’t represent the typical process, consider excluding outliers. Just be sure to record the removed outliers and explain why. But, the default position should be to leave them in unless you have a good rational for removing them.
I write more detail about the above in my ebook about regression. You might consider getting because you’re using for your study.
Best of luck with your study!
Hi Jim,
Could you please suggest any source where I can find why ordinal regression is preferable over multiple regression analysis. Even if you have written any thing on it that will be also great. I am in argument with my team to use ordinal but they says opposite so need some strong sources. I will be very thankful to your help!
Hi Syed,
When you have an ordinal dependent variable, you should use ordinal logistic regression. Ordinary least squares regression is unlikely to provide value results when you have an ordinal dependent variable. Ordinal logistic regression was designed with ordinal DVs in mind. This decision is based on the type of variable you’re using for the dependent variable.
First, ensure that you have an ordinal dependent variable. If you do, I’m not sure why your team would resist ordinal logistic regression?
I don’t have a reference on hand at the moment, but I will check when I can. However, any advanced regression textbook that covers ordinal logistic regression will state this clearly!
Best of luck with your analysis!
Hi Jim,
Thanks for your helpful blog.
I’m hoping you might be able to offer some advice on a project I’m working on to understand customer telephone contact. The sort of questions I’m looking to answer are as follows –
Are customers with certain characteristics more likely to contact us?
Do customers with certain characteristics contact us more frequently?
I have a snapshot from our customer database including customer characteristics such as age, gender, marital status, whether have children, geographical location, etc, as well as the number of telephone calls made in that month. One row for each customer.
My rough plan is to look at the data and run off some descriptive statistics (scatter plots, box plots, etc) to see how frequency of contact varies by different characteristics. Then depending on what that shows, I’m hoping to construct a regression model(s). This is where I’m a little stuck in terms of the type of model.
I’m planning to start by looking at the probability of a customer making a call (a binary logistic regression?) I think this would tell me which characteristics increase/decrease the probability of a customer making a call versus no call.
I’m then thinking an alternative would be to define the dependent variable on the basis of low/medium/high frequency callers and run a regression on that basis. I think an ordinal logistic regression?
Could I also use calls per customer per month as the dependent variable and run a standard linear regression? Or would this be classified as count data and require a different model?
Grateful for any thoughts.
Kate
Greetings Jim
I conducted a simple multiple regression. I firstly tested all the assumptions and then created the models. I would like to know how I can validate these models? Do I have to perform additional tests or can I use information from the model analysis outputs? NB. I do not want to run complex tests that are difficult to interpret. Kind regards and many thanks,
Mario
Hi Mario,
Unlike most statistical procedures, with regression analysis you need to check the assumptions after you fit a model. You can’t check the assumptions before fitting the model. Sometimes you can get a sense that it’ll be hard to satisfy the assumptions before fitting the model. For example, if your dependent variable is highly skewed, it’ll be harder to satisfy the assumptions. So, fit the models and then check the assumptions. To learn more about that process, read my post about Checking the OLS Assumptions.
When you say “validate” the model, there are several ways I can interpret that. One is that you want to ensure that it satisfies all of the assumptions. If you mean that, read my blog post above. It doesn’t involve performing any complex statistical tests.
However, often people will validate a model when they use it to make predictions. They want to ensure that the model predicts new observations as well as it predict the observations used during the model fitting process. If this is what you mean, you’ll need to have a training dataset and a validation dataset. Use the training dataset to fit the model. Then, apply the model to a separate validation dataset and assess how well the model predicts the new set of observations. This process is known as cross-validation.
Because your study involves regression analysis, you might consider buying my ebook: Regression Analysis: An Intuitive Guide. My ebook covers the material in more depth than the blog posts.
Best of luck with your analysis!
Hi Jim,
Thanks a lot for the excellent article!
Can I ask if I have a mix of continuous and categorical dependent variables, can I still use partial least square regression?
Thanks!
Hi Min,
As far as I’m aware, you can use PLS only with continuous dependent variables.
Thanks a lot Jim
Hi Jim you did not reply to me? 🙁
Hi Abbas,
So sorry about that! Sometimes when I get really busy some questions are accidentally missed. I’ll answer your previous question within a day.
Great post Jim. I really like the way you explain the different types of regression.
But I’m little bit confused here. What if my dependent variable is a score, say having value 1-100 & my independent variable is a categorical one? Is poisson regression ok in this case?
Hi Tanvir
Thanks for the kind words!
If you working with a score that has at least 10 values, you can often treat it as a continuous variable. Consequently, I’d try using linear regression.
Poisson regression wouldn’t really be the right choice because you’re not counting outcomes. That’s usually what you use Poisson regression for. Even then, if the mean is high enough, the Poisson distribution begins to approximate the normal distribution and you might still be able to use linear regression.
At any rate, I’d start with OLS regression. If the scores are highly skewed, you might need to transform the data.
Best of luck with your analysis!
Dear Jim,
What type of regression would you recommend in a case when the key independent variable is not regularly reported? My dependent variable is a firm’s investment policy reported every fiscal quarter. The independent variable of the interest is a debt covenant strictness reported just at the moment when the firm issues any loans. I’ve tried to use investment changes over next 4 quarters (8 quarters) after the loan inception as a dependent variable. But I’m not 100% sure that it is a correct empirical design. Is it possible to use as the dependent variable also the investment levels instead?
Thank you!
Best,
Anastasia
hi which one can be the best model when you want to establish the relcatiionship between tax consultant and tax compliance
Hi Prosper,
Usually you don’t pick the type of regression model based on the subject area. In other words, there is no model that is best for taxation studies. Take a look at the type of dependent and independent variables you will use in your model and then read through this post to see which models can handle those types of variables.
Best of luck with your study!
Hi Jim,
Loved your post.
I wanted a suggestion from you regarding my project work.
Could you please suggest which model I can use for regression.
Following is the brief overview of the problem I am facing:
My dependent variable(y) is a non-negative integer and independent variables are categorical as well as continuous variables.
There are 3 continuous variables and 4 categorical variables on which(y) would depend.
Could you please suggest which regression models should I test in this case?
Hi Aki,
If your non-negative integer is a count variable, look in the section for Count Dependent Variables for ideas.
Best of luck with your analysis!
Hi Jim,
I have been reading your articles on regression analysis – they have been super helpful for me as I am just learning about regression analysis!
I am writing a research proposal on what factors influence voters to still vote for their preferred party when they have knowledge of that parties involvement in corruption.
previous research has indicated that voters continue to support the part as they personally benefit from the corruption. Whereas other academic conclude that loyalties to certain parties are a result of a clouded vision causing cognitive dissonance. The most frequently argued factor as to why individuals forgive a corrupt official is that if the economy are secure and improving. Finally, the more extreme the individuals party is (far left or far right) the more likely they are to vote for another party due to a lack of viable alternatives.
I wanted to use a MCQ questionnaire to collect data on all these factors and treat them as independent variables and use regression analysis. Is this possible? and if so how should I go about this. I have spent about 2 weeks on the proposal and I am not totally sure
Best wishes
Catherine
Hi Catherine,
I’m so happy to hear that my regression articles have been helpful! If you like them, you might consider my ebook, which is all about regression analysis.
Yes, it is possible to include MCQs in an analysis. The manner in which you include them depends on the type of answers. If they’re categorical (nominal) variables, such as, “What type of damage? Scratch Dent Tear,” you’d include it as a categorical IV. If it’s an ordinal variable, such as Likert scale responses, they straddle the line and you can potentially enter them in the model as either categorical variables or continuous variables. It depends on the nature of the data and the goals or your analysis. My ebook talks about all of these aspect, categorical IVs and how to handle ordinal IVs.
Best of luck with your analysis. It sounds interesting!
Hello Jim
I trust you are well.
Could you perhaps tell me how I should go about testing the assumptions for generalized linear models.
Thanks,
Mario
Hi, does your book cover all the regression analyses you covered in this blog post?
Hi Josh,
The ebook focuses on linear regression. I cover many aspects of that including, continuous and categorical IVs, main and interaction effects, modeling curvature, checking assumptions, goodness-of-fit, predictions, etc. I offer a few glimpses into the other types of regression, but they’re not the focus.
I can foresee a future book going beyond linear regression and covering the other types in more detail.
To get a better idea of exactly what the book covers, go to my blog post about my ebook. At the bottom you’ll see screenshots of the full TOC.
You can also go to My Store and download a free sample of the ebook, which has the complete first two chapters.
I hope this helps!
Great thread, this was really informative and has made me rethink my current regression model.
My undergraduate dissertation project is on how perfectionist concerns, strivings, and coping skills (7 sub scales) influence sport acute injury occurrence and frequency. The perfectionism data cam in the form ofa composite score made of two subscales from different questionnaires with different item number (5 and 7 items). Thus, a standardised composite score was created. which regression analysis tests would you recommend for an SPSS newbie?
Thanks
Hi Jim,
It would be great to see the kinds of estimates that can be obtained from each of these models, and possibly some of the interpretations that might come with using one of these models. Thanks!
-Frank
Hi Frank,
In future posts, I’ll do that! For now, you can see examples of nonlinear regression and binary logistic regression. I’m always adding posts and will get to the others at some point.
Thanks for writing!
Hi Jim,
I’m having a bit of trouble selecting the right non-linear regression for my analysis. I have data that looks like an inhibition dose response curve with a variable slope. The X is not dose however it is time and I am trying to compare the rate of closure between multiple samples over time. If I do a non-linear fit of [inhibitor] vs response with a variable slope, the curve fitting fits my data very well. I was wondering if in this scenario using a generated hill slope would be sufficient to evaluate rate? Would there be a more appropriate regression or should I consider an alternative such as area under the curve?
Thank you so much for any advice!
Hi, I often find that in nonlinear regression models, subject-area knowledge becomes even more important than it usually is. In nonlinear models, you’re often trying to incorporate underlying physical or biological processes into the regression equation itself. I once saw biologists debate over which nonlinear model is best for plant growth. It turns out that the best model depends on a plant’s specific growth pattern. All of this is to say, I don’t feel like I’m qualified to answer your question. You should consult with subject-area experts and see how similar studies have tackled the problem. Sorry I couldn’t provide a more specific answer!
Hi Jim,
I’m searching for a regression model for the following: I have a categorical dependent variable which comes from a multiple options question, I mean, the individual was being asked for his preference about six different saving alternatives. He could choose all the alternatives that he prefers (he can choose even all the six alternatives). So, I don’t know which could be the type of regression that I could apply.
In conclusion, my dependent categorical variable is non-excluding so I can’t apply a multinomial logit or multinomial probit. Which other models could I use?
Thanks in advance.
Thanks Jim. Very appreciate it
Hello Jim
I have a question about my study. Both the IV and DV in my data are categorical. IVs (nominal) are 5 types of behaviors and DVs (ordinal) are 3 types of reasoning. I am interested in how IVs predict DVs. I am wondering if logistic regression fit my study? Thanks in advance for your efforts.
Hi Franco,
It sounds like you need to use ordinal logistic regression!
Hey Jim,
I got a query.
Suppose we are given voting turnout results (1 for yes, 0 for no) of 4 previous years. And we want to predict it for the next year, and hence we do not have the 2019 result column. Logistic regression need X train and Y train, but in this case we do not have a Y train. How should I proceed with this? Or should I use some other model?
Dear Jim, I came across this article of yours. Very informative. I am a student doing chiasma count on chromosomes. Now I want to compare and contrast different varieties just based on this count number. The count range is limited ranging from 2 to 8. I cannot go for Poisson distribution as it is known that the chiasma occurrence on chromosomes is not random. Infact, two pathways occur, one is non random and second is random, but most of the counts are accounted by non random pathway. I do not know this, just based on the count data. Can you suggest any analysis which can be done to find out differences between varieties.
Hi Jim!
Fantastic blog post. Thank you for laying this out so clearly.
For the regression I am trying to run, the independent variable is number of classes attended. The possible options being 1, 2, 3, and 4 classes attended. To me, this seems like an ordinal variable, but I am not sure.
Is it okay to proceed with a regular linear regression? The output variable is change in symptoms of depression (a continuous variable).
Thank you in advance!
Allie
Hi Allie,
That’s a count variable. Count variables can have non-negative integers. This type of variable is different from a ordinal variable.
An ordinal variable does have a natural order. You can represent ordinal variables using numbers. So, that looks similar to a count variable. But, suppose you are recording how runners finish in a race. You have the first, second, and third place runners that you record as 1, 2, and 3. However. the second place finisher doesn’t necessarily take twice as long as the first place finisher even though 2 is twice the value of 1. You only know that 2nd place is after 1st place. 3rd place is after 2nd and 1st, but it’s not necessarily three times as long as 1st.
Contrast that with counts of 1, 2, and 3. A count of 2 is twice a count of 1. A count of 3 is three times 1. The difference between 1 and 2 is the same as the difference between 2 and 3. Etc.
When you have count IVs and ordinal IVs, neither are quite continuous (ratio) nor are they quite categorical (nominal). You’ll have to decide which way to enter them into your model. That depends on the nature of your data and the goals of your analysis. I write about this in my regression ebook that I just released. But, that’s the issue in a nutshell.
It’s often OK to include this type of variable in a linear model. Just be sure, as always, to check those residual plots to be sure that your fitting an adequate model!
Hi Jim!
First of all many thanks for your helpful, comprehensive tutorial on regression. I have a question that I really appreciate if you reply to it. What is the best possible way to choose the most relevant features in a linear regression model? How we can check the features?
Thanks in advance for your time
Hi Shohreh,
I’m not sure what you mean by “relevant features”? That can range anything from the type of data that a type of regression can model to key statistical output. Can you be more precise what you’d like to know? Thanks!
Hi Jim,
Great and very helpful blog. the way you explain is really helpful.
I want to run a regression on SPSS using customer satisfaction scale (5 point scale) as dependent variable. and other variables as independent variables such as some are on 5 or 10 point scale, some are yes/no. so basically I want to check what are the variables that impact customer satisfaction. Could you please help me to find out which regression analysis I suppose to use?
Hi Abbas,
Sorry about the delay in replying!
Because your dependent variable is an ordinal variable, you need to use ordinal logistic regression. In SPSS, you should use either the PLUM (easier to use) or GENLIN procedure. You can find PLUM in the Output Management System Control Panel under Command Identifiers.
Best of luck with your analysis!
Hello Jim,
First and foremost, thanks a lot for writing this blog – it’s pure gold.
However, I have a dilemma.
I have the data with 5 variables and with 100 units of observation. Every 10 units in the data encompass specific time frame – so first 10 units encompass one year, then next 10 units encompass the following year etc. For each of those 10 units, I have a distribution of 100% for each variable. This means that each variable in the first 10 units makes 100%, then the next 10 units also 100% etc. My dilemma is that units reflect the distribution of something (which makes a sum of 100% for each year) so I am not even sure can this be considered a continuous variable. Here is an example: https://i.imgur.com/dxCQfHT.png
But, to be concrete, I would like to know whether I can use multivariate regression analysis with this data where I would focus on one variable as DV and the other as IV in order to see which IV impact the most DV.
Hi Chris,
Thanks for kind words about my blog! 🙂
It sounds like you want to perform regression analysis with time series data. There are no inherent problems with that but there are special issues you’ll need to address. For example, you’ll need to include time related independent variables in your model, such as the date and lagged variables. You’ll need more specific date information than just the year. You’ll need to know the time between observations. You’ll also need to pay particular attention to correlation between adjacent residuals. See assumption #4 in my post about OLS assumptions.
I don’t see a problem with having the total within a year sum up to 100% of that year. That’s going to be true with all time series data! At least if I understand what you’re saying.
I don’t see an inherent problem with doing what you want to do as long as you know the time between the observations. If there’s a constant time between observations (observations occurred at fixed intervals) you can probably use the ID column as an IV. Just be aware of the issues I mention. In addition to the time related variables, it’s OK to include other IVs. Regression with time series data is not my specialty, so that’s about as far as I can take you. But, it is doable.
If you can fit a good model, here’s a post that can help you determine which IV is the most important.
Hello Jim, thank you for your helpful posts. I am trying to find out influence statistics for 32 states in a country and the states are in scale, nominal variables. How do I conduct spss with that kind of variable?
Hi Bella,
I’m not completely sure what you’re referring to by influence statistics. Are you referring to variables that influence an outcome? Or, perhaps statistics that identify unusual observations in a regression model that have a large influence on the results if they were to be removed from the dataset?
Thanks a lot
Another questions:
can me use Spearman coefficient to check the correlation between binary variable and continues variable?
And can me use person chi square between nominal and nominal or ordinal variables?
Hi, no, you can’t use Spearman’s correlation for that. With binary data, you have two groups. You can use a 2-sample t-test to determine whether the difference between the means of the continuous variable for these two groups is statistically significant.
Hi Jim,
first, thanks for this great website! Very helpful.
I have a question: I have a dependent variable that has been recorded as nominal categories (i.e. defoliation of a tree 0-25%, 26-50%, 51-75%, 76-100%). Should this be treated as nominal and a nominal logistic regression should be used although the intervals are equal?
I have also computed an tree attack level score based on similar categories (defoliation, discoloration, amount of resin flow etc.) by summing them, resulting in scores between 5 and 16, can this score be treated as a continuous variable and ols be used?
Thank you very much if you have time to answer!
Best,
Samuli Junttila
Hi Samuli,
I’m glad to hear my website has been helpful!
Your dependent variable has categories that have a natural order, Consequently, your DV contains ordinal data and you should use ordinal logistic regression.
I’m assuming the tree attack level score is the independent variable? If so, it is most likely fine using it as a continuous predictor. It appears you have a good range of possible values (5-16) as long the actual scores use most of the values. However, you can’t use OLS because you have an ordinal DV.
Be sure to check those residual plots! That’s always important but even more so when you’re combining non-continuous variables to form a different continuous variable.
Best of luck with your analysis!
Hi Jim
Have a nice day
Thank you for above great article, I have some questions I hope to give me advice.
I will conducting study about Risk factors of Anemia among children (< 6 years ) in Sudan, the DV is childhood anemia I will classifying according to hemoglobin concentration level (HCL) as anemic (HCL=11 g/dl)
IV’s are: sex of the child, Birth order, child’s birth weight, child’s age in months, mother’s education level, mother’s age at birth …. etc
this data I will analysis in two stage, first stage I will using bivariate correlation to identify the association between DV (childhood anemia) and IV’s to check whether the IV’s are significantly associated with anemia or not.
In the second stage I will using the ordinal logistic regression between DV and IV’s ( which associated with DV in the first stage) to make a model, and chose the best model based on Akaike Information Criteria (AIC) and 2-log-Likelihood
The End.
My question Is this method it is correct?
Hi,
It sounds like your dependent variable is binary–anemic yes or no. If that’s true, you need to use binary logistic regression.
Additionally, Assuming the DV is binary you can’t use correlation to assess the relationship between the independent variables and the dependent variable. If you have the raw continuous HCL data, you can use correlation between the continuous IVs and HCL. Even though you’re using a binary form of HCL, that correlation with the continuous form can provide meaningful information. I would actually use scatterplots to graph the IVs by HCL. That will allow you see whether there is curvature you need to model.
You won’t be able to use correlation with your IVs like education level because that’s a categorical variable. However, you can use graphs like boxplots or individual value plots to see whether there are differences between levels.
Finally, assuming your DV is binary, use binary logistic regression. Assess the CIs for the odds ratio for each DV and determine whether it excludes 1. Check the residual plots!
Best of luck with your analysis!
Jim,
If you have say 6 IVs and they are all categorical, doesn’t it limit the conclusions somewhat? Wouldn’t a mix of continuous and categorical be better? In other words, isn’t there more information in using interval variables or at least keeping the number of categorical variables to a minimum?
thanks!
Skip