• Skip to secondary menu
  • Skip to main content
  • Skip to primary sidebar
  • Home
  • About Me
  • Contact Me

Statistics By Jim

Making statistics intuitive

  • Basics
  • Hypothesis Testing
  • Regression
  • ANOVA
  • Time Series
  • Fun
  • Glossary
  • Blog
  • My Store

Regression

Using Data Mining to Select Regression Models Can Create Serious Problems

By Jim Frost 12 Comments


Data mining and regression seem to go together naturally. I’ve described regression as a seductive analysis because it is so tempting and so easy to add more variables in the pursuit of a larger R-squared. In this post, I’ll begin by illustrating the problems that data mining creates. To do this, I’ll show how data mining with regression analysis can take randomly generated data and produce a misleading model that appears to have significant variables and a good R-squared. Then, I’ll explain how data mining creates these deceptive results and how to avoid them. [Read more…] about Using Data Mining to Select Regression Models Can Create Serious Problems

Filed Under: Regression Tagged With: analysis example, conceptual, interpreting results

Five Reasons Why Your R-squared can be Too High

By Jim Frost 16 Comments

When your regression model has a high R-squared, you assume it’s a good thing. You want a high R-squared, right? However, as I’ll show in this post, a high R-squared can occasionally indicate that there is a problem with your model. I’ll explain five reasons why your R-squared can be too high and how to determine whether one of them affects your regression model. [Read more…] about Five Reasons Why Your R-squared can be Too High

Filed Under: Regression Tagged With: conceptual

Overfitting Regression Models: Problems, Detection, and Avoidance

By Jim Frost 49 Comments

Overfitting a model is a condition where a statistical model begins to describe the random error in the data rather than the relationships between variables. This problem occurs when the model is too complex. In regression analysis, overfitting can produce misleading R-squared values, regression coefficients, and p-values. In this post, I explain how overfitting models is a problem and how you can identify and avoid it. [Read more…] about Overfitting Regression Models: Problems, Detection, and Avoidance

Filed Under: Regression Tagged With: conceptual

Guide to Stepwise Regression and Best Subsets Regression

By Jim Frost 13 Comments


Automatic variable selection procedures are algorithms that pick the variables to include in your regression model. Stepwise regression and Best Subsets regression are two of the more common variable selection methods. In this post, I compare how these methods work and which one provides better results. [Read more…] about Guide to Stepwise Regression and Best Subsets Regression

Filed Under: Regression Tagged With: analysis example, interpreting results

How to Interpret Regression Models that have Significant Variables but a Low R-squared

By Jim Frost 56 Comments


Does your regression model have a low R-squared? That seems like a problem—but it might not be. Learn what a low R-squared does and does not mean for your model. [Read more…] about How to Interpret Regression Models that have Significant Variables but a Low R-squared

Filed Under: Regression Tagged With: conceptual, graphs

How High Does R-squared Need to Be?

By Jim Frost 2 Comments

How high does R-squared need to be in regression analysis? That seems to be an eternal question. [Read more…] about How High Does R-squared Need to Be?

Filed Under: Regression Tagged With: conceptual

Making Predictions with Regression Analysis

By Jim Frost 30 Comments

If you were able to make predictions about something important to you, you’d probably love that, right? It’s even better if you know that your predictions are sound. In this post, I show how to use regression analysis to make predictions and determine whether they are both unbiased and precise. [Read more…] about Making Predictions with Regression Analysis

Filed Under: Regression Tagged With: analysis example, assumptions, interpreting results

Curve Fitting using Linear and Nonlinear Regression

By Jim Frost 31 Comments


In regression analysis, curve fitting is the process of specifying the model that provides the best fit to the specific curves in your dataset. Curved relationships between variables are not as straightforward to fit and interpret as linear relationships. [Read more…] about Curve Fitting using Linear and Nonlinear Regression

Filed Under: Regression Tagged With: analysis example, assumptions, conceptual, interpreting results

How To Interpret R-squared in Regression Analysis

By Jim Frost 91 Comments

R-squared is a goodness-of-fit measure for linear regression models. This statistic indicates the percentage of the variance in the dependent variable that the independent variables explain collectively. R-squared measures the strength of the relationship between your model and the dependent variable on a convenient 0 – 100% scale. [Read more…] about How To Interpret R-squared in Regression Analysis

Filed Under: Regression Tagged With: conceptual, interpreting results

How to Interpret P-values and Coefficients in Regression Analysis

By Jim Frost 194 Comments


P-values and coefficients in regression analysis work together to tell you which relationships in your model are statistically significant and the nature of those relationships. The coefficients describe the mathematical relationship between each independent variable and the dependent variable. The p-values for the coefficients indicate whether these relationships are statistically significant. [Read more…] about How to Interpret P-values and Coefficients in Regression Analysis

Filed Under: Regression Tagged With: analysis example, conceptual, interpreting results

R-squared Is Not Valid for Nonlinear Regression

By Jim Frost 17 Comments

Nonlinear regression is an extremely flexible analysis that can fit most any curve that is present in your data. R-squared seems like a very intuitive way to assess the goodness-of-fit for a regression model. Unfortunately, the two just don’t go together. R-squared is invalid for nonlinear regression. [Read more…] about R-squared Is Not Valid for Nonlinear Regression

Filed Under: Regression Tagged With: assumptions, conceptual

How to Interpret Adjusted R-Squared and Predicted R-Squared in Regression Analysis

By Jim Frost 104 Comments

R-squared tends to reward you for including too many independent variables in a regression model, and it doesn’t provide any incentive to stop adding more. Adjusted R-squared and predicted R-squared use different approaches to help you fight that impulse to add too many. The protection that adjusted R-squared and predicted R-squared provide is critical because too many terms in a model can produce results that you can’t trust. These statistics help you include the correct number of independent variables in your regression model. [Read more…] about How to Interpret Adjusted R-Squared and Predicted R-Squared in Regression Analysis

Filed Under: Regression Tagged With: analysis example, conceptual, interpreting results

How to Interpret the Constant (Y Intercept) in Regression Analysis

By Jim Frost 65 Comments


The constant term in regression analysis is the value at which the regression line crosses the y-axis. The constant is also known as the y-intercept. That sounds simple enough, right? Mathematically, the regression constant really is that simple. However, the difficulties begin when you try to interpret the meaning of the y-intercept in your regression output. [Read more…] about How to Interpret the Constant (Y Intercept) in Regression Analysis

Filed Under: Regression Tagged With: conceptual, graphs, interpreting results

Check Your Residual Plots to Ensure Trustworthy Regression Results!

By Jim Frost 55 Comments

Use residual plots to check the assumptions of an OLS linear regression model. If you violate the assumptions, you risk producing results that you can’t trust. Residual plots display the residual values on the y-axis and fitted values, or another variable, on the x-axis. After you fit a regression model, it is crucial to check the residual plots. If your plots display unwanted patterns, you can’t trust the regression coefficients and other numeric results.

In this post, I explain the conceptual reasons why residual plots help ensure that your regression model is valid. I’ll also show you what to look for and how to fix the problems. [Read more…] about Check Your Residual Plots to Ensure Trustworthy Regression Results!

Filed Under: Regression Tagged With: assumptions, conceptual, graphs

How to Interpret the F-test of Overall Significance in Regression Analysis

By Jim Frost 113 Comments


The F-test of overall significance indicates whether your linear regression model provides a better fit to the data than a model that contains no independent variables. In this post, I look at how the F-test of overall significance fits in with other regression statistics, such as R-squared. R-squared tells you how well your model fits the data, and the F-test is related to it. [Read more…] about How to Interpret the F-test of Overall Significance in Regression Analysis

Filed Under: Regression Tagged With: conceptual, interpreting results

Multicollinearity in Regression Analysis: Problems, Detection, and Solutions

By Jim Frost 151 Comments


Multicollinearity occurs when independent variables in a regression model are correlated. This correlation is a problem because independent variables should be independent. If the degree of correlation between variables is high enough, it can cause problems when you fit the model and interpret the results. [Read more…] about Multicollinearity in Regression Analysis: Problems, Detection, and Solutions

Filed Under: Regression Tagged With: analysis example, conceptual, interpreting results

The Difference between Linear and Nonlinear Regression Models

By Jim Frost 38 Comments

The difference between linear and nonlinear regression models isn’t as straightforward as it sounds. You’d think that linear equations produce straight lines and nonlinear equations model curvature. Unfortunately, that’s not correct. Both types of models can fit curves to your data—so that’s not the defining characteristic. In this post, I’ll teach you how to identify linear and nonlinear regression models. [Read more…] about The Difference between Linear and Nonlinear Regression Models

Filed Under: Regression Tagged With: conceptual

Standard Error of the Regression vs. R-squared

By Jim Frost 116 Comments


The standard error of the regression (S) and R-squared are two key goodness-of-fit measures for regression analysis. While R-squared is the most well-known amongst the goodness-of-fit statistics, I think it is a bit over-hyped. [Read more…] about Standard Error of the Regression vs. R-squared

Filed Under: Regression Tagged With: analysis example, interpreting results

Statistical Analysis of the Republican Establishment Split

By Jim Frost Leave a Comment

Back in 2014, House Speaker John Boehner resigned, and then Kevin McCarthy refused the position of Speaker of the House before the vote. The Republican’s search for a new speaker ultimately led to Paul Ryan. Simultaneously, the Republican Freedom Caucus was making the news with a potential shutdown of the government that was controversial even amongst some Republicans. [Read more…] about Statistical Analysis of the Republican Establishment Split

Filed Under: Regression Tagged With: analysis example, interpreting results

  • « Go to Previous Page
  • Go to page 1
  • Go to page 2

Primary Sidebar

Meet Jim

I’ll help you intuitively understand statistics by focusing on concepts and using plain English so you can concentrate on understanding your results.

Read More…

Buy My Introduction to Statistics eBook!

New! Buy My Hypothesis Testing eBook!

Buy My Regression eBook!

Subscribe by Email

Enter your email address to receive notifications of new posts by email.

    I won't send you spam. Unsubscribe at any time.

    Follow Me

    • FacebookFacebook
    • RSS FeedRSS Feed
    • TwitterTwitter
    • Popular
    • Latest
    Popular
    • How To Interpret R-squared in Regression Analysis
    • How to Interpret P-values and Coefficients in Regression Analysis
    • Measures of Central Tendency: Mean, Median, and Mode
    • Normal Distribution in Statistics
    • Multicollinearity in Regression Analysis: Problems, Detection, and Solutions
    • How to Interpret the F-test of Overall Significance in Regression Analysis
    • Understanding Interaction Effects in Statistics
    Latest
    • Using Applied Statistics to Expand Human Knowledge
    • Variance Inflation Factors (VIFs)
    • Assessing a COVID-19 Vaccination Experiment and Its Results
    • P-Values, Error Rates, and False Positives
    • How to Perform Regression Analysis using Excel
    • Coefficient of Variation in Statistics
    • Independent and Dependent Samples in Statistics

    Recent Comments

    • Sergio on How To Interpret R-squared in Regression Analysis
    • Laura Kukkonen on Choosing the Correct Type of Regression Analysis
    • Girma Gedamu on Choosing the Correct Type of Regression Analysis
    • Madhu on Assessing Normality: Histograms vs. Normal Probability Plots
    • Madhu on Normal Distribution in Statistics

    Copyright © 2021 · Jim Frost · Privacy Policy