Residual Sum of Squares (RSS) Explained

The residual sum of squares (RSS) measures the difference between your observed data and the model’s predictions. It is the portion of variability your regression model does not explain, also known as the model’s error. Use RSS to evaluate how well your model fits the data.

In least squares regression, the concept of the sum of squares (SS) is pivotal in quantifying several types of variability relating to a regression model. This statistical method assesses how effectively a dataset aligns with a specific function. The various kinds of SS are essential mathematical tools in identifying the function that most accurately mirrors the data.

Within this framework, the residual sum of squares is one of several types of SS. After fitting the model, it evaluates explicitly the amount of error between the dataset and the regression function. Learn about the other types of sums of squares.

Mathematically, the residual sum of squares formula evaluates the distance between the observations and their corresponding fitted values. Specifically, it’s an overall measure of the size of the residuals. Ordinary least squares (OLS) regression minimizes RSS, producing the best possible fit for any given model.

Let’s learn how to use the residual sum of squares formula to calculate RSS, how to interpret it, and about some of its limitations.

Residual Sum of Squares Formula

The residual sum of squares formula is the following:

Finding the residual sum of squares involves summing the squared distances between each data point (y_i) and its fitted value (ŷ_i).

To calculate the RSS, do the following:

1. Take the y value for each observation and subtract the model’s predicted value for it (ŷ). That finds the residual (y_i — ŷ_i) for each data point.
2. Square all the residuals. This process guarantees a positive value for all data points.
3. Sum the squared residuals.

As you can see, the procedure is literally the sum of the squared residuals. Alternatively, statisticians refer to this measure as the error sum of squares (SSE).

Note: Some notation uses RSS for regression SS instead of residual SS. Be aware of this potentially confusing acronym switch!

Interpreting RSS

It’s challenging to interpret a single residual sum of square value in isolation because it uses squared units and doesn’t adjust for sample size. However, you can compare RSS values between competing models for the same dataset. Typically, you want RSS to be as small as possible because it suggests that your model’s predictions are close to the actual observations. Smaller values reflect less model error and indicate that the regression function predicts values relatively close to the observed values. Your model fits the data well.

Zero is the smallest possible value, indicating a perfect fit with no error. The predicted values match the observed values exactly. However, you won’t see this condition with real-world data.

R-squared is the standard goodness-of-fit measure for linear models. Its formula incorporates RSS. Typically, you’ll evaluate R² rather than the RSS because it avoids some of RSS’s limitations, making it much easier to interpret. See the limitations section next for more about that.

Learn more about How to Interpret R-squared.

Limitations of the Residual Sum of Squares

While valuable in regression analysis, the residual sum of squares has constraints.

Outliers Can Bias the Model

Least squares regression minimizes the RSS, which sounds great. However, the squaring process in the residual sum of squares formula gives outliers markedly more influence during model fitting than regular data points, potentially biasing the model.

Learn How to Find the Least Squares Line.

Unstandardized

The residual sum of squares formula does not standardize the results, complicating interpretation. It does not adjust for the data’s scale nor accounts for the sample size. If you have a model that fits two different datasets equally well, scale and sample size differences can produce two vastly different RSS values.

Consequently, you cannot determine whether a single RSS value in isolation is relatively low or not. While you can compare RSS values between models within the same dataset, the lack of standardization prevents you from comparing values between datasets.

Does Not Assess Other Assumptions

A low residual sum of squares does not guarantee that the model satisfies various least squares assumptions, such as linearity, error independence, and homoscedasticity. These problems can produce results you can’t trust. A low RSS doesn’t indicate you can trust the results. Check the residual plots to assess the regression assumptions!

In summary, RSS measures the amount of error in a model. In fact, least squares regression minimizes RSS to produce the best model fit. However, its susceptibility to outliers, interpretation difficulties, and inability to address model assumptions are critical limitations.