Orthogonality

Orthogonality is a mathematical property that is beneficial for statistical models. It’s particularly helpful when performing factorial analysis of designed experiments.

Orthogonality has various mathematic and geometric definitions. In this post, I’ll define it mathematically and then explain its practical benefits for statistical models.

Terminology

First, here’s a bit of background terminology that you’ll encounter when discussing orthogonality.

In math, a matrix is a two-dimensional rectangular array of numbers with columns and rows. A vector is simply a matrix that has either one row or one column.

For a regression model, the columns in your dataset are the independent and dependent variables. These columns are vectors.

When I refer to a vector in this context, you can think of a datasheet column representing a variable. Orthogonality applies specifically to the independent variables.

Related post: Independent and Dependent Variables

Orthogonal Definition

Vectors are orthogonal when the products of their matching elements sum to zero. That’s a mouthful, but it’s pretty simple illustrating how to find orthogonal vectors.

Follow these steps to calculate the sum of the vectors’ products.

1. Multiply the first values of each vector.
2. Multiply the second values, and repeat for all values in the vectors.
3. Sum those products.

If the sum equals zero, the vectors are orthogonal.

Let’s work through an example. Below are two vectors, V1 and V2. Each vector has five values.

The table below multiplies the values in each vector and sums them.

Because the sum equals zero, these two vectors are orthogonal.

For the discussion about orthogonality in linear models below, consider each vector to be an independent variable.

Orthogonality in Regression and ANOVA models

Orthogonality provides essential benefits to linear models, even though that might not be obvious from the mathematic definition!

When independent variables are orthogonal, they are uncorrelated, which is beneficial. Statisticians refer to the correlation amongst independent variables as multicollinearity. A little bit is okay, but more can cause problems.

The best case is when there is no multicollinearity at all, which is an orthogonal model. Orthogonality indicates that the independent variables are genuinely independent. They are not associated at all—totally uncorrelated.

For orthogonal models, the coefficient estimates for the reduced model will be the same as those in the full model. In other words, you obtain the same estimated effects for the independent variables whether you test them individually or simultaneously. You can add or subtract the orthogonal variables without affecting the coefficients of the other variables. The same is true for including or excluding interaction effects.

Your interpretation is easier, and you’ll feel more confident about your results because the coefficients won’t change as you alter the model.

Alternatively, when the variables are not orthogonal, the coefficients can change when you adjust the variables in the model. The effects depend on the variables in the model to some degree. This condition can leave you feeling less sure about the correct effects!

Related post: Multicollinearity: Problems, Detection, and Solutions

Orthogonal Designs in Factorial Experiments

It might sound unlikely that there would be absolutely no correlation between independent variables. That the sum of the vectors’ products will equal zero exactly. And you’d be correct. There’s usually some correlation, even if just by chance. However, when you use statistical software to design an experiment, it uses an algorithm to create an orthogonal factorial design to meet your needs.

Factorial designs set up a variety of contrasts to see how they affect the outcome. For example, in a manufacturing process, researchers might include time, temperature, alloy, etc., as factors in an experiment about increasing the strength of their product. Typically, each factor has two levels, and the analysis compares the mean outcomes between them.

Factorial designs are special cases of ANOVA. Again, your statistical software uses an algorithm to set up factorial designs that are orthogonal. It’ll devise combinations of factor level settings for each experimental run that collectively produce orthogonality. This process ensures that each factor’s effect is estimated independently from the other factors.

To learn about factors and ANOVA, read my ANOVA Overview.

In the factorial design below, we have three factors, A, B, and C. Each factor has two settings. In the datasheet, a 1 represents one setting for a factor, while -1 is the other. In a real experiment, the analysts enter the observed outcomes in their own column.

By calculating the sums of the products for the factors, we can see this is an orthogonal design.

A designed experiment is orthogonal when the effects of the factors sum to zero across the other factors, allowing the analysis to estimate each one independently.