Orthogonality is a mathematical property that is beneficial for statistical models. It’s particularly helpful when performing factorial analysis of designed experiments.
Orthogonality has various mathematic and geometric definitions. In this post, I’ll define it mathematically and then explain its practical benefits for statistical models.
Terminology
First, here’s a bit of background terminology that you’ll encounter when discussing orthogonal matrices.
In math, a matrix is a two-dimensional rectangular array of numbers with columns and rows. A vector is simply a matrix that has either one row or one column.
For a regression model, the columns in your dataset are the independent and dependent variables. These columns are vectors.
When I refer to a vector in this context, you can think of a datasheet column representing a variable. Orthogonality applies specifically to the independent variables.
Related post: Independent and Dependent Variables
Orthogonal Definition
Vectors are orthogonal when the products of their matching elements sum to zero. That’s a mouthful, but it’s pretty simple illustrating how to find orthogonal vectors.
Follow these steps to calculate the sum of the vectors’ products.
- Multiply the first values of each vector.
- Multiply the second values, and repeat for all values in the vectors.
- Sum those products.
If the sum equals zero, the vectors are orthogonal.
Let’s work through an example. Below are two vectors, V1 and V2. Each vector has five values.
The table below multiplies the values in each vector and sums them.
Because the sum equals zero, the vectors are orthogonal.
For the discussion about orthogonality in linear models below, consider each vector to be an independent variable.
Orthogonality in Regression and ANOVA models
Orthogonality provides essential benefits to linear models, even though that might not be obvious from the mathematic definition!
When independent variables are orthogonal, they are uncorrelated, which is beneficial. Statisticians refer to the correlation amongst independent variables as multicollinearity. A little bit is okay, but more can cause problems.
The best case is when there is no multicollinearity at all, which is an orthogonal model. Orthogonality indicates that the independent variables are genuinely independent. They are not associated at all—totally uncorrelated.
For orthogonal models, the coefficient estimates for the reduced model will be the same as those in the full model. In other words, you obtain the same estimated effects for the independent variables whether you test them individually or simultaneously. You can add or subtract the orthogonal variables without affecting the coefficients of the other variables. The same is true for including or excluding interaction effects.
Your interpretation is easier, and you’ll feel more confident about your results because the coefficients won’t change as you alter the model.
Alternatively, when the variables are not orthogonal, the coefficients can change when you adjust the variables in the model. The effects depend on the variables in the model to some degree. This condition can leave you feeling less sure about the correct effects!
Related post: Multicollinearity: Problems, Detection, and Solutions
Orthogonal Designs in Factorial Experiments
It might sound unlikely that there would be absolutely no correlation between independent variables. That the sum of the vectors’ products will equal zero exactly. And you’d be correct. There’s usually some correlation, even if just by chance. However, when you use statistical software to design an experiment, it uses an algorithm to create an orthogonal factorial design to meet your needs.
Factorial designs set up a variety of contrasts to see how they affect the outcome. For example, in a manufacturing process, researchers might include time, temperature, alloy, etc., as factors in an experiment about increasing the strength of their product. Typically, each factor has two levels, and the analysis compares the mean outcomes between them.
Factorial designs are special cases of ANOVA. Again, your statistical software uses an algorithm to set up factorial designs that are orthogonal. It’ll devise combinations of factor level settings for each experimental run that collectively produce orthogonality. This process ensures that each factor’s effect is estimated independently from the other factors.
To learn about factors and ANOVA, read my ANOVA Overview.
In the factorial design below, we have three factors, A, B, and C. Each factor has two settings. In the datasheet, a 1 represents one setting for a factor, while -1 is the other. In a real experiment, the analysts enter the observed outcomes in their own column.
By calculating the sums of the products for the factors, we can see this is an orthogonal design.
A designed experiment is orthogonal when the effects of the factors sum to zero across the other factors, allowing the analysis to estimate each one independently.
Hello Jim, I hope all is right.
Can you kindly throw more light on how I can intuitively interpret polynomial orthogonal contrasts.
Also the question of when to use polynomial orthogonal contrasts instead of post-hoc tests.
Further, doesn’t the family-wise error affect comparisons made with orthogonal contrasts.
I will be glad to get your views.
Hello Jim,
How does the design and analysis of a 2×2 factorial experiment differ from that of a 2-way ANOVA.
Hi Collin,
A 2X2 factorial design and 2-way ANOVA are very similar. In fact, I’d say that 2X2 factorial designs are a subtype of the more general 2-way ANOVA. Both analyses have two independent variables and assess the mean effects and interactions on a continuous dependent variable. However, the two categorical IVs in 2-way ANOVA can have any number of levels whereas a 2X2 factorial design has 2 categorical IVs, and each one must have exactly two levels.
Additionally, a 2X2 factorial design tends to occur in an experimental environment where the run order is randomized and all possible combinations are represented equally. Those conditions may or may not occur in a 2-way ANOVA.
Finally, the general purpose behind 2-level factorial designs and 2-way ANOVA are somewhat different. In two-way ANOVA, the number of levels for each categorical variable is driven by the research needs and subject area knowledge. That’s also true for 2-level factorial designs. However, with these designs the goal is to provide useful insights with a modest number of runs per factor while not necessarily covering the entire factor space. Think of these 2-level factorial designs as your initial guides. They can help highlight the major trends and give you a sense of direction for any further experimenting you might want to do.
So, consider 2X2 factorial designs to be subtype of 2-way ANOVA. Additionally, 2 level factorial designs is a subtype of ANOVA in a similar way.
I hope that helps!
Great, explanation! is there a threshold for how much different from zero the products are allowed in order to don’t affect the analysis?
Hi Luã,
While perfect orthogonality is the best, there are thresholds for when that’s not possible. However, they’re discussed in terms of multicollinearity, which is the correlation amongst predictors. For more details about the thresholds and how to assess that, read my post, Multicollinearity: Problems, Detection, and Solutions.
Thanks Jim for always finding a simple way and language to explain complex statistical concepts
Nice read! Thank you, Jim.