Permutation Test

A permutation test is a nonparametric method used for grouped designs to test for differences between groups. It can serve as an alternative to traditional methods like the independent samples t-test, paired t-test, ANOVA, or repeated measures ANOVA, without relying on assumptions such as normality or equal variances. The basic idea is to repeatedly shuffle the groupings in the data and recalculate the test statistic each time to see how often random chance could produce results as extreme as the ones observed.

In other words, it takes the measured data values and randomly assigns them to different groups for each permutation, while keeping the same number of values in each group as the original design. This approach produces a custom distribution that represents what would happen under the null hypothesis, given the group structure of the experiment. Researchers then compare the actual result to this distribution to estimate a p-value.

For example, to test whether a new teaching method improves test scores, researchers could randomly reassign which students are considered part of the “new method” group and which are part of the “old method” group, many times. After each shuffle, they would calculate the difference in average scores. The proportion of shuffled differences that are as large (or larger) than the original observed difference would give the p-value.

Permutation tests are similar to bootstrapping in that both involve resampling the data to create a custom distribution. However, permutation tests shuffle group assignments to test a hypothesis about group differences, while bootstrapping resamples observations with replacement to estimate statistics like means, medians, or confidence intervals without necessarily testing a specific hypothesis.