Observational Studies Explained

Observational studies use samples to draw conclusions about a population when the researchers do not control the treatment, or independent variable, that relates to the primary research question.

In my previous post, I show how random assignment reduces systematic differences between experimental groups at the beginning of the study, which increases your confidence that the treatments caused any differences between groups you observe at the end of the study.

Unfortunately, using random assignment is not always possible. For these cases, you can conduct an observational study. In this post, learn about observational studies, why these studies must account for confounding variables, and how to do so. I’ll close this post by reviewing a published observational study about vitamin supplement usage.

Definition of Observational Studies

In an observational study, the researchers only observe the subjects and do not interfere or try to influence the outcomes. In other words, the researchers do not control the treatments or assign subjects to experimental groups. Instead, they observe and measure variables of interest and look for relationships between them. Usually, researchers conduct observational studies when it is difficult, impossible, or unethical to assign study participants to the experimental groups randomly. If you can’t randomly assign subjects to the treatment and control groups, then you observe the subjects in their self-selected states.

Randomized studies are better, and you should always randomize whenever possible. However, if randomization is not possible, science should not come to a halt. After all, we still want to learn things, discover relationships, and make discoveries. For these cases, observational studies are a good alternative.

Related post: Control Groups in Experiments

Examples of When to Use Observational Studies

Let’s start by looking at cases where random assignment is problematic. Despite that constraint, we can still conduct observational studies and draw conclusions about effects.

If you’re studying how depression affects performance of an activity, it’s impossible to assign subjects to the depression and control group randomly. However, you can have subjects with and without depression perform the activity and compare the results.

Or, imagine trying to assign subjects to cigarette smoking and non-smoking groups randomly?! However, you can observe people in both groups and assess the differences in health outcomes.

Suppose you’re studying a treatment for a disease. Ideally, you recruit a group of patients who all have the disease, and then randomly assign them to the treatment and control group. However, it’s unethical to withhold the treatment, which rules out a control group. Instead, you can compare patients who voluntarily do not use the medicine to those who do use it.

Later in this post, we’ll examine a study about vitamin supplement consumption and how that affects the risk of death. It’s possible to use random assignment to place each subject in either the vitamin treatment group or the control group. However, the study assesses vitamin consumption in 40,000 participants over the course of two decades. It’s unrealistic to enforce the treatment and control protocols over such a long time for so many people!

In all these examples, the researchers do not assign subjects to the experimental groups. Instead, they observe people who are already in these groups and compare the outcomes.

Drawbacks of Observational Studies

While observational studies get around the inability to assign subjects randomly, this approach opens the door to the problem of confounding variables. A confounding variable, or confounder, correlates with both the experimental groups and the outcome variable. Because there is no random process that equalizes the experimental groups in an observational study, confounding variables can systematically differ between groups when the study begins. Consequently, confounders can be the actual cause for differences in outcome at the end of the study rather than the primary variable of interest. If an experiment does not account for confounding variables, confounders can bias the results and make them untrustworthy.

Let’s see how this works. Imagine an observational study that compares people who take vitamin supplements to those who do not. People who use vitamin supplements voluntarily will tend to have other healthy habits that exist at the beginning of the study. These healthy habits are confounding variables. If there are differences in health outcomes at the end of the study, it’s possible that these healthy habits actually caused them rather than the vitamin consumption itself. In short, confounders confuse the results because they provide alternative explanations for the differences.

Despite the limitations, an observational study can be a valid approach. However, you must ensure that your research accounts for confounding variables. Fortunately, there are several methods for doing just that!

Accounting for Confounding Variables in Observational Studies

In a randomized study, randomization tends to equalize confounders between groups and, thereby, prevents problems. In my post about random assignment, I describe that process as an elegant solution for confounding variables. You don’t need to measure or even know which variables are confounders, and randomization will still mitigate their effects. Additionally, you can use control variables to keep the experimental conditions as consistent as possible.

On the other hand, observational studies don’t use random assignment and confounders can be distributed disproportionately. Consequently, experimenters need to know which variables are confounders, measure them, and then use a method to account for them. It involves more work and the additional measurements can increase the costs. And there’s always a chance that researchers will fail to identify a confounder, not account for it, and produce biased results. However, if randomization isn’t an option, then you probably need to consider an observational study.

Trait matching and statistically controlling confounders using multivariate procedures are two standard approaches for incorporating confounding variables.

Related post: Causation versus Correlation in Statistics

Matching in Observational Studies

Matching is a technique that involves selecting study participants with similar characteristics outside the variable of interest or treatment. Rather than using random assignment to equalize the experimental groups, the experimenters do it by matching observable characteristics. For every participant in the treatment group, the researchers find a participant with comparable traits to include in the control group. Matching subjects facilitates valid comparisons between those groups. The researchers use subject-area knowledge to identify characteristics that are critical to match.

For example, a vitamin supplement study using matching will select subjects who have similar health-related habits and attributes. The goal is that vitamin consumption will be the primary difference between the groups, which helps you attribute differences in health outcomes to vitamin consumption. However, the researchers are still observing participants who decide whether they consume supplements.

Matching has some drawbacks. The experimenters might not be aware of all the relevant characteristics they need to match. In other words, the groups might be different in an essential aspect that the researchers don’t recognize. For example, in the hypothetical vitamin study, there might be a healthy habit or attribute that affects the outcome that the researchers don’t measure and match. These unmatched characteristics might cause the observed differences in outcomes rather than vitamin consumption.

Using Multiple Regression in Observational Studies

Random assignment and matching use different methods to equalize the experimental groups. However, statistical techniques, such as multiple regression analysis, don’t try to equalize the groups but instead use a model that accounts for confounding variables. These studies statistically control for confounding variables.

In multiple regression analysis, including a variable in the model holds it constant while you vary the variable/treatment of interest. For information about this property, read my post When Should I Use Regression Analysis?

As with matching, the challenge is to identify, measure, and include all confounders in the regression model. Failure to include a confounding variable in a regression model can bias your results.

Next, we’ll look at a published study that uses multiple regression to account for confounding variables.

Related post: Independent and Dependent Variables in a Regression Model

Assessing a Vitamin Supplement Observational Study

Murso et al. (2011)* use a longitudinal observational study that ran 22 years to assess differences in death rates for subjects who used vitamin supplements regularly compared to those who did not use them. This study used surveys to record the characteristics of approximately 40,000 participants. The surveys asked questions about potential confounding variables such as demographic information, food intake, health details, physical activity, and, of course, supplement intake.

Because this is an observational study, the subjects decided for themselves whether they were taking vitamin supplements. Consequently, it’s safe to assume that supplement users and non-users might be different in other ways. From their article, the researchers found the following pre-existing differences between the two groups:

Supplement users had a lower prevalence of diabetes mellitus, high blood pressure, and smoking status; a lower BMI and waist to hip ratio, and were less likely to live on a farm. Supplement users had a higher educational level, were more physically active and were more likely to use estrogen replacement therapy. Also, supplement users were more likely to have a lower intake of energy, total fat, and monounsaturated fatty acids, saturated fatty acids and to have a higher intake of protein, carbohydrates, polyunsaturated fatty acids, alcohol, whole grain products, fruits, and vegetables.

Whew! That’s a long list of differences! Supplement users were different from non-users in a multitude of ways that are likely to affect their risk of dying. The researchers must account for these confounding variables when they compare supplement users to non-users. If they do not, their results can be biased.

Using Multiple Regression to Statistically Control for Confounders

To account for these initial differences, the researchers use regression analysis and include the confounding variables in the model.

The researchers present three regression models. The simplest model accounts only for age and caloric intake. Next, are two models that include additional confounding variables beyond age and calories. The first model adds various demographic information and seven health measures. The second model includes everything in the previous model and adds several more specific dietary intake measures. Using statistical significance as a guide for specifying the correct regression model, the researchers present the model with the most variables as the basis for their final results.

It’s instructive to compare the raw results and the final regression results.

Raw results

The raw differences in death risks for consumers of folic acid, vitamin B6, magnesium, zinc, copper, and multivitamins are NOT statistically significant. However, the raw results show a significant reduction in the death risk for users of B complex, C, calcium, D, and E.

However, those are the raw results that do not control for the long list of differences between the groups that exist at the beginning of the study. After using the regression model to control for the confounding variables statistically, the results change dramatically.

Adjusted results

Of the 15 supplements that the study tracked, researchers found consuming seven of these supplements were linked to a statistically significant INCREASE in death risk (p-value < 0.05): multivitamins (increase in death risk 2.4%), vitamin B6 (4.1%), iron (3.9%), folic acid (5.9%), zinc (3.0%), magnesium (3.6%), and copper (18.0%). Only calcium was associated with a statistically significant reduction in death risk of 3.8%.

In short, the raw results suggest that those who consume supplements either have the same or lower death risks than non-consumers. However, these results do not account for the multitude of healthier habits and attributes in the group that uses supplements.

In fact, these confounders seem to produce most of the apparent benefits in the raw results because, after you statistically control the effects of these confounding variables, the results worsen for those who consume vitamin supplements. The adjusted results indicate that most vitamin supplements actually increase your death risk!

This study illustrates how confounders biased the raw results to make the vitamin consumption outcomes look better than they really are.

In conclusion, if you can’t randomly assign subjects to the experimental groups, an observational study might be right for you. However, be aware that you’ll need to identify, measure, and account for confounding variables in your experimental design.

Reference

Jaakko Mursu, PhD; Kim Robien, PhD; Lisa J. Harnack, DrPH, MPH; Kyong Park, PhD; David R. Jacobs Jr, PhD; Dietary Supplements and Mortality Rate in Older Women: The Iowa Women’s Health Study; Arch Intern Med. 2011;171(18):1625-1633.