What is an Observational Study?
An observational study uses sample data to find correlations in situations where the researchers do not control the treatment, or independent variable, that relates to the primary research question. The definition of an observational study hinges on the notion that the researchers only observe subjects and do not assign them to the control and treatment groups. That’s the key difference between an observational study vs experiment. These studies are also known as quasi-experiments.
True experiments assign subject to the experimental groups where the researchers can manipulate the conditions. Unfortunately, random assignment is not always possible. For these cases, you can conduct an observational study.
In this post, learn about the types of observational studies, why they are susceptible to confounding variables, and how they compare to experiments. I’ll close this post by reviewing a published observational study about vitamin supplement usage.
Observational Study Definition
In an observational study, the researchers only observe the subjects and do not interfere or try to influence the outcomes. In other words, the researchers do not control the treatments or assign subjects to experimental groups. Instead, they observe and measure variables of interest and look for relationships between them. Usually, researchers conduct observational studies when it is difficult, impossible, or unethical to assign study participants to the experimental groups randomly. If you can’t randomly assign subjects to the treatment and control groups, then you observe the subjects in their self-selected states.
Observational Study vs Experiment
Randomized experiments provide better results than observational studies. Consequently, you should always use a randomized experiment whenever possible. However, if randomization is not possible, science should not come to a halt. After all, we still want to learn things, discover relationships, and make discoveries. For these cases, observational studies are a good alternative to a true experiment. Let’s compare the differences between an observational study vs. an experiment.
Random assignment in an experiment reduces systematic differences between experimental groups at the beginning of the study, which increases your confidence that the treatments caused any differences between groups you observe at the end of the study. In contrast, an observational study uses self-formed groups that can have pre-existing differences, which introduces the problem of confounding variables. More on that later!
In a randomized experiment, randomization tends to equalize confounders between groups and, thereby, prevents problems. In my post about random assignment, I describe that process as an elegant solution for confounding variables. You don’t need to measure or even know which variables are confounders, and randomization will still mitigate their effects. Additionally, you can use control variables in an experiment to keep the conditions as consistent as possible.
Observational Study vs Experiment
|Does not assign subjects to groups||Randomly assigns subjects to control and treatment groups|
|Does not control variables that can affect outcome||Administers treatments and controls influence of other variables|
|Correlational findings. Differences might be due to confounders rather than the treatment||More confident that treatments cause the differences in outcomes|
Related posts: Experimental Design: Definition and Examples and Control Groups in Experiments
Observational Study Examples
Consider using an observational study when random assignment for an experiment is problematic. This approach allows us to proceed and draw conclusions about effects even though we can’t control the independent variables. The following observational study examples will help you understand when and why to use them.
For example, if you’re studying how depression affects performance of an activity, it’s impossible to assign subjects to the depression and control group randomly. However, you can have subjects with and without depression perform the activity and compare the results in an observational study.
Or imagine trying to assign subjects to cigarette smoking and non-smoking groups randomly?! However, you can observe people in both groups and assess the differences in health outcomes in an observational study.
Suppose you’re studying a treatment for a disease. Ideally, you recruit a group of patients who all have the disease, and then randomly assign them to the treatment and control group. However, it’s unethical to withhold the treatment, which rules out a control group. Instead, you can compare patients who voluntarily do not use the medicine to those who do use it.
In all these observational study examples, the researchers do not assign subjects to the experimental groups. Instead, they observe people who are already in these groups and compare the outcomes. Hence, the scientists must use an observational study vs. an experiment.
Types of Observational Studies
The observational study definition states that researchers only observe the outcomes and do not manipulate or control factors. Despite this limitation, there various types of observational studies.
Cohort studies and case-control studies are two types of observational studies that medical researchers and epidemiologists frequently use to identify risk factors for a disease or medical condition.
- Cohort Study: A longitudinal observational study that follows a group who share a defining characteristic. These studies frequently determine whether exposure to risk factor affects an outcome over time.
- Case-Control Study: A retrospective observational study that compares two existing groups—the case group with the condition and the control group without it. Researchers compare the groups looking for potential risk factors for the condition.
Qualitative research studies are usually observational in nature, but they collect non-numeric data and do not perform statistical analyses.
Retrospective studies must be observational.
Later in this post, we’ll closely examine a quantitative observational study example that assesses vitamin supplement consumption and how that affects the risk of death. It’s possible to use random assignment to place each subject in either the vitamin treatment group or the control group. However, the study assesses vitamin consumption in 40,000 participants over the course of two decades. It’s unrealistic to enforce the treatment and control protocols over such a long time for so many people!
Drawbacks of Observational Studies
While observational studies get around the inability to assign subjects randomly, this approach opens the door to the problem of confounding variables. A confounding variable, or confounder, correlates with both the experimental groups and the outcome variable. Because there is no random process that equalizes the experimental groups in an observational study, confounding variables can systematically differ between groups when the study begins. Consequently, confounders can be the actual cause for differences in outcome at the end of the study rather than the primary variable of interest. If an experiment does not account for confounding variables, confounders can bias the results and create spurious correlations.
Performing an observational study can decrease the internal validity of your study but increase the external validity. Learn more about internal and external validity.
Let’s see how this works. Imagine an observational study that compares people who take vitamin supplements to those who do not. People who use vitamin supplements voluntarily will tend to have other healthy habits that exist at the beginning of the study. These healthy habits are confounding variables. If there are differences in health outcomes at the end of the study, it’s possible that these healthy habits actually caused them rather than the vitamin consumption itself. In short, confounders confuse the results because they provide alternative explanations for the differences.
Despite the limitations, an observational study can be a valid approach. However, you must ensure that your research accounts for confounding variables. Fortunately, there are several methods for doing just that!
Accounting for Confounding Variables in an Observational Study
Because observational studies don’t use random assignment, confounders can be distributed disproportionately between conditions. Consequently, experimenters need to know which variables are confounders, measure them, and then use a method to account for them. It involves more work, and the additional measurements can increase the costs. And there’s always a chance that researchers will fail to identify a confounder, not account for it, and produce biased results. However, if randomization isn’t an option, then you probably need to consider an observational study.
Trait matching and statistically controlling confounders using multivariate procedures are two standard approaches for incorporating confounding variables.
Related post: Causation versus Correlation in Statistics
Matching in Observational Studies
Matching is a technique that involves selecting study participants with similar characteristics outside the variable of interest or treatment. Rather than using random assignment to equalize the experimental groups, the experimenters do it by matching observable characteristics. For every participant in the treatment group, the researchers find a participant with comparable traits to include in the control group. Matching subjects facilitates valid comparisons between those groups. The researchers use subject-area knowledge to identify characteristics that are critical to match.
For example, a vitamin supplement study using matching will select subjects who have similar health-related habits and attributes. The goal is that vitamin consumption will be the primary difference between the groups, which helps you attribute differences in health outcomes to vitamin consumption. However, the researchers are still observing participants who decide whether they consume supplements.
Matching has some drawbacks. The experimenters might not be aware of all the relevant characteristics they need to match. In other words, the groups might be different in an essential aspect that the researchers don’t recognize. For example, in the hypothetical vitamin study, there might be a healthy habit or attribute that affects the outcome that the researchers don’t measure and match. These unmatched characteristics might cause the observed differences in outcomes rather than vitamin consumption.
Learn more about Matched Pairs Design: Uses & Examples.
Using Multiple Regression in Observational Studies
Random assignment and matching use different methods to equalize the experimental groups in an observational study. However, statistical techniques, such as multiple regression analysis, don’t try to equalize the groups but instead use a model that accounts for confounding variables. These studies statistically control for confounding variables.
In multiple regression analysis, including a variable in the model holds it constant while you vary the variable/treatment of interest. For information about this property, read my post When Should I Use Regression Analysis?
As with matching, the challenge is to identify, measure, and include all confounders in the regression model. Failure to include a confounding variable in a regression model can bias your results.
Next, we’ll look at a published observational study that uses multiple regression to account for confounding variables.
Related post: Independent and Dependent Variables in a Regression Model
Vitamin Supplement Observational Study Example
Let’s dig in-depth in an observational study example to see how it works, understand the role of confounders, and how to control for them.
Murso et al. (2011)* use a longitudinal observational study that ran 22 years to assess differences in death rates for subjects who used vitamin supplements regularly compared to those who did not use them. This study used surveys to record the characteristics of approximately 40,000 participants. The surveys asked questions about potential confounding variables such as demographic information, food intake, health details, physical activity, and, of course, supplement intake.
Because this is an observational study, the subjects decided for themselves whether they were taking vitamin supplements. Consequently, it’s safe to assume that supplement users and non-users might be different in other ways. From their article, the researchers found the following pre-existing differences between the two groups:
Supplement users had a lower prevalence of diabetes mellitus, high blood pressure, and smoking status; a lower BMI and waist to hip ratio, and were less likely to live on a farm. Supplement users had a higher educational level, were more physically active and were more likely to use estrogen replacement therapy. Also, supplement users were more likely to have a lower intake of energy, total fat, and monounsaturated fatty acids, saturated fatty acids and to have a higher intake of protein, carbohydrates, polyunsaturated fatty acids, alcohol, whole grain products, fruits, and vegetables.
Whew! That’s a long list of differences! Supplement users were different from non-users in a multitude of ways that are likely to affect their risk of dying. The researchers must account for these confounding variables when they compare supplement users to non-users. If they do not, their results can be biased.
This example illustrates a key difference between an observational study vs experiment. In a randomized experiment, the randomization would have equalized the characteristics of those the researchers assigned to the treatment and control groups. Instead, the study works with self-sorted groups that have numerous pre-existing differences!
Using Multiple Regression to Statistically Control for Confounders
To account for these initial differences in the vitamin supplement observational study, the researchers use regression analysis and include the confounding variables in the model.
The researchers present three regression models. The simplest model accounts only for age and caloric intake. Next, are two models that include additional confounding variables beyond age and calories. The first model adds various demographic information and seven health measures. The second model includes everything in the previous model and adds several more specific dietary intake measures. Using statistical significance as a guide for specifying the correct regression model, the researchers present the model with the most variables as the basis for their final results.
It’s instructive to compare the raw results and the final regression results.
The raw differences in death risks for consumers of folic acid, vitamin B6, magnesium, zinc, copper, and multivitamins are NOT statistically significant. However, the raw results show a significant reduction in the death risk for users of B complex, C, calcium, D, and E.
However, those are the raw results for the observational study, and they do not control for the long list of differences between the groups that exist at the beginning of the study. After using the regression model to control for the confounding variables statistically, the results change dramatically.
Of the 15 supplements that the study tracked in the observational study, researchers found consuming seven of these supplements were linked to a statistically significant INCREASE in death risk (p-value < 0.05): multivitamins (increase in death risk 2.4%), vitamin B6 (4.1%), iron (3.9%), folic acid (5.9%), zinc (3.0%), magnesium (3.6%), and copper (18.0%). Only calcium was associated with a statistically significant reduction in death risk of 3.8%.
In short, the raw results suggest that those who consume supplements either have the same or lower death risks than non-consumers. However, these results do not account for the multitude of healthier habits and attributes in the group that uses supplements.
In fact, these confounders seem to produce most of the apparent benefits in the raw results because, after you statistically control the effects of these confounding variables, the results worsen for those who consume vitamin supplements. The adjusted results indicate that most vitamin supplements actually increase your death risk!
This research illustrates the differences between an observational study vs experiment. Namely how the pre-existing differences between the groups allow confounders to bias the raw results, making the vitamin consumption outcomes look better than they really are.
In conclusion, if you can’t randomly assign subjects to the experimental groups, an observational study might be right for you. However, be aware that you’ll need to identify, measure, and account for confounding variables in your experimental design.
Jaakko Mursu, PhD; Kim Robien, PhD; Lisa J. Harnack, DrPH, MPH; Kyong Park, PhD; David R. Jacobs Jr, PhD; Dietary Supplements and Mortality Rate in Older Women: The Iowa Women’s Health Study; Arch Intern Med. 2011;171(18):1625-1633.
Love your content and has been very helpful!
Can you please advise the question below using an observational data set:
I have three years of observational GPS data collected on athletes (2019/2020/2021). Approximately 14-15 athletes per game and 8 games per year. The GPS software outputs 50+ variables for each athlete in each game, which we have narrowed down to 16 variables of interest from previous research.
1) Period (first half, second half, and whole game),
2) Position (two groups with three subgroups in each – forwards (group 1, group 2, group 3) and backs (group 1, group 2, group 3))
16 variables of interest – all numerical and scale variables. Some of these are correlated, but not all.
My understanding is that I can use a oneway ANOVA for each year on it’s own, using one factor at a time (period or position) with post hoc analysis. This is fine, if data meets assumptions and is normally distributed. This tells me any significant interactions between variables of interest with chosen factor. For example, with position factor, do forwards in group 1 cover more total running distance than forwards in group 2 or backs in group 3.
However, I want to go deeper with my analysis. If I want to see if forwards in group 1 cover more total running distance in period 1 than backs in group 3 in the same period, I need an additional factor and the oneway ANOVA does not suit. Therefore I can use a twoway ANOVA instead of 2 oneway ANOVA’s and that solves the issue, correct?
This is complicated further by looking to compare 2019 to 2020 or 2019 to 2021 to identify changes over time, which would introduce a third independent variable.
I believe this would require a threeway ANOVA for this observational data set. 3 factors – Position, Period, and Year?
Are there any issues or concerns you see at first glance?
I appreciate your time and consideration.
Grace Gibson says
Could an observational study use a correlational design.
e.g. measuring effects of two variables on happiness, if you’re not intervening.
Jim Frost says
Typically, with observational studies, you’d want to include potential confounders, etc. Consequently, I’ve seen regression analysis used more frequently for observational studies to be able to control for other things because you’re not using randomization. You could use correlation to observe the relationship. However, you wouldn’t be controlling for potential confounding variables. Just something to consider.
If I am to administer moderate doses of coffee for a hypothetical experiment, does it raise ethical concerns? Can I use random assignment for it?
Jim Frost says
I don’t see any inherent ethical problems here as long as you describe the participant’s experience in the experiment including the coffee consumption. They key with human subjects is “informed consent.” They’re agreeing to participate based on a full and accurate understanding of what participation involves. Additionally, you as a researcher, understand the process well enough to be able to ensure their safety.
In your study, as long as subject know they’ll be drinking coffee and agree to that, I don’t see a problem. It’s a proven safe substance for the vast majority of people. If potential subjects are aware of the need to consume coffee, they can determine whether they are ok with that before agreeing to participate.
Muhammad Hamza Sabir says
Really great article which explains observational and experimental study very well. It presents broad picture with the case study which helped a lot in understanding the core concepts. Thanks