5 Steps for Conducting Scientific Studies with Statistical Analyses

The scientific method is a proven procedure for expanding knowledge through experimentation and analysis. It is a process that uses careful planning, rigorous methodology, and thorough assessment. Statistical analysis plays an essential role in this process.

In an experiment that includes statistical analysis, the analysis is at the end of a long series of events. To obtain valid results, it’s crucial that you carefully plan and conduct a scientific study for all steps up to and including the analysis. In this blog post, I map out five steps for scientific studies that include statistical analyses.

Mapping out the Process for Scientific Studies

It’s vital that you understand the scientific method and know how to design a scientifically rigorous study that includes statistical analysis. Mistakes along the way can invalidate the results of your analysis. I’ve divided the process into five stages. Depending on the nature of your experiment, you might need to emphasize or deemphasize certain aspects.

For example, studies of physical phenomena will look very different than those in the social sciences. In the same vein, studies that use designed factorial experiments, observational studies, and surveys will all look different from each other. While studies can differ drastically, they all use aspects of the same roadmap I lay out.

This roadmap relates to scientific studies that include statistical analysis because my blog is all about statistical analysis. However, even purely qualitative studies will share many of the same steps. Learn about qualitative research.

The steps in a scientifically rigorous study are the following:

1. Research Phase.
   • Define the Problem and Research Question.
   • Literature Review.
2. Operationalize Phase.
   • Define your variables and measurement techniques.
   • Design the experimental methods.
3. Data Collection.
4. Statistically analyze data and draw conclusions.
5. Communicate the results.

Related post: The Importance of Statistics

Step 1: Research Your Study Area

Good scientific research depends on gathering a lot of information before you even start collecting data. You’ll need to investigate your subject-area to write a research question that your study can reasonably answer. Then, you’ll need to develop in-depth knowledge about other studies to devise a plan for conducting your study.

Define Your Research Question

The first step of your study is to formulate a research question. This is the question you want your study to answer. Research questions focus your experiment, help guide your decision-making process, and helps prevent side issues from distracting you from your goal.

Typically, researchers start with a broad topic and research the subject area. Determine what types of questions researchers have and have not answered. This process helps to narrow the broad topic down to a more specific research question. Determine what studies researchers have already performed and what literature already exists. Will you answer a new question or try to replicate previous research?

Your research question should be appropriate for your discipline. Consequently, the properties of suitable research questions vary significantly by subject area. For example, acceptable research questions look different for physics, psychology, biology, and political science. However, they have some common qualities.

Research questions must be clear and concise. Therefore, readers of your short research question should clearly understand the goal of your study. Additionally, ensure the scope of the inquiry is narrow enough that your research can reasonably answer it using available time and resources.

Typically, developing your research question often starts with a topic that you are interested in and involves some initial research. This preliminary research helps you craft an actionable research question. However, after you devise your question, you’ll need to conduct a much more in-depth review of the literature. And you will likely perform some iterative fine-tuning. During the literature review, you might find yourself tweaking the research question.

Literature Review

A literature review is a very extensive background investigation into your research question. There are two primary goals of a literature review for a scientific study that involves statistical analysis.

First, you need to understand fully the subject-area that contains your research question. What have other studies found? Identify the significant relationships and effects that the literature recognizes along with their size and direction. What variables and factors play a role?

In short, define the current state of scientific knowledge surrounding your research question. This process helps you determine how your study fits within the field, enables you to understand the thought processes behind similar studies, and provides you with a general sense of the findings thus far.

Secondly, you need information that helps you operationalize your study. Operationalization is the process of taking the general idea of your research question and creating an actionable plan that allows an experiment to answer the question. If your study includes statistical analysis, you’ll need to determine how other studies have used statistics to answered similar questions.

With that in mind, determine the following:

• What data did similar studies collect? Which variables?
• How did they measure the variables?
• How did they draw their sample?
• What methods did they use to analyze the data? Which analyses and experimental designs?

You’ll also want to learn about the strengths, weaknesses, and mistakes that other studies have made. Avoid the mistakes of others and build on their strengths!

The research phase should produce a research question, in-depth knowledge of the subject-area and relevant findings, and a thorough understanding of how other researchers have operationalized similar studies. This background information helps you design your own experiment.

Step 2: Operationalize Your Study Plan

Operationalizing a study is the process of taking your research question, using the background information you gathered, and a formulating an actionable plan. This plan includes everything from defining variables to how you’ll analyze the data.

Variables: What Will You Measure?

Studies that use statistics to answer questions require you to collect data in the form of variables that you’ll analyze. Consequently, you must define the variables that you will measure and decide how you’ll measure them. If you do not collect the correct data or measure it inaccurately, you might not be able to answer your research question. In fact, thanks to omitted variable bias, the variables you do not measure can impact the results for the variables that you do measure! If you can’t measure an important variable directly, consider measuring a proxy variable to use in its place. Take your time determining which variables you’ll need to measure to answer your research question.

For example, if you are studying depression, how will you define and measure depression? Your literature review should inform your decision about using an accepted definition for depression and choosing a scientifically validated methodology for assessing depression. Science builds on itself!

If you’re trying to predict depression, describe its relationships with other variables, or evaluate treatments, you’ll need to define those variables operationally and determine how you’ll measure them.

Types of Variables and Treatments

A study will have a dependent variable. This variable is the outcome you are studying. Typically, studies want to understand how changes in one or more independent variables affect the dependent variable. Depending on the type of experiment, the researchers will either control or not control the independent variables. If you control the variables, you’ll need to decide on the settings for the controllable variables.

Most studies include a treatment, intervention, or some other comparison it wants to make. You’ll need to define the treatment and ensure a system is in place to deliver it as required. That’s true not only for medical treatments but with any intervention.

For example, I participated in an exercise intervention study to determine whether it affects bone density. We defined our intervention as sessions that occur three times a week and consist of 30 impacts that are six times the subjects’ body weights. We had the procedures, equipment, and training in place to ensure our subjects received the intervention as we defined it.

Measurement Methodology: How Will You Take Measurements?

You’ll also need to specify how you will take measurements. What equipment will you use? How will you reduce other sources of variation?

Precision and accuracy are essential in research. Ensure that your plan describes how to obtain good measurements. For example, I once wrote a detailed equipment calibration document to ensure high quality measurements over the course of the study. For that study, good measurements depended on daily, standardized calibrations.

Be sure that your measurement instruments and test scores are valid. To learn more, read my post about Validity.

Create a Sampling Plan: How Will You Collect Samples for Studying?

Researchers must specify the particular population they’re studying. For example, will you include all levels of depression or only mild to severe cases?

After you define your population, you need to devise a plan for collecting a sample from that population. Your sample contains the people or objects that your study assesses. Studies that use inferential statistics take sample data and draw inferences about a population. However, these studies must gather samples in a manner that produces unbiased estimates. This process often involves a random sampling methodology because a method based on convenience can introduce bias.

Literature reviews will often reveal sample collection methodologies other researchers have used in your study area. Determine where and how you’ll collect the sample, including the date and time, location, and so on.

Finally, how much data should you collect? On the one hand, you want to collect enough data to have a reasonable chance of detecting a practically significant effect. On the other hand, you don’t want to obtain such a large sample that it wastes your time and resources. A power analysis helps you choose a sample size that strikes a balance between these two competing goals. However, to perform a power analysis, you need estimates for effect size and variability in the data. Again, look at your literature review!

Related post: Population, Parameters, and Samples

Design the Experimental Methods

You’ll need to define your hypothesis in a form amenable to statistical analysis and choose the appropriate analysis. Your hypothesis must be testable, which means that the data you collect will either support or reject the hypothesis. Determine the statistical analyses that can adequately test your hypotheses. These methodology decisions start at a very high level, such as choosing between a randomized experiment or an observational study. From there, you can work your way down to more fundamental questions.

For example, will you compare means, medians, proportions, or rates between groups? Or perhaps assess the relationship between nominal variables or continuous variables? All these issues affect the statistical analyses you can perform.

Additionally, there are the nuts and bolts for each type of analysis that you’ll need to decide. What significance level will you use? One-tailed or two-tailed hypothesis tests? If you use ANOVA, will you follow up with a post hoc test? If so, which one? What steps will you take to help determine that you’re observing causation rather than just correlation? Will you conduct a randomized experiment or an observational study?

Your plan should limit the number of analyses and models you’ll use. Each statistical test has an error rate. The more tests you perform, the higher the overall chances of a false result. Making these methodology decisions in advance helps you avoid using multiple techniques and cherry picking the best results and reduces data mining, which lowers the probability of finding chance correlations.

The operationalization stage should produce a plan that tells you what you’ll measure, how you’ll measure it, how you will collect a sample, your experimental design, the size of the sample, and how you’ll analyze the data.

Overall, you need to be sure that your experimental designs and methods are valid. To learn more, read my post about internal and external validity.

Step 3: Data Collection

At this point, you’ve operationalized your study and have a plan of action. After you make the necessary arrangements, you should be ready to collect data! Depending on the nature of your research, this can be quite a long process. Whether you’re in the lab measuring, out administering surveys in the field, or working with human subjects, data collection is often the portion of the study that takes the most time and work.

Often, you’ll need to set up the proper conditions to make measurements and verify that everything is working correctly. Perhaps you need to get the lab conditions just right and ensure the equipment is functioning properly to obtain valid measurements. Or, you’re going through a detailed process to obtain a truly random sample. Sometimes it is difficult to recruit a sufficient number of human subjects. The procedures might also involve training other personnel to perform tasks precisely as prescribed. I once had to create a training video to obtain consistent results!

While you’re generally working from your operational plan, it’s not uncommon to encounter surprises, and you’ll need to adapt. Hopefully, your subject-area knowledge and literature review help you anticipate most surprises, but the thing about science is that you’re often studying something that researchers haven’t fully studied before. Expect surprises!

Before collecting data, assess the accuracy and precision of your measurement system!

Step 4: Statistical Analysis

Like the data collection stage of your study, you should already have the analysis phase defined. If you’re “winging it,” you’re not doing it right! My entire blog is about statistical analysis, so I’m not going to restate all of it here. In a nutshell, be sure that you’re analyzing the data correctly, satisfying the assumptions where necessary, and drawing the proper conclusions.

However, there is a vital point to make here. Problems along the way can prevent you from making discoveries or invalidate the findings well before you even get to the statistical analysis. As the old saying goes, garbage in, garbage out. If you put garbage data into the statistical analysis, it’ll spit out garbage results. If all the steps leading up to your analysis are not carefully thought out and performed, you might not be able to trust the results or miss important findings. Science is all about getting all the details correct.

Step 5: Writing the Results

After you collect the data and analyze it, you need to write up the results to inform other researchers about what you’ve found. Indicate which hypotheses the data support, the overall conclusions, and what they represent in the framework of the scientific field or real-world setting. However, it involves more than just writing up the findings.

The scientific method works by replicating results—or the failure to do so. The scientific process tends to cause the correct answers for research questions to rise the top over time through successful replication. Conversely, it weeds out incorrect results after they fail to replicate.

Consequently, you’ll need to provide enough information about how you conducted your study so other researchers can repeat it and, hopefully, replicate the results. Typically, you’ll include aspects of the first four steps (background research, operationalization, data collection, and analysis) in the final write up. The standards vary by field, so you should see how studies in your area document themselves. In this manner, your research becomes part of the knowledgebase for future studies to build on—just like you did during your literature review! Additionally, all the details help other researchers determine the strengths and weaknesses of your study so they can interpret the results while understanding the context.

In closing, statistical analysis is a crucial step in the scientific process. The analysis objectively tells you which hypothesis the data favor. However, there is a long list of items before the statistical analysis that must all proceed correctly for you to be able to trust the results.

To learn about some of the challenges I faced early in my scientific research career, read my post about using applied statistics to expand human knowledge!