Anecdotal evidence is a story told by individuals. It comes in many forms that can range from product testimonials to word of mouth. It’s often testimony, or a short account, about the truth or effectiveness of a claim. Typically, anecdotal evidence focuses on individual results, is driven by emotion, and presented by individuals who are not subject area experts.
The following are examples of anecdotal evidence:
- Wow! I took this supplement and lost a lot of weight! This pill must work!
- I know someone who smoked for decades, and it never produced any significant illness. Those claims about smoking are exaggerated!
- This anti-aging cream took years off. It must be the best!
I’m sure you’ve heard that you can’t trust anecdotal evidence. Yet, we still ask our friends for recommendations about restaurants, travel destinations, auto mechanics, and so on. The tricky thing about anecdotal evidence is that even when an individual story is true, it can still be entirely misleading. How does that work?
In this post, I’ll show you why you can’t trust anecdotal evidence!
Statistical Methodology versus Anecdotal Evidence
The table below shows how statistical and scientific methodology are opposites of anecdotal evidence.
| Statistical methodology | Anecdotal evidence |
| Samples are large and representative. Typically, they are generalizable outside the sample. | Small, biased samples are not generalizable. |
| Scientists take precise measurements in controlled environments with calibrated equipment. | Unplanned observations are described orally or in writing. |
| Other relevant factors are measured and controlled. | Pertinent factors are ignored. |
| Strict requirements for identifying causal connections | Anecdotes assume causal relationships as a matter of fact. |
A quick look at the table should be enough to convince you that anecdotal evidence is not trustworthy! However, it’s even worse thanks to psychological factors that prime us for believing these stories.
Humans are more likely to tell and remember dramatic, extraordinary personal stories. Throw in some emotion, and you’re more likely to believe the story. In psychological terms, statistical analysis of data that are carefully collected from well designed experiments lacks that emotional kick. Sad but true.
Furthermore, if B follows A, our brains are wired to assume that A causes B.
Finally, anecdotal evidence cherry picks the best stories. You don’t hear about all of the unsuccessful cases because people are less likely to talk about them.
So, if Fred tells an emotional story about how he took a supplement and then lost a lot of weight, we’ll remember Fred’s story and assume that the supplement caused the weight loss. Unfortunately, we don’t hear from the other 10 people who took the supplement and didn’t lose weight. We also don’t know what else Fred might be doing to lose weight.
Collectively, these factors bias conclusions drawn from anecdotal evidence towards unusual outcomes and unjustified causal connections.
Next, I’ll illustrate the problems graphically and explain how statistics and the scientific method deals with them.
Related post: The Importance of Statistics
Graphically Illustrating the Shortcomings of Anecdotal Evidence
The graph below displays the results from anecdotal stories of people who took a hypothetical weight loss supplement. Think of this chart as a summary of the results presented in a TV commercial. We’ll even assume that these people are telling the complete truth. The supplement looks effective, right? They’ve lost a lot of weight! When you see the individuals and hear their emotional stories about weight loss, we want to believe that the supplement worked.
Regrettably, the graph doesn’t provide the full story. Remember, anecdotal evidence uses small non-random samples that aren’t generalizable beyond the sample. The individuals might have been cherry-picked for their narratives, or perhaps they presented the tales on their own initiative. Either way, it is a sample based on having a dramatic and emotionally compelling story. As the fine print says, their results are not typical!
Unfortunately, our minds are wired to believe this type of evidence. We place more weight on dramatic, personal stories.
A scientific study of the weight loss supplement
Now, let’s imagine that we conduct a scientific study using a more substantial, random sample that represents the broader population. We’ll also include a treatment and control group for comparison. We must go beyond a few compelling stories and get the bigger picture that scientific studies can provide.
In this graph, blue dots represent the supplement takers, and red dots represent those who didn’t take the supplement.
These results are not as impressive as the other graph. Some of those who took the supplement did lose the amount of weight shown in the TV ad, but many more lost much less weight. Those people didn’t come forward with their less exciting stories!
Furthermore, those who did not take the supplement fit the same pattern as those who did. Collectively, taking the supplement didn’t produce greater weight loss than the control group.
Because this study uses a random sample rather than a self-selected sample, we have reason to believe that we can apply these results outside of the sample.
Survivorship Bias is a type of sampling bias that relates strongly to how anecdotal evidence can lead you astray. Learn more about Survivorship Bias: Definition, Examples, and Avoiding.
How Statistics Beats Anecdotal Evidence
In statistics, there are two basic methods for determining whether a dietary supplement causes weight loss: observational studies and randomized controlled trials (RCTs).
In an observational study, scientists measure all pertinent variables in a representative sample, and then generate a statistical model that describes the role of each variable. For each subject, you measure variables such as basal metabolic rate, exercise, diet, health, etc., and the consumption of dietary supplements. After you factor in the role of all other relevant variables, you can determine whether the supplement correlates with weight loss. Anecdotal evidence provides none of this critical, contextual information.
Randomized controlled trials (RCTs) is the other method. RCTs are the gold standard because they allow you to draw causal conclusions about the treatment effect. After all, we want to determine whether the supplement causes the weight loss. RCTs assign subjects to treatment and control groups randomly. This process helps ensure that the groups are comparable when treatment begins. Consequently, treatment effects are the most likely cause for differences between groups at the end of the study.
Making decisions based on anecdotal evidence might not always be harmful. For example, if you ask a friend for a restaurant recommendation, the risk is low, especially if you know his/her tastes. However, if you’re making important decisions about things like finances, healthcare, and fitness, don’t base them anecdotal evidence. Look at scientific data and expert analysis even though they’re not as flashy as emotionally charged stories presented by relatable people!
If you find yourself being won over by anecdotal evidence, remind yourself that the results are not typical!
Has anecdotal evidence ever influenced you?
Hello! Thanks for the great article! Was looking to cite this for my research paper, may I know when did u publish this!
I agree with Adam(⬇️) 100%.
This should be required learning in all schools
I would like to know how this whole post is not simply a logical fallacy as you “cite” the human tendency to trust A over B.
Perhaps I’m just missing where you point to this psychological phenomenon, but otherwise I’m just taking your word for this “situational idiom.”
Where is your proof that this is an occurrence at all?
Is there a DSM or equivalent term for this event? An article?
Whilst I might find something on my own, I believe the burden to bolster your argument lies on the OP.
Dan,
Two points.
First, I almost didn’t approve your message because of its tone. On my website, people are free to discuss ideas and different viewpoints. But it is always with a positive, friendly tone. This is your one and only warning. Next time your comment will be discarded without further discussion.
Second, Google is your friend. In less than five minutes I found this peer reviewed journal article that is a meta-analysis of 61 papers. Their findings are “we establish that, in situations where emotional engagement is high (e.g., an issue associated with a severe threat, involving a health issue, or affecting oneself), statistical evidence is less influential than anecdotal evidence. . . . Factors which heighten emotional engagement render decision makers more susceptible to the anecdotal bias.”
This finding is consistent with what I write in this blog post. My blog is based on facts.
Traci H.Frelinga, Zhiyong Yang, Ritesh Saini, Omar S. Itani, Ryan Rashad Abualsamh, “When poignant stories outweigh cold hard facts: A meta-analysis of the anecdotal bias”, Organizational Behavior and Human Decision Processes, Volume 160, September 2020, Pages 51-67.
if we only use statistical evidence of something’s “goodness,” where’s the fun and adventure in THAT?
Hi Scott,
If you’re talking about choosing a restaurant based on anecdotal evidence, that can be fun! However, if you’re talking approving medications for public use, making decision on aircraft parts, or trying to predict who will default on a loan, you’ll want to go the statistical route because making the wrong choice on those is not fun and can also be deadly and/or costly!
Interesting article. The use of a visualization to illustrate how anecdotal evidence only shows a part of the total picture was particularly effective. I came upon this article after writing my own about anecdotal evidence (https://fecundity.blog/anecdotes-are-not-evidence.html) and wanted to compare with what others had written.
amazing! helped me so much!