These past years have provided a tremendous education in the ways that seemingly rigorous science can actually be highly misleading. Much of the problem traces to a simple idea: selection bias. Once you understand it, much of what we think we know about what is true, along with many studies showing it to be true, melt away.
Good science corrects for the problem, but it is still ubiquitous.
Let’s take a simple example.
Say you have a dog and you make its meals. You are careful about this and rather strict, making sure that you have all the right vitamins and minerals in the meals and each is the right size, and you stick with it daily. You monitor the results. You save money. The dog is healthy and happy.
But then someone comes along waving scientific literature on the topic. They point out that all of the studies show that dogs fed with home-cooked meals have a far higher rate of gastrointestinal problems. In fact, many studies show this, and veterinarians cite them in defense of commercial dog food.
They yell at you: Never feed dogs people food. It will make them sick. The only real path to a healthy pet is the bags at the store.
It’s intimidating, isn’t it? You think that you are doing the right thing, but the experts say otherwise. What might be going on? Well, the science is not wholly incorrect. It is likely true that dogs that are fed with commercial food are generally healthier than those who eat table scraps.
Why might that be? Simple. The subjects who buy the commercial stuff tend to be more scrupulous about their pet’s health. They measure. They feed at certain times per day. With a stable commercial product, there is less to go wrong. They are willing to spend more money on pet food.
On the other hand, people who report feeding dogs people food are more likely the neglectful ones, scraping the leftovers into the dog’s bowl and otherwise giving the dog whatever scraps fall from the table—could be tacos, mac and cheese, salad with dressing, or a big piece of meat with sauces and salts and whatever. They simply are not monitoring.
And sure enough, those people end up with pets at the doctor more often. That is precisely when the gastrointestinal problems are found and recorded.
That’s not what you do, but you are lumped in with the others in the selection bias. The truth might be the opposite of the study. The people who make stable and healthy meals for their pets are doing better for the dog’s health than those who use commercial food filled with grains and processed products. But you are not the sample being studied.
The study’s conclusions, then, are reversed by the problem of the built-in bias. In other words, the study was designed to produce the precise results that it did produce.
This is a major feature of so many scientific undertakings. They generalize from a population that includes a range of behaviors but is dominated by one feature and conclude fallacious recommendations from the whole.
Consider the example of red wine studies from years ago. They concluded that drinking one or two glasses of red wine per day can be good for the heart. They deduced this based on surveys that showed precisely that: People who drink red wine daily have fewer overall health problems than those who do not.
It took many years, but finally some researchers grew suspicious. They wondered whether the study was a victim of selection bias. Perhaps moderate drinkers were healthier than nondrinkers because they were from a higher socioeconomic status. They are wealthy and physically active. They are more likely to have health insurance, see doctors more often, and eat balanced diets. And they had another theory: The nondrinkers in the study might, in fact, be ex-drinkers who quit to stop health decline.
Sure enough, after reviewing more than 100 studies, researchers have more recently concluded that no amount of alcohol is good for you and that studies that say the opposite were making conclusions about causation that could not be supported by the studies’ results. Now, we find that the conventional wisdom has shifted again. It took decades, but now we are seeing that the studies were flawed from the beginning.
This is what shoddy science does. It makes us believe one thing that turns out not to be true once the selection bias is eliminated.
This problem has been huge for COVID-19 vaccine studies over the past three years. The question was a simple one. Among the populations that received the initial round plus boosters, against those who received neither, which had fewer health problems? You might think that constructing such a study would be simple.
It is not simple because you must compare likes to likes. You cannot compare relatively healthy populations with relatively less healthy populations. That would make a mess of the conclusions of the question you are trying to answer. But it turns out that this is precisely what many studies did. They committed the error of building in a “healthy vaccinee bias.”
As you might expect, the people who complied with the mandates and then got boosters tended to be of higher socioeconomic status, have health insurance, visit the doctor more often, and generally be more scrupulous about their health. The people who never got one might have done so because they didn’t trust the vaccine, but they also might not have had access to quality health care and might generally disregard their health.
The crucial point here is that even a small bias can ruin such a study. As it turns out, this is precisely what happened in most early studies. As time went on, more researchers got involved in looking more carefully. They discovered that once you eliminate the bias, the conclusions of the study are not only not supported but sometimes even reversed. This is especially true because more people grew suspicious over time, such that healthier populations started to say no.
They conclude: “Consistently over datasets and age categories, all cause mortality (ACM) was substantially lower in the vaccinated than unvaccinated groups regardless of the presence or absence of a wave of COVID-19 deaths. Moreover, the ACMs in groups more than 4 weeks from Doses 1, 2, or 3 were consistently several times higher than in those less than 4 weeks from the respective dose. The healthy vaccinee effect appears to be the only plausible explanation for this, which is further corroborated by a created mathematical model.”
That is a serious and substantial error. It causes the judgments to be reversed. What seems to be science is actually rooted in what is really a simple mistake. Sadly, this selection bias is a huge factor in many studies. Once you become aware of it, you can spot it rather quickly.
Let’s say a study appears today that says people who eat roasted brussels sprouts tend to live longer, healthier, and richer lives. Are you going to believe that the brussels sprouts cause this to happen or will you now clearly see that what is being tested here is a feature of the selection rather than an underlying cause?
Remember this: A life lived according to the dictates of “the science” puts your life decisions at the mercy of the integrity of the researchers. That might not be the wisest choice.
