In a recent post, I described a study that would settle any lingering doubts about vaccine safety or effectiveness. There were a few practical issues, not the least of which was that it would be despicably unethical because it would involve children dying from preventable diseases. It would also take decades, cost an insane amount of money, and involve completely uprooting families from their homes.
Obviously, the post was satirical, and my goal was to point out the reasons we haven’t (and never will) perform a double-blinded, randomized, placebo-controlled trial for our current vaccine schedule. Which is why I was astonished to read in the comment thread that a group had very recently published “this *exact* study.” Of course, I was curious to see the results, so I clicked on the link. I downloaded a copy to review, but you may have a hard time finding it–I’ll get to that in a bit.
“Study” Design
First things first. It’s not a study. It’s a survey. What’s the difference? Well, studies start with a specific question, and are designed to test an educated guess (“hypothesis”) about that question. This one didn’t.
Let’s start at the beginning:
We did not set out to test a specific hypothesis about the association between vaccination and health.
They lost me from hello. Choosing a hypothesis to test is basic. Really, really basic. It’s how science works, and kids learn that in elementary school. Admitting that you didn’t do it, and expecting someone to publish your work, is absurd.
After bypassing a really crucial step, they gathered their study participants. They did this by recruiting from homeschool organizations. First of all, I have nothing against homeschooling. My wife and I homeschooled one of our children for 3 years, and I think it’s a great option for some families. The problem is that homeschoolers are not a representative sample–this is known as selection bias. Homeschoolers are less likely than average families to seek medical care (as stated in this article as well as other surveys). The authors of the study report that this is the reason they made this decision–because, whether due to religious or personal beliefs or a lack of school-mandated immunizations, they are less likely to be immunized.
What the authors either failed to realize, or outright ignored, is that homeschoolers also less likely to receive medical care for a lot of other issues. ADHD may not be as much of a problem outside a traditional classroom setting. Learning disabilities or mild autism spectrum disorders may not be picked up because children are not compared to peers or observed as frequently by physicians or teachers. Some infections may be diagnosed less commonly because the parents don’t seek medical care (potentially simply because they don’t need a school excuse).
The authors invited various homeschool organizations to forward survey links to their members, and homeschool parents invited their friends:
As contact information on homeschool families was unavailable, there was no defined population or sampling frame from which a randomized study could be carried out, and from which response rates could be determined. However, the object of our pilot study was not to obtain a representative sample of homeschool children but a convenience sample of unvaccinated children of sufficient size to test for significant differences in outcomes between the groups…A number of homeschool mothers volunteered to assist NHERI promote the study to their wide circles of homeschool contacts.
In other words, they knew this wasn’t a representative or random sample, and they were ok with that. We don’t know how many families got invited to take the survey (because the researchers don’t either), but we do know that those who felt strongly that vaccines were unsafe would be more likely to respond. That’s a very common phenomenon–very few people will take the time to review a book that they thought was alright or a restaurant that served decent food. Those who feel strongly about a topic will be most likely to respond. And in a group biased toward not vaccinating, those who feel strongly are more likely to be opposed to vaccines–especially if the study group is recruiting their own friends.
There’s a table in the “Methods” section of nearly every scientific article, called “Table 1.” The purpose of it is to compare the two groups, essentially showing whether or not they are similar in important areas. Ideally, they would be as similar as possible except for the thing being tested. This article doesn’t even separate the groups–it just lists demographic information about the entire group of respondents. It’s entirely useless, except to point out that married, white, Christian, female college graduates are over-represented as compared to the general population. That’s important when generalizing the results to the whole population, but not nearly as important as how the two groups compare to each other.
The sample size of 415 families and 666 children is grossly inadequate. Serious adverse events to vaccines happen on a one-in-a-million scale. And the 261 unvaccinated children are not nearly enough that we should expect to see severe complications or deaths from the diseases that could have been prevented. If a sample size is too small, infrequent events may be over-represented or not present at all. Having the appropriate sample size is crucial, and depends on the question you are trying to answer–which nobody seemed to know.
Mothers who responded to the survey were asked questions about their children’s vaccine status and diagnoses. And that’s pretty much it for the data gathering. Nobody reviewed any medical charts, spoke with the children’s physicians, or verified any of the parent’s claims. The researchers relied solely on memories–which are notoriously, if fascinatingly, inaccurate. We know that parents of children who suffer from some traumatic event are more likely to remember the events surrounding that period–events such as a recent doctor’s visit in which vaccines were given. This is known as recall bias, and it’s why we should be cautious about trusting surveys based on unconfirmed memories.
The researchers assume that the children’s mothers have intimate knowledge of the vaccine schedule and can accurately determine whether their children are “fully vaccinated,” “partially vaccinated,” or “unvaccinated.” While “unvaccinated” is pretty straightforward, the other two groups are not. The immunization schedule isn’t all that simple–I still consult it from time-to-time–and relying on parents to click the right box just doesn’t seem wise.
This method also assumes that every child with a particular condition was diagnosed with that condition, and that the mother remembers it. But we’ve already established that these families are less likely to seek medical care. And within that subset of the population, those who chose not to vaccinate their children were seen by a doctor even less frequently–according to the study data. So how many learning disabilities, cases of mild autism, or children with ADHD went undiagnosed? How many ear infections–which can have effects on hearing and language development–were missed? Can we really compare diagnosed conditions in two groups who have unequal opportunities for diagnosis? (The answer, of course, is no.)
And finally, there’s no way of knowing whether a child had a preventable disease before or after he/she became “fully immunized” or “partially immunized” (whatever that means). Did the 7.9% of vaccinated children who had chickenpox get it before or after they were vaccinated for chickenpox? I have to assume before, because I’ve taken care of thousands of children, and–thanks to vaccines–I’ve seen exactly 2 with chickenpox (both in immunocompromised patients). So these were almost certainly either children who were unvaccinated at the time of the disease, or self-diagnosed diseases that weren’t chickenpox at all.
Results
Well, now for the results. Unfortunately, they are entirely worthless because of the flawed study design. But here they are anyway.
Vaccinated children (N=405), combining the partially and fully vaccinated, were significantly less likely than the unvaccinated to have had chickenpox (7.9% vs. 25.3%, p <0.001; Odds Ratio = 0.26, 95% Confidence Interval: 0.2, 0.4) and whooping cough (pertussis) (2.5% vs. 8.4%, p <0.001; OR 0.3, 95% CI: 0.1, 0.6), and less likely, but not significantly so, to have had rubella (0.3% vs. 1.9%, p = 0.04; OR 0.1, 95% CI: 0.01, 1.1). However, the vaccinated were significantly more likely than the unvaccinated to have been diagnosed with otitis media (19.8% vs. 5.8%, p <0.001; OR 3.8, 95% CI: 2.1, 6.6) and pneumonia (6.4% vs. 1.2%, p = 0.001; OR 5.9, 95% CI: 1.8, 19.7). No significant differences were seen with regard to hepatitis A or B, high fever in the past 6 months, measles, mumps, meningitis (viral or bacterial), influenza, or rotavirus (Table 2).
Vaccinated children were significantly more likely than the unvaccinated to have been diagnosed with the following: allergic rhinitis (10.4% vs. 0.4%, p <0.001; OR 30.1, 95% CI: 4.1, 219.3), other allergies (22.2% vs. 6.9%, p <0.001; OR 3.9, 95% CI: 2.3, 6.6), eczema/ atopic dermatitis (9.5% vs. 3.6%, p = 0.035; OR 2.9, 95% CI: 1.4, 6.1), a learning disability (5.7% vs. 1.2%, p = 0.003; OR 5.2, 95% CI: 1.6, 17.4), ADHD (4.7% vs. 1.0%, p = 0.013; OR 4.2, 95% CI: 1.2, 14.5), ASD (4.7% vs. 1.0%, p = 0.013; OR 4.2, 95% CI: 1.2, 14.5), any neurodevelopmental disorder (i.e., learning disability, ADHD or ASD) (10.5% vs. 3.1%, p <0.001; OR 3.7, 95% CI: 1.7, 7.9) and any chronic illness (44.0% vs. 25.0%, p <0.001; OR 2.4, 95% CI: 1.7, 3.3). No significant differences were observed with regard to cancer, chronic fatigue, conduct disorder, Crohn’s disease, depression, Types 1 or 2 diabetes, encephalopathy, epilepsy, hearing loss, high blood pressure, inflammatory bowel disease, juvenile rheumatoid arthritis, obesity, seizures, Tourette’s syndrome, or services received under the Individuals with Disabilities Education Act (Table 3).
So they say vaccines reduce the risk of chickenpox and pertussis (which can be fatal). The risk of rubella is reduced–I’m not sure why they say that isn’t significant, because scientists have traditionally agreed to define a p-value of <0.05 as significant. Perhaps they were unfamiliar with this nearly-universal scientific convention.
Vaccinated children were reported to be more likely to be diagnosed with ear infections, pneumonia, allergies, eczema, learning disabilities, autism, and ADHD. Remember, this is the number of children who were diagnosed with these things–quite possibly because they also went to the doctor more. And quite possibly, that diagnosis provided a way for the child to get the help they needed, whether it was relief from allergies, treatment for frequent ear infections, or ABA therapy for autism.
Authors/Funding/Publication
Note that until now I have evaluated only the methods used by the researchers. I started there because I feel that’s the most important part. Let’s take a look at the authors:
- Anthony R. Mawson, M.A., Dr.P.H.- Professor of Epidemiology and Biostatistics at Jackson State University. Dr. Mawson is a Doctor of Public Health–not a physician. He was a supporter of Andrew Wakefield, who used to be a physician until his license was revoked because of a fraudulent study in which he linked the MMR (measles, mumps, rubella) vaccine to gastrointestinal inflammation and implied that it may cause autism. Regarding that discredited study, Mawson wrote: “The paper, once understood in this light, as case series analysis, is truly remarkable, well written and brilliantly documented. It concluded by stating the hypothesis, based on parents’ reports, that the children’s’ signs and symptoms were temporally connected to MMR vaccination. Subsequent studies may not have substantiated the hypothesis; but that does not detract from or invalidate the merits of the paper as a case series and as, essentially, a hypothesis paper.”
- Brian D. Ray, PhD, MS- president of the National Home Education Research Institute. Holds a doctorate degree in science education.
- Azad R Bhuiyan, MD, MPH, PhD- Associate Professor of Epidemiology and Biostatistics at Jackson State University
- Binu Jacob–listed simply as a “former graduate student at Jackson State University.” A Google search revealed a cardiologist by that name, but I have no knowledge of whether that is the same person.
Funding
The authors don’t report any financial conflicts of interest, but at the end of the paper, you’ll find this note:
This study was supported by grants from Generation Rescue, Inc., and the Children’s Medical Safety Research Institute, charitable organizations that support research on children’s health and safety.
Generation Rescue is Jenny McCarthy’s organization that has been…let’s say, vocal, in alleging that vaccines cause autism. (They don’t.) CMSRI is another organization with similar, staunchly anti-vaccine views. For all the fuss about studies funded by pharmaceutical companies, nobody seemed to mind that this one was bankrolled by some heavily biased groups. I don’t know for sure how much funding the authors received, but at one point, there was a petition for $500,000. I’m not sure how this study could possibly have cost anywhere close to that (or if they actually got it). But they tried.
Peer Review
This paper is “peer-reviewed,” but the term “peer” applied loosely. One of the two–yes, only two–reviewers was Linda Mullin, DC, a chiropractor at Life University. Suffice it to say that a chiropractor is not the ideal choice to critically evaluate a study involving pediatrics, vaccines, or infectious diseases.
Publication
As far as publication goes, this study isn’t new. It was done in 2012, and initially accepted for publication in Frontiers in Public Health in 2016. It was shortly removed, before the publication date. The authors ended up paying to publish it on Open Access Text, but it was rejected from there as well, just days after publication. The fact that they couldn’t get a reputable journal to publish it says something; the fact that a pay-to-publish journal removed it after publication says even more. I’m uncertain if they received a refund–or, if they did, if they plan to return the money to their anti-vaccine supporters.
Conclusion
I’ve written a lot about vaccines in the past. Sometimes I feel like I’m beating the greasy spot on the ground where the dead horse used to be. In exchange, I’ve received no end of angry emails and comments, a couple non-specific death threats, and still no payments from any pharmaceutical company. I do my best to remain rational when responding to these comments (except those threatening harm).
I am firmly convinced that most parents who choose not to vaccinate do so out of legitimate fear and a desire to do what is best for their child. And while I disagree with their decision, I respect their motives. And because my goal with this blog is to help parents make informed decisions about their children’s healthcare, I keep going.
But when someone incites even more fear in those parents, I get angry. I get angry because their actions cause more stress and guilt for parents. Because their actions result in children suffering–and potentially dying–from preventable diseases. Because so much of my time with patients is spent trying to undo their damage–valuable time that could be spent discussing so many other important issues.
And when that “someone” includes doctoral-level epidemiologists, funded by outspoken anti-vaccine groups, who publish a study with no hypothesis and the worst methodology I’ve ever seen, implying that their skewed survey results overrule the mountains of evidence that we have for vaccine safety, I have to think they did it on purpose.
I agree completely that this so-called study is a textbook example of stacking the deck, but I’m a little curious about your comment”He has ties to Andrew Wakefield.”
That’s the sort of vague statement that annoys the ds out of me when antivaxxers use it. What kind of ties does Mawson have? Please explain.
Sorry–hadn’t added links yet.
http://archive.li/SpgMC
-Mawson
Thanks for the clarification; this doesn’t surprise me.
This is a remarkably level headed discussion.
Can you help me understand- you correctly pointed out that there’s no specific hypothesis tested here. What’s the hypothesis behind the one study demand? It seems like there’s some underlying assumption that any vaccine makes an irrevocable change in the child, but I’m not clear how and why.
Not sure what you mean.
There seems to be an assumption that there is a meaningful difference between having no vaccines and having any. But I don’t get how that would work. What’s the biological mechanism that would make these children different across many different things beyond different rates of the spectrum diseases vaccines prevent?
Depending on how you look at it, there is either not one, or very many. I’ve read some vague thoughts that tampering with the immune system could shift the immune response in the direction of allergy, that a specific antigen could trigger autoimmune disease, that the MMR vaccine could trigger gut inflammation, that the aluminum or thimerosal or unnamed “toxins” can do something…
Frankly, I don’t know that the ones leading this movement are all that concerned about biological plausibility. As long as it sounds sciencey enough and scares parents, it’s effective.
Thank you.
Sad but true.
Glad to see this stuff. IMO, the response to claims that vaccines cause autism has for too long been dominated by public health doctors and epidemiologists. Back in the day, when there were worries about the old whole-cell DTP, pediatricians, and especially pediatric neurologists were quite prominent.
http://www.thelancet.com/journals/lancet/article/PIIS0140-6736%2897%2911096-0/abstract
The retracted Wakefield paper?
Chad, thanks for this thoughtful and thorough review. I’ve included it in the roundup post:
“About those Homeschooled Unvaccinated Children are Healthier studies:
http://lizditz.typepad.com/i_speak_of_dreams/2017/05/about-those-homeschooled-unvaccinated-children-are-healthier-studies-.html