top of page
reliability and validity
Reliability and Validity
The Fundamentals of Science
How do you consume science? Although I enjoy mine with a side of eel sauce, I like research best with lots of reliability and validity. What do these terms mean, and why should you care?
Reliability relates to how consistently
results hold up over time, across people coding data, or within a tool used to collect data. These measures give us a sense of results’ reproducibility. Importantly, if results are not reliable, they cannot be valid. Validity is how accurately a study measured the concept it hoped to assess.
Look at the bullseye rolls. In the top roll, I placed the black roe with consistency, but I clearly missed the target. My roe placement was reliable, but not valid. On the bottom roll, I placed the roe consistently on the target. This roll is reliable and valid.
Why should you care about reliability/validity? Scrolling through your social media newsfeeds, you probably come across lots of articles describing interesting findings (e.g., Chocolate is good for you! Wine makes you live longer!). Although these articles tend to give a nice snapshot of the study they review, they often only present findings from the highlighted study.
Think about the debate over vaccines causing autism. In 1998 and 2002, one research group published 2 articles describing an observation of kids who were diagnosed with autism ~1 month after vaccination. These results were highly publicized. However, the studies had flaws and ethical concerns that led to misattribution of results. For example, the researchers didn’t consider the inherent similarity in age for when 1) kids get vaccines and 2) most cases of autism are diagnosed.
After many studies were unable to reproduce these findings, one of the articles was retracted. Since then, organizations have publicly pushed to debunk the myth that resulted from these articles. However, some parents still choose not to vaccinate their kids, worried that they will develop autism.
So before changing your own actions after reading one exciting study, you might want to take the results with a grain of salt (or sushi rice) until they are reproduced.
References
[1] Heale, R., & Twycross, A. (2015). Validity and reliability in quantitative studies. Evidence-based nursing, ebnurs-2015.
[2] Wakefield, A. J., Murch, S. H., Anthony, A., Linnell, J., Casson, D. M., Malik, M., ... & Valentine, A. (1998). RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children.
[3] Wakefield, A. J. (2002). Enterocolitis, autism and measles virus. Molecular Psychiatry, 7(S2), S44.
[4] Madsen, K. M., Hviid, A., Vestergaard, M., Schendel, D., Wohlfahrt, J., Thorsen, P., ... & Melbye, M. (2002). A population-based study of measles, mumps, and rubella vaccination and autism. New England Journal of Medicine, 347(19), 1477-1482.
[5] Taylor, L. E., Swerdfeger, A. L., & Eslick, G. D. (2014). Vaccines are not associated with autism: an evidence-based meta-analysis of case-control and cohort studies. Vaccine, 32(29), 3623-3629.
[6] Jain, A., Marshall, J., Buikema, A., Bancroft, T., Kelly, J. P., & Newschaffer, C. J. (2015). Autism occurrence by MMR vaccine status among US children with older siblings with and without autism. Jama, 313(15), 1534-1540.
[7] Autism Speaks, “What causes autism?”
[8] World Health Organization, “MMR and Autism”
[9] US Center for Disease Control, “Vaccines do not cause autism concerns”
[10] Tomeny, T. S., Vargo, C. J., & El-Toukhy, S. (2017). Geographic and demographic correlates of autism-related anti-vaccine beliefs on Twitter, 2009-15. Social Science & Medicine, 191, 168-175.
bottom of page