Levels of Evidence in OT
Occupational therapy practitioners turn to evidence-based practice (EBP) for many reasons.
We look for answers, treatment recommendations, and affirmation that we’re truly providing the best possible care to our clients.
But, the deeper we dig into the world of evidence-based practice, the more nuanced (and hopefully fun) it becomes.
Once you start immersing yourself in the world of EBP, you will quickly find that there are several ways of conceptualizing the levels of evidence—and, depending on your purpose and setting, you might find a certain model to be the most helpful.
In this article, we will explore why the levels of evidence matter in occupational therapy. We’ll also cover some standard ways of looking at levels of evidence—and we’ll take a look at the evidence pyramid I’ve been using for my journal club, which has been helpful for me in looking at OT-specific research.
Quick reminder: Why EBP matters for OT
Let’s take a step back and start with an actual definition of this crazy thing we call evidence-based practice.
One publication says: “Evidence-based practice (EBP) is the implementation of knowledge from clinical and professional expertise, patient-client circumstances and values, and best research evidence into current practice (Law & MacDermind, 2014; Straus, Richardson, Glasziou, & Haynes, 2005; Wong et al., 2014).
That’s a bit of a mouthful, but it’s essentially saying that EBP is looking at the assessments and interventions that research supports, and incorporating them into our care in a way that works for our patients’ needs.
Another consideration is WHY we really care about the evidence.
Considering how busy we are in our day-to-day practice, it’s worth looking critically at why this matters to us.
It turns out that EBP is associated with better outcomes, and it is more cost effective than non-evidence-based treatment approaches (Shin, Randolph, & Rauch, 2010; Straus et al., 2005). So, that’s a relief! By moving toward EBP, we’re likely streamlining our busy schedules and ultimately giving our patients better care (and hopefully saving them money in the process)!
Healthcare professionals, including OTs and OTAs, should really be thinking about what to cut, as much as what to add, when it comes to EBP.
The primary goal of EBP is to utilize research evidence to decrease the use of ineffective healthcare practices and ensure you are providing ethical, effective, and best treatment. (Baker & Tickle-Degnen, 2014; Law & MacDermind, 2014; Straus et al., 2005).
In other words, ditch the ineffective treatments and replace them with ones that are proven to work.
While this sounds obvious, it is actually much harder to change our professional habits than one may think. If you want to do some extra reading on this, I highly recommend the article, When Evidence Says No, But Doctors Say Yes.
Our professional organization is on board with EBP. EBP is a highly valued ideal of the American Occupational Therapy Association (AOTA), and is a pillar for their Vision 2025 (American Occupational Therapy Association, 2017; Gillen et al., 2017).
EBP is essential from a reimbursement perspective, too, as it is used as a framework for how interventions are evaluated by payers. In fact, EBP is considered the gold standard for reimbursement of services, especially with Medicare’s increased emphasis on value-based payment initiatives (Shin, Randolph, & Rauch, 2010; Straus et al., 2005).
Why levels of evidence matters for OTs
You are probably sufficiently convinced that EBP matters. So, let’s discuss “levels of evidence” and how they apply to EBP.
As we know from school, there are many ways to design a study, and there are certain presentations of evidence that are more helpful to your clinical practice than others.
In general, the levels of evidence serve as a mind map for conceiving which methodologies are most stringent and sound, and which ones should impact your practice most.
For example, an intervention is more evidence-based if a systematic review has indicated its efficacy, as opposed to that same intervention being supported by 100 promising case reports.
What are the different levels of evidence?
When searching for evidence-based information, you should generally select the highest level of evidence possible.
These are systematic reviews, meta-analyses, and/or critically-appraised topics. These sources of information have all gone through an evaluation process, and are considered “filtered.” Research articles that have not been evaluated for validity or efficacy are considered “unfiltered.”
The categorizations below are adapted from Kielhofner & Taylor, 2017.
Disclaimer: Keep in mind that the “levels of evidence” system isn't foolproof. A quality design (even a systematic review!) could be poorly executed and misleading, just as an obscure case study may actually align most closely with your particular patient—and therefore be the most helpful to you.
That said, let’s look at the levels of evidence. And, by levels of evidence, I mean essentially, the degree to which you can trust the research article, based on study design:
Filtered information (syntheses of previous studies)
Level I (Systematic Reviews and Meta-Analyses)
A systematic review is a structured review of all of the relevant literature on a topic. It is performed by searching for relevant articles, analyzing them for their validity and efficacy, and summarizing all of the literature results for a specific topic.
A meta-analysis is a systematic review that uses statistical methods to combine numerical results from many studies to summarize results. To help practitioners, the Cochrane Collaboration has its own database that contains even more helpful systematic reviews. The Cochrane Collaboration are experts in systematic reviews, and have even included their own level of rigor to help determine the efficacy of a treatment intervention. See what determinations the Cochrane Collaboration has stated about your favorite treatment intervention (if they examined it): Cochrane Collaboration
Level II (Critically Appraised Topics)
These are abbreviated versions of systematic reviews and meta-analyses. They are brief summaries of the best available evidence.
Unfiltered information (the studies themselves)
Level III (Randomized Controlled Trials)
In randomized controlled trials (RCTs), participants are randomly allocated to either an intervention group or a control group, and outcomes of both groups are compared. A double blind randomized control trial with a large number of participants has the strongest support for a “cause-and-effect” relationship with an intervention and the participants. (“Double blind” means all participants in both the intervention and control groups are “blind” to which group they’re in—and the administrators of the study are also “blind” to who is in each group.)
Level IV (Cohort Studies)
A cohort study (also called a longitudinal study) involves participants being assigned to a treatment or a control group. The assignment is not randomized. The study follows the groups of participants throughout a duration of time and gathers results. These studies are often not blind studies, and it is very difficult to control external variables that might impact the results.
Level V (Case-Control Studies and Case Series Reports)
A case-control study compares patients who have the outcome of interest (e.g., a disease) with patients who do not have the same outcome/control (in this example, a disease), and looks back retrospectively to examine the relationship between a particular variable (e.g., risk factor) and the outcome/control/disease. A more detailed example: comparing lifeguards who have developed skin cancer to lifeguards who do not have skin cancer and asking both groups the amount of sunscreen they wore.
These study designs are used during the early stages of research to help identify variables of interest. Case-control studies and case series reports often have a small number of participants, and are not generally randomized or controlled for confounding variables.
Level VI (Background Information and Expert Opinion)
Though every research project has a theoretical foundation, this study design on its own can be heavily influenced by personal beliefs and opinions. This evidence level is also deemed anecdotal evidence.
Common ways of organizing the levels of evidence
Now, if you were to Google levels of evidence, you might feel confused because there is not one standard evidence pyramid.
But, when you are looking at the various evidence pyramids out there, you should definitely see themes that you recognize (with systematic reviews being the highest quality of evidence, then RCTs, then case studies, etc.).
From there, there are nuanced differences in what is included and what is not included, based on the application of the pyramid.
For example, if you are focused on cutting-edge research as you try to bring a new technology to market, a pyramid that includes in-vitro and animal research would be really important to you, because this is where you will find cutting-edge advances happening.
Evidence pyramids that contain only research studies.
For us practitioners, though, the most important thing is probably having evidence filtered and explained regarding how it is applicable to our daily practice.
Most of us don’t have time to look at animal research and ponder what might come of it in 20 years. Therefore, a pyramid that highlights filtered information is probably the most helpful.
Research pyramids that divide evidence into filtered and unfiltered information
I also need to note that even though we tend to conceptualize levels of evidence in numerical form, there are also many examples of pyramids using numerical-alphabetical value. This is especially helpful for organizations who want to use a shorthand for the strength of evidence. For example, this treatment has level 1B studies, whereas this other one only has 3B level studies.
OT Potential Club levels of evidence pyramid
As I’ve been operating a journal club with weekly article reviews, I’ve started to hone in on a conceptual hierarchy of evidence for influential OT-related articles.
First, for practicing clinicians, articles that contain a synthesis of information seem to be the most helpful. The content in there is often geared towards actionable change in the clinic, which is what were are typically looking for as practitioners. So, for me, I like the structures where these articles go on the top of hierarchy.
Ultimately, I found this pyramid, which seems to originate from the University of Virginia Health Sciences Library, to match up with the evidence I was seeing most regularly.
For the studies, we’ve looked at, I’ve found that cross-sectional studies, animal research, article synthesis and expert opinions are not common in influential therapy-related articles, thus I have left them off this OT-directed pyramid. I actually ended up adding back in the category of mechanistic study, as this was a category we were encountering in the research.
Ultimately, we came up with this categorization:
This has given us a pyramid to categorize the studies we are looking at, based on the intent of the article. As a journal club, it is not within our purview to categorize the overall level of current research available or make a formal criticism of the execution of the study.
If you would like to see our pyramid in action, please join us at the OT Potential Club.
I hope that this review of the levels of evidence has empowered you to let go of the assumption that research hierarchies are intimidating and immutable. Rather, they are meant to be a helpful tool to assist in your clinical reasoning.
These research hierarchies attempt to provide a common language and some starting points for the most important step: analyzing and discussing how research can best serve your patients.
About the Authors
Bryden Giving, MAOT, OTR/L, is a recent graduate with a Masters of Occupational Therapy degree coupled with experience in research, evidence-based practice, and occupation-centered treatments for children and adults with developmental disabilities. Bryden's specialties include Autism and feeding therapy, and he aims to pursue a Doctorate in Public Health within four to five years with a concentration in health equity and social justice while continuing to advocate for a strengths-based approach when treating pediatric clients.
Sarah Lyon, OTR/L, is the founder of OT Potential, a platform for occupational therapy practitioners and students. In addition to an occupational therapy journal club, OT Potential hosts popular resources about salaries, evidence-based practice, job hunting, and general musings on the future of OT.
When not thinking about occupational therapy, Sarah can be found hanging out with her awesome family, enjoying the great outdoors, or frantically trying to finish her book club assignment.
American Occupational Therapy Association (AOTA). (2015). Critically appraised topic: Evidence for sound therapies for children with autism spectrum disorder. Retrieved from https://www.aota.org/Practice/Children-Youth/Evidence-based/CAT-ASD-Sound.aspx.
Baker, N., & Tickle-Degnen, L. (2014). Evidence-based practice: integrating evidence to inform practice. In B. A. B. Schell, G. Gillen, & M. E. Scaffa (Eds). Willard and Spackman's Occupational Therapy (12th ed., pp. 398-412). Philadelphia: Lippincott Williams & Wilkins.
Gillen, G., Lieberman, D., Stutzbach, M., & Arbesman, M. (2017). Five interventions/assessments our clients should question. OT Practice, 22(15), 19-20. Retrieved from https://www.aota.org/~/media/Corporate/Files/Secure/Publications/OTP/2017/OTP-Volume-22-Issue-15-bridge-obstacles.pdf
Shin, J., Randolph, G. W., & Rauch, S. D. (2010). Evidence-based medicine in otolaryngology, part 1: The multiple faces of evidence-based medicine. Otolaryngology – Head and Neck Surgery, 142, 637-646. doi:10.1016/j.otohns.2010.01.018
Straus, S. E., Richardson, W. S., Glasziou, P., & Haynes, R. B. (2005). How to practice and teach EBM. Evidence-Based Medicine. Third edition. Elservier, 13-29. doi:10.1177/1049731506296167
Wong, C., Odom, S. L., Hume, K. Cox, A. W., Fettig, A., Kucharczyk, S., ... Schultz, T. R. (2014). Evidence-based practices for children, youth, and young adults with Autism Spectrum Disorder. Retrieved from http://autismpdc.fpg.unc.edu/sites/autismpdc.fpg.unc.edu/files/2014-EBP-Report.pdf.