Levels of Evidence in OT

Occupational therapy practitioners turn to evidence-based practice (EBP) to look for answers, treatment recommendations, and affirmation that we’re truly providing the best possible care to our clients.

Once you start immersing yourself in the world of EBP, you will quickly find that there are several ways to conceptualize the levels of evidence—and, depending on your purpose and setting, you might find a certain model to be more helpful than the others.

In this article, we will explore why the levels of evidence matter in occupational therapy. We’ll also cover some standard ways of looking at levels of evidence, and we’ll take a look at the evidence pyramid I’ve been using for my journal club, which has been helpful as I look at OT-specific research.

Why the levels of evidence matter for OTs

Let’s discuss “levels of evidence” and how they apply to evidence-based practice in OT as well as knowledge translation.

Research evidence is a critical part of EBP, but there are certain presentations of evidence that are more helpful to your clinical practice than others.

The levels of evidence serve as a mind map for determining which methodologies are most stringent and sound—and which ones should impact your practice most.

For example, an intervention is more evidence-based if a systematic review has indicated its efficacy, as opposed to that same intervention being supported by 100 promising case reports.

What are the different levels of evidence? 

When searching for evidence-based information, you should generally seek out the highest level of evidence possible.

These are systematic reviews, meta-analyses, and/or critically-appraised topics. These sources of information have all gone through an evaluation process and are considered “filtered.”

Research articles that have not been evaluated for validity or efficacy are considered “unfiltered.” The categorizations below are adapted from the 2017 book, “Kielhofner’s Research in Occupational Therapy.”

Disclaimer: Keep in mind that the “levels of evidence” system isn’t foolproof. A quality design (even a systematic review!) could be poorly executed and misleading, just as an obscure case study may actually align most closely with your particular patient—and therefore be the most helpful. It is also important to note that a lower-level study in a prestigious OT-related journal may carry more weight than a systematic review in a less well-known journal.

That said, let’s look at the levels of evidence (that is, the degree to which you can trust any given research article, based on study design).

Filtered information (e.g., syntheses of previous studies) 

Level I (Systematic Reviews and Meta-analyses)

A systematic review is a structured evaluation of all of the relevant literature on a topic. It is performed by searching for relevant articles, analyzing them for validity and efficacy, and summarizing all of the literature results for a specific topic.

A meta-analysis is a systematic review that uses statistical methods to combine and summarize numerical results from many studies. The Cochrane Collaboration has its own database containing the most helpful systematic reviews for practitioners. They are experts in systematic reviews, and have even included their own level of rigor to help determine the efficacy of a treatment intervention. See what determinations the Cochrane Collaboration has stated about your favorite treatment intervention (if they examined it) here.

Level II (Critically Appraised Topics) 

These abbreviated versions of systematic reviews and meta-analyses offer brief summaries of the best available evidence.

Unfiltered information (e.g., the studies themselves)

Level III (Randomized Controlled Trials) 

In randomized controlled trials (RCTs), participants are randomly allocated to either an intervention group or a control group, and outcomes for both groups are compared. A double-blind randomized control trial with a large number of participants provides the strongest support for a “cause-and-effect” relationship with an intervention and the outcomes achieved. (“Double blind” means all participants in both the intervention and control groups are “blind” to which group they’re in—and the administrators of the study are also “blind” to who is in each group.)

Level IV (Cohort Studies)

In a cohort study (also called a longitudinal study), participants are assigned to either a treatment group or a control group. The assignment is not randomized. The study follows both groups of participants over a specified duration of time and gathers results. These studies often are not blind studies, and it is very difficult to control external variables that might impact the results.

Level V (Case-Control Studies and Case Series Reports) 

A case-control study compares patients who have the outcome of interest (e.g., a disease) with patients who do not have the same outcome/control (in this example, a disease), and looks back retrospectively to examine the relationship between a particular variable (e.g., risk factor) and the outcome/control/disease. For example, a case-control study might compare lifeguards who have developed skin cancer to lifeguards who do not have skin cancer, looking specifically at the amount of sunscreen each lifeguard wore while on the job.

These study designs are used during the early stages of research to help identify variables of interest. Case-control studies and case series reports often have a small number of participants, and are not generally randomized or controlled for confounding variables.

Level VI (Background Information and Expert Opinion) 

Though every research project has a theoretical foundation, this study design on its own can be heavily influenced by personal beliefs and opinions. This evidence level is also deemed anecdotal evidence.

Common ways of organizing the levels of evidence

Now, if you were to Google “levels of evidence,” the results might be confusing, because there is not one standard evidence pyramid.

But, as you explore the various evidence pyramids out there, you will definitely see some common themes (with systematic reviews being the highest quality of evidence, then RCTs, then case studies, etc.).

From there, there are nuanced differences in what types of evidence are and are not included, based on the application of the pyramid.

For example, if you are focused on cutting-edge research as you try to bring a new technology to market, a pyramid that includes in-vitro and animal research might be really important to you, because this is where you would find cutting-edge advances happening.

Evidence pyramids that contain only research studies

For us practitioners, though, it is more important for evidence to be filtered and explained (particularly with regard to how it applies to our daily practice).

Most of us don’t have time to look through animal research and ponder what might come of it in 20 years. Therefore, a pyramid that highlights filtered information is probably the most helpful.

Research pyramids that divide evidence into filtered and unfiltered information

I should also note that even though we tend to conceptualize levels of evidence in numerical form, there are many examples of pyramids that use numerical-alphabetical values. This is especially helpful for organizations that want to use a shorthand denoting the strength of evidence (e.g., “X treatment has level 1B studies, whereas Y treatment only has 3B studies.”).

In fact, the American Occupational Therapy Association (AOTA) Approved Provider Program uses a numerical-alphabetical hierarchy to assess evidence used in their professional development activities.

OT Potential Club levels of evidence pyramid

Through the numerous article reviews I have performed, I’ve developed my own conceptual hierarchy of evidence for influential OT-related articles.

First, for practicing clinicians, articles containing a synthesis of information seem to be the most helpful. Their content is often geared toward actionable change in the clinic, which is typically what we are looking to achieve. So, I prefer the structures where these articles sit at the top of the hierarchy.

I have found that the pyramid below, which appears to originate from the University of Virginia Health Sciences Library, most closely matches my preferred evidence hierarchy.

For the studies we’ve looked at, I’ve found that cross-sectional studies, animal research, article synthesis, and expert opinions are not common in influential therapy-related articles; thus, I have left them off of our OT-directed pyramid. I actually ended up adding back in the category of mechanistic study, as this was a category we were encountering in the research.

Ultimately, we came up with this categorization:

This has given us a pyramid to categorize the studies we are looking at based on the intent of the article, as it is not within our purview to categorize the overall level of current research available or to make a formal criticism of the execution of the study.

If you would like to see our pyramid in action, please join us in the OT Potential Club!

What about qualitative evidence?

You may notice that the evidence pyramids covered so far include only quantitative study designs. Qualitative research designs like ethnography, phenomenology, and grounded theory design are on the rise in health literature, and their findings can provide rich insights into the lives and experiences of the people and communities we work with.

But the flexible nature of qualitative study designs and the wide diversity of methods makes it trickier to assess the quality of qualitative findings—and to determine how they should inform OT practice.

It is worth noting that filtered qualitative information (e.g., systematic reviews and analyses) are becoming more popular in the literature, but even these reviewers struggle to define clear criteria for assessing the strength of qualitative studies. Most qualitative research is unfiltered and requires careful analysis.

Evaluating Qualitative Evidence

There is much debate about how qualitative evidence can and should be evaluated—and whether the knowledge gained through these studies can really be judged at all. Stories are captured and used very differently than statistics, and they cannot be measured by the same standards of validity and reliability.

A number of models and checklists have been proposed for assessing whether a qualitative study is strong or poor in its design and execution, but gold standard criteria have yet to rise to the top.

Here, we present two approaches: a hierarchical model and a checklist of questions for assessing the strength of qualitative study design and execution. Know that no one set of criteria can perfectly capture all the nuances of rigor in qualitative research, but these approaches offer a great starting place for feeling out the strength of a study and its relevance to your practice and clients.

Daly et al. (2007) offers this hierarchy for assessing the strength of qualitative studies, with the ideal studies being those which are most generalizable to other settings.

Level 1 (Generalizable Studies)

The ideal qualitative research for informing your evidence base is one that builds off existing literature and theoretical frameworks, provides detailed and comprehensive descriptions of data collection and analysis, and offers clear implications for practice and policy. Generalizable studies should clearly report on sampling saturation and diversity, as well as detail the study’s relevance to other populations and settings.

Level 2 (Conceptual Studies)

Existing theoretical concepts (like stigma, cultural identity, or gender differences) guide conceptual studies, and participants in these studies are selected to understand the concept through their lens. The goal of conceptual studies is to develop an overall account of the relationship between participants and the concept.

Level 3 (Descriptive Studies)

Descriptive studies, as their name suggests, describe rather than explain participant experiences and views, typically through direct quotes. This study type is often seen in mixed methodology research to give additional context and richness to quantitative data.

Level 4 (Single Case Study)

The “lowest” level of qualitative evidence comes from studies involving a small group or a single subject. Case studies can shed light on previously unexplored or underrepresented experiences in healthcare, but their very small sample sizes limit the generalizability of the findings to broader populations. They are often used as starting points for generating future research questions rather than providing direct evidence for practice or
policy.

Things to look for (and to look out for) in qualitative studies

In qualitative research, rigor is less about the type of study design and more about the researchers’ transparency, descriptiveness, and adherence to the methodology.

It can be tempting to skip straight to the quotes in a qualitative study, because quotes are compelling, moving, and often the most interesting part of qualitative research! But quotes alone do little to inform practice unless they are contextualized within the study by comprehensive and transparent descriptions.

Quality checklists are another imperfect, yet useful, way of assessing qualitative studies without hierarchy. The table below is synthesized from articles from 2020, 2007, and 2000. From these articles, here are five important considerations to look for when assessing a qualitative study:

In the video below, you can see an example of how this checklist and the hierachy above can be used by OT practitioners to evaluate qualitative research:

Conclusion

We hope this review of the levels of evidence has empowered you to let go of the assumption that research hierarchies are intimidating and immutable. Rather, they are meant to be a helpful tool to assist in your clinical reasoning.

These research hierarchies attempt to provide a common language and some starting points for the most important step: analyzing and discussing how research can best serve your patients.

Lastly, if you are interested in OT-related research, we hope you consider joining us in the OT Potential Club!

References

AOTA APP Levels and Strength of Evidence | AOTA. (2024).

Busetto, L., Wick, W., & Gumbinger, C. (2020). How to use and assess qualitative research methods. Neurological Research and Practice, 2(1), 14.

Daly, J., Willis, K., Small, R., Green, J., Welch, N., Kealy, M., & Hughes, E. (2007). A hierarchy of evidence for assessing qualitative health research. Journal of Clinical Epidemiology, 60(1), 43–49.

Mays, N., & Pope, C. (2000). Assessing quality in qualitative research. BMJ : British Medical Journal, 320(7226), 50–52.