This blog post is part of our Research Article of the Month series. For this month, we highlight “How Do Intervention Studies Measure the Relation Between Implementation Fidelity and Students’ Reading Outcomes,” an article published in the Elementary School Journal in 2023. Important words related to research are bolded, and definitions of these terms are included at the end of the article in the “Terms to Know” section.

What Did the Researchers Examine?

Educators consider many factors when delivering a reading intervention. How will students be grouped for instruction? What activities will they complete? How will the activities be ordered? How much time will they spend on each activity? Even with the most detailed plan, many factors can affect how the intervention is delivered. Student behavior, administrative support, teacher preparation, and the complexity of the intervention are only some of the factors that can affect the delivery of instruction. 

Therefore, when implementing a new reading intervention, many educators and researchers track the extent to which the intervention is delivered as intended. This is referred to as fidelity of implementation. The definitions of fidelity of implementation and its components vary, though many definitions include the following components: 

• Dosage: the amount and frequency with which a student receives instruction (e.g., number of minutes per session, number of sessions per week, number of weeks of intervention)
• Adherence: the extent to which intervention steps were implemented as planned
• Quality: how well the intervention was implemented

This study examines the relationship between fidelity of implementation and student reading outcomes. Understanding this relationship can help educators determine which interventions work, who they work for, and under which conditions they work best. 

The researchers sought to shed light on the relationship between fidelity of implementation and reading outcomes by addressing the following research questions:

• Which types of fidelity of implementation do researchers use to examine the relation between fidelity of reading interventions and students’ reading outcomes?
• How do researchers measure fidelity of implementation? 
• How does fidelity of implementation in reading interventions relate to students’ reading outcomes? 

What Did the Researchers Find?

Dosage

• Dosage was the most commonly used type of fidelity, reported in 32 studies.
• Dosage was conceptualized in many different ways, including total number of intervention minutes, average number of intervention minutes per week, cumulative number of words read, cumulative number of readings completed, percentage of readings completed per session, and total log-in time on a computer-based intervention.
• Researchers primarily used teachers’ self-reports or intervention logs to obtain data on dosage.
• The majority of studies (57%) showed no significant relation between dosage and student reading outcomes, some (32%) showed a positive relation, and a few (6%) showed a negative relation.

Adherence

• Adherence was the second most commonly used type of fidelity, with measures of adherence included in 20 studies.
• Most studies examined adherence through live observations with checklists, and a few studies used audio recordings. 
• Adherence to interventions was generally low (>80% in three studies, 60-80% in four studies, <60% in one study). Furthermore, across studies, there was variation in what researchers considered high, medium, and low fidelity.
• Forty-one percent of relations between adherence and student reading outcomes were nonsignificant, some relations were unclear (29%), some were positive (24%), and a few (3%) were negative.

Quality

• Quality was the least examined type of fidelity among the studies reviewed; measures of quality were included in six studies.
• Most quality data were collected through live observations conducted by members of research teams. Less common methods included audio recording and video observation.
• Fifty percent of relations between quality and student reading outcomes were unclear, some (30%) were negative, and a few (17%) were nonsignificant.

Adherence + Quality

• A combined measure of adherence and quality was reported in 19 studies. To measure adherence + quality, many authors had observers indicate the presence of various intervention components in instruction and then assign a score for how well the component was implemented.
• Most researchers used live observations with checklists to obtain data on adherence + quality. Others used audio recordings and field notes. 
• Measures of adherence + quality were primarily collected by members of research teams as opposed to school personnel. 
• Finally, for measures of adherence + quality, some (51%) showed no significant relation with student reading outcomes, some (41%) showed a positive relation, a few (6%) showed a negative relation, and a few (2%) were unclear.

The findings suggest that merely increasing the dosage of intervention may not consistently enhance student reading outcomes. However, the combination of adherence and quality showed possible positive associations with student reading outcomes. This combination appeared particularly beneficial in promoting reading skills for students with disabilities. 

For schools and teachers, understanding and improving intervention fidelity of implementation may lead to more effective student outcomes. Schools could use structured or standardized protocols for measuring fidelity, like dosage and adherence checklists. Fidelity data could help teachers understand which aspects of interventions to prioritize.

How Did the Researchers Find This?

The authors conducted a systematic review of 50 studies that examined the relationship between fidelity of implementation and students' reading outcomes. To be included in the review, the studies needed to:

• Be published in a peer-reviewed journal or be considered gray literature
• Focus on PK–12 students
• Examine a reading intervention delivered in a whole group, small group, or one-on-one setting
• Report effects on student reading outcomes (e.g., print awareness, phonological awareness, reading comprehension, vocabulary, or fluency)
• Measure a component of fidelity of implementation (e.g., dosage, adherence, or quality)

Researchers also took into account other variables in the studies that could affect students’ reading outcomes. These variables included:

• Study characteristics
  • Publication type (e.g., peer-reviewed article, dissertation, program report, conference proceeding)
  • Publication year
  • Type of data
  • Type of statistical analyses
• Intervention characteristics 
  • Reading focus area
  • Intervention provider (e.g., teacher, researcher, computer)
  • Student grouping (e.g., whole-group, small group, one-on-one)
  • Disability status of students
• Outcome measures 
  • Type of reading outcome (e.g., phonological awareness, fluency, comprehension) 
  • Type of measure (e.g., standardized or researcher-developed assessment)
• Fidelity measures 
  • Type of fidelity (e.g., dosage, adherence, quality)
  • Type of measure
  • Method of measurement (e.g., direct observation, self-evaluation, audio recordings)
  • Observer (e.g., teacher, researcher)

Due to differences in how studies conceptualized and measured fidelity of implementation, the researchers were unable to perform statistical analyses regarding the impacts of fidelity of implementation on students’ reading outcomes. Instead, the researchers provide a narrative overview of how these studies conceptualized and measured fidelity of implementation.

What Are the Limitations of This Paper?

This study explored a wide range of PK–12 reading interventions. The types of interventions varied significantly, from one-on-one tutoring to group-based instructional programs. Additionally, interventions focused on a wide range of reading skills, such as phonemic awareness, reading fluency, and comprehension.  This diversity means that the fidelity measures that work for one group may not be effective for another.  For example, high adherence to a reading program could yield positive outcomes for young readers but may be less effective or even yield negative outcomes for older readers. Further research is needed to determine the relationships between fidelity measures and specific types of reading interventions for different groups of students. 

In addition, some studies included in the systematic review used varying classifications for adherence and quality, without clear and consistent defining criteria. This inconsistency makes it challenging to compare results across studies and limits the ability to generalize findings. 

Terms to Know

Positive relation: If there is a positive relation between two variables, when one variable changes, the other variable also changes in the same direction. For example, if the length of intervention and reading comprehension are positively related, then when the length of reading intervention increases, student reading comprehension will also increase. 

Negative relation: If there is a negative relation between two variables, when one variable changes, the other variable also changes in the opposite direction. For example, if class size is negatively related to student participation, then when class size increases, student participation will decrease.

Nonsignificant: If a study’s findings are statistically significant, it means they are unlikely to be explained by chance alone. Conversely, if a study’s findings are nonsignificant, it is possible that they could be due to chance.

Gray literature: Gray literature refers to information produced outside of traditional publishing channels. It includes dissertations, program reports, conference proceedings, policy literature, and government documents. 

References

van Dijk, W., Lane, H. B., & Gage, N. A. (2023). How do intervention studies measure the relation between implementation fidelity and students’ reading outcomes? The Elementary School Journal, 124(1), 56-84. https://doi.org/10.1086/725672