Background: Interventions, whether simple or complex, are increasing in health care in response to the growing complexity and acuity of patient’s conditions. Monitoring the fidelity of implementing interventions is challenging. A common method to assess and monitor fidelity of intervention implementation is through a structured, reliable and valid instrument. Purpose: The purpose of this paper is to examine existing instruments measuring fidelity of intervention implementation in order to determine aspects of fidelity that have been assessed and reported on the reliability and validity of these instruments. Design: A descriptive review was conducted. Studies were included if they described and reported on the fidelity of intervention implementation instruments, their psychometric properties were published between 1980 and 2015. Methods: Data were extracted on the study characteristics, levels and aspects of fidelity and the psychometric properties, specifically the reliability and validity of the fidelity of intervention implementation instruments. Results: In total, 21 studies were included in the review. Overall results showed that some aspects and levels of fidelity of intervention implementation are included in the instruments. At the theoretical level, fidelity of intervention implementation is not accounted for majority of the studies and few explicitly reports on the use of instruments to evaluate intervention differentiation. At the operational level, interventionists’ adherence and competence are included in the instruments; however, participants’ engagement, exposure and enactment are not. The instruments demonstrate acceptable level of validity and reliability. Conclusion: Sustained focus on developing psychometrically sound instruments that account for all levels ( i.e. theoretical and operational) and aspects of fidelity of intervention implementation is imperative to strengthen the methodological literature for interventions research; and for researchers to correctly interpret research findings and to arrive at valid conclusions on the effectiveness of interventions, whether simple or complex.
Historically, intervention fidelity received little attention in intervention research because of the assumption that interventions were delivered in a standardized and consistent manner by interventionists who strictly followed the treatment protocol and manual [
Over the past three decades, there has been growing recognition of the importance to monitor and assess fidelity of implementing health interventions [
Intervention fidelity is conceptualized at two levels: theoretical and operational. At the theoretical level, intervention fidelity is related to the process of developing and designing an intervention. It refers to the correspondence between the intervention’s active ingredients and its components and activities. The active ingredients are identified in the intervention theory as the elements that characterize the intervention and are responsible for producing the changes in outcomes. The active ingredients are reflected in the components and activities comprising the intervention. At the operational level, intervention fidelity refers to the degree to which the intervention is delivered according to the original design and plan. The interventionists’ performance in delivering the intervention and the client’s exposure, engagement, and adherence to the intervention are necessary for ensuring successful implementation of its components and activities and in turn, effectiveness in instigating the desired changes [
Intervention fidelity affects the external and internal validity and statistical conclusions in intervention research [
A common strategy to monitor and assess for fidelity of intervention implementation is through a structured, valid and reliable instrument. The development and use of such instruments to assess the quality of the implementation of an intervention has been widely accepted in health intervention research. Although efforts have been made to develop instruments to assess fidelity of intervention implementation, limited research has been conducted on identifying the aspects of fidelity that are captured in these measures. In this descriptive review, we examined existing instruments measuring fidelity of intervention implementation to determine aspects of fidelity that they assess, and reported on the reliability and validity of these instruments.
Studies were included in the review if they met these criteria: original research study reporting on an instrument measuring fidelity of intervention implementation and its psychometric properties, published in the English language in peer-reviewed journals, dissertations or theses, between 1980 and 2015. The start date of 1980 provided a time period for publication of relevant papers, following the emphasis in the 1960s and 1970s on data collection, assessment, and reporting of intervention fidelity [
The databases used to identify the literature were: Health and Psychological Instruments (HAPI), Cumulative Index to Nursing and Allied Health Literature (CINAHL), Medline, Educational Resources Information Center (ERIC), Web of Science, Proquest Dissertations and Theses, and Google Scholar. The following keywords and Boolean operators were used to combine and refine the searches: (“fidelity” OR “integrity” OR “adherence” OR “implementation fidelity” OR “fidelity to treatment” OR “intervention fidelity” OR “intervention integrity” OR “intervention adherence”) AND (tool* OR instrument* OR questionnaire OR survey*) AND (valid* OR reliab*) AND (measure* OR evaluat* OR assess*). Also, reference lists of the selected articles were reviewed to identify additional publications.
Data were extracted from full papers on study characteristics, aspects of fidelity assessed by the instruments and psychometric properties of the instruments. Information presented in relevant sections of the papers was summarized and coded by the authors. Any difference was resolved through consensus. High level of agreement (> 80%) was attained.
The following information specific to the study characteristics was extracted: (a) author’s last name and year of publication; (b) discipline; (c) target population; (d) type of intervention under evaluation; and (e) number of components comprising the intervention.
Information was gathered on elements of theoretical and operational fidelity assessed and on strategies or items used to conduct the assessment. The information was derived from relevant methodological literature [
Two strategies have been proposed to examine theoretical fidelity. The first is applied when designing the intervention and consists of generating a matrix to link the intervention’s active ingredients with its components and activities; the matrix forms the basis for developing the items for measuring the implementation of the intervention’s activities [
Operational fidelity is assessed at two levels: interventionist and participant. For the interventionists, two elements of operational fidelity are examined: adherence and competence [
For participants, the elements of operational fidelity are: exposure, engagement and enactment. Exposure refers to the extent to which the participant is in contact with the intervention’s content. Exposure is often documented as the number of intervention sessions attended and duration of each session. Engagement is the extent to which the participants are involved in the intervention activities and captured through participants’ self-report and or interventionists’ observation of the activities completed during the intervention sessions (e.g. participation in group discussion). Adherence refers to the extent to which participants apply the activities or recommendations in the context of daily life such as exercising for 30 minutes five times per week [
The psychometric properties of instruments measuring fidelity of intervention implementation were evaluated using the methodological framework of Streiner and Norman (2008) [
The data pertaining to the study characteristics, identification of the aspects of fidelity and the psychometric properties of the instruments were analyzed descriptively using the Statistical Package for the Social Sciences (SPSS) Version 22.
The literature search yielded a total of 104,143 titles and abstracts (
The studies were published between 1996 and 2015. The reported disciplines included: nursing (n = 4, 19%), psychology (n = 4, 19%), psychiatry (n = 3, 14.2%), social work (n = 1, 4.8%), public health (n = 1, 4.8%), behavioral medicine (n = 1, 4.8%), rehabilitation science (specifically Occupational Therapy) (n = 2, 9.5%), education (n = 3, 14.3%), and health services (n = 2, 9.5%). The target population included: persons living with addictions (n = 4, 19%), pre-mature infants (n = 1, 4.8%), children (n = 4, 19%), older adults (n = 1, 4.8%), persons living with mental health challenges (n = 5, 24%), persons living with physical health challenges (n = 2, 10%), homeless persons (n = 1, 4.8%); couples (n = 1, 4.8%), parents of lower socioeconomic status (n = 1, 4.8%), and teachers providing social-emotional competence and behavioral support (n = 1, 4.8%). The type of interventions comprised of two or more components, such as educational (i.e. provision of information on addictions, mental health, and child safety) and behavioral (i.e. targeting behaviors related to addictions) and included: psychotherapy (n = 3, 14.3%); psycho-education (n = 1, 4.8%); cognitive behavioral therapy (CBT) (n = 2, 9.5%); emotion-focused therapy (n = 1, 4.8%); educational (n = 1, 4.8%) behavioral therapy (n = 6, 28.9%);
prevention-related (i.e. hospital related functioning decline among older adults) (n = 4, 19%); and technology (n = 3, 14.3%).
To examine theoretical fidelity, the authors of all studies (n = 21, 100%) generated a matrix to construct the items measuring fidelity. The active ingredients of the interventions were identified through various means, separately or in combination: experts, literature and review of the treatment protocol. In addition the content of the items was validated in majority of the studies (as reported in a later section). However, intervention differentiation was assessed in only three studies (14.3%).
Different elements of operational fidelity were represented in the instruments. Interventionists’ competency and adherence in delivering the intervention were most commonly assessed (n = 12, 57%). Specifically, interventionists’ adherence was measured by the respective instruments used in five studies (24%); interventionists’ general behavior was assessed in one study (4.8%); and the remaining three studies (14.3%) did not clearly indicate whether or not interventionists’ competency and adherence were measured. In the majority (n = 20, 95%) of the studies, participants’ exposure, engagement and enactment of the intervention were not accounted for in the instruments measuring the fidelity of intervention implementation. One study assessed participant’s engagement in the intervention through report by a third party and direct and indirect observations by the interventionist.
In the majority (n = 19, 91%) of the studies, the reliability of the intervention fidelity instruments was evaluated.
Internal consistency. Of the 19 studies, 13 (68.4%) reported on internal consistency of the intervention fidelity instruments using Cronbach’s alpha (α), and the remaining four did not provide empirical evidence. The Cronbach’s alpha ranged from 0.70 - 0.72 (acceptable) for nursing interventions; 0.70 - 0.99 (acceptable to excellent) for rehabilitation science interventions; 0.47 - 0.98 (unacceptable to excellent) for psychological interventions; 0.62 - 0.95 (acceptable to excellent) for psychiatry interventions; and 0.721 - 0.91 (good-excellent) for education interventions.
Inter-rater reliability. A total of 13 studies (68.4%) reported on inter-rater reliability using different coefficients. The Krippendorff’s α coefficient was 0.70 and 0.81 (good to excellent) for social work interventions. The Intra-class Correlation Coefficient ranged from 0.35 - 0.79 (unacceptable to acceptable) for psychiatry interventions; 0.71 - 0.95 (acceptable to excellent) for psychological interventions; 0.60 - 0.74 (questionable to acceptable) for nursing interventions; and 0.99 (excellent) for rehabilitation science interventions. The value of the Cohen’s Kappa coefficient was reported at 0.69 (good) for psychological interventions; 0.72 - 0.87 (good to very good) for behavioral interventions; and 0.66 - 0.96 (good to very good) for nursing interventions. The G coefficients ranged from 0.75 - 0.87 (acceptable) for rehabilitation science and education interventions. The percent of agreement ranged from 78.8% - 98% (acceptable) for nursing and education interventions.
Test re-test reliability. None of the 21 studies reported on test re-test reliability of the intervention fidelity instruments.
The majority (n = 18, 86%) of the studies reported on the validity of the intervention fidelity instruments.
Construct validity. About half of the studies (11 of 18, 61.1%) reported on construct validity of the instruments measuring fidelity of implementing psychology, psychiatry, health services, education, and rehabilitation interventions. However, not all provided empirical evidence to support the claim of construct validity. Of those that did, results demonstrated the instruments were able to discriminate between the different conditions (e.g. Cognitive Therapy and Supportive Expressive Dynamic Therapy; Twelve Step Facilitation, Clinical Management, and Cognitive Behavioral Therapy). Further, six studies (33.3%) reported on the relationships between the instruments’ subscales which captured the different aspects of fidelity with other measures of interventionists’ adherence to interventions such as Twelve Step Facilitation, Clinical Management scales; the association were relatively small in magnitude (Pearson’s r = −0.29 to −0.10; 0.23 - 0.63; 0.18 - 0.36; Spearman’s rho Correlation: rs 0.14 and 0.33).
Content validity. Most studies (10 of 18, 56%) reported on content validity of the instruments. This was done through expert judgment of the content of the items (n = 4, 22.2%). Agreement among experts was quantified in terms of the Cohen’s Kappa Coefficient (ҡ) in one study, and the Content Validity Index (CVI) in three studies. Overall, the CVIs were high (82.2% - 100%) implying the majority of the experts rated the items as relevant in capturing the key ingredients of the interventions.
This study represents a first attempt to examine and identify the aspects of fidelity that are represented in existing instruments measuring fidelity of intervention implementation and to report on the validity and reliability of these measures.
Overall, the findings of this review showed that majority of the instruments were developed to measure the fidelity of implementing a specific intervention. Although advantageous in explicitly representing the active ingredients, components and activities that characterize a particular intervention, specific measures have limited applicability to similar or different interventions [
Although there has been a proliferation of instruments for measuring fidelity of intervention implementation, the findings of this review pointed to several gaps. Theoretical fidelity was addressed primarily when developing the instruments by following a systematic process, which strengthens the validity and utility of the instruments. The process involved the identification of the active ingredients, components and activities, as well as the generation of a matrix, which was used to operationalize the intervention and guide the statement of items. However, a few of the studies explicitly reported use of the instrument to evaluate intervention differentiation. This finding may not be surprising as theoretical fidelity is often perceived as important during the stage of intervention design more so than implementation and evaluation. Assessment of intervention differentiation should be done in future research in order to determine the extent of contamination or dissemination of the intervention under evaluation to the control or comparison group, which may account for non-significant effects.
The findings of this review indicated that not all aspects of operational fidelity were captured in the instruments. Most instruments contained items that assessed interventionists’ competence or adherence, but not both. According to Hogue et al. (1996) [
The overall findings of this review have demonstrated that although some aspects of fidelity (i.e. interventionists’ adherence and competence) have been accounted for in instruments measuring fidelity of intervention implementation instruments, not all levels (i.e. theoretical) and aspects (i.e. participant engagement, exposure, and enactment) have been captured in the existing measures. Implementation of interventions, whether simple of complex, is one that requires the actions of both participants and interventionists in order to successfully attain the intervention goals and achieve the desired changes [
The instruments measuring fidelity of intervention implementation have shown acceptable levels of reliability and validity. This finding is consistent with a narrative review of generic intervention fidelity instruments for paediatric rehabilitation [
The results of this review highlight key implications for research and practice. First, the review can inform researchers and clinicians of the current reliable and valid intervention fidelity instruments to monitor and assess the implementation of theoretically similar interventions (i.e. CBT for persons with depression and anxiety). This will decrease the burden of developing new instruments [
There are several limitations that are noteworthy in this review. First, despite the extensive search strategy, only a limited number of papers that report on the development and use of instruments for measuring fidelity are found. Additional papers, published in languages other than English may have been missed. Second, there is variability in the reported reliability and validity of the instruments by researchers; some authors do not provide empirical evidence to support their claims, which limit information needed to inform researchers and clinicians intending to use the instrument of its psychometric prosperities.
There is a need for the development and use of objective, reliable and valid fidelity of intervention implementation instruments that capture all aspects of fidelity and enhance the validity of findings in interventions research [
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[
[