Association between Implementation and Effect in the Randomized Controlled Trial “Move for Well-Being in School” ()
1. Background
Physical activity (PA) has the potential to strengthen a wide range of positive dimensions related to both physical and psychological well-being among children and young people (Babic et al., 2014; Biddle, 2015; Landry & Driscoll, 2012). In addition to the long list of health benefits of PA, positive experiences with PA affect perceived physical competence (Christiansen et al., 2018)—an important aspect of overall mental health, not least for children and youth (Fox, 2000; Lubans et al., 2016). Physical capability has been assessed using the theoretical constructs of physical self-perceptions e.g. the Children’s Physical Self- Perception Profile (C-PSPP). In this paper, “physical self-worth” serves as the primary outcome measure.
School-based PA interventions can be particularly effective when multicomponent PA programs, using two or more strategies concurrently, are conducted (Demetriou & Höner, 2012; Dobbins, Husson, DeCorby, & LaRocca, 2013; Kriemler et al., 2011; Lai et al., 2014; Van Sluijs, McMinn, & Griffin, 2007). However, when evidence-informed programs are broadly implemented into school settings, the effectiveness is often disappointing (Durlak & DuPre, 2008; Van den Berg et al., 2017). To understand the discrepancy between evidence-based PA programmes and their effectiveness in real-world settings, authors often refer to unobserved implementation issues that lead to the observed program failure and lack of fidelity (Schaap, Bessems, Otten, Kremers, & Van Nassau, 2018). Several studies have addressed the challenges with implementing PA programmes in schools, identifying numerous barriers such as lack of time among, for instance, educators, lack of supportive school climate for PA and lack of available resources (Durlak & DuPre, 2008; Naylor et al., 2015).
Authors have found more than 60 variables that might influence implementation in real world settings, and employed more than 20 measures to capture several different components of implementation (Durlak, 2015). The golden standard of measuring implementation is yet to be fully agreed upon (Moore et al., 2015). At the same time, it is generally agreed upon that implementation research could be significantly improved by a more systematic approach to the use of investigative theory and frameworks (Eccles, Grimshaw, Walker, Johnston, & Pitts, 2005).
Reviews covering implementation issues in relation to school-based physical activity interventions, point out that current research to a large degree has focused on measuring effectiveness. Whereas factors that are related to the implementation of programmes or the effects of the implementation on primary and/or secondary effect measures, have been researched only sporadically (Domitrovich & Greenberg, 2000; McGoey, Root, Bruner, & Law, 2015, 2016). While there is a growing recognition of the need to conduct comprehensive process evaluations, and a growing number of studies focusing on exactly process aspects, the bulk of studies focus on programme effectiveness (Gray, Contento, & Koch, 2015; Naylor et al., 2015; Schaap et al., 2018; Van Sluijs et al., 2007).
The studies that address the correlation between implementation and primary outcome are, with few exceptions, efficacy studies (Naylor et al., 2015; Schaap et al., 2018). The general lack of such studies to successfully translate into real-world settings could, as described by Domitrovich and colleagues, be related to the fidelity and quality by which the intervention is implemented—which again is related to the possible effectiveness of the intervention (Domitrovich et al., 2008).
Reviews on school-based PA interventions suggest a rather consistent correlation between implementation and the effectiveness on primary intervention outcome (Durlak, 2015; Naylor et al., 2015; Quested, Ntoumanis, Thøgersen-Ntoumani, Hagger, & Hancox, 2017). In short, school-based PA interventions should not just be informed by the best available evidence regarding efficacy and/or effectiveness, they also need to be well-implemented to truly make a significant impact (Durlak, 2015).
In order to construct evidence applicable to the school setting it is important to evaluate the feasibility of the intervention as well as fidelity. Measuring implementation fidelity in relation to complex interventions is challenging (Craig et al., 2008; Moore et al., 2015). The intricate work to link implementation and effectiveness, especially in complex settings with multicomponent interventions, may explain the lack of research within the area (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Naylor et al., 2015).
Therefore, the aim of this paper is to assess the association between the degree of implementation of a multicomponent PA intervention and effectiveness on students’ physical self-worth in the RCT intervention Move for Well-being in School. Furthermore, we want to contribute to the limited research regarding implementation scores and discuss how these can be associated with intervention effects.
2. Methods
2.1. Study Design, Setting and Participants
This study looks at the association between implementation—measured through survey among educators—and change in students’ physical self-worth. The Move for Well-being in School (MWS) study used a cluster randomized controlled study design applying the RE-AIM framework as an overall framework to guide implementation and evaluation (Smedegaard, Christiansen, Lund-Cramer, Bredahl, & Skovgaard, 2016). The framework elements were furthermore guided by the work of Nayler et al. and Durlak and Dupree (Durlak & DuPre, 2008; Naylor et al., 2015). In 2015, 11 municipalities in the Central Denmark Region and Region of Southern Denmark were selected based on maximizing geographic spread, the difference in size and allocated budget for public schools and invited to participate in the study (Smedegaard et al., 2016). Seven municipalities consented and either contacted schools or allowed the research team to invite schools. Out of 126 possible schools in the 7 municipalities, 24 schools agreed to participate and were randomized taking into account school size, district socio-economic status and an even distribution of schools from each municipality in the intervention group and the control group. In the current study, the analyses are based on the 12 intervention schools that participated in the 1-school year interventions period. The enrolment procedure and the study design has been described in detail elsewhere (Smedegaard et al., 2016) as well as previously papers has addressed results from the MWS project (Christiansen et al., 2018; Christiansen, Clausen, Smedegaard, & Skovgaard, 2019; Holt, Smedegaard, Pawlowski, Skovgaard, & Christiansen, 2018; Smedegaard, Brondeel, Christiansen, & Skovgaard, 2017). A total of 148 educators were involved in the implementation of the MWS program during the school year of which 48 were PE teachers and 1485 students from 4th, 5th and 6th grade (10 - 13 years old) attended the 12 intervention schools at baseline.
The Move for Well-being in School study was notified and approved by the Danish Data Protection Agency (J.nr: 2014-54-0693) and the Danish Health Research Ethics Committee.
2.2. Intervention
To ensure high quality and ensure a socially inclusive PA environment, the approach to the physical activity intervention in the MWS PA program was grounded in Self-Determination theory and designed to target three innate psychological needs: competence, autonomy and relatedness in order to improve motivation for physical activity for all students (Ryan & Deci, 2000).
The intervention program consisted of initiatives targeting three settings for school-based physical activity: 1) Physical Education (PE) Classes; 2) In-class activities; 3) Recess activities. During PE classes, a minimum of 6 out of 8 specially designed PE courses each lasting 4 × 90 min were to be conducted throughout the school year. Central features across all PE courses included teamwork and individual competence development rather than competition. In-class activities consisted of minimum two brain breaks per day lasting at least 5 minutes, i.e. short physical activities briefly interrupting the regular courses. The in-class activities covered four different topics, dependent on the purpose chosen by the educators. The purpose could be a social focus, generate high energy/pulse, create relaxing/physical well-being, or stimulate coordination. For the recess activities, each school was provided with a bag of different sports equipment and toys to be used to initiate a variety of activities supported by teachers and teacher assistants during recess minimum three times 30 min/week. Furthermore, the schools were encouraged to introduce activity-enhancing initiatives such as opening up in-door areas for activities during rainy days, make rules for limiting use of electronic devices during recess etc. Lastly, three theme days, spread out during the school year, were to be conducted. The aim was to engage students in the development of activities and to initiate joint discussions on physical activity and general well-being in school (Smedegaard et al., 2016).
The deliverers of the intervention, teachers and teacher assistants teaching 4th to 6th grade, were equipped with a Tailored Activity Program, including educational materials, planning guides and course plans for incorporating PA throughout the school day. The Activity program was supported by a Competency Development Program consisting of four full-day workshops focusing on the underlying theoretical approach and opportunities to try out core activities in practice. Participation and teacher preparedness to deliver the intervention as a result of the Competence Development Program have been address elsewhere (Smedegaard et al., 2017). At each school, a local appointed Coordination Group guided and motivated the implementation of the intervention program receiving frequent support from the research team via information e-mails and local supervision visits throughout the intervention period (Smedegaard et al., 2016).
2.3. Data Collection, Aggregation and Analyses
2.3.1. Student Data
A total of 1485 students from 4th, 5th and 6th grade (10 - 13 years old) attended the 12 intervention schools at baseline. The students completed an online survey during school hours in the beginning of the school year (August 2015, baseline) and repeated nine months later at the end of the school year (May or June 2016, follow-up).
The outcome variable, physical self-worth was measured using the Children’s Physical Self-Perception Profile (Fox & Corbin, 1989; Whitehead, 1995). PSW was calculated as the mean of six items with four categories (1 indicating the lowest level and 4 the highest).
To be included in the analyses students had to provide data on PSW for at either baseline or follow-up. This resulted in a final dataset with 2331 observations (baseline or follow up) from 1269 students in 57 classes at 12 schools.
2.3.2. Educator Data
A total of 148 educators were involved in the implementation of the MWS program during the school year of which 48 were PE teachers. The average number of involved educators per school was 12 (range 5 - 17), and on average three of these were physical education teachers (range 1 - 6).
Educators were asked to complete an online questionnaire two months (T1), five months (T2) and nine months (T3) into the program period (Smedegaard et al., 2017). At T1, 141 educators answered of whom 42 were PE teachers, at T2, 135 educators answered of whom 48 were PE teachers and at T3, 139 educators answered of whom 39 were PE teachers.
The educator survey was used to evaluate the implementation of the intervention components at class level: in-class activities, PE-courses and recess activities. The aggregation and matching of data from educator level to student level is presented in Table 1.
The brain breaks were reported for each class in the educator questionnaire. In case there was more than one educator that reported on the same class, the sum was calculated. The brain break variable include missing values for the students in one class, because no one reported brain breaks here (n = 24 students).
The implementation of intervention specific PE-courses was assessed using one question concerning the proportion of taught lessons on a 5-point Likert-scale from none to all. The answers were grade specific and if several PE-teachers taught within the same grade the average was calculated. The PE variable includes missing values for eight classes (n = 171 students).
The implementation level of recess activities was assessed by the average of two items capturing number of days (0 - 5 days) with planning in individual classes before recess and with initiatives during recess for more classes in the schoolyard. Regarding the planning question, we identified in which classes the educators had taught from their response on brain breaks. If an educator were attached to more than one class, number of educator planning days was divided to the number of classes. Finally, if more educators were identified planning for the same class they were summed. Regarding initiatives during recess, it was assessed
![]()
Table 1. Questions, level and time points for educator survey.
at overall school level. Number of days at class level was found by averaging of all educators’ days divided by number of classes. Finally, the two calculated recess questions were summed to one total score at class level.
The association between implementation and PSW were analysed with a multilevel linear regression, including four levels of clustered observations (i.e. observations (baseline or follow-up) nested within students, classes, and schools), 4 variables (i.e. time and the three dose variables), and 3 interaction terms (i.e. time by the three dose variables) to estimate the evolution over time by different levels of intervention dose. The multilevel linear regressions model was estimated in R (version 3.2.2) using the “lme4” packages (Bates, Mächler, Bolker, & Walker, 2015). To help the interpretation of the regression coefficients, plots were provided for each dose variable. The plots depict the change of PSW (y-axis) over time (x-axis) for 3 levels of dose representative of the distribution of the doses: the mean and the mean ± 1 standard deviation (SD).
3. Results
Descriptive baseline characteristics are presented in Table 2. Students provided information on gender, primary language spoken at home, and family social class in three categories (high, medium, low) (Smedegaard et al., 2016). There was an equal distribution between boys and girls with a small overrepresentation of fourth graders compared to 5th and 6th graders. With respect to social class, the study sample is comparable with the Danish population as a whole (Smedegaard et al., 2017).
Table 3 presents the implementations scores for Brain Breaks, Physical Education and Recess, respectively. According to the implementation protocol, 10 brain breaks per week per class were the target. The results show an average of 5.9 brain breaks being delivered. There was, however, large variation from one to 19 brain breaks. For PE, an implementation score at 3.4 equals approximately half of the total lessons, which matches the implementation protocol. Finally, a score on 0.8 day a week where educators initiate or help plan recess activities is
![]()
Table 2. Baseline characteristics of participants included in the analyses.
quite lower than the three days described in the MWS protocol. We found low correlation scores between the components using Pearsons correlation, In-class vs PE (0.009), In-class vs Recess (0.313), PE vs Recess (0.197).
Table 4 presents the results of the multilevel linear regression for each of the three intervention components and their interaction with the time effects. The interaction terms indicate if a change in PSW is associated with the implementation score. Figures 1-3 visualize the results from Table 4 at mean and ±1 SD of the actual implementation dose.
The results show that there are no significant association between PSW and brain break implementation at neither baseline nor follow-up. Figure 1 indicates that classes receiving the largest amount of BB´s are made up of students with an insignificant lower PSW at baseline. This difference increased insignificantly at follow-up.
For PE there were a significant association at baseline between PSW and implementation. At follow-up, this difference between classes were slightly diminished (Figure 2). The classes receiving most MWS lessons are those with the highest average PSW at both baseline and follow-up.
Finally, results show no significant association between PSW and recess
![]()
Table 3. Descriptive values dose parameters for intervention group.
n = 1269.
![]()
Table 4. Multilevel linear regression for the intervention components
Multilevel linear regressions with 4 levels: 2331 observations nested within 1269 students, within 57 classes, within 12 schools; Time: difference between baseline and follow-up measure; PA: physical activity; *p < 0.05; **p < 0.01; ***p < 0.001.
implementation at baseline, but a significant positive interaction is found between time and implementation. This indicates that students who more frequently interacted with educators, who initiated and/or assisted in planning recess, significantly increased PSW.
Figures 1-3 Differential evolution in the outcome variable (PSW: Physical Self Worth) given for Brain Breaks, Physical Education and Recess, +/− 1 SD of the actual implementation dose.
4. Discussion
The aim of this study was to assess the association betweenimplementation of the Move for Well-being in School intervention and effect on students’ physical self-worth. Implementation was assessed for the three primary intervention components at class level. The three components were implemented at all schools to a variating degree. PE and brain breaks were implemented closest to the protocol, while the aimed at implementation level for recess was low.Such a pattern is not uncommon. In some studies, the range has been reported as high as 87% when comparing the lowest and highest implementation levels within the same studies, and 20% to 40% differences between providers e.g. educators or sites e.g. schools are not uncommon (Durlak & DuPre, 2008).
The regression analyses showed a significant association between PSW and implementation for PE at baseline and for recess at follow-up. The significant association between the degree of PE implementation and PSW at baseline could, in our opinion, have several causes. Firstly, implementation might have been easier in classes with more physical confident students. Some of the PE lesson plans, provided to the intervention schools, put higher demands on both teachers and students, which could have been a challenge in classes with overall lower physical confident students. Secondly, the difference in PSW at baseline may also be due to previously different PE practises, where teachers, more prone to implement, also had more focus on promoting PSW for all students in the classes before the intervention.
The significant interaction between recess implementation and time was the only intervention component, showing a relationship between implementation and effect. The explanation here according to the protocol is that the positive adult involvement in recess has induced new activities and social interaction, which have promoted PSW. The effort may have engaged students with lower motivation and self-worth to participate in different physical activities during recess. The recess component was the most difficult for schools to implement. At the same time it was very popular among students (Holt et al., 2018). While brain breaks and PE are mandatory and educator controlled, recess is largely students’ free time. In Denmark, educators are mainly passive observers during recess, and active engagement is rarely seen in 4th to 6th grade. This freedom is enjoyed by many students, which appreciate recess and the opportunities for self-organized activities. At the same time, a considerable number of students are challenged by the ‘freedom of recess’ and are either not invited into activities or do not seek an invitation. This group of children are supported by positive adult engagement during recess to achieve positive physical experience (Chris- tiansen et al., 2018; Holt et al., 2018; LadeKjær & Hjarbech, 2016). Whereas the educators generally found it difficult to intervene during recess due to time constraints, the challenges for in-class activities revolved around spending time away from the academic subject. Two reasons why adoption in PE might be less challenging are firstly the fact that PE teachers are PA experts; and secondly that the MWS PE-lesson should be conducted anyway and therefore the plans are seen as a help and not as extra work.
4.1. Implementation Scores in Multicomponent Interventions
We chose to assess the implementation for each of the primary components adjusted for the other two components. The first reason for doing so was that we found a low correlation between implementation in the three areas. Secondly, if all three components were compiled in one total implementation score we were not able to assess if the components had different associations with PSW. Another choice in this study was to keep the number of variables low in the implementation score. We limited ourselves to questions regarding dose delivered assessed via educator questionnaires. Other approaches in the implementation literature have included data from parents, students, school management (Askell-Williams, Dix, Lawson, & Slee, 2013; Saunders, Ward, Felton, Dowda, & Pate, 2006; Van Nassau et al., 2016). In the DOiT project, van Nassau and colleagues’ assessed 44 different components to form an implementation index score ranging from 8 - 31, related to the degree by which a curriculum-based PA intervention affected adolescents’ adiposity and energy balance-related behaviours (van Nassau et al., 2016). The DOiT authors conclude that their index is still experimental and due to limited numbers of schools in the project, the explanatory power is questionable. The KidsMatter project identified 37 different elements collectively scored between 37 - 226 based on survey data, to address the implementation index to mental health issues at approximately 100 Australian schools (Askell-Williams et al., 2013). The 37 elements were scored on a scale between 1 - 7 based on the theoretical model from Domitrovich et al. (2008). They found a weak correlation between the implementation scores and students well-being (Askell-Williams et al., 2013). The LEAP study classified schools as having either ‘high’ or ‘low’ implementation levels related to the participation of girls in vigorous PA activity at school (Saunders et al., 2006). The differentiation was based on a mixed-methods approach and was significantly associated with the girls’ participation in vigorous PA activity at school (Saunders et al., 2006).
Summing up, the literature on implementation scores to be associated with intervention effects offers different strengths and weaknesses. A score including many variables have the strength to make it possible to consider many aspects, but it can afterwards be a challenge to identify particular important implementation factors. A score using mixed methods can to a higher degree assess quality of implementation but faces the risk of different assessment criteria and interrater differences. Finally, the approach in the current study offers a specific assessment of dose delivered in the respective components, but do not take quality or other aspects into account.
4.2. Validity of the Implementation Score
In the current study, a number of methodological issues related to the validity of the implementation score were experienced. The main challenge in creating the class based score, was the matching of the educators actions to the individual classes. Implementation of brain break was assessed at individual class level and PE variables at the grade level. The recess assessment was not reported class or grade specific and we therefore used the brain breaks variable to match the questions regarding planning recess activities. This is an assumption, which can be discussed. The second recess question regarding initiatives during recess, we averaged all educators’ activities per class. This could potential underestimate the true number of days with educator involvement during recess, because these activities can involve several classes and the protocol did not state the number of active teachers during recess. If three educators initiated activities during recess at three different days at one school with six classes, the average per class would be 0.5, while the true number of days would be three. So even though we used a simple implementation score based on primarily dose delivered, it was not straight forward and several assumptions can be discussed. Especially, regarding the alignment of dose delivered and dose received at class level.
In our approach, we chose separate implementation scores for each component in the same regression model. This allowed us to assess the effect of each component on PSW and at the same time adjusting for the implementation of the other two components. A potential problem with this approach is that we do not take potential synergetic effects between components into consideration e.g. if a higher rate of implementation in more components is more effective than the single components alone. This aspect of multiple components design in school interventions and the issues around assessing proposed synergistically effects is a known complex challenge for process evaluations and implementation studies (Lloyd et al., 2017).
Implementation of school based PA interventions is complex in nature even though the intervention itself may seem simple. Documenting the implementation process and evaluating the association with primary outcomes is crucial to improve school-based interventions. In this study, we used a quantitative approach, computing a numeric score based on educator surveys. However, we recognize that this is a rather simple way of evaluating a complex intervention. In a previously published paper, we qualitatively evaluated the intervention process (Holt et al., 2018). Furthermore, in a separate published paper, we have discussed a range of implementation issues related to the MWS project (Smedegaard et al., 2017). These prior reporting’s gave valuable insight into the project black box and what happened at the schools. More studies on implementation process and impacts will contribute to a better understanding of possible links between implementation and primary outcome variables (Gray et al., 2015).
5. Conclusion
The aim of this paper was to assess the degree of implementation of a multicomponent physical activity intervention in order to explore the association between implementation and effect on PSW. Three components were implemented at all schools to a varying degree. For PE, half of the performed lessons were from the program, which was in line with the protocol aim. Classes with a higher PE implementation had a significantly higher PSW at baseline. An average of 5.8 brain breaks was implemented per class per week, and no significant differences were established between implementation and change in PSW. Recess implementation was the hardest for most schools to implement, but schools with a high level of recess implementation had a significant increase in PSW at follow-up. This indicates that if schools and teacher prioritize recess, help students plan, and initiate inclusive activities it may be possible to significantly increase PSW.
This paper explores an important aspect of school interventions, showing that implementation varies both at school and class level and that this potentially has an effect on outcome measures. Furthermore, the paper discusses several methodological difficulties in linking implementation and effects in multicomponent school-based activity interventions. This particular area of research needs further development to increase the understanding of how physical activity-related interventions are implemented and how this affects outcomes.
Acknowledgements
Pernille Lund-Cramer & Anne-Didde Holt, scientific assistants, for their participation in the project, data collection, distribution and daily sparring.
The authors would also like to thank the schools, educators and children involved.
Funding
The study was funded by a donation from the non-profit foundation TrygFonden, Denmark. The funding body has no role or authority in the conduct of the research project, except acceptance of adjustments to the research plan.
Availability of Data and Materials
The datasets collected and/or analysed during the current study are available from the corresponding authors upon reasonable request.
Authors’ Contributions
SS initiated this paper and wrote the draft. All of the authors contributed to writing the manuscript and critically reviewing its content. All authors approved the final version before submission.
Ethics Approval and Consent to Participate
Prior to the data collection, students and their parents received information about the nature and procedure of the study. Consent took the form of written passive consent from the parents and both written and oral passive consent from the students. It was emphasized that participation was voluntary, that respondents could withdraw consent at any time, and that data would be treated confidentially and anonymously. These points were also emphasized at the beginning of each questionnaire session. All data are stored and treated in accordance with Danish law for data protection, and the project is notified and approved by the Danish Data Protection Agency (J.nr: 2014-54-0693) and the Danish Health Research Ethics Committee (De Videnskabsetiske Komitéer for Region Syddanmark) and is on the ISRCTN registry (DOI https://www.isrctn.com/ISRCTN12496336).
Abbreviations
BB: Brain Breaks
MWS: Move for Wellbeing in School
PA: Physical Activity
PE: Physical Education
PSW: Physical Self-Worth
RCT: Randomized Controlled Trial
RE: Recess