Evaluation of Clinician Training in Autism Screening, Care Management, and Patient Education ()
1. Introduction
In 2021, researchers found that less than half of American physicians reported feeling confident in caring for patients with disability [1] . This is concerning given the high number of patients with disabilities [1] combined with the increasing prevalence of diagnoses such as autism [2] . Care for autistic patients often start with screening and diagnosis, which while recommended to occur between the ages of 18 - 24 months [3] , typically do not occur until 36 months or beyond [2] . Early diagnosis paired with timely access to services contributed to positive outcomes [4] . Autism diagnosis at a younger age is associated with better cognitive and language outcomes and decreased need of ongoing support compared to diagnosis at a later age [4] . Moreover, early intensive behavioral intervention for autistic children was found to positively impact intellectual quotient (IQ) and adaptive behaviors [5] .
Unfortunately, developmental evaluations and accessing long-term services (e.g., behavior therapy) often involve multiple time-consuming steps [6] . This process could be challenging for families to navigate. Research has shown that sociodemographic factors such as low socioeconomic status, belonging to racial/ethnic minority groups, and limited English proficiency are associated with delayed autism diagnosis [7] , though the latest Centers for Disease Control and Prevention data show a trend toward addressing this disparity [2] . Delayed diagnoses can be problematic as early support has been shown to contribute to positive outcomes [8] .
Emerging evidence suggests that greater investment is needed to increase the number of clinicians trained and skilled in developmental care [9] . Clinician shortage in pediatric subspecialists has been an issue for more than a decade [10] but the recent COVID-19 pandemic exacerbated barriers including poor coordination of service delivery and reduced allocation of time, financial resources, and staff [11] . Although barriers to care outside of individual clinicians’ control exist, equipping primary care clinicians (PCCs) with the ability to streamline the initial steps of an evaluation as well as counseling families about autism and the diagnostic journey could be an essential part of improving developmental care. Clinician training in autism-related care can positively impact patient outcomes and healthcare delivery by increasing access to time-sensitive developmental services to maximize benefits of early support.
Boosting Capacity to Screen and Care for Autistic Children ECHO Program (BCAEP)
Boosting Capacity to Screen and Care for Autistic Children ECHO Program (BCAEP) was developed based on the international Extension for Community Healthcare Outcomes (ECHO) [12] model to boost capacity of PCCs to deliver developmental services to youth in a Boston-area community-based health network of six health centers and a large safety-net medical center. This training was delivered to 47 clinicians and included topics under the following themes: 1) screening and diagnosis (e.g., diagnostic criteria and differential diagnosis), 2) care management and patient education, 3) patient experience (e.g., sleep challenges, co-occurring medical conditions), and 4) non-medical services (e.g., special education, support for transitioning from pediatric to adult services). Training topics for each of the 12 sessions are available in Appendix 1. The ECHO model has demonstrated effectiveness at improving knowledge, skills, and self-efficacy generally and within autism care [13] [14] [15] [16] .
BCAEP was led by a senior developmental and behavioral pediatrician with 30 years of experience and a developmental and behavioral pediatric advanced practice clinician with 19 years of experience. Attendees included pediatric care professionals (Table 1) from six community health centers and one hospital-based pediatric practice in the Boston-area. Twelve monthly 60-minute virtual
![]()
Table 1. Participant characteristics at pre-test (November 2020, n = 30), mid-point (May 2021, n = 19), and post-test (October 2021, n = 17).
1Other includes clinical social work, med-peds, and trauma.
sessions were delivered via Zoom during the COVID-19 pandemic from November 2020 to October 2021. Each session consisted of didactic and case discussion components. Didactic presentations were delivered live by two facilitators leading and answering questions. Per the ECHO model, a clinician presented a real de-identified patient case, which was discussed by all attendees [12] . Attendees could claim continuing medical education credit (CME); 212 credits were claimed by 32 clinicians.
Although well-established [17] , few ECHO autism program evaluations using mixed-methods approaches exist [18] , as studies used mainly quantitative pre-test, post-test design [15] [16] [19] - [24] . Conducting a mixed-methods evaluation that requires use of qualitative data collection strategies to put the quantitative findings in context can result in actionable recommendations for sustainability and replication [25] [26] . The current evaluation employed mixed methods to examine training program delivery and effectiveness in increasing clinicians’ capacity in autism screening, care management, and patient education. Additionally, burnout and professional fulfillment data were collected, being particularly relevant given the unique context in which training took place (i.e., COVID-19 pandemic).
2. Methods
The mixed-methods evaluation was guided by a logic model. Data collection and analysis were guided by the Normalization Process Theory (NPT), an implementation science framework that describes how a new practice becomes incorporated into an existing workflow [27] . NPT posits that implementation occurs through four constructs: coherence or participants’ sense-making of the new practice; cognitive participation or identification of components needed for the new practice; collective action or participants’ motivation and action steps needed to implement the new practice; and, reflexive working or participants’ assessment of the new practice [27] . A pre/post design without a control group was employed in the current evaluation. Quantitative data on participants’ self-reported ability and self-efficacy on autism care, as well as burnout and professional fulfillment, were collected before (pre-test), after the sixth session (midpoint), and immediately after the twelfth and last training session (post-test). Qualitative data on participants’ feedback about the training were collected at follow-up, approximately six months post-training. The evaluation was approved as exempt by the Boston University Medical Campus and Boston Medical Center Institutional Review Board (H-40718).
2.1. Surveys
Outcomes were assessed via a pre-test/post-test evaluation design through surveys of BCAEP clinicians. Surveys were administered via Qualtrics, took approximately 15 minutes to complete, and were voluntary. Surveys were sent before receiving training (pre-test, November 2020), after the sixth session (mid-point, April 2021), and immediately after the twelfth and final session (post-test, October 2021). Survey links were emailed to the 47 clinicians. Surveys consisted of the following components: 1) self-reported abilities, 2) self-efficacy to conduct autism screening, care management, and patient education, 3) burnout, and 4) professional fulfillment [28] . For the self-reported abilities and self-efficacy component, agreement with statements were assessed using a five-point Likert scale, with the scale ranging from strongest disagreement to strongest agreement or not important to extremely important, depending on the question. For the burnout survey, higher scores indicated more alignment with the statement on burnout and could be interpreted as feeling more pressure/burnt out.
2.2. Focus Group
Survey results combined with NPT [27] framework informed the focus group guide designed to contextualize quantitative findings. Qualitative data collection procedures are organized and presented using the Standards for Reporting Qualitative Research (SRQR) checklist [29] . Clinicians were invited via email to focus groups/interviews, which were conducted over three weeks in March 2021, approximately five months after program completion. Each focus group/interview lasted about 60 minutes. Five sessions were conducted virtually via Zoom and one session was conducted in-person. Additional details about the qualitative analysis are available elsewhere [30] .
2.3. Data Analysis
The final survey dataset contained 30 pre-test responses, 19 mid-point responses, and 17 post-test responses. Data were cleaned, and the pre-test and post-test surveys were matched on last four digits of the cell phone number, resulting in nine matched responses between pre-test and post-test. Analyses used the Wilcoxon test for unmatched pre-test/post-test data and the signed-rank analytic test for the matched pre-test/post-test data (Table 2 and Table 3).
![]()
Table 2. Clinician self-rated skill acquisition and self-efficacy at pre-test (November 2020, n = 30), mid-point (May 2021, n = 19), and post-test (October 2021, n = 17).
*p < 0.05. 1Wilcoxon test was conducted on unmatched samples in SAS. 2ASD (autism spectrum disorder).
![]()
Table 3. Responses to burnout and professional fulfillment survey [28] in participants at pre-program and post-program.
1Responses were analyzed in SAS using Wilcoxon test (unmatched sample). 2Higher mean scores in the burnout section of the survey indicate more alignment with the statement-scores that are higher can be interpreted as feeling more pressure/burnout than lower scores. 3Pre-program surveys were administered in November 2020. 4Post-program surveys were administered in October 2021.
Focus groups and interviews were audio-recorded, transcribed, de-identified, and verified by two authors. Transcripts were deductively coded [31] using all NPT [27] constructs and content analysis [32] was performed to categorize qualitative findings (Table 4, Table 5). Qualitative analysis was conducted using NVivo12 software [33] . Transcripts were independently coded by two authors who met and resolved differences via consensus; a third author served as tiebreaker.
3. Results
3.1. Quantitative Findings
3.1.1. Survey Respondent Characteristics
At pre-test, 30 participants represented behavioral health staff (13.33%, n = 4), nurse (6.67%, n = 2), nurse practitioner (13.33%, n = 4), physician (63.33%, n = 19), and medical support staff (3.33%, n = 1) (Table 1). Participants specialized in developmental and behavioral pediatrics (3.33%), family medicine (23.33%, n = 7), pediatrics (60%, n = 18), psychiatry (3.33%, n = 1), and other including clinical social work, med-peds, and trauma (10%, n = 3). Most participants (66.06%, n = 16) have between three to ten years of experience in practice.
3.1.2. Self-Reported Ability
Participants were assessed via a five-point Likert scale on self-reported ability related to autism screening, care management, and patient education (Table 2). Participants reported a significant increase in their perceived ability to administer level 1 autism screeners from 43.33% (n = 11) reporting above average ability or very able at pre-test compared to 52.94% (n = 15) at post-test (p = 0.0029). Participants reported a significant increase in their perceived ability to counsel families from 20% (n = 6) reporting average ability or very able at pre-test compared to 58.82% (n = 10) at post-test (p = 0.0065).
![]()
Table 4. The Standards for Reporting Qualitative Research (SRQR) 21-item checklist [29] .
a. Sample of a Table footnote (Table footnote is dispensable).
![]()
Table 5. Qualitative findings from focus groups and interviews with primary care clinicians organized using Normalization Process Theory [27] .
The largest increase from pre-test to post-test was self-reported ability to manage treatment of autistic patients with 13.33% (n = 4) of participants reported having above average ability or very able at pre-test compared to 82.36% (n = 14) at post-test (p < 0.0001). In the subsample of participants that matched from pre-test to post-test (n = 9), there was a statistically significant increase in self-rated ability to counsel families about autism (pre-test mean = 3.00, post-test mean = 3.89, p = 0.0313) and manage autistic patients’ care (pre-test mean = 2.56, post-test mean = 4.11, p = 0.0078).
3.1.3. Self-Efficacy
Participants expressed higher agreement with some self-efficacy statements at mid-point (n = 18) and more so at post-test (n = 17) compared to pre-test (n = 30), indicating increasing levels of confidence as the training progressed (Table 2). For example, more participants reported feeling confident about educating their patients about autism at post-test compared to pre-test. A majority of participants (70.59%, n = 12) strongly agreed at post-test that they felt confident educating patients about autism compared to 3.33% (n = 1) at pre-test (p = 0.0019). More than half (64.71%, n = 11) of participants also strongly agreed to feeling confident counseling patients on available treatments and realistic treatment expectations at post-test compared to 3.33% (n = 1) at pre-test (p = 0.0002).
Additionally, more participants reported feeling confident about caring for autistic patients at post-test compared to pre-test including feeling confident managing the primary care needs of autistic patients, feeling confident interpreting results of autism screening, and understanding the concerns of early intervention. Although there were increases in unmatched data, they were not statistically significant in the matched subsample (n = 9).
At post-test, participants reflected on how the training changed their practice and behaviors with many participants feeling confident in the processes, protocols, and workflows related to autism care of patients. For example, participants indicated feeling confident making referrals, observing behaviors, identifying children on the autism spectrum, and using the word “autism” in conversations. Many of these statements were further contextualized through the interviews and focus groups [30] .
3.1.4. Burnout and Professional Fulfillment
Unmatched analyses indicated changes in clinicians’ reporting of feeling sensitive to others’ feelings/emotions, interest in talking with their patients, and connected with their patients and colleagues (Table 3). In matched analyses (n = 9), clinicians reported at pre-program versus post-program feeling more empathetic with their patients (2.11 and 1.22, respectively, p = 0.0156); feeling more empathetic with their colleagues (2.33 and 1.22, respectively, p = 0.0391); and being more sensitive to others’ feelings or emotions (2.11 and 1.22, respectively, p = 0.0313). There were found no statistically significant change in professional fulfillment in both matched and unmatched analyses.
3.2. Qualitative Findings
3.2.1. Focus Group and Interview Participant Characteristics
Ten clinicians participated in the focus groups and nine completed the pre-focus group survey to assess professional characteristics of participants (i.e., role/title, medical specialty, years of experience, number of sessions attended). Participants represented physicians (44.44%, n = 4), nurse practitioners (44.44%, n = 4), administrative leadership (11.11%, n = 1), and specialties including pediatrics (77.78%, n = 7), psychiatry (11.11%, n = 1), and integrated behavior health (11.11%, n = 1). One-third (n = 3) of participants had 3 - 5 years of experience in practice and 55.55% (n = 5) had more than 10 years of experience. On average, participants attended ten out of 12 sessions. Themes that emerged from interviews and focus groups were organized according to NPT [27] constructs. We present key themes and illustrative quotes to highlight participant feedback and provide context to the surveys (Table 5).
3.2.2. Construct 1: Coherence
Coherence occurs when participants make sense of a new practice, why it is important, and how it fits into their practice [27] . Participants were highly motivated to engage in BCAEP because they had a previously established interest in autism, and some had noticed increasing numbers of their patients presenting with symptoms of autism but needed enhanced skills to feel confident working with these patients. According to one participant who had “been in practice for 25 years”, they noticed an increase in autistic patients in their practice “and… I’m talking the ones that are so obvious” [30] .
Such increase in developmental services also highlights the importance of collaboration. Participants noted the importance of engaging in bidirectional knowledge transfer with their colleagues. This internal motivation served as the driving force behind participants’ reflection on how their individual roles in the clinic can support the autism screening and follow-up and how they could improve existing procedures or start implementing new ones. One participant shared paying more attention to “…signs we might want to be noticing… so, for me, this is about how do I take that information from the trainings and put it into some practice” [30] .
3.2.3. Construct 2: Cognitive Participation
The construct cognitive participation [27] requires a focus on identifying the problem, working to solve the problem, and sustaining the work. Participants described components of their work that need to change in order to improve their individual practice as well as their clinic as a team. Participants reflected on their confidence regarding the extent to which they had the tools and expertise necessary to create change. For example, one participant “… did start doing CARS evaluations after the [BCAEP] started, partially because I just had more confidence…” [30] .
Participants fell into two categories: clinicians who were familiar with autism screening and those who were newer to the autism evaluations. Some had been using these tools for years while others just started to use the tools within their practice. For example, one participant had “trained in the CARS as a fellow, probably, you know, actually performed it with some small [number of patients], and then I was in the real-world practicing and using it…” [30] . Case discussions with other clinicians and presentations from knowledgeable leaders in the field during BCAEP provided participants who were newer to screening tools with more confidence.
3.2.3. Construct 3: Collective Action
The third step is collective action [27] , which refers to the action steps needed to implement new practices participants learned from BCAEP. For example, interacting with systems already in place (e.g., electronic medical record, special education system), barriers (e.g., COVID-19 pandemic, lack of services availability), and allocation of time, resources, and personnel responsibilities.
The biggest barriers were long wait times to get patients evaluated by a specialist and logistical complications resulting from the COVID-19 pandemic. Participants expressed wanting to expand these skills to PCCs to make evaluations more time-efficient and to ease the burden on the smaller group of individuals currently performing screenings. For example, dedicating “[even] four to six hours a week to just do… the screening diagnostic” had been helpful in getting patients seen by specialists more quickly [30] . Moreover, there needs to be infrastructure to support changes made to protocols within the clinic. For instance, one participant described the steps required to enter screening results into the electronic medical record, “We’re in kind of like a paper world. So, you get the paper screener… and then it’s put into the computer, or you can yourself put it in or have your medical assistant [or] whoever. Sometimes there are lags or drops in [that] process. I think the ultimate goal and what most clinics should be really doing in this day and age is… everything should… just [be inputted] directly into the computer….” [30] .
The process from screening, diagnosis, and services receipt involves many steps. It requires concrete systems of accountability and strong communication between everyone involved. One participant described that they “would love someone to walk [families] through this [Applied Behavior Analysis or ABA] process” because families may get “stuck at paperwork which is… the very beginning step” [30] .
Lastly, participants discussed collaborations with other clinicians. Since before the COVID-19 pandemic, there were “little cross collaboration there in pediatrics” [30] . During the pandemic, connection with other clinicians was even more important. One participant emphasized that it is “… important [for nurse practitioners to] stay connected... especially because we are going to continue to see increasing numbers of autistic children” [30] . Having a community could reduce feelings of isolation for clinicians. One participant reported that they “love[d] the fact that I could see into the thoughts and experiences of others… And to hear… what other practices are struggling with or doing well with. So that whole like community feel[ing], I thought was great” [30] .
3.2.4. Construct 4: Reflexive Working
Reflexive working is defined as participants’ assessment of the actions they took and whether they were useful or successfully implemented [27] . In the current evaluation, reflexive working applied to when participants evaluated 1) their implementation of strategies learned during the training, 2) training content, and 3) how training was delivered.
BCAEP was helpful in increasing rates of autism screening, which could catalyze referrals to specialists, thus reducing wait times. Access to external services such as ABA, however, remained scarce and could not be addressed by BCAEP alone. One participant reported that they were “continuing to have, unfortunately, a lot of trouble getting kids hooked in with ABA, and part of it is that [the insurance system] is claiming that I am not qualified to do autism evaluations...” [30] .
Additionally, participants noted the importance of effectively communicating with families, using transparent language. One participant “made a point of using the word [autism]… most parents were thinking it anyway, and if you don’t say it, then they continue to worry inside… they almost… appreciated hearing that” [30] .
Participants reported positive feedback about training delivery. Factors that increased training engagement included learning from other participants: “The process of [going] through a case presentation from another pediatrician at another health center about a tough diagnostic disclosure with a family with language and cultural and whatever barriers. Like that experience… I found the most valuable” [30] . Participants also offered additional topics for future trainings such as transition to adult services, ABA, and medications.
4. Discussion
Procuring services for autism is a complex and time-consuming process with different systemic barriers and bottlenecks [6] . Equipping PCCs with the ability and confidence to streamline the initial steps (i.e., screening) and counsel families about autism can help improve patients’ overall experience in accessing and sustaining care. Participants reported an increase in self-rated ability and self-efficacy to administer autism screeners, communicate with families about autism, manage care of autistic patients, and identify/refer to appropriate services for autistic patients.
4.1. Autism Screening and Care Management
The positive change observed in self-efficacy about autism screening and care management is consistent with past evaluations of other autism-focused ECHO programs [15] [16] [18] [22] [24] . In particular, increased self-reported ability and self-efficacy could lead to practice change [34] though evidence on practice change due to the ECHO model is mixed [13] [24] . Qualitative analysis using NPT allowed exploration into the barriers within a system of clinicians that, no matter how motivated and empowered to change practice, faced difficulties in implementing the lessons learned. Examples of specific challenges faced by participants in implementing knowledge and skills gained from BCAEP into their clinical practice include time constraints and workload demands as barriers to autism screenings. Specifically, limited time during patient appointments and gaps between appointments that made timely receipt of services challenging (e.g., state-funded early intervention stops at a child’s third birthday). These barriers coupled with an already overextended workforce could hinder access to services [6] . The current evaluation, however, showed that increasing confidence to perform these responsibilities quickly may overcome some of the environmental barriers to providing care to patients.
Initial screenings may not result in service access improvements without an efficient referral protocol [35] . Participants reported that there were long wait times for long-term services resulting in delays regardless of how confident and adept clinicians were in making referrals. BCAEP clinicians expressed workflow issues in ABA and school assessments and offered suggestions for future trainings, including involving specialists in training sessions. Improvements to clinic workflows and processes, expanding care capacity, and clear protocols for feedback loops across settings (i.e., clinics, schools, early intervention) are necessary to fully address autism care in pediatric patients.
4.2. Patient Education and Counseling
In both the pre-test and post-test surveys and follow-up focus groups, participants indicated an increased skill and comfort in communicating an autism diagnosis and educating families about services. Participants also indicated wanting to learn more about the specifics of behavior therapy so that they could better explain to families and increase their engagement in these services. Findings were consistent with those of a scoping review of 27 studies, which reported healthcare professionals’ desire for more training, information, and resources to care for autistic patients [36] .
Another barrier reported by participants was families’ refusal to see a specialist. This is consistent with the literature, which also shows sociocultural factors influencing patients’ willingness to see a specialist and receive an autism diagnosis [7] . One systematic review found that families reported experiencing stigma, associated consequences (e.g., self-esteem and mental well-being) [37] , and reduced access to autism services for underserved populations [7] . Complex payment systems, trauma, and cultural bias are barriers affecting patients’ ability to access services, which are largely outside clinician control [38] .
BCAEP clinicians regularly care for families with diverse sociocultural backgrounds making stigma and access to services for underserved populations an important focus in trainings. BCAEP clinicians qualitatively reported increased knowledge about addressing barriers as well. Sociocultural, economic, and systemic barriers faced by underserved families [37] highlighted the importance of intentional and culturally responsive patient/family education. An entire session was dedicated to discussing communicating an autism diagnosis and navigating the sociocultural factors that may affect how an autism diagnosis is received by the family.
4.3. Burnout and Professional Fulfillment
Survey responses on burnout suggested an increase in participants’ empathy with patients and colleagues as well as sensitivity toward others’ emotions from pre- to post-program. This finding aligned with other researchers’ conclusion that the ECHO model could be used to support frontline staff [39] and buffer clinician burnout [40] . Professional fulfillment indicators were not significantly affected.
4.4. Limitations
This evaluation has several limitations. First, it was conducted on a group of self-selected clinicians and may not be generalizable to other settings. Self-selecting participants may have higher motivation to learn about and interest in developmental and behavioral pediatrics, which may not represent all PCCs. Participants, however, represented clinics across a diverse health system and came from a variety of experience and roles. Second, the sample size was small as it was exploratory research for a newly-developed program, which limits generalizability as the participant group may not be representative of all PCCs. The small group made it possible to assess individual engagement in sessions and establish a cohesive community over the year-long project. Third, there was no comparison group though baseline information was collected prior to the program. Lack of a comparison group limited our ability to distinguish whether gains reported by participants were due to the training or other confounding factors, however the pre-test/post-test design does allow for shifts throughout the training to be captured. Finally, all data on participants’ ability to screen and care for autistic patients were self-reported and there were no data on screening rates at the clinics to measure longer-term impact of the training on practice change. Some participants, however, qualitatively reported an increase in autism screening rates in their clinics, which is consistent with another autism-focused ECHO program in underserved rural primary care practices [13] .
4.5. Conclusion
Accessing developmental services can be a time consuming and complicated journey for autistic patients and families. There are multi-level barriers from the child and family level (e.g., sociocultural factors) to systems and policy level (e.g., frequently changing insurance requirements). One barrier is lack of clinicians trained and skilled in developmental care [41] , which is more problematic since the COVID-19 pandemic with clinician shortages in pediatric subspecialists and greater need in children and families. Training programs designed to enhance self-reported ability and self-efficacy of primary care clinicians in autism screening, care, and patient education can improve access to crucial services for autistic patients. To be most effective and provide lessons for future application and trainings, however, these programs should be rigorously evaluated for implementation and achievement of outcomes. The results of this mixed-methods evaluation can inform future clinician training for successful delivery and effectiveness, resulting in more primary care clinicians who can conduct initial screening and are knowledgeable about developmental care to best manage a patient’s care and provide recommendations to families.
To our knowledge, this is the first mixed-methods evaluation conducted on an autism-focused ECHO program in a safety-net urban setting and in an all-virtual environment. BCAEP increased participants’ self-efficacy and self-reported ability to administer autism screening, manage care, and educate families about autism. There is also evidence for reduced burnout after program participation. Moreover, follow-up focus groups offered individual and clinic-level context on successful strategies for practice implementation. Competencies addressed in BCAEP may result in faster access to services and improved rapport with families, which is crucial as autism can have significant lifelong impacts that require care continuity [42] . Additional research is required to monitor the long-term outcomes related to participants’ knowledge retention, confidence in practice, and improvements to autism evaluations and patient education. Potential examples of improvement indicators of autism care include wait time to developmental evaluation and diagnosis receipt, as well as gathering feedback from families regarding their experience accessing developmental care. As resources allow, periodic follow-up with training participants allows for deeper understanding of the competencies that are put into practice and the barriers and facilitators to sustainability of practice change.
Acknowledgements
We would like to acknowledge the health system and health centers for their investment of time and participation in this project and evaluation. This evaluation was supported by the Deborah Munroe Noonan Memorial Research Fund. The funding sponsor did not have a role in study design, in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.
Appendix 1