Evaluation of a Remote Training Strategy

An increasing number of schools in rural settings are employing the multi-tier positive behavioral interventions and supports (PBIS) framework to improve school-climate. PBIS can also be used as a framework for the adoption and integration of evidence-based practices (EBPs) for children's mental health concerns. A key challenge is that school personnel need technical assistance (training plus ongoing consultation) in order to implement EBPs with fidelity. In urban and suburban schools, this support can be provided to school staff on site. However, providing ongoing on-site support is not feasible or sustainable in the majority of rural schools, due to their remote physical location. For this reason, video technology has been recommended for the training of behavioral health staff (BHS) in under-served rural communities.

Objectives: The purpose of this study is to test the effectiveness of a remote training platform with BHS in schools serving rural communities. Study Design: This study is a pilot randomized controlled trial to compare implementation and student outcomes of three training strategies: Control Condition (CC), Remote Video (RV), and Remote Video plus Coaching (RV+). Setting/Participants: Participants will be school personnel with and without prior mental health training and students in grades 4-8 in schools serving rural communities who are deemed at risk for externalizing and internalizing mental health disorders. The pilot trial will be conducted in 24 schools (8 in CC, 8 in RV, and 8 in RV+). It is estimated that a total of 72 behavioral health staff (3 staff per school- 24 per arm) and 312 students (13 students per school - 104 per arm) will participate in this study. Additionally, we estimate about 24 school administrators such as principals or assistant principals will approve school's participation. Study Interventions and Measures: Each school will be randomly assigned to receive one of three training supports: (a) Initial training (CC) (b) Initial training, plus video (RV) or (c) Initial training plus video, plus remote coaching (RV+). BHS in the RV+ condition will receive consultation from a Children's Hospital of Philadelphia (CHOP) consultant regarding implementation of EBPs, whereas BHS in the RV condition will be given access to the asynchronous training video modules. Participants in all three conditions will receive treatment manuals for the interventions they have chosen. The EBPs to be implemented by the BHS include: Coping Power Program (CPP), CBT for Anxiety Treatment in Schools (CATS), and Check-in/Check-out (CI/CO). All BHS will conduct one CPP or CATS group with 5 students of similar developmental level (e.g., a group of 4th & 5th graders), and that each school staff in CI/CO will implement the intervention with 4 separate individual students of any eligible school grade. Analyses: This study is designed to test the effectiveness of initial training compared to remote video training and remote video training plus coaching to train BHS to implement EBPs.

Positive-behavior Interventions and Supports, Evidence-based Practices, Tier 2, Mental Health, Rural Schools

School personnel will implement CATS or CPP and CICO with students in the school setting. They will participate in an initial live remote training to learn about implementing the three EBPs (CC).

School personnel will implement CATS or CPP and CICO with students in the school setting. They will participate in an initial live remote training and receive access to asynchronous video training modules about the EBPs (RV).

School personnel will implement CATS or CPP and CICO with students in the school setting. They will participate in an initial live remote training, receive access to asynchronous video training modules about the EBPs, and receive coaching support by study staff (RV+). The coaching will be from a study consultant regarding the implementation of EBPs.

CPP is an evidence-based intervention designed for students with externalizing behavior disorder. CPP consists of twelve 45-minute sessions. This EBP has been found to be effective at reducing aggressive behavior, covert delinquent behavior and substance abuse among aggressive boys, with gains maintained at one-year follow-up. Growth curve analyses showed that CPP had linear effects for three years after intervention on reductions in aggressive behavior and academic behavior problems.

CATS is an adaptation of Friends for Life (FRIENDS). The adapted protocol retains the core elements of evidence-based CBT for anxiety and the FRIENDS group format. Investigators implemented planned adaptations to the protocol based on collective experience. Changes were made to the language, cultural methods, number of sessions, and activities while maintaining the 5 essential components of the treatment. This resulted in a briefer (8-session) and more feasible, engaging and culturally appropriate protocol for urban under-resourced schools than the original FRIENDS.

CI/CO is a targeted, Tier 2 intervention for students at risk of developing externalizing and internalizing mental health disorders. CI/CO is designed to provide immediate feedback (i.e., at the end of each class period) to students, based on the use of a daily report card. This feedback is developmentally sensitive. CI/CO implementers meet individually with students for a brief 'check-in' in the morning and a brief 'check-out' in the afternoon. Research on the use of CI/CO has shown it to be effective in reducing externalizing and internalizing problems with elementary school students. CI/CO will be offered to individual students for a three-month period of time, which is the same time-frame needed for the implementation of CPP.

School personnel will participate in an initial live remote training and receive access to asynchronous video training modules.

School personnel will participate in an initial live remote training, receive access to asynchronous video training modules, and receive coaching support from study staff.

The primary endpoints related to school staff implementing the interventions are measures of content fidelity. Content fidelity for the group interventions CPP and CATS will be measured using the Content Fidelity Checklists that reflects each activity component of the session agenda of the treatment protocols. Raters use a yes/no response scale to indicate whether or not the implementer covered a particular component.

Content fidelity for CI/CO will be assessed via the Check-In/Check-Out Fidelity Checklist. The CI/CO Fidelity Checklist is completed by the consultant (research staff) on a monthly basis after collecting the completed Daily Progress Report (DPR) student forms for each student involved in CICO.

The primary endpoints related to school staff implementing the interventions are measures of content and process fidelity. Process fidelity for the group interventions CPP and CATS will be assessed via an adapted version of the rating system developed by Lochman and colleagues. 70 Ten items are rated on a scale of 0 to 4, with 0 being "Not At All" and 4 being "Very Often". Ratings are given on the extent to which school staff members delivered the intervention in an orderly fashion, using active learning strategies. etc. Exploratory factor analysis (EFA) yielded two factors on the 10-item scale.

Perceived adoption and penetration will be assessed via a survey completed by school staff. The adoption inventory will track the number of times each intervention is used per school, per condition. The penetration inventory will list EBP penetration at the student level (students receiving EBPs at Tier 2).

Student Mental Health Symptoms will be measured by comparing parent scores pre-intervention and post-intervention. The Behavior Assessment System for Children-3rd edition (BASC-3) is a 138-item, 4-point, Likert-type rating scale (N=Never, S=Sometimes, O=Often, A=Almost Always) for assessing parental report of child mental health functioning, standardized for ages 2.5 to 18 years.

Student Academic Engagement will be measured by comparing teacher scores pre-intervention and post-intervention. This is a 20-item, four-point instrument (1 = Not At All True, 2 = Not Very True, 3 = Sort Of True, 4 = Very True) with four sub-scales. Investigators will use the average score for each of the four scales at pre- and post-participation in group cognitive behavioral therapy (CBT) or Check-in/Check-out (CICO).

Student externalizing and internalizing problems will be measure by comparing student scores pre-intervention and post-intervention. The Behavior and Feeling Survey-Youth Report (YFS-SR) is a brief (10 items) measure of internalizing and externalizing problems rated on a 5-point scale. The YFS-SR has excellent psychometric properties, and it is sensitive to change. The YFS-SR provides three scores (Internalizing, Externalizing, Total).

Secondary endpoints are the mediators and moderators effects of consultation support on behavioral health staff fidelity and the qualitative examination of school staff's perceived feasibility, appropriateness and acceptability of the support they received.

Inclusion Criteria: Administrator: Any school principal or assistant principal from participating schools implementing PBIS. Behavioral Health Staff: Any counselor, social worker, or teacher from participating schools implementing PBIS who work with students in grades 4-8. Students: Inclusion of students to receive a Tier 2 intervention is as follows: attending one of the participating schools being in grades 4-8 identified by the Tier 2 team as not responding to Tier 1 intervention, thus needing Tier 2 support scoring ≥ 1 SD above the mean on the Emotional Symptoms or Conduct Problems scales of the Strength andDifficulties Questionnaire (SDQ) completed by a parent or a teacher The cut-off score level for the SDQ is appropriate for identifying students at risk for a behavioral/mental health disorder. We limit participation to students in grades 4-8 because the group EBPs are appropriate for this age group. Exclusion Criteria: Administrator: School staff who are not principals or assistant principals. Behavioral Health Staff: School staff who are not part of the PBIS team and who do not work with students in grades 4-8. Students: Students who do not meet screening or group participation criteria will not be included in the study. Students with a history of intellectual disability or serious developmental delays according to school records will not be included because they would be unlikely to benefit from the interventions used in the study. Students with a history of psychotic or autistic spectrum disorders as reported by parents will not be included. Subjects that do not meet all the enrollment criteria may not be enrolled. Any violations of these criteria must be reported in accordance with the Institutional Review Board (IRB) Policies and Procedures.

Eiraldi R, McCurdy BL, Khanna MS, Goldstein J, Comly R, Francisco J, Rutherford LE, Wilson T, Henson K, Farmer T, Jawad AF. Development and evaluation of a remote training strategy for the implementation of mental health evidence-based practices in rural schools: pilot study protocol. Pilot Feasibility Stud. 2022 Jun 17;8(1):128. doi: 10.1186/s40814-022-01082-4.