Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.

Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.

Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.

This project proposes to reduce disparities in care among disadvantaged racial/ethnic minority adolescents with ADHD by improving community therapist fidelity to evidence-based behavior therapy through a technology-assisted supervision intervention. In Y01, the research team will work with stakeholders to develop the proposed supervision intervention utilizing two novel technologies: Lyssn + Care4 (LC4S). In Y02, a preliminary clinical trial (N=72) will be conducted in three community mental health agencies in Miami, FL. Adolescent participants will be randomly assigned to receive supervision from a therapist who is trained in LCS4 or provides enhanced supervision as usual(ESAU)using a permuted block randomization strategy that randomizes within site. There will also be double randomization of agency therapists to supervisors. Supervisors will deliver both conditions and investigators will test for contamination to determine the integrity of this design prior to a future R01 that measures patient outcomes. Data from therapists, adolescents and their parents, and supervisors will be collected pre-training, post-training, weekly during service delivery, at EBT completion, and at the end of the trial. The proximal intervention target is therapist fidelity to EBT and the distal targets are service delivery outcomes that include quality, quantity, and speed of delivery. Investigators will also measure indices of consumer fit: cost, acceptability, feasibility, and fidelity to supervision procedures. Sources of data will be audio recorded therapy and supervision sessions, therapist and supervisor report, and project and electronic health records. In longitudinal analyses, time will be modeled as a person-specific variable representing months since baseline. Investigators will nest adolescents within therapists for all analyses.

Y02 of this study is a small (N=72) phase 1/phase 2 clinical trial of a supervision intervention designed to improve therapist treatment fidelity and subsequent service delivery outcomes. The parallel design includes random assignment of eligible and consenting patients at three community agencies to two active supervision intervention arms (LC4S or ESAU) using a permuted block randomization strategy that accounts for agency. Participants will receive behavioral interventions from community agency staff and their service utilization will be tracked using project and agency electronic health records. Agency therapists and supervisors will also be participants in this trial. Therapists will be randomized to receive either LC4S or ESAU from their supervisor (double randomization) using a block randomization strategy that accounts for site. Supervisors will administer both supervision conditions in this trial; however, investigators will systematically assess for contamination to assess whether this design is appropriate for a future R01. To minimize bias, adolescent and parent participants will not be informed of the group to which they have been assigned. However, full masking of therapists and supervisors is not feasible in this trial because both supervisors and therapists will know whether they are participating in technology-assisted supervision activities or standard supervision activities due to the nature of these conditions. However, the primary investigator will blind therapists and supervisors to our study hypotheses and the nature of outcome measures to minimize bias in the trial. Many of these measures will be observational and objective (i.e., therapy records, therapy audio recordings), which should reduce bias stemming from self-reports. Investigators will also assess whether there are group effects on therapist accuracy of self-report. All interventions will be delivered by agency staff, who will not be required to follow intervention delivery protocols because an outcome of this study is the extent to which agency therapists follow intervention procedures with guidance provided from their supervisors. Study assessments will be administered electronically via Care4 and data collection will be oversee by study staff. Each measure of fidelity will be analyzed using a separate mixed /growth model (Duncan et al., 1999). In this design, treatment sessions (level 1) are nested within adolescents (level 2), which are nested within therapists (level 3). Each adolescent attends up to 10 sessions; each therapist treats 3 adolescents. Supervisors serve as an additional higher level, but with so few supervisors (approximately 6), this will be addressed by including dummy predictor variables representing the supervisors. The direct effect of LC4S vs. ESAU on fidelity intercept and slope will be tested for each fidelity outcome (see Table 3). With time centered at the first session, the intercept reflects initial fidelity for the ESAU condition, the group effect reflects the initial fidelity difference between ESAU and LC4S, the time effect reflects the linear change in fidelity over time for the ESAU condition, and the interaction of time and group reflects the difference between ESAU and LC4S in linear change in fidelity over time. As part of the R34, in will estimate the intraclass correlation (ICC) and design effect for the clustering effect of therapist and supervisor on outcome to determine the extent to which additional clustering will be needed in a future R01. ICC ranges from 0 to 1, with larger values reflecting a larger proportion of variance at the higher levels (here, therapist and supervisor rather than adolescent). Investigators will also test both linear and non-linear slopes to ascertain the expected shape of the LGCA in a future R01. Time to therapist MI competence, proportion of EBT delivered by 10th session, and number of sessions and days to completion of the EBT will be modeled using regression (linear, logistic, or Poisson depending on the distribution of the resulting variables) with group as a predictor. Accuracy of therapist self-report will be analyzed using polynomial regression (Laird & LaFleur, 2013). Using the R package powerlmm, investigators have an estimate of .8 power to detect large effects (d = 0.8) representing group differences at the adolescent (level 2) level; investigators have .4 power to detect medium effects (d = .5) representing group differences at the adolescent level. Our hypotheses, however, are primarily at the level of the therapist (level 3), which has fewer units (i.e., 12 therapists per condition versus 36 adolescents per condition). As such, measures of both adolescent-level and therapist-level effects will be estimated. For power for the ICC calculation, investigators used the ICC.Sample.Size R package based on Zou (2012). With a sample size of 72 participants, 10 observations per participant, alpha equal to .05, and a two tailed test, investigators have greater than .9 power detect an ICC of 0.2. ICC values for cross-sectional data such as children within classrooms are approximately 0.2 (Hedges & Hedberg, 2007). Investigators expect excellent precision for estimating ICC values in this study.

Measurement-based supervision protocol that incorporates fidelity measurement from a machine learning tool and feedback reports from this tool into a standardized supervision protocol for behavior therapy to task-shift burdensome supervision tasks to a machine, reducing costs and improving precision of fidelity measurement for agencies.

ESAU therapists will be given standard, paper-based facilitation resources for STAND and will receive 4 hours of training on how to navigate these materials and self-assess fidelity. ESAU therapists will also be trained how to upload recordings into Care4 and complete self-assessments for each session. Supervisors will be given access to these data and recordings once uploaded (but not Lyssn scores or electronic facilitation resources).

Measurement-based supervision protocol that incorporates fidelity measurement from a machine learning tool and feedback reports from this tool into a standardized supervision protocol for behavior therapy to task-shift burdensome supervision tasks to a machine, reducing costs and improving precision of fidelity measurement for agencies.

Behavior therapy content: STAND treatment fidelity checklists (Sibley et al., 2013, 2016, 2019). This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.

Behavior therapy content:Motivational Interviewing Treatment Integrity (MITI) measure (Moyers et al., 2014) will measure MI integrity.This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.

Quality: (1) Time to Therapist MI Competence; MITI Benchmarks,(Moyers et al., 2014) Accuracy of therapist self-report on MITI indices and fidelity checklists (tested using Polynomial Regression)--will assess concordance between reporters over time. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.

Quantity: (2) Proportion of EBT delivered by 10th session Speed of Delivery: Number of sessions and days to completion of EBT. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.

Cost of implementing/subscribing to Care4/Lyssn; Supervisor/ therapist time , technical supports, stakeholder-identified costs

cost will be computed at the end of the year from study records and electronic heath records, an average of 9 months

Therapists and supervisors will complete an end of study questionnaire on the acceptability of the Lyssn Care4 Study protocol.

The Short Supervisory Relationship Questionnaire (Cliffe et al., 2006), which will be completed by therapists and supervisors.

We will adapt a project specific version of the Supervisor Adherence and Competence Scale (SACS; Martino et al., 2016), which will be completed by trained research assistants who listen to audio recordings. In addition, we will train Lyssn to evaluate supervision sessions on this checklist by the end of the trial for potential use as supervisor feedback tool in a future R01.

Therapist Supervision Attendance: Supervisor logs of supervision dates/times; cancelations/reasons for cancelation; to be developed Weekly Facilitation Resources Use-measure; and completion rate for weekly Care4 support tasks (uploading session audio file and completing fidelity self-report form prior to supervision) forms are very brief questionnaires completed by therapists and supervisors prior to uploading each session in Care4 to measure therapist engagement with Care4 and Lyssn integrations

Minutes discussing case on audio recording; supervisor log of prep work for case (not including research tasks). Tapes will be coded by RAs for length and supervision logs will be kept by the supervisors.

Semi-Structured exit interviews designed to collect qualitative perspectives on the LC4S intervention.

Inclusion Criteria: DSM-5 ADHD diagnosis, Enrollment in 6th-12th grade, IQ greater than 70, no history of autism spectrum disorder or thought disorder, client of one of two community mental health agencies Exclusion Criteria: Autism Spectrum/Thought Disorders, IQ<70