search
Back to results

Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.

Primary Purpose

Attention Deficit Hyperactivity Disorder

Status
Unknown status
Phase
Not Applicable
Locations
Study Type
Interventional
Intervention
Artificial Intelligence-Assisted Supervision Protocol
Sponsored by
Seattle Children's Hospital
About
Eligibility
Locations
Arms
Outcomes
Full info

About this trial

This is an interventional health services research trial for Attention Deficit Hyperactivity Disorder focused on measuring ADHD, Adolescents, Community Mental Health

Eligibility Criteria

11 Years - 17 Years (Child)All SexesDoes not accept healthy volunteers

Inclusion Criteria:

  • DSM-5 ADHD diagnosis, Enrollment in 6th-12th grade, IQ greater than 70, no history of autism spectrum disorder or thought disorder, client of one of two community mental health agencies

Exclusion Criteria:

  • Autism Spectrum/Thought Disorders, IQ<70

Sites / Locations

    Arms of the Study

    Arm 1

    Arm 2

    Arm Type

    Experimental

    No Intervention

    Arm Label

    Artificial Intelligence-Assisted Supervision Protocol

    Enhanced Supervision as Usual (ESAU) Condition

    Arm Description

    Measurement-based supervision protocol that incorporates fidelity measurement from a machine learning tool and feedback reports from this tool into a standardized supervision protocol for behavior therapy to task-shift burdensome supervision tasks to a machine, reducing costs and improving precision of fidelity measurement for agencies.

    ESAU therapists will be given standard, paper-based facilitation resources for STAND and will receive 4 hours of training on how to navigate these materials and self-assess fidelity. ESAU therapists will also be trained how to upload recordings into Care4 and complete self-assessments for each session. Supervisors will be given access to these data and recordings once uploaded (but not Lyssn scores or electronic facilitation resources).

    Outcomes

    Primary Outcome Measures

    Therapist STAND Fidelity
    Behavior therapy content: STAND treatment fidelity checklists (Sibley et al., 2013, 2016, 2019). This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.
    Therapist MITI Fidelity
    Behavior therapy content:Motivational Interviewing Treatment Integrity (MITI) measure (Moyers et al., 2014) will measure MI integrity.This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.

    Secondary Outcome Measures

    Service Delivery Quality
    Quality: (1) Time to Therapist MI Competence; MITI Benchmarks,(Moyers et al., 2014) Accuracy of therapist self-report on MITI indices and fidelity checklists (tested using Polynomial Regression)--will assess concordance between reporters over time. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.
    Service Delivery Quantity
    Quantity: (2) Proportion of EBT delivered by 10th session Speed of Delivery: Number of sessions and days to completion of EBT. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.

    Full Information

    First Posted
    September 24, 2021
    Last Updated
    November 13, 2021
    Sponsor
    Seattle Children's Hospital
    Collaborators
    Florida International University
    search

    1. Study Identification

    Unique Protocol Identification Number
    NCT05135065
    Brief Title
    Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.
    Official Title
    Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.
    Study Type
    Interventional

    2. Study Status

    Record Verification Date
    September 2021
    Overall Recruitment Status
    Unknown status
    Study Start Date
    November 18, 2021 (Anticipated)
    Primary Completion Date
    September 1, 2022 (Anticipated)
    Study Completion Date
    December 1, 2022 (Anticipated)

    3. Sponsor/Collaborators

    Responsible Party, by Official Title
    Principal Investigator
    Name of the Sponsor
    Seattle Children's Hospital
    Collaborators
    Florida International University

    4. Oversight

    Studies a U.S. FDA-regulated Drug Product
    No
    Studies a U.S. FDA-regulated Device Product
    No
    Data Monitoring Committee
    Yes

    5. Study Description

    Brief Summary
    This project proposes to reduce disparities in care among disadvantaged racial/ethnic minority adolescents with ADHD by improving community therapist fidelity to evidence-based behavior therapy through a technology-assisted supervision intervention. In Y01, the research team will work with stakeholders to develop the proposed supervision intervention utilizing two novel technologies: Lyssn + Care4 (LC4S). In Y02, a preliminary clinical trial (N=72) will be conducted in three community mental health agencies in Miami, FL. Adolescent participants will be randomly assigned to receive supervision from a therapist who is trained in LCS4 or provides enhanced supervision as usual(ESAU)using a permuted block randomization strategy that randomizes within site. There will also be double randomization of agency therapists to supervisors. Supervisors will deliver both conditions and investigators will test for contamination to determine the integrity of this design prior to a future R01 that measures patient outcomes. Data from therapists, adolescents and their parents, and supervisors will be collected pre-training, post-training, weekly during service delivery, at EBT completion, and at the end of the trial. The proximal intervention target is therapist fidelity to EBT and the distal targets are service delivery outcomes that include quality, quantity, and speed of delivery. Investigators will also measure indices of consumer fit: cost, acceptability, feasibility, and fidelity to supervision procedures. Sources of data will be audio recorded therapy and supervision sessions, therapist and supervisor report, and project and electronic health records. In longitudinal analyses, time will be modeled as a person-specific variable representing months since baseline. Investigators will nest adolescents within therapists for all analyses.
    Detailed Description
    Y02 of this study is a small (N=72) phase 1/phase 2 clinical trial of a supervision intervention designed to improve therapist treatment fidelity and subsequent service delivery outcomes. The parallel design includes random assignment of eligible and consenting patients at three community agencies to two active supervision intervention arms (LC4S or ESAU) using a permuted block randomization strategy that accounts for agency. Participants will receive behavioral interventions from community agency staff and their service utilization will be tracked using project and agency electronic health records. Agency therapists and supervisors will also be participants in this trial. Therapists will be randomized to receive either LC4S or ESAU from their supervisor (double randomization) using a block randomization strategy that accounts for site. Supervisors will administer both supervision conditions in this trial; however, investigators will systematically assess for contamination to assess whether this design is appropriate for a future R01. To minimize bias, adolescent and parent participants will not be informed of the group to which they have been assigned. However, full masking of therapists and supervisors is not feasible in this trial because both supervisors and therapists will know whether they are participating in technology-assisted supervision activities or standard supervision activities due to the nature of these conditions. However, the primary investigator will blind therapists and supervisors to our study hypotheses and the nature of outcome measures to minimize bias in the trial. Many of these measures will be observational and objective (i.e., therapy records, therapy audio recordings), which should reduce bias stemming from self-reports. Investigators will also assess whether there are group effects on therapist accuracy of self-report. All interventions will be delivered by agency staff, who will not be required to follow intervention delivery protocols because an outcome of this study is the extent to which agency therapists follow intervention procedures with guidance provided from their supervisors. Study assessments will be administered electronically via Care4 and data collection will be oversee by study staff. Each measure of fidelity will be analyzed using a separate mixed /growth model (Duncan et al., 1999). In this design, treatment sessions (level 1) are nested within adolescents (level 2), which are nested within therapists (level 3). Each adolescent attends up to 10 sessions; each therapist treats 3 adolescents. Supervisors serve as an additional higher level, but with so few supervisors (approximately 6), this will be addressed by including dummy predictor variables representing the supervisors. The direct effect of LC4S vs. ESAU on fidelity intercept and slope will be tested for each fidelity outcome (see Table 3). With time centered at the first session, the intercept reflects initial fidelity for the ESAU condition, the group effect reflects the initial fidelity difference between ESAU and LC4S, the time effect reflects the linear change in fidelity over time for the ESAU condition, and the interaction of time and group reflects the difference between ESAU and LC4S in linear change in fidelity over time. As part of the R34, in will estimate the intraclass correlation (ICC) and design effect for the clustering effect of therapist and supervisor on outcome to determine the extent to which additional clustering will be needed in a future R01. ICC ranges from 0 to 1, with larger values reflecting a larger proportion of variance at the higher levels (here, therapist and supervisor rather than adolescent). Investigators will also test both linear and non-linear slopes to ascertain the expected shape of the LGCA in a future R01. Time to therapist MI competence, proportion of EBT delivered by 10th session, and number of sessions and days to completion of the EBT will be modeled using regression (linear, logistic, or Poisson depending on the distribution of the resulting variables) with group as a predictor. Accuracy of therapist self-report will be analyzed using polynomial regression (Laird & LaFleur, 2013). Using the R package powerlmm, investigators have an estimate of .8 power to detect large effects (d = 0.8) representing group differences at the adolescent (level 2) level; investigators have .4 power to detect medium effects (d = .5) representing group differences at the adolescent level. Our hypotheses, however, are primarily at the level of the therapist (level 3), which has fewer units (i.e., 12 therapists per condition versus 36 adolescents per condition). As such, measures of both adolescent-level and therapist-level effects will be estimated. For power for the ICC calculation, investigators used the ICC.Sample.Size R package based on Zou (2012). With a sample size of 72 participants, 10 observations per participant, alpha equal to .05, and a two tailed test, investigators have greater than .9 power detect an ICC of 0.2. ICC values for cross-sectional data such as children within classrooms are approximately 0.2 (Hedges & Hedberg, 2007). Investigators expect excellent precision for estimating ICC values in this study.

    6. Conditions and Keywords

    Primary Disease or Condition Being Studied in the Trial, or the Focus of the Study
    Attention Deficit Hyperactivity Disorder
    Keywords
    ADHD, Adolescents, Community Mental Health

    7. Study Design

    Primary Purpose
    Health Services Research
    Study Phase
    Not Applicable
    Interventional Study Model
    Parallel Assignment
    Masking
    Outcomes Assessor
    Allocation
    Randomized
    Enrollment
    72 (Anticipated)

    8. Arms, Groups, and Interventions

    Arm Title
    Artificial Intelligence-Assisted Supervision Protocol
    Arm Type
    Experimental
    Arm Description
    Measurement-based supervision protocol that incorporates fidelity measurement from a machine learning tool and feedback reports from this tool into a standardized supervision protocol for behavior therapy to task-shift burdensome supervision tasks to a machine, reducing costs and improving precision of fidelity measurement for agencies.
    Arm Title
    Enhanced Supervision as Usual (ESAU) Condition
    Arm Type
    No Intervention
    Arm Description
    ESAU therapists will be given standard, paper-based facilitation resources for STAND and will receive 4 hours of training on how to navigate these materials and self-assess fidelity. ESAU therapists will also be trained how to upload recordings into Care4 and complete self-assessments for each session. Supervisors will be given access to these data and recordings once uploaded (but not Lyssn scores or electronic facilitation resources).
    Intervention Type
    Other
    Intervention Name(s)
    Artificial Intelligence-Assisted Supervision Protocol
    Intervention Description
    Measurement-based supervision protocol that incorporates fidelity measurement from a machine learning tool and feedback reports from this tool into a standardized supervision protocol for behavior therapy to task-shift burdensome supervision tasks to a machine, reducing costs and improving precision of fidelity measurement for agencies.
    Primary Outcome Measure Information:
    Title
    Therapist STAND Fidelity
    Description
    Behavior therapy content: STAND treatment fidelity checklists (Sibley et al., 2013, 2016, 2019). This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.
    Time Frame
    Weekly for an average of 9 months
    Title
    Therapist MITI Fidelity
    Description
    Behavior therapy content:Motivational Interviewing Treatment Integrity (MITI) measure (Moyers et al., 2014) will measure MI integrity.This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.
    Time Frame
    Weekly for an average of 9 months
    Secondary Outcome Measure Information:
    Title
    Service Delivery Quality
    Description
    Quality: (1) Time to Therapist MI Competence; MITI Benchmarks,(Moyers et al., 2014) Accuracy of therapist self-report on MITI indices and fidelity checklists (tested using Polynomial Regression)--will assess concordance between reporters over time. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.
    Time Frame
    Weekly for duration that case is active in the agency, an average of 9 months
    Title
    Service Delivery Quantity
    Description
    Quantity: (2) Proportion of EBT delivered by 10th session Speed of Delivery: Number of sessions and days to completion of EBT. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.
    Time Frame
    Weekly for duration that case is active in the agency, an average of 9 months
    Other Pre-specified Outcome Measures:
    Title
    Cost
    Description
    Cost of implementing/subscribing to Care4/Lyssn; Supervisor/ therapist time , technical supports, stakeholder-identified costs
    Time Frame
    cost will be computed at the end of the year from study records and electronic heath records, an average of 9 months
    Title
    Technology Acceptability
    Description
    Therapists and supervisors will complete an end of study questionnaire on the acceptability of the Lyssn Care4 Study protocol.
    Time Frame
    At end of the study, after an average of 9 months
    Title
    Satisfaction with supervision
    Description
    The Short Supervisory Relationship Questionnaire (Cliffe et al., 2006), which will be completed by therapists and supervisors.
    Time Frame
    At end of the study, after an average of 9 months
    Title
    Supervision Fidelity
    Description
    We will adapt a project specific version of the Supervisor Adherence and Competence Scale (SACS; Martino et al., 2016), which will be completed by trained research assistants who listen to audio recordings. In addition, we will train Lyssn to evaluate supervision sessions on this checklist by the end of the trial for potential use as supervisor feedback tool in a future R01.
    Time Frame
    Weekly, through study completion, an average of 9 months
    Title
    Therapist engagement
    Description
    Therapist Supervision Attendance: Supervisor logs of supervision dates/times; cancelations/reasons for cancelation; to be developed Weekly Facilitation Resources Use-measure; and completion rate for weekly Care4 support tasks (uploading session audio file and completing fidelity self-report form prior to supervision) forms are very brief questionnaires completed by therapists and supervisors prior to uploading each session in Care4 to measure therapist engagement with Care4 and Lyssn integrations
    Time Frame
    Weekly, through study completion, an average of 9 months
    Title
    Supervisor Burden
    Description
    Minutes discussing case on audio recording; supervisor log of prep work for case (not including research tasks). Tapes will be coded by RAs for length and supervision logs will be kept by the supervisors.
    Time Frame
    Weekly, through study completion, an average of 9 months
    Title
    Exit Interviews
    Description
    Semi-Structured exit interviews designed to collect qualitative perspectives on the LC4S intervention.
    Time Frame
    At end of the study, after an average of 9 months

    10. Eligibility

    Sex
    All
    Minimum Age & Unit of Time
    11 Years
    Maximum Age & Unit of Time
    17 Years
    Accepts Healthy Volunteers
    No
    Eligibility Criteria
    Inclusion Criteria: DSM-5 ADHD diagnosis, Enrollment in 6th-12th grade, IQ greater than 70, no history of autism spectrum disorder or thought disorder, client of one of two community mental health agencies Exclusion Criteria: Autism Spectrum/Thought Disorders, IQ<70
    Central Contact Person:
    First Name & Middle Initial & Last Name or Official Title & Degree
    Mercedes Ortiz, B.A
    Phone
    206-884-8260
    Email
    mercedes.ortizrodriguez@seattlechildrens.org
    Overall Study Officials:
    First Name & Middle Initial & Last Name & Degree
    Margaret H Sibley, Ph.D
    Organizational Affiliation
    Seattle Children's Research Institute
    Official's Role
    Principal Investigator

    12. IPD Sharing Statement

    Learn more about this trial

    Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.

    We'll reach out to this number within 24 hrs