Measurement Training and Feedback System: Family-Based Services (MTFS-FBS)

Measurement Training and Feedback System for Implementation of Family-Based Services for Adolescent Substance Use: A Cluster Randomized Trial Protocol

National mandates for increasing accountability in behavioral healthcare have created an urgent need for effective quality assurance (QA) procedures featuring pragmatic measures of treatment implementation quality in usual care (UC). The most practical method for assessing treatment quality in UC is therapist self-report; unfortunately, therapist-report measures of fidelity to evidence-based practices (EBPs) have demonstrated weak validity to date. This study addresses this need for effective QA procedures by first developing treatment quality assurance procedures designed to increase the adoption and quality of empirically supported family-based services (FBS) for adolescent substance use (ASU) in usual care, and then testing two system-level implementation strategies for installing the new quality procedures in ASU treatment sites. FBS have achieved the strongest evidence base for treating ASU and are a prime candidate for upgrading the quality of ASU services in various systems of care. FBS comprise both family participation in services, the systemic parameters wherein family members are included in assessment and treatment activities; and family therapy techniques, the specific interventions that clinicians use to directly target family members and family functioning for change. For FBS to fulfill their potential to enhance ASU treatment systems, FBS implementation must be supported by effective quality assurance procedures designed to ensure that FBS are delivered with fidelity. The proposed study will leverage a strong research-government partnership between the applicant organization and the New York single-state agency for SU services. The study will first develop innovative quality procedures (Aim 1) that use existing FBS quality metrics to promote high-fidelity FBS: Measurement Training and Feedback System for Implementation (MTFS-I), a pragmatic evidence-based method for increasing FBS quality by providing monthly feedback on therapist-reported FBS delivery along with brief online FBS training modules. The study will then experimentally compare two system-level implementation strategies designed to foster MTFS-I utilization in usual care for ASU. Core Training Only will contain two 3-hour training sessions: Mapping existing FBS and identifying site goals for FBS improvement; and Installing and sustaining the MTFS-I. Core + Facilitation is an additive strategy that will begin with the Core Training sessions and then continue with monthly facilitation meetings for one year to promote MTFS-I use and progress toward FBS improvement. The study will feature a three-group cluster randomized trial testing Core Training Only versus Core + Facilitation versus no-intervention Control in 9 ASU clinics across New York State. MTFS-I utilization data will be collected from Core Training and Core + Facilitation sites for one-year follow-up (after initial Core Training); FBS quality data on family participation and family therapy technique use will be collected from all sites over one-year follow-up; observational coding methods will verify therapist reports of FBS quality; and client outcome data for all sites will be retrieved from administrative data warehouses over one-year baseline and one-year follow-up. These data will enable between-condition comparisons of FBS delivery (Aim 2: MTFS-I utilization, family participation, family therapy technique use) and client outcomes (Aim 3: therapeutic goal achievement, substance use change). If study aims are achieved, investigators would be positioned to mount a fully powered RCT to test the effectiveness of these FBS quality improvement procedures at scale.

Discovering and disseminating effective methods to improve the quality of treatment services for adolescent substance use (ASU) within the national behavioral healthcare system is an urgent public health priority. Despite the strong legislative and policy focus on quality of care evident in the Children's Health Insurance Program Reauthorization Act of 2009 and the Affordable Care Act of 2010, recent comprehensive reports underscore the enduring "quality chasm" between behavioral treatments proven in controlled research versus those commonly practiced in usual care. ASU treatment quality in particular is considered mediocre to inadequate, due to a host of factors headlined by the absence or modest quality of evidence-based services, along with inadequate provider training, little quality monitoring, inattention to data-driven decision-making, and ineffective system-level policies for promoting existing quality mandates. Family-based services (FBS) are a prime candidate for upgrading the quality of the ASU treatment system. FBS comprise both family participation in services, the systemic parameters wherein family members are included in assessment and treatment activities; and family therapy techniques, the specific interventions that clinicians use to directly target family members and family functioning for change. FBS have reached the highest levels of empirical validation for ASU, posting an exemplary record of success in comparison to alternative evidence-based treatments as well as usual care, and FBS produce the largest average effect sizes by a large margin. Due in large part to this extensive evidence base, FBS have long been strongly endorsed by federal agencies, national associations, and policy-making groups. There is also incentive from ASU clinical providers and payers to deliver FBS, which are now approved for treating ASU and disruptive behavior disorders by federal and private insurance plans and regulatory agencies that govern licensed treatment providers. FBS are also widely endorsed by therapists treating youth in routine care. Thus there is impetus from all corners to expand FBS implementation in usual care for ASU. There are several well-documented barriers to dissemination of manualized FBS models, including the cost and complexity of delivering these models in everyday settings. In addition, for FBS to fulfill their potential to enhance the quality of ASU treatment in usual care, FBS implementation must be supported by effective quality assurance (QA) procedures designed to ensure that FBS are delivered with fidelity, that is, to the target population, by appropriately trained providers, and in accord with specified procedures. The time is propitious for developing such procedures in the current healthcare market, which is incentivized to establish reliable standards for quality care. First, there is growing demand for innovative quality indicators of behavioral treatment that assess appropriateness and potential effectiveness of care. Conventional quality indicators capture broad principles of behavioral care such as treatment assignment, retention and follow-up rates, referrals for ancillary care, and client safety. However, the emerging quality-of-care implementation framework advocates that fidelity to evidence-based treatments itself be considered a quality indicator. Second, in order to properly monitor treatment fidelity in usual care, there is urgent need to develop quality metrics that can reliably and pragmatically measure fidelity in front-line treatment settings. The implications of emerging scientific and policy mandates for improving treatment quality are clear with regard to FBS for ASU: Pragmatic QA procedures for ensuring high-fidelity FBS need to be developed, and these procedures need to be anchored by reliable FBS fidelity metrics. This study will develop pragmatic QA procedures designed to promote FBS adoption and quality in ASU treatment systems using a measurement feedback system (MFS). MFS is a performance feedback loop in which a given quality indicator is continuously monitored by the clinician to gauge case progress and support clinical decision-making. MFS feedback loops usually take the form of easy-to-digest data reports that provide summary appraisals of individual client progress on selected indicators in comparison to a desired benchmark. To date MFS has been used in mental health care to enable monitoring of client outcomes primarily-for example, therapists tracking weekly client-report depression scores compared to age-adjusted norms on a depression inventory. With adults, utilizing MFS has led to impressive gains in outcomes with diverse samples: preventing early treatment failure, reversing symptom deterioration, and enhancing overall outcomes. MFS research on youth is in its early stages but growing rapidly, with strong enthusiasm about reaping comparable benefits. Importantly, clinicians trained in MFS develop positive attitudes toward it. MFS successes for client outcomes have generated enthusiastic support for the value of developing complementary MFS procedures that provide measurement Training along with data-based Feedback on implementation: MTFS-I. Due to streamlined administration procedures and emerging evidence, MTFS-I has been labeled a promising QA strategy with broad dissemination potential for youth. MTFS-I has already been incorporated into manualized QA procedures to bolster fidelity to several varieties of manualized treatment; in a few of these instances, feedback reports contain data input from therapists on fidelity indicators. One study found that MTFS-I for clinician-reported session content during youth therapy increased the likelihood and rapidity of addressing that same content in future sessions. In keeping with primary goals of the R34 mechanism, this study will test a pragmatic MTFS-I designed to improve the quality of FBS implementation in ASU treatment sites. The MTFS-I will draw on technology from existing feedback systems, but also, be the first to target system-level FBS delivery for ASU. The four MTFS-I components, described in C4, are designed to increase the extensiveness (i.e., amount and frequency of) FBS implementation.

This study is a cluster randomized trial testing two additive implementation strategies for installing FBS quality improvement procedures in ASU outpatient treatment. Nine ASU treatment sites will be yoked and randomized into three conditions to enhance ecological validity; 1) Core Training Only; 2) Core Training and Facilitation; and 3) Control (no intervention). The implementation strategies (i.e., Core Training Only, Core + Facilitation), are system-level strategies for translating evidence-based interventions into multi-site service settings.

Core Training will be a blended strategy wherein multiple interventions targeting multiple agency roles are combined to help ASU treatment providers increase their FBS quality. Core Training will be conducted with administrators, supervisors, data managers, and clinical staff. It will contain two 3-hour sessions focusing on FBS System Mapping and MTFS-I Installation. FBS System Mapping will focus on process mapping of existing site policies and practices for assessing family functioning at intake, educating families about intervention strategies and activities, including family members in treatment sessions, and hiring and supporting clinicians trained in and routinely delivering FBS. MTFS-Installation will focus on introducing MTFS-I components and integrating principles of DDDM.

Core Training & Facilitation: FBS System Mapping and MTFS-I Installation plus monthly on-site meetings

Facilitation will consist of monthly on-site meetings to review progress toward FBS improvements, discuss MTFS-I feedback reports and online resource use, and provide intensive technical assistance for progressing toward site-selected FBS improvement goals. Meetings will involve a subset of site-delegated participants from Core Training sessions to serve as a local change team (quality improvement team). The efficacy of facilitation-guided change teams has been established. Although there are several models for creating change teams, at least three features are common: (1) members represent multiple organizational roles (supervisors, data managers, line staff); (2) focus on data-driven decisions; and (3) use of external expert advisors.

The Clinical Practice Standards for Adolescent Programs (CPS-AP) is a manualized set of ASU service delivery guidelines co-developed by OASAS and CASA. The Manual describes each of 12 Standards using the following format: (a) theoretical background illustrating the clinical and programmatic salience of the Standard for delivering quality ASU treatment; (b) 4-8 main components of the given Standard, with a cross-Standard focus on developmental science-based differences between adolescent versus adult treatment practices; (c) brief clinical descriptions and exemplars of each Standard. Throughout the study therapists will be asked to complete a web-based form to report on family participation via the CPS-AP (once per month).

The proposed study will utilize the 7-item Family Therapy (FT) scale from the 27-item Inventory of Therapy Techniques for Adolescent Behavior Problems (ITT-ABP). The ITT-ABP is a QA tool designed to collect post-session therapist-report data on implementation of four treatment approaches commonly deployed in front-line ASU care: FT, cognitive-behavioral therapy (CBT), motivational interviewing (MI), and drug counseling (DC). The ITT-ABP observational version mirrors the therapist-report version, containing the same 7 FT scale items. We will collect session audio recordings with all study families, and from among all recordings we will randomly select one session from the Early and Later phase of each case for coding with the observer version of the ITT-ABP.

Client outcomes will be extracted from the OASAS Client Data System. For analyses, we will extract non-identified data on all clients admitted to the 9 study sites during the one-year Baseline and FU periods. The categories to be analyzed under Outcome 3 are achievement of therapeutic goals and reduction of substance use. Treatment goal achievement is rated on a 3-point scale (1 = achieved; 2 = partially achieved; 3 = not achieved) for the following categories as applicable to a given client: Overall program goals; Alcohol use; Drug use; Education/Vocation; Social functioning; Emotional functioning; Family situation; Medical. To rate client change in SU during the course of treatment, therapists confer with clients and compare SU during the 30 days prior to discharge with SU reported at admission for up to three substances on the following scale: 1 = abstinence achieved; 2 = reduced usage; 3 = no change; 4 = increased usage; 5 = unknown usage.

Inclusion Criteria: Therapists treating adolescents aged 12-21 at partnering treatment sites Exclusion Criteria: -

Hogue A, Bobek M, Porter N, MacLean A, Bruynesteyn L, Jensen-Doss A, Henderson CE. Therapist Self-Report of Fidelity to Core Elements of Family Therapy for Adolescent Behavior Problems: Psychometrics of a Pragmatic Quality Indicator Tool. Adm Policy Ment Health. 2022 Mar;49(2):298-311. doi: 10.1007/s10488-021-01164-0. Epub 2021 Sep 2.

Hogue A, Porter N, Bobek M, MacLean A, Bruynesteyn L, Jensen-Doss A, Dauber S, Henderson CE. Online Training of Community Therapists in Observational Coding of Family Therapy Techniques: Reliability and Accuracy. Adm Policy Ment Health. 2022 Jan;49(1):139-151. doi: 10.1007/s10488-021-01152-4. Epub 2021 Jul 23.

Hogue A, Dauber S, Bobek M, Jensen-Doss A, Henderson CE. Measurement Training and Feedback System for Implementation of family-based services for adolescent substance use: protocol for a cluster randomized trial of two implementation strategies. Implement Sci. 2019 Mar 11;14(1):25. doi: 10.1186/s13012-019-0874-6.