Stepped Approach to Reducing Risk of Suicide in Primary Care (STARRS-PC)

National Institute of Mental Health (NIMH), Ohio State University, Holzer Gallipolis, Holzer Jackson, Holzer Athens, Eastglen Pediatrics, Signature Health Ashtabula, Signature Health Willoughby, Signature Health Painesville, AxessPointe - Arlington, Dayton Children's - Main Campus, Dayton Children's - Northwest Campus, Cornerstone Pediatrics, Kettering Pediatric and Family Care, Franklin Avenue Primary Care Center, Village Square Primary Care Center, Spring Valley Primary Care Center

Suicide is the second leading cause of death among young people aged 12-17 years in the United States, yet many youth at risk for suicide are not identified or go untreated. Stepped care approaches have been shown to be effective at reducing suicide risk in clinical settings, including primary care. The goal of this hybrid I stepped wedge effectiveness-implementation study is to test the effectiveness of a population-based quality improvement (QI) intervention, entitled STARRS-PC (Stepped Approach to Reducing Risk of Suicide in Primary Care) compared to treatment as usual (TAU), in reducing the risk of suicidal behavior among youth in the pediatric primary care setting. STARRS-PC implements a clinical pathway for youth at elevated risk for suicide in pediatric primary care clinics. Clinical pathways are tools used by health professionals to guide evidence-informed practice. The STARRS-PC pathway consists of three evidence-based suicide clinical care processes: risk detection, assessment and triage, and, if needed, follow-up transitional care. STARRS-PC is guided by the Practical, Robust Implementation, and Sustainability Model (PRISM), which allows for the study of factors that influence effective implementation of the suicide prevention clinical pathway and is focused on scalability. The main questions the study aims to answer are: Will STARRS-PC be more effective than TAU at reducing the rate of suicide attempt at 12 months post-baseline (primary outcome)? Will STARRS-PC be more effective than TAU at reducing suicidal ideation and non-suicidal self-injury, and improving family satisfaction at 12 months post-baseline (secondary outcomes)? What are the barriers and facilitators of effective implementation and sustainability of STARRS-PC?

This five-year multi-site study consists of the following essential elements: (1) recruitment of 2,324 adolescents ages 12-17 years inclusive at baseline, and their parent/guardian; (2) initial assessment of youth and parent/guardian participants using direct interviews and standardized questionnaires; and (3) follow-up assessments of all participants at 3-months, 6-months, and 1-year post-baseline. To achieve the study objectives, the project will be conducted in 16 pediatric primary care practices. The study consists of three phases of data collection: TAU, intervention, and sustainability. One thousand thirty-three suicidal youth will be enrolled during the TAU phase, and 1,291 suicidal youth will be enrolled during the intervention phase (total number of youth participants = 2,324). Youth and parent participants enrolled in the study will complete multiple questionnaires at four time points (baseline, 3-months, 6-months, and 12-months). Medical record reviews of youth participant files will occur throughout the study and be reviewed by research staff. The intervention phase is followed by a 6-month sustainability phase during which the participating practices will work on maintaining or further improving upon the anticipated gains made during the intervention phase. Phase 1: TAU TAU immediately precedes the intervention phase and ranges in length from 6 to 26 months. During TAU, participants will be treated according to usual and customary care, thus establishing each site's baseline rate of suicide risk screening and detection. Usual care at each participating practice involves screening for depression and suicide risk with the PHQ-9. This baseline rate will serve as the control for subsequent study phases. Phase 2: Intervention The intervention phase will range from 11 to 30 months. During this phase, each site will implement the clinical pathway, which includes universal screening for suicide risk using the Ask Suicide-Screening Questions (ASQ), a four-item questionnaire. Those who screen positive will then be evaluated using the Brief Suicide Safety Assessment (BSSA). The BSSA allows the clinician to assess the level of suicide risk to then choose how to proceed with clinical next steps. Clinicians will triage care based on three categories of risk: low-risk, further evaluation needed, and imminent-risk. The risk assessment will determine the type of care the individual then receives (e.g., mental health referrals, mental health evaluation and safety plan, tele-psychiatry crisis intervention consultation, send to ED). We will also conduct in-depth qualitative interviews with a purposive sample of providers (e.g., physicians, nurses, social workers) at each of the participating sites. Purposive sampling is a way to identify and recruit participants that are especially experienced with or knowledgeable about a phenomenon of interest. Provider participants will complete a series of questionnaires at three separate times: 1) prior to clinical pathway training, 2) after training, and 3) 6 months after training. Provider participants will also be trained on the quality improvement process for implementing the intervention strategy. Some provider participants will be asked to complete up to four qualitative interviews that will focus on the implementation process, barriers and facilitators to successful implementation of the clinical pathway, and lessons learned. Phase 3: Sustainability The intervention phase is followed by a 6-month sustainability phase during which the site will work on maintaining or further improving upon the anticipated gains made during the intervention phase. No participants will be recruited during the sustainability phase and this phase will not be used to evaluate intervention effectiveness. The specific aims include: Aim 1: To implement, study, and improve through a Plan-Do-Study-Act (PDSA) cycle, the STARRS-PC intervention that includes suicide risk detection, assessment and triage, and care management based on risk profile. PDSA is a systematic way to test a change that is implemented by breaking down the implementation process into specific steps, and then evaluating the outcome, improving on it, and testing again. Aim 2: Test the effectiveness of STARRS-PC compared to TAU on the primary patient outcome, suicide attempts, secondary patient outcomes (suicidal ideation, non-suicidal self-injury (NSSI), and family satisfaction) at 12 months post-baseline, as well as mediators and moderators, through a stepped wedge design. Hypotheses: STARRS-PC will significantly reduce the rate of: 1) suicide attempts (nonfatal and fatal) and 2) suicidal ideation, NSSI, and improve family satisfaction during the 12-month post-baseline follow-up period compared with TAU. Exploratory Aim 2a: To examine whether increases in provider knowledge, self-efficacy, and buy-in regarding suicide risk screening, assessment, and management will mediate the intervention effect on patient outcomes. Exploratory Aim 2b: To examine whether organization readiness and practice integration will moderate effects of intervention on patient outcomes. Aim 3: To identify barriers to and facilitators of implementation and sustainability of an intervention designed to improve suicide risk detection and risk management using qualitative interviews and surveys with clinical stakeholders at each study site.

Columbia Suicide Severity Rating Scale (C-SSRS) is a semi-structured interview that tracks suicidal ideation and behavior severity over time.

Pierce Suicide Intent Scale (SIS) is a questionnaire that assesses behavior and circumstances surrounding the suicide attempt, including plans, preparation, communicative elements, and concept of lethality of the chosen method.

This measure will be administered to youth only at baseline as well as at 3-, 6-, and 12-months after baseline to track NSSI over time. This is a youth only assessment.

Suicidal Ideation Questionnaire-JR is a 15-item modified version of the SIQ, a 30 item self-report measure of suicidal ideation severity in adolescents, that has been tested in a variety of cultural settings and has been used as an outcome measure in several pediatric treatment studies.

Client Satisfaction Questionnaire-8 (CSQ-8) is an 8-item self-report measure with items scored on a 1-4 scale (1=the lowest satisfaction, 4=the highest satisfaction). The CSQ assesses parent and youth perceptions of treatment acceptability and helpfulness.

AIM (Acceptability of Intervention Measure) is a 4-item measure used to determine the extent the parent and child finds the intervention pathway acceptable.

Inclusion Criteria (Youth and Parent): All patients between the ages of 12 - 17 years and their parent/guardian at time of consent who are identified as being at risk for suicide and receive primary care services at one of 16 participating sites are eligible for study inclusion. Youth will be recruited without regard to current or past histories of mental health problems. Youth with comorbid physical illness (e.g., asthma) and those receiving medication treatment for a comorbid physical or psychiatric condition will be eligible to participate provide they otherwise meet study entry criteria. Inclusion Criteria (Provider): Providers must work with patients at one of the participating PCCs. Exclusion Criteria (Youth and Parent): Subjects will be excluded for being: medically or cognitively unable to participate in study procedures without permanent residence or access to a telephone unable to speak English adequately to understand study procedures Exclusion Criteria (Provider): Unable to give consent Unable to speak English adequately to understand study procedures.

Data will be made available through the National Institute of Mental Health (NIMH) Data Archive (NDA) system.

Web-based Injury Statistics Query and Reporting System (WISQARS) [online]. National Center for Injury Prevention and Control. Accessed October 9, 2020. www.cdc.gov/injury/wisqars

Burns BJ, Costello EJ, Angold A, Tweed D, Stangl D, Farmer EM, Erkanli A. Children's mental health service use across service sectors. Health Aff (Millwood). 1995 Fall;14(3):147-59. doi: 10.1377/hlthaff.14.3.147.

Kessler RC, Berglund P, Borges G, Nock M, Wang PS. Trends in suicide ideation, plans, gestures, and attempts in the United States, 1990-1992 to 2001-2003. JAMA. 2005 May 25;293(20):2487-95. doi: 10.1001/jama.293.20.2487.

Rushton J, Bruckman D, Kelleher K. Primary care referral of children with psychosocial problems. Arch Pediatr Adolesc Med. 2002 Jun;156(6):592-8. doi: 10.1001/archpedi.156.6.592.

Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington (DC): National Academies Press (US); 2001. Available from http://www.ncbi.nlm.nih.gov/books/NBK222274/

Miller CL, Druss B. Datapoints: suicide and access to care. Psychiatr Serv. 2001 Dec;52(12):1566. doi: 10.1176/appi.ps.52.12.1566. No abstract available.

Brent DA, Perper JA, Moritz G, Allman C, Friend A, Roth C, Schweers J, Balach L, Baugher M. Psychiatric risk factors for adolescent suicide: a case-control study. J Am Acad Child Adolesc Psychiatry. 1993 May;32(3):521-9. doi: 10.1097/00004583-199305000-00006.

Shaffer D, Gould MS, Fisher P, Trautman P, Moreau D, Kleinman M, Flory M. Psychiatric diagnosis in child and adolescent suicide. Arch Gen Psychiatry. 1996 Apr;53(4):339-48. doi: 10.1001/archpsyc.1996.01830040075012.

Underwood JM, Brener N, Thornton J, Harris WA, Bryan LN, Shanklin SL, Deputy N, Roberts AM, Queen B, Chyen D, Whittle L, Lim C, Yamakawa Y, Leon-Nguyen M, Kilmer G, Smith-Grant J, Demissie Z, Jones SE, Clayton H, Dittus P. Overview and Methods for the Youth Risk Behavior Surveillance System - United States, 2019. MMWR Suppl. 2020 Aug 21;69(1):1-10. doi: 10.15585/mmwr.su6901a1.

Reinherz HZ, Tanner JL, Berger SR, Beardslee WR, Fitzmaurice GM. Adolescent suicidal ideation as predictive of psychopathology, suicidal behavior, and compromised functioning at age 30. Am J Psychiatry. 2006 Jul;163(7):1226-32. doi: 10.1176/ajp.2006.163.7.1226.

Harrington R, Bredenkamp D, Groothues C, Rutter M, Fudge H, Pickles A. Adult outcomes of childhood and adolescent depression. III. Links with suicidal behaviours. J Child Psychol Psychiatry. 1994 Oct;35(7):1309-19. doi: 10.1111/j.1469-7610.1994.tb01236.x.

Frankenfield DL, Keyl PM, Gielen A, Wissow LS, Werthamer L, Baker SP. Adolescent patients--healthy or hurting? Missed opportunities to screen for suicide risk in the primary care setting. Arch Pediatr Adolesc Med. 2000 Feb;154(2):162-8. doi: 10.1001/archpedi.154.2.162.

Cheung AH, Zuckerbrot RA, Jensen PS, Laraque D, Stein REK; GLAD-PC STEERING GROUP. Guidelines for Adolescent Depression in Primary Care (GLAD-PC): Part II. Treatment and Ongoing Management. Pediatrics. 2018 Mar;141(3):e20174082. doi: 10.1542/peds.2017-4082.

Zuckerbrot RA, Cheung A, Jensen PS, Stein REK, Laraque D; GLAD-PC STEERING GROUP. Guidelines for Adolescent Depression in Primary Care (GLAD-PC): Part I. Practice Preparation, Identification, Assessment, and Initial Management. Pediatrics. 2018 Mar;141(3):e20174081. doi: 10.1542/peds.2017-4081.

Cheung AH, Dewa CS, Levitt AJ, Zuckerbrot RA. Pediatric depressive disorders: management priorities in primary care. Curr Opin Pediatr. 2008 Oct;20(5):551-9. doi: 10.1097/MOP.0b013e32830fe3e3.

Kelleher KJ, Hohmann AA, Larson DB. Prescription of psychotropics to children in office-based practice. Am J Dis Child. 1989 Jul;143(7):855-9. doi: 10.1001/archpedi.1989.02150190105033.

Vannoy SD, Tai-Seale M, Duberstein P, Eaton LJ, Cook MA. Now what should I do? Primary care physicians' responses to older adults expressing thoughts of suicide. J Gen Intern Med. 2011 Sep;26(9):1005-11. doi: 10.1007/s11606-011-1726-5. Epub 2011 May 4.

Vannoy SD, Robins LS. Suicide-related discussions with depressed primary care patients in the USA: gender and quality gaps. A mixed methods analysis. BMJ Open. 2011 Jan 1;1(2):e000198. doi: 10.1136/bmjopen-2011-000198.

Schwartz-Lifshitz M, Zalsman G, Giner L, Oquendo MA. Can we really prevent suicide? Curr Psychiatry Rep. 2012 Dec;14(6):624-33. doi: 10.1007/s11920-012-0318-3.

O'Connor E, Gaynes BN, Burda BU, Soh C, Whitlock EP. Screening for and treatment of suicide risk relevant to primary care: a systematic review for the U.S. Preventive Services Task Force. Ann Intern Med. 2013 May 21;158(10):741-54. doi: 10.7326/0003-4819-158-10-201305210-00642.

McDowell AK, Lineberry TW, Bostwick JM. Practical suicide-risk management for the busy primary care physician. Mayo Clin Proc. 2011 Aug;86(8):792-800. doi: 10.4065/mcp.2011.0076. Epub 2011 Jun 27.

LeFevre ML; U.S. Preventive Services Task Force. Screening for suicide risk in adolescents, adults, and older adults in primary care: U.S. Preventive Services Task Force recommendation statement. Ann Intern Med. 2014 May 20;160(10):719-26. doi: 10.7326/M14-0589.

Raue PJ, Ghesquiere AR, Bruce ML. Suicide risk in primary care: identification and management in older adults. Curr Psychiatry Rep. 2014 Sep;16(9):466. doi: 10.1007/s11920-014-0466-8.

Bower P, Gilbody S. Stepped care in psychological therapies: access, effectiveness and efficiency. Narrative literature review. Br J Psychiatry. 2005 Jan;186:11-7. doi: 10.1192/bjp.186.1.11.

Asarnow JR, Jaycox LH, Duan N, LaBorde AP, Rea MM, Murray P, Anderson M, Landon C, Tang L, Wells KB. Effectiveness of a quality improvement intervention for adolescent depression in primary care clinics: a randomized controlled trial. JAMA. 2005 Jan 19;293(3):311-9. doi: 10.1001/jama.293.3.311.

Richardson LP, Ludman E, McCauley E, Lindenbaum J, Larison C, Zhou C, Clarke G, Brent D, Katon W. Collaborative care for adolescents with depression in primary care: a randomized clinical trial. JAMA. 2014 Aug 27;312(8):809-16. doi: 10.1001/jama.2014.9259.

Hogan MF, Grumet JG. Suicide Prevention: An Emerging Priority For Health Care. Health Aff (Millwood). 2016 Jun 1;35(6):1084-90. doi: 10.1377/hlthaff.2015.1672.

Asarnow JR, Jaycox LH, Tang L, Duan N, LaBorde AP, Zeledon LR, Anderson M, Murray PJ, Landon C, Rea MM, Wells KB. Long-term benefits of short-term quality improvement interventions for depressed youths in primary care. Am J Psychiatry. 2009 Sep;166(9):1002-10. doi: 10.1176/appi.ajp.2009.08121909. Epub 2009 Aug 3.

Wells KB, Sherbourne C, Schoenbaum M, Duan N, Meredith L, Unutzer J, Miranda J, Carney MF, Rubenstein LV. Impact of disseminating quality improvement programs for depression in managed primary care: a randomized controlled trial. JAMA. 2000 Jan 12;283(2):212-20. doi: 10.1001/jama.283.2.212. Erratum In: JAMA 2000 Jun 28;283(24):3204.

Wells K, Sherbourne C, Schoenbaum M, Ettner S, Duan N, Miranda J, Unutzer J, Rubenstein L. Five-year impact of quality improvement for depression: results of a group-level randomized controlled trial. Arch Gen Psychiatry. 2004 Apr;61(4):378-86. doi: 10.1001/archpsyc.61.4.378.

Kemper AR, Hostutler CA, Beck K, Fontanella CA, Bridge JA. Depression and Suicide-Risk Screening Results in Pediatric Primary Care. Pediatrics. 2021 Jul;148(1):e2021049999. doi: 10.1542/peds.2021-049999. Epub 2021 Jun 7.

Improvement IfH. How to Improve. Accessed October 18, 2020. http://www.ihi.org/resources/Pages/HowtoImprove/default.aspx

Horowitz LM, Bridge JA, Teach SJ, Ballard E, Klima J, Rosenstein DL, Wharff EA, Ginnis K, Cannon E, Joshi P, Pao M. Ask Suicide-Screening Questions (ASQ): a brief instrument for the pediatric emergency department. Arch Pediatr Adolesc Med. 2012 Dec;166(12):1170-6. doi: 10.1001/archpediatrics.2012.1276.

Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. 2nd ed. Jossey-Bass 2009.

Crowl A, Sharma A, Sorge L, Sorensen T. Accelerating quality improvement within your organization: Applying the Model for Improvement. J Am Pharm Assoc (2003). 2015 Jul-Aug;55(4):e364-74; quiz e375-6. doi: 10.1331/JAPhA.2015.15533.

Harwayne-Gidansky I, Bellis JM, McLaren SH, et al. Mannequin-Based Immersive Simulation Improves Resident Understanding of a Clinical Decision Rule. Simulation & Gaming. 2017;48(5):657-669. doi:10.1177/1046878117719483

Adams AJ, Wasson EA, Admire JR, Pablo Gomez P, Babayeuski RA, Sako EY, Willis RE. A Comparison of Teaching Modalities and Fidelity of Simulation Levels in Teaching Resuscitation Scenarios. J Surg Educ. 2015 Sep-Oct;72(5):778-85. doi: 10.1016/j.jsurg.2015.04.011. Epub 2015 May 20.

Fallucco EM, Seago RD, Cuffe SP, Kraemer DF, Wysocki T. Primary care provider training in screening, assessment, and treatment of adolescent depression. Acad Pediatr. 2015 May-Jun;15(3):326-32. doi: 10.1016/j.acap.2014.12.004. Epub 2015 Mar 29.

Ventre KM, Barry JS, Davis D, Baiamonte VL, Wentworth AC, Pietras M, Coughlin L, Barley G. Using in situ simulation to evaluate operational readiness of a children's hospital-based obstetrics unit. Simul Healthc. 2014 Apr;9(2):102-11. doi: 10.1097/SIH.0000000000000005.

Yager P, Collins C, Blais C, O'Connor K, Donovan P, Martinez M, Cummings B, Hartnick C, Noviski N. Quality improvement utilizing in-situ simulation for a dual-hospital pediatric code response team. Int J Pediatr Otorhinolaryngol. 2016 Sep;88:42-6. doi: 10.1016/j.ijporl.2016.06.026. Epub 2016 Jun 7.

Wallin K, Kelly F, Sembera K. Building High Reliability Through Simulation. In: Oster CA, Braaten JS, eds. High Reliability Organizations: A Healthcare Handbook for Patient Safety & Quality. 2 ed. Sigma Theta Tau International; 2021.

Brahmbhatt K, Kurtz BP, Afzal KI, Giles LL, Kowal ED, Johnson KP, Lanzillo E, Pao M, Plioplys S, Horowitz LM; PaCC Workgroup. Suicide Risk Screening in Pediatric Hospitals: Clinical Pathways to Address a Global Health Crisis. Psychosomatics. 2019 Jan-Feb;60(1):1-9. doi: 10.1016/j.psym.2018.09.003. Epub 2018 Sep 22.

Aguinaldo LD, Sullivant S, Lanzillo EC, Ross A, He JP, Bradley-Ewing A, Bridge JA, Horowitz LM, Wharff EA. Validation of the ask suicide-screening questions (ASQ) with youth in outpatient specialty and primary care clinics. Gen Hosp Psychiatry. 2021 Jan-Feb;68:52-58. doi: 10.1016/j.genhosppsych.2020.11.006. Epub 2020 Nov 13.

PhenX Toolkit: Mental Health Research Core Tier 1 Demographic and Social Environment Measures. RTI International. Accessed October 4, 2020. https://www.phenxtoolkit.org/sub-collections/view/1

Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001 Sep;16(9):606-13. doi: 10.1046/j.1525-1497.2001.016009606.x.

Cameron IM, Crawford JR, Lawton K, Reid IC. Psychometric comparison of PHQ-9 and HADS for measuring depression severity in primary care. Br J Gen Pract. 2008 Jan;58(546):32-6. doi: 10.3399/bjgp08X263794.

Posner K, Brent D, Lucas C, et al. Columbia-Suicide Severity Rating Scale (C-SSRS) New York State Psychiatric Institute https://cssrs.columbia.edu/wp-content/uploads/C-SSRS_Pediatric-SLC_11.14.16.pdf

Pierce DW. Suicidal intent in self-injury. Br J Psychiatry. 1977 Apr;130:377-85. doi: 10.1192/bjp.130.4.377.

Brent D, Emslie G, Clarke G, Wagner KD, Asarnow JR, Keller M, Vitiello B, Ritz L, Iyengar S, Abebe K, Birmaher B, Ryan N, Kennard B, Hughes C, DeBar L, McCracken J, Strober M, Suddath R, Spirito A, Leonard H, Melhem N, Porta G, Onorato M, Zelazny J. Switching to another SSRI or to venlafaxine with or without cognitive behavioral therapy for adolescents with SSRI-resistant depression: the TORDIA randomized controlled trial. JAMA. 2008 Feb 27;299(8):901-913. doi: 10.1001/jama.299.8.901. Erratum In: JAMA. 2019 Nov 5;322(17):1718.

March J, Silva S, Petrycki S, Curry J, Wells K, Fairbank J, Burns B, Domino M, McNulty S, Vitiello B, Severe J; Treatment for Adolescents With Depression Study (TADS) Team. Fluoxetine, cognitive-behavioral therapy, and their combination for adolescents with depression: Treatment for Adolescents With Depression Study (TADS) randomized controlled trial. JAMA. 2004 Aug 18;292(7):807-20. doi: 10.1001/jama.292.7.807.

Nguyen TD, Attkisson CC, Stegner BL. Assessment of patient satisfaction: development and refinement of a service evaluation questionnaire. Eval Program Plann. 1983;6(3-4):299-313. doi: 10.1016/0149-7189(83)90010-1.

Horwitz SM, Hoagwood K, Stiffman AR, Summerfeld T, Weisz JR, Costello EJ, Rost K, Bean DL, Cottler L, Leaf PJ, Roper M, Norquist G. Reliability of the services assessment for children and adolescents. Psychiatr Serv. 2001 Aug;52(8):1088-94. doi: 10.1176/appi.ps.52.8.1088.

Stiffman AR, Horwitz SM, Hoagwood K, Compton W 3rd, Cottler L, Bean DL, Narrow WE, Weisz JR. The Service Assessment for Children and Adolescents (SACA): adult and child reports. J Am Acad Child Adolesc Psychiatry. 2000 Aug;39(8):1032-9. doi: 10.1097/00004583-200008000-00019.

Association AP. DSM-5-TR Online Assessment Measures. Accessed July 21, 2022. https://www.psychiatry.org/psychiatrists/practice/dsm/educational-resources/assessment-measures

Chorpita BFS, Susan H. Child Depression and Anxiety RCADS 25 Accessed July 22, 2022. https://help.greenspacehealth.com/article/50-child-depression-and-anxiety-rcads-25

Herron J, Ticehurst H, Appleby L, Perry A, Cordingley L. Attitudes toward suicide prevention in front-line health staff. Suicide Life Threat Behav. 2001 Fall;31(3):342-7. doi: 10.1521/suli.31.3.342.24252.

Gamarra JM, Luciano MT, Gradus JL, Wiltsey Stirman S. Assessing Variability and Implementation Fidelity of Suicide Prevention Safety Planning in a Regional VA Healthcare System. Crisis. 2015;36(6):433-9. doi: 10.1027/0227-5910/a000345.

Hoekstra F, van Offenbeek MAG, Dekker R, Hettinga FJ, Hoekstra T, van der Woude LHV, van der Schans CP; ReSpAct group. Implementation fidelity trajectories of a health promotion program in multidisciplinary settings: managing tensions in rehabilitation care. Implement Sci. 2017 Dec 1;12(1):143. doi: 10.1186/s13012-017-0667-8.

Mullin DJ, Hargreaves L, Auxier A, Brennhofer SA, Hitt JR, Kessler RS, Littenberg B, Macchi CR, Martin M, Rose G, Trembath F, van Eeghen C. Measuring the integration of primary care and behavioral health services. Health Serv Res. 2019 Apr;54(2):379-389. doi: 10.1111/1475-6773.13117. Epub 2019 Feb 6.

Kessler RS, Auxier A, Hitt JR, Macchi CR, Mullin D, van Eeghen C, Littenberg B. Development and validation of a measure of primary care behavioral health integration. Fam Syst Health. 2016 Dec;34(4):342-356. doi: 10.1037/fsh0000227. Epub 2016 Oct 13.

Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: a psychometric assessment of a new measure. Implement Sci. 2014 Jan 10;9:7. doi: 10.1186/1748-5908-9-7.

Armat MR, Assarroudi A, Rad M, Sharifi H, Heydari A. Inductive and Deductive: Ambiguous Labels in Qualitative Content Analysis. The Qualitative Report. 2018;doi:10.46743/2160-3715/2018.2872

Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008 Apr;34(4):228-43. doi: 10.1016/s1553-7250(08)34030-6.

McCreight MS, Rabin BA, Glasgow RE, Ayele RA, Leonard CA, Gilmartin HM, Frank JW, Hess PL, Burke RE, Battaglia CT. Using the Practical, Robust Implementation and Sustainability Model (PRISM) to qualitatively assess multilevel contextual factors to help plan, implement, evaluate, and disseminate health services programs. Transl Behav Med. 2019 Nov 25;9(6):1002-1011. doi: 10.1093/tbm/ibz085.

Constas MA. Qualitative Analysis as a Public Event: The Documentation of Category Development Procedures. American Educational Research Journal. 1992;29(2):253-266. doi:10.3102/00028312029002253

Hemming K, Taljaard M, Grimshaw J. Introducing the new CONSORT extension for stepped-wedge cluster randomised trials. Trials. 2019 Jan 18;20(1):68. doi: 10.1186/s13063-018-3116-3.

Foster ED, Deardorff A. Open Science Framework (OSF). Journal of the Medical Library Association. 2017;105(2)doi:10.5195/jmla.2017.88

Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007 Feb;28(2):182-91. doi: 10.1016/j.cct.2006.05.007. Epub 2006 Jul 7.

Grambsch PM, Therneau TM. Proportional Hazards Tests and Diagnostics Based on Weighted Residuals. Biometrika. 1994;81(3):515-526. doi:10.2307/2337123

Flury BK, Riedwyl H. Standard Distance in Univariate and Multivariate Analysis. The American Statistician. 1986;40(3):249-251. doi:10.1080/00031305.1986.10475403

Austin PC. Using the Standardized Difference to Compare the Prevalence of a Binary Variable Between Two Groups in Observational Research. Communications in Statistics - Simulation and Computation. 2009;38(6):1228-1234. doi:10.1080/03610910902859574

Tingley D, Yamamoto T, Hirose K, Keele L, Imai K. mediation: R Package for Causal Mediation Analysis. Journal of Statistical Software. 2014;59(5):1 - 38. doi:10.18637/jss.v059.i05

Sales A. Review: mediation Package in R. Journal of Educational and Behavioral Statistics. 2016;42doi:10.3102/1076998616670371

Jakobsen JC, Gluud C, Wetterslev J, Winkel P. When and how should multiple imputation be used for handling missing data in randomised clinical trials - a practical guide with flowcharts. BMC Med Res Methodol. 2017 Dec 6;17(1):162. doi: 10.1186/s12874-017-0442-1.

Xie H, Gao W, Xing B, Heitjan DF, Hedeker D, Yuan C. Measuring the Impact of Nonignorable Missingness Using the R Package isni. Comput Methods Programs Biomed. 2018 Oct;164:207-220. doi: 10.1016/j.cmpb.2018.06.014. Epub 2018 Jul 4.

Nock MK, Green JG, Hwang I, McLaughlin KA, Sampson NA, Zaslavsky AM, Kessler RC. Prevalence, correlates, and treatment of lifetime suicidal behavior among adolescents: results from the National Comorbidity Survey Replication Adolescent Supplement. JAMA Psychiatry. 2013 Mar;70(3):300-10. doi: 10.1001/2013.jamapsychiatry.55.

Baio G, Copas A, Ambler G, Hargreaves J, Beard E, Omar RZ. Sample size calculation for a stepped wedge trial. Trials. 2015 Aug 17;16:354. doi: 10.1186/s13063-015-0840-9.

Agency for Healthcare Research and Quality (AHRQ). https://www.ahrq.gov/health-literacy/improve/precautions/tool2b.html#:~:text=The%20Plan%2DDo%2DStudy%2D,on%20it%2C%20and%20testing%20again

Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Adm Policy Ment Health. 2015 Sep;42(5):533-44. doi: 10.1007/s10488-013-0528-y.