Effectiveness and Implementation of mPATH-CRC

Effectiveness and Implementation of mPATH-CRC: a Mobile Health System for Colorectal Cancer Screening

Study Investigators are conducting this study to learn how to best implement a new iPad program in clinical practice.

The study team has developed mPATH-CRC (mobile PAtient Technology for Health-Colorectal Cancer), a patient-friendly iPad program used by individuals immediately before a routine primary care visit. To fully realize mPATH-CRC's potential to decrease CRC mortality, the program now must be implemented in primary care practices in a way that encourages routine and sustained use. However, while hundreds of mobile health (mHealth) tools have been developed in recent years, the optimal strategies for implementing and maintaining mHealth interventions in clinical practice are unknown. This study will compare the results of a "high touch" strategy to a "low touch" strategy using a Type III hybrid design and incorporating mixed methods to evaluate implementation, maintenance, and effectiveness of mPATH-CRC in a diverse sample of community-based practices. The study will be conducted in three phases: 1) in a cluster-randomized controlled trial of 22 primary care clinics, the study team will compare the implementation outcomes of a "high touch" evidence-based mHealth implementation strategy with a "low touch" implementation strategy; 2) in a nested pragmatic study, the study team will estimate the effect of mPATH-CRC on completion of CRC screening within 16 weeks of a clinic visit; and 3) by surveying and interviewing clinic staff and providers after implementation is complete, the study team will determine the factors that facilitate or impede the maintenance of mHealth interventions. This record refers to the cluster-randomized controlled trial of 22 primary care clinics.

English-speaking patients aged 50-74 who are seen in the study clinics randomized to mPATH-CRC utilizing the "high touch" Implementation strategy.

English-speaking patients aged 50-74 who are seen in the study clinics randomized to mPATH-CRC utilizing the "low touch" Implementation Strategy

Clinic personnel (e.g., administrators, nurses, providers) who are involved with the implementation of mPATH-CRC, in the study clinics randomized to mPATH-CRC utilizing the "high touch" Implementation strategy.

Clinic personnel (e.g., administrators, nurses, providers) who are involved with the implementation of mPATH-CRC, in the study clinics randomized to mPATH-CRC utilizing the "low touch" Implementation Strategy

mPATH-CRC is a self-administered iPad program that patients use in primary care clinics to help them receive CRC screening. The mPATH-CRC program also includes health questions to assist clinics with patient check-in, thereby incentivizing its use for all patients.

The "high touch" strategy consists of pre-implementation activities, training, and ongoing support. Pre-Implementation Activities Clinic champion identified. Study team meeting with clinic champion Implementation adaptations as needed for clinic flow Implementation Kick-Off (Day 1) • On-site training with key clinic personnel Months 1 - 6 Phone/email technical support, as needed. Access to web-based QA dashboard Monthly program usage report sent to clinic champions Scheduled phone-calls with clinic champion to review QA data and explore potential barriers. Implementation adaptations as needed for clinic flow Goal-triggered follow-up on-site trainings Additional on-site trainings as requested. Months 7 - 12 Phone/email technical support, as needed Access to web-based QA dashboard

Clinics randomized to receive the low touch implementation strategy will receive: Pre-Implementation Activities • N/A Implementation Kick-Off (Day 1) • On-site training with key clinic personnel Months 1 - 6 Phone/email technical support, as needed. Access to web-based QA dashboard Months 7 - 12 Phone/email technical support, as needed Access to web-based QA dashboard

mPATH-CRC Implementation: Proportion of all eligible patients, ages 50 - 74, who complete the mPATH-CRC program in the 6th month following the implementation date.

mPATH-CRC Reach: The proportion of patients, ages 50 - 74, who are given mPATH-CRC or have risk factors identified by mPATH-CheckIn in months 1-6 by varying socioeconomic strata (Describe strata)

The mean usage of mPATH-CRC among staff and providers over the first 6 months following implementation; usage is calculated for each staff/provider as the proportion of times mPATH-CRC is completed out of the total times mPATH-CRC should have been launched.

The proportion of patients aged 18 or older who complete mPATH-CheckIn in months 1-6; this outcome will be calculated overall and within socioeconomic strata

The mean usage of mPATH-CheckIn among staff and providers over the first 6 months following implementation; usage is calculated for front desk staff as the proportion of times mPATH-CheckIn is completed out of the total times mPATH-CheckIn should have been handed out; usage is calculated for nurses/providers as the proportion of times mPATH-CheckIn is completed and data is transmitted to the EHR out of the total times mPATH-CheckIn should have been handed out

The proportion of patients who use mPATH-CRC and request a CRC screening test who have a test ordered or have the order dismissed (i.e., "self-order" feature is used as designed) in months 1-6

The proportion of patients aged 50-74 who are eligible for CRC screening who complete mPATH-CRC or have risk factors identified by mPATH-CheckIn in months 7-12

The outcome is defined as the proportion of patients aged 50-74 who are eligible for CRC screening who have a CRC screening test ordered (colonoscopy, flexible sigmoidoscopy, fecal testing for blood, or fecal DNA testing) within 16 weeks of their index visit to the clinic. This outcome will also be compared between the pre- and post-implementation cohorts.

The proportion of patients aged 50-74 who are eligible for CRC screening who complete CRC screening within 16 weeks of their index visit to the clinic. Effectiveness is determined by comparing the proportion who complete screening in a pre-implementation cohort (months 12 - 4 before implementation) to a post-implementation cohort (months 1 - 8 after implementation).

These will be identified through semi-structured interviews. Interviews will explore how mPATH-CRC was incorporated in the clinic's work flow and factors that affected maintenance such as intervention adaptations, organizational characteristics, and the champion's role. Interviews will be conducted with four members of each selected clinic: the clinic champion, one clinician, one front desk team member, and one medical assistant/nursing team member.

The proportion of patients aged 50-74 who are eligible for CRC screening who complete mPATH-CRC or have risk factors identified by mPATH-CheckIn in months 1-5 following implementation

The Acceptability of Intervention Measure (AIM) is a 4-item measure scored on a 5-point scale and summed. The Range of Scores is from 4 to 20. Higher Scores indicate higher acceptability.

The Intervention Appropriateness Measure (IAM) is a 4-item measure scored on a 5-point scale and summed. The Range of Scores is from 4 to 20. Higher Scores indicate higher appropriateness.

The Feasibility of Intervention Measure (FIM) is a 4-item measure scored on a 5-point scale and summed. The Range of Scores is from 4 to 20. Higher Scores indicate higher feasibility.

The Acceptability of Intervention Measure (AIM) is a 4-item measure scored on a 5-point scale and summed. The Range of Scores is from 4 to 20. Higher Scores indicate higher acceptability.

The Intervention Appropriateness Measure (IAM) is a 4-item measure scored on a 5-point scale and summed. The Range of Scores is from 4 to 20. Higher Scores indicate higher appropriateness.

The Feasibility of Intervention Measure (FIM) is a 4-item measure scored on a 5-point scale and summed. The Range of Scores is from 4 to 20. Higher Scores indicate higher feasibility.

This study will include three distinct populations of participants: 1) healthcare providers and staff at primary care practices, 2) patients aged 18 and older seen in the participating study sites, and 3) patients aged 50-74 seen in the participating study sites who are eligible for CRC screening Patient Inclusion Criteria: Due for routine CRC screening, defined as: No colonoscopy within the prior 10 years No flexible sigmoidoscopy within the prior 5 years No CT colonography within the prior 5 years No fecal DNA testing within the prior 3 years No fecal blood testing (guaiac-based test with home kit or fecal immunochemical test) within the prior 12 months Patient Exclusion Criteria: Personal history of CRC First degree relative with CRC Personal history of colorectal polyps

We will share de-identified participant data that underlie the results reported in peer-reviewed publications (text, tables, figures, and appendices).

Investigators who provide a methodologically sound proposal to use the participant data in a meta-analysis.

Sharp L, Tilson L, Whyte S, O'Ceilleachair A, Walsh C, Usher C, Tappenden P, Chilcott J, Staines A, Barry M, Comber H. Cost-effectiveness of population-based screening for colorectal cancer: a comparison of guaiac-based faecal occult blood testing, faecal immunochemical testing and flexible sigmoidoscopy. Br J Cancer. 2012 Feb 28;106(5):805-16. doi: 10.1038/bjc.2011.580. Epub 2012 Feb 16.

Lin JS, Piper MA, Perdue LA, Rutter CM, Webber EM, O'Connor E, Smith N, Whitlock EP. Screening for Colorectal Cancer: Updated Evidence Report and Systematic Review for the US Preventive Services Task Force. JAMA. 2016 Jun 21;315(23):2576-94. doi: 10.1001/jama.2016.3332. Erratum In: JAMA. 2016 Aug 2;316(5):545. JAMA. 2016 Oct 4;316(13):1412.

Knudsen AB, Zauber AG, Rutter CM, Naber SK, Doria-Rose VP, Pabiniak C, Johanson C, Fischer SE, Lansdorp-Vogelaar I, Kuntz KM. Estimation of Benefits, Burden, and Harms of Colorectal Cancer Screening Strategies: Modeling Study for the US Preventive Services Task Force. JAMA. 2016 Jun 21;315(23):2595-609. doi: 10.1001/jama.2016.6828.

Inadomi JM. Screening for Colorectal Neoplasia. N Engl J Med. 2017 Jan 12;376(2):149-156. doi: 10.1056/NEJMcp1512286. No abstract available.

US Preventive Services Task Force; Bibbins-Domingo K, Grossman DC, Curry SJ, Davidson KW, Epling JW Jr, Garcia FAR, Gillman MW, Harper DM, Kemper AR, Krist AH, Kurth AE, Landefeld CS, Mangione CM, Owens DK, Phillips WR, Phipps MG, Pignone MP, Siu AL. Screening for Colorectal Cancer: US Preventive Services Task Force Recommendation Statement. JAMA. 2016 Jun 21;315(23):2564-2575. doi: 10.1001/jama.2016.5989. Erratum In: JAMA. 2016 Aug 2;316(5):545. JAMA. 2017 Jun 6;317(21):2239.

Qaseem A, Denberg TD, Hopkins RH Jr, Humphrey LL, Levine J, Sweet DE, Shekelle P; Clinical Guidelines Committee of the American College of Physicians. Screening for colorectal cancer: a guidance statement from the American College of Physicians. Ann Intern Med. 2012 Mar 6;156(5):378-86. doi: 10.7326/0003-4819-156-5-201203060-00010. Erratum In: Ann Intern Med. 2012 Jul 17;157(2):152.

Smith RA, Andrews K, Brooks D, DeSantis CE, Fedewa SA, Lortet-Tieulent J, Manassaram-Baptiste D, Brawley OW, Wender RC. Cancer screening in the United States, 2016: A review of current American Cancer Society guidelines and current issues in cancer screening. CA Cancer J Clin. 2016 Mar-Apr;66(2):96-114. doi: 10.3322/caac.21336. Epub 2016 Jan 21.

Rex DK, Boland CR, Dominitz JA, Giardiello FM, Johnson DA, Kaltenbach T, Levin TR, Lieberman D, Robertson DJ. Colorectal Cancer Screening: Recommendations for Physicians and Patients from the U.S. Multi-Society Task Force on Colorectal Cancer. Am J Gastroenterol. 2017 Jul;112(7):1016-1030. doi: 10.1038/ajg.2017.174. Epub 2017 Jun 6.

White A, Thompson TD, White MC, Sabatino SA, de Moor J, Doria-Rose PV, Geiger AM, Richardson LC. Cancer Screening Test Use - United States, 2015. MMWR Morb Mortal Wkly Rep. 2017 Mar 3;66(8):201-206. doi: 10.15585/mmwr.mm6608a1.

Gupta S, Sussman DA, Doubeni CA, Anderson DS, Day L, Deshpande AR, Elmunzer BJ, Laiyemo AO, Mendez J, Somsouk M, Allison J, Bhuket T, Geng Z, Green BB, Itzkowitz SH, Martinez ME. Challenges and possible solutions to colorectal cancer screening for the underserved. J Natl Cancer Inst. 2014 Apr;106(4):dju032. doi: 10.1093/jnci/dju032. Epub 2014 Mar 28.

Yao N, Alcala HE, Anderson R, Balkrishnan R. Cancer Disparities in Rural Appalachia: Incidence, Early Detection, and Survivorship. J Rural Health. 2017 Sep;33(4):375-381. doi: 10.1111/jrh.12213. Epub 2016 Sep 7.

Doubeni CA, Laiyemo AO, Major JM, Schootman M, Lian M, Park Y, Graubard BI, Hollenbeck AR, Sinha R. Socioeconomic status and the risk of colorectal cancer: an analysis of more than a half million adults in the National Institutes of Health-AARP Diet and Health Study. Cancer. 2012 Jul 15;118(14):3636-44. doi: 10.1002/cncr.26677. Epub 2012 Jan 3. Erratum In: Cancer. 2013 Jan 15;119(2):467-9.

James TM, Greiner KA, Ellerbeck EF, Feng C, Ahluwalia JS. Disparities in colorectal cancer screening: a guideline-based analysis of adherence. Ethn Dis. 2006 Winter;16(1):228-33.

Morgan JW, Cho MM, Guenzi CD, Jackson C, Mathur A, Natto Z, Kazanjian K, Tran H, Shavlik D, Lum SS. Predictors of delayed-stage colorectal cancer: are we neglecting critical demographic information? Ann Epidemiol. 2011 Dec;21(12):914-21. doi: 10.1016/j.annepidem.2011.09.002. Epub 2011 Oct 13.

Siegel R, Desantis C, Jemal A. Colorectal cancer statistics, 2014. CA Cancer J Clin. 2014 Mar-Apr;64(2):104-17. doi: 10.3322/caac.21220. Epub 2014 Mar 17.

Miller DP Jr, Brownlee CD, McCoy TP, Pignone MP. The effect of health literacy on knowledge and receipt of colorectal cancer screening: a survey study. BMC Fam Pract. 2007 Mar 30;8:16. doi: 10.1186/1471-2296-8-16.

Institute of Medicine (US) Committee on Health Literacy; Nielsen-Bohlman L, Panzer AM, Kindig DA, editors. Health Literacy: A Prescription to End Confusion. Washington (DC): National Academies Press (US); 2004. Available from http://www.ncbi.nlm.nih.gov/books/NBK216032/

Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011 Jul 19;155(2):97-107. doi: 10.7326/0003-4819-155-2-201107190-00005.

Oldach BR, Katz ML. Health literacy and cancer screening: a systematic review. Patient Educ Couns. 2014 Feb;94(2):149-57. doi: 10.1016/j.pec.2013.10.001. Epub 2013 Oct 14.

Beydoun HA, Beydoun MA. Predictors of colorectal cancer screening behaviors among average-risk older adults in the United States. Cancer Causes Control. 2008 May;19(4):339-59. doi: 10.1007/s10552-007-9100-y. Epub 2007 Dec 18.

Weinberg DS, Turner BJ, Wang H, Myers RE, Miller S. A survey of women regarding factors affecting colorectal cancer screening compliance. Prev Med. 2004 Jun;38(6):669-75. doi: 10.1016/j.ypmed.2004.02.015.

Berkowitz Z, Hawkins NA, Peipins LA, White MC, Nadel MR. Beliefs, risk perceptions, and gaps in knowledge as barriers to colorectal cancer screening in older adults. J Am Geriatr Soc. 2008 Feb;56(2):307-14. doi: 10.1111/j.1532-5415.2007.01547.x. Epub 2007 Dec 7.

Farmer MM, Bastani R, Kwan L, Belman M, Ganz PA. Predictors of colorectal cancer screening from patients enrolled in a managed care health plan. Cancer. 2008 Mar 15;112(6):1230-8. doi: 10.1002/cncr.23290.

Wee CC, McCarthy EP, Phillips RS. Factors associated with colon cancer screening: the role of patient factors and physician counseling. Prev Med. 2005 Jul;41(1):23-9. doi: 10.1016/j.ypmed.2004.11.004. Epub 2004 Dec 29.

Klabunde CN, Schenck AP, Davis WW. Barriers to colorectal cancer screening among Medicare consumers. Am J Prev Med. 2006 Apr;30(4):313-9. doi: 10.1016/j.amepre.2005.11.006.

Seeff LC, Nadel MR, Klabunde CN, Thompson T, Shapiro JA, Vernon SW, Coates RJ. Patterns and predictors of colorectal cancer test use in the adult U.S. population. Cancer. 2004 May 15;100(10):2093-103. doi: 10.1002/cncr.20276.

Ayres CG, Griffith HM. Perceived barriers to and facilitators of the implementation of priority clinical preventive services guidelines. Am J Manag Care. 2007 Mar;13(3):150-5.

Yarnall KS, Pollak KI, Ostbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention? Am J Public Health. 2003 Apr;93(4):635-41. doi: 10.2105/ajph.93.4.635.

Ostbye T, Yarnall KS, Krause KM, Pollak KI, Gradison M, Michener JL. Is there time for management of patients with chronic diseases in primary care? Ann Fam Med. 2005 May-Jun;3(3):209-14. doi: 10.1370/afm.310.

Katz ML, Broder-Oldach B, Fisher JL, King J, Eubanks K, Fleming K, Paskett ED. Patient-provider discussions about colorectal cancer screening: who initiates elements of informed decision making? J Gen Intern Med. 2012 Sep;27(9):1135-41. doi: 10.1007/s11606-012-2045-1. Epub 2012 Apr 5.

Zapka JM, Klabunde CN, Arora NK, Yuan G, Smith JL, Kobrin SC. Physicians' colorectal cancer screening discussion and recommendation patterns. Cancer Epidemiol Biomarkers Prev. 2011 Mar;20(3):509-21. doi: 10.1158/1055-9965.EPI-10-0749. Epub 2011 Jan 14.

Anhang Price R, Zapka J, Edwards H, Taplin SH. Organizational factors and the cancer screening process. J Natl Cancer Inst Monogr. 2010;2010(40):38-57. doi: 10.1093/jncimonographs/lgq008.

Taplin SH, Anhang Price R, Edwards HM, Foster MK, Breslau ES, Chollette V, Prabhu Das I, Clauser SB, Fennell ML, Zapka J. Introduction: Understanding and influencing multilevel factors across the cancer care continuum. J Natl Cancer Inst Monogr. 2012 May;2012(44):2-10. doi: 10.1093/jncimonographs/lgs008.

Miller DP Jr, Weaver KE, Case LD, Babcock D, Lawler D, Denizard-Thompson N, Pignone MP, Spangler JG. Usability of a Novel Mobile Health iPad App by Vulnerable Populations. JMIR Mhealth Uhealth. 2017 Apr 11;5(4):e43. doi: 10.2196/mhealth.7268.

Orr JA, King RJ. Mobile phone SMS messages can enhance healthy behaviour: a meta-analysis of randomised controlled trials. Health Psychol Rev. 2015;9(4):397-416. doi: 10.1080/17437199.2015.1022847. Epub 2015 May 28.

Hall AK, Cole-Lewis H, Bernhardt JM. Mobile text messaging for health: a systematic review of reviews. Annu Rev Public Health. 2015 Mar 18;36:393-415. doi: 10.1146/annurev-publhealth-031914-122855.

Ostrander RE, Thompson HJ, Demiris G. Using targeted messaging to increase physical activity in older adults: a review. J Gerontol Nurs. 2014 Sep;40(9):36-48. doi: 10.3928/00989134-20140324-03. Epub 2014 Mar 31.

Vodopivec-Jamsek V, de Jongh T, Gurol-Urganci I, Atun R, Car J. Mobile phone messaging for preventive health care. Cochrane Database Syst Rev. 2012 Dec 12;12(12):CD007457. doi: 10.1002/14651858.CD007457.pub2.

Hawley S, Lillie S, Cooper G, Elston Lafata J. Managed care patients' preferences, physician recommendations, and colon cancer screening. Am J Manag Care. 2014 Jul;20(7):555-61.

Inadomi JM, Vijan S, Janz NK, Fagerlin A, Thomas JP, Lin YV, Munoz R, Lau C, Somsouk M, El-Nachef N, Hayward RA. Adherence to colorectal cancer screening: a randomized clinical trial of competing strategies. Arch Intern Med. 2012 Apr 9;172(7):575-82. doi: 10.1001/archinternmed.2012.332.

Miller DP Jr, Spangler JG, Case LD, Goff DC Jr, Singh S, Pignone MP. Effectiveness of a web-based colorectal cancer screening patient decision aid: a randomized controlled trial in a mixed-literacy population. Am J Prev Med. 2011 Jun;40(6):608-15. doi: 10.1016/j.amepre.2011.02.019.

Miller DP, Denizard-Thompson N, Case LD, Weaver KE, Pignone MP, Lawler D, Troyer J, Spangler JG. The effect of a mobile health intervention (mPATH-CRC) on colorectal cancer screening in a diverse patient population. J Gen Intern Med. 2017 Apr;32(2 Supplement):S338

Sabatino SA, White MC, Thompson TD, Klabunde CN; Centers for Disease Control and Prevention (CDC). Cancer screening test use - United States, 2013. MMWR Morb Mortal Wkly Rep. 2015 May 8;64(17):464-8.

Bardus M, Smith JR, Samaha L, Abraham C. Mobile Phone and Web 2.0 Technologies for Weight Management: A Systematic Scoping Review. J Med Internet Res. 2015 Nov 16;17(11):e259. doi: 10.2196/jmir.5129.

Matthew-Maich N, Harris L, Ploeg J, Markle-Reid M, Valaitis R, Ibrahim S, Gafni A, Isaacs S. Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review. JMIR Mhealth Uhealth. 2016 Jun 9;4(2):e29. doi: 10.2196/mhealth.5127.

Joe J, Demiris G. Older adults and mobile phones for health: a review. J Biomed Inform. 2013 Oct;46(5):947-54. doi: 10.1016/j.jbi.2013.06.008. Epub 2013 Jun 25.

Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. 1 edition. Englewood Cliffs, N.J: Prentice Hall; 1985

Weaver KE, Ellis SD, Denizard-Thompson N, Kronner D, Miller DP. Crafting Appealing Text Messages to Encourage Colorectal Cancer Screening Test Completion: A Qualitative Study. JMIR Mhealth Uhealth. 2015 Nov 4;3(4):e100. doi: 10.2196/mhealth.4651.

Seale JP, Johnson JA, Clark DC, Shellenberger S, Pusser AT, Dhabliwala J, Sigman EJ, Dittmer T, Barnes Le K, Miller DP, Clemow D. A Multisite Initiative to Increase the Use of Alcohol Screening and Brief Intervention Through Resident Training and Clinic Systems Changes. Acad Med. 2015 Dec;90(12):1707-12. doi: 10.1097/ACM.0000000000000846.

Venkatesh V, Bala H. Technology Acceptance Model 3 and a Research Agenda on Interventions. Decis Sci. 2008 May 1;39(2):273-315.

Venkatesh V, Davis FD. A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Manag Sci. 2000 Feb;46(2):186-204.

Chan WV, Pearson TA, Bennett GC, Cushman WC, Gaziano TA, Gorman PN, Handler J, Krumholz HM, Kushner RF, MacKenzie TD, Sacco RL, Smith SC Jr, Stevens VJ, Wells BL. ACC/AHA Special Report: Clinical Practice Guideline Implementation Strategies: A Summary of Systematic Reviews by the NHLBI Implementation Science Work Group: A Report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Am Coll Cardiol. 2017 Feb 28;69(8):1076-1092. doi: 10.1016/j.jacc.2016.11.004. Epub 2017 Jan 26.

Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, Grimshaw JM. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014 Jan 17;9:14. doi: 10.1186/1748-5908-9-14.

Ford JH 2nd, Alagoz E, Dinauer S, Johnson KA, Pe-Romashko K, Gustafson DH. Successful Organizational Strategies to Sustain Use of A-CHESS: A Mobile Intervention for Individuals With Alcohol Use Disorders. J Med Internet Res. 2015 Aug 18;17(8):e201. doi: 10.2196/jmir.3965.

Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006 Feb;21 Suppl 2(Suppl 2):S1-8. doi: 10.1111/j.1525-1497.2006.00355.x.

Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006 Mar;29(1):126-53. doi: 10.1177/0163278705284445.

Shaw EK, Howard J, West DR, Crabtree BF, Nease DE Jr, Tutt B, Nutting PA. The role of the champion in primary care change efforts: from the State Networks of Colorado Ambulatory Practices and Partners (SNOCAP). J Am Board Fam Med. 2012 Sep-Oct;25(5):676-85. doi: 10.3122/jabfm.2012.05.110281.

Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013 Oct 2;8:117. doi: 10.1186/1748-5908-8-117.

Neta G, Glasgow RE, Carpenter CR, Grimshaw JM, Rabin BA, Fernandez ME, Brownson RC. A Framework for Enhancing the Value of Research for Dissemination and Implementation. Am J Public Health. 2015 Jan;105(1):49-57. doi: 10.2105/AJPH.2014.302206.

Hemming K, Girling AJ, Sitch AJ, Marsh J, Lilford RJ. Sample size calculations for cluster randomised controlled trials with a fixed number of clusters. BMC Med Res Methodol. 2011 Jun 30;11:102. doi: 10.1186/1471-2288-11-102. Erratum In: BMC Med Res Methodol. 2017 Jan 19;17 (1):8.

Rutterford C, Copas A, Eldridge S. Methods for sample size determination in cluster randomized trials. Int J Epidemiol. 2015 Jun;44(3):1051-67. doi: 10.1093/ije/dyv113. Epub 2015 Jul 13.

Murray DM, Varnell SP, Blitstein JL. Design and analysis of group-randomized trials: a review of recent methodological developments. Am J Public Health. 2004 Mar;94(3):423-32. doi: 10.2105/ajph.94.3.423.

Austin PC. A comparison of the statistical power of different methods for the analysis of cluster randomization trials with binary outcomes. Stat Med. 2007 Aug 30;26(19):3550-65. doi: 10.1002/sim.2813.

Ma J, Thabane L, Kaczorowski J, Chambers L, Dolovich L, Karwalajtys T, Levitt C. Comparison of Bayesian and classical methods in the analysis of cluster randomized controlled trials with a binary outcome: the Community Hypertension Assessment Trial (CHAT). BMC Med Res Methodol. 2009 Jun 16;9:37. doi: 10.1186/1471-2288-9-37.

Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002 Aug;27(4):299-309. doi: 10.1046/j.1365-2710.2002.00430.x.

Penfold RB, Zhang F. Use of interrupted time series analysis in evaluating health care quality improvements. Acad Pediatr. 2013 Nov-Dec;13(6 Suppl):S38-44. doi: 10.1016/j.acap.2013.08.002.

Helfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implement Sci. 2009 Jul 14;4:38. doi: 10.1186/1748-5908-4-38.

Kutner M, Greenburg E, Jin Y, Paulsen C. The Health Literacy of America's Adults: Results from the 2003 National Assessment of Adult Literacy. NCES 2006-483. Natl Cent Educ Stat [Internet]. 2006 [cited 2015 Sep 23];

U.S. Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Health Statistics. Summary Health Statistics: National Health Interview Survey, 2015 [Internet]. [cited 2017 Jun 16].

http://www.healthcareitnews.com/news/cerner-still-leads-ehr-marketshare-though-smaller-vendors-are-making-moves

Monegain B. Cerner still leads in EHR marketshare, though smaller vendors are making moves. Healthc IT News [Internet]. 2017 Apr 28 [cited 2017 Jun 22];

Berta W, Cranley L, Dearing JW, Dogherty EJ, Squires JE, Estabrooks CA. Why (we think) facilitation works: insights from organizational learning theory. Implement Sci [Internet]. 2015 Dec [cited 2017 May 8];10(1).

Maher L, Gustafson D, Evans A. Sustainability Model and Guide [Internet]. British National Health Service Modernization Agency; 2004 [cited 2017 Jun 9].

Soo S, Baker WB and GR. Role of Champions in the Implementation of Patient Safety Practice Change. Healthc Q [Internet]. 2009 Aug 15 [cited 2017 May 8];

Behavioral Risk Factor Surveillance System, 2015 Questionnaire [Internet]. Centers for Disease Control and Prevention (US); 2014 [cited 2016 Sep 30].