Effectiveness of Interactive Voice Response for COVID-19 Vaccination Training in the Democratic Republic of the Congo

Effectiveness of Interactive Voice Response for COVID-19 Vaccination Training in the Democratic Republic of the Congo

Effectiveness of Interactive Voice Response (IVR) for COVID-19 Vaccination Training of Frontline Health Workers: Experimental Evidence From the Democratic Republic of the Congo

This study seeks to evaluate the effectiveness of a remote training course delivered via interactive voice response (IVR) to the mobile phones of frontline providers in the Democratic Republic of the Congo (DRC) during the COVID-19 pandemic. The randomized trial will examine the impact of the training on health workers' knowledge, beliefs and intended behaviors related to COVID-19 vaccines and vaccine administration. Potential enrollees will be consented first, after which the participants assigned to the treatment group will be invited to participate in the training over a one month period, while the control group will receive the intervention following a month delay, which will allow for the comparison of knowledge and beliefs with a group that is controlled for bias introduced by time. We hypothesize that the training will lead to increased knowledge about vaccines, lower vaccine hesitancy, and increased preparedness to administer vaccines among trained health workers.

Background: As COVID-19 continues to pose a significant threat worldwide, training health workers to administer the COVID-19 vaccine and encourage uptake of the vaccine is essential to limiting the number of deaths and ultimately ending the pandemic. Providing adequate training to health workers in low- and middle-income countries (LMICs) had already been logistically and economically challenging, but the onset of the pandemic has intensified the challenge of providing critical and rapidly evolving health information to health workers at a time when it is needed most, particularly in low bandwidth areas where access to accurate sources of information are limited. A growing body of research suggests that effective and efficient training can be provided through mobile health (mHealth) solutions. The rapid expansion of mobile phone technology available throughout LMICs has enabled the scaling of health education delivery via mobile phone. However, while app-based learning has grown in popularity, drawbacks include limited access to smartphones and the data costs to access the internet, if available. Interactive voice response (IVR) systems pose an alternative solution by employing pre-recorded audio files in local languages, allowing users to receive information from mobile phones without the need for internet access. Aims: In this study, we investigate the effectiveness of an IVR training intervention for improving knowledge of COVID-19 vaccines and vaccine administration among health workers in the Democratic Republic of the Congo (DRC). We will also examine differences in effectiveness across subgroups including by geographic region, professional roles (e.g., nurses, physicians) and type of health worker facility. Methods: We are collaborating with Viamo, a global social enterprise that has established IVR systems in more than 23 countries, to conduct a randomized controlled trial to measure the effects of the IVR remote training on health workers' knowledge and beliefs about COVID-19 vaccines and preparedness to administer vaccines. In coordination with the DRC Ministry of Health, Viamo has identified roughly 9,000 facility-based health workers prioritized for participation. Participants will be randomized 1:1 into a treatment and a control group, stratified by geographic region and professional roles to ensure equal representation across regions and types of health workers. The participants in the treatment group will receive the intervention at the start of the study while the participants in the control group will receive the intervention following a one-month delay, which will allow for the comparison of knowledge, attitudes and practices with a group that is controlled for bias introduced by time. The duration of the study for participants will be roughly one month, which is the time it takes to complete the 5 required IVR training modules and a baseline and endline survey. Intervention: Training content is being developed by Viamo in collaboration with the Stanford Center for Health Education and the DRC Ministry of Public Health. The intervention consists of 5 IVR-based modules conveying information about COVID-19 vaccines and vaccine administration best practices. Module topics include the following: 1) the importance of COVID-19 vaccines and recipient eligibility, 2) overview of and differences between approved COVID-19 vaccines , 3) surveillance of adverse vaccine events, 4) organizing a vaccination site, and safe storage and handling COVID-19 vaccines, and 5) micro planning and communication strategies for COVID-19 vaccination efforts. The training will be offered in five local languages: French, Lingala, Swahili, Kikongo, and Tshiluba. Sample size calculation: The DRC Ministry of Health identified roughly 9,000 health workers located in 13 focal provinces for prioritization of the training. With a total sample size of 9,000 (including both arms), we estimate a minimum detectable effect size (MDES) of 5.6 percentage points (at 80 percent power and a 5 percent significance level). Because of the voluntary nature of the IVR training participation, we anticipate a response rate of lower than 50 percent. At 80 percent power and 5 percent significance, a sample size of 2900 participants (over both arms) is needed to detect a 10 percent difference. Data collection and training: Data will be collected via the IVR platform. Treatment and control groups will complete a baseline and endline survey administered at the same time. The baseline survey will include a brief knowledge assessment, questions about participants' vaccination status, beliefs about COVID-19 vaccines, and preparedness to administer vaccines. The baseline also includes demographic and background questions about participants' prior work experience. The endline survey includes a more extensive knowledge assessment and questions about participants' vaccination status, beliefs about COVID-19 vaccines, and preparedness to administer vaccines. All surveys will be completed directly by participants via IVR calls direct to participants' mobile phones. Surveys and training content will include audio prompts to overcome literacy barriers and will be offered in one of five local languages, selected by participants directly. Participants will be trained on how to use the IVR technology on their mobile devices at the start of the study. Callers interact with pre-recorded audio files in their local languages and can choose what information they want to receive using their mobile phone keypads ("For more information on prevention, please press 1"). While the intervention group will be invited to begin the IVR training immediately after completing the baseline survey, the control arm will have a roughly one month delay before receiving the intervention. Besides the short delay, every aspect for treatment and control arms will remain the same. Data Analysis: The primary analysis will be based on intention-to-treat at the unit of the individual CHW. The analysis will include randomization strata fixed effects and control for baseline covariates to improve precision of estimates. Because we will have granular by-module completion data across participants, we will also conduct a treatment-on-the-treated instrumental variables analysis to examine the impact of the training on outcomes, among participants completing the training.

This is a randomized trial with baseline covariate adjustment and stratification. The randomization unit is the health worker. The randomization will be stratified by province, health worker role, and whether the health worker holds a supervisory role to ensure balance of intervention versus control across characteristics. The following baseline characteristics will be measured and adjusted for in the main analysis of primary outcomes to increase statistical efficiency: 1) age, 2) gender, 3) years of work experience in healthcare, 4) prior vaccination experience, 5) facility type, 6) COVID-19 vaccination status, 7) COVID-19 vaccine knowledge, and 8) COVID-19 vaccine beliefs.

Since data will be entered directly by health workers via IVR, the outcomes assessor (ie: IVR platform) will have no knowledge of the interventions assigned to individual participants. Additionally, the control group health workers will not be informed they are part of a control arm, but rather that they will receive the training after a delay.

The participants in the treatment group will be invited to participate in a 5-module IVR training covering an overview of available COVID-19 vaccines and best practices for vaccine administration.

Participants in the control group will receive the intervention following a one-month delay allowing for a comparison of outcomes with the treatment group.

The intervention consists of 5 interactive voice response (IVR) modules conveying information about COVID-19 vaccines and vaccine administration best practices. Module topics include the following: 1) the importance of COVID-19 vaccines and recipient eligibility, 2) overview of and differences between approved COVID-19 vaccines , 3) surveillance of adverse vaccine events, 4) organizing a vaccination site, and safe storage and handling COVID-19 vaccines, and 5) micro planning and communication strategies for COVID-19 vaccination efforts. The training will be offered in five local languages: French, Lingala, Swahili, Kikongo, and Tshiluba.

The control arm will receive the COVID-19 Vaccine IVR Training after an estimated month-long delay. The control arm will be told upon completion of the baseline survey that the training will commence at a later estimated date and will not receive any additional information until the treatment group completes training.

Standardized score on an assessment of 10 knowledge questions about COVID-19 vaccines and vaccine administration, drawn directly from the IVR curriculum

Agreement (on 3 point likert scale) with statement that participant feels confident and prepared to administer COVID-19 vaccines. Scale options: 1-disagree (minimum), 2-unsure, 3-agree (maximum), with maximum representing the better outcome.

Agreement (on 3 point likert scale) with statement that COVID-19 vaccines are safe. Scale options: 1-disagree (minimum), 2-unsure, 3-agree (maximum), with maximum representing the better outcome.

Agreement (on 3 point likert scale) with statement that COVID-19 vaccines are effective. Scale options: 1-disagree (minimum), 2-unsure, 3-agree (maximum), with maximum representing the better outcome.

Inclusion Criteria: - Identified by the DRC Ministry of Health as priority health worker to receive the training Exclusion Criteria: - Not identified by the DRC Ministry of Health as priority health worker to receive the training

Arie S. Can mobile phones transform healthcare in low and middle income countries? BMJ. 2015 Apr 22;350:h1975. doi: 10.1136/bmj.h1975. No abstract available.

Agarwal S, Labrique A. Newborn health on the line: the potential mHealth applications. JAMA. 2014 Jul 16;312(3):229-30. doi: 10.1001/jama.2014.6371. No abstract available.