Randomized Trial of the Positive Action Program in Chicago Schools and Extension to Grade 8

This project focuses on social and character development of elementary and middle school-aged children and responds to an urgent national need that schools improve their capacity to address a range of student outcomes, including social skills, character, behavior, academic achievement and health outcomes. This study is a school-based randomized trial to evaluate the Positive Action program. The Positive Action program was designed to promote social and character development and improve behavior and school performance.

This project is one of seven in a multi-site trial of different programs that has been nationally implemented. Oregon State University (OSU) and University of Illinois at Chicago (UIC) are conducting a school-based randomized trial to evaluate the efficacy of the Positive Action program (PA) to find out how the program works, to determine the effectiveness of the Positive Action program on reducing negative behaviors (including health-related behaviors), increasing positive behaviors and improving academic achievement of elementary school students. The Positive Action program was designed to promote social and character development (respect, responsibility, altruism, civic virtue, prosocial behavior) in ways that reduce anti-social behavior (violence, substance use, delinquency) and improve school performance (attendance, test scores). Fourteen eligible schools selected during winter 2004 are comprised of 7 matched pairs (treatment and control); the schools were matched on a 'risk score' composed of multiple school characteristics. Students in grade 3 in the 2004-05 school year, their parents, and their teachers and principals were surveyed at baseline (Fall 2004), spring and fall of 2005, spring 2006 and spring 2007. Evaluation is based on multiple kinds of process, mediator variable and outcome data from school records (attendance, transience, grades, test performance, disciplinary actions and suspensions, and changes in school and student population characteristics), student records, student surveys, parent surveys, teacher ratings and surveys, and administrator surveys, collected from schools in both conditions (except information about delivery of the Positive Action program). The work being done at OSU is confined to Dr. Flay's overall supervision of all aspects of the project, and data analysis using de-identified data received from Dr. DuBois at UIC and research paper writing. The work being done at UIC, directed by Dr. David DuBois, includes all of the intervention work, data collection, data entering, and some data analysis and report writing. The U.S. Department of Education/IES hired a national contractor, Mathematica Policy Research, Inc. (MPR) to conduct core surveys at all sites of the multi-site trial through spring 2007. In addition, OSU/UIC is administering a site-specific student survey that is complementary to the multi-site surveys during all waves of data collection. As the project funding followed Dr. Flay's move from UIC to OSU in September 2005, OSU IRB provides a review for the overall project. As of April 2008, new funding allows continuation of the study through March 2012 and follows the target cohort of students through the end of 8th grade as they and their teachers and principals are surveyed fall 2008, spring 2009 and again along with their parents in spring 2010. Data collection for the continuation study also includes collection of height and weight of children and process evaluation data from students and teachers. There will be no involvement of MPR. Data collection was completed June 2010. Consent Rates and Mobility: Parental consent was obtained before students, parents or teachers completed surveys when students were in grade 3. Seventy-nine percent of parents provided consent at baseline. Students joining the study at later waves were consented at that time; consent rates for them ranged from 65% to 78% for Waves 2-5. All students were re-consented for the second phase of funding at Wave 6 (beginning of grade 7); consent rates were lower at Waves 6 through 8 ( ≈ 58 to 64%). This is consistent with previous studies that have found that consent rates drop as grade levels increase. The percentages of consenting parents who provided reports on their children were 72.3%, 58.9%, 52.2%, 50.5%, and 72.9% at Waves 1, 2, 4, 5 and 8, respectively. Two factors that likely increased parent response rate at Wave 8 were (1) an increase in the financial incentive for completing the parent report and (2) an intensive period of phone outreach to families to note the incentive increase and to encourage survey completion. Percentages of consented students for whom teachers completed ratings were 74.6%, 74.8%, 72.4%, 78.3%, 74.4%, and 92.7% for Waves 1, 2, 4, 5, 7, and 8, respectively. At Wave 8, we introduced an additional school-level incentive for 100% rates of teacher survey completion, which likely resulted in the increase in completed teacher ratings. Mobility patterns were identified using results from a latent class analysis in which a 5-class solution was found to be the most appropriate fit for the data: 1) stayers (average study duration of 5.72 years, N = 158), 2) temporary participants (1.30 years, only in grades 4 or 5; N=196), 3) late joiners (1.38 years; N=308); 4) early leavers (0.94 years; N=263), and 5) late leavers (3.23 years; N=287). Planned Statistical Analyses: Because the trial was cluster-focused, we assessed students who entered schools after the beginning of the trial (joiners), but did not follow individual students who stopped attending the study schools (leavers). From the standpoint of students, across time they could be considered a "dynamic" (i.e. changing) grade cohort. Multilevel models will be used to take into account variation at the school and student levels. Missing data will be addressed using the missing-at-random (MAR) assumption, as it is unlikely that a single unmeasured variable or set of variables would predict missingness for all students who left or joined the trial schools after randomization We propose a three-level (occasions of measurement nested within students nested within schools) growth-curve model for analyzing treatment effects on various student-level outcomes. These models will account for all observations and model school differences. This approach allows for a complete analysis of the multiple waves of available data and takes into account the patterns of change over time. Random-intercept growth-curve models will first be estimated. Following the random-intercept model, a random-coefficient model will be run to test whether there is significant variation in student change across time, rather than all students in each condition having the same change pattern. A Likelihood Ratio Chi-square (LR) test will be used to compare model fit with and without the random coefficient. If a model with a random time coefficient provides a significantly better fit for a given outcome, it will be reported as the final model. Intervention effects on scales collected only at later waves (Waves 5 or 6 onwards) will be tested with the intercept set at the endpoint (Wave 8) with the condition term indicating a possible difference in effects at the last (Wave 8). Because only 14 schools are in this trial, and the PA effect is tested at the school level in a cluster-randomized trial, we will conduct several sensitivity analyses. First, we will assess the statistical significance of the PA coefficient estimate and its standard error using the t-distribution with 12 degrees of freedom: 14 schools - 1 (the condition effect) - 1 = 12 df providing for a more conservative approach. A second approach will be a pair-level analysis, estimated as a four-level model: occasions of measurement nested within students, nested within schools, nested within matched pairs. In addition to the student-level survey data, several school-level archival measures will be analyzed. Because these data are at the school level, the growth-curve models will be two-level (observations within schools) rather than three-level. Because of the small amount of data (the number of schools times the number of waves) and the resulting power limitations, these analyses will use the random-intercept model only. We will test for moderation by gender and by student mobility. The moderation tests will reveal for whom the program has its effects; that is, these tests will allow us to assess whether program effects differ by gender or a child's mobility. We will not test for moderation by ethnicity because it is highly confounded with school, with 3 pairs of schools having a mostly African-American enrollment and 2 pairs of schools having a mostly Hispanic enrollment. While all 14 schools were retained throughout the CRCT, the student population in this trial was highly mobile. Thus, it is important to test for potential moderating effects of student mobility patterns. A recent approach to analyzing mobility patterns is latent class analysis (LCA). The mobility patterns described above can then be tested as a moderator of program effects; that is, examining whether students with different mobility patterns have different program effects. Mediation analyses will allow us to examine the PA program's mechanisms of action. We will first estimate the bivariate effect of X on Y without the mediator included in the model. Then, we will simultaneously estimate the direct effect of X on Y with the mediator included in the model, as well as the mediated effect, which consists of the effect of X on M × M on Y.

School health program, Positive health behavior, School-wide climate, Mental health, Self-esteem, Social support, Exercise, Nutrition, Parent involvement, Community involvement

The Positive Action program was implemented over 6 years, starting with Grade 3, then continuing through Grade 8.

Components of the Positive Action program (6 units) use research-supported educational strategies/methods that include active learning, positive classroom management, teacher training, detailed curriculum with almost daily lessons, school-wide climate-change program, and family program of parent support/involvement.

Local Site Scales - Student self-reported behaviors. School and Classroom Climate: Improved relationships

Current at time of asking, asked at beginning and end of grades 3 and 4, and end of grade 5, beginning and end of grade 7 and end of grade 8

Local Site Scales - Student self-reported behaviors. Expected Student Effects: Improvement in each: Attachment/Normative Beliefs; Attitudes Towards Behaviors; Self-Efficacy/Self-Esteem; Academic/Social Skills; Character and Social Development

Current at time of asking, asked at beginning and end of grades 3 and 4, and end of grade 5, beginning and end of grade 7 and end of grade 8

Local Site Scales - Student self-reported behaviors. Expected Impacts: Reduced Emotional Problems; Improved Health Behavior; Reduced Substance Use; Less Violence; Improved Grades and Test Scores

Current at time of asking, asked at beginning and end of grades 3 and 4, and end of grade 5, beginning and end of grade 7 and end of grade 8

Multi-site Scales - Student self-reported behaviors. School and Classroom Climate: Improved Relationships

Multi-site Scales - Student self-reported behaviors. Expected Student Effects: Improved Attitudes Towards Behaviors; Improved Self-Efficacy/Self-Esteem; Improved Character and Social Development

Multi-site Scales - Student self-reported behaviors. Expected Impacts: Reduced Emotional Problems; Fewer Behavioral/Disciplinary Problems

Multi-site Scales - Parent Reports of student behaviors. School and Classroom Climate: Improved Relationships; Increased Involvement of School with Parents and Community

Multi-site Scales - Parent Reports of student behaviors. Expected Student Effects: Improved Academic/Social Skills; Improved Character and Social Development; Fewer Behavioral/Disciplinary Problems

Multi-site Scales - Teacher Reports of student behaviors. School and Classroom Climate: Improved Relationships; Increased Amount/Quality Social/Character Development Activities used by School; Increased Involvement of School with Parents/Community

Multi-site Scales - Teacher Reports of student behaviors. Expected Student Effects: Improved Academic/Social Skills; Improved Character and Social Development

Multi-site Scales - Teacher Reports of student behaviors. Expected Impacts: Fewer Behavioral/Disciplinary Problems

Fidelity/Implementation Measures for Treatment Schools: Teacher online reports of Positive Action (PA) classroom activities

Inclusion Criteria: Public elementary schools Pre-K or K through grade 5 or 6; Schools were included in the study if they: Are community-based (that is, not magnet, academy, special ed., etc.), Have at least 60 students and two classrooms of grades 2, 3, 4, and 5, Have no more than 100 students or 3 classrooms per grade level, Have annual mobility rates no greater than 30% (meaning that approximately 15% move out of the school and 15% more into the school each year), Have at least 50% of students eligible for free or reduced price lunch, Are relatively low performing on standardized tests, Have not used Positive Action program in the last decade, Are not doing another social/character program. All students in the study cohorts - those in grade 3 in 2004-05 Exclusion Criteria: Reverse of inclusion criteria -

Ji P, Flay BR, Dubois DL, Brechling V, Day J, Cantillon D. Consent form return rates for third-grade urban elementary students. Am J Health Behav. 2006 Sep-Oct;30(5):467-74. doi: 10.5555/ajhb.2006.30.5.467.

Ji P, DuBois DL, Flay BR, Brechling V. "Congratulations, you have been randomized into the control group!(?)": issues to consider when recruiting schools for matched-pair randomized control trials of prevention programs. J Sch Health. 2008 Mar;78(3):131-9. doi: 10.1111/j.1746-1561.2007.00275.x.

Jarpe-Ratner E, Fagen M, Day J, Gilmet K, Prudowsky J, Neiger BL, DuBois DL, Flay BR. Using the community readiness model as an approach to formative evaluation. Health Promot Pract. 2013 Sep;14(5):649-55. doi: 10.1177/1524839913487538. Epub 2013 May 23.

Li KK, Washburn I, DuBois DL, Vuchinich S, Ji P, Brechling V, Day J, Beets MW, Acock AC, Berbaum M, Snyder F, Flay BR. Effects of the Positive Action programme on problem behaviours in elementary school students: a matched-pair randomised control trial in Chicago. Psychol Health. 2011 Feb;26(2):187-204. doi: 10.1080/08870446.2011.531574.

Washburn IJ, Acock A, Vuchinich S, Snyder F, Li KK, Ji P, Day J, DuBois D, Flay BR. Effects of a social-emotional and character development program on the trajectory of behaviors associated with social-emotional and character development: findings from three randomized trials. Prev Sci. 2011 Sep;12(3):314-23. doi: 10.1007/s11121-011-0230-9.

Lewis KM, Bavarian N, Snyder FJ, Acock A, Day J, Dubois DL, Ji P, Schure MB, Silverthorn N, Vuchinich S, Flay BR. Direct and Mediated Effects of a Social-Emotional and Character Development Program on Adolescent Substance Use. Int J Emot Educ. 2012 Apr;4(1):56-78.

Lewis KM, Schure MB, Bavarian N, DuBois DL, Day J, Ji P, Silverthorn N, Acock A, Vuchinich S, Flay BR. Problem behavior and urban, low-income youth: a randomized controlled trial of positive action in Chicago. Am J Prev Med. 2013 Jun;44(6):622-30. doi: 10.1016/j.amepre.2013.01.030.

Bavarian N, Lewis KM, Dubois DL, Acock A, Vuchinich S, Silverthorn N, Snyder FJ, Day J, Ji P, Flay BR. Using social-emotional and character development to improve academic outcomes: a matched-pair, cluster-randomized controlled trial in low-income, urban schools. J Sch Health. 2013 Nov;83(11):771-9. doi: 10.1111/josh.12093.

Lewis KM, DuBois DL, Bavarian N, Acock A, Silverthorn N, Day J, Ji P, Vuchinich S, Flay BR. Effects of Positive Action on the emotional health of urban youth: a cluster-randomized trial. J Adolesc Health. 2013 Dec;53(6):706-11. doi: 10.1016/j.jadohealth.2013.06.012. Epub 2013 Jul 26.

Bavarian N, Lewis KM, Holloway S, Wong L, Silverthorn N, DuBois DL, Flay BR, Siebert C. Mechanisms of Influence on Youth Substance Use for a Social-Emotional and Character Development Program: A Theory-Based Approach. Subst Use Misuse. 2022;57(12):1854-1863. doi: 10.1080/10826084.2022.2120359. Epub 2022 Sep 11.