Eric.ed.gov – Estimating Impacts on Students, Teachers, and Schools of Programs Targeted for Preschool through Transition to College–Lessons for Methodologists from the Designs of 117 Evaluations Being Conducted in the Investing in Innovation Fund (i3) Grant Program.

When presenting results from rigorous, field-based evaluations of educational interventions, researchers often reveal methodological barriers they face in designing studies to assessing program effectiveness. The evaluations funded by the Investing in Innovation Fund (i3) present a timely opportunity for identifying relevant and pressing issues arising in current education evaluations. With over $1 billion dollars invested in the implementation and evaluation of 117 education interventions, the i3 program represents a major investment by the U.S. Department of Education in educational interventions that will both serve students, teachers and schools, and generate evidence about intervention effectiveness. The breadth and number of i3 evaluations provides a unique opportunity to learn about the features of, and challenges encountered in, the design and implementation of the evaluations. Researchers from the National Evaluation of i3 (NEi3) are in a unique position to describe and summarize the barriers to high quality research the i3 evaluators face. The following research questions are explored in this study: (1) What designs are being used to evaluate the i3 interventions? (2) How has the choice of design varied by: Grant type (development, validation, and scaleup); intervention target ages/grades (e.g., pre-school interventions, elementary, middle, high school, transition to college); intervention target outcome domains (e.g., ELA, math, science, arts achievement, socio-emotional; behavior regulation; GPA, credits, attendance, course taking, FAFSA completion)? and (3) What are the threats to causal inference that most commonly arise in the design and implementation of the i3 evaluations?