Educational Evaluation and Policy Analysis recent issues

To Choose or not to Choose: High School Choice and Graduation in Chicago

Lauen, D. L. — Thu, 13 Aug 2009 16:17:00 PDT

School choice reforms have been proposed as ways to enhance efficiency, equity, and effectiveness in education. This study examines the consequences of participating in public high school choice in Chicago, a city with a wide variety of choice programs, including career academies, charter schools, magnet schools, and selective test-based college prep high schools. The analysis uses population-level administrative and survey data on all public school eighth graders enrolled in Chicago to estimate the effect of school choice participation on on-time graduation propensity (i.e., in 4 years). Techniques employed to estimate this effect include propensity score, catchment area fixed effects, and multilevel analysis. Results suggest that there is a modest positive graduation benefit from exercising school choice. There are no racial/ethnic differences in the choice benefit, but low-achieving students benefit less from high school choice than high-achieving students. In addition, students in high-poverty neighborhoods gain less from exercising choice than do students in low-poverty neighborhoods. These findings call into question the extent to which school choice enhances equity for low-achieving students and students in high-poverty neighborhoods.

A Longitudinal Investigation of the Relationship between Teachers' Self-Reports of Reform-Oriented Instruction and Mathematics and Science Achievement

Le, V.-N., Lockwood, J. R., Stecher, B. M., Hamilton, L. S., Martinez, J. F. — Thu, 13 Aug 2009 16:17:00 PDT

In the past two decades, several major initiatives were launched to improve mathematics and science education. One prominent feature in these efforts was a new approach to teaching mathematics and science, referred to as reform-oriented teaching. Although past studies suggest this approach may improve student achievement, the relationships between reform-oriented pedagogy and achievement were weak. The weak relationships may be partially attributable to the limited time frame in which reform-oriented teaching was examined (typically a 1-year period). This study explored the relationship between mathematics and science achievement and reform-oriented teaching over a 3-year period. Results suggested greater exposure to reform-oriented instruction was generally not significantly associated with higher student achievement but the effects became stronger with prolonged exposure to reform-oriented practices. Reform-oriented instruction showed stronger, positive relationships with open-ended measures than with multiple-choice tests in both mathematics and science and with problem-solving skills than with procedural skills in mathematics.

Do Inspections Improve Primary School Performance?

Luginbuhl, R., Webbink, D., de Wolf, I. — Thu, 13 Aug 2009 16:17:00 PDT

This article investigates the effect of Dutch primary school inspections on test scores. These inspections are assessments of the educational quality of schools. The authors use two approaches to estimate the effect of the inspections. The first approach, based on a standard fixed effects model, indicates that inspections lead to slightly better school performance: Test scores increase by 2% to 3% of a standard deviation. However, the fixed effects model might be sensitive to selection bias. The second approach exploits a sample of randomly selected schools originally drawn for the purpose of compiling the annual report of the state of Dutch education. Based on this approach, the authors find that inspections do no harm but seem to have little or no effect on student performance.

Is There a de Facto National Intended Curriculum? Evidence From State Content Standards

Porter, A. C., Polikoff, M. S., Smithson, J. — Thu, 13 Aug 2009 16:17:00 PDT

State content standards are the backbone of the standards-based reform movement. Content standards provide teachers with a set of guidelines for what students are expected to know and be able to do, defining the intended curriculum. And although the current 50-state system of education gives each state the task of setting content standards, there has been little empirical investigation of the similarities and differences among state content standards. This analysis uses the content analysis procedures of the Council of Chief State School Officers/State Collaboratives on Assessment and State Standards to consider whether there exists a de facto national curriculum as defined in state content standards. Data from English/language arts and reading (ELAR), science, and mathematics for Grades 4, 8, and K–8 are used. Results suggest considerable variability among states in the content of content standards, particularly in individual grades, but also for the aggregated standards. Further analysis suggests that state standards are no more well aligned to national professional standards (i.e., National Science Education, National Council of Teachers of Mathematics) than to other states’ standards. Still, there exists a small "core curriculum" across states in each content area. The level of focus of the state standards varies substantially across states, and the redundancy of the standards is such that alignment within state, across grades is often as high as alignment within grade, across states. Policy implications are briefly discussed.

How External Interveners Leverage Large-Scale Change: The Case of America's Choice, 1998-2003

Glazer, J. L. — Thu, 13 Aug 2009 16:17:00 PDT

This article discusses a dilemma that confronts reformers who seek to improve the education system yet operate from a position outside that system. On one hand, "external" reformers have found that the implementation of school-level programs is attenuated by a lack of coherence and support in schools' environment. Districts and states often enact policies and pursue initiatives that are not consistent with the aims and practices of intervention designs. On the other hand, when reformers redraw the boundaries of their operations to include districts and states, they are beset with an array of new problems inherent in such partnerships. The current discussion explores the dynamics of this dilemma by examining the experience of the America's Choice School Design between the years 1998 and 2003. The discussion concludes by arguing that while interveners cannot ignore schools' environments, their efforts to gain leverage on them will never be more than partially effective.

An Examination of the Precision and Technical Accuracy of the First Wave of Group-Randomized Trials Funded by the Institute of Education Sciences

Spybrook, J., Raudenbush, S. W. — Thu, 13 Aug 2009 16:17:00 PDT

This article examines the power analyses for the first wave of group-randomized trials funded by the Institute of Education Sciences. Specifically, it assesses the precision and technical accuracy of the studies. The authors identified the appropriate experimental design and estimated the minimum detectable standardized effect size (MDES) for each study under plausible assumptions about intra-class correlations, covariate-outcome correlations, and explanatory effects of blocking. The MDESs ranged from .18 to .40 for studies funded by the National Center for Education Evaluation and Regional Assistance (NCEE) and from .20 to 1.0 for studies funded by the National Center for Education Research (NCER). They evaluated the technical accuracy by comparing the estimated MDES to the MDES stated in the proposal. Studies funded by the NCEE tended to be very accurate: estimated MDESs and stated MDESs were similar, whereas studies funded by the NCER were less accurate. However, the technical accuracy of the NCER studies improved over time.

Boosting Student Achievement: The Effect of Comprehensive School Reform on Student Achievement

Gross, B., Booker, T. K., Goldhaber, D. — Wed, 06 May 2009 10:03:14 PDT

Between the late 1980s and early 2000s, schools, districts, states, and the federal government devoted enormous resources to the implementation of Comprehensive School Reform (CSR) models. With more than 1.6 billion federal dollars distributed through the Comprehensive School Reform Demonstration (CSRD) project and its successor, the CSR project, states and districts made CSR adoption a central reform strategy for their lowest performing schools. Today, however, federal funding for CSR has dried up, and this policy has been left behind with few explicit efforts to assess the effect of these CSR funds on schools. In this article, the authors look back on this federal reform initiative and the effect it had on Texas students. Using promising analytic techniques for nonexperimental studies to investigate the effects of federal CSR awards on student achievement, the authors find that CSRD funding did not significantly effect students’ reading performance and that its effect on math performance varied across different student types.

What Works in College Outreach: Assessing Targeted and Schoolwide Interventions for Disadvantaged Students

Domina, T. — Wed, 06 May 2009 10:03:14 PDT

By offering information, counseling, and tutoring, college outreach programs attempt to smooth the path between high school and higher education for at-risk students. But do these program work? This paper uses longitudinal data from the Education Longitudinal Study to construct two quasi-experiments to assess the effectiveness of college outreach. The first compares outreach program participants with a propensity score matched sample of program non-participants to measure the effects of targeted college outreach programs. The second assesses the effects of school-wide college outreach programs by comparing students in school-wide outreach high schools with students in a matched sample of high schools that offer no formal outreach. The results suggest that targeted outreach programs do little to change the educational experiences of participating students. However, there is limited evidence to suggest that school-wide outreach programs may have modest "spill-over" effects, improving the educational outcomes of relatively unengaged students at participating schools.

How Does Test Exemption Affect Schools' and Students' Academic Performance?

Jennings, J. L., Beveridge, A. A. — Wed, 06 May 2009 10:03:14 PDT

Analyzing data from a large urban district in Texas, this study examines how high-stakes test exemptions alter officially reported scores and asks whether test exemption has implications for the academic achievement of special education students. Test exemption inflated overall passing rates but especially affected the passing rates of African American and Hispanic students because these students were more likely to be exempted. Furthermore, our results suggest that tested special education students in Grades 3 through 8 performed better academically than they would have if they were not tested. However, taking the high-stakes test provided no academic benefit to special education students in Grades 9 through 11.

Toward Policy-Relevant Benchmarks for Interpreting Effect Sizes: Combining Effects With Costs

Harris, D. N. — Wed, 11 Mar 2009 14:18:15 PDT

The common reporting of effect sizes has been an important advance in education research in recent years. However, the benchmarks used to interpret the size of these effects—as small, medium, and large—do little to inform educational administration and policy making because they do not account for program costs. The author proposes an approach to establishing cost-effectiveness benchmarks rooted in an explicit economics-based decision-making framework and assumptions about the decision-making context. To be considered large, the ratio of effects to costs must be at least as large as the ratios for substitute interventions. Evidence related to class size, prekindergarten, and other interventions is discussed to illustrate the calculation of the cost-effectiveness ratios, how the evidence can be used to develop benchmarks, and how the benchmarks can be useful for researchers and policy makers. The development of benchmarks is intended to encourage cost-effectiveness analysis as a standard part of policy analysis, thereby providing more evidence to increase the validity of the benchmarks and, ultimately, improving policy decisions. Recent cost-effectiveness research in health care policy illustrates the potential value of cost-effectiveness benchmarks in education.

Do Community Colleges Provide a Viable Pathway to a Baccalaureate Degree?

Long, B. T., Kurlaender, M. — Wed, 11 Mar 2009 14:18:15 PDT

Community colleges have become an important entryway for students intending to complete baccalaureate degrees. However, many question the viability of the transfer function and wonder whether students suffer a penalty for starting at 2-year institutions. The authors examined how the outcomes of community college entrants compared with those of similar students who initially entered 4-year institutions within the Ohio public higher education system. Using a detailed data set, the authors tracked outcomes for 9 years and used multiple strategies to deal with selection issues: propensity score matching and instrumental variables. The results suggest that straightforward estimates are significantly biased, but even after accounting for selection, students who initially began at community colleges were 14.5% less likely to complete bachelor’s degrees within 9 years.

Reading Instruction Time and Homogeneous Grouping in Kindergarten: An Application of Marginal Mean Weighting Through Stratification

Hong, G., Hong, Y. — Wed, 11 Mar 2009 14:18:15 PDT

A kindergartner’s opportunities to develop reading and language arts skills are constrained by the amount of time allocated to reading instruction. In the meantime, the student’s engagement in learning tasks may increase if the instruction has been adapted to his or her prior ability through homogeneous grouping. This study investigates whether the grouping effects on kindergartners’ reading growth depend on the amount of reading instruction time and the intensity of grouping. To answer the study’s research questions requires causal inferences about concurrent multivalued instructional treatments. The authors develop a procedure of applying the method of marginal mean weighting through stratification to multilevel educational data. Results from the Early Childhood Longitudinal Study Kindergarten cohort data set lend support to the theoretical hypothesis that when teachers allocate a substantial amount of time to reading instruction, homogeneous grouping helps kindergartners to gain more in reading. The authors find no effect of homogeneous grouping when the total amount of reading time is limited. They also find that the benefit of increasing reading instruction time becomes evident only if kindergarten teachers adapt instruction through homogeneous grouping.

A Randomized Field Trial of the Fast ForWord Language Computer-Based Training Program

Borman, G. D., Benson, J. G., Overman, L. — Wed, 11 Mar 2009 14:18:15 PDT

This article describes an independent assessment of the Fast ForWord Language computer-based training program developed by Scientific Learning Corporation. Previous laboratory research involving children with language-based learning impairments showed strong effects on their abilities to recognize brief and fast sequences of nonspeech and speech stimuli, but generalization of these effects beyond clinical settings and student populations and to broader literacy measures remains unclear. Implementing a randomized field trial in eight urban schools, we generated impact estimates from separate intent-to-treat and treatment-on-the-treated analyses of the literacy outcomes of second- and seventh-grade students who were more generally at risk for poor reading and language outcomes. There were some problems of implementation in the field setting, and the Fast ForWord Language program did not, in general, help students in these eight schools improve their language and reading comprehension test scores.

Surveying the Landscape of Teacher Education in New York City: Constrained Variation and the Challenge of Innovation

Wed, 10 Dec 2008 09:36:54 PST

In this article, the authors describe the state of teacher education in and around the large and diverse school district of New York City. Using multiple data sources, including program documents, interviews, and surveys of teachers, this study attempts to explore the characteristics of programs that prepare elementary teachers of New York City public schools, including the kinds of programs that exist, who enters these different programs, who teaches in the programs, and what characterizes the core curriculum. A central question concerns the amount of variation that exists in the preparation of elementary teachers for a single, large school district. Despite the number and variety of programs that exist to prepare elementary teachers, the authors found the overall curriculum and structure of teacher education to be more similar than different. To understand this lack of variation, the authors draw on organizational theory, particularly, the concept of institutional isomorphism, to examine the case of teacher education. The authors conclude with recommendations for what it might take to change the landscape of teacher education in the context of a large urban district.

Contributions of Qualitative Research to Research on Teacher Qualifications

Kennedy, M. M. — Wed, 10 Dec 2008 09:36:54 PST

The influence of teachers’ qualifications on their teaching practice has been subject to debate. Literature reviews do not settle these debates, partly because the literature is uneven and partly because reviews capture only narrow slices of literature. In particular, many reviews eliminate qualitative studies. Yet without examining qualitative evidence, variations in quantitative findings are difficult to interpret, disappointing findings are difficult to understand, and plausible explanations of patterns are in short supply. The present review focuses on qualitative studies and compares their findings with those from quantitative literature. The author finds that the qualitative literature agrees with quantitative literature in its inability to distinguish between teachers with different types of certificates or different teacher education backgrounds. On the other side, the author finds more evidence of benefit from content knowledge than quantitative studies have typically found. The qualitative studies also reveal competing influences and offer hypotheses about why the outcomes look the way they do.

Scaling Up an Early Reading Program: Relationships Among Teacher Support, Fidelity of Implementation, and Student Performance Across Different Sites and Years

Wed, 10 Dec 2008 09:36:54 PST

Successful implementation of evidence-based educational practices at scale is of great importance but has presented significant challenges. In this article, the authors address the following questions: How does the level of on-site technical assistance affect student outcomes? Do teachers’ fidelity of treatment implementation and their perceptions of school climate mediate effects on student performance? Using a randomized control trial at scale, the authors examine Kindergarten Peer Assisted Learning Strategies, which previously has been shown to be effective in increasing student reading achievement. Analyzing data from 2 years and three sites, the analyses show that the level of on-site technical support has significant effects on reading achievement gains, are robust across multiple sites, and are mediated by fidelity of implementation within teachers’ classrooms.

A Multisite Cluster Randomized Field Trial of Open Court Reading

Borman, G. D., Dowling, N. M., Schneck, C. — Wed, 10 Dec 2008 09:36:54 PST

In this article, the authors report achievement outcomes of a multisite cluster randomized field trial of Open Court Reading 2005 (OCR), a K–6 literacy curriculum published by SRA/McGraw-Hill. The participants are 49 first-grade through fifth-grade classrooms from predominantly minority and poor contexts across the nation.Blocking by grade level within schools, the trial includes 27 classrooms receiving the OCR curricular materials and professional development and 22 "business-as-usual" control classrooms. Multilevel analyses of classroom-level effects of assignment to OCR reveal statistically significant treatment effects on all three of the Comprehensive Test of Basic Skills, 5th edition, Terra Nova literacy posttests. The OCR effect sizes are d = 0.16 for the Reading Composite, d = 0.19 for Vocabulary, and d = 0.12 for Reading Comprehension. These effects achieved across 27 classrooms and 5 schools demonstrate the potential for replicating improved literacy outcomes through the scale-up of OCR.

Acknowledgements

Wed, 10 Dec 2008 09:36:54 PST