A Tale of Two Interventions

"In much of the rest of the world there is evolutionary change, grounded in the assumption that if professionals keep working at something, they can make continual improvements. In China and Japan, for example, curricula change much less frequently and much more slowly than in the United States. To begin with, these curricula are carefully conceived and known to be reasonably effective. These curricula are refined on the basis of classroom observations and student performance. Teachers make the curriculum a collaborative object of study, working to find better ways to teach lessons or to improve them. In that way, gradual and sustained improvements are made...."
-Alan H. Schoenfeld, Professor of Education, UC, Berkeley

How learning happens inside a classroom is affected by so many factors. The curriculum is one small factor influencing learning outcomes in schools. A written curriculum is likewise not necessarily identical to what is being delivered to students. There are individuals called teachers who implement the curriculum. And there is no denying that some teachers, if not most, feel that pedagogy supported by experimental research may look inviting in literature, but often fails in their own classrooms.

Education is perhaps so complex that it is wise to take a conservative perspective and focus only on small changes. Gigantic pedagogical reforms that promise too much often fall flat. It is also quite common to see interventions working so well in a controlled setting, but not delivering when brought to a larger scale or a more realistic setting.

But there are interventions that work even in the large scale. Pasi Sahlberg wrote the following in the Washington Post:
"...many education visitors to Finland expect to find schools filled with Finnish pedagogical innovation and state-of-the-art technology. Instead, they see teachers teaching and pupils learning as they would in any typical good school in the United States. Some observers call this “pedagogical conservatism” or “informal and relaxed” because there does not appear to be much going on in classrooms. 
The irony of Finnish educational success is that it derives heavily from classroom innovation and school improvement research in the United States. Cooperative learning and portfolio assessment are examples of American classroom-based innovations that have been implemented in large scale in the Finnish school system."
There are interventions backed by evidence-based research that do work. In this post, we look at two interventions that demonstrate encouraging results in the initial pilot studies to gain insights on what makes an innovation successful in a larger scale. One intervention (Number Rockets) seems to work across a larger sample while a second one (Successful Intelligence) fails when scaled up.

Successful Intelligence can be described by the following excerpt from Sternberg and Grigorenko:
"Many students could learn more effectively than they do now if they were taught in a way that better matched their patterns of abilities. Teaching for successful intelligence provides a way to create such a match. It involves helping all students capitalize on their strengths and compensate for or correct their weaknesses. It does so by teaching in a way that balances learning for memory, analytical, creative, and practical thinking...."
The above does sound quite promising if not outright inspiring. "Teaching in a way that balances learning for memory, analytical, creative and practical thinking" is an ideal no teacher or parent would deny. These objectives require the teacher, for instances, to help students compare and contrast two different solutions to a math problem (analytical), discover on their own an explanation behind a natural phenomenon (creative), and apply what they have learned in math to budgeting their allowance (practical).  These are just specific examples but it should be quite clear that Teaching for Successful Intelligence goes into even greater depth than teaching critical thinking. And it works, at least, in controlled settings.

Sternberg and coworkers have examined whether Teaching for Successful Intelligence works on a larger scale (more than 7000 4th grade students from 223 schools across the US, located in nine states). The following figure summarizes their findings.

Above copied from Sternberg et al.
 The study compares Successful Intelligence against other teaching methods, namely Memory (M), Critical Thinking (CT) and "Teaching as usual" (TAU). The vertical axis on the above graph is a measure of how effective a teaching method is compared to Successful Intelligence (SI). A positive number means that the method other than SI is better. These comparisons are made across different lessons or units. There are five lessons in Language Arts: Wonders of Nature, True Wonders, Lively Biographics, Journeys, and It's a Mystery. There are three lessons in Math: Equivalent Fractions, Measurement, and Geometry. There are two lessons in Science: The Nature of Light and Magnetism. Out of the 23 comparisons, only seven are below the 0.00 line. Out of the 12 lessons, Successful Intelligence works best only in 2. It sounds so good but it does not work.

To end this post with a much more optimistic look, another intervention is worth mentioning. Number Rockets is an intervention designed for first grade pupils who are having difficulties in arithmetic.

Number Rockets from the Vanderbilt Kennedy Center
 Number Rockets is a tutoring program consisting of 48 forty-minute sessions over 16 weeks that focus on (a) identifying and writing numbers, (b) identifying more and less objects, (c) sequencing numbers, (d) using <, >, and =, (e) skip counting by l0s, 5s, and 2s, and (f) place values. Similar to the previous intervention, Number Rockets also shows promising results in small controlled studies. Gersten and coworkers have investigated whether Number Rockets also works on a much larger scale (almost 1000 at-risk students from 77 schools). The results are summarized in the following figure:

From the above graph, it can be seen that the effect size of the intervention is about 0.33 (The mean score of the intervention group is about a third of its standard deviation higher than the mean score of the control group). This is a moderate and significant size effect. Gersten et al. therefore conclude in their abstract, "Intervention students showed significantly superior performance on a broad measure of mathematics proficiency."

Some pedagogical interventions do work. The one illustrated here that works seems to carry less loftier goals. It simply aims to help a seven year old child who is struggling with math.