"Bear in mind that the wonderful things you learn in your schools are the work of many generations, produced by enthusiastic effort and infinite labor in every country of the world. All this is put into your hands as your inheritance in order that you may receive it, honor it, add to it, and one day faithfully hand it to your children. Thus do we mortals achieve immortality in the permanent things which we create in common." - Albert Einstein

Monday, April 15, 2013

Good Teachers Can Make a Difference, How About Bad Ones?

I have had my share of bad teachers (of course, this is based only on my opinion) while I was a student. But I do remember the good ones. A good teacher can indeed have a long term impact on a student. I guess one can wonder what would have happened if all I had as a student were good teachers. Kevin Drum in a recent article on Mother Jones provides an estimate to this question. Drum's article, "Firing Bad Teachers Has Surprisingly Meager Effects", makes use of a working paper from the National Bureau of Economic Research entitled "The Long Term Impacts of Teachers: Teacher Value Added and Student Outcomes in Adulthood", which has the following abstract:
Are teachers’ impacts on students’ test scores (“value-added”) a good measure of their quality? This question has sparked debate largely because of disagreement about (1) whether value-added (VA) provides unbiased estimates of teachers’ impacts on student achievement and (2) whether high-VA teachers improve students’ long-term outcomes. We address these two issues by analyzing school district data from grades 3-8 for 2.5 million children linked to tax records on parent characteristics and adult outcomes. We find no evidence of bias in VA estimates using previously unobserved parent characteristics and a quasi-experimental research design based on changes in teaching staff. Students assigned to high-VA teachers are more likely to attend college, attend higher- ranked colleges, earn higher salaries, live in higher SES neighborhoods, and save more for retirement. They are also less likely to have children as teenagers. Teachers have large impacts in all grades from 4 to 8. On average, a one standard deviation improvement in teacher VA in a single grade raises earnings by about 1% at age 28. Replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students’ lifetime income by more than $250,000 for the average class- room in our sample. We conclude that good teachers create substantial economic value and that test score impacts are helpful in identifying such teachers.
Drum highlights the finding that replacing a bottom 5% teacher ("the worst of the worst") results only in a 1% increase in earnings for a student at age 28. Drum heavily emphasizes the 1% figure, explaining the "meager" word in his Mother Jones' article. The bottom 5% teachers are not teaching in one school so the chances that a student gets more than one teacher from this group is statistically unlikely. Thus, Drum concludes at the end of the article:
Am I the only one who finds that surprisingly meager? I'm all for getting rid of horrible teachers, but if anything, this study makes me put a lower priority on this than I used to—especially considering how difficult it would be to carry out a policy like this. I was surprised that this study got such a euphoric reception when it was published, and I still am.
The above observation does deliver a serious blow against this cover story on Newsweek:

March 2010 Newsweek issue

There are about 3 million teachers in elementary and middle schools in the United States. Firing 5% means laying off 150,000. This is a huge number and, as Drum points out, will only have a meager effect.

There are indeed far better ways of improving teacher quality. A lot can be achieved when teachers are given time to work together so that, for example, they can compare notes. Teachers can observe other teachers and this practice helps both, the one observing and the one being observed. Teachers must be afforded time to reflect, prepare and review. For the Philippines, teacher quality is, without doubt, a crucial issue in addressing the problems Philippine basic education faces. DepEd secretary Luistro noted:
“In government, whether in teaching or in other areas, no one has ever been dismissed for incompetence. People have been dismissed for corruption, but not for incompetence. In most schools, teachers with tenure, however incompetent, cannot be fired without due cause.
So we have to work with them. We have to continue to train and retrain them to be able to do K to 12 well.”
This statement acknowledges that it is difficult to fire bad teachers. However, it should be made clear that even if that option is possible, it is not going to work. What is truly needed is realizing what "training" and "retraining" really encompass. One must review if regional workshops or seminars are really effective. My initial guess is that these are of very low utility. Teachers really face highly individualized cases and efforts and time are probably better spent at the school level - in terms of greater involvement of the principal as well as collaboration among teachers in a school. This brings us back to one of the seven elements of Finland's education reform:

Diversity: The school network is based on the idea of inclusive education that promotes diversity in schools and classrooms. Steering of teaching and learning has never been based on written standards, but rather upon guidelines encouraging creative solutions within increasingly diverse social and human environments.








17 comments:

  1. "Drum highlights the finding that replacing a bottom 5% teacher ("the worst of the worst") results only in a 1% increase in earnings for a student at age 28"

    no, that is not the paper or drum's claim. the paper's claim is: "Replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students’ lifetime income by more than $250,000 for the average class- room in our sample. "

    Question: is 250K in lifetime income LARGE or SMALL?

    this paper tells us what US lifetime income is:http://www.treasury.gov/resource-center/economic-policy/Documents/rp2007-01.pdf

    check out table: lifetime incomes range from 233K (women, minority, notblack) to approx 1mill (white males)



    as far as i see it, chetty et all are correct -- these are LARGE. not small. it will more than DOUBLE a median minority woman's lifetime income.

    ReplyDelete
  2. This is what Drum wrote:

    "I remember reading this study when it first came out, and I had a very different reaction. Basically, the researchers looked at data on teachers and test scores starting around 1990, and linked it up with the incomes of students later in life. They had a big dataset and they did all the usual controls. Then they estimated the gains to income from firing a terrible teacher (one in the bottom 5 percent) and replacing him with an average teacher. Long story short, once you account for real-world frictions (namely that we can't perfectly identify the terrible teachers), they estimate that making this change in a single grade level produces an income gain among students of slightly more than 1 percent at age 28.

    Now, the first thing to say about this is that a result this small, even if it's statistically significant, should be treated with extreme caution. There are just a ton of confounding variables that could be at play here, and even a small missing factor could swamp this study's findings."

    And from the Harvard paper:

    We Örst evaluate Hanushekís (2009, 2011) proposal to replace the 5 percent of teachers with the lowest value-added with teachers of average quality. To calculate the impacts of such a policy, note that a teacher in the bottom 5% of the true VA distribution is on average 2.04 standard deviations below the mean teacher quality. Therefore, replacing a teacher in the bottom 5% with an average teacher generates a gain per student of $9;422 per student in present value at age
    12, or $267;000 for a class of average size (28.3 students).

    Again, please stop misinforming the readers of this blog. I do not know what your agenda is, but you did not even see that the total figure your are quoting above (250K) does not correspond to a student but to 28 students. That is why it is an effect of 1% on the lifetime income of an individual. It is SMALL! How could 250K be taken as 1% when that the range of lifetime incomes is 200K to 1 million and yet the Harvard paper says 1%. It is because you are blind and you did not see that the total amount correspond to an entire classroom and not just one student. You are failing miserably in math and reading comprehension.

    ReplyDelete
  3. 1) it doesnt matter if its 250 if PER STUDENT or PER CLASSROOM.

    i chose the lifetime income of a person to illustrate the gains from firing a teacher from the bottom 5% VA.

    note: i did NOT claim that it would raise income of a person by 250k? did i? pls point to where i did.

    what i did was to compare 250k lifetime income, to measures of lifetime income of select median individuals.

    all i did was argue (correctly) that losing the lowest 5% (VA) would be equivalent to he increasing would be equivaluent to a minority woman's lifetime income. that is all. and on that basis, i found it to be quite large.

    now, if u want to say that that is an inappropriate comparison, make the argument. i'll defend mine.

    now, why make that comparison in the first place. think about it: the value of replacing a poor teacher with a median teacher (according to VA) is EQUIVALENT TO 100% of a minority, non-black woman's LIFETIME INCOME, or 25% of a white male's LIFE TIME INCOME.

    now, the value of the teacher should be measured per classroom, as the teacher never teaches only ONE STUDENT. hence, you must aggregate up student benefits to the classroom level.

    1b) here's another way to think about it. if replacing a bottom teacher is "worth" 250k worth of lifetime income, then you can compare it to the lifetime income for a TEACHER. now, the 230k figure is probably lower, but isnt too far from the lifetime income of the public school teacher.

    so, you can interpret the gains of replace the poor teacher be the same magnitude as if you hired ANOTHER TEACHER. (ie. its as if you had hired two teachers instead of one).

    2) in your original blog, you did not cite the claim from the paper correctly. The claim is simple and straighforward "On average, a one standard deviation improvement in teacher VA in a single grade raises earnings by about 1%
    at age 28. Replacing a teacher whose VA is in the bottom 5% with an average teacher would increase
    the present value of students’ lifetime income by more than $250,000 for the average class- room in our sample"

    meanwhile, you said "Drum highlights the finding that replacing a bottom 5% teacher ("the worst of the worst") results only in a 1% increase in earnings for a student at age 28."

    anyone who can read can see that they arent the same claim. (admittedly, this is a minor point)

    for one, the bottom 5% is 2 standard deviations, and the 1% figure is for a 1std dev change. (yes, this is a minor point, but i wanted to point out that your claim is not the same claim as the paper's).

    3) actually, the true cost-benefit of any policy should be relative returns. If you can point out ANY POLICY, that is worth/valued at more 250k PER TEACHER, then policy should do that first.

    do you know any?

    4) i have no agenda. if you can prove that any of my points 1-3 are wrong, i'd be glad to say i'm wrong.

    ReplyDelete
  4. I did not incorrectly cite anyone. You could look at the exact quotes above. Be honest.

    ReplyDelete
  5. The reason why your quantitative comparison is not fair is because you lifted a number 250K, which corresponds to more than two dozen individuals and compare this against the lifetime earnings of an individual. Of course, the number will turn out to be significant, What 28 people lose can be at the same order of magnitude as what one individual earns during a lifetime. This loses the actual percentage for each person. That percentage is important. A fair comparison would be 250K versus the lifetime earnings of 28 individuals - and this will then match the same 1% that Drum cites. Why is this important? Because 1% could easily be noise. Lifetime earnings can be changed by other factors by more than 1%. Absolute values such 250,000 dollars can be misleading - since this has been amplified by multiplying each student's loss of earnings by the number of students. Why stop at multiplying it by the number of students, why not multiplying this by number of years to even reach a much more staggering number like millions, just to have more impact; A bad teacher working for four years. results in 1 million dollars loss in earnings. Or why even stop there - let the teacher teach for 20 years - that is a loss of 5 million dollars. This math is highly deceiving because these numbers no matter how big they are, is still 1% of total earnings. And 1% is easily noise in this complicated world. That is the point.

    ReplyDelete
  6. And if I am misquoting Drun, why did Drum write the following:

    "Now, the first thing to say about this is that a result this small, even if it's statistically significant, should be treated with extreme caution. There are just a ton of confounding variables that could be at play here, and even a small missing factor could swamp this study's findings.

    But it's worse than that. Even if you assume these results are correct, they struck me not as optimistic, but as shockingly low. We're not talking about a small change in teacher quality here. We're talking about a huge change: an entire year spent with a perfectly acceptable teacher instead of one who's the worst of the worst. And even at that, it only made a difference of 1 percent in future income. That's it?"

    I am citing Drum correctly.

    ReplyDelete
  7. "Why stop at multiplying it by the number of students, why not multiply this by number of years to even reach a much more staggering number like millions, just to have more impact;"


    you can't do that because the notion of PRESENT VALUE already aggregates over years.


    so you compare present values with other present values.

    ReplyDelete
  8. "you lifted a number 250K, which corresponds to more than two dozen individuals and compare this against the lifetime earnings of an individual. Of course, the number will turn out to be significant, What 28 people lose can be at the same order of magnitude as what one individual earns during a lifetime. "

    In this case, NO, because a teacher, on ave (or median) doesnt teach ONE student, but 30 at a time. so the effect of replacing ONE teacher is spread out among 30 students.

    Its a management issue.

    Lets say you are a manager, and you find that if you replace the poorest performing worker by a median worker, you get X in excess profits.

    Is X BIG or SMALL? It is BIG if X is big relative to the cost of a (marginal) worker.



    if a worker costs X, the excess profits are the same magnitude as if you hired ONE MORE WORKER.


    what you DO NOT DO is argue. "you know what, since i have 100 workers, the benefit of doing the replacing is only X/100. small!"


    its a mistake because the additional profit to the manager is NOT X/100. its X.

    ReplyDelete
  9. then both of you mis-cited the original paper.

    ReplyDelete
  10. Lets put this issue another way.


    Lets assume its 250k PER STUDENT. thats not large -- it is implausible.


    this is saying that a teacher replacement policy can create gains/returns up to 30 times its cost (i.e the cost of replacing a teacher), at zero risk.


    implausible.

    ReplyDelete
  11. See how GabbyD misinforms::

    "Lets put this issue another way.

    Lets assume its 250k PER STUDENT. thats not large -- it is implausible.

    this is saying that a teacher replacement policy can create gains/returns up to 30 times its cost (i.e the cost of replacing a teacher), at zero risk.

    implausible.

    to put it another way still: if merely replacing a teacher can DOUBLE the LIFETIME earnings of a person, thats TOO BIG to be believable.

    think about it. you will meet this teacher only one year in your WHOLE LIFE, but this teacher has the potential to DOUBLE your lifetime earnings. that is larger than the return to college, which has arguably more proximal impact to your lifetime income.

    here's another branch of econ that deals with income effects of an intervention when you are child. surely these should have larger effects than replacing a teacher... and they do!

    -------------------------

    And then GabbyD writes later:

    note: i did NOT claim that it would raise income of a person by 250k? did i? pls point to where i did.

    ________________

    And then later claims that both Drum and I cited incorrectly the paper.

    ________________

    Neither Drum and I incorrectly cited the paper. That is why the abstract has been quoted in this post. One must not make a mountain out of a pebble by playing with math.The link to the original paper is provided, one should look at the figures representing the data and examine closely the values on the y-axis. The differences are very small (1%) and points are drawn for each 5 percentile (without deliberately showing the spread within this percentile). There are no error bars and only bins of collected points are shown. One must imagine the scatter within this data to see how big 1% really is in the overall picture. If one does that, one should see that 1% is really within the noise of the data.

    ReplyDelete
  12. misinforms?


    i did say LET US ASSUME, did i not?


    the point is to assume it, and then we will find that it is indeed large -- so large as to be impossible!

    ReplyDelete
  13. So, point of fact: did you (or drum) miscite the paper.

    yes. its not a big mistake (or a "mountain" as you aver), but its a mistake nontheless. When you write: "Drum highlights the finding that replacing a bottom 5% teacher ("the worst of the worst") results only in a 1% increase in earnings for a student at age 28."


    that is not the same as chetty etal's claim. its a difference between 1 standard dev and 2stnd dev. Thats a big difference.


    Note, that i admitted to a typo earlier, and i completely explained my arguments for it below. if you wish to counter, please do so. i eagerly anticipate your thoughts.


    now, i see your latest comment wants to talk about 1% and noise. If you want to talk about econometrics, we can do so. but its a good idea to agree on "easier" ideas first, such as what the claims are, and whether those claims are BIG or SMALL.


    parenthetically, i'd note that the noise issue is addressed and factored into their results. I wonder why you agree with a journalist (drum) more than the econometrican (chetty). since you are a scientist, i thought you would be more skeptical of journalists ' claims over a scientist.


    but we cant talk about econometrics if you want.

    ReplyDelete
  14. now, why do i care? i care because, like you, i love the philippines. and i'd like to bring the best that scientific research can bring to tough questions on education policy (and econ policy in general).

    that requires 2 things:

    1) i dont have ANY PRECONCEIVED IDEAS about what is good or not. I'll let research guide me.

    2) I need to understand exactly what the research is telling me.

    as a scientist, i hope you'd subscribe to 1) and 2). I ask you now: do you?

    now, if you wanted to critique the study, I have one. A critique that chetty etal also share.



    the policy experiment ASSUMES that you can costlessly replace a poor performing teacher with a median teacher.


    there is nothing in the anaylsis that tells you this is possible. chetty et al, address this issue somewhat in their paper, and as you'll suspect, the gains from replacing a poor VA teacher depends ENTIRELY on how good the replacement is.

    ReplyDelete
  15. Also, i'd like to add, that i have never disparaged your character and your intelligence, in the same way that you have mine.

    its ok -- the science is first, my feelings a far second.

    in fact, i think you are erudite, eloquent, and potentially influential in the philippines. thats why i'm spending time talking about philippine educ reform with you.

    ReplyDelete
  16. You started by claiming that I incorrectly cited Kevin Drum of Mother Jones. That is not correct. I hope you have indeed read the article written by Kevin Drum. The title of his article, I thought, was very clear, with regard to what he wanted to say. Thus, I do not understand why you started with your very first comment with that accusation in mind.

    Yes, Drum does deviate from what Chetty, Friedman and Rockoff wrote in their paper. The deviation, however, is not with the data or the findings. The difference lies in the interpretation of what policy changes entail. Chetty et al. interpret their findings as evidence that policies based on test scores which may define teacher quality can lead to improvements in the future quality of life of students. Drum does not subscribe to that and neither do I. These are big data and for a scientist, these are indeed fascinating but one must exercise great caution in drawing conclusions from big data that carry significant scatter and a large room for other factors. One important thing here to note is that the data have been obtained without a clear policy in place yet. Human beings respond to policies. When stakes are increased, gaming the system can occur, and at this point, both Atlanta and the District of Columbia are now under investigation of cheating.

    Understanding these studies is not easy. These are not simple studies and education is not a simple system. I chose Drum's article specifically because he brought this complicated study into something easier to grasp, zeroing on what the major finding of the study really means. The data presented by Chetty et al. are not questioned. These are products of good scientific work. Unfortunately, the implications are controversial. You thought that when I kept multiplying the numbers by years - it was inappropriate. It was "super-multiplicative-aggregation" (I am borrowing a term used by Professor of Education at Rutgers, Bruce D. Baker), which is quite interesting since one of the authors (Friedman) is quoted in a NY Times article as saying:

    “If you leave a low value-added teacher in your school for 10 years, rather than replacing him with an average teacher, you are hypothetically talking about $2.5 million in lost income,” said Professor Friedman, one of the coauthors.

    Diane Ravitch, in her comment regarding the study by Chetty et al., hits the right button, in my opinion:

    "Here are some obvious conclusions from the study: Teachers are really important. They make a lasting difference in the lives of their students. Some teachers are better than other teachers. Some are better at raising students' test scores.

    The problems of the study are not technical, but educational."

    I hope this helps you understand where I am coming from.

    ReplyDelete
  17. "You started by claiming that I incorrectly cited Kevin Drum of Mother Jones"

    no. i said : "no, that is not the paper or drum's claim. the paper's claim is:"

    so i said you misstated either drum or the paper. frankly, i care less about drum, and more about the paper.

    "Drum does not subscribe to that and neither do I."

    uhm, he does, but he calls the magnitude SMALL. i have provided arguments that they are large, but not too large as to be impossible.

    in other words, i think drum is wrong, for reasons based on mgt, economics and logic. if you disagree, i invite you to write your arguments down.

    and here is my question to you, a simple question that deals with the philosophy of science and policy making:

    if you dont subscribe to a "good" finding, doesnt it behoove you to think about WHY? if this is a "good" study, and I AGREE IT IS, shouldn't we be bayesian thinkers? when we get new, "good" info, shouldnt that make us change our priors (prior beliefs)?

    let me rewrite my earlier comment:

    "i'd like to bring the best that scientific research can bring to tough questions on education policy (and econ policy in general).

    that requires 2 things:

    1) i dont have ANY PRECONCEIVED IDEAS about what is good or not. I'll let research guide me.

    2) I need to understand exactly what the research is telling me.

    as a scientist, i hope you'd subscribe to 1) and 2). I ask you now: do you?"

    finally, diane ravitch's comment makes no sense to me. there are valuations of the gains from introducing better tteachers. this policy, potentially, has large effects on the education system and welfare (in terms of lifetime income).

    so when she says "The problems of the study are not technical, but educational.", i have no clue what she means.

    ReplyDelete