"Bear in mind that the wonderful things you learn in your schools are the work of many generations, produced by enthusiastic effort and infinite labor in every country of the world. All this is put into your hands as your inheritance in order that you may receive it, honor it, add to it, and one day faithfully hand it to your children. Thus do we mortals achieve immortality in the permanent things which we create in common." - Albert Einstein

Wednesday, March 26, 2014

Exam Questions: Those That Are Pure

Writing questions in an exam is not an easy task. Each question needs to be thoughtfully considered to see whether it is in fact assessing the right thing. When I was looking at practice questions for the Graduate Record Examination (GRE) back in 1987, I did not particularly like a question in the analytical section. It was a series of questions that followed a paragraph introducing and explaining the rules of a baseball game in the United States. Unlike basketball, a game I was more familiar with in the Philippines, substitution in a baseball game is permanent. The questions in the exam exploited this difference highlighting the rules regarding "pinch-hitting", "pinch-running", and "relief pitching". I thought this question was unfair. It was not only assessing one's analytical reasoning skills. Anyone who was familiar with the game clearly had a leg up. Anyone who grew up with America's favorite pastime sport definitely had an advantage.

It is therefore not surprising to see that scores in standardized exams are correlated with factors other than learning (poverty, race, gender) if the background or experiences of a child factor in the exam. There are "pure" questions, those that are assessing something very specific. Unfortunately, as problems become more complex, it becomes difficult to write questions that are independent of a student's background. The following is perhaps one of the very few examples of "pure" questions in math:


There are 100 in this list. These are not all unique. Some are in fact repeated. To some, this list is "drill and kill", an excessive repetition of simple and isolated skills. There is no "real world" connection. In a previous post, "National Achievement Test - Should We Abolish Standardized Testing?", a paper by W. James Popham is highlighted. In Popham's article, there is one sample math question deemed as "pure":

Above copied from W. James Popham, "Why Standardized Tests Don't Measure Educational Quality".
Compared to the previous "drill and kill" questions, the above is already "impure". The first sentence alone introduces "pears". Pears are from temperate zones so as a child growing up in the Philippines, I would not have been familiar with this fruit. Seeing something foreign right at the beginning of a question can easily produce anxiety. It does not matter whether it is relevant to what the question is really asking. In addition, the above question already uses words thus, in a way, it is likewise assessing reading comprehension.

The idea that tests should probe deeper can be a double-edged sword. By making questions more complex, extraneous factors can be introduced. The observed correlation between poverty and low scores in standardized exams therefore needs a second look. Do poor children really score lower than rich kids? Researchers in Germany recently published a paper that addresses a similar question:



The following are the highlights of this paper:

Highlights

Advantages for service class students in pure and realistic item category.
Significant class differences more likely in realistic than in pure items.
33% of pure and 62% of realistic items showed significant class differences.
Further item characteristics did not deliver a more systematic explanation.
In all, analyses provide mixed evidence to support research assumptions.


In this paper, working class students are those children who come from families whose parents are "skilled, semi-skilled and unskilled manual workers, workers in agricultural production, lower grade technicians, and supervisors of manual workers." The service class pupils are from privileged families. Their parents are "higher and lower grade professionals, administrators, officials, and higher grade technicians". Math questions in the international standardized exam TIMSS 2007 have been classified as either "pure (PM)", "weak/minimal real-life references (RM-)", "moderate real-life references (RM), or "strong real-life references (RM+)". To appreciate this classification, the following examples have been provided:


The important finding of this paper is that questions labeled with strong real-life references (RM+) show a markedly larger difference in performance between service and working children. It is true that there is a difference in performance across the board. Poor children also make more mistakes with "pure" questions, but the difference in performance is much bigger when the questions include elements from real life. Real-life questions seem to test more than the math - these questions seem to perform excellently in assessing one's social class as well.





No comments:

Post a Comment