"Bear in mind that the wonderful things you learn in your schools are the work of many generations, produced by enthusiastic effort and infinite labor in every country of the world. All this is put into your hands as your inheritance in order that you may receive it, honor it, add to it, and one day faithfully hand it to your children. Thus do we mortals achieve immortality in the permanent things which we create in common." - Albert Einstein

Thursday, July 20, 2017

The Accuracy of Students' Surveys

Some colleges still provide questionnaires to students to evaluate their instructors. In educational research, students' self-reports are often used. It is therefore necessary to gauge how accurate these reports are. After all, a teacher knows quite well that even in basic recall tests, not all students get a perfect score. And these tests taken by students are of course high stakes than a simple survey which neither rewards nor punishes a student for accuracy. Since surveys still require some bit of cognitive skills, it is also possible that errors in students' surveys will not be random but systematic. The chance that a student who does not well academically will likewise not be so accurate in answering questions in a survey is high. In an instructor evaluation, it is probably not surprising to see results correlate with the students' academic performance. Yet, these surveys are sometimes used for merit decisions for the instructor. How reliable then are students' voices? In the Philippines, young children are even seen protesting against the Duterte administration.

Above copied from Bobi Tiglao Facebook page

Rosen and coworkers have recently investigated the accuracy of students' surveys. They find that as the questions become more potentially sensitive, accuracy dramatically falls:
We find that students are reasonably good reporters of course-taking patterns but poor reporters of more potentially sensitive questions, including when the student completed Algebra I and the grade earned in the course. We find that lack of accuracy in student survey reports is consistently related to several student characteristics.
The lack of accuracy is an important finding but more troubling is the observation that inaccuracy correlates with student characteristics. And this is vividly seen when students are asked to report the grades they received:

Above copied from Rosen et al.

In the above table, the numbers in bold are the percentages of correct responses for each letter grade. One can see that accuracy correlates with higher grades. However, this is probably much more than just a correlation with a student's cognitive ability since the errors are not random. A student is more likely to report incorrectly a higher grade than a lower one. One can see this with "C" students, they are more likely to report incorrectly a "B" than a "D" (47.7 versus 6.2, respectively). Responding to a survey means revealing something about ourselves. When such information becomes sensitive and potentially reflecting something not so good about ourselves, we are naturally inclined not to answer truthfully.

No comments:

Post a Comment