Do We Know How to Evaluate Teaching

Given that teaching is one important factor that affects learning, it is surprising that we are so much in the dark when it comes to how to evaluate teaching. In fact, even in institutions of higher education where one may expect administrators to be more knowledgeable, practices known to be ineffective in evaluating teaching are still in place. Take, for instance, student evaluation of teaching. A number of universities, including Georgetown, are still using student course evaluations in spite of the evidence that student evaluation is not correlated with student learning.

How my students evaluated me
There is indeed a disconnect between educators and research when it comes to student evaluation of teaching (SET). A team of researchers from Canada has recently attempted to distill what we already know about SET and its relationship to learning outcomes. After a thorough and careful analysis of studies previously made on SET, the conclusion is that SET is not related to student learning.

Above copied from Studies in Educational Evaluation
The above, unfortunately, is not really new. Nobel laureate Carl Wieman wrote in the Change magazine more than a year ago:
Student course evaluations. There are literally thousands of articles on student course evaluations, and within them one can find supporting evidence for all possible opinions. Student evaluations do provide useful information: They are often used to flag instructors who have anomalously low scores in order to understand and address the reasons for the negative student opinions, which is appropriate. However, they have several critical failings with regard to the criteria listed above that should prevent them from serving as the primary method for evaluating the quality of teaching. 
First, they have some fundamental limitations that transcend any details. It is impossible for a student (or anyone else) to judge the effectiveness of an instructional practice except by comparing it with others that they have already experienced. If all they have experienced are lectures, they cannot meaningfully evaluate the effectiveness of lectures relative to other practices. This prevents student evaluations from encouraging or rewarding the adoption of more effective research-based teaching methods when such practices are seldom used at an institution.
The criteria Wieman listed are validity, meaningful comparisons, fairness, practicality, and chance for improvement. The validity criterion is clearly not met as Wiemen pointed out, "There is another basic limitation of student evaluations that ask how much was learned in the course, which most do. People are poor at evaluating their own learning, because it is difficult to know what you do not know. The accuracy of this evaluation is also sensitive to the level of expertise of the respondent."

Wieman therefore proposes with Sarah Gilbert a more effective way of evaluating teaching. This proposal is published in CBE Life Sciences Education. This evaluation scheme basically looks at practices implemented by an instructor and gives points to those practices that have been shown by research as beneficial to student learning.
Table 1. Teaching practices inventory categories
I. Course information provided (including learning goals or
II. Supporting materials provided
III. In-class features and activities
IV. Assignments
V. Feedback and testing
VI. Other (diagnostics, pre–post testing, new methods with
measures, etc.)
VII. Training and guidance of TAs
VIII. Collaboration or sharing in teaching
The following are examples provided by Wieman and Gilbert:

Looking at the above table, one can already anticipate a problem. The third column lists references that demonstrate benefits of a particular teaching practice. This is research. The main problem is that there is a huge disconnect between research and practice when it comes to student evaluation of teaching. How much more if one considers the entire field of teaching practices.

Thus, for years to come, we can only expect student evaluation of teaching to continue....