Measuring Teaching Quality of Secondary Mathematics and Science Residents: A Classroom Observation Framework

From Section:
Assessment & Evaluation
Mar. 01, 2019

Source: Journal of Teacher Education, Volume: 70 issue: 2, page(s): 139-154

In this paper, the authors report the reliability of the UCLA-IMPACT (Inspiring Minds through a Professional Alliance of Community Teachers) classroom observation rubric in secondary math (ICOR [IMPACT Classroom Observation Rubric]-Math) and secondary science (ICOR [IMPACT Classroom Observation Rubric]-Science) to assess teachers’ classroom practice and describe the rubrics’ formative use to
(a) provide a shared vision of humanizing pedagogy that dually reflect sound pedagogical strategies,
(b) foster the development of reflective practitioners, and
(c) inform program improvement anchored by social justice and humanizing pedagogy.

Research Focus
This study focuses on assessing the reliability of ICOR-Math and ICOR-Science classroom observation rubrics in secondary settings and the use of the rubric data to inform the authors’ teacher education program. The authors report the results of their efforts to develop and test these rubrics as part of a systematic evaluation of teaching quality among IMPACT resident teachers in their program.
From 2010 to 2013, teacher educators observed more than 300 mathematics and science residents using successive versions of the rubrics. The authors report the results of an empirical study investigating the extent to which trained experienced raters assessed instruction along the dimensions of the ICOR-Math and ICOR-Science instantiated in the observation protocols with appropriate levels of reliability. They investigated the psychometric properties of the observation protocols (sources of measurement error, reliability, and precision) to assess their ability to support inferences about the quality of math and science instruction on the field.

Research Questions
1: What are the sources of error variation affecting the scores given by trained, experienced raters?
2: What is the reliability of average scores under varying scoring scenarios, with the intended purpose of applying these understandings toward teacher development and improvement?

Results and Discussion
The authors’ report that the ICOR-Math and ICOR-Science, based on humanizing pedagogy, Common Core State Standards (CCSS) and Next Generation Science Standards (NGSS), inform content-specific observation rubrics and provide a powerful-shared vision of practice for IMPACT resident teachers in their teacher education program. In practice, the data from the rubrics allow faculty and mentors to provide targeted formative feedback to resident teachers based on rubric ratings.
To systematically assess teaching quality using the rubrics, it was necessary to determine the reliability of ratings data and note that these observed reliability coefficients were derived from a smallscale study and, thus, can most effectively be tied to programmatic evaluation and feedback.
While results are promising, the authors understand that training and clarity in language may further improve the reliability of ICOR-Science and ICOR-Math. Rater selection and resource selection for training are essential. It is clear that exemplars of each rubric level are necessary and that novel practices need further elaboration, for example, high-level task and characteristics of a democratic classroom.

Implications and Conclusion
The authors conclude that the ICOR-Math and ICOR-Science make theories and the program’s conception of good teaching visible. The rubrics provide a common language and public vision around humanizing pedagogy that allow university faculty, resident teachers, and their mentors to engage in powerful reflection around practice.
These tools helped teachers talk about their practice in relation to theory and the new standards, facilitating authentic connections between the university and schools. They discuss three specific changes to the teacher education program based on the development of the ICOR-Math and ICOR-Science:
(a) articulation of core practices,
(b) providing feedback to candidates, and
(c) program evaluation.
Based on ICOR-Math and ICOR-Science, the teacher education program faculty and field supervisors identified core practices that candidates would learn in methods courses and practice in the field.
The rubrics were used to anchor program spaces where mentor teachers in the partner schools modeled core practices for the residents.
ICOR-Math and ICOR-Science focused on formative feedback when observing residents in the field. ICOR-Math and ICOR-Science changed how field supervisors and mentor teachers gave feedback to residents in the field. Some used the observation rubric to engage residents in reflective conversation, following the classroom observation.

Some residents reported using the observation rubric as a self-assessment tool. This use facilitated conversations in the field about what aspects of a lesson went well, or did not go well, and next steps for improvement, and provided a direct tie to content methods courses at the university.
These rubrics are currently used by the program for formative development and support of teachers in the program during their first semester as teachers of record.
The authors conclude that while no rubric can capture the nuances of space, time, culture, and context, ICOR-Math and ICOR-Science are useful tools for making visible the work of humanizing pedagogy in CCSS- and NGSS-aligned classrooms.
The focus on social justice through a lens of equity, access, and connections to students is novel for extant rubrics and they believe necessary in accessing and providing opportunities to every student.

They indicate that the results of this study contribute to teacher development and the field of classroom observation rubrics through its focus on content and the epistemological practices that define math and science in their program; it demonstrates that programs can develop their own tools for assessment and growth with limited resources by calling on program and university assets, alumni, and established partnerships. 

Updated: Jul. 31, 2019
Assessment | Observations | Preservice teacher education | Social justice | Values