Source: Professional Development in Education, Vol. 37, No. 1, February 2011, 143–157.
(Reviewed by the Portal Team)
The purpose of this article is to address the issue of Continuing Professional Development (CPD) evaluation in education. The paper focuses on what are often called ‘level’ models for evaluating development and training. Such models draw on the influential work of Kirkpatrick and Guskey, which attempt to trace the processes through which CPD interventions achieve outcomes.
The authors have used the term ‘level models’ to draw on an evaluation tradition which posits that programme design and implementation involve a series of inter-related components and the role of evaluation is to assess one or more of these components and the inter-relationships between them.
The aims of the article are threefold:
1. to consider the ways in which level models have been articulated and critiqued;
2. to explain how our own evaluation work has been influenced by these models and critiques;
3. to stand back from these models’ use in practice to consider some more fundamental ontological and epistemological questions to which they give rise but which are not often discussed in evaluation reports.
The authors discussed various level models which have been used in the literature in a variety of ways.
This discussion suggests that level models raise two key questions.
First, what causative relationships are assumed to hold between a training or development experience and various kinds of potential outcomes?
Secondly, how does the experience interact with situational factors associated with individuals and organizational arrangements, and how do these interactions affect outcomes?
This led the authors to develop a model of the effects of CPD programmes using a broader set of types of variables. The frame for the model is constructed around the following sets of key variables, and their interactions:
● Interventions: the CPD activities themselves.
● Antecedents: those factors associated with individual participants that affect their ability to benefit from the opportunities offered to them.
● Moderating factors: variables in the school and wider environment that influence whether, and how, the interventions lead, via the achievement of intermediate outcomes to producing final outcomes. These factors help to explain why apparently similar activities have different consequences for different individuals, teams and schools.
● Intermediate outcomes: those outcomes of the CPD activities that are conceived to be pre-conditions for the achievement of the final outcomes, particularly learning and changes in participant behaviour.
● Final outcomes: the intended effects of the CPD activities, primarily relating to effects on organisations, teachers and students.
The authors conclude that level models can be used in a number of ways such as positivist tradition, realist perspective and constructivist perspective.
Furthermore, level models can be used with differing emphases, such as methodological approach (from quantitative to qualitative); the manner in which the data are to be judged or valued (from unitary – by the commissioner or the evaluator – to plural); and the user focus of the evaluation effort (from instrumental to enlightenment) (Alkin and Ellett, 1985).
However, the choices made about the level models' use will need to reflect both theoretical choices and practical considerations.
References
Alkin, M. and Ellett, F., 1985. Evaluation models and their development. In: T. Husen and N. Postlethwaite, eds. International encyclopaedia of education: research and studies. sOxford: Pergamon, 1760–1766.
Guskey, T., 2000. Evaluating professional development. Thousand Oaks, CA: Corwin Press.
Kirkpatrick, D., 1998. Evaluating training programmes: the four levels. 2nd ed. San Francisco, CA: Berrett-Koehler.