From Theory to Practice-: Applying the Relational Systems Evaluation to Evaluation Capacity Building
Author: Su-Ching Lin( Graduate Institute of Education, National Changhua University of Education ), Fu-Nan Su (Taichung Municipal Li-jen Junior High School)
Vol.&No.：Vol. 69, No. 1
Date：March 2024
Pages：99-130
DOI：https://doi.org/10.6209/JORIES.202403_69(1).0004
Abstract：
Program evaluation involves the systematic collection of information about a program to improve its quality (Rossi et al., 2004). Program evaluators may be program stakeholders rather than neutral third parties; this is particularly the case when educators are asked to design and implement curriculum reforms. In the case of Taiwan’s 12-year compulsory education curriculum, teachers are expected to design, implement, and evaluate the curriculum (the program) to ensure quality. This study used relational systems evaluation (RSE) as a model for evaluative capacity building (ECB) to equip teachers in different fields to evaluate interdisciplinary programs. Unlike a traditional evaluation model, which views the program as something set in stone to be implemented, RSE is based on a picture where a program and its implementation inform and shape each other for the program’s objectives to be achieved from the ground up (Bollinger, 2021). RSE is a cyclic process in which the local program context is evaluated in light of the relationship of program components to the whole. The changes to be effected, the scope of the evaluation, and the evaluative competencies to be developed are laid out in partnership with stakeholders using systems evaluation protocol (SEP). SEP, RSE, and ECB conceive of educators as both program practitioners and program evaluators who must thus have strong evaluative knowledge and skills (Trochim & Urban, 2021).
RSE is derived from theoretical approaches such as evolutionary evaluation (EE), relational developmental systems theory, and systems thinking (ST). In EE, all programs are viewed as cycles, and evaluation is integral to each cycle. In evaluating a system, evaluators must select an appropriate evaluation model that fits the current stage of the program’s cycle (Trochim & Urban, 2021). In RSDT, a program’s environment drives development and change, and evaluators must consider whether the program requires adjustment based on the local environment or context (Bornstein, 2006). Finally, in ST, evaluators must consider the dynamic relationship between parts of the program to the whole program when implementing changes. Moreover, evaluators must establish program boundaries to determine the scope of the evaluation, identify the causes of strengths and weaknesses, and determine the critical outcomes to be measured. In RSE, all these processes are seen to build evaluative competencies in a collaborative SEP process (Chauveron et al., 2021; Urban, Archibald et al., 2021).
ECB is a dynamic and intentional process for improving individual evaluation motivation, knowledge, skills, and attitudes; strengthening a team’s ability to conduct evaluations; continually optimizing evaluation quality; and making evaluation routine (Buckley et al., 2015; Clinton, 2014; Labin et al., 2012). This study analyzed three evaluative domains: evaluative thinking (ET), cultural competency (CC), and evaluative capacity (EC). Buckley et al. (2015) observe that ET is the core of ECB, which requires continual analysis and rethinking. Without ET, reviews stagnate, fail to get at essential qualities, and induce inappropriate or incorrect corrections. CC refers to the position and attitude of the evaluator toward the culture of the organization. CC recognizes that any decision made during the evaluation process reflects the values of the evaluator, from a statement of the purpose of the evaluation to the collection and analysis of data and interpretation and application of the results (Centers for Disease Control and Prevention [CDC], 2014; Frierson et al., 2002). EC refers to the basic competence of the evaluator in identifying stakeholders; establishing the intention, purpose, problem, and scope of the evaluation; selecting or developing assessment tools; collecting and analyzing evaluation data; and using evaluation results to improve a program (Frierson et al., 2002).
Although several studies have discussed the implementation of ECB, empirical research on this topic is limited (Wandersman, 2014). Moreover, only two empirical studies have applied the RSE model to ECB (Chauveron et al., 2021; Urban, Linver et al., 2021). These studies have indicated that RSE is an effective model for enhancing participants’ ET and EC. However, no studies applying RSE have been conducted in Taiwan. This study had two primary goals: (1) To implement an RSE-ECB model to improve reforms of Taiwan’s 12-year compulsory education system, and (2) to translate the theoretical insights from these models into practical steps or programs through a series of workshops and self-directed learning modules. Through this process, the study will expand the literature on ECB by exploring its application to Taiwan’s unique cultural landscape of compulsory education reform.
In ECB, the evaluation skills individuals acquire must be implemented in their workplace with the support of leaders in their organization (Chauveron et al., 2021; Labin et al., 2012); for teachers, these leaders would be their principal or department heads. Therefore, this study recruited a total of eight teachers in the disciplines of science and technology, art, and the humanities from one rural and one urban junior high school in addition to the principals of these schools. The study enabled these stakeholders to collaborate in employing ECB in the design, implementation, and assessment of a program of curriculum reform grounded in RSE theories.
The secondary goals of this study were to (1) analyze participant responses to determine the effectiveness of the ECB program; (2) analyze changes in participants’ ET, CC, and EC; (3) analyze participants’ application of RSE perspectives in developing school-wide curriculum evaluation measures; and (4) analyze participants’ views on the RSE model. This study adopted a mixed-method design. This study collected data on ET, CC, and EC scale scores and from ECB workshop satisfaction questionnaires, feedback sheets, e-portfolio professional growth files, and interview records. The participants provided positive feedback about the ECB workshops, and their ET, CC, and EC scores significantly improved. The school-wide curriculum evaluation measures they developed reflected RSE perspectives and the unique demands of each school’s curriculum. According to the participants, compared with the traditional linear evaluation model, which only incorporated feedback at the end of the school year, the RSE model reflected the essential objectives of the curriculum and enabled real-time feedback for program revisions. Furthermore, the RSE model helped practitioners determine whether the curriculum met the needs of students and the community. In short, the RSE model provided a useful and comprehensible curriculum evaluation method for teachers and principals, although its implementation was not without challenges.
Keywords：cultural competency, evaluative thinking, evaluative capacity, evaluative capacity building, relational systems evaluation