Is Using a SRI to Determine the Fate of Teachers or Commencement of Work Suitable? A Mixture IRT Analysis
Author: Ming-Ci Tseng(Department of Curriculum Design and Human Potentials Development, National Dong Hwa University), Haw-Jeng Chiou (College of Management, National Taiwan Normal University), Te-Sheng Chang (Department of Curriculum Design and Human Potentials Development, National Dong Hwa University), Pao-Feng Lo (Department of Curriculum Design and Human Potentials Development, National Dong Hwa University)
Vol.&No.：Vol. 58, No. 1
Date：March 2013
Pages：91-116
DOI：10.3966/2073753X2013035801004
Abstract：
This study examines the effects of variability in student ratings regarding instruction on decision-making for faculty teaching evaluation. A total of 6,111 undergraduate students from 173 classes in a university on the east coast of Taiwan were included in the research sample.
This study is different from previous studies regarding student ratings for instruction that are constructed in classical test theory. We use item response theory to analyze the heterogeneity of students, to rigorously examine the effects on student ratings regarding instruction. The results show that teachers may easily exceed the teaching criterion score set by the university when not considering the heterogeneity of the student ratings. However, the different latent types of the variability of student ratings may be important for interpreting the results of different student rating scores. The recommendations for university teaching and student ratings regarding instruction are created based on the results from this study.
Keywords：student ratings of instruction, mixture IRT analysis
《Full Text》
References：
- 周祝瑛（2009）。政大教師教學評鑑中「教學意見調查表」之研究。取自http://www3.nccu.edu. tw/~iaezcpc/C-Teaching%20Survey%20Form%20Research.htm【Chou, C.-P. (2009). National Cheng Chi University evaluation of teaching. Retrieved from http://www3.nccu. edu.tw/~iaezcpc/C-Teaching%20Survey%20Form%20Research.htm】
- 張德勝（2002）。學生評鑑教師教學：理論、實務與態度。臺北市：揚智。【Chang, T.-S. (2002). Student ratings of instruction: Theory, practice and attitudes. Taipei, Taiwan: Yang-Chih Book.】
- 張德勝、邱于真、羅寶鳳（2011）。題目順序對學生評鑑教師教學與學生自評影響之探索性研究。測驗學刊，58（2），367-389。【Chang, T.-S., Qiu, Y.-Z., & Lo, P.-F. (2011). The exploratory study on the effects of question order on student ratings of instruction and student self-evaluation. Psychological Testing, 58(2), 367-389.】
- 曾明基、邱于真、張德勝、羅寶鳳（2011）。學生認知歷程對學生評鑑教師教學的影響：階層線性模式分析。課程與教學季刊，14（3），157-180。【Tseng, M.-C., Qiu, Y.-Z., Chang, T.-S., & Lo, P.-F. (2011). HLM analysis of the effects of cognitive process on student ratings of instruction. Curriculum & Instruction Quarterly, 14(3), 157-180.】
- 曾明基、張德勝、羅寶鳳、邱于真（2012）。學生評鑑教師教學題目安排順序不同對學生評鑑教師的影響：MI與MMI分析取向。測驗學刊，59（1），131-156。【Tseng, M.-C., Chang, T.-S., Lo, P.-F., & Qiu, Y.-Z. (2012). The effects of question order difference on student ratings of instruction: MI and MMI analysis approaches. Psychological Testing, 59(1), 131-156.】
» More
- 周祝瑛（2009）。政大教師教學評鑑中「教學意見調查表」之研究。取自http://www3.nccu.edu. tw/~iaezcpc/C-Teaching%20Survey%20Form%20Research.htm【Chou, C.-P. (2009). National Cheng Chi University evaluation of teaching. Retrieved from http://www3.nccu. edu.tw/~iaezcpc/C-Teaching%20Survey%20Form%20Research.htm】
- 張德勝（2002）。學生評鑑教師教學：理論、實務與態度。臺北市：揚智。【Chang, T.-S. (2002). Student ratings of instruction: Theory, practice and attitudes. Taipei, Taiwan: Yang-Chih Book.】
- 張德勝、邱于真、羅寶鳳（2011）。題目順序對學生評鑑教師教學與學生自評影響之探索性研究。測驗學刊，58（2），367-389。【Chang, T.-S., Qiu, Y.-Z., & Lo, P.-F. (2011). The exploratory study on the effects of question order on student ratings of instruction and student self-evaluation. Psychological Testing, 58(2), 367-389.】
- 曾明基、邱于真、張德勝、羅寶鳳（2011）。學生認知歷程對學生評鑑教師教學的影響：階層線性模式分析。課程與教學季刊，14（3），157-180。【Tseng, M.-C., Qiu, Y.-Z., Chang, T.-S., & Lo, P.-F. (2011). HLM analysis of the effects of cognitive process on student ratings of instruction. Curriculum & Instruction Quarterly, 14(3), 157-180.】
- 曾明基、張德勝、羅寶鳳、邱于真（2012）。學生評鑑教師教學題目安排順序不同對學生評鑑教師的影響：MI與MMI分析取向。測驗學刊，59（1），131-156。【Tseng, M.-C., Chang, T.-S., Lo, P.-F., & Qiu, Y.-Z. (2012). The effects of question order difference on student ratings of instruction: MI and MMI analysis approaches. Psychological Testing, 59(1), 131-156.】
- Asparouhov, T., & Muthén, B. (2008). Multilevel mixture models. In G. R. Hancock & K. M. Samuelsen (Eds.), Advances in latent variable mixture models (pp. 27-51). Charlotte, NC: Information Age.
- Beran, T., & Violato, C. (2005). Ratings of university teacher instruction: How much do student and course characteristics really matter? Assessment & Evaluation in Higher Education, 30(6), 593-601. doi:10.1080/02602930500260688
- Bowling, N. A. (2008). Does the relationship between student ratings of course easiness and course quality vary across schools? The role of school academic rankings. Assessment & Evaluation in Higher Education, 33(4), 455-464. doi:10.1080/02602930701562965
- Celeux, G., & Soromenho, G. (1996). An entropy criterion for assessing the number of clusters in a mixture model. Journal of Classification, 13(2), 195-212. doi:10.1007/BF01246098
- Chang, T.-S. (2005). The validity and reliability of student ratings: Comparison between paper- pencil and online survey. Chinese Journal of Psychology, 47(2), 113-125.
- Chang, T.-S., & Ross, W. (2006). The influence of instructor, student, and course characteristics on student ratings: What do faculty and students believe? Journal of National Hualien University of Education, 23, 1-28.
- Cohen, E. H. (2005). Student evaluations of course and teacher: Factor analysis and SSA approaches. Assessment & Evaluation in Higher Education, 30(2), 123-136. doi:10.1080/0260293042000 264235
- Cohen, A. S., & Bolt, D. M. (2005). A mixture model analysis of differential item functioning. Journal of Educational Measurement, 42(2), 133-148. doi:10.1111/j.1745-3984.2005.00007
- d’Apollonia, S., & Abrami, P. C. (1997). Navigating student ratings of instruction. American Psychologist, 52(11), 1198-1208. doi:10.1037/0003-066X.52.11.1198
- Donald, J. G. (2006). Enhancing the quality of teaching in Canada. New Directions for Higher Education, 2006(133), 23-31. doi:10.1002/he.202
- Embretson, S., & Yang, X. (2006). Item response theory. In J. L. Green, G. Camilli, & P. B. Elmore (Eds.), Handbook of complementary methods in education research (pp. 385-409). Mahwah, NJ: Lawrence Erlbaum Associates.
- Frühwirth-Schnatter, S. (2006). Finite mixture and markov switching models. New York, NY: Springer.
- Greenwald, A. G., & Gillmore, G. M. (1997). Grading leniency is a removable contaminant of student ratings. American Psychologist, 52(11), 1209-1217. doi:10.1037/0003-066X.52.11.1209
- Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nijhoff.
- Kreber, C. (2006). Setting the context: The climate of university teaching and learning. New Directions for Higher Education, 2006(133), 5-11. doi:10.1002/he.200
- Marsh, H. W. (2007). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases, and usefulness. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319-383). Dordrecht, the Netherlands: Springer. doi:10.1007/1-4020-5742-3_9
- Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47(2), 149-174. doi:10.1007/BF02296272
- Masters, G. N. (1985). A comparison of latent trait and latent class analysis of Likert-type data. Psychometrika, 50(1), 69-82. doi:10.1007/BF02294149
- Masters, G. N. (1988). The analysis of partial credit scoring. Applied Measurement in Education, 1(4), 279-297. doi:10.1207/s15324818ame0104_2
- McDonald, R. P., & Mok, M. C. (1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30(1), 23-40. doi:10.1207/s15327906mbr3001_2
- McKeachie, W. J. (1997). Student ratings: The validity of use. American Psychologist, 52(11), 1218-1225. doi:10.1037/0003-066X.52.11.1218
- McLachlan, G., & Peel, D. (2000). Finite mixture models. New York, NY: John Wiley & Sons.
- Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159-176. doi:10.1177/014662169201600206
- Muthén, B. (2006). Should substance use disorders be considered as categorical or dimensional? Addiction, 101, 6-16. doi:10.1111/j.1360-0443.2006.01583.x
- Muthén, B., & Asparouhov, T. (2006). Item response mixture modeling: Application to tobacco dependence criteria. Addictive Behaviors, 31(6), 1050-1066. doi:10.1016/j.addbeh.2006.03.026
- Muthén, B., Asparouhov, T., & Rebollo, I. (2006). Advances in behavioral genetics modeling using Mplus: Applications of factor mixture modeling to twin data. Twin Research and Human Genetics, 9(3), 313-324. doi:10.1375/183242706777591317
- Muthén, L. K. (2010, February 11). Re: Model fit index WRMR [Online forum comment]. Retrieved from http://www.statmodel.com/discussion/messages/9/5096.html
- Muthén, L. K., & Muthén, B. O. (2006). Mplus user’s guide (4th ed.). Los Angeles, CA: Author.
- Nylund, K. L., Asparouhov, T., & Muthén, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A monte carlo simulation study. Structural Equation Modeling, 14(4), 535-569. doi:10.1080/10705510701575396
- Rice, R. E. (2006). Enhancing the quality of teaching and learning: The U.S. experience. New Directions for Higher Education, 2006(133), 13-22. doi:10.1002/he.201
- Rost, J. (1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14(3), 271-282. doi:10.1177/014662169001400305
- Rost, J. (1997). Logistic mixture models. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 449-463). New York, NY: Springer. doi:10.1007/978-1-4757-2691-6_26
- Sara, J. F., & Christine, D. (2006). Non normal and categorical data in structural equation modeling. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (pp. 269-314). Greenwich, CT: Information Age.
- Smit, A., Kelderman, H., & Flier, H. (1999). Collateral information and mixed Rasch models. Methods of Psychological Research Online, 4(3), 19-32.
- Spooren, P., & Mortelmans, D. (2006). Teacher professionalism and student evaluation of teaching: Will better teachers receive higher ratings and will better students give higher ratings? Educational Studies, 32(2), 201-214. doi:10.1080/03055690600631101
- Vincent, F. F., & Kennon, M. S. (2003). Student psychological need satisfaction and college teacher-course evaluations. Educational Psychology, 23(3), 235-247. doi:10.1080/01443410 32000060084
- Vincent, F. F., & Kennon, M. S. (2008). Teacher support, student motivation, student need satisfaction, and college teacher course evaluations: Testing a sequential path model. Educational Psychology, 28(6), 711-724. doi:10.1080/01443410802337794
- von Davier, M., & Yamamoto, K. (2004). Partially observed mixtures of IRT models: An extension of the generalized partial-credit model. Applied Psychological Measurement, 28(6), 389-406. doi:10.1177/0146621604268734
- Webler, W. D. (2006). German policy perspectives on enhancing the quality of student learning by university teaching. New Directions for Higher Education, 2006(133), 53-62. doi:10.1002/he. 205
- Yen, W. M., & Fitzpatrick, A. R. (2006). Item response theory. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 111-153). Westport, DC: National Council on Measurement in Education.