（專題）數位學習實驗研究品質評估與現況分析：以行動學習為例
作者:李漢岳(國立臺灣師範大學心測中心)、楊介銘(國立臺灣師範大學心測中心)、宋曜廷(國立臺灣師範大學教育心理與輔導學系)
卷期：62卷第2期
日期：2017年6月
頁碼：31-60
DOI：10.6209/JORIES.2017.62(2).02
摘要：
由於近年來行動裝置與教育應用軟體的快速發展，探討行動裝置輔助學習效果的實驗研究大量增加。然而，目前尚缺乏以評估實驗研究品質為主題所進行的回顧性研究。因此，本研究的目的即為廣泛蒐集ERIC和SSCI資料庫自2003年至2013年間，197篇以行動裝置輔助學習的實驗研究，並探討其在實驗設計嚴謹度、統計考驗之過程與結果的正確性，以及成效評估工具的品質等議題。研究結果顯示，實驗設計以準實驗研究占最多數（61%），但有四分之一的準實驗研究未考量實驗組與控制組之起點能力均等性問題。統計結果效度部分則發現，半數以上研究未符合統計基本假設，七成研究所採用之樣本數偏低，使其統計考驗力低落，效果值估計情形未臻精準。成效評估工具品質部分，則發現半數研究者未提供相關的測驗信度與效度資訊。最後，本研究根據研究結果與討論，對未來研究者提出相關建議。
關鍵詞：行動學習、研究品質、實驗設計
《詳全文》
參考文獻：
- 張菀珍、葉榮木（2014）。應用多媒體行動學習系統輔助大學生情緒管理學習成效與評量之探究。教育科學研究期刊，59（4），99-136。doi:10.6209/JORIES.2014.59(4).04【Chang, W.-J., & Yeh, Z.-M. (2014). Exploration of learning effectiveness and assessment of emotion management for college students by using a multimedia mobile learning system. Journal of Research in Education Sciences, 59(4), 99-136. doi:10.6209/JORIES.2014.59(4).04】
- Ardito, C., Costabile, M. F., de Angeli, A., & Lanzilotti, R. (2012). Enriching archaeological parks with contextual sounds and mobile technology. ACM Transactions on Computer-Human Interaction, 19(4), 1-30. doi:10.1145/2395131.2395136
- Aron, A., & Aron, E. N. (1999). Statistics for psychology (2nd ed.). Upper Saddle River, NJ: Prentice-Hall.
- Basoglu, E. B., & Akdemir, O. (2010). A comparison of undergraduate students English vocabulary learning: Using mobile phones and flash cards. Turkish Online Journal of Educational Technology, 9(3), 1-7. Retrieved from http://www.tojet.net
- Bebell, D., & O’Dwyer, L. M. (2010). Educational outcomes and research from 1:1 computing settings. Journal of Technology, Learning, and Assessment, 9(1), 5-15. Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1606
» 展開更多
- 張菀珍、葉榮木（2014）。應用多媒體行動學習系統輔助大學生情緒管理學習成效與評量之探究。教育科學研究期刊，59（4），99-136。doi:10.6209/JORIES.2014.59(4).04【Chang, W.-J., & Yeh, Z.-M. (2014). Exploration of learning effectiveness and assessment of emotion management for college students by using a multimedia mobile learning system. Journal of Research in Education Sciences, 59(4), 99-136. doi:10.6209/JORIES.2014.59(4).04】
- Ardito, C., Costabile, M. F., de Angeli, A., & Lanzilotti, R. (2012). Enriching archaeological parks with contextual sounds and mobile technology. ACM Transactions on Computer-Human Interaction, 19(4), 1-30. doi:10.1145/2395131.2395136
- Aron, A., & Aron, E. N. (1999). Statistics for psychology (2nd ed.). Upper Saddle River, NJ: Prentice-Hall.
- Basoglu, E. B., & Akdemir, O. (2010). A comparison of undergraduate students English vocabulary learning: Using mobile phones and flash cards. Turkish Online Journal of Educational Technology, 9(3), 1-7. Retrieved from http://www.tojet.net
- Bebell, D., & O’Dwyer, L. M. (2010). Educational outcomes and research from 1:1 computing settings. Journal of Technology, Learning, and Assessment, 9(1), 5-15. Retrieved from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1606
- Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014). A meta-analysis of blended learning and technology use in higher education: From the general to the applied. Journal of Computing in Higher Education, 26(1), 87-122. doi:10.1007/s12528 -013-9077-3
- Billings, E. S., & Mathison, C. (2012). I get to use an iPod in school? Using technology-based advance organizers to support the academic success of English learners. Journal of Science Education and Technology, 21(4), 494-503. doi:10.1007/s10956-011-9341-0
- Brooks, G., Miles, J. N. V., Torgerson, C. J., & Torgerson, D. J. (2006). Is an intervention using computer software effective in literacy learning? A randomised controlled trial. Educational Studies, 32(2), 133-143. doi:10.1080/03055690500416116
- Burston, J. (2015). Twenty years of MALL project implementation: A meta-analysis of learning outcomes. ReCALL, 27(1), 4-20. doi:10.1017/S0958344014000159
- Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multritrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105. doi:10.1037/h0046016
- Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Chicago, IL: Rand McNally.
- Chao, P.-Y., & Chen, G.-D. (2009). Augmenting paper-based learning with mobile phones. Interacting with Computers, 21(3), 173-185. doi:10.1016/j.intcom.2009.01.001
- Chen, N.-S., Hsieh, S.-W., & Kinshuk. (2008). Effects of short-term memory and content representation type on mobile language learning. Language Learning & Technology, 12(3), 93-113. Retrieved from http://llt.msu.edu/vol12num3/chenetal.pdf
- Chen, Y.-S., Kao, T.-C., & Sheu, J.-P. (2005). Realizing outdoor independent learning with a butterfly-watching mobile learning system. Journal of Educational Computing Research, 33(4), 395-417. doi:10.2190/0PAB-HRN9-PJ9K-DY0C
- Cheung, A. C.-K., & Slavin, R. E. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research Review, 9, 88-113. doi:10.1016/j.edurev.2013.01.001
- Chu, H.-C., Hwang, G.-J., Huang, S.-X., & Wu, T.-T. (2008). A knowledge engineering approach to developing e-libraries for mobile learning. Electronic Library, 26(3), 303-317. doi:10.1108/026 40470810879464
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates.
- Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49(12), 997-1003. doi:10. 1037/0003-066X.49.12.997
- D’Alonzo, K. T. (2004). The Johnson-Neyman procedure as an alternative to ANCOVA. West Journal of Nursing Research, 26(7), 804-812. doi:10.1177/0193945904266733
- Dupont, W. D., & Plummer, W. D. (1990). Power and sample size calculations: A review and computer program. Controlled Clinical Trials, 11(2), 116-128. doi:10.1016/0197-2456(90)90005-M
- Edwards, C. M., Rule, A. C., & Boody, R. M. (2013). Comparison of face-to-face online mathematics learning of sixth graders. Journal of Computers in Mathematics and Science Teaching, 32(1), 25-47. Retrieved from https://www.learntechlib.org/p/39231
- Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191. doi:10.3758/BF03193146
- Ferguson, G. A., & Takane, Y. (1989). Statistical analysis in psychology and education (6th ed.). New York, NY: McGraw-Hill.
- Fleischer, H. (2012). What is our current understanding of one-to-one computer projects: A systematic narrative research review. Educational Research Review, 7(2), 107-122. doi:10.1016/ j.edurev.2011.11.004
- Fraenkel, J. R., Wallen, N. E., & Hyun, H. (2011). How to design and evaluate research in education. New York, NY: McGraw-Hill Education.
- Frohberg, D., Goth, C., & Schwabe, G. (2009). Mobile learning projects– A critical analysis of the state of the art. Journal of Computer Assisted Learning, 25(4), 307-331. doi:10.1111/j.1365- 2729.2009.00315.x
- Gentry, T. (2008). PDAs as cognitive aids for people with multiple sclerosis. American Journal of Occupational Therapy, 62(1), 18-27. doi:10.5014/ajot.62.1.18
- Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42(3), 237-288. doi:10.2307/1169991
- Hedman, L., & Sharafi, P. (2004). Early use of internet-based educational resources: Effects on students’ engagement modes and flow experience. Behaviour & Information Technology, 23(2), 137-146. doi:10.1080/01449290310001648251
- Howell, D. C. (2002). Statistical methods for psychology (5th ed.). Pacific Grove, CA: Duxbury.
- Hsi, S. (2003). A study of user experiences mediated by nomadic web content in a museum. Journal of Computer Assisted Learning, 19(3), 308-319. doi:10.1046/j.0266-4909.2003.jca_023.x
- Hsu, Y.-C., & Ching, Y.-H. (2013). Mobile computer-supported collaborative learning: A review of experimental research. British Journal of Educational Technology, 44(5), E111-E114. doi:10. 1111/bjet.12002
- Huck, S. W. (2008). Reading statistics and research (5th ed.). Boston, MA: Pearson Education.
- Hung, J.-L., & Zhang, K. (2012). Examining mobile learning trends 2003-2008: A categorical meta-trend analysis using text mining techniques. Journal of Computer Higher Education, 24(1), 1-17. doi:10.1007/s12528-011-9044-9
- Hwang, G.-J., Shi, Y.-R., & Chu, H.-C. (2011). A concept map approach to developing collaborative mindtools for context-aware ubiquitous learning. British Journal of Educational Technology, 42(5), 778-789. doi:10.1111/j.1467-8535.2010.01102.x
- Hwang, G.-J., & Tsai, C.-C. (2011). Research trends in mobile and ubiquitous learning: A review of publications in selected journals from 2001 to 2010. British Journal of Educational Technology, 42(4), E65-E70. doi:10.1111/j.1467-8535.2011.01183.x
- Keppel, G. (1991). Design and analysis: A researcher’s handbook (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
- Kerawalla, L., O’Connor, J., Underwood, J., duBoulay, B., Holmberg, J., Luckin, R., …Tunley, H. (2007). Exploring the potential of the homework system and tablet PCs to support continuity of numeracy practices between home and primary school. Educational Media International, 44(4), 289-303. doi:10.1080/09523980701680904
- Kert, S. B. (2011). The use of SMS support in programming education. Turkish Online Journal of Educational Technology, 10(2), 268-273. Retrieved from http://files.eric.ed.gov/fulltext/ EJ932245.pdf
- Ketamo, H. (2003). An adaptive geometry game for handheld devices. Educational Technology & Society, 6(1), 83-95. Retrieved from http://ifets.info/journals/6_1/ketamo.pdf
- Kirk, R. E. (1995). Experimental design procedures for the behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole.
- Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56(5), 746-759. doi:10.1177/0013164496056005002
- Kondo, M., Ishikawa, Y., Smith, C., Sakamoto, K., Shimomura, H., & Wada, N. (2012). Mobile assisted language learning in university EFL courses in Japan: Developing attitudes and skills for self-regulated learning. ReCALL, 24(2), 169-187. doi:10.1017/S0958344012000055
- Kulik, C.-L. C., & Kulik, J. A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 7(1-2), 75-94. doi:10.1016/0747-5632(91)90030-5
- Liao, C. Y.-K. (2007). Effects of computer-assisted instruction on students’ achievement in Taiwan: A meta-analysis. Computers & Education, 48(2), 216-233. doi:10.1016/j.compedu.2004.12.005
- Lin, T.-J., Duh, H. B.-L., Li, N., Wang, H.-Y., & Tsai, C.-C. (2013). An investigation of learners’ collaborative knowledge construction performances and behavior patterns in an augmented reality simulation system. Computers & Education, 68, 314-321. doi:10.1016/j.compedu.2013. 05.011
- Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
- Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. New York, NY: Oxford University press.
- Liu, C.-C., Chou, C.-C., Liu, B.-J., & Yang, J.-W. (2006). Improving mathematics teaching and learning experiences for hard of hearing students with wireless technology-enhanced classrooms. American Annals of the Deaf, 151(3), 345-355. doi:10.1353/aad.2006.0035
- Liu, T.-Y., & Chu, Y.-L. (2010). Using ubiquitous games in an English listening and speaking course: Impact on learning outcomes and motivation. Computers & Education, 55(2), 630-643. doi:10. 1016/j.compedu.2010.02.023
- Liu, C.-C., & Hong, Y.-C. (2007). Providing hearing-impaired students with learning care after classes through smart phones and the GPRS network. British Journal of Educational Technology, 38(4), 727-741. doi:10.1111/j.1467-8535.2006.00656.x
- Liu, T.-C., Lin, Y.-C., & Paas, F. (2014). Effects of prior knowledge on learning from different compositions of representations in a mobile learning environment. Computers & Education, 72, 328-338. doi:10.1016/j.compedu.2013.10.019
- Looi, C.-K., Zhang, B., Chen, W., Seow, P., Chia, G., Norris, C., & Soloway, E. (2011). 1:1 mobile inquiry learning experience for primary science students: A study of learning effectiveness. Journal of Computer Assisted Learning, 27(3), 269-287. doi:10.1111/j.1365-2729.2010.00390.x
- Mayr, S., Erdfelder, E., Buchner, A., & Faul, F. (2007). A short tutorial of GPower. Tutorials in Quantitative Methods for Psychology, 3(2), 51-59. doi:10.20982/tqmp.03.2.p051
- Morris, N. P., Ramsay, L., & Chauhan, V. (2012). Can a tablet device alter undergraduate science students’ study behavior and use of technology? Advances in Physiology Education, 36(2), 97-107. doi:10.1152/advan.00104.2011
- Oberg, A., & Daniels, P. (2013). Analysis of the effect a student-centred mobile learning instructional method has on language acquisition. Computer Assisted Language Learning, 26(2), 177-196. doi:10.1080/09588221.2011.649484
- Pagano, R. R. (2007). Understanding statistics in the behavioral sciences (8th ed.). Belmont, CA: Thomson Higher Education.
- Penuel, W. R. (2006). Implementation and effects of one to one computing initiatives: A research synthesis. Journal of Research on Technology in Education, 38(3), 329-348. doi:10.1080/ 15391523.2006. 10782463
- Rockinson-Szapkiw, A. J., Courduff, J., Carter, K., & Bennett, D. (2013). Electronic versus traditional print textbooks: A comparison study on the influence of university students’ learning. Computers & Education, 63, 259-266. doi:10.1016/j.compedu.2012.11.022
- Roschelle, J., Rafanan, K., Bhanot, R., Estrella, G., Penuell, B., Nussbaum, M., & Claro, S. (2010). Scaffolding group explanation and feedback with handheld technology: Impact on students’ mathmatics learning. Educational Technology Research and Development, 58(4), 399-419. doi:10.1007/s11423-009-9142-9
- Sandberg, J., Maris, M., & de Geus, K. (2011). Mobile English learning: An evidence-based study with fifth graders. Computers & Education, 57(1), 1334-1347. doi:10.1016/j.compedu.2011.01.015
- Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton-Mifflin.
- Sink, C. A., & Mvududu, N. H. (2010). Statistical power, sampling, and effect sizes: Three keys to research relevancy. Counseling Outcome Research and Evaluation, 1(2), 1-18. doi:10.1177/ 2150137810373613
- Slavin, R. E. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15-21. doi:10.3102/0013189X031007015
- Slavin, R. E. (2003). A reader’s guide to scientifically based research. Educational Leadership, 60(5), 12-16. Retrieved from http://www.ascd.org/publications/educational-leadership/feb03/vol60/ num05/A-Reader’s-Guide-to-Scientifically-Based-Research.aspx
- Slavin, R., & Madden, N. A. (2011). Measures inherent to treatments in program effectiveness reviews. Journal of Research on Educational Effectiveness, 4(4), 370-380. doi:10.1080/19345747. 2011.558986
- Solhaug. (2009). Two configurations for accessing classroom computers: Differential impact on students’ critical reflections and their empowerment. Journal of Computer Assisted Learning, 25(5), 411-422. doi:10.1111/j.1365-2729.2009.00318.x
- Sung, Y.-T., Chang, K.-E., Lee, Y.-H., & Yu, W.-C. (2008). Effects of a mobile electronic guidebook on visitors’ attention and visiting behaviors. Educational Technology & Society, 11(2), 67-80. Retrieved from http://www.ifets.info/journals/11_2/7.pdf
- Sung, Y.-T., Chang, K.-E., & Liu, T.-C. (2016). The effects of integrating mobile devices with teaching and learning on students’ learning performance: A meta-analysis and research synthesis. Computers & Education, 94, 252-275. doi:10.1016/j.compedu.2015.11.008
- Sung, Y.-T., Chang, K.-E., & Yang, J.-M. (2015). How effective are mobile devices for language learning? A meta-analysis. Educational Research Review, 16, 68-84. doi:10.1016/j.edurev.2015. 09.001
- Tabachnick, B., & Fidell, L. (2007). Using multivariate statistics. Boston, MA: Pearson/Allyn & Bacon.
- Tan, T.-H., Liu, T.-Y., & Chang, C.-C. (2007). Development and evaluation of an RFID-based ubiquitous learning environment for outdoor learning. Interactive Learning Environments, 15(3), 253-269. doi:10.1080/10494820701281431
- U.S. Department of Education. (2002). Strategic plan 2002-2007. Retrieved from http://www2. ed.gov/about/reports/strat/plan2002-07/index.html
- Valentine, J. C., & Cooper, H. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13(2), 130-149. doi: 10.1037/1082-989X.13.2.130
- Valentine, J. C., & McHugh, C. M. (2007). The effects of attrition on baseline comparability in randomized experiments in education: A meta-analysis. Psychological Methods, 12(3), 268-282. doi:10.1037/1082-989X.12.3.268
- What Works Clearinghouse. (2014). WWC procedures and standards handbook version 3.0. Retrieved from http://ies.ed.gov/ncee/wwc/Handbooks
- Wolbers, K. A., Dostal, H. M., Graham, S., Cihak, D., Kilpatrick, J. R., & Saulsburry, R. (2015). The writing performance of elementary students receiving strategic and interactive writing instruction. Journal of Deaf Studies and Deaf Education, 20(4), 385-398. doi:10.1093/deafed/ env022
- Wong, L.-H., & Looi, C.-K. (2011). What seams do we remove in mobile assisted seamless learning? A critical review of the literature. Computers & Education, 57(4), 2364-2381. doi:10.1016/j. compedu.2011.06.007
- Wu, W.-H., Wu, Y.-C., Chen, C.-Y., Kao, H.-Y., Lin, C.-H., & Huang, S.-H. (2012). Review of trends from mobile learning studies: A meta-analysis. Computers & Education, 59(2), 817-827. doi: 10.1016/j.compedu.2012.03.016
- Yen, J.-C., Lee, C.-Y., & Chen, I-J. (2012). The effects of image-based concept mapping on the learning outcomes and cognitive processes of mobile learners. British Journal of Educational Technology, 43(2), 307-320. doi:10.1111/j.1467-8535.2011.01189.x
- Zurita, G., & Nussbaum, M. (2004). Computer supported collaborative learning using wirelessly interconnected handheld computers. Computers & Education, 42(3), 289-314. doi:10.1016/j. compedu.2003.08.005
- Zurita, G., Nussbaum, M., & Salinas, R. (2005). Dynamic grouping in collaborative learning supported by wireless handhelds. Educational Technology & Society, 8(3), 149-161. Retrieved from http://www.ifets.info/journals/8_3/14.pdf
Journal directory listing - Volume 62 (2017) - Journal of Research in Education Sciences【62(2)】June (Special Issue: Digital Learning: Retrospect and Prospect)
(Special Issue) Quality Assessment and Situational Analysis of Experimental E-Learning Designs:A Case Study of Mobile Learning
Author: Han-Yueh Lee(Research Center for Psychological and Educational Testing, National Taiwan Normal University), Je-Ming Yang(Research Center for Psychological and Educational Testing, National Taiwan Normal University), Yao-Ting Sung(Department of Educational Psychology and Counseling, National Taiwan Normal University)
Vol.&No.：Vol. 62, No.2
Date：June 2017
Pages：31-60
DOI：10.6209/JORIES.2017.62(2).02
Abstract：
Because of the substantial development of mobile devices and educational software in recent years, the results of mobile learning–based interventions represent a popular research topic for investigation. Although experiment quality is the basis of empirical research, few studies have explored this issue. Thus, the present study investigated the shortfalls in existing experimental research designs related to mobile learning over the past decade and offers suggestions in this paper. The researchers collected data from all 197 experimental studies on mobile learning published in the Education Resources Information Center and Institute of Science Index from 2003 to 2013. The findings of the present study are described as follows: (1) Quasi-experimental designs represent the most frequently used design type (61%); however, among the quasi-experimental studies, 25% did not consider baseline equivalence. (2) Over half of the studies may not have met basic statistical assumptions, and approximately 70% used insufficient sample sizes, leading to low statistical power and imprecise effect size estimation. (3) Half of the studies did not provide information on test reliability and validity. Finally, this paper discusses the results and their implications for future research and practice.
Keywords：experimental design, mobile learning, research quality