Comparison of ability estimates under computerized adaptive testing and linear computer-based test: Implications for assessment use in Africa

##plugins.themes.academic_pro.article.main##

Adeola Ayodeji Famoroti

Abstract

Introduction: Computer Based-Test (CBT) is seemed more preferred in ability estimation in this 21st century to traditional method called Paper and Pencil Test (PPT). The CBT consisted of two major types namely Computerized Adaptive Testing (CAT) and Linear Computer Based test (LCBT). The wider acceptability that CAT is receiving in the recent time calls for more empirical research in the area.


Purpose: The purpose of this study was to establish whether there is a significant difference between the mean of the ability estimates under LCBT and ability estimates under CAT and also to establish whether there is any significant agreement between the ability estimates under the two testing (LCBT and CAT) modes that could suggest that CAT can replace LCBT.


Methodology: Causal comparative type of non-experimental design was used. The study was carried out in private secondary schools in Ife East Local Government in Osun state, Nigeria. The study population consisted of 442 Junior Secondary School 3 students who have completed the Junior Secondary School Social Studies syllabus and also completed two stages of the study. The items used were standardized using 3PL model of Item Response Theory to establish all the necessary psychometric properties.


Results: The results showed that CAT has a mean value of (M = 0.646, STD = 0.8799) and LCBT has mean value of (M = 0.148, STD = 0.8538). The observed mean difference (M = 0.479, STD = 0.862) in examinees’ ability estimates under CAT and LCBT was statistically significant, t(441) = 12.129, p = .000. Also, there is significant agreement between the two ability estimates under LCBT and CAT (upper limit value = 2.248, lower limit value = -1.232, bias value = 0.5). The study concluded that ability estimate under CAT was better than that of LCBT; and also, that CAT testing mode can replace LCBT.


Recommendations: The study recommended that CAT should be adopted for use by the examining bodies and higher institutions of learning. Also, that seminar, workshop, and capacity building programs be organized for the concerned educational stakeholders with a view to widening their knowledge in the new area.

##plugins.themes.academic_pro.article.details##

How to Cite
Famoroti, A. A. (2023). Comparison of ability estimates under computerized adaptive testing and linear computer-based test: Implications for assessment use in Africa. Journal of Educational Research in Developing Areas, 4(1), 1-11. https://doi.org/10.47434/JEREDA.4.1.2023.1

References

  1. An, X. M., & Yung, Y. F. (2014). Item response theory: What it is and how you can use the IRT procedure to apply it. SAS institute.
  2. Bandele, S. O. (2019). Evaluation of general studies computer-based tests in universities in South West Nigeria. International Journal of Interdisciplinary Research Methods 6(3), 19-28.
  3. Bennett, R. E. (2015). The changing nature of educational assessment. Review of Research in Education, 39(1), 370-407.
  4. Bugbee, A. C. & Bernt, F. M. (1990). Testing by computer: Findings in six years of use 1982- 1988. Journal of Research on Computing in Education. 23(1), 87-100.
  5. Cisar, S. M., Radosav, D., Markoski, B., Pinter, R., & Cisar, P. (2010). Computer adaptive testing of student knowledge. Journal of Acta Polytechnica Hungarica, 7(4), 139-152.
  6. Eid, G. K. (2005). An investigation into the effects and factors influencing computer-based online math problem-solving in primary schools. Journal of Educational Technology Systems, 33(3), 223-240.
  7. Ejim, S. (2017). An overview of computer based test. Research proposal of student graduate member, IEEE – Nigeria section, Abubakar Tafawa Balewa University. DOI: 10.13140/RG.2.2.32040.88326. https://www.researchgate.net/publication/324208499_Overview_of_Computer_Based_Tests?enrichId=rgreq-16f145e6470c4cdb7e015ff78adda7c0-XXX&enrichSource=Y292ZXJQYWdlOzMyNDIwODQ5OTtBUzo2MTE2NTEzNDI1ODE3NjZAMTUyMjg0MDQ1MjkyMw%3D%3D&el=1_x_2&_esc=publicationCoverPdf
  8. Flowers, C., Kim, D., Lewis, P., & Davis, V. C. (2011). A comparison of computer-based testing and pencil-and-paper testing for students with a read-aloud accommodation. Journal of Special Education Technology, 26(1), 1-12.
  9. Garas, S. & Hassan, M. (2018) Student performance on computer-based tests versus paper-based tests in introductory financial accounting: UAE evidence. Academy of Accounting and Financial Studies Journal. 22(2), 1– 14.
  10. Garcia, E., Aryal, S., Spence-Almaguer, E., Rohr, D. & Walters, S. T. (2018). Use of the IRT model to validate test items from a technology assisted health coaching program. Open Journal of Statistics, 8, 519-532. https://doi.org/10.4236/ojs.2018.83034.
  11. Haahr, J. H. & Hansen, M. E. (2006). Adult skills assessment in Europe: Feasibility study. Policy and business analysis, final report.
  12. Han, K. C. T. (2018). Conducting simulation studies for computerized adaptive testing using SimulCAT: an instructional piece. Journal of Educational Evaluation for Health Professions, 15(20), 1-11.
  13. Jeong, H. (2012). A comparative study of scores on computer-based tests and paper-based tests. Behaviour & Information Technology, 33(4), 410-422. https://doi.org/10.1080/0144929x.2012.710647.
  14. Jimoh, R.G., Abduljaleel, K.S., & Kawu, Y.K. (2012). students’ perception of computer-based test (CBT) for examining undergraduate Chemistry courses. Journal of Emerging Trends in Computing and Information Sciences, 3(2), 125- 134.
  15. Khoshsima, M. H., Hosseini, S., & Hashemi, A. (2017). Cross-mode comparability of computer-based testing (CBT) versus paper-pencil based testing (PPT): An investigation of testing administration mode among Iranian intermediate EFL Learners Toroujeni1. English Language Teaching, 10 (2), 64-72.
  16. Luecht, R. M. & Sireci, S. G. (2011). A review of models for computer-based testing. College board research report 2011-12. https://files.eric.ed.gov/fulltext/ED562580.pdf
  17. Magis, D. & Barrada, J. R. (2017). Computerized adaptive testing with R: Recent updates of the package catR. Journal of Statistical Software. 76(1), 1-19. https://doi.org/10.18637/jss.v076.c01.
  18. Magyar, A. (2015). Comparing the measurement effectiveness of computer-based linear and adaptive tests. Dissertation theses university of Szeged faculty of arts doctoral school of education information and communication technologies in education.
  19. Merritt, S. (2009). Tips for computer based multiple choice tests (CBT, CBE and CAT). https://teneceblog.wordpress.com/2018/07/02/tips-for-computer-based-multiple-choice-tests-cbt-cbe-cat/
  20. Nebel, I-T., Klemma, T., Fasshauer, M., Muller, U., Verlohren, H., Klaiberg,
  21. A., and Paschke, R. (2004). Comparative analysis of conventional and an adaptive computer-based hypoglycaemia education programs. Patient education and counseling 53 (2004) 315–318. https://doi.org/10.1016/j.pec.2003.03.003.
  22. Okocha, F., Eyiolorunshe Toluwani, T., & Owolabi, S. (2017). Student perception and acceptance of computer-based testing: A case study of Landmark university students. Journal of Digital Innovations & Contemporary Research in Science, Engineering & Technology, 5(1), 25-32.
  23. Olumorin, O. C., Fakomogbon, A. M., Fasasi, A. Y., Olawale, O. C. & Olafare, O. F. (2013). Computer based tests: A system of assessing academic performance in university of Ilorin, Ilorin, Nigeria. American Academic & Scholarly Research Journal, 5(2), 110-117.
  24. Osadebe, P. U., & Esegbue, T. O. (2018). Evaluation of students’ academic performance in JAMB Chemistry test under the computer-based testing and paper pencil media in Delta state university. International Journal of Advanced Research. 6(5), 415–426.
  25. Rotou, O., Patsula, L., Steffen, M. & Rizavi, S. (2007). Comparison of multistage tests with computerized adaptive and paper-and-pencil tests. ETS research report. ETS, Princeton, NJ.
  26. Sadiq, F. I., & Onianwa, C. U. (2011). Towards a scalable web assessment system for post university matriculation examination in Nigeria. African Journal of Computer & ICT, 4 (2), 25-30. http://www.ajocict.net.
  27. Seo, D. G. & Choi, J. (2018). Post-hoc simulation study of computerized adaptive testing for the Korean Medical Licensing Examination. Journal of Educational Evaluation for Health Professions, https://doi.org/10.3352/jeehp.2018.15.14.
  28. Schnipke, D. L., & Reese, L. M. (1997). A comparison of testlet-based test designs for computerized adaptive testing. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
  29. Sheu, A.L. (2019). Test mode effect on students’ scores in an educational assessment course at the University of Ilorin, Nigeria. Journal of Humanity and Education (JOHE), 4(1), 130-147.
  30. Thiessen, U. I., & Mislevy, B. (2000). Testing algorithms. Computerized adaptive testing. In H. Wainer, (Eds.), A Primer. Lawrence Erlbaum Associates.
  31. Thurlow, M., Lazarus, S. S., Albus, D., & Hodgson, J. (2010). Computer-based testing: Practices and considerations Synthesis Report 78). University of Minnesota, National Center on Educational Outcomes.
  32. Vispoel, W. P., Rocklin, T. R., & Wang, T. (1994). Individual differences and test administration procedures: a comparison of fixed-item, computerized adaptive, and self-adapted testing. Applied Measurement in Education, 7, 53e79. http://dx.doi.org/10.1207/s15324818ame0701_5.
  33. Wang, H. (2010). Comparability of computerized adaptive and paper-pencil tests. http://images.pearsonassessments.com/images/tmrs/tmrs_rg/Bulletin_13.pdf,
  34. Wang, H. & Shin, D. (2010). Comparability of computerized adaptive and paper-pencil tests. Test, measurement & research service bulletin.https://pdfs.semanticscholar.org/6f1e/bcee90a66fcc969c4caba1362bedb39d505a.pdf
  35. Wang, K. (2017). A fair comparison of the performance of computerized adaptive testing and multistage adaptive testing. A dissertation submitted to Michigan State University in partial fulfilment of the requirements for the degree of Measurement and Quantitative Methods – Doctor of Philosophy.
  36. Weiss, D. J., & Kingsbury, G. G. (1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21, 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x.
  37. Yagcı M., Ekiz. H., ve Gelbal, S. (2011). Çevrimiçi sınav ortamlarının öğrencilerin akademik başarılarına etkisi. 5th international computer and instructional technologies symposium, Elazığ, Turkey, 22-24 September.