Integrating technology and non-routine problems: the design, development, and validation of a computer-aided assessment for non-routine problem-solving
DOI:
https://doi.org/10.33830/ijdmde.v2i1.11721Keywords:
computer-aided assessment, Design and Development Research, Higher-order thinking skills, Non-routine problemsAbstract
The increasing demand for higher-order thinking skills in 21st-century education underscores the need for innovative resources, as traditional materials often focus on routine problems with limited opportunities for non-routine problem-solving. This study designed, developed, and validated a computer-aided assessment for non-routine problem-solving using the Design and Development Research method, conducted in three phases: needs analysis, design and development, and evaluation. Data collection included a literature review, systematic design and development, expert validation, and user feedback, analyzed through descriptive statistics and thematic analysis. Findings show that the computer-aided assessment supports non-routine problem-solving and addresses limitations of traditional assessments. The needs analysis identified gaps in current methodologies and the benefits of technology-enhanced assessments. The design and development phase produced an assessment promoting cognitive engagement, while the evaluation phase gathered expert validation and user feedback. Five experts in information technology and mathematics education provided positive evaluations of the assessment’s functionality, accessibility, and cognitive support. Feedback from 15 students across three educational institutions indicated enhanced engagement, independent learning, and deeper mathematical understanding. Although minor technical issues were noted, they did not significantly impact the computer-aided assessment’s quality. The validated computer-aided assessment shows potential as a digital tool for fostering non-routine problem-solving and interactive learning. It suggests that computer-aided assessments with non-routine problems may enhance higher-order thinking skills by promoting cognitive engagement and personalized learning. Future research could explore its long-term impact, adaptability across disciplines, and potential for adaptive learning and collaborative features to maximize educational effectiveness.
References
Ahmed, S.K. (2024). How to choose sampling technique and determine sample size for research: A simplified guide for researchers. Oral Oncology Reports, 12:100662. https://doi.org/ 10.1016/j.oor.2024.100662
Alqurni, J. S. (2023). Evaluating the user interface and usability approaches for e-learning systems. International Journal of Information Technology and Web Engineering, 18(1), 1-25. https://doi.org/10.4018/IJITWE.333638
Almuhanna, M. A. (2024). Teachers' perspectives of integrating AI-powered technologies in K-12 education for creating customized learning materials and resources. Education and Information Technologies, 1-29. https://doi.org/10.1007/s10639-024-13257-y
Andrade, R.R., Fortes, E.C., & Mabilangan, R.A. (2020). Problem solving heuristics and mathematical abilities of heterogeneous learners. Universal Journal of Educational Research 8(11), 5114-5126. https://doi.org/10.13189/ujer.2020.081111
Anstey, L., & Watson, G. (2018). Rubric for elearning tool evaluation. Centre for Teaching and Learning Western University. https://teaching.uwo.ca/pdf/elearning/Rubric-for-eLearning-Tool-Evaluation.pdf
Atlintas, O. (2022). The Use of open-ended items for giving feedback during the formative assessment process. International Journal of Assessment Tools in Education 9, Special Issue, 109–125. https://doi.org/10.21449/ijate.1132981
Barana, A., Boetti, G. & Marchisio, M. (2022). Self-assessment in the development of mathematical problem-solving skills. Education Sciences, 2022, 12, 81. https://doi.org/10.3390/edusci12020081
Bearman, M., Nieminen, J.H., & Ajjawi, R. (2023). Designing assessment in a digital world: An organising framework. Assessment & Evaluation in Higher Education, 48(3), 291–304. https://doi.org/10.1080/02602938.2022.2069674
Berisha, V. (2015). The general characteristics of mathematics textbooks for lower secondary school in Kosovo. International Journal of Novel Research in Education and Learning, 292), 19-23. https://www.noveltyjournals.com
Bianchi, N., Song, H., & Lu, Y. (2022). The effect of computer-assisted learning on students’ long-term development. Journal of Development Economics, 158, 102919.
https://doi.org/10.1016/j. jdeveco.2022.102919
Boyatzis, R.E. (1998). Transforming qualitative information: The matic analysis and code development. Sage Publications, Inc. https://psycnet.apa.org/record/1998-08155-000
Braun, V., & Clarke, V. (2006). Using thematic analysis in psy chology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Creswell, J.W., & Creswell, J.D. (2018). Research design: qualitative, quantitative, and mixed methods approaches (5th ed.). SAGE Publications, Inc. https://spada.uns.ac.id/pluginfile.php/510378/mod_resource/content/1/creswell.pdf
Davies, B., Smart, T., Geraniou, E. & Crisan, C. (2022). STACKification: automating assessments in tertiary mathematics. Twelfth Congress of the European Society for Research in Mathematics Education (CERME12), Feb 2022, Bozen-Bolzano, Italy. Hal-03750584. https://hal.science/hal-03750584v1/document
Dyrvold, A., & Bergvall, I. (2023). Computer-based assessment in mathematics: Issues about validity. LUMAT: International Journal on Math, Science and Technology Education, 11(3), 49-76. https://doi.org/10.31129/LUMAT.11.3.1877
Elbyaly, M.Y.H. & Elfeky, A.I.M. (2023). The impact of problem-solving programs in developing critical thinking skills. European Chemical Bulletin, 12(6), 6636-6642. https://doi.org/10.31838/ecb/2023.12.si6.588
Elliott, V. (2018). Thinking about the coding process in qualitative data analysis. Qualitative Report, 23(11), 2850–2861. https://doi.org/10.46743/2160-3715/2018.3560
Ellwood, B. (2021). Spacing math practice across multiple sessions improves students’ test scores and helps them accurately judge their learning. PsyPost.
Emeny, W. G., Hartwig, M. K., & Rohrer, D. (2021). Spaced mathematics practice improves test scores and reduces overconfidence. Applied Cognitive Psychology. https://doi.org/10.1002/acp.3814
Evans, C. M. (2020). Measuring student success skills: A review of the literature on collaboration. Dover, NH: National Center for the Improvement of Educational Assessment. https://files.eric.ed.gov/fulltext/ED607774.pdf
Fan, L. & Zhu, Y. (2000). Problem solving in singaporean secondary mathematics textbooks. The Mathematics Educator, 5(1), 117-141. https://repository.nie.edu.sg/server/api/core/bitstreams/ 978a61aa-a672-4f1a-ba95-605fdb26d399/content
Govindasamy, P., Abdullah, N., & Ibrahim, R. (2023). Employing modified design and development research (DDR) approach to develop creativity clay module to teach special educational needs (SEN) students with learning disabilities. International Journal of Academic Research in Business and Social Sciences, 13(3), 461 – 476. https://doi.org/10.6007/IJARBSS/v13-i3/15773
Gunesekera, A.I., Bao, Y. & Kibelloh, M. (2019). The role of usability on e-learning user interactions and satisfaction: a literature review. Journal of Systems and Information Technology, 21(3), 368-394. https://doi.org/10.1108/JSIT-02-2019-0024
Hasbullah, N.A., Fabil, N., Yusoff, A.M., & Azmi, A.S. (2022). Adapting a design and development research (DDR) approach in designing the mobile application model for tarannum subject. Advances in Humanities and Contemporary Studies 3(2), 193-204. https://doi.org/10.30880/ahcs.2022.03.02.018
Hughes, M., Salamonson, Y., & Metcalfe, L. (2020). Student engagement using multiple-attempt “Weekly Participation Task” quizzes with undergraduate nursing students. Nurse Education in Practice, 46, 1-6. https://doi.org/10.1016/j.nepr.2020.102803
Ikart, E. (2019). Survey questionnaire survey pretesting method: An evaluation of survey questionnaire via expert reviews technique. Asian Journal of Social Science Studies, 4(2). https://doi.org/10.20849/ajsss.v4i2.565
Jaya, S., Zaharudin, R., Hashim, N. A., Zaid, S. M., Ithnin, M. A., Mapjabil, J., & Nordin, M. N. (2021). Employing design and development research (DDR) approach in designing next generation learning spaces (NGLS) in teachers' pedagogy and technology tools. Review of International Geographical Education Online, 11(7), 1237–1246. https://doi.org/10.48047/rigeo.11.07.116
Jhangiani, R.S., Chiang, I. A., Cuttler, C., & Leighton, D.C. (2019). Research methods in psychology (4th ed.). Kwantlen Polytechnic University. https://kpu.pressbooks.pub/psychmethods4e/
Julie, H., Sanjaya, F., & Anggoro, A. Y. (2017). The students’ ability in the mathematical literacy for uncertainty problems on the PISA adaptation test. AIP Conference Proceedings, 1868(1):050026. https://doi.org/10.1063/1.4995153
Kablan, Z. & Uğur, S. S. (2019). The relationship between routine and non-routine problem solving and learning styles. Educational Studies, 1-16. https://doi.org/110.1080/03055698.2019.1701993
Kartikasari, I. A., & Usodo, B. (2022). The Effectiveness open-ended learning and creative problem solving models to teach creative thinking skills. Pegem Journal of Education and Instruction, 12(4), 29-38.. https://doi.org/ 10.47750/pegegog.12.04.04
Kinnear, G., Jones, I., Sangwin, C., Alarfaj, M., Davies, B., Fearn, S., ... & Wong, T. (2024). A collaboratively-derived research agenda for e-assessment in undergraduate mathematics. International Journal of Research in Undergraduate Mathematics Education, 10(1), 201-231. https://doi.org/10.1007/s40753-022-00189-6
Kolovou, A., van den Heuvel-Panhuizen, M., & Bakker, A. (2009). Non-routine problem solving tasks in primary school mathematics textbooks – a needle in a haystack. Mediterranean Journal for Research in Mathematics Education, 8 (2), 31-68. https://Kolovou-etal_2009_MJRME_Textbook-analysis%20.pdf
Kragt, D., & Day, D. V. (2020). Predicting leadership competency development and promotion among high-potential executives: The role of leader identity. Frontiers in psychology, 11, 1816. https://doi.org/10.3389/fpsyg.2020.01816
Kumari, S., Gautam, H., Nityadarshinin, N., Das, B. K. & Chaudhry. (2021). Online classes versus traditional classes? Comparison during COVID 19. Journal of Education and Health Promotion, 10, 1-4. https://doi.org/10.4103/jehp.jehp_317_21
Kundu, A. & Bej, T. (2021). Experiencing e-assessment during COVID-19: an analysis of Indian students’ perception. Higher Education Evaluation and Development, 15(2), 114-134. https://www.emerald.com/insight/2514-5789. htm
Kurniawan, H., Darmono, P. B., Mursalin, M., Shang, Y., Weinhandl, R., & Sharm, R. (2022). Describe mathematical creative thinking skills and problem-solving strategies by prospective teacher students on non-routine problems. International Journal of Trends in Mathematics Education Research, 5(2), 119-124. https://doi.org/10.33122/ijtmer.v5i2.140
Lee, H. Y., Chen, P. H., Wang, W. S., Huang, Y. M., & Wu, T. T. (2024). Empowering ChatGPT with guidance mechanism in blended learning: Effect of self-regulated learning, higher-order thinking skills, and knowledge construction. International Journal of Educational Technology in Higher Education, 21(1), 1-28. https://doi.org/10.1186/s41239-024-00447-4
Legarde, M. A. (2022). Students’ common errors in solving routine & non-routine problems: a mixed method analysis. International Journal of Multidisciplinary Research and Analysis, 5(2), 541-547. https://doi.org/10.47181/ijmra/v5-i2-42
Liebenberg, L., Jamal, A., & Ikeda, J. (2020). Extending youth voices in a participatory thematic analysis approach. International Journal of Qualitative Methods, 19, 1609406920934614. https://doi.org/10.1177/1609406920934614
Makmuri, Aziz, T. A. & Kharis, S. A. A. (2021). Characteristics of problems for developing higher-order thinking skills in mathematics. Journal of Physics: Conference Series, 1-9. https://doi.org/ 10.1088/1742-6596/1882/1/ 012074
Manopo & Lisarani, V. (2021). A Comparative analysis of non-routine problems in mathematics textbooks of Indonesia and Singapore. EDUMATIKA: Jurnal Riset Pendidikan Matematika, 4(1). https://doi.org/10.32939/ejrpm.v4i1.750
Marriott, P. (2009). Students’ evaluation of the use of online summative assessment on an undergraduate financial accounting module. British Journal of Educational Technology, 40(2), 237-254. https://doi.org/10.1111/j.1467-8535.2008.00924.x
Mogari, D., & Lupahla, N. (2013). Mapping a group of Northern Namibian grade 12 learners’ algebraic non-routine problem solving skills. African Journal of Research in Mathematics, Science and Technology Education, 17(1/2), 94-105. https://doi.org/10.1080/10288457.2013.826974
Monrat, N., Phaksunchai, M., & Chonchaiya, R. (2022). Developing students’ mathematical critical thinking skills using open-ended questions and activities based on student learning preferences. Education Research International, 2022, Article ID 330036. https://doi.org/10.1155/2022/3300363
Muhammad, N., Ozuem, W., Howell, K., & Ranfagni, S. (2023). A Step-by-step process of thematic analysis to develop a conceptual model in qualitative research. International Journal of Qualitative Methods, 22, 1–18. https://doi.org/10.1177/16094069231205789
Padzil, M.R., Karim, A.A., & Husnin, H. (2021). Employing DDR to design and develop a flipped classroom and project based learning module to applying design thinking in design and technology. International Journal of Advanced Computer Science and Applications, 12(9), 791-798. https://doi.org/10.14569/IJACSA.2021.0120988
Pamungkas, A.P., & Widodo, P. (2019). Implementing online course evaluation rubric on e-learning ecologies: innovative approaches to teaching and learning for the digital age. Proceedings of the International Conference on Online and Blended Learning 2019 (ICOBL 2019) volume 440, 138-141. https://doi.org/10.2991/assehr.k.200521.030
Pegg, J. (2010). Promoting the acquisition of higher order skills and understandings in primary and secondary mathematics, In Research Forum. https://research.acer.edu.au/cgi/viewcontent.cgi?article=1081&context=research_conference
Pham, A. T. (2022). University students' attitudes towards the application of quizizz in learning english as a foreign language. International Journal of Emerging Technologies in Learning, 17(19). http://doi.org/10.3991/ijet.v17i19.32235
Pitorini, D. E., Suciati, & Harlita (2024). Students’ critical thinking skills using an e-module based on problem-based learning combined with socratic dialogue. Journal of Learning Development, 11(1), 52-65. https://doi.org/10.56059/jl4d.v11i1.1014
Plomp, T. (2007). Educational design research: an introduction. an introduction to educational design research. Proceedings of the seminar conducted at the East China Normal University, Shanghai (PR China), November 23-26, 2007, 9-35. https://ris.utwente.nl/ws/portalfiles/portal/14472302/ Introduction_20to_20education_20design_20research.pdf
Rasila, A., Malinen, J., & Tiitu, H. (2015). On automatic assessment and conceptual understanding. Teaching Mathematics and its Applications, 34(3), 149−159. https://doi.org/10.1093/teamat/hrv013
Richey, R. C., & Klein, J. D. (2007). Design and Development Research: Methods, strategies, and issues. https://openlibrary.org/books/OL11281240M/Design_and_Development_Research
Rusmini, R., Suyono, & Agustini, R. (2021). Profile of Argumentation ability of undergraduate students in chemistry education based on non-routine problems. E3S Web of Conferences. https://doi.org/10.1051/e3sconf/ 202132806007
Sahrir, M. S., Alias, N. A., Ismail, Z., & Osman, N. (2012). Employing design and development research (DDR): approaches in the design and development of online arabic vocabulary learning games prototype. Turkish Online Journal of Educational Technology, 11(2), 108-119. Retrieved from https://files.eric.ed.gov/fulltext/EJ989018.pdf
Saro, J. M. et al. (2022). New normal education: strategies, methods, and trends of teaching-learning on students’ perspectives and effectiveness. Psychology and Education: A Multidisciplinary Journal, 5(1), 259-265. https://doi.org/10.5281/zenodo. 7242770
Shawan, M., Osman, S. & Abu, M. S. (2021). Difficulties in solving non-routine problems: preliminary analysis and results. ASM Science Journal, 16, 1-14. https://doi.org/10.32802/asmscj.2021.800
Sikurajapathi, I., Henderson, K., & Gwynllyw, R. (2020). Using e-Assessment to Address Mathematical Misconceptions in Engineering Students. International Journal of Information and Education Technology, 10(5), 356-361. https://doi.org/10.18178/ijiet.2020.10.5.1389
Su, C. Y. & Guo, Y. Q. (2021). Factors impacting university students’ online learning experiences during the COVID-19 epidemic. Journal of Computer Assisted Learning, 37, 1578-1590. https://doi.org/10.1111/jcal.12555
Thomas, D. R. (2006). A general inductive approach for analyzing qualitative evaluation data. American Journal of Evaluation, 27(2), 237–246. https://doi.org/10.1177/ 1098214005283748
Tonbuloglu, B. (2022). State of research on e-assessment in education: a bibliometric analysis. Turkish Online Journal of Distance Education-TOJDE, 24(4), 276-301. https://doi.org/10.17718/tojde.1184218
Truong, K. D. & Tran, B-C. N. (2021). Higher order thinking skills in teaching academic writing: suggestions for application. Theodore Maria School of Arts, Assumption University, 101-124. http://www.assumptionjournal.au.edu/index.php/newEnglishTeacher/article/view/ 5581/3329
Usta, N. (2020). Evaluation of preservice teacher’s skills in solving non-routine mathematical problems through various strategies. Asian Journal of Education and Training, 6(3), 362-383. https://doi.org/10.20448/ journal.522.2020.63.362.383
Van der Kleij, F. M., Feskens, R.C.W., & Eggen, T.J.J.M. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: a meta-analysis. Review of Educational Research, 85(4), 1-37. https://doi.org/10.3102/0034654314564881
Van Zanten, M. & van den Heuvel-Panhuizen, M. (2018). Opportunity to learn problem solving in Dutch primary school mathematics textbooks. ZDM. https://doi.org/10.1007/s11858-018-0973-x
Wang, J. (2011). Re-examining test item issues in the TIMSS Mathematics and Science Assessments. School Science and Mathematics, 111(7), 334-344. https://doi.org/10.1111/j.1949-8594.2011.00096.x
Wang, Y., Kai, S. & Baker, R. S. (2020). Early detection of wheel-spinning in assistments. International conference on artificial intelligence in education, 574-585. https://doi.org/10.1007/978-3-030-52237-7_46
Wijaya, T., Hidayat, W., & Zhou, Y. (2020). Development of Interactive Learning Video on Linear Program. Universal Journal of Educational Research, 8(12A), 7530 - 7538. https://doi.org/ 10.13189/ujer.2020.082537
Xu, W., &Zammit,K.(2020).Applying thematic analysis to education: A hybrid approach to interpreting data in practitioner research. Inter national Journal of Qualitative Methods, 19, 1609406920918810. https://doi.org/10.1177/1609406920918810
Yazgan, Y. (2015). Sixth graders and non-routine problems: Which strategies are decisive for success? Educational Research and Reviews, 10(13), 107-1816. https://doi.org/10.5897/ERR2015.2230
Yolida, B., Marpaung, R. R. T., & Lengkana, D. (2023). Pre-service teachers’: analysis and constraints in making hots-based multiple choice questions. Journal of Education for Sustainability and Diversity, 2(1), 59-79. https://doi.org/10.57142/jesd.v2i1.59
Yu, R. & Cai, X. (2022). Impact of immediacy of feedback on continuous intentions to use online learning from the student perspective. Frontiers in Psychology, 13(865680). https://doi.org/10.3389/fpsyg.2022.865680
Zakaria, Z. A., Abdullah, N. & Mahir, N. A. (2022). The perceptiosn of students on online assessment during covid-19 pandemic. International Journal of Languages and Education, 1(1), 65-73. https://doi.org/xxx.0000/jcicom.vol1no1.9
Zulkipli & Saidah, M. (2024). How do junior high school students solve problems in non routine mathematics problems. Journal of Education for Sustainability and Diversity, 3(1), 65-76. https://doi.org/10.57142/jesd.v3i1.616
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Marivic G. Molina, Erna V. Yabut
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
