Comprehensive Analysis of a Multiple-Choice Algebra Assessment
Abstract
Abstract
This study developed and validated a 70-item multiple-choice College and Advanced Algebra (Math 101) midterm test designed to diagnose least-mastered competencies. Development followed a three-phase process: (1) preparation, involving syllabus mapping and a Table of Specifications (TOS) to ensure balanced content coverage and guide item writing; (2) development, which included expert review of content and technical quality and a tryout administration; and (3) implementation, featuring uniform test administration procedures, scoring verification, item analysis, and reliability estimation. Expert evaluations indicated strong content validity (M = 3.71, Highly Relevant), while student feedback supported face validity (M = 3.83, Highly Acceptable). In the tryout subsample (n = 30) drawn from the main cohort, preliminary reliability was acceptable (KR-20/α = .715). Using the full dataset (n = 129), internal consistency improved to a good level (α = .853). Item analysis resulted in 48 retained items, 20 for revision, and 2 rejected, supporting continued refinement and item-bank development.
Keywords
Full Text:
PDFUsage metrics (updated daily, overnight Central European Time)
Article pageviews (filtered): 288
PDF full-text accesses (filtered): 97
References
Abate, T., & Mishore, E. (2024). Alignment analysis between teacher-made tests with the learning objectives in a selected school of central regional state of Ethiopia. Heliyon, 10, e31869.
https://doi.org/10.1016/j.heliyon.2024.e31869
Basantes-Andrade, A., López-Gutiérrez, J.C., Mora Grijalva, M., & Ricardo, Y. (2023). Validity and reliability of the questionnaire of academic knowledge of teachers of basic general education. F1000Research, 12, 642.
https://doi.org/10.12688/f1000research.134261.1
Bujang, M.A., Lim, C.H., Yahya, F.H., Bujang, M.S., Abu Bakar Sidik, T.M.I.T., Sheikh Abdul Hamid, N.A., & Kueh, Y.C. (2024). Sample size determination for conducting a pilot study to assess reliability of a questionnaire. Restorative Dentistry & Endodontics, 49, e3.
https://doi.org/10.5395/rde.2024.49.e3
Cañeda, M., Gamaya, A.F., & Baring, M. (2024). Ensuring validity and reliability in Algebra midterm assessment: A comprehensive approach to test development and analysis. Journal of Interdisciplinary Perspectives, 2, 362-372.
https://doi.org/10.69569/jip.2024.0497
Chappuis, J., & Stiggins, R.J. (2021). An introduction to student-involved assessment for learning (7th ed.). New York: Pearson.
Dalawi, I., Isa, M.R., Chen, X.W., Azhar, Z.I., & Aimran, N. (2023). Development of the Malay language of understanding, attitude, practice and health literacy questionnaire on Covid-19 (MUAPHQ C-19): Content validity & face validity analysis. BMC Public Health, 23, 1131.
https://doi.org/10.1186/s12889-023-16044-5
Downing, S.M. (2002). Construct-irrelevant variance and flawed test questions: Do multiple-choice item-writing principles make any difference? Academic Medicine, 77, S103-S104.
https://doi.org/10.1097/00001888-200210001-00032
El-Khouly, M.A., Taher, H.R., Abboushi Abboushi, M.Y., Elgayar, A.E., Lskaan, S.M., Abd El-Wahab, A.M.M., Farrag, N.S., & Ibrahim, N.F. (2025). Assessment of knowledge, attitudes, practices, and barriers to Evidence-Based Practice (EBP) among healthcare providers at Port Said: An exploratory sequential mixed method study. BMC Medical Education, 26, 125.
https://doi.org/10.1186/s12909-025-08398-8
Farmer, C., Thurm, A., Das, T., Bebin, E.M., Bernstein, J.A., Berry-Kravis, E., & Sahin, M. (2025). Which score for what? Operationalizing standardized cognitive test performance for the assessment of change. American Journal on Intellectual and Developmental Disabilities, 130, 344-361.
https://doi.org/10.1352/1944-7558-130.5.344
Ferrando, P.J., Lorenzo-Seva, U., & Bargalló-Escrivà, M.T. (2023). Gulliksen’s pool: A quick tool for preliminary detection of problematic items in item factor analysis. PLOS ONE, 18, e0290611.
https://doi.org/10.1371/journal.pone.0290611
Folger, T.D., Bostic, J.D., & Krupa, E.E. (2023). Defining test-score interpretation, use, and claims: Delphi study for the validity argument. Educational Measurement: Issues and Practice, 42, 22-38.
https://doi.org/10.1111/emip.12569
Gilbert, G.E., & Prion, S. (2016). Making sense of methods and measurement: Lawshe’s content validity index. Clinical Simulation in Nursing, 12, 530-531.
https://doi.org/10.1016/j.ecns.2016.08.002
Haladyna, T.M., Downing, S.M., & Rodriguez, M.C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15, 309-334.
https://doi.org/10.1207/S15324818AME1503_5
Hillsley, K. (2025). Question format is the best predictor of item discrimination: A multivariable analysis. Journal of Microbiology & Biology Education, 26, e00205-25.
https://doi.org/10.1128/jmbe.00205-25
Hoekstra, R., Vugteveen, J., Warrens, M.J., & Kruyen, P.M. (2019). An empirical analysis of alleged misunderstandings of coefficient alpha. International Journal of Social Research Methodology, 22, 351-364.
https://doi.org/10.1080/13645579.2018.1547523
Kalkbrenner, M.T. (2021). A practical guide to instrument development and score validation in the social sciences: The measure approach. Practical Assessment, Research, and Evaluation, 26, 1.
https://doi.org/10.7275/svg4-e671
Khan, H.F., Qayyum, S., Beenish, H., Khan, R.A., Iltaf, S., & Faysal, L.R. (2025). Determining the alignment of assessment items with curriculum goals through document analysis by addressing identified item flaws. BMC Medical Education, 25, 200.
https://doi.org/10.1186/s12909-025-06736-4
Kılıç, A.F., & Uysal, İ. (2022). To what extent are item discrimination values realistic? A new index for two-dimensional structures. International Journal of Assessment Tools in Education, 9, 728-740.
https://doi.org/10.21449/ijate.1098757
Kim, Y.H. (2023). Item difficulty index, discrimination index, and reliability of multiple-choice questions in the national comprehensive basic science examination. Journal of Educational Evaluation for Health Professions, 20, 31.
https://doi.org/10.3352/jeehp.2023.20.31
Longe, I.O., & Maharaj, A. (2023). Investigating students’ understanding of complex number and its relation to algebraic group using and APOS theory. Journal of Medives: Journal of Mathematics Education IKIP Veteran Semarang, 7, 117-134.
https://doi.org/10.31331/medivesveteran.v7i1.2332
Lu, Y., & Sireci, S.G. (2007). Validity issues in test speededness. Educational Measurement: Issues and Practice, 26, 29-37.
https://doi.org/10.1111/j.1745-3992.2007.00106.x
Mamolo, L.A. (2021). Development of an achievement test to measure students’ competency in general mathematics. Anatolian Journal of Education, 6, 79-90.
https://doi.org/10.29333/aje.2021.616a
Masuwai, A., Zulkifli, H., & Hamzah, M.I. (2024). Evaluation of content validity and face validity of a teacher self-assessment instrument using expert review and indices. Cogent Education, 11, 2308410.
https://doi.org/10.1080/2331186X.2024.2308410
McCrossan, P., Nicholson, A., & McCallion, N. (2022). Minimum accepted competency examination: Test item analysis. BMC Medical Education, 22, 400.
https://doi.org/10.1186/s12909-022-03475-8
Mustafa, S., & Hamid, O.E. (2026). Psychometric item/question analysis of multiple-choice questions in fixed prosthodontics exam. BMC Medical Education, 26, 86.
https://doi.org/10.1186/s12909-025-08429-4
Padua, R.N., & Santos, R.G. (1997). Educational evaluation and measurement: Theory, practice, and application. Quezon City, Philippines: Katha.
Peeters, M.J., & Harpe, S.E. (2020). Updating conceptions of validity and reliability. Research in Social and Administrative Pharmacy, 16, 1127-1130.
https://doi.org/10.1016/j.sapharm.2019.11.017
Qhibi, A.D., de Jager, T., & White, C.J. (2025). Horizontal alignment between Grade 9 mathematics content standards and summative assessment tasks. Eurasia Journal of Mathematics, Science and Technology Education, 21, em2597.
https://doi.org/10.29333/ejmste/16043
Rashwan, N.I., Aref, S.R., Nayel, O.A., & Rizk, M.H. (2024). Postexamination item analysis of undergraduate pediatric multiple-choice questions exam: Implications for developing a validated question bank. BMC Medical Education, 24, 168.
https://doi.org/10.1186/s12909-024-05153-3
Raymond, J., Dai, D.W., & McAllister, S. (2025). The interpretation-use argument: The essential ingredient for high quality assessment design and validation. Advances in Health Sciences Education, 30, 1313-1332.
https://doi.org/10.1007/s10459-024-10392-6
Reeves, T.D., & Marbach-Ad, G. (2016). Contemporary test validity in theory and practice: A primer for discipline-based education researchers. CBE –Life Sciences Education, 15, rm1.
https://doi.org/10.1187/cbe.15-08-0183
Rezigalla, A.A., Eleragi, A.M.E.S.A., Elhussein, A.B., Alfaifi, J., ALGhamdi, M.A., Al Ameer, A.Y., Yahia, A.I.O., Mohammed, O.A., & Adam, M.I.E. (2024). Item analysis: The impact of distractor efficiency on the difficulty index and discrimination power of multiple-choice items. BMC Medical Education, 24, 445.
https://doi.org/10.1186/s12909-024-05433-y
Rudolph, M.J., Daugherty, K.K., Ray, M.E., Shuford, V.P., Lebovitz, L., & DiVall, M.V. (2019). Best practices related to examination item construction and post-hoc review. American Journal of Pharmaceutical Education, 83, 7204.
https://doi.org/10.5688/ajpe7204
Sadeghi, P., Pourabbas, A., Dehghani, G., & Katebi, K. (2025). Quantitative and qualitative item analysis of exams of basic medical sciences departments of Tabriz University of Medical Sciences in 2023. BMC Medical Education, 25, 937.
https://doi.org/10.1186/s12909-025-07539-3
Sireci, S., & Benítez, I. (2023). Evidence for test validation: A guide for practitioners. Psicothema, 35, 217-226.
https://doi.org/10.7334/psicothema2022.477
Taber, K.S. (2018). The use of Cronbach’s alpha when developing and reporting research instruments in science education. Research in Science Education, 48, 1273-1296.
https://doi.org/10.1007/s11165-016-9602-2
Taherdoost, H. (2016). Validity and reliability of the research instrument: How to test the validation of a questionnaire/survey in a research. International Journal of Academic Research in Management, 5, 28-36.
https://doi.org/10.2139/ssrn.3205040
Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.
Xiao, L., & Hau, K.-T. (2023). Performance of coefficient alpha and its alternatives: Effects of different types of non-normality. Educational and Psychological Measurement, 83, 5-27.
https://doi.org/10.1177/00131644221088240
Zubairi, N.A., AlAhmadi, T.S., Ibrahim, M.H., Hegazi, M.A., & Gadi, F.U. (2025). Effective use of item analysis to improve the reliability and validity of undergraduate medical examinations: Evaluating the same exam over many years. A different approach. Pakistan Journal of Medical Sciences, 41, 810-815.
https://doi.org/10.12669/pjms.41.3.10693
DOI: https://doi.org/10.7358/ecps-2026-033-cama
Copyright (©) 2026 Matthew Estudillo Cañeda – Editorial format and Graphical layout: copyright (©) LED Edizioni Universitarie

This work is published on this website under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Journal of Educational, Cultural and Psychological Studies (ECPS)
Registered by Tribunale di Milano (19/05/2010 n. 278)
Online ISSN 2037-7924 - Print ISSN 2037-7932
Research Laboratory on Didactics and Evaluation - Department of Education - "Roma Tre" University
Editor-in-Chief: Gaetano Domenici
Editorial Board: Eleftheria Argyropoulou - Massimo Baldacci - Joao Barroso - Richard Bates - Christofer Bezzina - Pietro Boscolo - Sara Bubb - Giuseppina Castellana - Jean-Émile Charlier - Lucia Chiappetta Cajola - Jean-Louis Derouet - Peter Early - Giancarlo Fortino - Constance Katz - James Levin - Pietro Lucisano - Roberto Maragliano - Massimo Marcuccio - Arianna Lodovica Morini - Romuald Normand - Michael Osborne - Donatella Palomba - Nazarena Patrizi - Michele Pellerey - Vitaly V. Rubtzov - Jaap Scheerens - Noah W. Sobe - Francesco Susi - Giuseppe Spadafora - Pat Thomson - Roberto Trinchero - Ira Vannini
Editorial Staff: Fabio Alivernini - Guido Benvenuto - Anna Maria Ciraci - Massimiliano Fiorucci - Luca Mallia - Massimo Margottini - Giovanni Moretti - Carla Roverselli
Editorial Secretary: Conny De Vincenzo
© 2001-2026 LED Edizioni Universitarie di Lettere Economia Diritto






