Email this article The Street Lamp Paradox: Analysing Students’ Evaluation of Teaching through Qualitative and Quantitative Approaches
Abstract
This paper presents a study about Students Evaluation of Teaching (SET). Student surveys delivered at the end of the academic course is an approach currently applied in all Italian university courses. However, the quality of teaching is a multi-layered phenomenon whose evaluation might require additional methods. In this study, concerning a course offered during the academic year 2016/2017, we propose the qualitative analysis of students’ final essays, which includes the students’ opinion about teaching quality. The qualitative evaluation of teaching is then compared with the results of the standard SET survey. 48 students filled in the survey, while 47 delivered their final essay (a corpus of about 650 text pages). Our study shows that while the standard survey provides an overall picture useful at institutional level for Quality Assurance (QA), the qualitative approach captures an accurate account of students’ reactions, sensitive to the pedagogical approach adopted. This provides additional information on the students’ perspectives regarding the specific features of the course. Our case study suggests that the integration of traditional SET survey with qualitative teaching evaluation approaches, at least for innovative courses based on socio-constructivist learning, might provide information, overlooked in the Italian SET survey, that is useful for Quality Enhancement (QE) of teaching.
Keywords
Full Text:
PDFReferences
Allendoerfer, C., Wilson, D., Plett, M., Bates, R. A., Smith, T. F., & Veilleux, N. M. (2016). Student perceptions of faculty support: Do class size or institution type matter? In ASEE Annual Conference and Exposition, Conference Proceedings (Vol. 2016-June).
Balzaretti, N., & Vannini, I. (2018). Promuovere la qualità della didattica universitaria* La Formative Educational Evaluation in uno studio pilota dell’Ateneo bolognese. ECPS Journal-Journal of Educational, Cultural and Psychological Studie, 2018(18), 187–214. https://doi.org/http://dx.doi.org/10.7358/ecps-2018-018-balz
Bernardi, L., Boncori, L., Gola, M., Magni, C., & Ottaviani, M. G. (1998). Valutazione della didattica da parte degli studenti. Rapporto finale del gruppo di ricerca. Roma.
Bertaccini, B. (2015). Il sistema italiano di valutazione della didattica universitaria: analisi critica della normativa vigente. (The Italian approach in evaluating the quality of the academic teaching system: a critical review of the current regulation). RIV Rassegna Italiana Di Valutazione, 63, 7–22. https://doi.org/10.3280/RIV2015-063002
Bertaccini, B., Giusti, A., & Petrucci, A. (2018). Students’ opinions on teaching and services provided by the Italian Universities: a proposal for a new evaluation scheme. Quality & Quantity. https://doi.org/10.1007/s11135-018-0801-y
Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students’ evaluations of professors. Education Review, 2014(41), 71–88.
Buck, D. (1998). Student Evaluations of Teaching Measure the Intervention, Not the Effect. American Psychologist, 1224–1226.
Chiandotto, B., & Gola, M. (2002). Proposta di un insieme minimo di domande per la valutazione della didattica da parte degli studenti frequentanti. Roma. Retrieved from http://www2.unibas.it/pqa/images/DOCUMENTI/2002.07.00 - CNVSU Proposta Questionario (Doc 09 2002).pdf
Chiandotto, B., Grilli, L., & Rampichini, C. (2005). Valutazione dei processi formativi di terzo livello: contributi metodologici. Firenze.
Creswell, J. W., & Plano Clark, V. L. (2011). Designing and Conducting Mixed Methods Research. London: SAGE Publications.
Crotty, M. (1998). The foundations of social research : meaning and perspective in the research process. Sage Publications.
D’Esposito, M. R. (2002). Valutazione della didattica e dei servizi nel sistema universita. In Atti della Giornata di Studio, 31 maggio 2002. Fisciano.
Ehlers, U.-D. (2004). Quality in e-learning from a learner’s perspective. EuropeaJournal for Distance and Open Learning, 1, 73–90.
Elassy, N. (2015). The concepts of quality, quality assurance and quality enhancement. Quality Assurance in Education, 23(3), 250–261. https://doi.org/10.1108/QAE-11-2012-0046
Fabbris, L. (2015). La misura della student satisfaction per la valutazione della qualità della didattica.
Falchikov, N. (2005). Improving Assessment through Student Involvement. London: Routledge–Falmer.
Felisatti, E., Giovanna Del Gobbo, U. di F., Pietro, M. Di, Lombardo, B. M., Perroteau, I., Zabalza, M., & Capogna, S. (2018). Linee di indirizzo per lo sviluppo professionale del docente e strategie di valutazione della didattica in Università. Roma.
Author (2012).
Author (2005).
Author et al. (2019).
Author et al. (2012).
Author et al. (2013).
Gibbs, G. (2010). Dimension of quality in higher education. York, UK. https://doi.org/ISBN 978-1-907207-24-2
Grion, V., & Tino, C. (2018). Verso una “valutazione sostenibile” all’università: percezioni di efficacia dei processi di dare e ricevere feedback fra pari. Lifelong Lifewide Learning, 14(31), 38–55.
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, California: sage.
Harvey, L. (2004). Analytic Quality Glossary, Quality Research International.
Harvey, L. (2007). The epistemology of quality. Perspectives in Education, 25(3), 1–13.
Harvey, L., & Green, D. (1993). Defining Quality. Assessment & Evaluation in Higher Education, 18(1), 9–34. https://doi.org/10.1080/0260293930180102
Jonassen, D. H. (1992). Evaluating constructivistic learning. In M. T. Duffy & D. H. Jonassen (Eds.), Constructivism and the technology of instruction (pp. 137–148). Hillsdale, New Jersey: Lawrence Erlbaum Associates Publishers.
Kaplan, A. (1973). The conduct of inquiry. Transaction Publishers.
Landis, J. R., & Koch, G. G. (1977). The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159. https://doi.org/10.2307/2529310
Lattuca, L. R., & Domagal‐Goldman, J. M. (2007). Using qualitative methods to assess teaching effectiveness. New Directions for Institutional Researches, 2007(136 Special Issue: Using Qualitative Methods in Institutional Assessment), 81–93. https://doi.org/10.1002/ir.233
Laurillard, D. (1993). Rethinking University Teaching: A Framework for the Effective Use of Educational Technology. Routledge.
Laurillard, D. (2012). Teaching as a Design Science: Building Pedagogical Patterns for Learning and Technology. London: Routledge.
Lincoln, Y., Lynham, S., & Guba, N. (2011). Paradigmatic controversies, contradictions, and emerging confluences, revisited. In N. K. Denzin & Y. S. Lincoln (Eds.), The Sage Handbook of Qualitative Research (4th Editio, pp. 97–128). Thousand Oaks, CA: SAGE Publications.
Macdonald, R. (2006). The use of evaluation to improve practice in learning and teaching. Innovations in Education and Teaching International, 43(1), 3–13. https://doi.org/10.1080/14703290500472087
Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: a peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122. https://doi.org/10.1080/02602938.2013.795518
Ory, J. C. (2001). Faculty thoughts and concerns about student ratings. New Directions for Teaching and Learning, 2001(87), 3–15. https://doi.org/10.1002/tl.23
Pagani, L., & Seghieri, C. (2002). A statistical analysis of teaching effectiveness from students’ point of view. In A. Mrvar & A. Ferligoj (Eds.), Developments in Statistic (pp. 197–208). Ljubljana: FDV. Retrieved from https://www.researchgate.net/publication/228979953_A_statistical_analysis_of_teaching_effectiveness_from_students’_point_of_view
Patton, M. Q. (1990). Qualitative evaluation and research methods (3rd ed.). Thousand Oaks, CA, USA: Sage Publications.
Raban, C. (2007). Assurance versus enhancement: Less is more? Journal of Further and Higher Education, 31(1), 77–85. https://doi.org/10.1080/03098770601167948
Rampichini, C., Grilli, L., & Petrucci, A. (2004). Analysis of university course evaluations: from descriptive measures to multilevel models. Statistical Methods & Applications, 2004(13), 357–373. https://doi.org/10.1007/s10260-004-0087-1
Richards, L., Morse, J. M., Gatti, F., & Graffigna, G. (2009). Fare ricerca qualitativa : prima guida. Angeli. Retrieved from https://www.francoangeli.it/ricerca/Scheda_Libro.aspx?ID=16858
Rienties, B. (2014). Understanding academics’ resistance towards (online) students evaluation. Assessment and Evaluation in Higher Education, 39(8), 987–1001. https://doi.org/10.1080/02602938.2014.880777
Author et al. (2017).
Shaw, I. (1999). Qualitative evaluation. London: Sage Publications. https://doi.org/10.4135/9781849209618
Spencer, K. J., & Schmelkin, L. P. (2002). Student Perspectives on Teaching and its Evaluation. Assessment & Evaluation in Higher Education, 27(5).
Spooren, P., Brockx, B., & Mortelmans, D. (2013). On the Validity of Student Evaluation of Teaching: The State of the Art. Review of Educational Research, 83, 598–642. https://doi.org/10.3102/0034654313496870
Spooren, P., & Christiaens, W. (2017). I liked your course because I believe in (the power of) student evaluations of teaching (SET). Students’ perceptions of a teaching evaluation process and their relationships with SET scores. Studies in Educational Evaluation, 2017(54), 43–49.
Topping, K. (1998). Peer Assessment Between Students in Colleges and Universities. Review of Educational Research, 68(3), 249–276. https://doi.org/10.3102/00346543068003249
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty’s teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, (54), 22–42. https://doi.org/http://dx.doi.org/10.1016/j.stueduc.2016.08.007
Vaismoradi, M., Turunen, H., & Bondas, T. (2013, September). Content analysis and thematic analysis: Implications for conducting a qualitative descriptive study. Nursing and Health Sciences. https://doi.org/10.1111/nhs.12048
Williams, J. (2016). Quality assurance and quality enhancement: is there a relationship? Quality in Higher Education, 22(2), 97–102. https://doi.org/10.1080/13538322.2016.1227207
Yang, N. (2015). Quality Teaching in Large University Classes: Designing Online Collaboration among Learners for Deep Understanding. University of Trento.
DOI: https://doi.org/10.7358/ecps-2020-021-ghis
Copyright (©) 2020 Patrizia Maria Margherita Ghislandi, Juliana Elisa Raffaghelli, Albert Sangrà, Giuseppe Ritella – Editorial format and Graphical layout: copyright (©) LED Edizioni Universitarie
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Journal of Educational, Cultural and Psychological Studies (ECPS)
Registered by Tribunale di Milano (19/05/2010 n. 278)
Online ISSN 2037-7924 - Print ISSN 2037-7932
Research Laboratory on Didactics and Evaluation - Department of Education - "Roma Tre" University
Executive Editor: Gaetano Domenici - Associate Executive Editor & Managing Editor: Valeria Biasci
Editorial Board: Eleftheria Argyropoulou - Massimo Baldacci - Joao Barroso - Richard Bates - Christofer Bezzina - Paolo Bonaiuto - Lucia Boncori - Pietro Boscolo - Sara Bubb - Carlo Felice Casula - Jean-Émile Charlier - Lucia Chiappetta Cajola - Carmela Covato - Jean-Louis Derouet - Peter Early - Franco Frabboni - Constance Katz - James Levin - Pietro Lucisano - Roberto Maragliano - Romuald Normand - Michael Osborne - Donatella Palomba - Michele Pellerey - Clotilde Pontecorvo - Vitaly V. Rubtzov - Jaap Scheerens - Noah W. Sobe - Francesco Susi - Giuseppe Spadafora - Pat Thomson
Editorial Staff: Fabio Alivernini - Guido Benvenuto - Anna Maria Ciraci - Massimiliano Fiorucci - Luca Mallia - Massimo Margottini - Giovanni Moretti - Carla Roverselli
Editorial Secretary:Nazarena Patrizi
© 2001 LED Edizioni Universitarie di Lettere Economia Diritto