Reliability of ChatGPT-5.0 as an Automated Essay Scoring Tool: What Matters?

Thanh Trinh

doi:10.33830/ijelp.v3i2.13551

Thanh Trinh
HCMC University of Technology and Education

Keywords: Automated essay scoring, Reliability measurements, Prompting engineering, Data feeding

Abstract

The aims of the present study were twofold: Exploring the variabilities of ChatGPT-5.0’s capabilities of rubric-based essay scoring across three prompting designs and two essay feeding methods; and testing the reliability of ChatGPT-generated scores against human ratings. Drawing upon three reliability measurements, including: Spearman’s correlations, Intraclass Correlations (Koraishi, 2024; Bui & Barrot, 2025) and quadratic weighted kappa (QWK) ( Mizumoto & Eguchi, 2023; Poole & Coss, 2024), the findings revealed that although the reliability coefficients ranged from moderate to substantial, the essay scoring abilities of ChatGPT-5.0 depended greatly upon users’ expertise to engineer prompts and their choices of essay feeding. This study highlights the importance of continued effort in the validation of this technology as an automated essay scoring tool and emphasizes the irreplaceability of human professional judgment in this field.

Published Nov 10, 2025

DOI:

https://doi.org/10.33830/ijelp.v3i2.13551

How to Cite

Trinh, T. (2025). Reliability of ChatGPT-5.0 as an Automated Essay Scoring Tool: What Matters?. International Journal of English Language and Pedagogy, 3(2), 44–59. https://doi.org/10.33830/ijelp.v3i2.13551

Download Citation

Issue

Vol. 3 No. 2 (2025): International Journal of English Language and Pedagogy (IJELP)

Section

Articles

Similar Articles

Aldi Hardian Maulana, Astuti Azis, Improving Students Speaking Ability Using Story Telling Methods in Class XI IPA SMAN 5 Kundur , International Journal of English Language and Pedagogy: Vol. 2 No. 2 (2024): International Journal of English Language and Pedagogy (IJELP)
Satrio Binusa Suryadi, Faiz Ushbah Mubarok, Atiqah Nurul Asri, Farida Ulfa, Usman Nurhasan, Anugrah Nur Rahmanto, Risa Juliadilla, Identifying EFL Students Speaking Performance: Foreign Language Anxiety Among Non-English Department Students , International Journal of English Language and Pedagogy: Vol. 2 No. 2 (2024): International Journal of English Language and Pedagogy (IJELP)
Suriani Djafar, Syukri, Muhammad Yunus, The DEVELOPING THE SPEAKING SKILLS OF THE STUDENTS THROUGH DRAMA TECHNIQUE , International Journal of English Language and Pedagogy: Vol. 1 No. 1 (2023): International Journal of English Language and Pedagogy (IJELP)
Lukytta Gusti Acfira, Hisyam Ichsan, Students' Perceptions and Learning Needs: A Fundamental Aspect in Designing English Flashcards for Accountants , International Journal of English Language and Pedagogy: Vol. 2 No. 2 (2024): International Journal of English Language and Pedagogy (IJELP)
Gia Arya Azzahra, Enhancing Junior High School Students' Reading Comprehension: The Effectiveness of Directed Reading Thinking Activity Technique , International Journal of English Language and Pedagogy: Vol. 2 No. 1 (2024): International Journal of English Language and Pedagogy (IJELP)
Lailatul Masruroh, Using KWL (Know, Want, and Learned) Strategy to Improve Students’ Reading Comprehension , International Journal of English Language and Pedagogy: Vol. 2 No. 1 (2024): International Journal of English Language and Pedagogy (IJELP)
Parussaya Kiatkheeree, Writing errors of EFL undergraduate students in the online writing course , International Journal of English Language and Pedagogy: Vol. 2 No. 1 (2024): International Journal of English Language and Pedagogy (IJELP)
Agustina Syafriani, Research Test on Formative Assessment Product , International Journal of English Language and Pedagogy: Vol. 1 No. 1 (2023): International Journal of English Language and Pedagogy (IJELP)
Dianna Suzieanna Mohamad Shah, Mohamad Safwat Ashahri Mohd Salim, Mohd Nur Fitri Mohd Salim, Siti Hadianti, Nurul Isra Fauziah, Rahma Dewi Hartati, Explainer Videos for Language Learning: A Comparative Study between UiTM and UT , International Journal of English Language and Pedagogy: Vol. 3 No. 2 (2025): International Journal of English Language and Pedagogy (IJELP)
Alifia Chaerunnisa, An Analysis on Language Learning Strategies Used by Successful Language Learners in Speaking , International Journal of English Language and Pedagogy: Vol. 1 No. 2 (2023): International Journal of English Language and Pedagogy

<< < 1 2 3 4 5 > >>

You may also start an advanced similarity search for this article.

Keywords: Automated essay scoring, Reliability measurements, Prompting engineering, Data feeding

Abstract

Most read articles by the same author(s)

Similar Articles