Transformative Practices: Integrating Automated Writing Evaluation in Higher Education Writing Classrooms - A Systematic Review

Indri Astutik(1), Utami Widiati(2), Devinta Puspita Ratri(3), Peggy Magdalena Jonathans(4), Nurkamilah Nurkamilah(5*), Yeni Mardiyana Devanti(6), Zaldi Harfal(7)

(1) Faculty of Letters, Universitas Negeri Malang; Faculty of Teacher Training and Education, Universitas Muhammadiyah Jember
(2) Faculty of Letters, Universitas Negeri Malang
(3) Faculty of Letters, Universitas Negeri Malang; Faculty of Cultural Studies, Universitas Brawijaya
(4) Faculty of Letters, Universitas Negeri Malang; Faculty of Teacher Training and Education, Universitas Kristen Artha Wacana
(5) Faculty of Teacher Training and Education, Universitas Muhammadiyah Jember, Indonesia; College of Education, the Pennsylvania State University, USA
(6) Faculty of Teacher Training and Education, Universitas Muhammadiyah Jember
(7) Warner School of Education, University of Rochester
(*) Corresponding Author

Abstract

This systematic literature review explores the utilization of Automated Writing Evaluation (AWE) as a writing scoring tool over a five-year period from 2016 to 2020, focusing on its role in the transformation and integration of learning tools for pedagogical purposes. Transformation refers to the significant changes and advancements in teaching methods, particularly in adapting to new educational technologies and approaches, while integration involves the seamless incorporation of AWE systems into these evolving instructional practices to enhance the effectiveness of writing instruction. The study aims to analyze the various types of AWE employed in academic research, track trends in AWE technology strategies, and investigate students’ perceptions of AWE in both scoring and instructional contexts. Additionally, it aims to uncover the benefits and limitations associated with AWE implementation in writing instruction. Examining 19 journal articles, this review identifies fourteen types of AWE utilized by researchers and tracks advancements in machine learning within the field. The findings reveal positive student perceptions of AWE, citing its usefulness, efficiency, and linguistic accuracy in scoring and instruction. Benefits of AWE implementation include improved linguistic accuracy, enhanced writing performance, increased student engagement, and the provision of reliable and valid feedback. Moreover, AWE demonstrates effectiveness in scoring and feedback provision, with potential short- and long-term effects on student learning. However, limitations of AWE are also noted, including student distrust of feedback and a preference for human raters over AWE-generated scores. This review provides valuable insights into the multifaceted role of AWE in writing instruction, highlighting its potential benefits and areas for improvement.

Keywords

automated writing evaluation; educational technology; employs holistic; integration of learning; student engagement; transformation learning; transformative pratice

Full Text:

PDF

References

Abidin, N. L. F., Dwiningsih, K., Jehwae, P., & Sari, C. K. (2024). Leveraging technology to improve learning independence in chemistry: A study on Moodle integration. Indonesian Journal on Learning and Advanced Education (IJOLAE), 6(3), 365-386.

Bitchener, John. 2012. “A Reflection on ‘the Language Learning Potential’ of Written CF.” Journal of Second Language Writing 21(4):348–63. doi: 10.1016/j.jslw.2012.09.006.

Bridgeman, Brent, and Chaitanya Ramineni. 2017. “Design and Evaluation of Automated Writing Evaluation Models: Relationships with Writing in Naturalistic Settings.” Assessing Writing 34:62–71. doi: 10.1016/j.asw.2017.10.001.

Bridgeman, Brent, Catherine Trapani, and Yigal Attali. 2012. “Comparison of Human and Machine Scoring of Essays: Differences by Gender, Ethnicity, and Country.” Applied Measurement in Education 25(1):27–40. doi: 10.1080/08957347.2012.635502.

Chapelle, C. .. 1999. “Research Questions for a CALL Research Agenda: A Reply to Rafael Salaberry.” Language Learning & Technology 37(1):108–13.

Chapelle, Carol A., and Erik Voss. 2016. “20 Years of Technology and Language Assessment in Language Learning & Technology.” Language Learning and Technology 20(2):116–28.

Chen, Chi Fen Emily, and Wei Yuan Eugene Cheng. 2008. “Beyond the Design of Automated Writing Evaluation: Pedagogical Practices and Perceived Learning Effectiveness in Efl Writing Classes.” Language Learning and Technology 12(2):94–112.

Cheng, Gary. 2017. “The Impact of Online Automated Feedback on Students’ Reflective Journal Writing in an EFL Course.” The Internet and Higher Education 34:18–27. doi: 10.1016/j.iheduc.2017.04.002.

Cotos, E. 2015. “AWE for Writing Pedagogy: From Healthy Tension to Tangible Prospects. Special Issue on Assessment for Writing and Pedagogy.” Writing & Pedagogy 7(2–3):197–231.

Dikli, Semire. 2006. “An Overview of Automated Scoring of Essays.” The Journal of Technology, Learning and Assessment 5(1 SE-Articles).

Dikli, Semire, and Susan Bleyle. 2014. “Automated Essay Scoring Feedback for Second Language Writers: How Does It Compare to Instructor Feedback?” Assessing Writing 22:1–17. doi: 10.1016/j.asw.2014.03.006.

Godshalk, F. .., F. Swineford, and W. .. Coffman. 1966. The Measurement of Writing Ability. New York: College Entrance Examination Board.

Graham, Steve, Michael Hebert, and Karen R. Harris. 2015. “Formative Assessment and Writing: A Meta-Analysis.” The Elementary School Journal 115(4):523–47.

Hutchison, Dougal. 2007. “An Evaluation of Computerised Essay Marking for National Curriculum Assessment in the UK for 11‐year‐olds.” British Journal of Educational Technology 38(6):977–89. doi: 10.1111/j.1467-8535.2006.00686.x.

Koltovskaia, Svetlana. 2020. “Student Engagement with Automated Written Corrective Feedback (AWCF) Provided by Grammarly: A Multiple Case Study.” Assessing Writing 44:100450. doi: 10.1016/j.asw.2020.100450.

Laborda, Jesus Garcia. 2007. “Introducing Standardized EFL/ESL Exams.” Language Learning and Technology 11(2):3–9.

Lachner, Andreas, Christian Burkhart, and Matthias Nückles. 2017. “Formative Computer-Based Feedback in the University Classroom: Specific Concept Maps Scaffold Students’ Writing.” Computers in Human Behavior 72:459–69. doi: 10.1016/j.chb.2017.03.008.

Li, Zhi, Hui-Hsien Feng, and Aysel Saricaoglu. 2017. “The Short-Term and Long-Term Effects of AWE Feedback on ESL Students’ Development of Grammatical Accuracy.” CALICO Journal 34(3):355–75.

Liao, Hui-Chuan. 2016a. “Enhancing the Grammatical Accuracy of EFL Writing by Using an AWE-Assisted Process Approach.” System 62:77–92. doi: 10.1016/j.system.2016.02.007.

Liao, Hui-Chuan. 2016b. “Using Automated Writing Evaluation to Reduce Grammar Errors in Writing.” ELT Journal 70(3):308–19. doi: 10.1093/elt/ccv058.

Lim, Fei Victor, and Jean Phua. 2019. “Teaching Writing with Language Feedback Technology.” Computers and Composition 54:102518. doi: 10.1016/j.compcom.2019.102518.

Link, Stephanie, Mohaddeseh Mehrzad, and Mohammad Rahimi. 2022. “Impact of Automated Writing Evaluation on Teacher Feedback, Student Revision, and Writing Improvement.” Computer Assisted Language Learning 35(4):605–34. doi: 10.1080/09588221.2020.1743323.

Liu, Q., S. Zhang, Q. Wang, and W. Chen. 2018. “Mining Online Discussion Data for Understanding Teachers Reflective Thinking.” IEEE Transactions on Learning Technologies 11(2):243–54. doi: 10.1109/TLT.2017.2708115.

Liu, Sha, and Antony John Kunnan. 2016. “Investigating the Application of Automated Writing Evaluation to Chinese Undergraduate English Majors: A Case Study of WriteToLearn.” CALICO Journal 33(1):71–91.

Mohsen, Mohammed Ali, and Abdulaziz Alshahrani. 2019. “The Effectiveness of Using a Hybrid Mode of Automated Writing Evaluation System on Efl Students’ Writing.” Teaching English with Technology 19(1):118–31.

Onojah, A. A., Onojah, A. O., Olumorin, C. O., & Omosewo, E. O. (2021). Secondary School Teachers’ Accessibility to Internet Facilities for Advanced Instruction in Nigeria. Indonesian Journal on Learning and Advanced Education (IJOLAE), 3(2), 86-95.

Qian, Leyi, Yali Zhao, and Yan Cheng. 2020. “Evaluating China’s Automated Essay Scoring System IWrite.” Journal of Educational Computing Research 58(4):771–90. doi: 10.1177/0735633119881472.

Ranalli, Jim, Stephanie Link, and Evgeny Chukharev-Hudilainen. 2017. “Automated Writing Evaluation for Formative Assessment of Second Language Writing: Investigating the Accuracy and Usefulness of Feedback as Part of Argument-Based Validation.” Educational Psychology 37(1):8–25. doi: 10.1080/01443410.2015.1136407.

Roscoe, Rod D., Joshua Wilson, Adam C. Johnson, and Christopher R. Mayra. 2017. “Presentation, Expectations, and Experience: Sources of Student Perceptions of Automated Writing Evaluation.” Computers in Human Behavior 70:207–21. doi: 10.1016/j.chb.2016.12.076.

Salaberry, Rafael. 1999. “CALL in the Year 2000: Still Developing the Research Agenda.” Language Learning & Technology 3(1):104–7.

Saricaoglu, Aysel. 2019. “The Impact of Automated Feedback on L2 Learners’ Written Causal Explanations.” ReCALL 31(2):189–203. doi: 10.1017/S095834401800006X.

Shermis, Mark D. 2014. “State-of-the-Art Automated Essay Scoring: Competition, Results, and Future Directions from a United States Demonstration.” Assessing Writing 20:53–76. doi: 10.1016/j.asw.2013.04.001.

Shermis, Mark D., and Ben Hamner. 2013. “Contrasting State-of-the-Art Automated Scoring of Essays.” Pp. 313–46 in Handbook of automated essay evaluation. Routledge.

Silva, Pedro. 2017. “Scaffolding Assignments: Analysis of AssignMentor as a Tool to Support First Year Students’ Academic Writing Skills.” E-Learning and Digital Media 14(1–2):86–97. doi: 10.1177/2042753017695652.

Sulistyanto, H., Anif, S., Sutama, S., Narimo, S., Sutopo, A., Haq, M. I., & Nasir, G. A. (2022). Education application Testing Perspective to Empower Students' Higher Order Thinking Skills Related to the Concept of Adaptive Learning Media. Indonesian Journal on Learning and Advanced Education (IJOLAE), 4(3), 257-271.

Sulistyanto, H., Sumardjoko, B., Haq, M. I., Zakaria, G. A. N., Narimo, S., Astuti, D., Adhantoro, M. S., Setyabudi, D. P., Sidiq, Y., & Ishartono, N. (2023). Impact of Adaptive Educational Game Applications on Improving Student Learning: Efforts to Introduce Nusantara Culture in Indonesia. Indonesian Journal on Learning and Advanced Education (IJOLAE), 5(3), 249–261. https://doi.org/10.23917/ijolae.v5i3.23004

Stevenson, Marie. 2016. “A Critical Interpretative Synthesis: The Integration of Automated Writing Evaluation into Classroom Writing Instruction.” Computers and Composition 42:1–16. doi: https://doi.org/10.1016/j.compcom.2016.05.001.

Stevenson, Marie, and Aek Phakiti. 2014. “The Effects of Computer-Generated Feedback on the Quality of Writing.” Assessing Writing 19:51–65. doi: https://doi.org/10.1016/j.asw.2013.11.007.

Stevenson, Marie, and Aek Phakiti. 2019. “Automated Feedback and Second Language Writing.” Pp. 125–42 in Feedback in second language writing: Contexts and issues.

Tian, Lili, and Yu Zhou. 2020. “Learner Engagement with Automated Feedback, Peer Feedback and Teacher Feedback in an Online EFL Writing Context.” System 91:102247. doi: https://doi.org/10.1016/j.system.2020.102247.

Ullmann, Thomas Daniel. 2019. “Automated Analysis of Reflection in Writing: Validating Machine Learning Approaches.” International Journal of Artificial Intelligence in Education 29(2):217–57. doi: 10.1007/s40593-019-00174-2.

Wang, Elaine Lin, Lindsay Clare Matsumura, Richard Correnti, Diane Litman, Haoran Zhang, Emily Howe, Ahmed Magooda, and Rafael Quintana. 2020. “ERevis(Ing): Students’ Revision of Text Evidence Use in an Automated Writing Evaluation System.” Assessing Writing 44:100449. doi: https://doi.org/10.1016/j.asw.2020.100449.

Wang, Zhijie. 2022. “Computer-Assisted EFL Writing and Evaluations Based on Artificial Intelligence: A Case from a College Reading and Writing Course.” Library Hi Tech 40(1):80–97. doi: 10.1108/LHT-05-2020-0113.

Warschauer, Mark, and Paige Ware. 2006. “Automated Writing Evaluation: Defining the Classroom Research Agenda.” Language Teaching Research 10(2):157–80. doi: 10.1191/1362168806lr190oa.

Wilson, Joshua, and Amanda Czik. 2016. “Automated Essay Evaluation Software in English Language Arts Classrooms: Effects on Teacher Feedback, Student Motivation, and Writing Quality.” Computers & Education 100:94–109. doi: https://doi.org/10.1016/j.compedu.2016.05.004.

Wolfe, E. M. 2005. “Uncovering Rater’s Cognitive Processing and Focus Using Think-Aloud Protocols.” Journal of Writing Assessment 2(1):37–56.

Zhang, Zhe (Victor). 2020. “Engaging with Automated Writing Evaluation (AWE) Feedback on L2 Writing: Student Perceptions and Revisions.” Assessing Writing 43:100439. doi: https://doi.org/10.1016/j.asw.2019.100439.

Zhang, Zhe (Victor), and Ken Hyland. 2018. “Student Engagement with Teacher and Automated Feedback on L2 Writing.” Assessing Writing 36:90–102. doi: https://doi.org/10.1016/j.asw.2018.02.004.

DOI: 10.23917/ijolae.v6i3.23675

Article Metrics

Abstract view(s): 102 time(s)
PDF: 66 time(s)

Refbacks

  • There are currently no refbacks.