AWE for Formative Purposes in the EFL Context:
A Study of Multiple Revisions in Academic Abstract Writing
Abstract
The use of automated writing evaluation (AWE) tools for writing practices has become a central issue in English as a second and foreign language teaching contexts. Researchers appear to agree that students’ positive outcomes hinge upon their integration with formative purposes. Nonetheless, few studies have addressed multiple revisions and automated feedback provision in students’ academic writing development. In this context, an instructional treatment for the integration of this technology has been designed. Explicit instruction on academic writing, AWE workshops, and practice activities remain at its core. Hence, this paper examines the effects of self- and AWE-mediated writing revisions on undergraduate learners’ academic writing performance over time and at different levels of proficiency. A series of repeated-measures analysis of variance (ANOVA) were computed to study participants’ syntactic complexity, readability and language issues, and lexical diversity outcomes. Results revealed improvement in some of the dependent variables, that is, language issues reduction and increased type-token ratio mean scores, which could represent an initial step towards reconsidering the provision of automated written feedback.
Downloads
References
Alameddine, M. M., & Mirza, H. S. (2016). Teaching academic writing for advanced level grade 10 English. Procedia-Social and Behavioral Sciences, 232, 209–216. https://doi.org/10.1016/j.sbspro.2016.10.048
Barkaoui, K. (2016). What and when second-language learners revise when responding to timed writing tasks on the computer: The roles of task type, second language proficiency, and keyboarding skills. Modern Language Journal, 100(1), 320–340. https://doi.org/10.1111/modl.12316
Biber, D., Conrad, S., & Leech, G. (2002). Longman student grammar of spoken and written English. Harlow, Essex: Pearson Education.
Bitchener, J., & Storch, N. (2016). Written corrective feedback for L2 development (Vol. 96). Multilingual Matters.
Casal, J. E., & Lee, J. J. (2019). Syntactic complexity and writing quality in assessed first-year L2 writing. Journal of Second Language Writing, 44, 51–62. https://doi.org/10.1016/j.jslw.2019.03.005
Chambers, L. (2011). Composition and revision in computer-based written assessment. Research Notes, 43, 25–32.
Chapelle, C. A., & Voss, E. (2017). Utilizing Technology in Language Assessment. In: Shohamy, E., Or, I., May, S. (Eds.), Language Testing and Assessment. Encyclopedia of Language and Education. Springer, Cham. https://doi.org/10.1007/978-3-319-02261-1_10
Chapelle, C. A., Cotos, E., & Lee, J. (2015). Validity arguments for diagnostic assessment using automated writing evaluation. Language Testing, 32(3), 385–405. https://doi.org/10.1177/0265532214565386
Chen, C., & Cheng, W. (2008). Beyond the design of automated writing evaluation: pedagogical practices and perceived learning effectiveness in EFL writing classes. Language Learning & Technology, 12(2), 94–112. http://dx.doi.org/10125/44145
Chen, J., & Zhang, L. J. (2019). Assessing student-writers’ self-efficacy beliefs about text revision in EFL writing. Assessing Writing, 40(2), 27–41. https://doi.org/10.1016/j.asw.2019.03.002
Cohen, J. (1988). Statistical power analysis for the behavioural sciences (2nd ed.). New York: Academic Press.
Council of Europe (2020). Common European framework of reference for languages: Learning, teaching, assessment – Companion volume, Council of Europe Publishing, Strasbourg. Retrieved January 3, 2024, from http://www.coe.int/lang-cefr
Corazza, A., Maggio, V., & Scanniello, G. (2018). Coherence of comments and method implementations: A dataset and an empirical investigation. Software Quality Journal, 26(2), 751–777. https://doi.org/10.1007/s11219-016-9347-1
Cotos, E. (2014). Genre-based automated writing evaluation for L2 research writing: From design to evaluation and enhancement. Palgrave Macmillan.
Cotos, E. (2023). Automated Feedback on Writing. In O. Kruse, C. Rapp, C. Anson, K. Benetos, E. Cotos, A. Devitt, A. Shibani (Eds.), Digital Writing Technologies in Higher Education (pp. 347–364). Springer, Cham. https://doi.org/10.1007/978-3-031-36033-6_22
Crossley, S. A., & McNamara, D. S. (2014). Does writing development equal writing quality? A computational investigation of syntactic complexity in L2 learners. Journal of Second Language Writing, 26, 66–79. https://doi.org/10.1016/j.jslw.2014.09.006
Davidson, P. (2019). What is academic writing? In W. Zoghbor, S. Al Alami, & T. Alexiou, (Eds.), Proceedings of the 1st applied linguistics and language teaching and learning in a globalized world (ALLT) Conference (pp. 12–20). Zayed University Press.
Dikli, S. (2006). An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5(1), 1–35. Retrieved December 8, 2023 from https://ejournals.bc.edu/index.php/jtla/article/view/1640
Dizon, G., & Gayed, J. M. (2021). Examining the impact of Grammarly on the quality of mobile L2 writing. The JALT CALL Journal, (17)2, 74–92. https://doi.org/10.29140/jaltcall.v17n2.336
Eckstein, G., & Bell, L. (2023). Dynamic written corrective feedback in first-year composition: Accuracy and lexical and syntactic complexity. RELC Journal, 54(3), 630–647. https://doi.org/10.1177/00336882211061624
Eleyan, D., Othman, A., & Eleyan, A. (2020). Enhancing software comments readability using Flesch reading ease score. Information, 11(9), 430. https://doi.org/10.3390/info11090430
Flesch, R. (1979). How to write plain English: A book for lawyers and consumers. Harper and Row.
Goldin, I., Narciss, S., Foltz, P., & Bauer, M. (2017). New directions in formative feedback in interactive learning environments. International Journal of Artificial Intelligence in Education, 27(3), 385−392. https://doi.org/10.1007/s40593-016-0135-7
Gregori-Signes, C., & Clavel-Arroitia, B. (2015). Analysing lexical density and lexical diversity in university students’ written discourse. Procedia-Social and Behavioral Sciences, 198, 546–556. https://doi.org/10.1016/j.sbspro.2015.07.477
Guo, Q., Feng, R., & Hua, Y. (2021). How effectively can EFL students use automated written corrective feedback (AWCF) in research writing? Computer Assisted Language Learning, 1–20. https://doi.org/10.1080/09588221.2021.1879161
Hayes, J. R. (2012). Modeling and remodeling writing. Written Communication, 29(3), 369–388. https://doi.org/10.1177/0741088312451260
Herrington, A., & Moran, C. (2012). Writing to a machine is not writing at all. In N. Elliot & L. Perelman (Eds.), Writing in the 21st Century: Essays in honor of Edward M. White (pp. 219–232). New York: Hampton Press.
Hockly, N. (2019). Automated writing evaluation. ELT Journal, 73(1), 82−88. https://doi.org/10.1093/elt/ccy044
Huang, S., & Renandya, W. A. (2020). Exploring the integration of automated feedback among lower-proficiency EFL learners. Innovation in Language Learning and Teaching, 14(1), 15−26. https://doi.org/10.1080/17501229.2018.1471083
Jingxin, G., & Razali, A. B. (2020). Tapping the potential of Pigai automated writing evaluation (AWE) program to give feedback on EFL writing. Universal Journal of Educational Research, 8(12B), 8334–8343. https://doi.org/10.13189/ujer.2020.082638
Johansson, V. (2008). Lexical diversity and lexical density in speech and writing: A developmental perspective. Working papers/Lund University, Department of Linguistics and Phonetics, 53, 61–79.
Kaharuddin, A. (2020). Contributions of technology, culture, and attitude to English learning motivation during COVID-19 outbreaks. Systematic Reviews in Pharmacy, 11(11), 76–84.
Kobayashi, H., & Rinnert, C. (2001). Factors relating to EFL writers’ discourse level revision skills. International Journal of English Studies, 1(2), 71–101. Retrieved from https://revistas.um.es/ijes/article/view/48111
Koltovskaia, S. (2020). Student engagement with automated written corrective feedback (AWCF) provided by Grammarly: A multiple case study. Assessing Writing, 44, 100450. https://doi.org/10.1016/j.asw.2020.100450
Kyle, K. (2016). Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication [Doctoral Dissertation, Georgia State University]. ScholarWorks.
Lee, C., Cheung, W. K. W., Wong, K. C. K., & Lee, F. S. L. (2013). Immediate web-based essay critiquing system feedback and teacher follow-up feedback on young second language learners’ writings: An experimental study in a Hong Kong secondary school. Computer Assisted Language Learning, 26(1), 39–60. https://doi.org/10.1080/09588221.2011.630672
Lee, Y-J. (2020). The long-term effect of automated writing evaluation feedback on writing development. English Teaching, 75(1), 67–92. https://doi.org/10.15858/engtea.75.1.202003.67
Li, J., Link, S., & Hegelheimer, V. (2015). Rethinking the role of automated writing evaluation (AWE) feedback in ESL writing instruction. Journal of Second Language Writing, 27, 1–18. https://doi.org/10.1016/j.jslw.2014.10.004
Li, W., Lu, Z., & Liu, Q. (2020). Syntactic complexity development in college students’ essay writing based on AWE. CALL for widening participation: Short papers from EUROCALL, 190–194. https://doi.org/10.14705/rpnet.2020.48.1187
Li, Z. (2021). Teachers in automated writing evaluation (AWE) system-supported ESL writing classes: Perception, implementation, and influence. System, 99, 102505. https://doi.org/10.1016/j.system.2021.102505
Li, Z., Dursun, A., & Hegelheimer, V. (2017). Technology and L2 writing. In C. A. Chapelle & S. Sauro (Eds.), The handbook of technology and second language teaching and learning (pp. 77–92). John Wiley & Sons.
Liao, H. C. (2016). Enhancing the grammatical accuracy of EFL writing by using an AWE-assisted process approach. System, 62, 77−92. https://doi.org/10.1016/j.system.2016.02.007
Lillis, T., & Curry, M. J. (2006). Professional academic writing by multilingual scholars: Interactions with literacy brokers in the production of English-medium texts. Written Communication, 23(1), 3–35. https://doi.org/10.1177/0741088305283754
Link, S., Mehrzad, M., & Rahimi, M. (2020). Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement. Computer Assisted Language Learning, 33, 1–30. https://doi.org/10.1080/09588221.2020.1743323
Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4), 474–496. https://doi.org/10.1075/ijcl.15.4.02lu
McCarthy, K. S., Roscoe, R. D., Allen, L. K., Likens, A. D., & McNamara, D. S. (2022). Automated writing evaluation: Does spelling and grammar feedback support high-quality writing and revision? Assessing Writing, 52, 100608. https://doi.org/10.1016/j.asw.2022.100608
McNamara, D. S., Graesser, A. C., McCarthy, P., & Cai, Z. (2014). Automated Evaluation of Text and Discourse with Coh-Metrix. Cambridge: Cambridge University Press.
Muñoz-Luna R. (2015). Main ingredients for success in L2 academic writing: Outlining, drafting and proofreading. PLoS ONE, 10(6), 1–15. https://doi.org/10.1371/journal.pone.0128309
Nasseri, M., & Thompson, P. (2021). Lexical density and diversity in dissertation abstracts: Revisiting English L1 vs. L2 text differences. Assessing Writing, 47, 100511. https://doi.org/10.1016/j.asw.2020.100511
Norris, J. M., & Ortega, L. (2009). Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied linguistics, 30(4), 555–578. https://doi.org/10.1093/applin/amp044
Nova, M. (2018). Utilizing Grammarly in evaluating academic writing: A narrative research on EFL students’ experience. Premise: Journal of English Education and Applied Linguistics 7(1), 80–96. http://dx.doi.org/10.24127/pj.v7i1.1332
Ranalli, J. (2018). Automated written corrective feedback: How well can students make use of it? Computer Assisted Language Learning, 31(7), 653−674. https://doi.org/10.1080/09588221.2018.1428994
Ranalli, J., Link, S., & Chukharev-Hudilainen, E. (2017). Automated writing evaluation for formative assessment of second language writing: Investigating the accuracy and usefulness of feedback as part of argument-based validation. Educational Psychology, 37(1), 8–25. https://doi.org/10.1080/01443410.2015.1136407
Richardson, J. T. (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational research review, 6(2), 135–147. https://doi.org/10.1016/j.edurev.2010.12.001
Rodríguez-Peñarroja, M. (2022). Automated writing evaluation: Present challenges and future directions. In N. Martínez León, I. Mosquera Gende, & A. J. Moreno Guerrero (Eds.), Tic Docentes (pp. 357–370). Thomson Reuters Aranzadi.
Saricaoglu, A. (2019). The impact of automated feedback on L2 learners’ written causal explanations. ReCALL, 31(2), 189–203. https://doi.org/10.1017/S095834401800006X
Sengupta, S. (2000). An investigation into the effects of revision strategy instruction on L2 secondary school learners. System, 28(2), 97–113. https://doi.org/10.1016/S0346-251X(99)00063-9
Soleimani, H., & Moqimi, J. (2023). Teaching ESL/EFL Writing Skill Website Evaluation: The Purdue Online Writing Lab and Pro Writing Aid. Interdisciplinary Studies in English Language Teaching, 2(1), 127–145. https://doi.org/10.22080/iselt.2023.25121.1049
Spring, J., Davidson, P., & Palmer, D. (2010). Why we should not be teaching academic writing through the rhetorical pattern approach. In D. Anderson & C. Coombe (Eds.), Cultivating real writers: Emerging theory and practice for adult Arab learners (pp. 53–64). HCT Press.
Stevenson, M., & Phakiti, A. (2014). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51–65. https://doi.org/10.1016/j.asw.2013.11.007
Stevenson, M., Schoonen, R., & de Glopper, K. (2006). Revising in two languages: A multi- dimensional comparison of online writing revisions in L1 and FL. Journal of Second Language Writing, 15(3), 201–233. https://doi.org/10.1016/j.jslw.2006.06.002
Strobl, C., Ailhaud, E., Benetos, K., Devitt, A., Kruse, O., Proske, A., & Rapp, C. (2019). Digital support for academic writing: A review of technologies and pedagogies. Computers & Education, 131, 33–48. https://doi.org/10.1016/j.compedu.2018.12.005
Thi, N. K., & Nikolov, M. (2023). Effects of teacher, automated, and combined feedback on syntactic complexity in EFL students’ writing. Asian-Pacific Journal of Second and Foreign Language Education, 8(1), 6. https://doi.org/10.1186/s40862-022-00182-1
Truscott, J. (2007). The effect of error correction on learners’ ability to write accurately. Journal of second language writing, 16(4), 255–272. https://doi.org/10.1016/j.jslw.2007.06.003
University of Cambridge Local Examinations Syndicate (UCLES) (2001). Quick placement test. Oxford University Press.
Wallwork, A. (2011). English for writing research papers. Springer.
Wang, Y. J., Shang, H. F., & Briody, P. (2013). Exploring the impact of using automated writing evaluation in English as a foreign language university students’ writing. Computer Assisted Language Learning, 26(3), 234–257. https://doi.org/10.1080/09588221.2012.655300
Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157–180. https://doi.org/10.1191/1362168806lr190oa
Waycott, J., Bennett, S., Kennedy, G., Dalgarno, B., & Gray, K. (2010). Digital divides? Student and staff perceptions of information and communication technologies. Computers & Education, 54(4), 1202–1211. https://doi.org/10.1016/j.compedu.2009.11.006
Wilson, J., & Roscoe, R. D. (2020). Automated writing evaluation and feedback: Multiple metrics of efficacy. Journal of Educational Computing Research, 58(1), 87−125. https://doi.org/10.1177/0735633119830764
Xu, J., & Zhang, S. (2022). Understanding AWE feedback and English writing of learners with different proficiency levels in an EFL classroom: A sociocultural perspective. The Asia-Pacific Education Researcher, 31(4), 357–367. https://doi.org/10.1007/s40299-021-00577-7
Zhang, J., & Zhang, L. J. (2022). The effect of feedback on metacognitive strategy use in EFL writing. Computer Assisted Language Learning, 37(5–6), 1–26. https://doi.org/10.1080/09588221.2022.2069822
Zhang, Z. V. (2017). Student engagement with computer-generated feedback: A case study. ELT Journal, 71(3), 317–328. https://doi.org/10.1093/elt/ccw089
Zhang, Z. V. (2020). Engaging with automated writing evaluation (AWE) feedback on L2 writing: Student perceptions and revisions. Assessing Writing, 43, 100439. https://doi.org/10.1016/j.asw.2019.100439
Zhang, Z. V., & Hyland, K. (2018). Student engagement with teacher and automated feedback on L2 writing. Assessing Writing, 36, 90–102. https://doi.org/10.1016/j.asw.2018.02.004
Zheng, Y. & Yu, S. (2018). Student engagement with teacher written corrective feedback in EFL writing: A case study of Chinese lower-proficiency students. Assessing Writing, 37, 13−24. https://doi.org/10.1016/j.asw.2018.03.001
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
The works published in this journal are subject to the following terms:
1. The Publications Services at the University of Murcia (the publisher) retains the property rights (copyright) of published works, and encourages and enables the reuse of the same under the license specified in item 2.
2. The works are published in the electronic edition of the magazine under a Creative Commons Attribution Non-commercial Share Alike 4.0.
3.Conditions of self-archiving. Authors are encouraged to disseminate pre-print (draft papers prior to being assessed) and/or post-print versions (those reviewed and accepted for publication) of their papers before publication, because it encourages distribution earlier and thus leads to a possible increase in citations and circulation among the academic community.
RoMEO color: green
