Improving logical flow in English-as-a-foreign-language learner essays by reordering sentences. (English) Zbl 07702962
Summary: Argumentation is ubiquitous in everyday discourse, and it is a skill that can be learned. In our society, it is also one that must be learned: education systems all over the world agree on the importance of argumentation skills. However, writing effective argumentation is difficult, and even more so if it has to be expressed in a foreign language. Existing artificial intelligence systems for language learning can help learners: they can provide objective feedback (e.g., concerning grammar and spelling), as well as providing learners with opportunities to identify errors and subsequently improve their texts. Even so, systems aiming at higher discourse-level skills, such as persuasiveness and content organisation, are still limited. In this article, we propose the novel task of sentence reordering for improving the logical flow of argumentative essays. To train such a computational system, we present a new corpus called ICNALE-AS2R, containing essays written by English-as-foreign-language learners from various Asian countries, that have been annotated with argumentative structure and sentence reordering. We also propose a novel method to automatically reorder sentences in imperfect essays, which is based on argumentative structure analysis. Given an input essay and its corresponding argumentative structure, we cast the reordering task as a traversal problem. Our sentence reordering system first determines the pairwise ordering relation between pairs of sentences that are connected by argumentative relations. In the second step, the system traverses the argumentative structure that has been augmented with pairwise ordering information, in order to generate the final output text. Empirical evaluation shows that in the task of reconstructing the final reordered essays in the dataset, our reordering system achieves .926 and .879 in longest common subsequence ratio and Kendall’s Tau metrics, respectively. The system is also able to perform the reordering operation selectively, that is, it reorders sentences when necessary and retains the original input order when it is already optimal.
MSC:
| 68Txx | Artificial intelligence |
Keywords:
argumentative structure; corpus; English-as-foreign-language learners; intelligent tutoring systems; sentence reorderingReferences:
| [1] | Al-Khatib, K.; Wachsmuth, H.; Hagen, M.; Stein, B., Patterns of argumentation strategies across topics, (Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Copenhagen, Denmark (2017)), 1351-1357 |
| [2] | Aristotle Kennedy, G., On Rhetoric: A Theory of Civil Discourse (1991), Oxford University Press |
| [3] | Bacha, N. N., Teaching the academic argument in a university EFL environment, J. Engl. Acad. Purpos., 9, 229-241 (2010) |
| [4] | Bamberg, B., What makes a text coherent, College Compos. Commun., 34, 417-429 (1983) |
| [5] | Barzilay, R.; Elhadad, N.; McKeown, K. R., Inferring strategies for sentence ordering in multidocument news summarization, J. Artif. Intell. Res., 17, 35-55 (2002) · Zbl 1056.68140 |
| [6] | Barzilay, R.; Lapata, M., Modeling local coherence: an entity-based approach, Comput. Linguist., 34, 1-34 (2008) |
| [7] | Ben-Anath, D., The role of connectives in text comprehension, Columbia Univ. Work. Papers TESOL Appl. Linguist., 5, 1-27 (2005) |
| [8] | Blair, J. A., Groundwork in the Theory of Argumentation (2012), Springer |
| [9] | Britt, M. A.; Larson, A. A., Constructing representations of arguments, J. Mem. Lang., 48, 794-810 (2003) |
| [10] | Carenini, G.; Moore, J. D., Generating and evaluating evaluative arguments, Artif. Intell., 170, 925-952 (2006) |
| [11] | Chen, X.; Qiu, X.; Huang, X., Neural sentence ordering (2016), CoRR |
| [12] | Cho, K.; MacArthur, C., Student revision with peer and expert reviewing, Learn. Instr., 20, 328-338 (2010) |
| [13] | Choi, E.; Palomaki, J.; Lamm, M.; Kwiatkowski, T.; Das, D.; Collins, M., Decontextualization: making sentences stand-alone, Trans. Assoc. Comput. Linguist., 9, 447-461 (2021) |
| [14] | Connor, U., New directions in contrastive rhetoric, TESOL Q., 36, 493-510 (2002) |
| [15] | Cui, B.; Li, Y.; Chen, M.; Zhang, Z., Deep attentive sentence ordering network, (Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018), Association for Computational Linguistics: Association for Computational Linguistics Brussels, Belgium), 4340-4349 |
| [16] | Cullen, S.; Fan, J.; van der Brugge, E.; Elga, A., Improving analytical reasoning and argument understanding: a quasi-experimental field study of argument visualization, NPJ Sci. Learn., 3 (2018) |
| [17] | Devlin, J.; Chang, M. W.; Lee, K.; Toutanova, K., BERT: pre-training of deep bidirectional transformers for language understanding, (Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (2019), Association for Computational Linguistics: Association for Computational Linguistics Minneapolis, Minnesota), 4171-4186 |
| [18] | Dijk, T. A.V.; Kintsch, W., Strategies of Discourse Comprehension (1983), Academic Press: Academic Press New York |
| [19] | Dozat, T.; Manning, C. D., Deep biaffine attention for neural dependency parsing, (Proceedings of the International Conference on Learning Representations (2017)), 1-8 |
| [20] | El Baff, R.; Wachsmuth, H.; Al Khatib, K.; Stede, M.; Stein, B., Computational argumentation synthesis as a language modeling task, (Proceedings of the 12th International Conference on Natural Language Generation (2019), Association for Computational Linguistics: Association for Computational Linguistics Tokyo, Japan), 54-64 |
| [21] | Ermakova, L.; Mothe, J.; Firsov, A., A metric for sentence ordering assessment based on topic-comment structure, (Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (2017), Association for Computing Machinery), 1061-1064 |
| [22] | Feng, V. W.; Lin, Z.; Hirst, G., The impact of deep hierarchical discourse structures in the evaluation of text coherence, (Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, Dublin City University and Association for Computational Linguistics, Dublin, Ireland (2014)), 940-949 |
| [23] | Fleiss, J. L., Measuring nominal scale agreement among many raters, Psychol. Bull., 76, 378-382 (1971) |
| [24] | Garing, A. G., (Coherence in Argumentative Essays of First Year College of Liberal Arts Students at de la Salle University (2014), DLSU Research Congress) |
| [25] | Gong, J.; Chen, X.; Qiu, X.; Huang, X., End-to-end neural sentence ordering using pointer network (2016), CoRR |
| [26] | Grosz, B. J.; Sidner, C. L., Attention, intentions, and the structure of discourse, Comput. Linguist., 12, 175-204 (1986) |
| [27] | Guinaudeau, C.; Strube, M., Graph-based local coherence modeling, (Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2013), Association for Computational Linguistics: Association for Computational Linguistics Sofia, Bulgaria), 93-103 |
| [28] | Halliday, M. A.K.; Hasan, R., Cohesion in English (1976), Longman: Longman London and New York |
| [29] | Herdan, G., Type-Token Mathematics (1960), The Hague: The Hague Mouton · Zbl 0163.40904 |
| [30] | Hirst, G.; Budanitsky, A., Correcting real-word spelling errors by restoring lexical cohesion, Nat. Lang. Eng., 11, 87-111 (2005) |
| [31] | Hovy, E. H., Approaches to the Planning of Coherent Text, 83-102 (1991), Springer US: Springer US Boston, MA |
| [32] | Hovy, E. H., Automated discourse generation using discourse structure relations, Artif. Intell., 63, 341-385 (1993) |
| [33] | Iida, R.; Tokunaga, T., Building a corpus of manually revised texts from discourse perspective, (Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14) (2014), European Language Resources Association (ELRA): European Language Resources Association (ELRA) Reykjavik, Iceland), 936-941 |
| [34] | Invanic, R., Discourses of writing and learning to write, Lang. Educ., 18, 220-245 (2004) |
| [35] | Ishikawa, S., The ICNALE and sophisticated contrastive interlanguage analysis of asian learners of english, Learn. Corp. Stud. Asia World, 1, 91-118 (2013) |
| [36] | Ishikawa, S., The ICNALE edited essays: a dataset for analysis of l2 english learner essays based on a new integrative viewpoint, Engl. Corpus Linguist., 25, 1-14 (2018) |
| [37] | Johns, A. M., The esl student and the revision process: some insights from scheme theory, J. Basic Writ., 5, 70-80 (1986) |
| [38] | Kaplan, R. B., Cultural thought patterns in inter-cultural education, Lang. Learn., 16, 1-20 (1966) |
| [39] | Kendall, M. G., A new measure of rank correlation, Biometrika, 30, 81-93 (1938) · Zbl 0019.13001 |
| [40] | Kleijn, S.; Maat, H. L.P.; Sanders, T. J., Comprehension effects of connectives across texts, readers, and coherence relations, Discourse Process., 56, 447-464 (2019) |
| [41] | Lai, A.; Tetreault, J., Discourse coherence in the wild: a dataset, evaluation and methods, (Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue (2018), Association for Computational Linguistics: Association for Computational Linguistics Melbourne, Australia), 214-223 |
| [42] | Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; Soricut, R., ALBERT: a lite BERT for self-supervised learning of language representations, (Proceedings of the International Conference on Learning Representations (2020)), 1-17 |
| [43] | Lapata, M., Probabilistic text structuring: experiments with sentence ordering, (Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (2003), Association for Computational Linguistics: Association for Computational Linguistics Sapporo, Japan), 545-552 |
| [44] | Lapata, M., Automatic evaluation of information ordering: Kendall’s tau, Comput. Linguist., 32, 471-484 (2006) · Zbl 1234.68418 |
| [45] | Lawrence, J.; Reed, C., Argument mining: a survey, Comput. Linguist., 45, 765-818 (2020) |
| [46] | Li, J.; Jurafsky, D., Neural net models of open-domain discourse coherence, (Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (2017), Association for Computational Linguistics: Association for Computational Linguistics Copenhagen, Denmark), 198-209 |
| [47] | Liddy, E. D., The discourse-level structure of empirical abstracts: an exploratory study, Inf. Process. Manag., 27, 55-81 (1991) |
| [48] | Lin, Z.; Ng, H. T.; Kan, M. Y., Automatically evaluating text coherence using discourse relations, (Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (2011), Association for Computational Linguistics: Association for Computational Linguistics Portland, Oregon, USA), 997-1006 |
| [49] | Lippi, M.; Torroni, P., Argumentation mining: state of the art and emerging trends, ACM Trans. Internet Technol., 16 (2016) |
| [50] | Liu, L., Rhetorical education through writing instruction across cultures: a comparative analysis of select online instructional materials on argumentative writing, J. Second Lang. Writ., 14, 1-18 (2005) |
| [51] | Logeswaran, L.; Lee, H.; Radev, D. R., Sentence ordering and coherence modeling using recurrent neural networks, (AAAI (2018)), 5285-5292 |
| [52] | Louis, A.; Nenkova, A., A coherence model based on syntactic patterns, (Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (2012), Association for Computational Linguistics: Association for Computational Linguistics Jeju Island, Korea), 1157-1168 |
| [53] | Mann, W.; Thompson, S., Rhetorical structure theory: toward a functional theory of text organization, Text, 8, 243-281 (1988) |
| [54] | Matsumura, K.; Sakamoto, K., A structure analysis of japanese efl students’ argumentative paragraph writing with a tool for annotation discourse relation, (The Bulletin of the Writing Reserch Group (2021)), JACET Kansai Chapter 14 |
| [55] | McNemar, Q., Note on the sampling error of the difference between correlated proportions or percentages, Psychometrika, 12, 153-157 (1947) |
| [56] | Mesgar, M.; Strube, M., A neural local coherence model for text quality assessment, (Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018), Association for Computational Linguistics: Association for Computational Linguistics Brussels, Belgium), 4328-4339 |
| [57] | National Commission on Writing in American Schools and Colleges, The neglected R: the need for a writing revolution (2003), retrieved 20 October 2021 |
| [58] | Noreen, E. W., Computer Intensive Methods for Testing Hypotheses: An Introduction (1989), Wiley: Wiley New York |
| [59] | Peldszus, A.; Stede, M., From argument diagrams to argumentation mining in texts: a survey, Int. J. Cogn. Inf. Nat. Intell., 7, 1-31 (2013) |
| [60] | Persing, I.; Davis, A.; Ng, V., Modeling organization in student essays, (Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (2010), Association for Computational Linguistics: Association for Computational Linguistics Cambridge, MA), 229-239 |
| [61] | Persing, I.; Ng, V., Modeling argument strength in student essays, (Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (2015), Association for Computational Linguistics: Association for Computational Linguistics Beijing, China), 543-552 |
| [62] | Pishdad, L.; Fancellu, F.; Zhang, R.; Fazly, A., How coherent are neural models of coherence?, (Proceedings of the 28th International Conference on Computational Linguistics (2020), International Committee on Computational Linguistics: International Committee on Computational Linguistics Barcelona, Spain), 6126-6138, (Online) |
| [63] | Prabhumoye, S.; Salakhutdinov, R.; Black, A. W., Topological sort for sentence ordering, (Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (2020), Association for Computational Linguistics), 2783-2792, Online |
| [64] | Putra, J. W.G.; Teufel, S.; Matsumura, K.; Tokunaga, T., TIARA: a tool for annotating discourse relations and sentence reordering, (Proceedings of the 12th Language Resources and Evaluation Conference (2020), European Language Resources Association: European Language Resources Association Marseille, France), 6912-6920 |
| [65] | Putra, J. W.G.; Teufel, S.; Tokunaga, T., Parsing argumentative structure in English-as-foreign-language essays, (Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications (2021), Association for Computational Linguistics), 97-109, Online |
| [66] | Putra, J. W.G.; Teufel, S.; Tokunaga, T., Annotating argumentative structure in english-as-foreign-language learner essays, Nat. Lang. Eng., 1-37 (2021) |
| [67] | Putra, J. W.G.; Teufel, S.; Tokunaga, T., Multi-task and multi-corpora training strategies to enhance argumentative sentence linking performance, (Proceedings of the 8th Workshop in Argument Mining (2021), Association for Computational Linguistics), Online |
| [68] | Putra, J. W.G.; Tokunaga, T., Evaluating text coherence based on semantic similarity graph, (Proceedings of TextGraphs-11: the Workshop on Graph-Based Methods for Natural Language Processing (2017), Association for Computational Linguistics: Association for Computational Linguistics Vancouver, Canada), 76-85 |
| [69] | Rabinovich, E.; Nisioi, S.; Ordan, N.; Wintner, S., On the similarities between native, non-native and translated texts, (Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2016), Association for Computational Linguistics: Association for Computational Linguistics Berlin, Germany), 1870-1881 |
| [70] | Reimers, N.; Gurevych, I., Sentence-BERT: sentence embeddings using Siamese BERT-networks, (Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (2019)), 3982-3992 |
| [71] | Sanders, T. J., Coherence, causality and cognitive complexity in discourse, (Proceedings/Actes SEM-05, First International Symposium on the Exploration and Modelling of Meaning, Toulouse, France (2005)), 105-114 |
| [72] | Segal, E. M.; Duchan, J. F.; Scott, P. J., The role of interclausal connectives in narrative structuring: evidence from adults’ interpretations of simple stories, Discourse Process., 14, 27-54 (1991) |
| [73] | Shute, V. J., Focus on formative feedback, Rev. Educ. Res., 78, 153-189 (2008) |
| [74] | Silva, T., Toward an understanding of the distinct nature of L2 writing: the esl research and its implications, TESOL Q., 27, 657-677 (1993) |
| [75] | Stab, C.; Gurevych, I., Parsing argumentation structures in persuasive essays, Comput. Linguist., 43, 619-659 (2017) |
| [76] | Strobl, C.; Ailhaud, E.; Benetos, K.; Devitt, A.; Kruse, O.; Proske, A.; Rapp, C., Digital support for academic writing: a review of technologies and pedagogies, Comput. Educ., 131, 33-48 (2019) |
| [77] | Tarjan, R., Edge-disjoint spanning trees and depth-first search, Acta Inform., 6, 171-185 (1976) · Zbl 0307.05104 |
| [78] | Todd, R. W.; Thienpermpool, P.; Keyuravong, S., Measuring the coherence of writing using topic-based analysis, Assess. Writ., 9, 85-104 (2004) |
| [79] | Wachsmuth, H.; Al-Khatib, K.; Stein, B., Using argument mining to assess the argumentation quality of essays, (Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (2016), The COLING 2016 Organizing Committee: The COLING 2016 Organizing Committee Osaka, Japan), 1680-1691 |
| [80] | Wachsmuth, H.; Stede, M.; El Baff, R.; Al-Khatib, K.; Skeppstedt, M.; Stein, B., Argumentation synthesis following rhetorical strategies, (Proceedings of the 27th International Conference on Computational Linguistics (2018), Association for Computational Linguistics: Association for Computational Linguistics Santa Fe, New Mexico, USA), 3753-3765 |
| [81] | Wambsganss, T.; Caines, A.; Buttery, P., ALEN app: argumentative writing support to foster English language learning, (Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022) (2022), Association for Computational Linguistics: Association for Computational Linguistics Seattle, Washington), 134-140 |
| [82] | Webber, B.; Joshi, A., Discourse structure and computation: past, present and future, (Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries (2012), Association for Computational Linguistics: Association for Computational Linguistics Jeju Island, Korea), 42-54 |
| [83] | Weischedel, R. M.; Voge, W. M.; James, M., An artificial intelligence approach to language instruction, Artif. Intell., 10, 225-240 (1978) |
| [84] | Yanase, T.; Miyoshi, T.; Yanai, K.; Sato, M.; Iwayama, M.; Niwa, Y.; Reisert, P.; Inui, K., Learning sentence ordering for opinion generation of debate, (Proceedings of the 2nd Workshop on Argumentation Mining (2015), Association for Computational Linguistics: Association for Computational Linguistics Denver, CO), 94-103 |
| [85] | Yin, Y.; Song, L.; Su, J.; Zeng, J.; Zhou, C.; Luo, J., Graph-based neural sentence ordering, (Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (2019)), 5387-5393 |
| [86] | Yuan, Z.; Briscoe, T., Grammatical error correction using neural machine translation, (Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016), Association for Computational Linguistics: Association for Computational Linguistics San Diego, California), 380-386 |
| [87] | Zhang, F.; Hashemi, H. B.; Hwa, R.; Litman, D., A corpus of annotated revisions for studying argumentative writing, (Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2017), Association for Computational Linguistics: Association for Computational Linguistics Vancouver, Canada), 1568-1578 |
| [88] | Zhang, F.; Litman, D., Annotation and classification of argumentative writing revisions, (Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications (2015)), 133-143 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
