Annotation and Classification of Evidence and Reasoning Revisions in Argumentative Writing

Published in: Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications, pages 75–84 (July 2020). doi: 10.18653/v1/2020.bea-1.7

Posted on RAND.org on February 03, 2021

by Tazin Afrin, Elaine Lin Wang, Diane Litman, Lindsay Clare Matsumura, Richard Correnti

Read More

Access further information on this document at Proceedings of the 15th Workshop on Innovative Use of NLP for Building Educational Applications

This article was published outside of RAND. The full text of the article can be found at the link above.

Automated writing evaluation systems can improve students' writing insofar as students attend to the feedback provided and revise their essay drafts in ways aligned with such feedback. Existing research on revision of argumentative writing in such systems, however, has focused on the types of revisions students make (e.g., surface vs. content) rather than the extent to which revisions actually respond to the feedback provided and improve the essay. We introduce an annotation scheme to capture the nature of sentence-level revisions of evidence use and reasoning (the 'RER' scheme) and apply it to 5th- and 6th-grade students' argumentative essays. We show that reliable manual annotation can be achieved and that revision annotations correlate with a holistic assessment of essay improvement in line with the feedback provided. Furthermore, we explore the feasibility of automatically classifying revisions according to our scheme.

Research conducted by

This report is part of the RAND Corporation External publication series. Many RAND studies are published in peer-reviewed scholarly journals, as chapters in commercial books, or as documents published by other organizations.

The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND's publications do not necessarily reflect the opinions of its research clients and sponsors.