Computer Science ›› 2023, Vol. 50 ›› Issue (3): 315-322.doi: 10.11896/jsjkx.220100137

• Artificial Intelligence • Previous Articles     Next Articles

Chinese Argumentative Writing Quality Evaluation Based on Multi-perspective Modeling

HE Yaqiong, JIANG Feng, CHU Xiaomin, LI Peifeng   

  1. School of Computer Science and Technology,Soochow University,Suzhou,Jiangsu 215006,China
  • Received:2022-01-14 Revised:2022-08-14 Online:2023-03-15 Published:2023-03-15
  • About author:HE Yaqiong,born in 1997,postgra-duate,is a member of China Computer Federation.His main research interests include natural language processing and so on.
    LI Peifeng,born in 1971,Ph.D,professor,Ph.D supervisor,is a member of China Computer Federation.His main research interests include natural language processing and machine learning.
  • Supported by:
    National Natural Science Foundation of China(61836007,62006167) and Priority Academic Program Development of Jiangsu Higher Education Institutions(PAPD).

Abstract: Automated essay scoring is a task that replaces manual grading for students’ essays,where rich semantics,rigorous organization,and reasonable logic are important considering factors.Most previous studies only consider the semantics or organization of the essay from a single perspective,lacking considering higher-level factors such as logic.Therefore,this paper proposes a multi-perspective evaluation framework(MPE) to more objective and reliable evaluate the essay from semantics,organization,and logic.MPE first utilizes the pre-trained model to encode sentence and obtain three levels semantic information to evaluate the essay's semantic expression.Then,it combines sentence function identification and paragraph function identification to evaluate the essay′s organization.Moreover,MPE evaluates the essay's logic by calculating the coherence between paragraphs.Finally,the framework scores the essay by integrating these three evaluation perspectives.Experimental results show that the proposed multi-perspective evaluation framework can effectively score the essays at various qualities,outperforming all the baselines.

Key words: Multi-perspective, Essay score, Argumentation, XLNet, Global coherence

CLC Number: 

  • TP391
[1]MESGAR M,STRUBE M.A neural local coherence model fortext quality assessment[C]//Proceedings of the 2018 Confe-rence on Empirical Methods in Natural Language Processing.2018:4328-4339.
[2]LIU J,XU Y,ZHU Y.Automated essay scoring based on two-stage learning[J].arXiv:1901.07744,2019.
[3]YANG Y,ZHONG J.Automated essay scoring via example-based learning[C]//International Conference on Web Enginee-ring.Cham:Springer,2021:201-208.
[4]CHEN H,HE B.Automated essay scoring by maximizing human-machine agreement [C]//Proceedings of the 2013 Confe-rence on Empirical Methods in Natural Language Processing.2013:1741-1752.
[5]SOMASUNDARAN S,BURSTEIN J,CHODOROW M.Lexical chaining for measuring discourse coherence quality in test-taker essays[C]//The 25th International Conference on Computa-tional Linguistics:Technical papers(COLING 2014).2014:950-961.
[6]YANNAOUDAKIS H,BRISCOE T,MEDLOCK B.A newdataset and method for automatically grading ESOL texts[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:Human Language Technologies.2011:180-189.
[7]PHANDI P,CHAI K M A,NG H T.Flexible domain adaptation for automated essay scoring using correlated linear regression[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.2015:431-439.
[8]ALIKANIOTIS D,YANNAOUDAKIS H,REI M.Automatictext scoring using neural networks[J].arXiv:1606.04289,2016.
[9]TAGHIPOUR K,NG H T.A neural approach to automated essay scoring[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.2016:1882-1891.
[10]DONG F,ZHANG Y.Automatic features for essay scoring-an empirical study[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.2016:1072-1077.
[11]DONG F,ZHANG Y,YANG J.Attention-based recurrent convolutional neural network for automatic essay scoring[C]//Proceedings of the 21st Conference on Computational Natural Language Learning(CoNLL 2017).2017:153-162.
[12]SOMASUNDARAN S,FLOR M,CHODOROW M,et al.To-wards evaluating narrative quality in student writing[J/OL].Transactions of the Association for Computational Linguistics,2018,6:91-106.https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00007/43428/Towards-Evaluating-Narrative-Qua-lity-In-Student.
[13]PERSING I,NG V.Modeling stance in student essays[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers).2016:2174-2184.
[14]MATHIAS S,BHATTACHARYYA P.Thank “Goodness”! A Way to Measure Style in Student Essays[C]//Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications.2018:35-41.
[15]KE Z,INAMDAR H,LIN H,et al.Give me more feedback II:Annotating thesis strength and related attributes in student essays[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019:3994-4004.
[16]PERSING I,DAVIS A,NG V.Modeling organization in student essays[C]//Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing.2010:229-239.
[17]SONG W,SONG Z,LIU L,et al.Hierarchical Multi-task Lear-ning for Organization Evaluation of Argumentative Student Essays[C]//IJCAI.2020:3875-3881.
[18]CHEN Y.Convolutional neural network for sentence classification[D].Canadian:University of Waterloo,2015.
[19]SHI X J,CHEN Z,WANG H,et al.Convolutional LSTM network:A machine learning approach for precipitation nowcasting[C]//Advances in Neural Information Processing Systems.2015:802-810.
[20]LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-basedlearning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[21]WON Y.The Prediction of Writing Scores Using Vocabulary Features in ESL University Students’ Essays[J].Modern English Education Society,2019,20(4):31-40.
[22]LE Q,MIKOLOV T.Distributed representations of sentences and documents[C]//International Conference on Machine Learning.PMLR,2014:1188-1196.
[23]LIU C,ZHAO S,VOLKOVS M.Unsupervised document embedding with cnns[J].arXiv:1711.04168,2017.
[24]WU L,YEN I E H,XU K,et al.Word mover's embedding:From word2vec to document embedding[J].arXiv:1811.01713,2018.
[25]YANG Z,DAI Z,YANG Y,et al.XLNet:Generalized autore-gressive pretraining for language understaning[J/OL].Advances in neural information processing systems,2019,32.https://proceedings.neurips.cc/paper/2019/hash/dc6a7e655d7e5840e66733e9ee67cc69-Abstract.html.
[26]ATTALI Y,BURSTEIN J.Automated essay scoring with e-ra-ter© V.2[J/OL].The Journal of Technology,Learning and Assessment,2006,4(3).https://ejournals.bc.edu/index.php/jtla/article/view/1650.
[27]LIANG M C.A study of coherence in EFL learners’ writtenproduction [J].Modern Foreign Languages,2006,29(3):284-292.
[28]LOUIS A,NENKOVA A.A coherence model based on syntactic patterns[C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning.2012:1157-1168.
[29]MILTSAKAKI E,KUKICH K.Evaluation of text coherence for electronic essay scoring systems[J].Natural Language Engineering,2004,10(1):25-55.
[30]LIAO D,XU J,LI G,et al.Hierarchical Coherence Modeling for Document Quality Assessment[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2021,35(15):13353-13361.
[31]XU W C.Cohesion,coherence and quality in English composition[J].Journal of GuangZhou University,2000(5):71-75.
[32]MA G G.A comparative analysis of the linguistic features ofEnglish composition between Chinese and American College Students[J].Foreign Language Teaching Research,2002,34(5):345-350.
[33]ZHU Y S.Halliday's standard of discourse coherence is misunderstood by the outside world and its own shortcomings[J].Foreign Language Teaching and Research,1997(1):23-27.
[34]MCNAMARA D S,LOUWERSE M M,GRAESSER A C.CohMetrix:Automated cohesion and coherence scoresto predict text readability an-d facilitate comprehension[R].Technical report,Institute for Intelligent Systems,University of Memphis,Memphis,TN,2002.
[35]VASWANIA,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Advances in Neural Information Processing Systems.2017:5998-6008.
[36]乐乐.问题[OL].http://www.leleketang.com/zuowen/287886.shtml.
[1] TAN Li-xing, WANG Fu-jun. On Logic of Graded Argumentation System [J]. Computer Science, 2020, 47(5): 27-31.
[2] WEI Bin. Argument Game Model for Gradual Argumentation Semantic [J]. Computer Science, 2017, 44(4): 256-262.
[3] LIU Bin,YAO Li,HAO Zhi-yong and GONG Yong. Research on Argument Ontology for Computational Argumentation [J]. Computer Science, 2014, 41(2): 226-231.
[4] LI Shan-shan and CAO Cun-gen. Commonsense Knowledge Analysis Approach Based on Event Preconditions and Effects [J]. Computer Science, 2013, 40(4): 185-192.
[5] . Union Argumentation Framework Based on Subject Trustworthiness [J]. Computer Science, 2011, 38(11): 179-186.
[6] FU Yi-xing,QI Xue-tian,YAO Li ,WANG Yan-juan. Study of Decision Support Technology Based on Value-based Argumentation Framework [J]. Computer Science, 2011, 38(10): 205-208.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!