Computer Science ›› 2021, Vol. 48 ›› Issue (6): 63-70.doi: 10.11896/jsjkx.200500138

• Database & Big Data & Data Science • Previous Articles     Next Articles

Data Analysis of OpenReview

ZHANG Ming-yang, WANG Gang, PENG Qi, ZHANG Yan-feng   

  1. School of Computer Science and Engineering,Northeastern University,Shenyang 110169,China
  • Received:2020-05-27 Revised:2020-08-18 Online:2021-06-15 Published:2021-06-03
  • About author:ZHANG Ming-yang,born in 1998,under-graduate.Her main research interests include data mining and machine learning.(may_zh@foxmail.com)
    ZHANG Yan-feng,born in 1982,professor,Ph.D supervisor,is a senior member of China Computer Federation.His main research interests include big data mining,large-scale machine learning and distributed systems.
  • Supported by:
    National Natural Science Foundation of China (61672141),Key R&D Program of Liaoning Province(2020JH2/10100037)and Fundamental Research Funds for the Central Universities(N181605017,N181604016).

Abstract: The current academic evaluation system in China has been criticized for several years.It is crucial to build a fair,impartial and open academic evaluation system for creating a good academia environment.In recent years, the emergence of OpenReview,an open review website for academic papers,has brought a new idea to the evaluation of academic papers.It employs double-blind review process and makes all the submissions and reviews publicly accessible,which strengthens the supervision of the review process and improves paper review’s fairness and openness,and makes OpenReview widely used in top AI conferences.This paper collects 5527 submissions and their 16853 reviews from the OpenReview platform and performs several big data analysis tasks.It mainly focuses on the submissions from Chinese scholars and the reviews written by Chinese scholars,and obtains seve-ral interesting results.These results are helpful for understanding the characteristics of Chinese scholars and can provide insightful suggestions to improve our academic evaluation system.

Key words: Acceptance rate, OpenReview, Peer review, Text analysis

CLC Number: 

  • TP391.1
[1]ALIAKSANDR B,JOSEPH R W,CLAUDIO B,et al.Alternatives to peer review:novel approaches for research evaluation[J].Computational neuroscience,2011,5:66.
[2]CLARK K,LUONG M T,LE Q V,et al.ELECTRA:Pre-trai-ning Text Encoders as Discriminators Rather Than Generators[J].arXiv:2003.10555,2020.
[3]DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding[J].arXiv:1810.04805,2018.
[4]Artificial-intelligence-terminology[EB/OL].https://github.com/jiqizhixin/Artficial Intelligence-Terminology.
[5]JOE H,WARD J.Hierarchical grouping to optimize an objective function[J].Journal of the American Statistical Association,1963,58(301):236-244.
[6]CECI S,DOUGLAS P.Peer review:A study of reliability[J].Change:The Magazine of Higher Learning,1982,14(6):44-48.
[7]LEE K,BOYD E,HOLROYD L J,et al.Predictors of publication:characteristics of submitted manuscripts as sociated with acceptance at major biomedical journals[J].Med,2006,184(12):621-626.
[8]LYNCH J R,CUNNINGHAM M R,WARME W J,et al.Commercially funded and united states-based research is more likely to be published;good-quality studies with negative outcomes are not[J].Bone Joint Surg,2007,89(5):1010-1018.
[9]CECI S J,WILLIAMS W M.Understanding current causes of women’s under representation inscience[J].Proceedings of the National Academy of Sciences,2011,108(8):3157-3162.
[10]KANG D,AMMAR W,DALVI B,et al.A dataset of peer reviews(peerread):Collection,insights and NLP applications[C]//NAACL.2018.
[11]WANG K,WAN X.Sentiment analysis of peer review texts for scholarly papers[C]//SIGIR.2018:175-184.
[12]GHOSAL T,VERMA R,EKBAL A,et al.Deepsentipeer:Harnessing sentiment in review texts to recommend peer review decisions[C]//ACL.2019:1120-1130.
[13]PRICE S,FLACH P A.Computational support for academicpeer review:A perspective from artificial intelligence[J].Communication of the ACM,2017 60(3):70-79.
[14]CHARLIN L,ZEMEL R.The toronto paper matching system:an automated paper reviewer assignment system[C]//ICLM.2013.
[15]MROWINSKI M J,FRONCZAK P,FRONCZAK A,et al.Artificial intelligence in peer review:How can evolutionary computation support journal editors[J].PloS one,2017,12(9).
[16]STELMAKH I,SHAH N B,SINGH A.On testing for biases in peer review[C]//ACM EC Workshop on Mechanism Design for Social Good.2019:5287-5297.
[17]BIRUKOU A,WAKELING J R,BARTOLINI C,et al.Alternatives to peer review:novel approaches for research evaluation [J].BMJ,2015.
[18]GAO Y,EGER S,KUZNETSOV I,et al.Does My RebuttalMatter?Insights from a Major NLP Conference[C]//Procee-dings of the 2019 Conference of the North.2019.
[19]LI S,ZHAO W X,YIN E J,et al.A neural citation count prediction model based on peer review text[C]//EMNLP.2019:4913-4923.
[1] LI Jian-lan, PAN Yue, LI Xiao-cong, LIU Zi-wei, WANG Tian-yu. Chinese Commentary Text Research Status and Trend Analysis Based on CiteSpace [J]. Computer Science, 2021, 48(11A): 17-21.
[2] LIU Hui-qing, GUO Yan-bu, LI Hong-ling, LI Wei-hua. Short Text Feature Extension Method Based on Bayesian Networks [J]. Computer Science, 2019, 46(11A): 66-71.
[3] LI Wei-jiang and WANG Feng. Method of Query Expansion Based on LCA Prune Semantic Tree [J]. Computer Science, 2015, 42(Z6): 479-483.
[4] LIU Jin-long, WU Bin, CHEN Zhen and SHEN Chong-wei. Research on Influence of Micro Blogging Based on Field Division [J]. Computer Science, 2015, 42(5): 42-46.
[5] LI Min,WANG Xiao-cong,ZHANG Jun and LIU Zheng-jie. Study on Check-in and Related Behaviors of Location-based Social Network Users [J]. Computer Science, 2013, 40(10): 72-76.
[6] WANG Zhao-long,LI Xia,XU Rui-fang. Multi-keywords Query Expansion with OLCA Based Concept Tree Pruning [J]. Computer Science, 2010, 37(4): 132-.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!