计算机科学 ›› 2021, Vol. 48 ›› Issue (6): 63-70.doi: 10.11896/jsjkx.200500138

• 数据库&大数据&数据科学 • 上一篇    下一篇

学术论文公开评审平台数据分析

张明阳, 王刚, 彭起, 张岩峰   

  1. 东北大学计算机科学与工程学院 沈阳110169
  • 收稿日期:2020-05-27 修回日期:2020-08-18 出版日期:2021-06-15 发布日期:2021-06-03
  • 通讯作者: 张岩峰(zhangyf@mail.neu.edu.cn)
  • 基金资助:
    国家自然科学基金(61672141);辽宁省重点研发计划(2020JH2/10100037);中央高校基本科研业务费(N181605017,N181604016)

Data Analysis of OpenReview

ZHANG Ming-yang, WANG Gang, PENG Qi, ZHANG Yan-feng   

  1. School of Computer Science and Engineering,Northeastern University,Shenyang 110169,China
  • Received:2020-05-27 Revised:2020-08-18 Online:2021-06-15 Published:2021-06-03
  • About author:ZHANG Ming-yang,born in 1998,under-graduate.Her main research interests include data mining and machine learning.(may_zh@foxmail.com)
    ZHANG Yan-feng,born in 1982,professor,Ph.D supervisor,is a senior member of China Computer Federation.His main research interests include big data mining,large-scale machine learning and distributed systems.
  • Supported by:
    National Natural Science Foundation of China (61672141),Key R&D Program of Liaoning Province(2020JH2/10100037)and Fundamental Research Funds for the Central Universities(N181605017,N181604016).

摘要: 我国目前的学术评价体制饱受诟病,因而构建公平、公正、公开的学术评价体制对营造良好的学术环境至关重要。近年来,学术论文公开评审网站OpenReview的出现给学术论文评价带来了一种新思路。论文评审采用双盲机制,论文及审稿意见向大众公开,这在一定程度上加强了同行对审稿过程的监督,提高了学术评价的公平性。该模式已经被部分人工智能领域的国际顶级会议所采用。文中爬取了OpenReview平台上的5527篇论文投稿及16853条评审意见数据进行大数据分析,并重点关注了参与其中的中国投稿人和中国评审人的数据分析,得到若干统计分析结果,这些统计结果对了解我国科研人员的特点、改善我国学术评价体制具有一定的参考价值。

关键词: OpenReview, 录用率, 同行评审, 文本分析, 文本情感分析

Abstract: The current academic evaluation system in China has been criticized for several years.It is crucial to build a fair,impartial and open academic evaluation system for creating a good academia environment.In recent years, the emergence of OpenReview,an open review website for academic papers,has brought a new idea to the evaluation of academic papers.It employs double-blind review process and makes all the submissions and reviews publicly accessible,which strengthens the supervision of the review process and improves paper review’s fairness and openness,and makes OpenReview widely used in top AI conferences.This paper collects 5527 submissions and their 16853 reviews from the OpenReview platform and performs several big data analysis tasks.It mainly focuses on the submissions from Chinese scholars and the reviews written by Chinese scholars,and obtains seve-ral interesting results.These results are helpful for understanding the characteristics of Chinese scholars and can provide insightful suggestions to improve our academic evaluation system.

Key words: Acceptance rate, OpenReview, Peer review, Text analysis

中图分类号: 

  • TP391.1
[1]ALIAKSANDR B,JOSEPH R W,CLAUDIO B,et al.Alternatives to peer review:novel approaches for research evaluation[J].Computational neuroscience,2011,5:66.
[2]CLARK K,LUONG M T,LE Q V,et al.ELECTRA:Pre-trai-ning Text Encoders as Discriminators Rather Than Generators[J].arXiv:2003.10555,2020.
[3]DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding[J].arXiv:1810.04805,2018.
[4]Artificial-intelligence-terminology[EB/OL].https://github.com/jiqizhixin/Artficial Intelligence-Terminology.
[5]JOE H,WARD J.Hierarchical grouping to optimize an objective function[J].Journal of the American Statistical Association,1963,58(301):236-244.
[6]CECI S,DOUGLAS P.Peer review:A study of reliability[J].Change:The Magazine of Higher Learning,1982,14(6):44-48.
[7]LEE K,BOYD E,HOLROYD L J,et al.Predictors of publication:characteristics of submitted manuscripts as sociated with acceptance at major biomedical journals[J].Med,2006,184(12):621-626.
[8]LYNCH J R,CUNNINGHAM M R,WARME W J,et al.Commercially funded and united states-based research is more likely to be published;good-quality studies with negative outcomes are not[J].Bone Joint Surg,2007,89(5):1010-1018.
[9]CECI S J,WILLIAMS W M.Understanding current causes of women’s under representation inscience[J].Proceedings of the National Academy of Sciences,2011,108(8):3157-3162.
[10]KANG D,AMMAR W,DALVI B,et al.A dataset of peer reviews(peerread):Collection,insights and NLP applications[C]//NAACL.2018.
[11]WANG K,WAN X.Sentiment analysis of peer review texts for scholarly papers[C]//SIGIR.2018:175-184.
[12]GHOSAL T,VERMA R,EKBAL A,et al.Deepsentipeer:Harnessing sentiment in review texts to recommend peer review decisions[C]//ACL.2019:1120-1130.
[13]PRICE S,FLACH P A.Computational support for academicpeer review:A perspective from artificial intelligence[J].Communication of the ACM,2017 60(3):70-79.
[14]CHARLIN L,ZEMEL R.The toronto paper matching system:an automated paper reviewer assignment system[C]//ICLM.2013.
[15]MROWINSKI M J,FRONCZAK P,FRONCZAK A,et al.Artificial intelligence in peer review:How can evolutionary computation support journal editors[J].PloS one,2017,12(9).
[16]STELMAKH I,SHAH N B,SINGH A.On testing for biases in peer review[C]//ACM EC Workshop on Mechanism Design for Social Good.2019:5287-5297.
[17]BIRUKOU A,WAKELING J R,BARTOLINI C,et al.Alternatives to peer review:novel approaches for research evaluation [J].BMJ,2015.
[18]GAO Y,EGER S,KUZNETSOV I,et al.Does My RebuttalMatter?Insights from a Major NLP Conference[C]//Procee-dings of the 2019 Conference of the North.2019.
[19]LI S,ZHAO W X,YIN E J,et al.A neural citation count prediction model based on peer review text[C]//EMNLP.2019:4913-4923.
[1] 李建兰, 潘岳, 李小聪, 刘子维, 王天宇.
基于CiteSpace的中文评论文本研究现状与趋势分析
Chinese Commentary Text Research Status and Trend Analysis Based on CiteSpace
计算机科学, 2021, 48(11A): 17-21. https://doi.org/10.11896/jsjkx.210300172
[2] 刘慧清, 郭延哺, 李红灵, 李维华.
基于贝叶斯网的短文本特征扩展方法
Short Text Feature Extension Method Based on Bayesian Networks
计算机科学, 2019, 46(11A): 66-71.
[3] 刘金,吴 斌,陈 震,沈崇玮.
基于领域划分的微博用户影响力分析
Research on Influence of Micro Blogging Based on Field Division
计算机科学, 2015, 42(5): 42-46. https://doi.org/10.11896/j.issn.1002-137X.2015.05.008
[4] 李敏,王晓聪,张军,刘正捷.
基于位置的社交网络用户签到及相关行为研究
Study on Check-in and Related Behaviors of Location-based Social Network Users
计算机科学, 2013, 40(10): 72-76.
[5] 井晓阳,罗飞,王亚棋.
汉语语音合成技术综述
Overview of the Chinese Voice Synthesis Technique
计算机科学, 2012, 39(Z11): 386-390.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!