Computer Science

Computer Science ›› 2023, Vol. 50 ›› Issue (4): 149-158.doi: 10.11896/jsjkx.211200175

• Artificial Intelligence • Previous Articles     Next Articles

Comprehensive Survey of Loss Functions in Knowledge Graph Embedding Models

SHEN Qiuhui1, ZHANG Hongjun2, XU Youwei1, WANG Hang1, CHENG Kai2   

  1. 1 School of Graduate,Army Engineering University of PLA,Nanjing 210007,China
    2 College of Command and Control Engineering,Army Engineering University of PLA,Nanjing 210000,China
  • Received:2021-12-15 Revised:2022-09-05 Online:2023-04-15 Published:2023-04-06
  • About author:SHEN Qiuhui,born in 1987,Ph.D,lecturer.Her main research interests include knowledge graph and cloud computing.
    ZHANG Hongjun,born in 1963,professor,Ph.D supervisor.His main research interests include data engineering and military modeling & simulation.
  • Supported by:
    National Natural Science Foundation of China(61806221).

PDF (PC)

557

Abstract: Due to its rich and intuitive expressivity,knowledge graph has received much attention of many scholars. A lot of works have been accumulated in knowledge graph embedding. The results of the works have played an important role in some fields, such as e-commerce, finance,medicine, transportation and intelligent Q & A. In the knowledge graph embedding model, the loss function plays a key role in its training stage. Based on the existing research of knowledge graph embeddings, this paper classifies the loss functions used in the model into six categories: hinge loss, logistic loss, cross entropy loss, log likelihood loss, negative sampling loss and mean square error loss. The prototype formula and physical meaning of loss functions and their expansion, evolution and application in knowledge graph embedding models are analyzed in detail one by one.Based on the above,the usage, efficiency and convergence of various loss functions in the static and dynamic knowledge graph scenarios are comprehensively analyzed and evaluated. According to the analysis results, combined with the development and application trend of knowledge graph and the current situation of loss functions,the future works of loss functions are discussed.

Key words: Knowledge graph, Loss function, Embedding model, Knowledge representation learning

CLC Number: 

  • TP311
[1]BORDES A,USUNIER N,GARCIA-DURAN A,et al.Translating embeddings for modeling multi-relational data[C]//Neural Information Processing Systems(NIPS).2013:1-9.
[2]WANG Z,ZHANG J,FENG J,et al.Knowledge graph embedding by translating on hyperplanes[C]//AAAI Conference on Artificial Intelligence.2014:1112-1119.
[3]SUN Z,VASHISHTH S,SANYAL S,et al.A Re-evaluation of Knowledge Graph Completion Methods[EB/OL].(2020-07-08)[2021-12-13].https://arxiv.org/abs/1911.03903v3.
[4]NICKEL M,TRESP V,KRIEGEL H P.A three-way model for collective learning on multi-relational data[C]//International Conference on International Conference on Machine Learning.2011:809-816.
[5]HAYASHI K,SHIMBO M.On the Equivalence of Holographicand Complex Embeddings for Link Prediction[C]//Proceedings of the 55th Annual Meeting of the Association for Computa-tional Linguistics(ACL).2017:554-559.
[6]NICKEL M,ROSASCO L,POGGIO T.Holographic embed-dings of knowledge graphs[C]//AAAI Conference on Artificial Intelligence.2016:1955-1961.
[7]TROUILLON T,WELBL J,RIEDEL S,et al.Complex embeddings for simple link prediction[C]//International Conference on Machine Learning.PMLR,2016:2071-2080.
[8]TROUILLON T,NICKEL M.Complex and Holographic Em-beddings of Knowledge Graphs:A Comparison[EB/OL].(2017-07-23)[2021-12-13].https://arxiv.org/abs/1707.01475.
[9]BORDES A,GLOROT X,WESTON J,et al.Joint learning ofwords and meaning representations for open-text semantic parsing[C]//Artificial Intelligence and Statistics.PMLR,2012:127-135.
[10]BORDES A,WESTON J,COLLOBERT R,et al.Learningstructured embeddings of knowledge bases[C]//AAAI Confe-rence on Artificial Intelligence.2011:301-306.
[11]GARCÍA-DURÁN A,BORDES A,USUNIER N.Effectiveblending of two and three-way interactions for modeling multi-relational data[C]//Joint European Conference on Machine Learning and Knowledge Discovery in Databases.2014:434-449.
[12]YANG B,YIH W,HE X,et al.Embedding entities and relations for learning and inference in knowledge bases[C]//International Conference on Learning Representations(ICLR).2015:1-13.
[13]FENG J,HUANG M,WANG M,et al.Knowledge graph em-bedding by flexible translation[C]//International Conference on Principles of Knowledge Representation and Reasoning.2016:557-560.
[14]LIN Y,LIU Z,SUN M,et al.Learning entity and relation embeddings for knowledge graph completion[C]//AAAI Confe-rence on Artificial Intelligence.2015:2181-2187.
[15]XIE R,LIU Z,SUN M.Representation learning of knowledgegraphs with hierarchical types[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence.2016:2965-2971.
[16]XIE R,LIU Z,JIA J,et al.Representation learning of knowledge graphs with entity descriptions[C]//AAAI Conference on Artificial Intelligence.2016:2659-2665.
[17]XIE R,LIU Z,LUAN H,et al.Image-embodied knowledge representation learning[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence.2017:3140-3146.
[18]DASGUPTA S,RAY S N,TALUKDAR P.Hyte:Hyperplanebased temporally aware knowledge graph embedding[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing(EMNLP).2018:2001-2011.
[19]JI G,HE S,XU L,et al.Knowledge graph embedding via dy-namic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.2015:687-696.
[20]JI G,LIU K,HE S,et al.Knowledge graph completion withadaptive sparse transfer matrix[C]//AAAI Conference on Artificial Intelligence.2016:985-991.
[21]XIE Q,MA X,DAI Z,et al.An Interpretable Knowledge Transfer Model for Knowledge Base Completion[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.2017:950-962.
[22]XIAO H,HUANG M,ZHU X.From one point to a manifold:knowledge graph embedding for precise link prediction[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence.2016:1315-1321.
[23]HE S,LIU K,JI G,et al.Learning to represent knowledgegraphs with gaussian embedding[C]//Proceedings of the 24th ACM International on Conference on Information and Know-ledge Management.2015:623-632.
[24]NGUYEN D Q,SIRTS K,QU L,et al.STransE:a novel embedding model of entities and relationships in knowledge bases[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.2016:460-466.
[25]LIN Y,LIU Z,LUAN H,et al.Modeling Relation Paths forRepresentation Learning of Knowledge Bases[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.2015:705-714.
[26]FAN M,ZHOU Q,CHANG E,et al.Transition-based know-ledge graph embedding with relational mapping properties[C]//Proceedings of the 28th Pacific AsiaConference on Language,Information and Computing.2014:328-337.
[27]EBISU T,ICHISE R.Toruse:Knowledge graph embedding on a lie group[C]//AAAI Conference on Artificial Intelligence.2018:1819-1826.
[28]LEBLAYJ,CHEKOL M W.Deriving validity time in knowledge graph[C]//Proceedings of the 26th International World Wide Web Conference.2018:1771-1776.
[29]XIAO H,HUANG M,HAO Y,et al.TransA:An adaptive approach for knowledge graph embedding[EB/OL].(2015-09-28)[2021-12-13].https://arxiv.org/abs/1509.05490.
[30]QIAN W,FU C,ZHU Y,et al.Translating embeddings forknowledge graph completion with relation attention mechanism[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence.2018:4286-4292.
[31]SOCHER R,CHEN D,MANNING C D,et al.Reasoning with neural tensor networks for knowledge base completion[C]//International Conference on Neural Information Processing Systems-Volume 1.2013:926-934.
[32]CHAMI I,WOLF A,JUAN D C,et al.Low-Dimensional Hy-perbolic Knowledge Graph Embeddings[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.2020:6901-6914.
[33]VU T,NGUYEN T D,NGUYEN D Q,et al.A capsule network-based embedding model for knowledge graph completion and search personalization[C]//Proceedings of the 2019 Confe-rence of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Vo-lume 1(Long and Short Papers).2019:2180-2189.
[34]LIU H,WU Y,YANG Y.Analogical inference for multi-rela-tional embeddings[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70.2017:2168-2178.
[35]KAZEMI S M,POOLE D.SimplE embedding for link prediction in knowledge graphs[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems.2018:4289-4300.
[36]ZHANG S,TAY Y,YAO L,et al.Quaternion knowledge graph embeddings[EB/OL].(2019-10-31)[2021-12-13].https://arxiv.org/abs/1904.10281.
[37]NGUYEN T D,NGUYEN D Q,PHUNG D.A Novel Embed-ding Model for Knowledge Base Completion Based on Convolutional Neural Network[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 2(Short Papers).2018:327-333.
[38]XU C,LI R.Relation Embedding with Dihedral Group inKnowledge Graph[C]//Proceedings of the 57th Annual Mee-ting of the Association for Computational Linguistics.2019:263-272.
[39]BALAZEVIC I,ALLEN C,HOSPEDALES T.TuckER:Tensor Factorization for Knowledge Graph Completion[C]//Procee-dings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing(EMNLP-IJCNLP).2019:5188-5197.
[40]BALAZEVIC I,ALLEN C,HOSPEDALES T.Multi-relational poincaré graph embeddings[J].Advances in Neural Information Processing Systems,2019,32:4463-4473.
[41]DETTMERS T,MINERVINI P,STENETORP P,et al.Convolutional 2d knowledge graph embeddings[C]//AAAI Confe-rence on Artificial Intelligence.2018:1811-1818.
[42]SHANG C,TANG Y,HUANG J,et al.End-to-end structure-aware convolutional networks for knowledge base completion[C]//Conference on Artificial Intelligence.2019:3060-3067.
[43]BALAŽEVIĆ I,ALLEN C,HOSPEDALES T M.Hypernetwork knowledge graph embeddings[C]//International Conference on Artificial Neural Networks.Cham:Springer,2019:553-565.
[44]SCHLICHTKRULL M,KIPF T N,BLOEM P,et al.Modeling relational data with graph convolutional networks[C]//Euro-pean Semantic Web Conference.Cham:Springer,2018:593-607.
[45]JIANG X,WANG Q,WANG B.Adaptive convolution for multi-relational learning[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies,Volume 1(Long and Short Papers).2019:978-987.
[46]PEZESHKPOUR P,CHEN L,SINGH S.Embedding Multimodal Relational Data for Knowledge Base Completion[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.2018:3208-3218.
[47]ZHU C,CHEN M,FAN C,et al.Learning from History:Mode-ling Temporal Knowledge Graphs with Sequential Copy-Generation Networks[EB/OL].(2021-03-05)[2021-12-13].https://arxiv.org/abs/2012.08492.
[48]YAO L,MAO C,LUO Y.KG-BERT:BERT for knowledgegraph completion[EB/OL].(2019-09-11)[2021-12-13].https://arxiv.org/abs/1909.03193v2.
[49]NICKEL M,KIELA D.Poincaré embeddings for learning hierarchical representations[C]//Proceedings of the 31st Interna-tional Conference on Neural Information Processing Systems.2017:6341-6350.
[50]SHI B,WENINGER T.ProjE:Embedding projection for know-ledge graph completion[C]//AAAI Conference on Artificial Intelligence.2017:1236-1242.
[51]XUE Y,YUAN Y,XU Z,et al.Expanding holographic embeddings for knowledge completion[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems.2018:4496-4506.
[52]LIN Y,LIU Z,SUN M.Knowledge representation learning with entities,attributes and relations[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence.2016:2866-2872.
[53]GARCIA-DURAN A,NIEPERT M.Kblrn:End-to-end learning of knowledge base representations with latent,relational,and numerical features[EB/OL].(2018-06-11)[2021-12-13].https://arxiv.org/abs/1709.04676.
[54]QU M,TANG J.Probabilistic logic neural networks for reaso-ning[EB/OL].(2019-10-29)[2021-12-13].https://arxiv.org/abs/1906.08495.
[55]XIAO H,HUANG M,ZHU X.TransG:A Generative Model for Knowledge Graph Embedding[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers).2016:2316-2325.
[56]ZHANG W,PAUDEL B,ZHANG W,et al.Interaction embeddings for prediction and explanation in knowledge graphs[C]//Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining.2019:96-104.
[57]TRIVEDI R,FARAJTABAR M,WANG Y,et al.Know-Evolve:Deep Reasoning in Temporal Knowledge Graphs[EB/OL].(2017-06-21)[2021-12-13].https://arxiv.org/abs/1705.05742v1.
[58]SADEGHIAN A,ARMANDPOUR M,COLAS A,et al.Chro-noR:Rotation Based Temporal Knowledge Graph Embedding[EB/OL].(2021-03-18)[2021-12-13].https://arxiv.org/abs/2103.10379.
[59]JIN W,QU M,JIN X,et al.Recurrent Event Network:Auto regressive Structure Inference over Temporal Knowledge Graphs[EB/OL].(2020-10-06)[2021-12-13].https://arxiv.org/abs/1904.05530v4.
[60]JENATTON R,LE ROUX N,BORDES A,et al.A latent factor model for highly multi-relational data[C]//Advances in Neural Information Processing Systems 25(NIPS 2012).2012:3176-3184.
[61]LIU Q,JIANG H,EVDOKIMOV A,et al.Probabilistic reaso-ning via deep learning:Neural association models[EB/OL].(2016-08-03)[2021-12-13].https://arxiv.org/abs/1603.07704.
[62]GUTMANN M U,HYVÄRINEN A.Noise-contrastive estimation of unnormalized statistical models,with applications to na-tural image statistics[J].The Journal of Machine Learning Research,2012(13):307-361.
[63]ANDRIY M,YEE W T.A fast and simple algorithm for training neural probabilistic language models[EB/OL].(2012-06-27)[2021-12-13].https://arxiv.org/abs/1206.6426.
[64]MIKOLOV T,SUTSKEVER I,CHEN K,et al.Distributed representations of words and phrases and their compositionality[C]//Conference and Workshop on Neural Information Proces-sing Systems(NIPS).2013:3111-3119.
[65]GUO L,SUN Z,HU W.Learning to exploit long-term relational dependencies in knowledge graphs[C]//International Confe-rence on Machine Learning.PMLR,2019:2505-2514.
[66]SUN Z,DENG Z H,NIE J Y,et al.RotatE:Knowledge Graph Embedding by Relational Rotation in Complex Space[C]//International Conference on Learning Representations.2018:1-18.
[67]ZHANG Z,CAI J,ZHANG Y,et al.Learning hierarchy-aware knowledge graph embeddings for link prediction[C]//AAAI Conference on Artificial Intelligence.2020:3065-3072.
[68]CHANG K W,YIH W,YANG B,et al.Typed tensor decomposition of knowledge bases for relation extraction[C]//Procee-dings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP).2014:1568-1579.
[69]KAZEMI S M,GOEL R,JAIN K,et al.Representation Learning for Dynamic Graphs:A Survey[J].Journal of Machine Learning Research,2020(21):1-73.
[70]LIU Z Y,SUN M S,LIN Y K,et al.Knowledge Representation Learning:A Review[J].Journal of Computer Research and Development,2016,53(2):247-261.
[71]WANG Q,MAO Z,WANG B,et al.Knowledge graph embedding:A survey of approaches and applications[J].IEEE Tran-sactions on Knowledge and Data Engineering,2017,29(12):2724-2743.
[72]PAULHEIM H.Knowledge graph refinement:A survey of approaches and evaluation methods[J].Semantic Web,2017,8(3):489-508.
[73]CAI H,ZHENG V W,CHANG K C.A comprehensive survey of graph embedding:Problems,techniques,and applications[J].IEEE Transactions on Knowledge and Data Engineering,2018,30(9):1616-1637.
[74]LIN Y,HAN X,XIE R,et al.Knowledge representation lear-ning:A quantitative review[EB/OL].(2018-12-28)[2021-12-13].https://arxiv.org/abs/1812.10901.
[75]GESESE G A,BISWAS R,SACK H.A Comprehensive Survey of Knowledge Graph Embeddings with Literals:Techniques and Applications[C]//Proceedings of DL4KG2019-Workshop on Deep Learning for Knowledge Graphs.2019:131.
[76]CHEN X,JIA S,XIANG Y.A review:Knowledge reasoningover knowledge graph[J].Expert Systems with Applications,2020,141(5):1-21.
[77]TAY Y,TUAN L A,PHAN M C,et al.Multi-task neural network for non-discrete attribute prediction in knowledge graphs[C]//Proceedings of the 2017 ACM on Conference on Information and Knowledge Management.2017:1029-1038.
[78]CHEN X,CHEN M,SHI W,et al.Embedding uncertain know-ledge graphs[C]//AAAI Conference on Artificial Intelligence.2019:3363-3370.
[79]ALI M,BERRENDORF M,HOYT C T,et al.Bringing Light Into the Dark:A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework[EB/OL].(2021-11-01)[2021-12-13].https://arxiv.org/abs/2006.13365.
[1] LI Shujing, HUANG Zengfeng. Mixed-curve for Link Completion of Multi-relational Heterogeneous Knowledge Graphs [J]. Computer Science, 2023, 50(4): 172-180.
[2] MA Tinghuai, SUN Shengjie, RONG Huan, QIAN Minfeng. Knowledge Graph-to-Text Model Based on Dynamic Memory and Two-layer Reconstruction Reinforcement [J]. Computer Science, 2023, 50(3): 12-22.
[3] WANG Jingbin, LAI Xiaolian, LIN Xinyu, YANG Xinyi. Context-aware Temporal Knowledge Graph Completion Based on Relation Constraints [J]. Computer Science, 2023, 50(3): 23-33.
[4] CHEN Fuqiang, KOU Jiamin, SU Limin, LI Ke. Multi-information Optimized Entity Alignment Model Based on Graph Neural Network [J]. Computer Science, 2023, 50(3): 34-41.
[5] LIU Xinwei, TAO Chuanqi. Method of Java Redundant Code Detection Based on Static Analysis and Knowledge Graph [J]. Computer Science, 2023, 50(3): 65-71.
[6] CHEN Shurui, LIANG Ziran, RAO Yanghui. Fine-grained Semantic Knowledge Graph Enhanced Chinese OOV Word Embedding Learning [J]. Computer Science, 2023, 50(3): 72-82.
[7] JIANG Chuanyu, HAN Xiangyu, YANG Wenrui, LYU Bohan, HUANG Xiaoou, XIE Xia, GU Yang. Survey of Medical Knowledge Graph Research and Application [J]. Computer Science, 2023, 50(3): 83-93.
[8] LI Zhifei, ZHAO Yue, ZHANG Yan. Survey of Knowledge Graph Reasoning Based on Representation Learning [J]. Computer Science, 2023, 50(3): 94-113.
[9] WANG Xiaofei, FAN Xueqiang, LI Zhangwei. Improving RNA Base Interactions Prediction Based on Transfer Learning and Multi-view Feature Fusion [J]. Computer Science, 2023, 50(3): 164-172.
[10] LIU Zejing, WU Nan, HUANG Fuqun, SONG You. Hybrid Programming Task Recommendation Model Based on Knowledge Graph and Collaborative Filtering for Online Judge [J]. Computer Science, 2023, 50(2): 106-114.
[11] LI Junlin, OUYANG Zhi, DU Nisuo. Scene Text Detection with Improved Region Proposal Network [J]. Computer Science, 2023, 50(2): 201-208.
[12] SHAN Zhongyuan, YANG Kai, ZHAO Junfeng, WANG Yasha, XU Yongxin. Ontology-Schema Mapping Based Incremental Entity Model Construction and Evolution Approach of Knowledge Graph [J]. Computer Science, 2023, 50(1): 18-24.
[13] RONG Huan, QIAN Minfeng, MA Tinghuai, SUN Shengjie. Novel Class Reasoning Model Towards Covered Area in Given Image Based on InformedKnowledge Graph Reasoning and Multi-agent Collaboration [J]. Computer Science, 2023, 50(1): 243-252.
[14] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[15] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.