基于全局属性注意力神经过程模型的数据补全研究

doi:10.11896/jsjkx.210800038

计算机科学 ›› 2022, Vol. 49 ›› Issue (10): 111-117.doi: 10.11896/jsjkx.210800038

• 数据库＆大数据＆数据科学 • 上一篇下一篇

基于全局属性注意力神经过程模型的数据补全研究

程恺¹, 刘满^1,2, 王之腾¹, 毛绍臣¹, 申秋慧¹, 张宏军¹

1 中国人民解放军陆军工程大学指挥控制工程学院南京 210007
2 中国人民解放军73131部队福建漳州 363000

收稿日期:2021-08-03 修回日期:2022-01-13 出版日期:2022-10-15 发布日期:2022-10-13
通讯作者: 张宏军(jsnjzhj_lgdx@163.com)
作者简介:(chengkai911@126.com)
基金资助:
国家自然科学基金(61806221)

Study on Data Filling Based on Global-attributes Attention Neural Process Model

CHEN Kai¹, LIU Man^1,2, WANG Zhi-teng¹, MAO Shao-chen¹, SHEN Qiu-hui¹, ZHANG Hong-jun¹

1 Institute of Command and Control Engineering,Army Engineering University of PLA,Nanjing 210007,China
2 73131 Unit of PLA,Zhangzhou,Fujian 363000,China

Received:2021-08-03 Revised:2022-01-13 Online:2022-10-15 Published:2022-10-13
About author:CHENG Kai,born in 1983,Ph.D,asso-ciate professor.His main research in-terests include data analysis and data mining,artificial intelligence and planning techniques.
ZHANG Hong-jun,born in 1963,Ph.D,professor,Ph.D supervisor.His main research interests include data & knowledge engineering and computer simulation theory.
Supported by:
National Natural Science Foundation of China(61806221).

摘要/Abstract

摘要： 注意力神经过程(Attentive Neural Process,ANP)模型采用生成模型的方法,以样本的任意局部上下文点为输入,输出整个样本的分布函数,从而模仿高斯过程回归完成数据补全任务。样本的属性信息可以为样本的生成提供重要信息,然而ANP模型忽略了对属性信息的使用。受条件变分自动编码机(CVAE)模型以标签为条件控制样本生成的启发,文中提出了全局属性注意力神经过程(Global-attribute Attentive Neural Process,GANP),将样本属性嵌入到编码器网络,从而使浅层变量隐含样本属性信息。同时,在解码器网络中加入样本属性作为特征,使模型的生成样本更为准确,特别是当输入上下文点数量稀少时,属性信息能够帮助模型生成更清晰、准确的样本。最后,从定性和定量两个方面证明了GANP性能的优越性,可以看出该模型扩展了NP家族模型的应用范围,从而更灵活、快速、准确地解决只有部分上下文信息时整个样本的数据补全问题。

关键词: 神经过程, 交叉注意力, 变分推断, 高斯过程, 全局属性

Abstract: The attention neural process(ANP) model which adopts the method of generative model,takes any number context points of the sample as input,and outputs the distribution function of the entire sample,so as to approximate the function of Gaussian process regression(GPR) to realize the data fullfilling task.In reality,many scenes or datasets containe the attributes or labels data which are critical for generating the missing data.However,the ANP ignores full use of them.Inspired by CVAE model which control sample generation with lable as condition,this paper proposes global attribute attentional neural process(GANP),which embeds sample attributes or labels into ANP network to make the model generate samples more accurately,especially when the number of input context points are scarce.In detail,the sample attributes are embedded into the encoder network,so that the latent variables contain the sample attribute information.At the same time,the sample attributes are added as features in the decoder network to help generate more accurate samples.Finally,experimental results prove the superiority of GANP in both qualitative and quantitative,and it also reveals that GANP expands the application of NP families which can solve the Gaus-sian process regression problem more flexibly,quickly and accurately.

Key words: Neural process, Cross attention, Variational inference, Gaussian process, Global attribute

中图分类号:

TP391.4

程恺, 刘满, 王之腾, 毛绍臣, 申秋慧, 张宏军. 基于全局属性注意力神经过程模型的数据补全研究[J]. 计算机科学, 2022, 49(10): 111-117. https://doi.org/10.11896/jsjkx.210800038

CHEN Kai, LIU Man, WANG Zhi-teng, MAO Shao-chen, SHEN Qiu-hui, ZHANG Hong-jun. Study on Data Filling Based on Global-attributes Attention Neural Process Model[J]. Computer Science, 2022, 49(10): 111-117. https://doi.org/10.11896/jsjkx.210800038

参考文献

[1]RASMUSSEN C E.Gaussian processes in machine learning[C]//Summer School on Machine Learning.Berlin:Springer,2003:63-71.
[2]QUINONERO-CANDELA J,RASMUSSEN C E.A unifyingview of sparse approximate Gaussian process regression[J].The Journal of Machine Learning Research,2005,6(65):1939-1959.
[3]GARNELO M,SCHWARZ J,ROSENBAUM D,et al.Neuralprocesses[J].arXiv:1807.01622,2018.
[4]WANG J X,KURTH-NELSON Z,TIRUMALA D,et al.Lear-ning to reinforcement learn[J].arXiv:1611.05763,2016.
[5]FINN C,ABBEEL P,LEVINE S.Model-agnostic meta-learning for fast adaptation of deep networks[C]//International Confe-rence on Machine Learning.PMLR,2017:1126-1135.
[6]KIM H,MNIH A,SCHWARZ J,et al.Attentive neural processes[J].arXiv:1901.05761,2019.
[7]GORDON J,BRUINSMA W P,FOONG A Y K,et al.Convolutional conditional neural processes[J].arXiv:1910.13556,2019.
[8]SINGH G,YOON J,SON Y,et al.Sequential neural processes[J].arXiv:1906.10264,2019.
[9]KUMAR S.Spatiotemporal Modeling using Recurrent NeuralProcesses[D].Pittsburgh,PA:Carnegie Mellon University,2019.
[10]ZHU J,QIN S,WANG W,et al.Probabilistic trajectory prediction for autonomous vehicles with attentive recurrent neural process[J].arXiv:1910.08102,2019.
[11]KUMAR A,ESLAMI S M,REZENDE D J,et al.Consistentgenerative query networks[J].arXiv:1807.02033,2018.
[12]MA Y,WANG S Q,MAO Y X.Path Planning of Mobile Robot Based on Neural Process-particle Swarm Optimization[J].Journal of Hubei University of Technology,2020,35(1):17-20.
[13]SUN X L,GUO Y,LI N,et al.Missing data imputing algorithm based on modified neural process[J].Journal of University of Chinese Academy of Sciences,2021,38(2):280-287.
[14]LECUN Y,BOTTOU L,BENGIO Y,et al.Gradient-based lear-ning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324.
[15]LIU Z,LUO P,WANG X,et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision.2015:3730-3738.
[16]BLUNDELL C,CORNEBISE J,KAVUKCUOGLU K,et al.Weight uncertainty in neural network[C]//International Conference on Machine Learning.PMLR,2015:1613-1622.
[17]GARNELO M,ROSENBAUM D,MADDISON C,et al.Conditional neural processes[C]//International Conference on Machine Learning.PMLR,2018:1704-1713.
[18]KINGMA D P,WELLING M.Auto-encoding variational bayes[J].arXiv:1312.6114,2013.
[19]DOERSCH C.Tutorial on variational autoencoders[J].arXiv:1606.05908,2016.
[20]SOHN K,LEE H,YAN X.Learning structured output repre-sentation using deep conditional generative models[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.2015:3483-3491.
[21]VASWANI A,SHAZEER N,PARMAR N,et al.Attention isall you need[C]//Advances in Neural Information Processing Systems.2017:5998-6008.
[22]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.

相关文章 10

[1]	杨玥, 冯涛, 梁虹, 杨扬. 融合交叉注意力机制的图像任意风格迁移 Image Arbitrary Style Transfer via Criss-cross Attention 计算机科学, 2022, 49(6A): 345-352. https://doi.org/10.11896/jsjkx.210700236
[2]	谈馨悦, 何小海, 王正勇, 罗晓东, 卿粼波. 基于Transformer交叉注意力的文本生成图像技术 Text-to-Image Generation Technology Based on Transformer Cross Attention 计算机科学, 2022, 49(2): 107-115. https://doi.org/10.11896/jsjkx.210600085
[3]	楼浩锋, 张端. 高斯过程下的CMA-ES在医学图像配准中的应用 Gaussian Process Assisted CMA-ES Application in Medical Image Registration 计算机科学, 2018, 45(11A): 234-237.
[4]	黄熠,王娟. PSO-GP中文文本情感分类方法研究 Research on Chinese Texts Sentiment Classification Approach Based on PSO-GP 计算机科学, 2017, 44(Z6): 446-450. https://doi.org/10.11896/j.issn.1002-137X.2017.6A.100
[5]	惠景丽,潘巍,吴康康,周晓英. 基于非对称变邻域粗糙集模型的属性约简 Attribute Reduction Based on Asymmetric Variable Neighborhood Rough Set 计算机科学, 2015, 42(6): 282-287. https://doi.org/10.11896/j.issn.1002-137X.2015.06.059
[6]	杜占玮杨永健肖敏白媛. 一种基于自适应高斯过程的基线计算算法 Baseline Algorithm Based on Adaptive Gaussian Process Machine Learning 计算机科学, 2012, 39(11): 79-82.
[7]	刘长红,杨扬,陈勇. 增量式人体姿态映射模型的学习方法 Incrementally Learning Human Pose Mapping Model 计算机科学, 2010, 37(3): 268-270.
[8]	田江，顾宏. 一种孤立点挖掘的混合核方法 Hybrid Method for Outliers Detection Using GPLVM and SVM 计算机科学, 2010, 37(3): 245-247.
[9]	周文云,刘全,李志涛. 一种大规模离散空间中的高斯强化学习方法 Gaussian Processes Reinforcement Learning Method in Large Discrete States Space 计算机科学, 2009, 36(8): 247-249.
[10]	王秀美高新波. 基于判别特征加权的GPLVM算法计算机科学, 2009, 36(3): 189-192.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于全局属性注意力神经过程模型的数据补全研究

Study on Data Filling Based on Global-attributes Attention Neural Process Model

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 10

Metrics

本文评价

推荐阅读 0