计算机科学 ›› 2019, Vol. 46 ›› Issue (7): 206-210.doi: 10.11896/j.issn.1002-137X.2019.07.031

• 人工智能 • 上一篇    下一篇

具有生物真实性的强抗噪性神经元激活函数

麦应潮,陈云华,张灵   

  1. (广东工业大学计算机学院 广州510006)
  • 收稿日期:2018-05-22 出版日期:2019-07-15 发布日期:2019-07-15
  • 作者简介:麦应潮(1994-),男,硕士,主要研究方向为神经形态计算、机器学习;陈云华(1977-),女,博士,副教授,CCF会员,主要研究方向为深度学习、神经形态计算,E-mail:yhchen@gdut.edu.cn(通信作者);张 灵(1968-),女,博士,教授,主要研究方向为模式识别、智能化信息处理、人工智能等。
  • 基金资助:
    广东省自然科学基金(2016A030313713),广东省交通运输厅科技项目(科技-2016-02-030),广东省科技计划项目(2013B040500008)资助

Bio-inspired Activation Function with Strong Anti-noise Ability

MAI Ying-chao,CHEN Yun-hua,ZHANG Ling   

  1. (School of Computers,Guangdong University of Technology,Guangzhou 5110006,China)
  • Received:2018-05-22 Online:2019-07-15 Published:2019-07-15

摘要: 当前人工神经网络虽然在图像识别等方面媲美人脑,但因其所采用的激活函数ReLU和Softplus等只是对生物神经元输出响应特性的高度简化与模拟,使其在抗噪性、不确定性信息处理及功耗等方面与人脑仍存在巨大差距。通过分析生物神经元仿真实验,以其响应特性为基础,引入反映每个神经元随机性的参数η,构建出一种具有生物真实性的强抗噪性激活函数Rand Softplus。最后将该激活函数应用于深度残差网络,并基于人脸表情数据集对其进行验证。结果表明,在输入无噪声或具有少量噪声时,文中提出的激活函数与当前主流激活函数的识别精度基本持平,当输入包含较大噪声时,文中所提激活函数的识别精度远高于其他激活函数,表现出了良好的抗噪性能。

关键词: LIF模型, 激活函数, 抗噪性, 神经网络

Abstract: Although the artificial neural network is almost comparable to the human brain in image recognition,the activation functions such as ReLU and Softplus are only highly simplified and simulated for the output response characte-ristics of biological neurons.There is still a huge gap between the artificial neural network and the human brain in many aspects,such as noise resistance,uncertainty information processing and power consumption.In this paper,based on the simulation experiments of biological neurons and their response characteristics,a strong anti-noise activation function Rand Softplus with biological authenticity was constructed by defining and calculating parameters η,which reflects the randomness of each neuron.Finally,the activation function was applied to the depth residuals network and verified by facial expression dataset.The results show that the recognition accuracy of the activation function proposed in this paper is almost equal to the current mainstream activation function when there is no noise or a small amount of noise,and when the input contains a large amount of noise,it shows good anti-noise performance.

Key words: Activation function, Anti-noise, Leaky integrate-and-fire model, Neural networks

中图分类号: 

  • TP183
[1]LECUN Y,BENGIO Y,HINTON G.Deep learning [J].Nature,2015,521(7553):436.
[2]SCHMIDHUBER J.Deep learning in neural networks:An overview[J].Neural Networks,2015,61:85-117.
[3]BADJATIYA P,GUPTA S,GUPTA M,et al.Deep Learning for Hate Speech Detection in Tweets[C]∥Proceedings of the 26th International Conference on World Wide Web Companion.Perth:International World Wide Web Conferences Steering Committee,2017:759-760.
[4]KIM B K,LEE H,ROH J,et al.Hierarchical Committee of Deep CNNs with Exponentially-Weighted Decision Fusion for Static Facial Expression Recognition[C]∥Proceedings of the 2015 ACM on International Conference on Multimodal Interaction.Seattle:ACM,2015:427-434.
[5]HODGKIN A L,HUXLEY A F.A quantitative description of membrane current and its application to conduction and excitation in nerve[J].The Journal of Physiology,1952,117(4):500-544.
[6]STEIN R B.A Theoretical analysis of neuronal variability[J].Biophysical Journal,1965,5(2):173-194.
[7]IZHIKEVICH E M.Which model to use for cortical spiking neurons[J].IEEE Transactions on Neural Networks,2004,15(5):1063-1070.
[8]ACHARD P,DE SCHUTTER E.Complex parameter landscape for a complex neuron model[J].PLOS Computational Biology,2006,2(7):794-804.
[9]GLOROT X,BENGIO Y.Understanding the difficulty of trai- ning deep feedforward neural networks[C]∥Proceedings of the thirteenth International Conferenceon Artificial Intelligenceand Statistics.Sardinia:PMLR,2010:249-256.
[10]GLOROT X,BORDES A,BENGIO Y.Deep sparse rectifier neural networks[C]∥Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics.Fort Lauderdale:PMLR,2011:315-323.
[11]ERHAN D,BENGIO Y,COURVILLE A,et al.Why does unsupervised pre-training help deep learning?[J].Journal of Machine Learning Research,2010,11:625-660.
[12]TROTTIER L,GIGU P,CHAIB-DRAA B.Parametric expo- nential linear unit for deep convolutional neural networks[C]∥Machine Learning and Applications.Cancun:IEEE Press,2017:207-214.
[13]STROMATIAS E,NEIL D,PFEIFFER M,et al.Robustness of spiking deep belief networks to noise and reduced bit precision of neuro-inspired hardware platforms[J].Frontiers in neuroscience,2015,9:222.
[14]LIU Q,FURBER S.Noisy softplus:a biology inspired activation function[C]∥InInternational Conference on Neural Information Processing.Kyoto:Springer,2016:405-412.
[15]LA C G,GIUGLIANO M,SENN W,et al.The response of cortical neurons to in vivo-like input current:theory and experiment:I.Noisy inputs with stationary statistics[J].Biological Cybernetics,2008,99(5):279-301.
[16]GEWALTIG M O,DIESMANN M.Nest (neural simulation tool)[J].Scholarpedia,2007,2(4):1430.
[17]DAVISON A P,BRUDERLE D,EPPLER J M,et al.PyNN:a common interface for neuronal network simulators[J].Frontiers in Neuroinformatics,2009,2(11):204-123.
[18]LUCEY P,COHN J F,KANADE T,et al.The Extended Cohn-Kanade Dataset (CK+):A complete dataset for action unit and emotion-specified expression[C]∥Computer Vision and Pattern Recognition Workshops.San Francisco:IEEE Press 2010:94-101.
[19]CALVO M G,LUNDQVIS D.Facial expressions of emotion (KDEF):Identification under different display-duration conditions[J].Behavior Research Methods,2008,40(1):109-115.
[20]WHITEHILL J,MOVELLAN J R.A discriminative approach to frame-by-frame head pose tracking[C]∥IEEE International Conference on Automatic Face & Gesture Recognition.Amsterdam:IEEE Press,2008:1-7.
[21]HE K,ZHANG X,REN S,et al.Deep residual learning for ima- ge recognition[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,Boston:IEEE Press,2016:770-778.
[1] 周芳泉, 成卫青.
基于全局增强图神经网络的序列推荐
Sequence Recommendation Based on Global Enhanced Graph Neural Network
计算机科学, 2022, 49(9): 55-63. https://doi.org/10.11896/jsjkx.210700085
[2] 周乐员, 张剑华, 袁甜甜, 陈胜勇.
多层注意力机制融合的序列到序列中国连续手语识别和翻译
Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion
计算机科学, 2022, 49(9): 155-161. https://doi.org/10.11896/jsjkx.210800026
[3] 宁晗阳, 马苗, 杨波, 刘士昌.
密码学智能化研究进展与分析
Research Progress and Analysis on Intelligent Cryptology
计算机科学, 2022, 49(9): 288-296. https://doi.org/10.11896/jsjkx.220300053
[4] 李宗民, 张玉鹏, 刘玉杰, 李华.
基于可变形图卷积的点云表征学习
Deformable Graph Convolutional Networks Based Point Cloud Representation Learning
计算机科学, 2022, 49(8): 273-278. https://doi.org/10.11896/jsjkx.210900023
[5] 郝志荣, 陈龙, 黄嘉成.
面向文本分类的类别区分式通用对抗攻击方法
Class Discriminative Universal Adversarial Attack for Text Classification
计算机科学, 2022, 49(8): 323-329. https://doi.org/10.11896/jsjkx.220200077
[6] 王润安, 邹兆年.
基于物理操作级模型的查询执行时间预测方法
Query Performance Prediction Based on Physical Operation-level Models
计算机科学, 2022, 49(8): 49-55. https://doi.org/10.11896/jsjkx.210700074
[7] 陈泳全, 姜瑛.
基于卷积神经网络的APP用户行为分析方法
Analysis Method of APP User Behavior Based on Convolutional Neural Network
计算机科学, 2022, 49(8): 78-85. https://doi.org/10.11896/jsjkx.210700121
[8] 朱承璋, 黄嘉儿, 肖亚龙, 王晗, 邹北骥.
基于注意力机制的医学影像深度哈希检索算法
Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism
计算机科学, 2022, 49(8): 113-119. https://doi.org/10.11896/jsjkx.210700153
[9] 檀莹莹, 王俊丽, 张超波.
基于图卷积神经网络的文本分类方法研究综述
Review of Text Classification Methods Based on Graph Convolutional Network
计算机科学, 2022, 49(8): 205-216. https://doi.org/10.11896/jsjkx.210800064
[10] 闫佳丹, 贾彩燕.
基于双图神经网络信息融合的文本分类方法
Text Classification Method Based on Information Fusion of Dual-graph Neural Network
计算机科学, 2022, 49(8): 230-236. https://doi.org/10.11896/jsjkx.210600042
[11] 齐秀秀, 王佳昊, 李文雄, 周帆.
基于概率元学习的矩阵补全预测融合算法
Fusion Algorithm for Matrix Completion Prediction Based on Probabilistic Meta-learning
计算机科学, 2022, 49(7): 18-24. https://doi.org/10.11896/jsjkx.210600126
[12] 杨炳新, 郭艳蓉, 郝世杰, 洪日昌.
基于数据增广和模型集成策略的图神经网络在抑郁症识别上的应用
Application of Graph Neural Network Based on Data Augmentation and Model Ensemble in Depression Recognition
计算机科学, 2022, 49(7): 57-63. https://doi.org/10.11896/jsjkx.210800070
[13] 张颖涛, 张杰, 张睿, 张文强.
全局信息引导的真实图像风格迁移
Photorealistic Style Transfer Guided by Global Information
计算机科学, 2022, 49(7): 100-105. https://doi.org/10.11896/jsjkx.210600036
[14] 戴朝霞, 李锦欣, 张向东, 徐旭, 梅林, 张亮.
基于DNGAN的磁共振图像超分辨率重建算法
Super-resolution Reconstruction of MRI Based on DNGAN
计算机科学, 2022, 49(7): 113-119. https://doi.org/10.11896/jsjkx.210600105
[15] 刘月红, 牛少华, 神显豪.
基于卷积神经网络的虚拟现实视频帧内预测编码
Virtual Reality Video Intraframe Prediction Coding Based on Convolutional Neural Network
计算机科学, 2022, 49(7): 127-131. https://doi.org/10.11896/jsjkx.211100179
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!