Abstract: The working mechanism of deep neural networks can be intuitively uncovered by visualization technique.Visualizing deep neural networks can provide the interpretability on the decision made by the black box model,which is critically important in many fields,such as medical diagnosis and autopilot.Current existing works are mostly based on the activation maximization technique,which optimizes the input,the hidden feature map or the original image,in condition to the neuron that we want to observe.Qualitatively,the change in the input value can be taken as explanation when the neuron has reached nearly the maximum activation value.However,such method lacks the quantitative analysis of deep neural networks.To fill this gap,this paper proposes two meta methods,namely,structure visualization and rule-based visualization.Structure visualization works by visualizing from the shallow layers to the deep layers,and find that neurons in shallow layers learn global characteristics while neurons in deep layers learn more specific features.The rule-based visualization includes intersection and difference selection rule,and it is helpful to find the existence of shared neurons and inhibition neurons that learns the common features of different categories and inhibits unrelated features respectively.Experiments on two representative deep networks,namely the convolutional network VGG and the residual network ResNet,by using ImageNet and COCO datasets.Quantitative analysis shows that ResNet and VGG are highly sparse in representation.Thus,by removing some low activation-value “noisy” neurons,the networks can keep or even improve the classification accuracy.This paper discovers the Latent representation of deep neural networks by visualizing and quantitatively analyzing hidden features,thus providing guidance and reference for the design of high-performance deep neural networks.

Key words: Deep neural network, Feature visualization, Inhibition neuron, Internal representation, Shared neuron