Abstract: Top-k Sparsification Method with error compensation is one of the state-of-the-art technologies in the training of distributed deep neural networks(DNNs).This technique aims to reduce the amount of communication by dynamically transmitting only parts of the gradients in each iteration,with the amount of transmitted gradients depending on the value of k.Although a smaller k can speed up training time,it may degrade the test accuracy,even with error compensation,known as the speed-accuracy dilemma.Based on the observation that the increase speed of the training accuracy and test accuracy have a dynamic correlation over time,this paper presents AdaTopK－an adaptive Top-k compressor with convergence guarantees.AdaTopK can dynamically adjust the value of k to accelerate the training speed while keeping or enhancing the test accuracy.Extensive experiments in the static and dynamic network scenarios show that AdaTopK can reduce 29% training time over the baseline without compression,while reducing 15% training time over DC2.

Key words: Distributed training, Network compression, Sparsification, Deep neural networks, Error compensation