AUTHOR=Berthiaume David , Paffenroth Randy , Guo Lei TITLE=Understanding Deep Learning: Expected Spanning Dimension and Controlling the Flexibility of Neural Networks JOURNAL=Frontiers in Applied Mathematics and Statistics VOLUME=6 YEAR=2020 URL=https://www.frontiersin.org/journals/applied-mathematics-and-statistics/articles/10.3389/fams.2020.572539 DOI=10.3389/fams.2020.572539 ISSN=2297-4687 ABSTRACT=

Neural networks (NN) provide state-of-the-art performance in many problem domains. They can accommodate a vast number of parameters and still perform well when classic machine learning techniques provided with the same number of parameters would tend to overfit. To further the understanding of such incongruities, we develop a metric called the expected spanning dimension (ESD) which allows one to measure the intrinsic flexibility of an NN. We analyze NNs from the small, in which the ESD can be exactly computed, to large real-world networks with millions of parameters, in which we demonstrate how the ESD can be numerically approximated efficiently. The small NNs we study can be understood in detail, their ESD can be computed analytically, and they provide opportunities for understanding their performance from a theoretical perspective. On the other hand, applying the ESD to large-scale NNs sheds light on their relative generalization performances and provides suggestions as to how such NNs may be improved.