Layer wise Scaled Gaussian Priors for Markov Chain Monte Carlo sampling (MCMC) Deep Bayesian Neural Networks

Devesh Jawla ^1* John Kelleher ²

• ¹ Technological University Dublin, Dublin, Ireland
• ² Trinity College Dublin, Dublin, County Dublin, Ireland

Previous work has demonstrated that initialization is very important for both fitting a neural network by gradient descent methods, as well as for Variational inference of Bayesian Neural networks. In this work we investigate how Layer wise Scaled Gaussian Priors perform with Markov chain Monte Carlo trained Bayesian Neural networks. From our experiments on 8 classifications datasets of various complexity, the results indicate that using Layer wise Scaled Gaussian Priors makes the sampling process more efficient as compared to using an Isotropic Gaussian Prior, an Isotropic Cauchy Prior, or an Isotropic Laplace Prior. We also show that the cold posterior effect does not arise when using a either an Isotropic Gaussian or a layer wise Scaled Prior for small feed forward Bayesian Neural Networks. Since Bayesian Neural networks are becoming popular due to their advantages such as uncertainty estimation, and prevention of over-fitting, this work seeks to provide improvements in the efficiency of Bayesian Neural Networks learned using Markov Chain Monte Carlo methods.