Skeletal muscle represents the largest body-composition component in humans. In addition to its primary function in the maintenance of upright posture and the production of movement, it also plays important roles in many other physiological processes, including thermogenesis, metabolism and the secretion of peptides for communication with other tissues. Research attempting to unveil these processes has traditionally focused on muscle fibers, i.e., the contractile muscle cells. However, it is a frequently overlooked fact that muscle fibers reside in a three-dimensional scaffolding that consists of various collagens, glycoproteins, proteoglycans, and elastin, and is commonly referred to as extracellular matrix (ECM). While initially believed to be relatively inert, current research reveals the involvement of ECM cells in numerous important physiological processes. In interaction with other cells, such as fibroblasts or cells of the immune system, the ECM regulates muscle development, growth and repair and is essential for effective muscle contraction and force transmission. Since muscle ECM is highly malleable, its texture and, consequently, physiological roles may be affected by physical training and disuse, aging or various diseases, such as diabetes. With the aim to stimulate increased efforts to study this still poorly understood tissue, this narrative review summarizes the current body of knowledge on (i) the composition and structure of the ECM, (ii) molecular pathways involved in ECM remodeling, (iii) the physiological roles of muscle ECM, (iv) dysregulations of ECM with aging and disease as well as (v) the adaptations of muscle ECM to training and disuse.