Editorial: Geometries of learning

Editorial on the Research Topic
Geometries of learning

Despite the widespread application and empirical success of deep neural networks, the theoretical development of these networks remains an under-explored area. The lack of theory prevents rigorous explanation and prediction of networks' performance and hinders confident deployment of the networks especially in safety-critical applications. Geometric analysis provides a unique perspective to bridge this gap between the practice and the theory of deep neural networks. As complex real-world data often lie on low-dimensional manifolds, geometries of these underlying structures provide informative insights for understanding the behavior of networks fitting to the data.

In this Research Topic of Frontiers in Computer Science on Geometries of Learning, we aim to present the latest advances on inspecting deep learning through the lens of geometry. We have followed a rigorous peer review process and collected 12 articles that showcase the depth and breadth of current approaches across applications of geometric analysis across diverse problem domains from adversarial defense to object pose estimation. A summary of the collection is introduced below.

In the article titled “Leveraging linear mapping for model-agnostic adversarial defense” Jamil et al. shed light on generalization of adversarial defense by demonstrating the existence of a linear mapping between adversarial image representations across different deep neural networks. Based on this insight, the authors further demonstrate a model-agnostic adversarial defense strategy by mapping adversarial representations from diverse models into a canonical space where adversarial representation can be identified accurately using a simple linear classifier without re-training.

In the article titled “Probabilistic and semantic descriptions of image manifolds and their applications”, Tu et al. model image manifolds as probability density functions using advanced deep generative models including normalizing flows and diffusion models. The authors further introduce a model based on the variational auto encoder to demonstrate semantic disentanglement on the image manifold and demonstrate reliable density estimation by enforcing semantic consistency as an application in adversarial robustness against patch attacks.

In the article titled “Integrating geometries of ReLU feedforward neural networks”, Liu Y. et al. introduce a toolbox that computes geometric properties of ReLU feedforward neural networks for characterizing networks' partition of the input space. The geometric analysis framework based on polyhedral decomposition lays a foundation for comprehensive analysis of network geometries and their implications on model behavior.

In the article titled “Locally linear attributes of ReLU neural networks”, Sattelberg et al. investigate ReLU neural networks which partitions the input space into convex polytopes. By studying the evolution of the polytope structure and affine transformations associated, the authors present insights on how the number of the polytopes can be reduced and how similar structures observed across different networks, suggesting improved network design for reducing complexity and enhancing generalization.

In the article titled “An algorithm for computing Schubert varieties of best fit with applications”, Karimov et al. present a geometric tool based on Schubert variety to present a collection of subspaces of a fixed vector space. The authors further introduce integration of the representation as a mathematically interpretable abstract node for artificial neural networks. The proposed algorithms demonstrated on classification problems suggest the existence of a best dimension for the representative subspace, as exceeding the dimension leads to no improvement in accuracy.

In the article titled “Exploring fMRI RDMs: enhancing model robustness through neurobiological data”, Pickard et al. introduce a large public benchmark consisting biological representations for classic vision datasets. With curated data, metrics and analysis based on representational dissimilarity matrices and Fiedler partitioning, the authors demonstrate the use of the benchmark to facilitate research on inspecting the alignment between neural network representations and neurobiological representations from fMRI and how it relates to network performance and robustness.

In the article titled “Manifold-driven decomposition for adversarial robustness”, Zhang et al. investigate the adversarial risk and robustness-accuracy trade-off of machine learning models from the manifold perspective. Considering decomposing adversarial risk into normal adversarial risk and in-manifold adversarial risk, the authors present theoretical findings on bounding the adversarial risks as well as empirical validation of the findings on synthetic and real-world datasets.

In the article titled “On-manifold projected gradient descent”, Mahler et al., present a novel solution for generating on-manifold adversarial data, by leveraging mathematical rigorous tools to approximate the data manifold and its tangent directions for sample perturbation. The perturbed samples are further projected back to the data manifold, resulting in adversarial samples which effectively confuse trained classifiers. The misclassification can be further explained based on the semantic basis of the manifold.

In the article titled “Leveraging diffusion models for unsupervised out-of-distribution detection on image manifold”, Liu Z. et al. propose a novel solution for unsupervised out-of-distribution detection by performing image inpainting with in domain diffusion models. The diffusion model serves as a mapping to its training manifold and the distance between mapped image and the original image serves an indicative metric for out-of-distribution detection, as supported by empirical experiments across images with diverse characteristics.

In the article titled “Orthogonality and graph divergence losses promote disentanglement in generative models”, Shukla et al. improve deep generative models by integrating an architecture promoting separation of latent space, an orthonormality constraint modeling statistical independence of latent attributes and a differentiable graph divergence loss promoting manifold preservation. The proposed solution achieves disentangled representations and controlled generation as demonstrated in experiments on 3D shape datasets.

In the article titled “Implications of data topology for deep generative models”, Jin et al. study the ability of various deep generative models to model complex data topologies. With experiments using synthetic data, the authors demonstrate the limitations of models with normal assumptions on latent distribution and demonstrate improved abilities of recent models including chart auto encoders and denoising diffusion probabilistic models. The work further identifies challenges including the limitations of distribution-based metrics for assessing deep generative models with respect to capturing underlying data topologies.

In the article titled “Recovering manifold representations via unsupervised meta-learning”, Gong et al. address the challenge of learning complex data manifold without uniformly or densely sampled data by leveraging novel episodic sampling strategies to improve auto encoders' generalization. The authors adopt topological and geometric metrics based on persistent homology to demonstrate the quantitative improvement on manifold reconstruction on 6-D object pose estimation benchmarks.

In summary, the selected papers highlight the cutting-edge developments in theories, methods, tools and datasets for geometric understanding of deep learning across different applications. Based on the insightful findings from the collection, we anticipate promising future research in multiple directions such as scaling up geometric analysis for larger network architectures including transformers which serve the core of recent developments in language modeling, promoting geometric and topological interpretation of learning with direct links to semantics, and expanding the study for a wide range of applications involving diverse data manifolds at real-world complexities.

The guest editor team would like to extend our sincere gratitude to all the authors, reviewers and the editorial team for their valuable contribution to this Research Topic. Our special thanks go to Dr. Bruce Draper for his vision and leadership in the research and discussions that laid the foundation of this Research Topic as well as his guidance and support throughout the editing process.

Author contributions

YG: Writing – review & editing, Writing – original draft. PT: Writing – original draft, Writing – review & editing. PF: Writing – original draft, Writing – review & editing. AD: Writing – review & editing, Writing – original draft.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.