Alon Loeffler1
Ruomin Zhu1
Joel Hochstetter1
Mike Li2Kaiwei Fu1
Adrian Diaz-Alvarez3
Tomonobu Nakayama3
James M. Shine2
Zdenka Kuncic1*- 1School of Physics, The University of Sydney, Sydney, NSW, Australia
- 2Central Clinical School, The University of Sydney, Sydney, NSW, Australia
- 3International Center for Materials Nanoarchitectonics (WPI-MANA), National Institute for Materials Science (NIMS), Tsukuba, Japan
Graph theory has been extensively applied to the topological mapping of complex networks, ranging from social networks to biological systems. Graph theory has increasingly been applied to neuroscience as a method to explore the fundamental structural and functional properties of human neural networks. Here, we apply graph theory to a model of a novel neuromorphic system constructed from self-assembled nanowires, whose structure and function may mimic that of human neural networks. Simulations of neuromorphic nanowire networks allow us to directly examine their topology at the individual nanowire–node scale. This type of investigation is currently extremely difficult experimentally. We then apply network cartographic approaches to compare neuromorphic nanowire networks with: random networks (including an untrained artificial neural network); grid-like networks and the structural network of C. elegans. Our results demonstrate that neuromorphic nanowire networks exhibit a small–world architecture similar to the biological system of C. elegans, and significantly different from random and grid-like networks. Furthermore, neuromorphic nanowire networks appear more segregated and modular than random, grid-like and simple biological networks and more clustered than artificial neural networks. Given the inextricable link between structure and function in neural networks, these results may have important implications for mimicking cognitive functions in neuromorphic nanowire networks.
1. Introduction
1.1. Graph Theory Applications
Graph theory is a framework used to represent complex networks mathematically, whereby network components are represented as nodes (N) and connections between components are represented as edges (E) (Boccaletti et al., 2006). Since the 1950s, graph theory has been applied to networks in a variety of fields, including social networks (Harary and Norman, 1953), progression of disease (Eubank et al., 2004; Mason and Verwoerd, 2007), transport networks (Wakabayashi and Iida, 1992), the internet (Albert et al., 2011), and many others. Graph theory has largely been employed to study the structure of networks, known as structural connectivity. Measures such as the path length (PL), clustering coefficient (CCoeff), participant coefficient (PCoeff), within-module degree z-Score (MZ), degree and small worldness (see Box 1 for definitions), are useful characterizations of the structural properties of a network (Strogatz, 2001; Estrada and Hatano, 2008; Grayson et al., 2016). In many cases, analyzing the structure of a network is the first step to understanding its function (Strogatz, 2001).
Box 1. Graph Theory Terms
Clustering Coefficient (CCoeff): A measure of how much nodes in a graph tend to cluster together. This reflects the proportion of nodes connected to node N that are also connected to each other (Verweij et al., 2014).
Degree (DEG): The number of edges connected to a node, N.
Hubs: Areas through which large amounts of information flow to reach from one part of a network to another (Types of hubs and non-hub nodes are described in Figure 6).
Within-Module Degree z-Score (MZ): Measures how well connected a node is to other nodes in the same module (or cluster/community). This demonstrates whether the node is a hub in the network (i.e., much of the information flows through this node) (Guimerà and Amaral, 2005). Guimera and Nunes Amaral define MZ > 2.5 as hub-like nodes, and MZ < 2.5 as non-hub nodes.
Modularity: A measure of network segregation into distinct modules (or clusters/communities) that have sparse connections between each module (Cohen and D'Esposito, 2016).
Participant Coefficient (PCoeff): Measures how homogeneous a node's edges are distributed across modules (or clusters/communities). Nodes are divided into two classes: (1) High PCoeff: connector nodes with many global edges across modules (strong between-module and weak within-module connectivity Rubinov and Sporns, 2010; Cohen and D'Esposito, 2016); and (2) Low PCoeff: provincial/local nodes with mostly edges that connect nodes within a module (strong within-module and weak between-module connectivity) (Joyce et al., 2010; Van Diessen et al., 2014; Bertolero et al., 2015).
Path Length (PL): Measures the minimal number of edges of all possible node connections in a network (Van Diessen et al., 2014; Verweij et al., 2014).
Small–worldness: A type of network architecture in which local clustering is combined with short path length. This architecture offers important advantages for network functionality, ranging from synchronizability to information flow (Oliveira et al., 2014; Muldoon et al., 2016).
Small–world Propensity: Introduced by (Muldoon et al., 2016), used to account for potential variations in connection strength in a network, by measuring how clustering and path length differ from random and grid-like networks.
Graph theory measures have been applied to the study of biological networks, including the brain structure of organisms such as the neural networks of C. elegans (Achacoso and Yamamoto, 1991; Yan et al., 2017) and Macaque monkeys (Achard et al., 2006), in attempt to better understand their function. Biological networks typically demonstrate a small–world architecture (see Box 1 for definition). Small–worldness has been shown to allow for high efficiency of synchronized and parallel information transfer between neural regions (Bullmore and Sporns, 2009). Within such a system, shorter paths from node to node (with few longer sparse connections) may provide more efficient communication across an entire system, thereby facilitating dynamical processes that require global coordination and information flow (Watts and Strogatz, 1998; Strogatz, 2001). For instance, regions with short path length and high clustering coefficient confer an ability to transfer information quickly between a large number of nodes. Contrastingly, areas with long path lengths and low clustering may allow for sparse connections between individual clusters in a network, resulting in a slower spread of information over greater distance (Strogatz, 2001; Bullmore and Sporns, 2012; Muldoon et al., 2016). Understanding these distinct structural features within biological neural networks has allowed researchers to infer that such networks may utilize different structural properties to communicate under separate time scales (e.g., fast local synchronization within dense regions and slow global communication between dense regions; Chow and Kokotovic, 1985; Tahmassebi et al., 2017).
The commencement of the Human Connectome Project in 2005 (Sporns et al., 2005) has driven a surge in techniques and studies to map the structure and function of the human brain network (Sporns et al., 2002; Bertolero et al., 2015; Farahani et al., 2019; Gilson et al., 2019). Many such studies apply graph theory to analyse the connectivity within and between regions of the brain (Bullmore and Sporns, 2009). While the networks of simple organisms such as C. elegans are composed of only 270–300 individual neurons (Yan et al., 2017), the human brain is a much larger network, composed of tens of billions of neurons (although the exact number is contested), each of which has around ten thousand synapses (Koch, 2004; Shepherd, 2004). The sheer number of neural components makes it extremely difficult to model human neural networks graphically. Therefore, much of the graph theory analysis on human neural networks is applied to large collection of neurons, or even entire regions of the brain (e.g., Bassett and Bullmore, 2006; Gilson et al., 2019).
1.2. Neuromorphic Systems: Mimicking the Brain in Hardware
In parallel to developments in neuroscience, the engineering community has spurred the development of neuromorphic systems that can mimic the function of human neurons in hardware (Vianello et al., 2017). Carver Mead's pioneering efforts to emulate biological information processing using analog circuits (instead of logic gates used in digital computing) and leveraging the inherent device physics of Metal Oxide Semiconductor Field Effect Transistors (MOSFETs) established a new paradigm in computing hardware (Mead, 1990). Today, neuromorphic computing encompasses the use of novel nanotechnologies such as non-volatile memory devices and memristors (memory-resistors) that can mimic synapse-like memory and spiking temporal characteristics (Yang et al., 2013; Burr et al., 2017; Ziegler et al., 2018; Roy et al., 2019). Because of their unconventional “beyond von Neumann” architecture, which substantially reduces power requirements, such devices are also attractive for implementing Artificial Neural Network (ANN) algorithms, which require computationally-intensive training to learn input-output relationships, thereby mimicking neurons and synaptic connections in software (Xu et al., 2018).
Similarly, neuromorphic chips [e.g., IBM's TrueNorth (Merolla et al., 2014; Akopyan et al., 2015) and Intel's Loihi (Davies et al., 2018)] have been developed specifically as ANN accelerators, although their neuromorphic hardware attributes are limited to the integration of processing and memory to reduce power requirements. More generally, a limitation of neuromorphic in-memory computing hardware systems is their restriction to a regularized grid-like array structure that emphasizes the role of individual synapse-like elements (e.g., memristors), rather than the network architecture as a whole.
This limits potential advantages arising from structure–function integration in a distributed network, such as in a small–world architecture seen in biological neural networks (Bullmore and Sporns, 2009; Chialvo, 2010). It is likely that due to their conventional grid-like array structure, most neuromorphic systems lack the emergent dynamical properties that are characteristic of neural network circuitry (e.g., memory, learning, and even intelligence). Such emergent properties are attributed to the complexity of neural networks and the interplay between structure and function (Hagmann et al., 2008; Chialvo, 2010; Bassett and Gazzaniga, 2011). It is important to note that factors other than topology may influence emergent behavior (e.g., learning rules specifically designed for ability acquisition; Chollet, 2019). However, much of the literature exploring emergence in complex systems, including biological networks, emphasizes the role of topology, and structural properties as key to understanding emergence (Angeline, 1994; Chialvo, 2010; Pascual-García, 2016; Dumitrescu et al., 2017).
We previously introduced a novel neuromorphic system comprised of self–assembled nanowires whose structure and function (in response to electrical stimulation) mimic that of biological neural networks (Kuncic et al., 2018; Diaz-Alvarez et al., 2019). In these networks, each junction between nanowires provides a non-linear synaptic function in a similar manner as an atomic switch (Terabe et al., 2005; Ohno et al., 2011). Rather than focusing on the controllability of individual synapses like ANNs or other neuromorphic systems, our Atomic Switch-like Networks (ASNs) mimic the complex topology of biological neural networks, by mimicking biological self–assembly to form similarly complex networks comprised of nanowires (synthetic neurons) and junctions (synthetic synapses) (Stieg et al., 2012; Diaz-Alvarez et al., 2019).
Previous studies have shown that ASNs exhibit emergent properties such as non-linear dynamics, recurrence and capacity for learning, which arise from the complexity of the networks, as well as the properties of the atomic switch-like junctions (Terabe et al., 2005; Avizienis et al., 2012; Kuncic et al., 2018). Such properties are essential for brain-like function (Avizienis et al., 2012). However, due to the complexity of ASNs, it is highly difficult to understand or predict the impact and interactions of the networks' structure and functions from experimental data alone. Furthermore, due to the networks' self-assembled structure, it is experimentally difficult to control the topology to measure how it influences dynamics. It is also extremely difficult to use imaging-based techniques such as or electron microscopy (e.g., White et al., 1986; Eberle and Zeidler, 2018) reconstructions to unpack the structural connectivity of ASNs, as it is impossible to tell whether or not intersecting wires form a junction between them. We therefore have developed a computational model that simulates the structure experimental ASNs, based on functional, experimental validation (Kuncic et al., 2018; Diaz-Alvarez et al., 2019). For the purposes of the present study, we use this model solely to construct simulated self-assembled networks for structural analysis. ASNs are made of a fixed nanowire structure that does not change under electrical activation. Our simulations allow us to visualize each wire and connection individually in a graphical representation, and to easily alter them, either by changing the positioning and lengths of individual wires and junctions, or manipulating the density and dispersion of the networks. Consequently, our model enables us to examine the structural properties of specific sections of the network, which is currently impossible to do experimentally, as well as different realizations of nanowire networks.
Here, we apply graph theory measures to simulated ASNs with varying topologies. This allows us to examine the topological properties of ASNs, and compare them with a range of real-world networks. These include the simple organism C. elegans, as well as random and grid-like networks.
2. Methods
2.1. Construction of Simulated ASNs
To explore the topology of ASNs, we generated multiple networks with different structural parameters (see Figure 1 for visualizations). Hardware ASNs acquire a complex network structure through bottom-up self-assembly (Avizienis et al., 2012; Stieg et al., 2012; Diaz-Alvarez et al., 2019), similar to neural network growth in the brain. To simulate this self-assembly, we modeled nanowires as 1D objects of length uniformly drawn from a normal distribution of specified average wire length (mean of distribution, ranging from 6 to 9 μm) and wire dispersion (ratio of standard deviation to the mean, ranging from 0 to 50%). These wires were randomly placed within a 2D plane of fixed size (30 × 30 μm), with horizontal and vertical positions of the wire centers generated from a uniform spatial distribution. The angular orientation of each wire was generated from a uniform distribution on 2π. A junction was modeled at each intersection point between nanowires (Kuncic et al., 2018; Diaz-Alvarez et al., 2019). The connectivity was mapped to a graph adjacency matrix representation with nodes corresponding to nanowires and edges corresponding to junctions. In real networks, not every intersection between wires need necessarily form a junction. It is, however, practically difficult to determine where individual junctions exist in the self-assembled networks (Diaz-Alvarez et al., 2019). In our simulated networks, the simplifying assumption that all intersections result in junctions has negligible effect on network functionality when compared to experimental measurements of hardware ASNs (see Supplementary Materials).
Figure 1. Neuromorphic nanowire networks. (a) Optical microscopy image of an actual self–assembled network of nanowires. The length of wires varies from~6 to 50μm in this image. (b) Simulated 500 nw (6,065 junction) network generated by our model. The length of wires in the simulated networks varies from 6 to 9μm.
For each of the networks, the following parameters were varied: number of nanowires (i.e., 100, 500, 1, 000, or 2, 000 nws), average nanowire length (6 – 9 μm), and dispersion of wire length (0, 10, 20, and 50% of average nanowire length). Using this process, we generated a total of 39 different combinations of networks. All simulated networks were constructed in Matlab v2018a and Python v3.7.3. All structural connectivity measures were taken from the open-source Brain Connectivity Toolbox (Rubinov and Sporns, 2010) and NetworkX (Hagberg et al., 2008) packages.
To contextualize the structural connectivity of our ASNs, we simulated the topology of Watts-Strogatz networks ranging from random to grid-like, and C. elegans. Graph theory measures were applied to the connectivity data of each ASN, as well as to each of the Watts-Strogatz and C. elegans networks (see Figure 2 for graphical representations of all networks). We also included a fully-connected ANN similar to a random Watts-Strogatz network. Next, we compared global clustering coefficient and average path lengths (Watts and Strogatz, 1998). We also calculated the small–world propensity values for each network to establish an unbiased (see Box 1) measure of small–worldness in all networks (Muldoon et al., 2016). Finally, we mapped 100 and 500 nw ASNs, as well as C. elegans and correspondingly sized random and grid-like WS networks, on the Guimerà and Amaral (2005) cartographic plane to compare participant coefficient and within-module degree z-score. This allowed us to examine the modularity and integration of the networks (Guimerà and Amaral, 2005; Power et al., 2013; Bertolero et al., 2015).
Figure 2. Graphical representations of sample networks: Grid-like Watts-Strogatz (WS) networks (β = 0; left); random Watts-Strogatz networks (β = 1; center-left); ASNs (center-right); C. elegans and fully-connected ANN networks (right). Nodes represent nanowires (or neurons for C. elegans and ANN), while edges represent junctions (or synapses for C. elegans and virtual synapses for ANN).
2.2. Construction of Random and Grid-Like Watts-Strogatz Networks
To create a series of Watts-Strogatz networks, we first created a ring lattice with N nodes of mean degree 2k, where 2k = mean degree of the corresponding ASN with N nodes. In the Watts-Strogatz networks, each node is connected to its k nearest neighbors on either side. For each edge, E, in the graph, we then rewire the target node to k other nodes in the network with probability β. When β = 0, no edges are rewired and the model returns a locally-clustered ring lattice. We term this network grid-like, as its non-graphical representation is formed from a grid-shaped lattice. In contrast, when β = 1, all of the edges are rewired and the ring lattice is transformed into a random graph (MathWorks, 2016). We varied β from 0 to 1 in steps of 0.05, leaving 21 networks ranging from completely Grid-Like (β = 0) to completely Random (β = 1), for each size N. A β of 0.2 is denoted as displaying “small–world” characteristics (Watts and Strogatz, 1998). Furthermore, to compare ASNs with a WS random-like ANN model, we constructed a 5-layer ANN, with 10 input nodes, 10 output nodes and 160 nodes in each middle layer. Every node in each layer is connected to every node in its parent and child layers (hence the term “fully-connected”). However, no nodes are connected within layers.
2.3. Construction of C. elegans Networks
Neuronal connectivity data of the simple nematode C. elegans (277 neurons and 2,105 synaptic connections) was adapted from Achacoso and Yamamoto (1991), and electron microscope reconstructions by White et al. (1986).
3. Results
3.1. Small–Worldness
We compared the structures of multiple unique ASNs across four sizes (a total of 39 networks comprised of 100, 500, 1,000, or 2,000 nanowires) with a fully-connected ANN, a C. elegans network, and Watts-Strogatz random/grid-like networks across four sizes and 21 varying β parameters (one network for each β, and for each size). See Table 1 for a full statistical description of each network.
Table 1. Mean and Standard Deviation (SD) for ASNs, WS random networks, WS grid-like networks of different sizes, and C. elegans.
Figure 3 shows a comparison of path lengths and path distances between 100 and 500 nw ASNs, and a C. elegans network. Figure 4 shows a comparison of global mean path length and global clustering coefficient for each of the networks studied. Larger ASNs have similar mean path length to C. elegans, but higher clustering. However, ASN networks of similar size to C. elegans have a higher average path length.
Figure 3. Path length comparison of sample 100 and 500 nw ASNs with C. elegans. (A) Path length of each node from a randomly selected peripheral node for sample 100, 500 nw and C. elegans networks. (B) Distribution of path lengths from all node pairs in each sample network, including the average and median path length distributions, for 100, 500 nw and C. elegans networks.
Figure 4. Watts and Strogatz (1998) cartographic plane: global clustering coefficient and global mean path length. Each large dot represents 39 ASNs of varying parameters, with colors representing the network size (number of nanowire nodes). The small dots are Watts-Strogatz networks, rewired from completely random (β = 1) to grid-like (β = 0). Beta values decrease from bottom to top. The large yellow square measures the C. elegans network, and the large pink triangle is a 500-node fully-connected ANN.
ASNs are also more clustered and have a longer mean path length than random WS networks (β = 1). Compared to grid-like WS networks (β = 0), ASNs tend to be less clustered with generally shorter path lengths. Compared to a fully–connected ANN of 500 nodes, ASNs display much higher clustering, and longer path lengths.
Using path length and clustering coefficient to estimate small–worldness Watts and Strogatz (1998), ASNs would fall in the small–world category, with relatively low path length and high clustering. Recently a measure called small–world propensity has been employed to consider potential drawbacks of the Watts–Strogatz method (see Box 1; Muldoon et al., 2016).
One-way ANOVAs were conducted to compare the small–world propensity of 100 nw ASNs with 100 node WS random-like, 100 node WS grid-like and C. elegans networks. There was a significant difference between groups [F(3,10) = 47.16, p < 0.001] (where F is the ratio of mean square values of each group). Post-hoc analysis using the Bonferonni correction for multiple comparisons indicated that 100 nw ASNs had higher small–world propensity [Mean(M) = 0.69, Standard Deviation(SD) = 0.04] than random networks (M = 0.29, SD = 0) and grid-like networks (M = 0.29, SD = 0), but there was no significant difference between ASNs and C. elegans (M = 0.55, SD = 0; see Supplementary Materials for boxplots and multiple comparison graphs). We repeated these ANOVAs for 500 nw ASNs and 500 node WS networks [F(3,9) = 182.16, p < 0.001]. Post-hoc analysis indicated that 500 nw ASNs (M = 0.68, SD = 0.02) had higher small–world propensity than random networks (M = 0.29, SD = 0), grid-like networks (M = 0.29, SD = 0) and C. elegans (M = 0.55, SD = 0). Figure 5 shows a visual difference between ASNs and other networks.
Figure 5. Average small–world propensity values for Watts-Strogatz (WS), 500-node ANN, C. elegans and ASNs of varying sizes (number of nanowires, nw). Averages for WS were taken across all 21 β parameters from 0 to 1, with error bars reflecting standard deviation across β parameters. Averages for ASNs were taken from 39 networks with varying parameters as described in the methods section, with error bars reflecting standard deviation across network parameters.
3.2. Modularity and Integration
We used MZ and PCoeff measures to plot ASNs on a Guimerà and Amaral (2005) cartographic space (see Figure 6A for 100 and 500 nw ASN values, and Supplementary Figure 5 for 1,000 and 2,000 nw ASN values). Briefly, this involves calculating the modular assignment of each node (see Box 1), and then estimating each nodes' topological role, relative to the modular assignment: high MZ = high within-node connection (segregation) and high PCoeff = high between-node connection (integration). When combined, these measures exhibit the modularity and hub characteristics of a network. Each region in this space classifies a node in a network as a specific type. Almost all the nodes in all sizes of ASNs were categorized as ultra-peripheral (PCoeff = 0), peripheral (MZ < 2.5, 0 < PCoeff < 0.62), and non-hub connector regions (MZ < 2.5, 0.62 < PCoeff < 0.80). There were also a very few nodes that fell in the provincial hub region (MZ > 2.5, and 0 < PCoeff < 0.30).
Figure 6. Guimera-Amaral (2005) Cartographic-Space: Within-Module Degree z-Score (MZ) and Participant Coefficient (PCoeff). The dark-gray region (bottom left) refers to ultra-peripheral nodes (i.e., nodes with only one or few connections within a module); The light red region refers to peripheral nodes (i.e., nodes that are non-hubs and are only connected within a module); The green region refers to non-hub connector nodes (i.e., nodes that are connected to other modules but are not hubs); The blue region (bottom right) refers to non-hub kinless nodes (i.e., non-hub nodes not belonging to a particular module); The yellow region (top left) refers to provincial hubs (i.e., hub nodes that are highly connected within a module, but not between modules); The cream region refers to connector hubs (i.e., nodes that are highly connected between modules, but not necessarily within modules); the light-gray region (top right) refers to kinless hubs (i.e., hubs not belonging to a particular module). Regions are adapted from Guimerà and Amaral (2005). See Box 1 for example nodes. (A) 500 nw ASN networks. Each colored dot is the mean MZ and PCoeff for the most dense (Avg DEG = 40.66) variations of ASNs. (B) 500 node WS networks and C. elegans networks. Squares represent the MZ and PCoeff for C. elegans. Stars and crosses represent the MZ and PCoeff values for WS networks with β = 0 and 1, respectively. (C) Average PCoeff and MZ scores for each type of network.
We compared the PCoeff and MZ of ASNs to both WS networks, and a biological system such as the C. elegans (see Figure 6B). One-way ANOVAs were conducted to compare the PCoeff and MZ of 100 nw ASNs with 100 node WS random-like, 100 node WS grid-like and C. elegans networks. There was a significant difference between groups [F(3,1,517) = 112.64, p < 0.001] for PCoeff, but there was no significant difference for MZ. Post-hoc analysis using the Bonferonni correction for multiple comparisons indicated that 100 nw ASNs had lower PCoeff (M = 0.22, SD = 0.23) than C. elegans (M = 0.41, SD = 0.21) and random networks (M = 0.57, SD = 0.17), but there was no significant difference between ASNs and grid-like networks. We repeated these ANOVAs for 500 nw ASNs and 500 node WS networks, and the results were largely unchanged (see Supplementary Materials).
The structures of WS random-type networks tend to have higher PCoeff values (see Table 1 for means and standard deviations), mainly in the PCoeff > 0.8 region (non-hub kinless nodes). They also have some examples of MZ > 2.5 in the connector and kinless hub regions, but mainly MZ < 2.5. WS grid-like networks have lower PCoeff values, typically limited to ultra-peripheral and peripheral regions. C. elegans networks cover a greater portion of the cartographic space, although most of the nodes tend to fall within the peripheral and non-hub regions (see Guimerà and Amaral, 2005 for more examples of biological PCoeff/MZ distributions).
4. Discussion
ASNs exhibit a small–world structure, characterized by relatively short mean path length, alongside high clustering (Sporns et al., 2002; Sporns and Zwi, 2004). When compared with random or grid-like Watts–Strogatz networks, ASNs demonstrate more biological-like small–worldness features. While both random networks and ASNs have short path lengths, ASNs show higher clustering. In studies on human neural networks, it has been suggested that a small–world network is ideal, for example, for synchronizing neural activity between brain regions (Latora and Marchiori, 2001; Reijneveld et al., 2007; Verweij et al., 2014). In turn, this reflects the capacity for high global efficiency of parallel information transfer between such regions (Bullmore and Sporns, 2009). ASNs may therefore have capacity for efficient, synchronized and parallel information transfer across the entire network, similar to that of biological systems.
However, the structure of wiring within and between regions/clusters, as highlighted by PCoeff and MZ measures, may be different from biological systems such as C. elegans (see Supplementary Figure 3 for comparison with human node types). In biological neural networks, PCoeff and MZ are used to identify whether particular nodes play a hub-like role in the network. Hubs are central areas through which large amounts of information is trafficked to reach different parts of a network (van den Heuvel and Sporns, 2013). They are characterized by high connectivity to other network regions, as well as central positioning in the network. MZ scores have been used to denote hub status (e.g., z-Score < 2.5), while PCoeff values are used to classify the type of hub (Guimerà and Amaral, 2005; Joyce et al., 2010). Our results are consistent with previous studies showing that nodes in C. elegans have many peripheral and non-hub connector nodes, but also some hub-type provincial and connector nodes (Achacoso and Yamamoto, 1991; Guimerà and Amaral, 2005; Power et al., 2013). Such networks maintain a balance between integration and segregation of modules. In contrast, random WS networks are largely comprised of highly integrated, non-hub nodes, with a few hub-type nodes. This reflects a network with few modules. Grid-like WS networks are entirely comprised of non-hub, ultra-peripheral/peripheral type nodes, with very little integration or even modularity, as they have no connector nodes to connect between any modules that may exist.
How do ASNs fit within this space? Our results suggest that ASNs have a high proportion of peripheral, non-hub type nodes, similar to grid-like graphs. However, ASNs also have many non-hub connector nodes, which grid-like graphs lack. This means that ASNs are highly segregated, but also have many connections between modules, although they are weaker than within-module connections. Therefore they also have higher modularity than C. elegans. Random networks, on the other hand are highly integrated and have very few connector or peripheral nodes. Therefore ASNs have greater segregation than random networks, and higher modularity than both random and grid-like networks. ASNs involve a balance between integration and segregation, that is biased toward the presence of highly clustered, tight-knit modules with sparse inter-connectivity.
However the modularity and segregation of ASNs do not seem to reflect that of an organism like C. elegans. The nematode network has a greater balance between segregation and integration than ASNs, although likely with less modularity. Even ASNs that are highly dense only have a few hub-type nodes, meaning that most of the network's capacity to transfer information occurs in segregated modules, with sparse links between modules. Networks like the C. elegans would likely have fewer modules, with more central hub-type nodes that are responsible for directing information flow to and from the segregated modules of the network (hence the term hub).
Optimization of the structure of ASNs to represent biological-like networks may be desirable in the future, to allow for more biological-like capacities. For instance, increasing the size of the networks, and allowing for a greater balance between sparse and dense connections may allow for a more equivalent distribution of MZ and PCoeff scores, as well as increasing small–worldness even more. If these parameters are changed, it may be possible to construct nanowire networks that are even more representative of a biological system. However, it may be that ASNs currently demonstrate similar functionality to biological organisms, but with a uniquely highly modular and segregated structure that has more emphasis on peripheral-type nodes. In such a case, nodes within a particular module or cluster may communicate more within that module than with nodes outside it, yet still produce dynamics that are similar to biological systems (Kuncic et al., 2018). Due to the high small–worldness that ASNs demonstrate, it may be possible that these types of networks place much less importance on hub-type nodes or regions, as many other small–world complex networks do (e.g., Guimerà and Amaral, 2005; van den Heuvel and Sporns, 2013; Verweij et al., 2014). We plan to investigate the functional connectivity of ASNs in a future study, to understand how similar the interplay between structure and function in these networks may be to biological systems and other real-world networks.
4.1. Conclusion
Neuromorphic nanowire networks demonstrate a small–world architecture that is similar to the biological system of C. elegans, and is distinct from random or grid-like networks (including untrained artificial neural networks). However, they also appear to be comprised of nodes that are equivalent to peripheral or non-hub nodes in a biological system, while being more segregated and modular, and less reliant on hubs of information flow. In future studies, investigating the functional properties of neuromorphic nanowire networks under electrical activation, coupled with altering the topology of these networks, will provide new insights into the interplay between structural and functional connectivity in a way that is extremely difficult experimentally. This may bring us closer to better understanding the physical components that may give rise to emergent dynamical behaviors of neural-network-like structures; behaviors that, in turn, enable cognitive functions such as learning and memory, or even intelligence.
Data Availability Statement
The code used to generate the datasets analyzed for this study can be found on Github. Datasets are also available on request from the corresponding author.
Author Contributions
Data collection, writing, and editing was conducted mainly by AL, with support from RZ, JH, ML, and KF. AD-A and TN helped with code and experimental verification of the simulation model. JS and ZK supervised the project and provided writing, editing, and feedback for the manuscript.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2020.00184/full#supplementary-material
References
Achacoso, T. B., and Yamamoto, W. S. (1991). Ay's Neuroanatomy of C. elegans for Computation, Vol. 1991. Boca Raton, FL: CRC Press.
Achard, S., Salvador, R., Whitcher, B., Suckling, J., and Bullmore, E. (2006). A resilient, low-frequency, small-world human brain functional network with highly connected association cortical hubs. J. Neurosci. 26, 63–72. doi: 10.1523/JNEUROSCI.3874-05.2006
Akopyan, F., Sawada, J., Cassidy, A., Alvarez-Icaza, R., Arthur, J., Merolla, P., et al. (2015). “TrueNorth: Design and Tool Flow of a 65 mW 1 Million Neuron Programmable Neurosynaptic Chip,” in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 34, 1537–1557.
Albert, R., Jeong, H., and Barabási, A. L. (2011). “Diameter of the world-wide web,” in The Structure and Dynamics of Networks, Vol. 9781400841356 (Princeton, NJ: Princeton University Press), 182.
Angeline, P. J. (1994). Genetic programming and emergent intelligence. Adv. Genet. Programm 1, 75–98.
Avizienis, A. V., Sillin, H. O., Martin-Olmos, C., Shieh, H. H., Aono, M., Stieg, A. Z., et al. (2012). Neuromorphic atomic switch networks. PLoS ONE 7:e42772. doi: 10.1371/journal.pone.0042772
Bassett, D. S., and Bullmore, E. (2006). Small-world brain networks. Neuroscientist 12, 512–523. doi: 10.1177/1073858406293182
Bassett, D. S., and Gazzaniga, M. S. (2011). Understanding complexity in the human brain. Trends Cogn. Sci. 15, 200–209. doi: 10.1016/j.tics.2011.03.006
Bertolero, M. A., Thomas Yeo, B. T., and D'Esposito, M. (2015). The modular and integrative functional architecture of the human brain. Proc. Natl. Acad. Sci. U.S.A. 112, E6798–E6807. doi: 10.1073/pnas.1510619112
Boccaletti, S., Latora, V., Moreno, Y., Chavez, M., and Hwang, D. U. (2006). Complex networks: Structure and dynamics. Phys. Rep. 424:175–308. doi: 10.1016/j.physrep.2005.10.009
Bullmore, E., and Sporns, O. (2009). Complex brain networks: Graph theoretical analysis of structural and functional systems. Nat. Rev. Neurosci. 10, 186–198. doi: 10.1038/nrn2575
Bullmore, E., and Sporns, O. (2012). The economy of brain network organization. Nat. Rev. Neurosci. 13:336–349. doi: 10.1038/nrn3214
Burr, G. W., Shelby, R. M., Sebastian, A., Kim, S., Kim, S., Sidler, S., et al. (2017). Neuromorphic computing using non-volatile memory. Adv. Phys. X 2, 89–124. doi: 10.1080/23746149.2016.1259585
Chialvo, D. R. (2010). Emergent complex neural dynamics. Nat. Phys. 6, 744–750. doi: 10.1038/nphys1803
Chollet, F. (2019). On the Measure of Intelligence. arXiv [Preprint]. arXiv:1911.01547. Available online at: https://arxiv.org/abs/1911.01547
Chow, J. H., and Kokotovic, P. V. (1985). “Time scale modeling of sparse dynamic networks,” in IEEE Transactions on Automatic Control, Vol. 30, 714–722.
Cohen, J. R., and D'Esposito, M. (2016). The segregation and integration of distinct brain networks and their relationship to cognition. J. Neurosci. 36, 12083–12094. doi: 10.1523/JNEUROSCI.2965-15.2016
Davies, M., Srinivasa, N., Lin, T. H., Chinya, G., Cao, Y., Choday, S. H., et al. (2018). Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38, 82–99. doi: 10.1109/MM.2018.112130359
Diaz-Alvarez, A., Higuchi, R., Sanz-Leon, P., Marcus, I., Shingaya, Y., Stieg, A. Z., et al. (2019). Emergent dynamics of neuromorphic nanowire networks. Sci. Rep. 9:14920. doi: 10.1038/s41598-019-51330-6
Dumitrescu, A. T., Oltean, E., Merezeanu, D., and Dobrescu, R. (2017). “Emergence in hierarchical complex systems structured as social networks,” in Proceedings - 2017 21st International Conference on Control Systems and Computer, CSCS 2017 (Bucharest: Institute of Electrical and Electronics Engineers Inc.), 426–431.
Eberle, A. L., and Zeidler, D. (2018). Multi-beam scanning electron microscopy for high-throughput imaging in connectomics research. Front. Neuroanat. 12:112. doi: 10.3389/fnana.2018.00112
Estrada, E., and Hatano, N. (2008). Communicability in complex networks. Phys. Rev. E Stat. 77:036111. doi: 10.1103/PhysRevE.77.036111
Eubank, S., Guclu, H., Kumar, V. S., Marathe, M. V., Srinivasan, A., Toroczkai, Z., et al. (2004). Modelling disease outbreaks in realistic urban social networks. Nature 429, 180–184. doi: 10.1038/nature02541
Farahani, F. V., Karwowski, W., and Lighthall, N. R. (2019). Application of graph theory for identifying connectivity patterns in human brain networks: a systematic review. Front. Neurosci. 13:585. doi: 10.3389/fnins.2019.00585
Gilson, M., Kouvaris, N. E., Deco, G., Mangin, J. F., Poupon, C., Lefranc, S., et al. (2019). Network analysis of whole-brain fMRI dynamics: A new framework based on dynamic communicability. NeuroImage 201:116007. doi: 10.1016/j.neuroimage.2019.116007
Grayson, D. S., Bliss-Moreau, E., Machado, C. J., Bennett, J., Shen, K., Grant, K. A., et al. (2016). The rhesus monkey connectome predicts disrupted functional networks resulting from pharmacogenetic inactivation of the amygdala. Neuron 91, 453–466. doi: 10.1016/j.neuron.2016.06.005
Guimerà, R., and Amaral, L. A. (2005). Functional cartography of complex metabolic networks. Nature 433, 895–900. doi: 10.1038/nature03288
Hagberg, A., Swart, P., and Schult, D. A. (2008). “Exploring network structure, dynamics, and function using networkX,” in Proceedings of the 7th Python in Science Conference (SciPy2008), eds G. Varoquaux, T. Vaught, and J. Millman (Pasadena, CA), 11–15.
Hagmann, P., Cammoun, L., Gigandet, X., Meuli, R., Honey, C. J., Van Wedeen, J., et al. (2008). Mapping the structural core of human cerebral cortex. PLoS Biol. 6:e159. doi: 10.1371/journal.pbio.0060159
Harary, F., and Norman, R. Z. (1953). Graph Theory as a Mathematical Model in Social Science. Ann Arbor, MI: University of Michigan, Institute for Social Research.
Joyce, K. E., Laurienti, P. J., Burdette, J. H., and Hayasaka, S. (2010). A new measure of centrality for brain networks. PLoS ONE 5:e12200. doi: 10.1371/journal.pone.0012200
Koch, C. (2004). Biophysics of Computation: Information Processing in Single Neurons: Computational Neuroscience Series. Oxford, UK: Oxford University Press, Inc. Available online at: https://dl.acm.org/doi/book/10.5555/1137840
Kuncic, Z., Marcus, I., Sanz-Leon, P., Higuchi, R., Shingaya, Y., Li, M., et al. (2018). “Emergent brain-like complexity from nanowire atomic switch networks: towards neuromorphic synthetic intelligence,” in Proceedings of the IEEE Conference on Nanotechnology, Vol. 2018 (Cork: IEEE), 1–3.
Latora, V., and Marchiori, M. (2001). Efficient behavior of small-world networks. Phys. Rev. Lett. 87, 198701: doi: 10.1103/PhysRevLett.87.198701
Mason, O., and Verwoerd, M. (2007). Graph theory and networks in Biology. IET Syst. Biol. 1, 89–119. doi: 10.1049/iet-syb:20060038
MathWorks (2016). Build Watts-Strogatz Small World Graph Model. Available online at: https://au.mathworks.com/help/matlab/math/build-watts-strogatz-small-world-graph-model.html
Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., et al. (2014). A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345, 668– 673. doi: 10.1126/science.1254642
Muldoon, S. F., Bridgeford, E. W., and Bassett, D. S. (2016). Small-world propensity and weighted brain networks. Sci. Rep. 6:22057. doi: 10.1038/srep22057
Ohno, T., Hasegawa, T., Tsuruoka, T., Terabe, K., Gimzewski, J. K., and Aono, M. (2011). Short-term plasticity and long-term potentiation mimicked in single inorganic synapses. Nat. Mater. 10, 591–595. doi: 10.1038/nmat3054
Oliveira, C. L., Morais, P. A., Moreira, A. A., and Andrade, J. S. (2014). Enhanced flow in small-world networks. Phys. Rev. Lett. 112:148701. doi: 10.1103/PhysRevLett.112.148701
Pascual-García, A. (2016). A topological approach to the problem of emergence in complex systems. arXiv [Preprint]. arXiv:1610.02448. Available online at: https://arxiv.org/abs/1610.02448
Power, J. D., Schlaggar, B. L., Lessov-Schlaggar, C. N., and Petersen, S. E. (2013). Evidence for hubs in human functional brain networks. Neuron 79, 798–813. doi: 10.1016/j.neuron.2013.07.035
Reijneveld, J. C., Ponten, S. C., Berendse, H. W., and Stam, C. J. (2007). The application of graph theoretical analysis to complex networks in the brain. Clin. Neurophys. 118, 2317–2331. doi: 10.1016/j.clinph.2007.08.010
Roy, K., Jaiswal, A., and Panda, P. (2019). Towards spike-based machine intelligence with neuromorphic computing. Nature 575, 607–617. doi: 10.1038/s41586-019-1677-2
Rubinov, M., and Sporns, O. (2010). Complex network measures of brain connectivity: Uses and interpretations. NeuroImage 52, 1059–1069. doi: 10.1016/j.neuroimage.2009.10.003
Shepherd, G. M. (2004). The Synaptic Organization of the Brain. Oxford, UK: Oxford University Press.
Sporns, O., Tononi, G., and Edelman, G. M. (2002). Theoretical neuroanatomy and the connectivity of the cerebral cortex. Behav. Brain Res. 135, 69–74. doi: 10.1016/s0166-4328(02)00157-2
Sporns, O., Tononi, G., and Kötter, R. (2005). The human connectome: a structural description of the human brain. PLoS Comput. Biol. 1:e42. doi: 10.1371/journal.pcbi.0010042
Sporns, O., and Zwi, J. D. (2004). The small world of the cerebral cortex. Neuroinformatics 2, 145–162. doi: 10.1385/NI:2:2:145
Stieg, A. Z., Avizienis, A. V., Sillin, H. O., Martin-Olmos, C., Aono, M., and Gimzewski, J. K. (2012). Emergent criticality in complex turing B-type atomic switch networks. Adv. Mater. 24, 286–293. doi: 10.1002/adma.201103053
Tahmassebi, A., Pinker-Domenig, K., Wengert, G., Lobbes, M., Stadlbauer, A., Romero, F. J., et al. (2017). “Dynamical graph theory networks techniques for the analysis of sparse connectivity networks in dementia,” in Proceedings Volume 10216, Smart Biomedical and Physiological Sensor Technology XIV (Anaheim, CA), 1021609.
Terabe, K., Hasegawa, T., Nakayama, T., and Aono, M. (2005). Quantized conductance atomic switch. Nature 433, 47–50. doi: 10.1038/nature03190
van den Heuvel, M. P. and Sporns, O. (2013). Network hubs in the human brain. Trends Cogn. Sci. 17, 683–696. doi: 10.1016/j.tics.2013.09.012
Van Diessen, E., Zweiphenning, W. J., Jansen, F. E., Stam, C. J., Braun, K. P., and Otte, W. M. (2014). Brain network organization in focal epilepsy: a systematic review and meta-analysis. PLoS ONE 9:e114606. doi: 10.1371/journal.pone.0114606
Verweij, I. M., Romeijn, N., Smit, D. J., Piantoni, G., Van Someren, E. J., and van der Werf, Y. D. (2014). Sleep deprivation leads to a loss of functional connectivity in frontal brain regions. BMC Neurosci. 15:88. doi: 10.1186/1471-2202-15-88
Vianello, E., Garbin, D., Bichler, O., Piccolboni, G., Molas, G., Salvo, B. D., et al. (2017). Advances in Neuromorphic Hardware Exploiting Emerging Nanoscale Devices, Vol. 31, Cognitive Systems Monographs. New York, NY: Springer India.
Wakabayashi, H., and Iida, Y. (1992). Upper and lower bounds of terminal reliability of road networks:an efficient method with boolean algebra. J. Natural Disaster Sci. 14, 29–44.
Watts, D. J., and Strogatz, S. H. (1998). Collective dynamics of ‘small-world’ networks. Nature 393, 440–442. doi: 10.1038/30918
White, J. G., Southgate, E., Thomson, J. N., and Brenner, S. (1986). The structure of the nervous system of the nematode Caenorhabditis elegans. Philos. Trans. R. Soc. B Biol. Sci. 314, 1–340. doi: 10.1098/rstb.1986.0056
Xu, X., Ding, Y., Hu, S. X., Niemier, M., Cong, J., Hu, Y., et al. (2018). Scaling for edge inference of deep neural networks perspective. Nat. Electr. 1, 216–222. doi: 10.1038/s41928-018-0059-3
Yan, G., Vértes, P. E., Towlson, E. K., Chew, Y. L., Walker, D. S., Schafer, W. R., et al. (2017). Network control principles predict neuron function in the Caenorhabditis elegans connectome. Nature 550, 519–523. doi: 10.1038/nature24056
Yang, J. J., Strukov, D. B., and Stewart, D. R. (2013). Memristive devices for computing. Nat. Nanotechnol. 8, 13–24. doi: 10.1038/nnano.2012.240
Keywords: neuromorphic, atomic-switch networks, nanowires, topology, complex networks, structural connectivity, graph theory, artificial neural networks
Citation: Loeffler A, Zhu R, Hochstetter J, Li M, Fu K, Diaz-Alvarez A, Nakayama T, Shine JM and Kuncic Z (2020) Topological Properties of Neuromorphic Nanowire Networks. Front. Neurosci. 14:184. doi: 10.3389/fnins.2020.00184
Received: 30 November 2019; Accepted: 19 February 2020;
Published: 06 March 2020.
Edited by:
Stefano Brivio, Institute for Microelectronics and Microsystems (CNR), ItalyReviewed by:
Martin Ziegler, Technische Universität Ilmenau, GermanyGaribaldi Pineda García, University of Sussex, United Kingdom
Copyright © 2020 Loeffler, Zhu, Hochstetter, Li, Fu, Diaz-Alvarez, Nakayama, Shine and Kuncic. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Zdenka Kuncic, zdenka.kuncic@sydney.edu.au