The Network Nullspace Property for Compressed Sensing of Big Data Over Networks

We present a novel condition, which we term the network nullspace property, which ensures accurate recovery of graph signals representing massive network-structured datasets from few signal values. The network nullspace property couples the cluster structure of the underlying network-structure with the geometry of the sampling set. Our results can be used to design efficient sampling strategies based on the network topology.

1. Introduction

A recent line of work proposed efficient convex optimization methods for recovering graph signals which represent label information of network structured datasets (cf. [1, 2]). These methods rest on the hypothesis that the true underlying graph signal is nearly constant over well-connected subsets of nodes (clusters).

In this paper, we introduce a novel recovery condition, termed the network nullspace property (NNSP), which guarantees convex optimization to accurately recovery of clustered (“piece-wise constant”) graph signals from knowledge of its values on a small subset of sampled nodes. The NNSP couples the clustering structure of the underlying data graph to the locations of the sampled nodes via interpreting the underlying graph as a flow network.

The presented results apply to an arbitrary partitioning, but are most useful for a partitioning such that nodes in the same cluster are connected with edges of relatively large weights, whereas edges between clusters have low weights. Our analysis reveals that if cluster boundaries are well-connected (in a sense made precise) to the sampled nodes, then accurate recovery of clustered graph signals is possible by solving a convex optimization problem.

Most of the existing work on graph signal processing applies spectral graph theory to define a notion of band-limited graph signals, e.g., based on principal subspaces of the graph Laplacian matrix, as well as sufficient conditions for recoverability, i.e., sampling theorems, for those signals [3, 4]. In contrast, our approach does not rely on spectral graph theory, but involves structural (connectivity) properties of the underlying data graph. Moreover, we consider signal models which amount to clustered (piece-wise constant) graph signals. These models, which have also been used in Chen et al. [5] for approximating graph signals arising in various applications.

The closest to our work is [6, 7], which provide sufficient conditions such that a variant of the Lasso method (network Lasso) accurately recovers clustered graph signals from noisy observations. However, in contrast to this line of work, we assume the graph signal values are only observed on a small subset of nodes.

The NNSP is closely related to the network compatibility (NCC) condition, which has been introduced by a subset of the authors in Jung et al. [8] for analyzing the accuracy of network Lasso. The NCC is a stronger condition in the sense that once the NCC is satisfied, the NNSP is also guaranteed to hold.

2. Problem Formulation

Many important applications involve massive heterogeneous datasets comprised heterogeneous data chunks, e.g., mixtures of audio, video and text data [9]. Moreover, datasets typically contain mostly unlabeled data points; only a small fraction is labeled data. An efficient strategy to handle such heterogenous datasets is to organize them as a network or data graph whose nodes represent individual data points.

2.1. Graph Signal Representation of Data

In what follows we consider datasets which are represented by a weighted data graph ${G}=({V},{E},\text{W})$ with nodes ${V}=\left\{1,\dots ,N\right\}$, each node representing an individual data point. These nodes are connected by edges $\left\{i,j\right\}\in {E}$. In particular, given some application-specific notion of similarity, the edges of the data graph ${G}$ connect similar data points $i,j\in {V}$ by an edge $\left\{i,j\right\}\in {E}$. In some applications it is possible to quantify the extent to which data points are similar, e.g., via the distance between sensors in a wireless sensor network [10]. Given two similar data points $i,j\in {V}$, we quantify the strength of their connection $\left\{i,j\right\}\in {E}$ by a non-negative edge weight W_{i, j} ≥ 0 which we collect in the symmetric weight matrix $\text{W}\in {ℝ}_{+}^{N\times N}$.

In what follows we will silently assume that the data graph ${G}$ is oriented by declaring for each edge $\left\{i,j\right\}\in {E}$ one node as the head e⁺ and the other node as the tail e⁻. For the oriented data graph we define the directed neighborhoods of a node $i\in {V}$ as ${{N}}^{+}(i):=\left\{j\in {N}(i):e=\left\{i,j\right\}\in {E},\mathrm{\text{ and }}i=e^{+}\right\}$ and ${{N}}^{-}(i):=\left\{j\in {N}(i):e=\left\{i,j\right\}\in {E},\mathrm{\text{ and }}i=e^{-}\right\}$. We highlight that the orientation of the data graph ${G}$ is not related to any intrinsic property of the underlying data set. In particular, the weight matrix W is symmetric since the weights W_{i, j} are associated with undirected edges $\left\{i,j\right\}\in {E}$. However, using an (arbitrary but fixed) orientation of the data graph will be notationally convenient in order to formulate our main results.

Beside the edges structure ${E}$, network-structured datasets typically also carry label information which induces a graph signal defined over ${G}$. We define a graph signal x[·] over the graph ${G}=({V},{E},\text{W})$ as a mapping ${V}\to ℝ$, which associates (labels) every node $i\in {V}$ with the signal value x[i] ∈ ℝ. In a supervised machine learning application, the signal values x[i] might represent class membership in a classification problem or the target (output) value in a regression problem. We denote the space of all graph signals, which is also known as the vertex space (cf. [11]), by ${ℝ}^{{V}}$.

2.2. Graph Signal Recovery

We aim at recovering (learning) a graph signal $x\left[·\right]\in {ℝ}^{{V}}$ defined over the data graph ${G}$, from observing its values ${\left\{x\left[i\right]\right\}}_{i\in {M}}$ on a (small) sampling set ${M}:=\left\{i_1,\dots ,i_M\right\}\subseteq {V}$, where typically M ≪ N.

The recovery of the entire graph signal x[·] from the incomplete information provided by the signal samples ${\left\{x\left[i\right]\right\}}_{i\in {M}}$ is possible under a clustering assumption, which is also underlying many supervised machine learning methods [12]. This assumption requires the signal values or labels of data points which are close, with respect to the data graph topology, to be similar. More formally, we expect the underlying graph signal $\text{x}\left[·\right]\in {ℝ}^{{V}}$ to have a relatively small total variation (TV)

$\begin{array}{l}\left|\right|x\left[·\right]|{|}_{\text{TV}}:={\displaystyle \sum _{\left\{i,j\right\}\in {E}}}{W}_{i,j}|x\left[i\right]-x\left[j\right]|.\end{array}$

The total variation of the graph signal x[·] obtained over a subset ${S}\subseteq {E}$ of edges is denoted $\left|\right|x\left[·\right]|{|}_{{S}}:=\sum _{\left\{i,j\right\}\in {S}}{W}_{i,j}|x\left[j\right]-x\left[i\right]|$.

Some well-known examples of clustered graph signals include low-pass signals in digital signal processing where time samples at adjacent time instants are strongly correlated and close-by pixels in images tend to be colored likely. The class of graph signals with a small total variation are sparse in the sense of changing significantly over few edges only. In particular, if we stack the signal differences x[i] − x[j] (across the edges $\left\{i,j\right\}\in {E}$) into a big vector of size $\left|{E}\right|$, then this vector is sparse in the ordinary sense of having only few significantly large entries [13].

In order to recover a signal with small TV ||x[·]||_TV, from its signal values ${\left\{x\left[i\right]\right\}}_{i\in {M}}$, a natural strategy is:

$\begin{array}{lll}\widehat{x}\left[·\right]& \in {\displaystyle \underset{\tilde{x}\left[·\right]\in {ℝ}^{{V}}}{arg min}}\left|\right|\tilde{x}\left[·\right]|{|}_{\text{TV}}\mathrm{\text{s.t.}}\tilde{x}\left[i\right]=x\left[i\right]\mathrm{\text{ for all }}i\in {M}.& (1)\end{array}$

There exist highly efficient methods for solving convex optimization problems of the form (1) (cf.[14–16] and the references therein).

3. Recovery Conditions

The accuracy of any learning method which is based on solving (1) depends on the deviations between the solutions $\widehat{x}\left[·\right]$ of the optimization problem (1) and the true underlying graph signal $x\left[·\right]\in {ℝ}^{{V}}$. In what follows, we introduce the NNSP as a sufficient condition on the sampling set and graph topology such that any solution $\widehat{x}\left[·\right]$ of (1) accurately resembles an underlying clustered (piece-wise constant) graph signal of the form (cf. [5])

$\begin{array}{ll}x\left[i\right]={\displaystyle \sum _{{C}\in {F}}}{a}_{{C}}{{I}}_{{C}}\left[i\right]\text{ with }{{I}}_{{C}}\left[i\right]:=\{\begin{array}{c}1\mathrm{\text{ for }}i\in {C}\\ 0\mathrm{\text{ else}}.\hfill \end{array}& (2)\end{array}$

The signal model (2) is defined using a fixed partition ${F}=\left\{{{C}}_1,\dots ,{{C}}_{\left|{F}\right|}\right\}$ of the entire data graph ${G}$ into disjoint clusters ${{C}}_l\subseteq {V}$. The signal model (2) has been studied in Chen et al. [5], where it was demonstrated that it allows, compared to band-limited graph signal models, for more accurate approximation of datasets obtained from weather stations.

While our analysis applies to an arbitrary partition ${F}$, our results are most useful for partitions where the nodes within clusters ${{C}}_l$ are connected by many edges with large weight, while nodes of different clusters are loosely connected by few edges with small weights. Such reasonable partitions can be obtained by one of the recently proposed highly scalable clustering methods, e.g., [17, 18].

We highlight that the knowledge of the partition ${F}$ underlying the signal model (2) is only required for the analysis of signal recovery methods (such as sparse label propagation [15]), which are based on solving the recovery problem (1), However, the partition is not required for the actual implementation of those methods, as the recovery problem (1) itself does not involve the partition.

We will characterize a partition ${F}$ by its boundary

$\begin{array}{ll}\partial {F}:=\left\{\left\{i,j\right\}\in {E}:i\in {{C}}_l,j\in {{C}}_{l^{\prime }},\mathrm{\text{ with }}l\ne l^{\prime }\right\}\subseteq {E},& (3)\end{array}$

which is the set of edges connecting nodes from different clusters. We highlight that the recovery problem (1) not require knowledge of the partition ${F}$.

3.1. Network Nullspace Property

Consider a clustered graph signal $x\left[·\right]\in {ℝ}^{{V}}$ of the form (2). We observe its values x[i] at the sampled nodes $i\in {M}$ only. In order to have any chance for recovering the complete signal only from the samples ${\left\{x\left[i\right]\right\}}_{i\in {M}}$ we have to restrict the nullspace of the sampling set, which we define as

$\begin{array}{ll}{K}({M}):=\left\{\tilde{x}\left[·\right]\in {ℝ}^{{V}}:\tilde{x}\left[i\right]=0\mathrm{\text{ for all }}i\in {M}\right\}.& (4)\end{array}$

Thus, the nullspace ${K}({M})$ contains exactly those graph signals which vanish at all nodes of the sampling set ${M}$. Clearly, we have no chance in recovering any signal $\widehat{x}\left[·\right]$ which belongs to the nullspace ${K}({M})$ as it cannot be distinguished from the all-zero signal $\tilde{x}\left[i\right]=0$, for all nodes $i\in {V}$, which result in exactly the same (vanishing) measurements $\tilde{x}\left[i\right]=\widehat{x}\left[i\right]=0$ for all $i\in {M}\subseteq {V}$.

In order to define the NNSP, we need the notion of a flow with demands [19].

Definition 1. Given a graph ${G}=({V},{E},\text{W})$, a flow with demands g[i] ∈ ℝ, for $i\in {V}$, is a mapping $f\left[·\right]:{E}\to {ℝ}_{+}$ satisfying the conservation law

$\begin{array}{ll}{\displaystyle \sum _{j\in {{N}}^{+}(i)}}f\left[\left\{i,j\right\}\right]-{\displaystyle \sum _{j\in {{N}}^{-}(i)}}f\left[\left\{i,j\right\}\right]=g\left[i\right]& (5)\end{array}$

at every node $i\in {V}$.

For a more detailed discussion of the concept of network flows, we refer to [19]. In this paper, we use the concept of network flows in order to characterize the connectivity properties or topology of a data graph ${G}=({V},{E},\text{W})$ by interpreting the edge weights W_{i, j} as capacity constraints that limit the amount of flow along the edge $\left\{i,j\right\}\in {E}$.

In particular, the notion of network flows with demands allows to adapt the nullspace property, introduced within the theory of compressed sensing [20, 21] for sparse signals, to the problem of recovering clustered graph signals (cf. (2)).

Definition 2. Consider a partition ${F}=\left\{{{C}}_1,\dots ,{{C}}_{\left|{F}\right|}\right\}$ of pairwise disjoint subsets of nodes (clusters) ${{C}}_l\subseteq {V}$ and a set of sampled nodes ${M}\subseteq {V}$. The sampling set ${M}$ is said to satisfy the NNSP relative to the partition ${F}$, denoted NNSP-${F}$, if for any signature ${\sigma }_e\in {\left\{-1,1\right\}}^{\partial {F}}$, which assigns the sign σ_e to a boundary edge $e\in \partial {F}$, there is a flow f[e]

• with demands g[i] = 0, for $i\notin {M}$,

• its values satisfy

$\begin{array}{l}f\left[e\right]=\kappa {\sigma }_e{W}_{e}fore\in \partial {F}withsome\kappa >1,and\\ f\left[e\right]\le W_{e}fore\in {E}\backslash \partial {F}.& (6)\end{array}$

It turns out that a sampling set ${M}$ satisfies NNSP-${F}$ for a given partition ${F}$ of the data graph, then the nullspace ${K}({M})$ (cf. (4)) of the sampling set cannot contain a non-zero clustered graph signal of the form (2).

A naive verification of the NNSP involves a search over all signatures, whose number is around $2^{\left|\partial {F}\right|}$, which might be intractable for large data graphs. However, similar to many results in compressed sensing, we expect using probabilistic models for the data graph to render the verification of NNSP tractable [20]. In particular, we expect that probabilistic statements about how likely the NNSP is satisfied for random data graphs (e.g., conforming to a stochastic block model) can be obtained easily.

3.2. Exact Recovery of Clustered Signals

Now we are ready to state our main result, i.e., the NNSP ensures the solution (1) to be unique and to coincide with the true underlying clustered graph signal of the form (2).

Theorem 3. Consider a clustered graph signal $x_c\left[·\right]\in {X}$ (cf. (2)) which is observed only at the sampling set ${M}\subseteq {V}$. If the sampling set ${M}$ satisfies NNSP-${F}$, then the solution of (1) is unique and coincides with x_c[·].

Proof: see Appendix.

Thus, if we sample a clustered graph signal x[·] (cf. (2)) on a sampling set which satisfies NNSP-${F}$, we can expect convex recovery algorithms, which are based on solving (1) to accurately recover the true underlying graph signal x[·].

A partial converse. The recovery condition provided by Theorem 3 is essentially tight, i.e., if the sampling set does not satisfy NNSP-${F}$, then they are solutions to (1) which are different from the true underlying clustered graph signal.

Consider a clustered graph signal

$\begin{array}{ll}{x}_{\text{c}}\left[i\right]=1·{{I}}_{{{C}}_1}\left[i\right]+2·{{I}}_{{{C}}_2}\left[i\right]& (7)\end{array}$

defined over a chain graph ${{G}}_{\text{chain}}$ containing an even number N of nodes (Figure 1). We partition the graph into two equal-sized clusters ${{F}}_{\text{c}}=\left\{{{C}}_1,{{C}}_2\right\}$ with ${{C}}_1=\left\{1,\dots ,N/2\right\}$ and ${{C}}_2=\left\{N/2+1,\dots ,N\right\}$. The edges within clusters are connected by edges with unit weight, while the single edge {N/2, N/2+1} connecting the two clusters has weight 1/δ. Let us assume that we sample the graph signal x_c[i] on the sampling set ${{M}}_{\text{c}}=\left\{1,N\right\}$.

FIGURE 1

Figure 1. A clustered graph signal $x\left[i\right]=a_1{{I}}_{{{C}}_1}\left[i\right]+a_2{{I}}_{{{C}}_2}\left[i\right]$ (cf. (2)) defined over a chain graph ${{G}}_{\text{chain}}$ which is partitioned into two equal-size clusters ${{C}}_1$ and ${{C}}_2$ which consist of consecutive nodes. The edges connecting nodes within the same cluster have weight 1, while the single edge connecting nodes from different clusters has weight 1/δ.

For any δ > 1, the sampling set ${{M}}_{\text{c}}$ satisfies the NNSP-${{F}}_{\text{c}}$ with κ = δ > 1 (cf. (6)). Thus, as long as the boundary edge has weight 1/δ with δ > 1, Theorem 3 guarantees that the true clustered graph signal x_c[i] can be perfectly recovered via solving (1).

If, on the other hand, the weight of the boundary edge is 1/δ with some δ ≤ 1, then the sampling set ${{M}}_{\text{c}}$ does not satisfy the NNSP-${{F}}_{\text{c}}$. In this case, as can be verified easily, the true graph signal x_c[i] is not the unique solution to (1) anymore. Indeed, for δ ≤ 1 it can be shown that the graph signal (7) has a TV norm at least as large as the graph signal $x^{\prime }\left[i\right]=(1-\frac{i-1}{N-1}{)x}_{\text{c}}\left[1\right]+\frac{i-1}{N-1}{x}_{\text{c}}\left[N\right]$, which linearly interpolates between the sampled signal values x_c[1] and x_c[N].

3.3. Recovery of Approximately Clustered Signals

The scope of Theorem 3 is somewhat limited as it applies only to graph signals which are precisely of the form (2). We now state a more general result applying to any graph signal $x\left[·\right]\in {ℝ}^{{V}}$.

Theorem 4. Consider a graph signal $x\left[·\right]\in {ℝ}^{{V}}$ which is observed only at the sampling set ${M}$. If NNSP-${F}$ holds with κ = 2 in (6), any solution $\widehat{\text{x}}$ of (1) satisfies (cf. (2))

$\begin{array}{ll}\left|\right|\widehat{x}\left[·\right]-x\left[·\right]|{|}_{\text{TV}}\le 6{\displaystyle \underset{\text{a}\in {ℝ}^{\left|{F}\right|}}{min}}||x\left[·\right]-{\displaystyle \sum _{{C}\in {F}}}{a}_{{C}}{{I}}_{{C}}\left[·\right]|{|}_{\text{TV}}.& (8)\end{array}$

Proof: see Appendix.

Thus, as long as the underlying graph signal x[·] can be well approximated by a clustered signal of the form (2), any solution $\widehat{x}\left[·\right]$ of (1) is a graph signal which varies significantly only over the boundary edges $\partial {F}$. We highlight that the error bound (8) only controls the TV (semi-)norm of the error signal $\widehat{x}\left[·\right]-x\left[·\right]$. In particular, this bound does not directly allow to quantify the size of the global mean squared error $(1/N)\sum _{i\in {V}}{(\widehat{x}\left[i\right]-x\left[i\right])}^2$. However, the bound (8) allows to characterize identifiability of the underlying partition ${F}$. Indeed, if the signal values $a_{{C}}$ in (2) satisfy ${min}_{{C}\in {F}}|a_{{C}}-a_{{{C}}^{\prime }}|{min}_{e\in \partial {F}}{W}_e\ge \left|\right|\widehat{x}\left[·\right]-\widehat{x}\left[·\right]|{|}_{\text{TV}}$, we can read off the cluster boundaries from the signal differences $\widehat{x}\left[i\right]-\widehat{x}\left[j\right]$ (over edges $\left\{i,j\right\}\in {E}$).

One particular use of Theorems 3, 4 is to guide the choice for the sampling set ${M}$. In particular, one should aim at sampling nodes such that the NNSP is likely to be satisfied. According to the definition of the NNSP, we should sample nodes which are well connected (in the sense of allowing for a large flow) to the boundary edges which connect different clusters. This approach has been studied empirically in [22, 23], verifying accurate recovery by efficient convex optimization methods using sampling sets which satisfy the NNSP (cf. Definition 2) with high probability.

4. Numerical Experiments

We now verify the practical relevance of our theoretical findings by means of two numerical experiments. The first experiment is based on a synthetic data set whose underlying data graph is a chain graph ${{G}}_{\mathrm{\text{chain}}}$. A second experiment revolves around a real-world data set describing the roadmap of Minnesota [5, 24].

4.1. Chain Graph

We generated a synthetic data set whose data graph is a chain graph ${{G}}_{\text{chain}}$. This chain graph contains $\left|{V}\right|=100$ nodes which are connected by $\left|{E}\right|=99$ undirected edges {i, i + 1}, for i ∈ {1, …, 99} and partitioned into $\left|{F}\right|=10$ equal-size clusters ${F}={\left\{{{C}}_l\right\}}_{l=1,2,\dots ,10}$, each cluster containing 10 consecutive nodes. The edges connecting nodes in the same cluster have weight W_{i, j} = 4, while those connecting different clusters have weight W_{i, j} = 2. For this data graph we generated a clustered graph signal x[i] of the form with alternating coefficients a_l ∈ {1, 5}.

The graph signal x[i] is observed only at the nodes belonging to a sampling set, which is either ${{M}}_1$ or ${{M}}_2$. The sampling set ${{M}}_1$ contains exactly one node from each cluster ${{C}}_l$ and thus, as can be verified easily, satisfies the NNSP (cf. Definition 2). While having the same size as ${{M}}_1$, the sampling set ${{M}}_2$ does not contain any node of clusters ${{C}}_2$ and ${{C}}_4$.

In Figure 2, we illustrate the recovered signals obtained by solving (1) using the sparse label propagation (SLP) algorithm [15], which is fed with signal values on the sampling set (being either ${{M}}_1$ or ${{M}}_2$). The signal recovered from the sampling set ${{M}}_1$, which satisfies the NNSP, closely resembles the true underlying clustered graph signal. In contrast, the sampling set ${{M}}_2$, which does not satisfy the NNSP, results in a recovered signal which significantly deviates from the true signal.

FIGURE 2

Figure 2. Clustered graph signal x[i] along with the recovered signals obtained from sampling sets ${{M}}_1$ and ${{M}}_2$.

4.2. Minnesota Roadmap

The second data set, with associated data graph ${{G}}_{\text{min}}$, represents the roadmap of Minnesota [5]. The data graph ${{G}}_{\text{min}}$ consists of $\left|{V}\right|=2642$ nodes, and $\left|{E}\right|=3303$ edges. We generate a clustered graph signal defined over ${{G}}_{\text{min}}$ by randomly selecting three different nodes {i₁, i₂, i₃} which are declared as “cluster centres” of the clusters ${{C}}_1,{{C}}_2,{{C}}_3$. The remaining nodes ${V}\backslash \left\{i_1,i_2,i_3\right\}$ are then associated to the cluster whose center is nearest in the sense of smallest geodesic distance. The edges connecting nodes within the same cluster have weight W_{i, j} = 4, and those connecting different clusters have weight W_{i, j} = 2.

We use SLP to recover the entire graph signal from its values obtained for the nodes in a sampling set. Two different choices ${{M}}_1$ and ${{M}}_2$ for the sampling set are considered: The sampling set ${{M}}_1$ is based on the NNSP and consists of all nodes which are adjacent to the boundary edges between two different clusters. In contrast, the sampling set ${{M}}_2$ is obtained by selecting uniformly at random a total of $\left|{{M}}_1\right|$ nodes from ${{G}}_{\text{min}}$, i.e., we ensure $\left|{{M}}_1\right|=\left|{{M}}_2\right|$.

The resulting MSE is 0.0023 for the sampling set ${{M}}_1$ (conforming with NNSP), while the recovery using the random sampling set ${{M}}_2$ incurred an average (over 100 i.i.d. simulation runs) MSE of 0.0502. In Figure 3, we depict the recovered graph signals using signal samples from either ${{M}}_1$ or ${{M}}_2$ (one typical realization). Evidently, the recovery using the sampling set ${{M}}_1$ (which is guided by the NNSP) results in a more accurate recovery compared to using the random sampling set ${{M}}_2$.

FIGURE 3

Figure 3. True graph signal (A) and recovered signals for the Minnesota roadmap data set obtained (B) when using sampling set ${{M}}_1$, (C) or sampling set ${{M}}_2$.

5. Conclusions

We considered the problem of recovering clustered graph signals, defined over complex networks, from observing its signal values on a small set of sampled nodes.

By applying tools from compressed sensing, we introduced the NNSP as a sufficient condition on the graph topology and sampling set such that a convex recovery method is accurate. This recovery condition is based on the connectivity properties of the underlying network. In particular, it requires the existence of certain network flows with the edge weights of the data graph being interpreted as capacities.

The NNSP involves both, the sampling set and the cluster structure of the data graph. Roughly speaking it requires to sample more densely near the boundaries between different clusters. This intuition has be verified by means of numerical experiments on synthetic and real-world datasets.

Our work opens up several avenues for future research. In particular, it would be interesting to analyze how likely the NNSP holds for certain random network models and sampling strategies. The tightness of the resulting recovery guarantees could then be contrasted with fundamental lower bounds obtained from an information-theoretic approach to minimax-estimation. Moreover, we would like to study variations of the SLP recovery method which are more suitable for classification problems.

Author Contributions

AJ: Designing research. Deriving main theoretical results. Writing paper. MH: Numerical experiments. Proofreading paper.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

Parts of the material underlying this work has been presented in Jung et al. [25]. However, this paper significantly extends [25] by providing detailed proofs as well as extended discussions of the main results. This manuscript is available as a pre-print [26] at the following address: https://arxiv.org/abs/1705.04379. Copyright of this pre-print version rests with the authors.

References

1. Jung A, Berger P, Hannak G, Matz G. Scalable graph signal recovery for big data over networks. In: Proceeddings of IEEE-SP Workshop on Signal Processing Advances in Wireless Communications (SPAWC). Edinburgh (2016).

Google Scholar

2. Hannak G, Berger P, Matz G, Jung A. Efficient graph signal recovery over big networks. In: Proceedings of Asilomar Conference Signals, Systems, Computers. Pacific Grove, CA (2016). p. 1–6.

Google Scholar

3. Marques AG, Segarra S, Leus G, Ribeiro A. Sampling of graph signals with successive local aggregation. IEEE Trans Signal Process. (2016) 64:1832–43. doi: 10.1109/TSP.2015.2507546

CrossRef Full Text | Google Scholar

4. Chen S, Varma R, Sandryhaila A, Kovačević J. Discrete signal processing on graphs: sampling theory. IEEE Trans Signal Process. (2015) 63:6510–23. doi: 10.1109/TSP.2015.2469645

CrossRef Full Text | Google Scholar

5. Chen S, Varma R, Sandryhaila A, Kovačević J. Representations of piecewise smooth signals on graphs. In: Proceedings of IEEE ICASSP 2016. Shanghai (2016).

Google Scholar

6. Sharpnack J, Rinaldo A, Singh A. Sparsistency of the edge lasso over graphs. In: Proceedings 15th International Conference on Artificial Intelligence and Statistics (AISTATS). La Palma (2012).

Google Scholar

7. Wang Y-X, Sharpnack J, Smola A J, Tibshirani R J. Trend filtering on graphs. J Mach Lear Res. (2016) 17:120–45.

Google Scholar

8. Jung A, Tran N, Mara A. When is network lasso accurate? Front Appl Math Stat. (2018) 3:28. doi: 10.3389/fams.2017.00028

CrossRef Full Text | Google Scholar

9. Cui S, Hero A, Luo ZQ, Moura J. editors. Big Data over Networks. La Palma: Cambridge University Press (2016).

10. Zhao F, Guibas LJ. Wireless Sensor Networks: An Information Processing Approach. Amsterdam: Morgan Kaufmann (2004).

Google Scholar

11. Diestel R. Graph Theory. New York, NY: Springer (2000).

12. Chapelle O, Schölkopf B, Zien A. editors. Semi-Supervised Learning. Cambridge, MA: The MIT Press (2006).

Google Scholar

13. Donoho DL. Compressed sensing. IEEE Trans Inf Theory (2006) 52:1289–306. doi: 10.1109/TIT.2006.871582

CrossRef Full Text | Google Scholar

14. Zhu Y. An augmented ADMM algorithm with application to the generalized lasso problem. J Comput Graph Stat. (2017) 26:195–204. doi: 10.1080/10618600.2015.1114491

CrossRef Full Text | Google Scholar

15. Jung A, Hero AO III Mara, A, Jahromi B. Semi-supervised learning via sparse label propagation. arXiv (2017) 315–39. Available online at: https://arxiv.org/abs/1612.01414

Google Scholar

16. Chambolle A, Pock T. A first-order primal-dual algorithm for convex problems with applications to imaging. J Math Imaging Vis. (2011) 40:120–45. doi: 10.1007/s10851-010-0251-1

CrossRef Full Text | Google Scholar

17. Spielman D, Teng S-H. A local clustering algorithm for massive graphs and its application to nearly linear time graph partitioning. SIAM J Comput. (2013) 42:1–26. doi: 10.1137/080744888

CrossRef Full Text | Google Scholar

18. Fortunato S. Community detection in graphs. Phys Rep. (2010) 486:75–174. doi: 10.1016/j.physrep.2009.11.002

CrossRef Full Text | Google Scholar

19. Kleinberg J, Tardos E. Algorithm Design. New York, NY: Addison Wesley (2006).

Google Scholar

20. Foucart S, Rauhut H. A Mathematical Introduction to Compressive Sensing. New York, NY: Springer (2012).

Google Scholar

21. Eldar YC. Sampling Theory - Beyond Bandlimited Systems. Cambridge: Cambridge University Press (2015).

Google Scholar

22. Mara A, Jung A. Recovery conditions and sampling strategies for network lasso. In: Proceedings of Asilomar Conference Signals, Systems, Computers. Pacific Grove, CA (2017).

Google Scholar

23. Basirian S, Jung A. Random walk sampling for big data over networks. In: Proceedings of International Conference on Sampling Theory and Applications (SampTA). Tallinn (2017). p. 427–431.

Google Scholar

24. Gleich D. MatlabBGL - A Matlab Graph Library Cambridge, MA: The MIT Press (2007).

25. Jung, A, Heimowitz, A, and Eldar, YC The network nullspace property for compressed sensing over networks. In: Proceedings of International Conference on Sampling Theory and Applications (SampTA). Tallinn (2017).

Google Scholar

26. Jung A, Hulsebos M. The network nullspace property for compressed sensing of big data over networks. arXiv (2017).

Google Scholar

27. Nam S, Davies M, Elad M, Gribonval R. The cosparse analysis model and algorithms. Appl Comput Harmon Anal. (2013) 34:30–56. doi: 10.1016/j.acha.2012.03.006

CrossRef Full Text | Google Scholar

28. Kabanava M, Rauhut H. Cosparsity in compressed sensing. In: Boche H, Calderbank R, Kutyniok G, Vybiral J, editors, Compressed Sensing and Its Applications. New York, NY: Springer (2015). p. 315–39.

Google Scholar

29. Kabanava M, Rauhut H. Analysis ℓ₁-recovery with frames and gaussian measurements. Acta Appl Math. (2015) 140:173–95. doi: 10.1007/s10440-014-9984-y

CrossRef Full Text | Google Scholar

30. Tellegen BDH. A general network theorem with applications. Philips Res Rep. (1952) 7:259–69.

Google Scholar

Appendix

The proofs for Theorem 3 and Theorem 4 rely on recognizing the recovery problem (1) as an analysis ℓ₁-minimization problem [27]. A sufficient condition for analysis ℓ₁-minimization to deliver the correct solution x[·] is given by the analysis nullspace property [27, 28]. In particular, the sampling set ${M}$ is said to satisfy the stable analysis nullspace property w.r.t. an edge set ${S}\subseteq {E}$ if

$\begin{array}{ll}\left|\right|u\left[·\right]|{|}_{{E}\backslash {S}}\ge \kappa |\left|u\left[·\right]\right|{|}_{{S}}\mathrm{\text{for any }}u\left[·\right]\in {K}({M}),& (9)\end{array}$

for some constant κ > 1.

Lemma 5. Consider a data graph ${G}$ and fixed partitioning ${F}=\left\{{{C}}_1,\dots ,{{C}}_{\left|{F}\right|}\right\}$ of its nodes into $\left|{F}\right|$ clusters ${{C}}_l$. We observe a clustered graph signal x[·] with $x\left[i\right]=\sum _{l=1}^{\left|{F}\right|}{a}_l{{I}}_{{{C}}_l}\left[i\right]$ at the sampled nodes ${M}\subseteq {V}$. If (9) holds for ${S}=\partial {F}$, then (1) has a unique solution given by x[·].

Proof. Consider a graph signal $\widehat{x}\left[·\right]$, which is different from the true underlying graph signal x[·], being feasible for (1), i.e, $\widehat{x}\left[i\right]=x\left[i\right]$ for all sampled nodes $i\in {M}$. Then, the difference $u\left[i\right]:=\widehat{x}\left[i\right]-x\left[i\right]$ belongs to the kernel ${K}({M})$ (cf. (4)). Note that, since x[i] is constant for all nodes $i\in {{C}}_l$ in the same cluster,

$\begin{array}{ll}\widehat{x}\left[i\right]-\widehat{x}\left[j\right]=u\left[i\right]-u\left[j\right]\mathrm{\text{, for any edge}}\left\{i,j\right\}\in {E}\backslash \partial {F}.& (10)\end{array}$

By the triangle inequality,

$\begin{array}{l}\left|\right|\widehat{x}\left[·\right]|{|}_{\partial {F}}\ge |\left|x\left[·\right]\right|{|}_{\partial {F}}-\left|\right|u\left[·\right]|{|}_{\partial {F}}=|\left|x\left[·\right]\right|{|}_{\text{TV}}-\left|\right|u\left[·\right]|{|}_{\partial {F}},\end{array}$

and, since $\left|\right|\widehat{x}\left[·\right]|{|}_{\text{TV}}=|\left|\widehat{x}\left[·\right]\right|{|}_{\partial {F}}+\left|\right|\widehat{x}\left[·\right]|{|}_{{E}\backslash \partial {F}}$, in turn

$\begin{array}{l}\left|\right|\widehat{x}\left[·\right]|{|}_{\text{TV}}=|\left|\widehat{x}\left[·\right]\right|{|}_{\partial {F}}+\left|\right|\widehat{x}\left[·\right]\left|{|}_{{E}\backslash \partial {F}}\stackrel{(10)}{=}\right|\left|\widehat{x}\left[·\right]\right|{|}_{\partial {F}}+u\left[·\right]|{|}_{{E}\backslash \partial {F}}\\ \ge \left|\right|x\left[·\right]|{|}_{\text{TV}}-|\left|u\left[·\right]\right|{|}_{\partial {F}}+\left|\right|u\left[·\right]\left|{|}_{{E}\backslash \partial {F}}\stackrel{(9)}{>}\right|\left|x\left[·\right]\right|{|}_{\text{TV}}.\end{array}$

Thus, we have shown that any graph signal $\widehat{x}\left[·\right]$ which is different from the true underlying graph signal x[·] but coincides with it at all sampled nodes $i\in {M}$, must have a larger TV norm than the true signal x[·] and therefore cannot be optimal for the problem (1). □

The next result extends Lemma 5 to graph signals $x\left[·\right]\in {ℝ}^{{V}}$ which are not exactly clustered, but which can be well approximated by a clustered signal of the form (2).

Lemma 6. Consider a data graph ${G}$ and a fixed partition ${F}=\left\{{{C}}_1,\dots ,{{C}}_{\left|{F}\right|}\right\}$ of its nodes into disjoint clusters ${{C}}_l\subseteq {V}$. We observe a graph signal $x\left[·\right]\in {ℝ}^{{V}}$ at the sampling set ${M}\subseteq {V}$. If (9) holds for ${S}=\partial {F}$ and κ = 2, any solution $\widehat{x}\left[·\right]$ of (1) satisfies

$\begin{array}{ll}\left|\right|x\left[·\right]-\widehat{x}\left[·\right]|{|}_{\text{TV}}\le 6{\displaystyle \underset{a_l\in ℝ}{min}}|x\left[·\right]-{\displaystyle \sum _{l=1}^{\left|{F}\right|}}{a}_l{{I}}_{{{C}}_l}\left[·\right]{|}_{\text{TV}}.& (11)\end{array}$

Proof. The argument closely follows the proof of [29, Theorem 8]. First note that any solution $\widehat{x}\left[·\right]$ of (1) obeys

$\begin{array}{ll}\left|\right|\widehat{x}\left[·\right]|{|}_{\text{TV}}\le |\left|x\left[·\right]\right|{|}_{\text{TV}},& (12)\end{array}$

since x[·] is trivially feasible for (1). From (12), we have

$\begin{array}{ll}\left|\right|\widehat{x}\left[·\right]|{|}_{{S}}+|\left|\widehat{x}\left[·\right]\right|{|}_{{E}\backslash {S}}\le \left|\right|x\left[·\right]|{|}_{{S}}+|\left|x\left[·\right]\right|{|}_{{E}\backslash {S}}.& (13)\end{array}$

Since $\widehat{x}\left[·\right]$ is feasible for (1), i.e., $\widehat{x}\left[i\right]=x\left[i\right]$ for every sampled node $i\in {M}$, the difference $v\left[·\right]:=\widehat{x}\left[·\right]-x\left[·\right]$ belongs to ${K}({M})$ (cf. (4)). Applying the triangle inequality to (13),

$\begin{array}{ll}\left|\right|v\left[·\right]|{|}_{{E}\backslash {S}}\le |\left|v\left[·\right]\right|{|}_{{S}}+2\left|\right|x\left[·\right]|{|}_{{E}\backslash {S}}.& (14)\end{array}$

Combining (14) with (9) (for the signal u[·] = v[·]),

$\begin{array}{ll}\left|\right|v\left[·\right]|{|}_{{E}\backslash {S}}\le 4|\left|x\left[·\right]\right|{|}_{{E}\backslash {S}}.& (15)\end{array}$

Using (9) again,

$\begin{array}{l}\left|\right|x\left[·\right]-\widehat{x}\left[·\right]|{|}_{\text{TV}}=|\left|v\left[·\right]\right|{|}_{\text{TV}}=\left|\right|v\left[·\right]|{|}_{{S}}+|\left|v\left[·\right]\right|{|}_{{E}\backslash {S}}\\ \stackrel{(9)}{\le }(3/2)\left|\right|v\left[·\right]\left|{|}_{{E}\backslash {S}}\stackrel{(15)}{\le }6\right|\left|x\left[·\right]\right|{|}_{{E}\backslash {S}}.\end{array}$

For any clustered graph signal x_c[·] of the form $x_{\text{c}}\left[i\right]=\sum _{l=1}^{\left|{F}\right|}{a}_l{{I}}_{{{C}}_l}\left[i\right]$, we have x_c[i]−x_c[j] = 0 for any $\left\{i,j\right\}\in {E}\backslash {S}$ (note that ${S}=\partial {F}$) and, in turn,

$\begin{array}{l}\left|\right|x\left[·\right]+x_{\text{c}}\left[·\right]|{|}_{\text{TV}}=||x\left[·\right]+x_{\text{c}}\left[·\right]|{|}_{{E}\backslash {S}}+\left|\right|x\left[·\right]+x{\left[·\right]}_c|{|}_{{S}}\\ \ge \left|\right|x\left[·\right]+x{\left[·\right]}_c|{|}_{{E}\backslash {S}}=|\left|x\left[·\right]\right|{|}_{{E}\backslash {S}}.\end{array}$

□

Let us now render Lemma 5 and Lemma 6 for clustered graph signals x[·] of the form (2) by stating a condition on the graph topology and sampling set ${M}$ which ensures (9).

Lemma 7. If a sampling set ${M}$ satisfies NNSP-${F}$, then it also satisfies the stable analysis nullspace property (9).

Proof. Consider a signal $u\left[·\right]\in {K}({M})$ which vanishes at all sampled nodes, i.e., $u\left[i\right]=0\mathrm{\text{for all}}i\in {M}$. We will now show that $\left|\right|u\left[·\right]|{|}_{{E}\backslash \partial {F}}\ge 2|\left|u\left[·\right]\right|{|}_{\partial {F}}$.

Let us assume that for each boundary edge $e\in \partial {F}$, the flow f[·] in Definition 2 has the same sign as u[e⁺]−u[e⁻]. We are allowed to assume this since according to Definition 2, if there exists a flow with f[e′] > 0 for a boundary edge $e^{\prime }\in \partial {F}$, there is another flow $\tilde{f}\left[·\right]$ with $\tilde{f}\left[e^{\prime }\right]=-f\left[e^{\prime }\right]$ for the same edge $e^{\prime }\in \partial {F}$, but otherwise identical to f[·], i.e., $\tilde{f}\left[e\right]=f\left[e\right]$ for all $e\in {E}\backslash \left\{e^{\prime }\right\}$.

Next, we construct an augmented graph ${{G}}^{\prime }$ by adding an extra node s to the data graph ${G}$ which is connected to all sampled nodes $i\in {M}$ via an edge e_i = {s, i} which is oriented such that $e_i^{+}=s$. We assign to each edge e_i = {s, i} the flow f[e_i] = g[i] (cf. (5)). It can be verified easily that the flow over the augmented graph has zero demands for all nodes. Thus, we can apply Tellegen's theorem [30] to obtain $\left|\right|u\left[·\right]|{|}_{{E}\backslash \partial {F}}\ge 2|\left|u\left[·\right]\right|{|}_{\partial {F}}$. □

We obtain Theorem 3 by combining Lemma 7 with Lemma 5. In order to verify Theorem 4 we note that, by Lemma 7, the NNSP according to Definition 2 implies the stable nullspace property (9) for ${S}=\partial {F}$. Therefore, we can invoke Lemma 6 to reach (8).