Structured Sparsity of Convolutional Neural Networks via Nonconvex Sparse Group Regularization

Abstract

Convolutional neural networks (CNN) have been hugely successful recently with superior accuracy and performance in various imaging applications, such as classification, object detection, and segmentation. However, a highly accurate CNN model requires millions of parameters to be trained and utilized. Even to increase its performance slightly would require significantly more parameters due to adding more layers and/or increasing the number of filters per layer. Apparently, many of these weight parameters turn out to be redundant and extraneous, so the original, dense model can be replaced by its compressed version attained by imposing inter- and intra-group sparsity onto the layer weights during training. In this paper, we propose a nonconvex family of sparse group lasso that blends nonconvex regularization (e.g., transformed , , and ) that induces sparsity onto the individual weights and regularization onto the output channels of a layer. We apply variable splitting onto the proposed regularization to develop an algorithm that consists of two steps per iteration: gradient descent and thresholding. Numerical experiments are demonstrated on various CNN architectures showcasing the effectiveness of the nonconvex family of sparse group lasso in network sparsification and test accuracy on par with the current state of the art.

1 Introduction

Deep neural networks (DNNs) have proven to be advantageous for numerous modern computer vision tasks involving image or video data. In particular, convolutional neural networks (CNNs) yield highly accurate models with applications in image classification [28, 39, 77, 95], semantic segmentation [13, 49], and object detection [30, 72, 73]. These large models often contain millions of weight parameters that often exceed the number of training data. This is a double-edged sword since on one hand, large models allow for high accuracy, while on the other, they contain many redundant parameters that lead to overparametrization. Overparametrization is a well-known phenomenon in DNN models [6, 17] that results in overfitting, learning useless random patterns in data [96], and having inferior generalization. Additionally, these models also possess exorbitant computational and memory demands during both training and inference. Consequently, they may not be applicable for devices with low computational power and memory.

Resolving these problems requires compressing the networks through sparsification and pruning. Although removing weights might affect the accuracy and generalization of the models, previous works [25, 54, 66, 81] demonstrated that many networks can be substantially pruned with negligible effect on accuracy. There are many systematic approaches to achieving sparsity in DNNs, as discussed extensively in Refs. 14 and 15.

Han et al. [26] proposed to first train a dense network, prune it afterward by setting the weights to zeroes if below a fixed threshold, and retrain the network with the remaining weights. Jin et al. [32] extended this method by restoring the pruned weights, training the network again, and repeating the process. Rather than pruning by thresholding, Aghasi et al. [1, 2] proposed Net-Trim, which prunes an already trained network layer by layer using convex optimization in order to ensure that the layer inputs and outputs remain consistent with the original network. For CNNs in particular, filter or channel pruning is preferred because it significantly reduces the amount of weight parameters required compared to individual weight pruning. Li et al. [43] calculated the sums of absolute weights of the filters of each layer and pruned the ones with the smallest sums. Hu et al. [29] proposed a metric called average percentage of zeroes for channels to measure their redundancies and pruned those with highest values for each layer. Zhuang et al. [105] developed discrimination-aware channel pruning that selects channels that contribute to the network’s discriminative power.

An alternative approach to pruning a dense network is learning a compressed structure from scratch. A conventional approach is to optimize the loss function equipped with either the or regularization, which drives the weights to zeroes or to very small values during training. To learn which groups of weights (e.g., neurons, filters, channels) are necessary, group regularization, such as group lasso [93] and sparse group lasso [76], are equipped to the loss function. Alvarez and Salzmann [4] and Scardapane et al. [75] applied group lasso and sparse group lasso to various architectures and obtained compressed networks with comparable or even better accuracy. Instead of sharing features among the weights as suggested by group sparsity, exclusive sparsity [104] promotes competition for features between different weights. This method was investigated by Yoon and Hwang [92]. In addition, they combined it with group sparsity and demonstrated that this combination resulted in compressed networks with better performance than their original counterparts. Non-convex regularization has also been examined. Louizos et al. [54] proposed a practical algorithm using probabilistic methods to perform regularization on CNNs. Ma et al. [61] proposed integrated transformed , a convex combination of transformed and group lasso, and compared its performance against the aforementioned group regularization methods.

In this paper, we propose a family of group regularization methods that balances both group lasso for group-wise sparsity and nonconvex regularization for element-wise sparsity. The family extends sparse group lasso by replacing the penalty term with a nonconvex penalty term. The nonconvex penalty terms considered are , , transformed , and SCAD. The proposed family is supposed to yield a more accurate and/or more compressed network than sparse group lasso since suffers various weaknesses due to being a convex relaxation of . We develop an algorithm to optimize loss functions equipped with the proposed nonconvex, group regularization terms for DNNs.

2 Model and Algorithm

2.1 Preliminaries

The most common regularizer used for DNNs is regularization , also known as weight decay. It prevents overfitting and improves generalization because it enforces the weights to decrease proportionally to their magnitudes [40]. Sparsity can be imposed by pruning weights whose magnitudes are below a certain threshold at each iteration during training. However, an alternative regularizer is the norm , also known as the lasso penalty [78]. The norm is the tightest convex relaxation of the penalty [20, 23, 82] and it yields a sparse solution that is found on the corners of the 1-norm ball [27, 52]. Theoretical results justify the norm’s ability to reconstruct sparse solution in compressed sensing. When a sensing matrix satisfies the restricted isometry property, the norm recovers the sparse solution exactly with high probability [11, 23, 82]. On the other hand, the null space property is a necessary and sufficient condition for minimization to guarantee exact recovery of sparse solutions [16, 23]. Being able to yield sparse solutions, the norm has gained popularity in other types of inverse problems such as compressed imaging [33, 57] and image segmentation [34, 35, 42] and in various fields of applications such as geoscience [74], medical imaging [33, 57], machine learning [10, 36, 67, 78, 89], and traffic flow network [91]. Unfortunately, element-wise sparsity by or regularization in CNNs may not yield meaningful speedup as the number of filters and channels required for computation and inference may remain the same [86].

To determine which filters or channels are relevant in each layer, group sparsity using the group lasso penalty [93] is considered. The group lasso penalty has been utilized in various applications, such as microarray data analysis [62], machine learning [7, 65], and EEG data [46]. Suppose a DNN has L layers, so the set of weight parameters W is divided into L sets of weights: . The weight set of each layer is divided into groups (e.g., channels or filters): . The group lasso penalty applied to is formulated aswhere corresponds to the weight parameter with index i in group g in layer l and the term denotes the number of weight parameters in group g in layer l. Because group sizes vary, the constant is multiplied in order to rescale the norm of each group with respect to the group size, ensuring that each group is weighed uniformly [65, 76, 93]. The group lasso regularizer imposes the norm on each group, forcing weights of the same groups to decrease altogether at every iteration during training. As a result, the groups of weights are pruned when their norms are negligible, resulting in a highly compact network compared to element-sparse networks.

As an alternative to group lasso that encourages feature sharing, exclusive sparsity [104] enforces the model weight parameters to compete for features, making the features discriminative for each class in the context of classification. The regularization for exclusive sparsity isNow, within each group, sparsity is enforced. Because exclusivity cannot guarantee the optimal features since some features do need to be shared, exclusive sparsity can be combined with group sparsity to form combined group and exclusive sparsity (CGES) [92]. CGES is formulated aswhere is a parameter for balancing exclusivity and sharing among features.

To obtain an even sparser network, element-wise sparsity and group sparsity can be combined and applied together to the training of DNNs. One regularizer that combines these two types of sparsity is the sparse group lasso penalty [76], which is formulated aswhereSparse group lasso simultaneously enforces group sparsity by having the regularizer and element-wise sparsity by having the norm. This regularizer has been used in machine learning [83], bioinformatics [48, 103], and medical imaging [47].

Figure 1 demonstrates the differences between lasso, group lasso, and sparse group lasso applied to a weight matrix connecting a 5-dimensional input layer to a 10-dimensional output layer. In white, the entries are zero’ed out; in gray; the entries are not. Unlike lasso, group lasso results in a more structured method of pruning since three of the five neurons can be zero’ed out. Combined with regularization on the individual weights, sparse group lasso allows for more weights in the remaining two neurons to be pruned.

FIGURE 1

2.2 Nonconvex Sparse Group Lasso

We recall that the norm is the tightest convex relaxation of the penalty, given bywherewhen applied to the weight set of layer l. The penalty is non-convex and discontinuous. In addition, any -regularized problem is NP-hard [23]. These properties make developing convergent and tractable algorithms for -regularized problems difficult, thereby making -regularized problems better alternatives to solve. However, the -regularized problems have been shown to recover better solutions in terms of sparsity and/or accuracy than do -regularized problems in various applications, such as compressed sensing [56], image restoration [8, 12, 19, 55, 102], MRI reconstruction [80], and machine learning [56, 94]. In particular, -regularized inverse problems were demonstrated to be more robust against Poisson noise than are -regualarized inverse problems [100].

A continuous alternative to the penalty is the SCAD penalty term [22, 58], given bywherefor and . This penalty term enjoys three properties – unbiasedness, sparsity, and continuity – while the norm, on the other hand, has only sparsity and continuity [22]. In linear and logistic regression, SCAD was shown to outperform in variable selection [22]. SCAD has been applied to wavelet approximation [5], bioinformatics [9, 84], and compressed sensing [64].

The transformed penalty term [68] also enjoys the properties of unbiasedness, sparsity, and continuity [58]. In fact, the regularizer is not just continuous but Lipschitz continuous [98]. The term is given bywhereIn addition, it interpolates the and penalties through the parameter a [98] becauseThe transformed penalty term was investigated and was shown to outperform in compressed sensing [79, 97, 98], deep learning [45, 61, 87], matrix completion [99], and epidemic forecasting [45].

Another Lipschitz continuous, nonconvex regularizer is the penalty given bywhere . In a series of works [50–52, 90], the penalty term with yields better solutions than does in various compressed sensing applications especially when the sensing matrix is highly coherent or it violates the restricted isometry property condition. To guarantee exact recovery of sparse solution, only requires a relaxed variant of the null space property [79]. Furthermore, is more robust against impulsive noise in yielding sparse, accurate solutions for inverse problems than is [44]. Besides compressed sensing, it has been utilized in image denoising and deblurring [53], image segmentation [71], image inpainting [63], and hyperspectral demixing [21]. In deep learning application, the regularization was used to learn permutation matrices [59] for ShuffleNet [60, 101].

Due to the advantages and recent successes of the aforementioned nonconvex regularizers, we propose to replace the norm in Eq. 5 with nonconvex penalty terms. Hence, we propose a family of group regularizers called nonconvex sparse group lasso. The family includes the following:Using these regularizers, we expect to obtain a sparser and/or more accurate network than from using the original sparse group lasso. The norm can also be replaced with other nonconvex penalties not mentioned in this paper. Refer to Refs. 3 and 85 to see other nonconvex penalties. However, we focus on the aforementioned nonconvex regularizers because they have closed-form proximal operators required by our proposed algorithm described in the next section.

2.3 Notations and Definitions

2.4 Numerical Optimization

We develop a general algorithm framework to solvewhere , is either or one of the nonconvex regularizers Eqs. 10–13, and is the corresponding sparsity-inducing regularizer. Throughout the paper, our assumption on Eq. 14 is the following:

Assumption 1. The function is continuously differentiable with respect to for each .

By introducing an auxiliary variable for (14), we have a constrained optimization problem:The constraints can be relaxed by adding the quadratic penalty terms with so that we haveWith β fixed, Eq. 16 can be solved by alternating minimization:To solve Eq. 17a, we simultaneously update for by gradient descentwhere is the learning rate and is the subdifferential of with respect to . In practice, Eq. 18 is performed using stochastic gradient descent (or one of its variants) with mini-batches due to the large-size computation dealing with the amount of data and weight parameters that a typical DNN has.

To update V, we see that Eq. 17b can be rewritten asThe proximal operators for the considered regularizers are thresholding functions as their closed-form solutions, and as a result, the V update simplifies to thresholding W. The regularization functions and their corresponding proximal operators are summarized in Table 1.

Incorporating the algorithm that solves the quadratic penalty problem Eq. 16, we now develop a general algorithm to solve Eq. 14. We solve a sequence of quadratic penalty problems Eq. 16 with where . This will yield a sequence so that , a solution to (14). This algorithm is based on the quadratic penalty method [69] and the penalty decomposition method [56]. The algorithm is summarized in Algorithm 1.

An alternative algorithm to solve Eq. 14 is proximal gradient descent [70]. By this method, the update for , is

Using this algorithm results in weight parameters with some already zero’ed out.

However, the advantage of our proposed algorithm lies in Eq. 17a, written more specifically asWe see that this step performs exact weight decay or regularization on weights whenever . On the other hand, when , the effect of regularization is mitigated on the corresponding weight based on the absolute difference . Using regularization was shown to give superior pruning results in terms of accuracy by Han et al. [26]. Our proposed algorithm can be perceived as an adaptive regularization method, where Eq. 17b identifies which weights to perform exact regularization on and Eq. 17a updates and regularizes the weights accordingly.

2.5 Convergence Analysis

To establish convergence for the proposed algorithm, the results below state that the accumulation point of the sequence generated by Eqs 17a and 17b is a block-coordinate minimizer, and an accumulation point generated by Algorithm 1 is a sparse feasible solution to (15). Proofs are provided in Section 5. Unfortunately, the feasible solution generated may not be a local minimizer of Eq. 15 because the loss function is nonconvex. However, it was shown in [18] that a similar algorithm to Algorithm 1, but for fixed β in a bounded interval, generates an approximate global solution with high probability for a one-layer CNN with ReLu activation function.

Theorem 2. Let be a sequence generated by the alternating minimization algorithm Eqs. 17a and 17b, where is , , transformed , , or SCAD. If is an accumulation point of , then is a block-coordinate minimizer of Eq. 16. that isTheorem 3. Let be a sequence generated by Algorithm 1. Suppose that is uniformly bounded. If is an accumulation point of , then is a feasible solution to Eq. 15, that is .

Remark: To safely ensure that is uniformly bounded in practice, we can find a feasible solution to (15) and impose a bound M such thatIf , then we set . This strategy is based on Ref. 56. However, in our numerical experiments, we have not yet encountered to diverge.

3 Numerical Experiments

3.1 Application to Deep Neural Networks

We compare the proposed nonconvex sparse group lasso against four other methods as baselines: group lasso, sparse group lasso (), CGES proposed in Ref. 92, and the group variant of regularization (denoted as for simplicity) proposed in Ref. 54. is optimized using the same algorithm proposed for nonconvex sparse group lasso. For the group terms, the weights are grouped together based on the filters or output channels, which we will refer to as neurons. We trained various CNN architectures on MNIST [41] and CIFAR 10/100 [38]. The MNIST dataset consists of 60k training images and 10k test images. MNIST is trained on two simple CNN architectures: LeNet-5-Caffe [31, 41] and a 4-layer CNN with two convolutional layers (32 and 64 channels, respectively) and an intermediate layer of 1000 fully connected neurons. CIFAR 10/100 is a dataset that has 10/100 classes split into 50k training images and 10k test images. It is trained on Resnets [28] and wide Resnets [95]. Throughout all of our experiments, for , we set as suggested in [22]; for , we set as suggested in Ref. 99; and for , we set as suggested by the literatures [50–52, 90]. For CGES, we have . Because the optimization algorithms do not drive most, if not all, the weights and neurons to zeroes, we have to set them to zeroes when their values are below a certain threshold. In our experiments, if the absolute weights are below , we set them to zeroes. Then, weight sparsity is defined to be the percentage of zero weights with respect to the total number of weights trained in the network. If the normalized sum of the absolute values of the weights of the neuron is less than , then the weights of the neuron are set to zeroes. Neuron sparsity is defined to be the percentage of neurons whose weights are zeroes with respect to the total number of neurons in the network.

3.1.1 MNIST Classification

MNIST is trained on Lenet-5-Caffe, which has four layers with 1,370 total neurons and 431,080 total weight parameters. All layers of the network are applied with strictly the same type of regularization. No other regularization methods (e.g., dropout and batch normalization) are used. The network is optimized using Adam [37] with initial learning rate 0.001. For every 40 epochs, the learning rate decays by a factor of 0.1. We set the regularization parameter to the following values: for . For and nonconvex sparse group lasso, we set , and for every 40 epochs, β increases by a factor of . The network is trained for 200 epochs across 5 runs.

Table 2 reports the mean results for test error, weight sparsity, and neuron sparsity across five runs of Lenet-5-Caffe trained after 200 epochs. We see that although CGES has the lowest test errors at and the largest weight sparsity for all , nonconvex sparse group lasso’s test errors and weight sparsity are comparable. Additionally, nonconvex sparse group lasso’s neuron sparsity is nearly two times larger than the neuron sparsity attained by CGES. Across all parameters and methods, SGL₀ with attains the best average test error of 0.630 with average weight sparsity 95.7% and neuron sparsity 80.7%. Furthermore, its test error is lower than the test errors of other nonconvex sparse group lasso regularization methods for all α’s tested. Generally, and nonconvex sparse group lasso outperform regularization proposed by Louizos et al. [54] and group lasso by average weight and neuron sparsity.

Table 3 reports the mean results for test error, weight sparsity, and neuron sparsity of the Lenet-5-Caffe models with the lowest test errors from the five runs. According to the results, the best test errors are attained by at ; at ; and CGES at . For average weight sparsity, attains the largest weight sparsity at . For average neuron sparsity, the largest values are attained by at ; by at ; and by at . Although does not outperform all the other methods across the board, its results are still comparable to the best results. Overall, we see that nonconvex sparse group lasso outperforms in test error, weight sparsity, and neuron sparsity and group lasso in weight and neuron sparsity.

MNIST is also trained on a 4-layer CNN with two convolutional layers with 32 and 64 channels, respectively, and an intermediate layer with 1000 neurons. Each convolutional layer has a convolutional filters. The 4-layer CNN has 2,120 total neurons and 1,087,010 total weight parameters. All layers of the network are applied with strictly the same type of regularization. The network is optimized with the same settings as Lenet-5-Caffe. However, the regularization parameter is different: we have for . For and nonconvex sparse group lasso, we set and for every 40 epochs, β increases by a factor of . The network is trained for 200 epochs across 5 runs.

Table 4 reports the mean results for test error, weight sparsity, and neuron sparsity across five runs of the 4-layer CNN models trained after 200 epochs. Although CGES consistently has the highest weight sparsity, it does not yield the most accurate models until when . Moreover, its neuron sparsity is smaller than the neuron sparsity by group lasso, , and nonconvex group lasso when . has the highest neuron sparsity for all α’s given, but its test errors are much greater. When , yields the most accurate models at while yields one at . Overall, we see that nonconvex group lasso has comparable weight sparsity and neuron sparsity as group lasso and .

Table 5 reports the mean results for test error, weight sparsity, and neuron sparsity of the 4-layer CNN models with the lowest test errors from the five runs. At , and have the lowest test errors, but their weight sparsity are exceeded by CGES and their neuron sparsity are exceeded by . At , has the lowest test error, but its weight sparsity and neuron sparsity are exceeded by CGES and , respectively. At , has the lowest test error, but has the largest weight sparsity with comparable test error. At , CGES has the lowest test error, but its weight sparsity is exceeded by group lasso, , and the nonconvex group lasso regularizers, which all have slightly higher test error. At , the neuron sparsity of CGES is comparable to the neuron sparsity of group lasso, , and the nonconvex group lasso regularizers. At , group lasso has the highest neuron sparsity, but nonconvex group lasso has slightly lower neuron sparsity. In general, weight sparsity of nonconvex group lasso is comparable to or larger than the weight sparsity of group lasso and .

3.1.2 CIFAR Classification

CIFAR 10/100 is trained on Resnet-40 and wide Resnet with depth 28 and width 10 (WRN-28-10). Resnet-40 has approximately 570,000 weight parameters and 1520 neurons while WRN-28-10 has approximately 36,500,000 weight parameters and 10,736 neurons. The networks are optimized using stochastic gradient descent with initial learning rate 0.1. After every 60 epochs, learning rate decays by a factor of 0.2. Strictly the same type of regularization is applied to the weights of the hidden layer where dropout is utilized in the residual block. We vary the regularization parameter . For Resnet-40, we have for CIFAR 10 and for CIFAR 100. For and nonconvex sparse group lasso, we set for Resnet-40 and for WRN-28-10. For every 20 epochs, β increases by a factor of . The networks are trained for 200 epochs across 5 runs. We excluded regularization by Louizos et al. [54] because it was unstable for the provided α’s. Furthermore, we only analyze the models with the lowest test errors since the test errors did not stabilize by the end of the 200 epochs in our experiments.

Table 6 reports mean test error, weight sparsity, and neuron sparsity across the Resnet-40 models trained on CIFAR 10 with the lowest test errors from the five runs. Group lasso has the lowest test errors for all α’s provided while CGES, , and nonconvex sparse group lasso are higher by at most 1.1%. When , CGES has the largest weight sparsity while , have larger weight sparsity than does group lasso. At , has the largest weight sparsity. At , has the largest weight sparsity with comparable test error as the nonconvex group lasso regularizers. For neuron sparsity, has the largest at while has the largest at . However, at , group lasso has the largest neuron sparsity. For all α’s tested, has higher weight sparsity and neuron sparsity than does but with comparable test error.

Table 7 reports mean test error, weight sparsity, and neuron sparsity across the Resnet-40 models trained on CIFAR 100 with the lowest test errors from the five runs. Group lasso has the lowest test errors for while CGES has the lowest test error at . However, the weight sparsity and the neuron sparsity of group lasso are lower than the sparsity of and some of the nonconvex sparse group lasso regularizers. CGES has the lowest neuron sparsity across all α’s. Among the nonconvex group lasso penalties, has the best test errors, which are lower than the test errors of for all α’s except 2.5.

Table 8 reports mean test error, weight sparsity, and neuron sparsity across the WRN-28-10 models trained on CIFAR 10 with the lowest test errors from the five runs. The best test errors are attained by at ; by CGES at ; and by at . Weight sparsity of CGES outperforms the other methods only when , but it underperforms when . Weight sparsity levels between group lasso and nonconvex group lasso are comparable across all α. For neuron sparsity, attains the largest values at . Nevertheless, the other nonconvex sparse group lasso methods have comparable neuron sparsity. Overall, , , , and outperform group lasso in test error while having similar or higher weight and neuron sparsity.

Table 9 reports mean test error, weight sparsity, and neuron sparsity across the WRN-28-10 models trained on CIFAR 100 with the lowest test errors from the five runs. According to the results, the best test errors are attained by CGES when ; by when ; and by when . Although CGES has the largest weight sparsity for , we see that its test error increases as α increases. When , the best weight sparsity is attained by , but the other methods have comparable weight sparsity. The best neuron sparsity is attained by CGES at ; by at ; and by at . The neuron sparsity among the nonconvex sparse group lasso methods are comparable. For , we see that and nonconvex sparse group lasso outperform group lasso in test error across α while having comparable weight and neuron sparsity.

3.2 Algorithm Comparison

We compare the proposed Algorithm 1 with direct stochastic gradient descent, where the gradient of the regularizer is approximated by backpropagation, and proximal gradient descent, discussed in Section 2.4, by applying them to on Lenet-5 trained on MNIST. The parameter setting for this CNN is discussed in Section 3.1.1. Table 10 reports the mean results for test error, weight sparsity, and neuron sparsity across five models trained after 200 epochs while Figure 2 provides visualizations. Table 11 and Figure 3 record mean statistics for models with the lowest test errors from the five runs. According to the results, proximal stochastic gradient descent attains the highest level of weight sparsity and neuron sparsity for models trained after 200 epochs and models with the lowest test error. However, their test errors are the highest among the three algorithms. On the other hand, our proposed algorithm attains the lowest test errors. For models trained after 200 epochs, the weight sparsity and neuron sparsity attained by Algorithm 1 are comparable to the sparsity attained by direct stochastic gradient descent. For models with the lowest test errors generated from their respective runs, the weight sparsity and neuron sparsity by the proposed algorithm are better than the sparsity by direct stochastic gradient descent. Therefore, our proposed algorithm generates the most accurate model with satisfactory sparsity among the three algorithms for sparse regularization.

FIGURE 2

FIGURE 3

4 Conclusion and Future Work

In this work, we propose nonconvex sparse group lasso, a nonconvex extension of sparse group lasso. The norm in sparse group lasso on the weight parameters is replaced with a nonconvex regularizer whose proximal operator is a thresholding function. Taking advantage of this property, we develop a new algorithm to optimize loss functions regularized with nonconvex sparse group lasso for CNNs in order to attain a sparse network with competitive accuracy. We compare the proposed family of regularizers with various baseline methods on MNIST and CIFAR 10/100 on different CNNs. The experimental results demonstrate that in general, nonconvex sparse group lasso generates a more accurate and/or more compressed CNN than does group lasso. In addition, we compare our proposed algorithm to direct stochastic gradient descent and proximal gradient descent on Lenet-5 trained on MNIST. The results show that the proposed algorithm to solve yields a satisfactorily sparse network with lower test error than do the other two algorithms.

According to the numerical results, there is no single sparse regularizer that outperforms all other on any CNN trained on a given dataset. One regularizer may perform well in one case while it may perform worse on a different case. Due to the myriad of sparse regularizers to select from and the various parameters to tune, especially for one CNN trained on a given dataset, one direction is to develop an automatic machine learning framework that efficiently selects the right regularizer and parameters. In recent works, automatic machine learning can be represented as a matrix completion problem [88] and a statistical learning problem [24]. These frameworks can be adapted for selecting the best sparse regularizer, thus saving time for users who are training sparse CNNs.

5 Proofs

We provide proofs for the results discussed in Section 2.5.

5.1 Proof of Theorem 2

By Eqs 17a and 17b, for each , we havefor all W, andfor all V. By Eq. 23, we havefor each . Altogether, we havefor each , so is nonincreasing. Since for all , its limit exists. From Eqs. 22–24, we haveTaking the limit gives usSince is an accumulation point of , there exists a subsequence K such thatBecause is lower semicontinuous and , there exists such that implies for each . Using this result along with Eq. 23, we obtainfor . As , we haveby continuity, so it follows that .

For notational convenience, letBy Eq. 22, we haveBecause exists, the sequence is bounded. If is , transformed , or SCAD, then is bounded. If is , then is coercive. If is , then is bounded above by . Overall, this follows that bounded as well. Hence, there exists a further subsequence such that exists. So, we obtainafter applying Eq. 26 in the third inequality and by continuity in the last equality.

Taking the limit over the subsequence in Eq. 30 and applying Eq. 31, we obtainby continuity. Adding on both sides yieldswhich follows that . This completes the proof.

5.2 Proof of Theorem 3

Because is an accumulation point, there exists a subsequence K such that . If is uniformly bounded, there exists M such that for all . Then we haveAs a result,Taking the limit over , we havewhich follows that . As a result, is a feasible solution to Eq. 15.

References

• 1.

  AghasiAAbdiANguyenNRombergJ. Net-trim: convex pruning of deep neural networks with performance guarantee. In: Advances in Neural Information Processing Systems; 2017 Nov 23; Long Beach, CA. Pasadena, CA: NeurIPS (2017). p. 3177–86. 10.5555/3294996.3295077

  • CrossRef
  • Google Scholar

• 2.

  AghasiAAbdiARombergJ. Fast convex pruning of deep neural networks. SIAM J Math Data Sci (2020). 2:158–188. 10.1137/19m1246468

  • CrossRef
  • Google Scholar

• 3.

  AhnMPangJ-SXinJ. Difference-of-convex learning: directional stationarity, optimality, and sparsity. SIAM J Optim. (2017). 27:1637–1665. 10.1137/16m1084754

  • CrossRef
  • Google Scholar

• 4.

  AlvarezJMSalzmannM. Learning the number of neurons in deep networks In: Advances in Neural Information Processing Systems; 2018 Oct 11; Barcelona, Spain. Pasadena, CA: NeurIPS (2016). p. 2270–8.

  • Google Scholar

• 5.

  AntoniadisAFanJ. Regularization of wavelet approximations. J Am Stat Assoc. (2001). 96:939–67. 10.1198/016214501753208942

  • CrossRef
  • Google Scholar

• 6.

  BaJCaruanaR. Do deep nets really need to be deep?Adv Neural Inf Process Syst. (2014). 2:2654–62. 10.5555/2969033.2969123

  • CrossRef
  • Google Scholar

• 7.

  BachFR. Consistency of the group lasso and multiple kernel learning. J Mach Learn Res. (2008). 9:1179–225. 10.5555/1390681.1390721

  • CrossRef
  • Google Scholar

• 8.

  BaoCDongBHouLShenZZhangXZhangX. Image restoration by minimizing zero norm of wavelet frame coefficients. Inverse Problems. (2016). 32:115004. 10.1088/0266-5611/32/11/115004

  • CrossRef
  • Google Scholar

• 9.

  BrehenyPHuangJ. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Ann Appl Stat. (2011). 5:232. 10.1214/10-aoas388

  • CrossRef
  • Google Scholar

• 10.

  CandèsEJLiXMaYWrightJ. Robust principal component analysis?J ACM. (2011). 58:1–37. 10.1145/1970392.1970395

  • CrossRef
  • Google Scholar

• 11.

  CandèsEJRombergJKTaoT. Stable signal recovery from incomplete and inaccurate measurements. Commun Pure Appl Math. (2006). 59:1207–23. 10.1002/cpa.20124

  • CrossRef
  • Google Scholar

• 12.

  ChanRHChanTFShenLShenZ. Wavelet algorithms for high-resolution image reconstruction. SIAM J Sci Comput. (2003). 24:1408–32. 10.1137/s1064827500383123

  • CrossRef
  • Google Scholar

• 13.

  ChenLCPapandreouGKokkinosIMurphyKYuilleAL (2018). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell.40, 834–84810.1109/TPAMI.2017.2699184

  • CrossRef
  • Google Scholar

• 14.

  ChengYWangDZhouPZhangT. A survey of model compression and acceleration for deep neural networks. Preprint repository name [Preprint] (2017). Available from: https://arxiv.org/abs/1710.09282.

  • Google Scholar

• 15.

  ChengYWangDZhouPZhangT. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Process Mag. (2018). 35:126–36. 10.1109/msp.2017.2765695

  • CrossRef
  • Google Scholar

• 16.

  CohenADahmenWDeVoreR. Compressed sensing and best k-term approximation. J Am Math Soc. (2009). 22:211–31. 10.1090/S0894-0347-08-00610-3

  • CrossRef
  • Google Scholar

• 17.

  DentonELZarembaWBrunaJLeCunYFergusR. Exploiting linear structure within convolutional networks for efficient evaluation. Adv Neural Inf Process Syst. (2014). 1:1269–77. 10.5555/2968826.2968968

  • CrossRef
  • Google Scholar

• 18.

  DinhTXinJ. Convergence of a relaxed variable splitting method for learning sparse neural networks via ℓ₁,ℓ₀, and transformed-ℓ₁ penalties. In: Proceedings of SAI Intelligent Systems Conference. Springer International Publishing (2020). p. 360–374.

  • Google Scholar

• 19.

  DongBZhangY. An efficient algorithm for ℓ₀ minimization in wavelet frame based image restoration. J Sci Comput. (2013). 54:350–68. 10.1007/s10915-012-9597-4

  • CrossRef
  • Google Scholar

• 20.

  DonohoDLEladM. Optimally sparse representation in general (nonorthogonal) dictionaries via ℓ₁ minimization. Proc Natl Acad Sci USA. (2003). 100:2197–202. 10.1073/pnas.0437847100

  • CrossRef
  • Google Scholar

• 21.

  EsserELouYXinJ. A method for finding structured sparse solutions to nonnegative least squares problems with applications. SIAM J Imag Sci (2013). 6:2010–46. 10.1137/13090540x

  • CrossRef
  • Google Scholar

• 22.

  FanJLiR. Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc. (2001). 96:1348–60. 10.1198/016214501753382273

  • CrossRef
  • Google Scholar

• 23.

  FoucartSRauhutH. An invitation to compressive sensing. A mathematical introduction to compressive sensing.New York, NY: Birkhäuser (2013). p. 1–39.

  • Google Scholar

• 24.

  GuptaRRoughgardenT. A pac approach to application-specific algorithm selection. SIAM J Comput. (2017). 46:992–1017. 10.1137/15m1050276

  • CrossRef
  • Google Scholar

• 25.

  HanSMaoHDallyWJ. Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding (2015). Available from: https://arxiv.org/abs/1510.00149.

  • Google Scholar

• 26.

  HanSPoolJTranJDallyW. Learning both weights and connections for efficient neural network. Adv Neural Inf Process Syst. (2015). 1:1135–43. 10.5555/2969239.2969366

  • CrossRef
  • Google Scholar

• 27.

  HastieTTibshiraniRFriedmanJ. The elements of statistical learning: data mining, inference, and prediction.New York, NY: Springer Science & Business Media (2009). 745 p.

  • Google Scholar

• 28.

  HeKZhangXRenSSunJ. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2016 Jun 27–30; Las Vegas, NV. New York, NY: IEEE (2016). p 770–8.

  • Google Scholar

• 29.

  HuHPengRTaiY-WTangC-K. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures (2016). Available from: https://arxiv.org/abs/1607.03250.

  • Google Scholar

• 30.

  HuangJRathodVSunCZhuMKorattikaraAFathiAet alSpeed/accuracy trade-offs for modern convolutional object detectors. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2017 Jul 21–26; Honolulu, HI. New York, NY: IEEE (2017). p. 7310–1.

  • Google Scholar

• 31.

  JiaYShelhamerEDonahueJKarayevSLongJGirshickRet alCaffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia (ACM); 2014 Jul 20; Berkeley, CA. Berkeley, CA: UC Berkeley EECS (2014). p. 675–8.

  • Google Scholar

• 32.

  JinXYuanXFengJYanS. Training skinny deep neural networks with iterative hard thresholding methods (2016). Available from: https://arxiv.org/abs/1607.05423.

  • Google Scholar

• 33.

  JungHYeJCKimEY. Improvedk-tBLAST and k-t SENSE using FOCUSS. Phys Med Biol. (2007). 52:3201. 10.1088/0031-9155/52/11/018

  • CrossRef
  • Google Scholar

• 34.

  JungM. Piecewise-Smooth image Segmentation models with L¹ data-fidelity Terms. J Sci Comput. (2017). 70:1229–61. 10.1007/s10915-016-0280-z

  • CrossRef
  • Google Scholar

• 35.

  JungMKangMKangM. Variational image segmentation models involving non-smooth data-fidelity terms. J Sci Comput. (2014). 59:277–308. 10.1007/s10915-013-9766-0

  • CrossRef
  • Google Scholar

• 36.

  KimCKlabjanD. A simple and fast algorithm for L1-norm kernel PCA. IEEE Trans Patt Anal Mach Intell. (2019). 42:1842–55. 10.1109/TPAMI.2019.2903505

  • CrossRef
  • Google Scholar

• 37.

  KingmaDPBaJ. Adam: a method for stochastic optimization. (2014). Available from: https://arxiv.org/abs/1412.6980.

  • Google Scholar

• 38.

  KrizhevskyAHintonG. Learning multiple layers of features from tiny images (2009). p. 60. Available from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.222.9220&rep=rep1&type=pdf.

  • Google Scholar

• 39.

  KrizhevskyASutskeverIHintonGE. Imagenet classification with deep convolutional neural networks. Commun ACM. (2012). 60:1097–105. 10.1145/3065386

  • CrossRef
  • Google Scholar

• 40.

  KroghAHertzJA. A simple weight decay can improve generalization. Adv Neural Inf Process Syst. (1992). 4:950–957. 10.5555/2986916.2987033

  • CrossRef
  • Google Scholar

• 41.

  LeCunYBottouLBengioYHaffnerP. Gradient-based learning applied to document recognition. Proc IEEE. (1998). 86:2278–324. 10.1109/5.726791

  • CrossRef
  • Google Scholar

• 42.

  LiFOsherSQinJYanM. A multiphase image segmentation based on fuzzy membership functions and l1-norm fidelity. J Sci Comput. (2016). 69:82–106. 10.1007/s10915-016-0183-z

  • CrossRef
  • Google Scholar

• 43.

  LiHKadavADurdanovicISametHGrafHP. Pruning filters for efficient convnets (2016). Available from: https://arxiv.org/abs/1608.08710.

  • Google Scholar

• 44.

  LiPChenWGeHNgMK. ℓ₁−αℓ₂ minimization methods for signal and image reconstruction with impulsive noise removal. Inv Problems. (2020). 36:055009. 10.1088/1361-6420/ab750c

  • CrossRef
  • Google Scholar

• 45.

  LiZLuoXWangBBertozziALXinJ. A study on graph-structured recurrent neural networks and sparsification with application to epidemic forecasting. World congress on global optimization.Cham, Switzerland: Springer (2019). p. 730–9.

  • Google Scholar

• 46.

  LimMAlesJMCottereauBRHastieTNorciaAM. Sparse EEG/MEG source estimation via a group lasso. PloS One. (2017). 12:e0176835. 10.1371/journal.pone.0176835

  • CrossRef
  • Google Scholar

• 47.

  LinDCalhounVDWangY-P. Correspondence between fMRI and SNP data by group sparse canonical correlation analysis. Med Image Anal. (2014). 18:891–902. 10.1016/j.media.2013.10.010

  • CrossRef
  • Google Scholar

• 48.

  LinDZhangJLiJCalhounVDDengH-WWangY-P. Group sparse canonical correlation analysis for genomic data integration. BMC bioinf. (2013). 14:1–16. 10.1186/1471-2105-14-245

  • CrossRef
  • Google Scholar

• 49.

  LongJShelhamerEDarrellT. Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition; 2015 Jun 7–15; Boston, MA. New York, NY: IEEE (2015). p. 3431–40.

  • Google Scholar

• 50.

  LouYOsherSXinJ. Computational aspects of constrained minimization for compressive sensing. Modelling, computation and optimization in information systems and management sciences.Cham, Switzerland: Springer (2015). p. 169–80.

  • Google Scholar

• 51.

  LouYYanM. Fast L1-L2 minimization via a proximal operator. J Sci Comput. (2018). 74:767–85. 10.1007/s10915-017-0463-2

  • CrossRef
  • Google Scholar

• 52.

  LouYYinPHeQXinJ. Computing sparse representation in a highly coherent dictionary based on difference of L₁ and L₂. J Sci Comput. (2015). 64:178–96. 10.1007/s10915-014-9930-1

  • CrossRef
  • Google Scholar

• 53.

  LouYZengTOsherSXinJ. A weighted difference of anisotropic and isotropic total variation model for image processing. SIAM J Imag Sci. (2015). 8:1798–823. 10.1137/14098435x

  • CrossRef
  • Google Scholar

• 54.

  LouizosCWellingMKingmaDP. Learning sparse neural networks through regularization (2017). Available from: https://arxiv.org/abs/1712.01312.

  • Google Scholar

• 55.

  LuJQiaoKLiXLuZZouY. ℓ₀-minimization methods for image restoration problems based on wavelet frames. Inverse Probl. (2019). 35:064001. 10.1088/1361-6420/ab08de

  • CrossRef
  • Google Scholar

• 56.

  LuZZhangY. Sparse approximation via penalty decomposition methods. SIAM J Optim. (2013). 23:2448–78. 10.1137/100808071

  • CrossRef
  • Google Scholar

• 57.

  LustigMDonohoDPaulyJM. Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn Reson Med. (2007). 58:1182–95. 10.1002/mrm.21391

  • CrossRef
  • Google Scholar

• 58.

  LvJFanY. A unified approach to model selection and sparse recovery using regularized least squares. Ann Stat. (2009). 37:3498–528. 10.1214/09-aos683

  • CrossRef
  • Google Scholar

• 59.

  LyuJZhangSQiYXinJ. Autoshufflenet: learning permutation matrices via an exact Lipschitz continuous penalty in deep convolutional neural networks. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. New York, NY: Association for Computing Machinery (2020). p. 608–16.

  • Google Scholar

• 60.

  MaNZhangXZhengH-TSunJ. “Shufflenet v2: practical guidelines for efficient CNN architecture design”. Computer Vision – ECCV 2018. Cham: Springer International Publishing (2018). p. 122–38.

  • Google Scholar

• 61.

  MaRMiaoJNiuLZhangP. Transformed ℓ₁ regularization for learning sparse deep neural networks. Neur Netw (2019). 119:286–9810.1016/j.neunet.2019.08.01

  • CrossRef
  • Google Scholar

• 62.

  MaSSongXHuangJ. Supervised group lasso with applications to microarray data analysis. BMC bioinf. (2007). 8:60. 10.1186/1471-2105-8-60

  • CrossRef
  • Google Scholar

• 63.

  MaT-HLouYHuangT-ZZhaoX-L. Group-based truncated model for image inpainting. In: IEEE international conference on image processing (ICIP); 2018 Feb 22; Beijing, China. New York, NY: IEEE (2017). p. 2079–83.

  • Google Scholar

• 64.

  MehranianARadHSRahmimAAyMRZaidiH. Smoothly clipped absolute deviation (SCAD) regularization for compressed sensing MRI using an augmented Lagrangian scheme. Magn Reson Imag. (2013). 31:1399–411. 10.1016/j.mri.2013.05.010

  • CrossRef
  • Google Scholar

• 65.

  MeierLVan De GeerSBühlmannP. The group lasso for logistic regression. J Roy Stat Soc B. (2008). 70:53–71. 10.1111/j.1467-9868.2007.00627.x

  • CrossRef
  • Google Scholar

• 66.

  MolchanovDAshukhaAVetrovD (2017). Variational dropout sparsifies deep neural networks. In Proceedings of the 34th International Conference on Machine Learning; Sydney, Australia. Sydney, NSW, Australia: JMLR. 2498–507.

  • Google Scholar

• 67.

  NieFWangHHuangHDingC. Unsupervised and semi-supervised learning via ℓ₁-norm graph. In: 2011 international conference on computer vision (IEEE) (2011). 2268–73.

  • Google Scholar

• 68.

  NikolovaM. Local strong homogeneity of a regularized estimator. SIAM J Appl Math. (2000). 61:633–58. 10.1137/s0036139997327794

  • CrossRef
  • Google Scholar

• 69.

  NocedalJWrightS. Numerical optimization.New York, NY: Springer Science & Business Media (2006). 651 p.

  • Google Scholar

• 70.

  ParikhNBoydS. Proximal algorithms. FNT Optimization. (2014). 1:127–239. 10.1561/2400000003

  • CrossRef
  • Google Scholar

• 71.

  ParkFLouYXinJ. A weighted difference of anisotropic and isotropic total variation for relaxed mumford-shah image segmentation. In: 2016 IEEE International Conference on Image Processing (ICIP); 2016 Sep 25–28; Phoenix, AZ. New York, NY: IEEE (2016). 4314 p.

  • Google Scholar

• 72.

  ParkhiOMVedaldiAZissermanA. Deep face recognition. In: Proceedings of the british machine vision conference. Cambridge, UK: BMVA Press (2015). p. 41.1–41.12.

  • Google Scholar

• 73.

  RenSHeKGirshickRSunJ. Faster R-CNN: Towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst (2015). 39:91–99. 10.1109/TPAMI.2016.2577031

  • CrossRef
  • Google Scholar

• 74.

  SantosaFSymesWW. Linear inversion of band-limited reflection seismograms. SIAM J Sci Stat Comput. (1986). 7:1307–30. 10.1137/0907087

  • CrossRef
  • Google Scholar

• 75.

  ScardapaneSComminielloDHussainAUnciniA. Group sparse regularization for deep neural networks. Neurocomputing. (2017). 241:81–9. 10.1016/j.neucom.2017.02.029

  • CrossRef
  • Google Scholar

• 76.

  SimonNFriedmanJHastieTTibshiraniR. A sparse-group lasso. J Comput Graph Stat. (2013). 22:231–45. 10.1080/10618600.2012.681250

  • CrossRef
  • Google Scholar

• 77.

  SimonyanKZissermanA. Very deep convolutional networks for large-scale image recognition (2015). Available from: https://arxiv.org/abs/1409.1556.

  • Google Scholar

• 78.

  TibshiraniR. Regression shrinkage and selection via the lasso. J Roy Stat Soc B. (1996). 58:267–88. 10.1111/j.2517-6161.1996.tb02080.x

  • CrossRef
  • Google Scholar

• 79.

  TranHWebsterC. A class of null space conditions for sparse recovery via nonconvex, non-separable minimizations. Res Appl Math. (2019). 3:100011. 10.1016/j.rinam.2019.100011

  • CrossRef
  • Google Scholar

• 80.

  TrzaskoJManducaABorischE. Sparse MRI reconstruction via multiscale L₀-continuation. In: 2007 IEEE/SP 14th workshop on statistical signal processing. New York, NY: IEEE (2007). p. 176–80.

  • Google Scholar

• 81.

  UllrichKMeedsEWellingM. Soft weight-sharing for neural network compression. Stat. (2017). 1050:9.

  • Google Scholar

• 82.

  VershyninR. High-dimensional probability: An introduction with applications in data science.Cambridge, UK: Cambridge University Press (2018). 296 p.

  • Google Scholar

• 83.

  VincentMHansenNR. Sparse group lasso and high dimensional multinomial classification. Comput Stat Data Anal. (2014). 71:771–86. 10.1016/j.csda.2013.06.004

  • CrossRef
  • Google Scholar

• 84.

  WangLChenGLiH. Group scad regression analysis for microarray time course gene expression data. Bioinformatics. (2007). 23:1486–94. 10.1093/bioinformatics/btm125

  • CrossRef
  • Google Scholar

• 85.

  WenFChuLLiuPQiuRC. A survey on nonconvex regularization-based sparse and low-rank recovery in signal processing, statistics, and machine learning. IEEE Access. (2018). 6:69883–906. 10.1109/access.2018.2880454

  • CrossRef
  • Google Scholar

• 86.

  WenWWuCWangYChenYLiH. Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems; 2016 Aug 12; Barcelona, Spain. NeurIPS. Red Hook, NY: Curran Associates Inc. (2016). p. 2074–82.

  • Google Scholar

• 87.

  XueFXinJ. Learning sparse neural networks via ℓ₀ and Tℓ₁ by a relaxed variable splitting method with application to multi-scale curve classification. World congress on global optimization.Cham, Switzerland: Springer (2019). p. 800–809.

  • Google Scholar

• 88.

  YangCAkimotoYKimDWUdellM. Oboe: Collaborative filtering for automl model selection. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. New York, NY: ACM (2019). p. 1173–183.

  • Google Scholar

• 89.

  YeQZhaoHLiZYangXGaoSYinTet alL1-Norm Distance minimization-based fast robust twin support vector κ-plane Clustering. IEEE Trans Neural Netw Learn Syst. (2018). 29:4494–503. 10.1109/TNNLS.2017.2749428

  • CrossRef
  • Google Scholar

• 90.

  YinPLouYHeQXinJ. Minimization of ℓ1-2 for Compressed Sensing. SIAM J Sci Comput. (2015). 37:A536–63. 10.1137/140952363

  • CrossRef
  • Google Scholar

• 91.

  YinPSunZJinW-LXinJ. ℓ₁-minimization method for link flow correction. Transp Res Part B Methodol. (2017). 104:398–408. 10.1016/j.trb.2017.08.006

  • CrossRef
  • Google Scholar

• 92.

  YoonJHwangSJ. Combined group and exclusive sparsity for deep neural networks. In: Proceedings of the 34th international conference on machine learning; Sydney, Australia. JMLR (2017). 3958–66.

  • Google Scholar

• 93.

  YuanMLinY. Model selection and estimation in regression with grouped variables. J Roy Stat Soc B. (2006). 68:49–67. 10.1111/j.1467-9868.2005.00532.x

  • CrossRef
  • Google Scholar

• 94.

  YuanX-TLiPZhangT. Gradient hard thresholding pursuit. J Mach Learn Res. (2017). 18:166–1. 10.5555/3122009.3242023

  • CrossRef
  • Google Scholar

• 95.

  ZagoruykoSKomodakisN. Wide residual networks (2016). Available from: https://arxiv.org/abs/1605.07146.

  • Google Scholar

• 96.

  ZhangCBengioSHardtMRechtBVinyalsO. Understanding deep learning requires rethinking generalization (2016). Available from: https://arxiv.org/abs/1611.03530.

  • Google Scholar

• 97.

  ZhangSXinJ. Minimization of transformed L-1 penalty: Closed form representation and iterative thresholding algorithms. Commun Math Sci. (2017). 15:511–37. 10.4310/cms.2017.v15.n2.a9

  • CrossRef
  • Google Scholar

• 98.

  ZhangSXinJ. Minimization of transformed L₁ penalty: theory, difference of convex function algorithm, and robust application in compressed sensing. Math Program. (2018). 169:307–36. 10.1007/s10107-018-1236-x

  • CrossRef
  • Google Scholar

• 99.

  ZhangSYinPXinJ. Transformed schatten-1 iterative thresholding algorithms for low rank matrix completion. Commun Math Sci. (2017). 15:839–62. 10.4310/cms.2017.v15.n3.a12

  • CrossRef
  • Google Scholar

• 100.

  ZhangXLuYChanT. A novel sparsity reconstruction method from Poisson data for 3d bioluminescence tomography. J Sci Comput. (2012). 50:519–35. 10.1007/s10915-011-9533-z

  • CrossRef
  • Google Scholar

• 101.

  ZhangXZhouXLinMSunJ. Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE (2018). p. 6848–56.

  • Google Scholar

• 102.

  ZhangYDongBLuZ. ℓ₀ minimization for wavelet frame based image restoration. Math Comput. (2013). 82:995–1015. 10.1090/S0025-5718-2012-02631-7

  • CrossRef
  • Google Scholar

• 103.

  ZhouHSehlMESinsheimerJSLangeK. Association screening of common and rare genetic variants by penalized regression. Bioinformatics. (2010). 26:2375. 10.1093/bioinformatics/btq448

  • CrossRef
  • Google Scholar

• 104.

  ZhouYJinRHoiS. Exclusive lasso for multi-task feature selection. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics; Montreal, Canada. JMLR: W&CP (2010). p. 988–95.

  • Google Scholar

• 105.

  ZhuangZTanMZhuangBLiuJGuoYWuQet alDiscrimination-aware channel pruning for deep neural networks. In: Advances in Neural Information Processing Systems; 2018 Dec 2–8; Sardinia, Italy. San Diego, CA: NeurIPS (2018). p. 875–86.

  • Google Scholar