Multi-Task Learning Based on Stochastic Configuration Networks

When the human brain learns multiple related or continuous tasks, it will produce knowledge sharing and transfer. Thus, fast and effective task learning can be realized. This idea leads to multi-task learning. The key of multi-task learning is to find the correlation between tasks and establish a fast and effective model based on these relationship information. This paper proposes a multi-task learning framework based on stochastic configuration networks. It organically combines the idea of the classical parameter sharing multi-task learning with that of constraint sharing configuration in stochastic configuration networks. Moreover, it provides an efficient multi-kernel function selection mechanism. The convergence of the proposed algorithm is proved theoretically. The experiment results on one simulation data set and four real life data sets verify the effectiveness of the proposed algorithm.

1 Introduction

In supervised machine learning, we often encounter situations that establishing models for several related tasks, such as searching cancer sites, identifying cancer types, judging cancer stages and so on, based on cancer image data. Generally, these tasks are undertaken separately, which we refer to as single-task supervised learning (STSL) in traditional machine learning (Ben-David and Schuller, 2003). These models do not consider the correlation among multiple tasks so some common information in model parameters or data features is lost. In particular, when the training sample size of a single task is insufficient, it is difficult for STSL to capture enough information, which results in poor generalization performance. Multi-task supervised learning (MTSL) provides a solution for such a situation. It improves the performance of each task by setting shared representations among related tasks (Baxter, 2000; Argyriou et al., 2007; Liu et al., 2017). In a sense, a very important reason why human beings can learn based on a small number of samples is that human beings can make full use of various senses to obtain enough information and synthesize relevant information. MTSL is one of the ways to realize this idea.

Classical MTSL can be roughly divided into two categories, namely, MTSL based on constraint sharing and MTSL based on parameter sharing. In relation to the first method, Argyriou et al. (2008) proposed the MTL-L₂₁ based on regularization strategies, that was achieved by adding a regularization term for all the tasks’ objective function coefficients on the cost function. But this method performs poorly when data features have the problem of collinearity. To reduce the impact of this problem, Chen et al. (2012) added a quadratic regularization term for all the tasks’ objective function coefficients based on MTL-L₂₁. In 2015, Duong et al. (2015) used L₂ distance to regularize the parameters in their multi-task neural networks, so that each task has similar but different model parameters. In 2017, Yang and Hospedales (2017) used the trace norm to implement Duong’s model. In 2019, Oliveira et al. (2019) attempted to conceive a group LASSO with asymmetric transference formulation in multi-task learning, looking for the best of both worlds in a framework that admits the overlap of groups. Since all of these MTSL methods need to learn sparse features, their performance is not ideal when the data has few features. The MTSL methods based on parameter sharing (Caruana, 1997; Jacob et al., 2009) are not affected by this problem. In 1997, Caruana (Caruana, 1997) proposed a MTSL method (MTL) based on backpropagation neural networks. He mirrored the correlation information by sharing the input and hidden layer neurons among different tasks. In 1998, Lecun et al. (1998) used convolutional neural networks, named as LeNet-5, for document recognition on the basis of MTL. Their results clearly demonstrated the advantages of training a recognizer at the word level, rather than training it on presegmented, hand-labeled, isolated characters. In 2018, Ma et al. (2018) proposed multi-gate mixture-of-experts (MMoE), which adapted the mixture-of-experts (MoE) structure to multi-task learning by sharing the expert submodels across all tasks, while also had a gating network trained to optimize each task. In 2021, Zhang et al. (Zhang et al., 2021) developed a programming framework, AutoMTL, which generates compact multi-task models given an arbitrary input backbone convolutional neural network model and a set of tasks. However, these methods have high computational complexity and poor learning performance when the training samples are insufficient.

To address the aforementioned problems, this paper proposes a MTSL method based on a constraint sharing framework of stochastic configuration networks (SCNs) proposed by Wang et al. (Wang and Li, 2017a; Wang and Li, 2017b) Instead of the complex gradient descent method for solving the weight parameters of hidden layer nodes in general neural networks, SCNs use a supervision mechanism to stochastically configure these parameters. This stochastic configuration mechanism greatly reduces the computational complexity. Inspired by this idea, we establish a multi-task supervised learning algorithm based on stochastic configuration radial basis networks (MTSL-SCRBN). The main contributions of this study are as follows.

1. We combine constraint sharing of SCNs and parameter sharing of MTSL organically. The shared parameters are stochastically configured under certain constraint, which has low computational complexity. At the same time, to improve learning performance, the radial basis functions (Powell, 1987; Broomhead and Lowe, 1988) with different scale parameters are used as the basis functions to replace the original sigmoid functions of SCNs.

2. Two types of difficult to choice hyper parameters of the proposed model, the scale parameters and the centers of the radial basis functions, are stochastically configured during the learning process.

The rest of the paper is organized as follows. In Section 2, we briefly review MTL-L₂₁, MTEN, MTL, and SCNs. Section 3 details our proposed algorithm MTSL-SCRBN and proves its convergence. The experimental results of these algorithms on one simulation data set and five real data sets are detailed in Section 4. Section 5 summarizes this paper.

2 Related Work

Firstly, we introduce some notations. Suppose that there are M supervised learning tasks. The samples of the m-th task are given by,

$\left\{\left({\mathbf{x}}_1^m,y_1^m\right),\dots ,\left({\mathbf{x}}_{N_m}^m,y_{N_m}^m\right)\right\},(1)$

where ${\mathbf{x}}_i^m={[x_{i,1}^m,\dots ,x_{i,d}^m]}^{\text{T}}\in R^d$, $y_i^m\in R$, i = 1, …, N_m, m = 1, …, M, and T means transpose transform.

2.1 Multi-Task Learning Methods Based on Constraint Sharing

Inspired by group sparsity, Argyrios et al. (Argyriou et al., 2008) proposed the MTL-L₂₁ method to learn the correlation among multiple tasks under a regularization strategy. It can be described as the following optimization problem,

$V_{MTL-L_{21}}^{\ast }=\underset{V\in R^{d\times M}}{\mathrm{arg\; min}}\sum _{m=1}^M\Vert X^m{\mathbf{v}}^m-{\mathbf{y}}^m{\Vert }_F^2+\lambda \Vert V{\Vert }_{\mathrm{2,1}},$

where$X^m={[{\mathbf{x}}_1^m,\dots ,{\mathbf{x}}_{N_m}^m]}^{\text{T}}\in R^{N_m\times d}$, ${\mathbf{y}}^m={[y_1^m,\dots ,y_{N_m}^m]}^{\text{T}}\in R^{N_m}$, ‖ ⋅‖_F is the Frobenius norm, V = [v¹, …, v^M] on behalf of the model coefficient matrix, ${\mathbf{v}}^m={[v_1^m,\dots ,v_d^m]}^{\text{T}}$ represents the m-th column of V, which is the coefficient vector of the m-th task. λ represents the regularization coefficient and $\Vert V{\Vert }_{\mathrm{2,1}}={\sum }_{i=1}^d\sqrt{{\sum }_{m=1}^M{(v_i^m)}^2}$. For the input ${\tilde{\mathbf{x}}}^m$ of the m-th task, MTL-L₂₁ gives the predicted value $f({\tilde{\mathbf{x}}}^m)={\tilde{\mathbf{x}}}^{m^{\text{T}}}{\mathbf{v}}_{MTL-L_{21}}^{m\ast }$.

When data features have the problem of collinearity, MTL-L₂₁ will have an unstable prediction performance. Xi Chen et al. (Chen et al., 2012) proposed the MTEN method by adding another quadratic regularization term for the objective function coefficients of all tasks on the basis of MTL-L₂₁. It can be described as the following optimization problem,

$V_{\mathit{MTEN}}^{\ast }=\underset{V\in R^{d\times M}}{\mathrm{arg\; min}}\sum _{m=1}^M\frac{1}{2n}\Vert X^m{\mathbf{v}}^m-{\mathbf{y}}^m{\Vert }_F^2+\lambda \rho \Vert V{\Vert }_{\mathrm{2,1}}+\frac{\lambda \left(1-\rho \right)}{2}\Vert V{\Vert }_F^2,$

where ρ ∈ [0, 1] represents the elastic net mixing parameter. For the input ${\tilde{\mathbf{x}}}^m$ of the m-th task, MTEN gives the predicted value $f({\tilde{\mathbf{x}}}^m)={\tilde{\mathbf{x}}}^{m^{\text{T}}}{\mathbf{v}}_{\mathit{MTEN}}^{m\ast }$.

In the case of insufficient data features and data size, the two algorithms MTL-L₂₁ and MTEN cannot obtain enough information by learning sparse features, which leads to poor prediction performance.

2.2 Multi-Task Learning Methods Based on Parameter Sharing

Caruana (1997) implemented MTSL on backpropagation nets by sharing input and hidden layer neurons among different tasks. Essentially, this method optimizes the choice of function space by the correlation among tasks and obtains better internal weight parameters.

Figure 1 shows the process of traditional backpropagation nets to deal with four related tasks. This method ignores the information among related tasks. Especially in the case of insufficient data samples, these models may have problems such as over-fitting.

FIGURE 1

FIGURE 1. Single-task backpropagation nets.

Figure 2 shows the multi-task backpropagation net (MTL) conceived by Caruana. In MTL, each task shares input and hidden layer neurons.

FIGURE 2

FIGURE 2. The multi-task backpropagation net.

Compared with the data form given in Eq. 1, the data form suitable for MTL is that different tasks have the same input,

$X^1=\dots =X^M=X={\left[{\mathbf{x}}_1,\dots ,{\mathbf{x}}_N\right]}^{\text{T}}\in R^{N\times d},$

and the output is,

$Y=\left[{\mathbf{y}}^1,\dots ,{\mathbf{y}}^M\right]\in R^{N\times M},{\mathbf{y}}^m={\left[y_1^m,\dots ,y_N^m\right]}^{\text{T}}\in R^N.$

MTL can be described as follows

$\left({\mathit{\beta }}^{\ast },W^{\ast },{\mathbf{b}}^{\ast }\right)=\underset{\mathbf{b}\in {\mathit{R}}^S}{\underset{\mathit{\beta }\in R^{S\times M}W\in R^{S\times d}}{\text{arg min}}}\sum _{m=1}^M‖{\mathbf{y}}^m-\sum _{j=1}^S{\mathbf{g}}_j{\beta }_j^m‖,$

where S is the number of hidden layer nodes, ${\beta }_j^m$ is the external weight parameter of the m-th task in the j-th hidden layer node. ${\mathbf{g}}_j≔{\mathbf{g}}_j(X)={[g({\mathbf{x}}_1^{\text{T}}{\mathbf{w}}_j+b_j),\dots ,g({\mathbf{x}}_N^{\text{T}}{\mathbf{w}}_j+b_j)]}^{\text{T}}$ (here g represents the sigmoid function), ${\mathbf{w}}_j={[w_{j,1},\dots ,w_{j,d}]}^{\text{T}}$ and b_j represent the internal weight parameter vector and the bias internal weight parameter shared by the backpropagation net. β ∈ R^N×M, W ∈ R^d×N, b ∈ R^N are the corresponding parameter matrix. For the input ${\tilde{\mathbf{x}}}^m$ of the m-th task, MTL gives the predicted value $f({\tilde{\mathbf{x}}}^m)={\sum }_{j=1}^{S}g({\tilde{\mathbf{x}}}^{m^{\text{T}}}{\mathbf{w}}_j^{\ast }+b_j^{\ast }){\beta }_j^{m\ast }$.

From a mathematical point of view, the essence of the backpropagation net is the gradient descent algorithm. In single-task supervised learning, the backpropagation net may fall into local optimum. However, in multi-task supervised learning, the local optimum of different tasks is in different positions, and the interaction among tasks can help the hidden layer to escape from local optimums (Caruana, 1997).

2.3 Stochastic Configuration Networks

Wang and Li (Wang and Li, 2017a; Wang and Li, 2017b) proposed supervised stochastic configuration networks, and implemented SCNs using three algorithms SC-i (i=I,II,III). SC-i starts with a small network structure (Tin-Yan Kwok and Dit-Yan Yeung, 1997), and uses a supervision mechanism to add hidden layer neurons until the model meets a predetermined error criterion. Since SC-III performs the best of the three algorithms, we next describe the implementation of the SC-III algorithm.

Suppose a SC-III with L − 1 hidden layer nodes has already been constructed, that is,

$f^{L-1}\left({\mathbf{x}}^{\text{T}}\right)=\sum _{j=1}^{L-1}g\left({\mathbf{x}}^{\text{T}}{\mathbf{w}}_j^{\ast }+b_j^{\ast }\right){\mathrm{\beta }}_j^{L-1},\left(L=\mathrm{2,3},\dots ,f^0\in \left[\mathrm{0,0},\dots ,0\right]\in R^q\right),$

where ${\mathrm{\beta }}_j^{L-1}=[{\beta }_{j,1}^{L-1},\dots ,{\beta }_{j,q}^{L-1}]\in R^q$ represents the optimal external weight parameter of the j-th hidden layer node, ${\mathbf{w}}_j^{\ast }$ and $b_j^{\ast }$ represent the optimal internal weight parameters of the j-th hidden layer node.

For training data set $X={[{\mathbf{x}}_1,\dots ,{\mathbf{x}}_N]}^{\text{T}}\in R^{N\times d}$, Y = [y¹, …, y^q] ∈ R^N×q, where ${\mathbf{x}}_i={[x_{i,1},\dots ,x_{i,d}]}^{\text{T}}$. Let $f^{L-1}≔f^{L-1}(X)={\sum }_{j=1}^{L-1}{\mathbf{g}}_j(X){\mathrm{\beta }}_j^{L-1}$ and ${\mathit{e}}^{L-1}≔Y-f^{L-1}≔[{\mathbf{e}}_1^{L-1},\dots ,{\mathbf{e}}_q^{L-1}]$, which is the residual error matrix of the (L − 1)-th hidden layer node. If ‖e^L−1‖_F does not meet the predetermined error criteria, SC-III needs to generate a new hidden layer node, that is, stochastically configure internal weight parameters w_L, b_L from an uniform distribution U^d+1[ − Φ, Φ], Φ > 0. For new variables,

${\xi }_n^L=\frac{{⟨{\mathbf{e}}_n^{L-1},{\mathbf{g}}_L⟩}^2}{\Vert {\mathbf{g}}_L{\Vert }_2^2}-\left(1-r-{\mu }_L\right)\Vert {\mathbf{e}}_n^{L-1}{\Vert }_2^2,n=1,\dots ,q,$

if $\underset{n}{\min }({\xi }_n^L)\ge 0$, then w_L, b_L are considered to meet the condition, otherwise w_L, b_L need to be configured again. With the qualified internal weight parameters ${\mathbf{w}}_L^{*}$ and $b_L^{*}$, SC-III obtains the optimal external weight parameter vector by the following optimization problem,

${\mathit{\beta }}^L=\underset{\mathit{\beta }\in R^{L\times q}}{\mathrm{arg\; min}}{‖Y-\sum _{j=1}^L{\mathbf{g}}_j{\mathrm{\beta }}_j‖}_F^2.$

The leading model $f^L(\mathbf{x})={\sum }_{j=1}^{L}g({\mathbf{x}}^{\text{T}}{\mathbf{w}}_j^{*}+b_j^{*}){\mathrm{\beta }}_j^L$ will have an improved residual error. Repeat the above steps to add hidden layer nodes until the residual error meets the predetermined error criteria.

3 Multi-Task Supervised Learning Based on Stochastic Configuration Radial Basis Networks

In this section, we introduce the proposed MTSL-SCRBN algorithm.

3.1 Model Introduction

In order to combine SCNs and MTL organically, we need to change the data form given in Eq. 1. In MTSL-SCRBN, first, we require each task to have a same number of samples, namely, N₁ =…= N_M =: N. (If the number of samples for each task is different, this requirement can be achieved by random sampling.) Then we merge the input data of different tasks into a new input data, that is, the i-th new input data is ${\mathbf{X}}_i^{\text{T}}:=({\mathbf{x}}_i^{1^{\text{T}}},\dots ,{\mathbf{x}}_i^{M^{\text{T}}})\in R^{1\times Md}$, where ${\mathbf{x}}_i^m\in R^d$ is the i-th input of the m-th task, for i = 1, …, N and m = 1, …, M. The corresponding i-th new output data is ${\mathbf{y}}_i:={(y_i^1,\dots ,y_i^M)}^T$, where $y_i^m$ is the original output of ${\mathbf{x}}_i^m$ in the m-th task. The goal of our MTSL-SCRBN is to establish an appropriate model from R^Md to R^M based on these data $\{{({\mathbf{X}}_i,{\mathbf{y}}_i)}_{i=1}^N\}$.

In order to obtain good learning performance, we use the following radial basis function k_σ(x, x′) as model’s basis function,

$k_{\sigma }\left(\mathbf{x},{\mathbf{x}}^{\prime }\right)=\exp \left(-\frac{\Vert \mathbf{x}-{\mathbf{x}}^{\prime }{\Vert }^2}{2{\sigma }^2}\right),$

where x is the input, x′ represents the center and σ is the scale parameter.

Suppose f⁰ = [0, 0, …, 0] ∈ R^M, for L = 2, 3, …, it is assumed that a MTSL-SCRBN with L − 1 hidden layer nodes has already been constructed as follows,

$f^{L-1}\left(\mathbf{X}\right)=\sum _{j=1}^{L-1}\left[k_{{\sigma }_j^{\ast }}\left({\mathbf{x}}^1,{\mathbf{x}}^{j\ast }\right),\dots ,k_{{\sigma }_j^{\ast }}\left({\mathbf{x}}^M,{\mathbf{x}}^{j\ast }\right)\right]{\mathit{\beta }}_j^{L-1},$

where ${\mathbf{X}}^{\text{T}}=({\mathbf{x}}^{1^{\text{T}}},\dots ,{\mathbf{x}}^{M^{\text{T}}})$ is a new input formed by the inputs of M tasks, and ${\mathit{\beta }}_j^{L-1}=[{\mathrm{\beta }}_j^{1,L-1},\dots ,{\mathrm{\beta }}_j^{M,L-1}]\in R^{M\times M}$,where ${\mathrm{\beta }}_j^{m,L-1}\in R^M$ represents the optimal external weight parameter vector of the m-th task in the j-th hidden layer node, x^j* and ${\sigma }_j^{*}$ are the optimal center and the optimal scale parameter of the radial basis function in the j-th hidden layer node, respectively. Different from the traditional learning of radial basis neural network, in our MTSL-SCRBN, the optimal centers and the optimal scale parameters at each step are randomly assigned by a shared supervision mechanism given in the following. This is simple to implement and easy to obtain a learning model with good performance.

Denote ${\mathbf{k}}_j^m≔{[k_{{\sigma }_j^{\ast }}({\mathbf{x}}_1^m,{\mathbf{x}}^{j\ast }),\dots ,k_{{\sigma }_j^{\ast }}({\mathbf{x}}_N^m,{\mathbf{x}}^{j\ast })]}^{\text{T}},K_j≔[{\mathbf{k}}_j^1,\dots ,{\mathbf{k}}_j^M]\in R^{N\times M}$ and $f^{L-1}≔{\sum }_{j=1}^{L-1}{K}_j{\mathit{\beta }}_j^{L-1}$, $Y={[{\mathbf{y}}_1,\dots ,{\mathbf{y}}_N]}^{\text{T}}\in R^{N\times M}$. Then, let e^L−1≔Y − f^L−1≔[e^1,L−1, …, e^M,L−1] be the residual error matrix of the (L − 1)-th hidden layer node. If ‖e^L−1‖_F does not meet the predetermined error criteria, MTSL-SCRBN needs to generate a new hidden layer node, that is, stochastically configure the scale parameter σ_L from U[0, Ω], Ω > 0 and the center of the radial basis function x^L from $\{{\mathbf{x}}_i^m:i=1,\dots ,N,m=1,\dots ,M\}$.

Similar to that in SCNs, we introduce a variable ξ^m,L in our multi-task learning case as follows,

${\xi }^{m,L}=\frac{1}{M}\frac{{⟨{\mathbf{e}}^{m,L-1},{\mathbf{k}}_L^m⟩}^2}{\Vert {\mathbf{k}}_L^m{\Vert }_2^2}-\left(1-r-{\mu }_{L,r}\right)\Vert {\mathbf{e}}^{m,L-1}{\Vert }_2^2.$

Here ${\mathbf{k}}_L^m={[k_{{\sigma }_L}({\mathbf{x}}_1^m,{\mathbf{x}}^L),\dots ,k_{{\sigma }_L}({\mathbf{x}}_N^m,{\mathbf{x}}^L)]}^{\text{T}}$, 0 < r < 1 is a given constant and ${\mu }_{L,r}=\frac{1-r}{L+1}$.

If ${\sum }_{m=1}^M{\xi }^{m,L}\ge 0$, then σ_L, x^L are considered to meet the condition, otherwise, σ_L, x^L need to be configured again. With the qualified parameters ${\sigma }_L^{*}$ and x^L∗, MTSL-SCRBN obtains the optimal external weight parameter vector by the following optimization problem,

${\mathbf{\beta }}^L=\underset{\mathbf{\beta }\in R^{LM\times M}}{\mathrm{arg\; min}}{‖Y-\sum _{j=1}^L{K}_j{\mathit{\beta }}_j‖}_F^2.$

The leading model,

$f^L\left(\mathbf{X}\right)=\sum _{j=1}^L\left[k_{{\sigma }_j^{\ast }}\left({\mathbf{x}}^1,{\mathbf{x}}^{j\ast }\right),\dots ,k_{{\sigma }_j^{*}}\left.\left({\mathbf{x}}^M,{\mathbf{x}}^{j\ast }\right)\right)\right]{\mathit{\beta }}_j^L,$

will have an improved residual error. Repeat the above steps to add hidden layer nodes until the residual error meets the predetermined error criteria.

The above implementation process of the proposed MTSL-SCRBN algorithm is described as follows.

Algorithm 1. The MTSL-SCRBN algorithm

Notice that in step 19, we calculate the parameter matrix based on the standard least squares method,

${\mathbf{\beta }}^L=\underset{\mathbf{\beta }\in R^{LM\times M}}{\mathrm{arg\; min}}{‖Y-\sum _{j=1}^L{K}_j{\mathit{\beta }}_j‖}_F^2={\mathbf{K}}_L^{†}Y,$

where ${\mathbf{K}}_L^{†}$ is the Moore-Penrose generalized inverse (Lancaster and Tismenetsky, 1985) of K_L. The setting of μ_L,r in step 7 and the updating idea of r in step 14 can be referred to literature (Wang and Li, 2017a).

3.2 The Convergence Theorem of the MTSL-SCRBN Algorithm

We extend the method in (Wang and Li, 2017a) to the multi-task learning framework of this paper and prove the convergence of the proposed algorithm.

Theorem 1. Assume that there are some p_k ∈ R⁺, satisfying $0<\Vert {\mathbf{k}}_j^m{\Vert }_2<p_{\mathbf{k}}$. Given 0 < r < 1 and a non-negative real value sequence {μ_L} with lim_L→+∞μ_L = 0 and μ_L ≤ (1 − r). For L = 2, 3 …, denoted by

${\mathrm{\delta }}^L=\sum _{m=1}^M{\delta }^{m,L},$

${\delta }^{m,L}=\left(1-r-{\mu }_L\right)\Vert {\mathbf{e}}^{m,L-1}{\Vert }_2^2.$

If the basis function ${\mathbf{k}}_L^m$ is generated to satisfy the following inequality,

$\sum _{m=1}^M{⟨{\mathbf{e}}^{m,L-1},{\mathit{k}}_L^m⟩}^2\ge \frac{M^2}{2M-2}{p}_{\mathbf{k}}^2{\mathrm{\delta }}^L,(2)$

and the external weight parameter vector is evaluated by,

${\mathbf{\beta }}^L={\left[{\mathit{\beta }}_1^L,\dots ,{\mathit{\beta }}_L^L\right]}^{\text{T}}=\underset{\mathbf{\beta }\in R^{LM\times M}}{\mathrm{arg\; min}}{‖Y-\sum _{j=1}^L{K}_j{\mathit{\beta }}_j‖}_F^2.$

Then, we have lim_L→+∞‖Y − f^L‖_F = 0.

Proof of Theorem 1. Define intermediate values

${\tilde{\mathrm{\beta }}}_L^{m,L}={\left[\frac{\langle {\mathit{e}}^{m,L-1},{\mathit{k}}_L^1\rangle }{M\Vert {\mathit{k}}_L^1{\Vert }_2^2},\dots ,\frac{\langle {\mathit{e}}^{m,L-1},{\mathit{k}}_L^M\rangle }{M\Vert {\mathit{k}}_L^M{\Vert }_2^2}\right]}^{\text{T}},$

and ${\tilde{\mathit{e}}}^L={\mathit{e}}^{L-1}-K_L{\tilde{\mathit{\beta }}}_L^L$, with ${\tilde{\mathit{\beta }}}_L^L=[{\tilde{\mathrm{\beta }}}_L^{1,L},\dots ,{\tilde{\mathrm{\beta }}}_L^{M,L}],{\mathit{e}}^0=Y$.It is clear that $\Vert {\mathit{e}}^L{\Vert }_F^2\le \Vert {\tilde{\mathit{e}}}^L{\Vert }_F^2\le \Vert {\mathit{e}}^{L-1}{\Vert }_F^2\le \Vert {\tilde{\mathit{e}}}^{L-1}{\Vert }_F^2$, where L = 2, 3, …. So $\{\Vert {\mathit{e}}^{L-1}{\Vert }_F^2\}$ is monotonically decreasing and convergent. Hence, we have,

$\begin{array}{ll}\hfill & \Vert {\mathit{e}}^L{\Vert }_F^2-\left(r+{\mu }_L\right)\Vert {\mathit{e}}^{L-1}{\Vert }_F^2\hfill \\ \hfill \le & \Vert {\tilde{\mathit{e}}}^L{\Vert }_F^2-\left(r+{\mu }_L\right)\Vert {\mathit{e}}^{L-1}{\Vert }_F^2\hfill \\ \hfill =& \sum _{m=1}^M\left(⟨{\mathbf{e}}^{m,L-1}-K_L{\tilde{\mathrm{\beta }}}_L^{m,L},{\mathit{e}}^{m,L-1}-K_L{\tilde{\mathrm{\beta }}}_L^{m,L}⟩-\left(r+{\mu }_L\right)⟨{\mathit{e}}^{m,L-1},{\mathit{e}}^{m,L-1}⟩\right)\hfill \\ \hfill =& \left(1-r-{\mu }_L\right)\Vert {\mathit{e}}^{L-1}{\Vert }_F^2-\sum _{m=1}^M\left(2⟨{\mathit{e}}^{m,L-1},K_L{\tilde{\mathrm{\beta }}}_L^{m,L}⟩-⟨K_L{\tilde{\mathrm{\beta }}}_L^{m,L},K_L{\tilde{\mathrm{\beta }}}_L^{m,L}⟩\right)\hfill \\ \hfill \le & \left(1-r-{\mu }_L\right)\Vert {\mathit{e}}^{L-1}{\Vert }_F^2-\frac{2M-2}{M^2}\sum _{m=1}^M\left(\frac{{⟨{\mathit{e}}^{m,L-1},{\mathit{k}}_L^1⟩}^2}{\Vert {\mathit{k}}_L^1{\Vert }_2^2}+\cdots +\frac{{⟨{\mathit{e}}^{m,L-1},{\mathit{k}}_L^M⟩}^2}{\Vert {\mathit{k}}_L^M{\Vert }_2^2}\right)\hfill \\ \hfill \le & \left(1-r-{\mu }_L\right)\Vert {\mathit{e}}^{L-1}{\Vert }_F^2-\frac{2M-2}{M^2}\sum _{m=1}^M\frac{{⟨{\mathit{e}}^{m,L-1},{\mathit{k}}_L^m⟩}^2}{\Vert {\mathit{k}}_L^m{\Vert }_2^2}\hfill \\ \hfill =& {\mathrm{\delta }}^L-\frac{2M-2}{M^2}\sum _{m=1}^M\frac{{⟨{\mathit{e}}^{m,L-1},{\mathit{k}}_L^m⟩}^2}{\Vert {\mathit{k}}_L^m{\Vert }_2^2}\hfill \\ \hfill \le & {\mathrm{\delta }}^L-\frac{2M-2}{M^2}\frac{{\sum }_{m=1}^M{⟨{\mathit{e}}^{m,L-1},{\mathit{k}}_L^m⟩}^2}{p_{\mathit{k}}^2}\hfill \\ \hfill \le & 0.\hfill \end{array}$

Then, the following inequality holds,

$\Vert {\mathit{e}}^L{\Vert }_F^2\le r\Vert {\mathit{e}}^{L-1}{\Vert }_F^2+{\gamma }_L,\left({\gamma }_L={\mu }_L\Vert {\mathit{e}}^{L-1}{\Vert }_F^2\ge 0\right).$

Since lim_L→+∞μ_L = 0, and 0 < r < 1, we have ${\lim }_{L\to +\infty }\Vert {\mathit{e}}^L{\Vert }_F^2=0$, and lim_L→+∞‖e^L‖_F = 0.

Remark 1. Unlike SC-III, we relax the condition for the configuration parameters in the formula (∗). SC-III requires each task to meet the inequality conditions, but MTSL-SCRBN only requires the sum of all tasks to satisfy the inequality condition. The rationality of this condition will also be verified in the experiment results of next section.

4 Experiment Results

In order to show the effectiveness of the proposed algorithm, this section uses the classical STSL algorithms SVM (Cortes and Vapnik, 1995), SC-III (Wang and Li, 2017a) and seven MTSL algorithms MTSL-SCRBN, MTL (Caruana, 1997), MTEN (Chen et al., 2012), DMTRL (Yang and Hospedales, 2017), MMoE (Ma et al., 2018), GAMTL (Oliveira et al., 2019), AUTOMTL (Zhang et al., 2021) to perform comparative experiments. All calculations are conducted using Python 3.6.5 on a computer with 2.60 GHz CPU and 8 GB RAM. The input features are scaled into [ − 1, 1] and the output remains unchanged. All the results reported in this paper take averages over 20 independent trials, except for the SVM and MTEN algorithms, which have fixed experiment results. The accuracy (ACC) and root mean square error (RMSE) are chosen as the classification and regression evaluation indicators, where

$RMSE={\left[\frac{1}{N\times M}\sum _{m=1}^M\sum _{i=1}^N{\left({\widehat{y}}_i^m-y_i^m\right)}^2\right]}^{\frac{1}{2}},$

with $y_i^m$ and ${\widehat{y}}_i^m$ representing the target output and the learner’s output of i − th sample for task m respectively.

For different data sets, some algorithms used in the following experiments can stochastically configure hyperparameters within specified ranges or determine parameters by cross-validation. Table 1 gives the specific selection range of each parameter.

TABLE 1

TABLE 1. Parameter description.

4.1 Experimental Results and Analysis on Simulated Data

The simulation data set selected in this paper is generated by the following five functions, which we refer as five tasks,

$\begin{array}{ll}\hfill \text{\hspace{0.17em}Task\hspace{0.17em}}1\text{:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}{f}_1\left(x\right)& =2e^{-{\left(10x-4\right)}^2}+5e^{-{\left(80x-40\right)}^2}+3e^{-{\left(80x-20\right)}^2}+4e^{-{\left(90x-60\right)}^2}\hfill \\ \hfill \text{\hspace{0.17em}Task\hspace{0.17em}}2\text{:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}{f}_2\left(x\right)& =10e^{-{\left(10x-4\right)}^2}+25e^{-{\left(80x-40\right)}^2}+15e^{-{\left(80x-20\right)}^2}+20e^{-{\left(90x-60\right)}^2}\hfill \\ \hfill \text{\hspace{0.17em}Task\hspace{0.17em}}3\text{:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}{f}_3\left(x\right)& =18e^{-{\left(10x-4\right)}^2}+45e^{-{\left(80x-40\right)}^2}+27e^{-{\left(80x-20\right)}^2}+36e^{-{\left(90x-60\right)}^2}.\hfill \\ \hfill \text{\hspace{0.17em}Task\hspace{0.17em}}4\text{:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}{f}_4\left(x\right)& =26e^{-{\left(10x-4\right)}^2}+65e^{-{\left(80x-40\right)}^2}+39e^{-{\left(80x-20\right)}^2}+52e^{-{\left(90x-60\right)}^2}\hfill \\ \hfill \text{\hspace{0.17em}Task\hspace{0.17em}}5\text{:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}{f}_5\left(x\right)& =34e^{-{\left(10x-4\right)}^2}+85e^{-{\left(80x-40\right)}^2}+51e^{-{\left(80x-20\right)}^2}+68e^{-{\left(90x-60\right)}^2}\hfill \end{array}$

Figure 3 depicts the distributions of five functions on [0, 1]. As we can see, when the independent variables of the five functions are the same, the function values follow similar trends. Therefore, learning the function values of five functions with the same independent variable can be regarded as a multi-task learning. Here, we independently extract 100 one-dimensional input data from the same uniform distribution, then calculate the corresponding function values according to these five functions, and add white Gaussian noise with a standard deviation of 0.01 to form 100 five-dimensional output data. In the following experiments, we randomly select 70% of the data as training data and 30% of the data as test data. From the figures of the five functions, it can be seen that only 70 training samples are not enough to achieve good single-task learning results. We verify this point by experimenting with single-task and multi-task algorithms.

FIGURE 3

FIGURE 3. Distribution of the simulation data set.

Firstly, we compare the learning performance of the proposed MTSL-SCRBN with other two STSL algorithms, SVM and SC-III, on five tasks. For MTSL-SCRBN, these five tasks are combined to learn together. The training and test RMSEs on five tasks for these three methods are given in Table 2. Clearly, the proposed multi-task learning model can product better performance on each task than the two STSL models, which only use 70 samples to learn each task independently. Furthermore, we show the learning effects of the three algorithms on Task 1 in Figure 4. It is can be seen that the proposed MTSL-SCRBN has good learning performance where the data changes dramatically.

TABLE 2

TABLE 2. The results of MTSL-SCRBN, SC-III and SVM on the simulation data set.

FIGURE 4

FIGURE 4. Prediction performance of MTSL-SCRBN, SC-III and SVM on Task 1 of simulation data set.

Next, the comparison results of three MTSL algorithms, MTSL-SCRBN, MTL, MTEN, on the simulation data set are recorded in Table 3 and Figure 5. In Table 3, the values in parentheses represent the standard deviations of 20 experiments’ results. According to these results, compared with MTEN and MTL, the proposed MTSL-SCRBN has better approximation ability.

TABLE 3

TABLE 3. The results of MTSL-SCRBN, MTL, MTEN on the simulation data set.

FIGURE 5

FIGURE 5. Prediction performance of MTSL-SCRBN, MTL, MTEN on Task 1 of simulation data set.

4.2 Experimental Results and Analysis on Benchmark Datasets

This subsection further compares seven MTSL algorithms on four benchmark datasets. They are MTL, MTEN, DMTRL, MMoE, GAMTL, AUTOMTL and the proposed MTSL-SCRBN. According to the characteristics of data sets and algorithms, different algorithms will be selected for comparative analysis on different data sets. The four benchmark datasets include three regression problems on the stock portfolio performance data set, the bionic robot data set SARCOS and the School data set, one classification problem on the Mnist data set from Yann LeCun¹ The basic information of the four datasets are summarized in Table 4.

TABLE 4

TABLE 4. Descriptions of benchmark datasets.

Firstly, we compare the performance of MTL, MTEN, DMTRL, MMoE, GAMTL and MTSL-SCRBN on different sizes of three regression data sets. For the three data sets, we randomly choose 15/30/3,500 samples outside the training set as test set, respectively. At the same time, we select 5 tasks, all of which have more than 230 samples, from 139 tasks in School data set. Table 5 below shows the specific experiment results. As we can see, the performance of each algorithm tends to be better and more stable with an increasing number of training samples. Furthermore, the proposed MTSL-SCRBN algorithm exhibits good performance even with a small number of training observations.

TABLE 5

TABLE 5. The comparison results of six MSTL algorithms on three data sets.

Then, in order to further verify the performance of MTSL-SCRBN for classification cases, we compare the results of MTSL-SCRBN, DMTRL, MMoE, GAMTL and AUTOMTL on the Mnist data set. We randomly choose 50/100/150 samples from each task in the Mnist data set as training set, and 10000 samples in the remaining samples as test set. Considering that this is a high dimensional small sample problem, we firstly reduce the dimensionality of the data set, and then use MTSL-SCRBN for training and prediction. There are many dimensionality reduction methods, such as Principal Component Analysis(PCA) (Pearson, 1901), Latent Dirichlet Allocation(LDA) (Blei et al., 2003), Sequential Markov Blanket Criterion (SMBC) (Pratama et al., 2017), Auto Encoder (Hinton and Salakhutdinov, 2006) and so on. Here, we use the performance of the dimensionality-reduced data in the MTSL-SCRBN as the selection criterion, and choose Auto Encoder to reduce the dimension of original data set into 30 dimensions. It can be seen from Table 6 that the performance of MTSL-SCRBN which uses dimensionality reduction data set is a little bit better than that of other Multi-task deep learning algorithms, but as the sample size increases, the performance of the two algorithms gradually approaches.

TABLE 6

TABLE 6. The accuracy of MTSL-SCRBN, DMTRL, MMoE, GAMTL and AUTOMTL on Mnist data set.

4.3 Comparison Experiment Results for Different Activation Functions

The previous results show that the proposed MTSL-SCRBN algorithm is effective for multi-task learning in the case of small samples. This subsection mainly discusses the impact of selecting different activation functions on algorithm performance. Here we select other three usually used activation functions. They are sigmoid function, Tanh function and ReLU function. After replacing the radial basis functions in MTSL-SCRBN with these three functions respectively, the model names are respectively called MTSL-SCSGM, MTSL-SCTANH and MTSL-SCReLU. We choose to conduct comparative experiments on the Stock and SARCOS data sets.

For different data sets, the parameters contained in each algorithm need to be randomly set or cross verified within a certain range. The specific selection range of each parameter is given in Table 7.

TABLE 7

TABLE 7. Parameter description for the four models.

Table 8 depicts the RMSE results of stochastic configuration multi-task learning models based on four different activation functions. Under the training samples with different sample sizes in the two data sets, the MTSL-SCRBN, which based on radial basis functions, has certain advantages over other three models in terms of performance.

TABLE 8

TABLE 8. The comparison results of different activation functions based models on two data sets.

5 Conclusion

In this paper, we propose a multi-task supervised learning framework based on stochastic configuration radial basis network. It can be effectively used in classification and regression problems when a single task has a small number of samples. The series experiment results on the four data sets show the proposed MTSL-SCRBN achieves a good performance compared with some existing methods.

Interesting areas for further directions include using the proposed algorithm in hyperspectral remote sensing image classification and other related research areas, considering the impact of using different activation functions in the network, and trying to explore the range of the sample size of the data set to use the multi-task learning method.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

X-MD: Conceptualization, Methodology, Formal analysis, Supervision, Project administration. XK: Writing-original draft, Software, Investigation. XZ: Validation, Writing—review and editing.

Funding

This work was supported in part by the Characteristic and Preponderant Discipline of Key Construction Universities in Zhejiang Province (Zhejiang Gongshang University- Statistics) and the special fund for basic business expenses of colleges and universities in Zhejiang Province.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

¹These four datasets can be obtained from http://archive.ics.uci.edu/ml/datasets/Stock+portfolio+performance, http://gaussianprocess.org/gpml/data, http://bristol.ac.uk/cmm/learning/support/datasets and http://yann.lecun.com/exdb/mnist/, respectively.

References

Argyriou, A., Evgeniou, T., and Pontil, M. (2008). Convex Multi-Task Feature Learning. Mach. Learn 73, 243–272. doi:10.1007/s10994-007-5040-8

CrossRef Full Text | Google Scholar

Argyriou, A., Evgeniou, T., and Pontil, M. (2007). “Multi-task Feature Learning,” in Advances in Neural Information Processing Systems, 41–48. doi:10.7551/mitpress/7503.003.0010

CrossRef Full Text | Google Scholar

Baxter, J. (2000). A Model of Inductive Bias Learning. Jair 12, 149–198. doi:10.1613/jair.731

CrossRef Full Text | Google Scholar

Ben-David, S., and Schuller, R. (2003). “Exploiting Task Relatedness for Multiple Task Learning,” in Learning Theory and Kernel Machines (Springer), 567–580. doi:10.1007/978-3-540-45167-9_41

CrossRef Full Text | Google Scholar

Blei, D. M., Ng, A. Y., and Jordan, M. I. (2003). Latent Dirichlet Allocation. J. Mach. Learn. Res. 3, 993–1022. doi:10.1162/jmlr.2003.3.4-5.993

CrossRef Full Text | Google Scholar

Broomhead, D., and Lowe, D. (1988). Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks. Malvern, United Kingdom: Royal Signals and Radar Establishment Malvern United Kingdom. Tech. rep.

Google Scholar

Caruana, R. (1997). Multitask Learning. Mach. Learn. 28, 41–75. doi:10.1023/a:1007379606734

CrossRef Full Text | Google Scholar

Chen, X., He, J., Lawrence, R., and Carbonell, J. (2012). “Adaptive Multi-Task Sparse Learning with an Application to Fmri Study,” in Proceedings of the 2012 SIAM International Conference on Data Mining, 212–223. doi:10.1137/1.9781611972825.19

CrossRef Full Text | Google Scholar

Cortes, C., and Vapnik, V. (1995). Support-vector Networks. Mach. Learn 20, 273–297. doi:10.1007/BF00994018

CrossRef Full Text | Google Scholar

Duong, L., Cohn, T., Bird, S., and Cook, P. (2015). “Low Resource Dependency Parsing: Cross-Lingual Parameter Sharing in a Neural Network Parser,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2, 845–850. Short Papers. doi:10.3115/v1/p15-2139

CrossRef Full Text | Google Scholar

Hinton, G. E., and Salakhutdinov, R. R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science 313, 504–507. doi:10.1126/science.1127647

PubMed Abstract | CrossRef Full Text | Google Scholar

Jacob, L., Vert, J., and Bach, F. (2009). “Clustered Multi-Task Learning: A Convex Formulation,” in Advances in Neural Information Processing Systems, 745–752.

Google Scholar

Lancaster, P., and Tismenetsky, M. (1985). The Theory of Matrices: With Applications. Elsevier.

Google Scholar

Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based Learning Applied to Document Recognition. Proc. IEEE 86, 2278–2324. doi:10.1109/5.726791

CrossRef Full Text | Google Scholar

Liu, T., Tao, D., Song, M., and Maybank, S. J. (2017). Algorithm-dependent Generalization Bounds for Multi-Task Learning. IEEE Trans. Pattern Anal. Mach. Intell. 39, 227–241. doi:10.1109/TPAMI.2016.2544314

PubMed Abstract | CrossRef Full Text | Google Scholar

Ma, J., Zhao, Z., Yi, X., Chen, J., Hong, L., and Chi, E. (2018). “Modeling Task Relationships in Multi-Task Learning with Multi-Gate Mixture-Of-Experts,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom: Association for Computing Machinery), 1930–1939. doi:10.1145/3219819.3220007

CrossRef Full Text | Google Scholar

Oliveira, S. H. G., Gonçalves, A. R., and Von Zuben, F. J. (2019). “Group Lasso with Asymmetric Structure Estimation for Multi-Task Learning,” in Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 (Macao, China: International Joint Conferences on Artificial Intelligence Organization), 3202–3208. doi:10.24963/ijcai.2019/444

CrossRef Full Text | Google Scholar

Pearson, K. (1901). Liii. on lines and planes of closest fit to systems of points in space. Lond. Edinb. Dublin Philosophical Mag. J. Sci. 2, 559–572. doi:10.1080/14786440109462720

CrossRef Full Text | Google Scholar

Powell, M. (1987). Radial Basis Functions for Multivariable Interpolation: A Review. Oxford, United Kingdom: Algorithms for approximation, 143–167.

Google Scholar

Pratama, M., Lu, J., Lughofer, E., Zhang, G., and Er, M. J. (2017). An incremental learning of concept drifts using evolving type-2 recurrent fuzzy neural networks. IEEE Trans. Fuzzy Syst. 25, 1175–1192. doi:10.1109/tfuzz.2016.2599855

CrossRef Full Text | Google Scholar

Tin-Yan Kwok, T., and Dit-Yan Yeung, D. (1997). Objective functions for training new hidden units in constructive neural networks. IEEE Trans. Neural Netw. 8, 1131–1148. doi:10.1109/72.623214

PubMed Abstract | CrossRef Full Text | Google Scholar

Wang, D., and Li, M. (2017). Robust stochastic configuration networks with kernel density estimation for uncertain data regression. Inf. Sci. 412-413, 210–222. doi:10.1016/j.ins.2017.05.047

CrossRef Full Text | Google Scholar

Wang, D., and Li, M. (2017). Stochastic configuration networks: Fundamentals and algorithms. IEEE Trans. Cybern. 47, 3466–3479. doi:10.1109/tcyb.2017.2734043

PubMed Abstract | CrossRef Full Text | Google Scholar

Yang, Y., and Hospedales, T. (2017). “Deep multi-task representation learning: A tensor factorisation approach,” In International Conference on Learning Representations.

Google Scholar

Zhang, L., Liu, X., and Guan, H. (2021). Automtl: A Programming Framework for Automated Multi-Task Learning, 13076. ArXiv abs/2110.

Google Scholar