Optimal Rates for the Regularized Learning Algorithms under General Source Condition

We consider the learning algorithms under general source condition with the polynomial decay of the eigenvalues of the integral operator in vector-valued function setting. We discuss the upper convergence rates of Tikhonov regularizer under general source condition corresponding to increasing monotone index function. The convergence issues are studied for general regularization schemes by using the concept of operator monotone index functions in minimax setting. Further we also address the minimum possible error for any learning algorithm.

1. Introduction

Learning theory [1–3] aims to learn the relation between the inputs and outputs based on finite random samples. We require some underlying space to search the relation function. From the experiences we have some idea about the underlying space which is called hypothesis space. Learning algorithms tries to infer the best estimator over the hypothesis space such that f(x) gives the maximum information of the output variable y for any unseen input x. The given samples ${\{x_i,y_i\}}_{i=1}^m$ are not exact in the sense that for underlying relation function f(x_i) ≠ y_i but f(x_i) ≈ y_i. We assume that the uncertainty follows the probability distribution ρ on the sample space X × Y and the underlying function (called the regression function) for the probability distribution ρ is given by

$f_{\rho }(x)={\displaystyle {\int }_Y}yd\rho (y|x),x\in X,$

where ρ(y|x) is the conditional probability measure for given x. The problem of obtaining estimator from examples is ill-posed. Therefore, we apply the regularization schemes [4–7] to stabilize the problem. Various regularization schemes are studied for inverse problems. In the context of learning theory [2, 3, 8–10], the square loss-regularization (Tikhonov regularization) is widely considered to obtain the regularized estimator [9, 11–16]. Gerfo et al. [6] introduced general regularization in the learning theory and provided the error bounds under Hölder's source condition [5]. Bauer et al. [4] discussed the convergence issues for general regularization under general source condition [17] by removing the Lipschitz condition on the regularization considered in Gerfo et al. [6]. Caponnetto and De Vito [12] discussed the square-loss regularization under the polynomial decay of the eigenvalues of the integral operator L_K with Hölder's source condition. For the inverse statistical learning problem, Blanchard and Mücke [18] analyzed the convergence rates for general regularization scheme under Hölder's source condition in scalar-valued function setting. Here we are discussing the convergence issues of general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator in vector-valued framework. We present the minimax upper convergence rates for Tikhonov regularization under general source condition Ω_{ϕ, R}, for a monotone increasing index function ϕ. For general regularization the minimax rates are obtained using the operator monotone index function ϕ. The concept of effective dimension [19, 20] is exploited to achieve the convergence rates. In the choice of regularization parameters, the effective dimension plays the important role. We also discuss the lower convergence rates for any learning algorithm under the smoothness conditions. We present the results in vector-valued function setting. Therefore, in particular they can be applied to multi-task learning problems.

The structure of the paper is as follows. In the second section, we introduce some basic assumptions and notations for supervised learning problems. In Section 3, we present the upper and lower convergence rates under the smoothness conditions in minimax setting.

2. Learning From Examples: Notations and Assumptions

In the learning theory framework [2, 3, 8–10], the sample space Z = X × Y consists of two spaces: The input space X (locally compact second countable Hausdorff space) and the output space (Y, 〈·, ·〉_Y) (the real separable Hilbert space). The input space X and the output space Y are related by some unknown probability distribution ρ on Z. The probability measure can be split as ρ(x, y) = ρ(y|x)ρ_X(x), where ρ(y|x) is the conditional probability measure of y given x and ρ_X is the marginal probability measure on X. The only available information is the random i.i.d. samples z = ((x₁, y₁), …, (x_m, y_m)) drawn according to the probability measure ρ. Given the training set z, learning theory aims to develop an algorithm which provides an estimator f_z : X → Y such that f_z(x) predicts the output variable y for any given input x. The goodness of the estimator can be measured by the generalization error of a function f which can be defined as

$\begin{array}{rc}{E}(f):={{E}}_{\rho }(f)={\displaystyle {\int }_Z}V(f(x),y)d\rho (x,y),& \left(1\right)\end{array}$

where V : Y × Y → ℝ is the loss function. The minimizer of ${E}(f)$ for the square loss function $V(f(x),y)=\left|\right|f(x)-y|{|}_Y^2$ is given by

$\begin{array}{rc}{f}_{\rho }(x):={\displaystyle {\int }_Y}yd\rho (y|x),& \left(2\right)\end{array}$

where f_ρ is called the regression function. The regression function f_ρ belongs to the space of square integrable functions provided that

$\begin{array}{rc}{\displaystyle {\int }_Z}\left|\right|y|{|}_Y^{2}d\rho (x,y)<\infty .& \left(3\right)\end{array}$

We search the minimizer of the generalization error over a hypothesis space ${H}$,

$\begin{array}{rc}{f}_{{H}}:=\underset{f\in {H}}{\mathrm{\text{argmin}}}\left\{{\displaystyle {\int }_Z}||f(x)-y|{|}_Y^{2}d\rho (x,y)\right\},& \left(4\right)\end{array}$

where f_${H}$ is called the target function. In case f_ρ ∈ ${H}$, f_${H}$ becomes the regression function f_ρ.

Because of inaccessibility of the probability distribution ρ, we minimize the regularized empirical estimate of the generalization error over the hypothesis space ${H}$,

$\begin{array}{rc}{f}_{\text{z},\lambda }:=\underset{f\in {H}}{\mathrm{\text{argmin}}}\left\{\frac{1}{m}{\displaystyle \sum _{i=1}^m}||f(x_i)-y_i|{|}_Y^2+\lambda ||f|{|}_{{H}}^2\right\},& \left(5\right)\end{array}$

where λ is the positive regularization parameter. The regularization schemes [4–7, 10] are used to incorporate various features in the solution such as boundedness, monotonicity and smoothness. In order to optimize the vector-valued regularization functional, one of the main problems is to choose the appropriate hypothesis space which is assumed to be a source to provide the estimator.

2.1. Reproducing Kernel Hilbert Space as a Hypothesis Space

Definition 2.1. (Vector-valued reproducing kernel Hilbert space) For non-empty set X and the real Hilbert space (Y, 〈·, ·〉_Y), the Hilbert space (${H}$, 〈·, ·〉_${H}$) of functions from X to Y is called reproducing kernel Hilbert space if for any x ∈ X and y ∈ Y the linear functional which maps f ∈ ${H}$ to 〈y, f(x)〉_Y is continuous.

By Riesz lemma [21], for every x ∈ X and y ∈ Y there exists a linear operator K_x : Y → ${H}$ such that

${\langle y,f(x)\rangle }_Y={\langle K_{x}y,f\rangle }_{{H}},\forall f\in {H}.$

Therefore, the adjoint operator $K_x^{*}:{H}\to Y$ is given by $K_x^{*}f=f(x)$. Through the linear operator K_x : Y → ${H}$ we define the linear operator K(x, t) : Y → Y,

$K(x,t)y:=K_{t}y(x).$

From Proposition 2.1 [22], the linear operator $K(x,t)\in {L}(Y)$ (the set of bounded linear operators on Y), K(x, t) = K(t, x)* and K(x, x) is non-negative bounded linear operator. For any m ∈ ℕ, {x_i: 1 ≤ i ≤ m} ∈ X, {y_i: 1 ≤ i ≤ m} ∈ Y, we have that ${\displaystyle \sum _{i,j=1}^m}\langle y_i,K(x_i,x_j)y_j\rangle \ge 0$. The operator valued function $K:X\times X\to {L}(Y)$ is called the kernel.

There is one to one correspondence between the kernels and reproducing kernel Hilbert spaces [22, 23]. So a reproducing kernel Hilbert space ${H}$ corresponding to a kernel K can be denoted as ${H}$_K and the norm in the space ${H}$ can be denoted as ||·||_${H}$K. In the following article, we suppress K by simply using ${H}$ for reproducing kernel Hilbert space and ||·||_${H}$ for its norm.

Throughout the paper we assume the reproducing kernel Hilbert space ${H}$ is separable such that

(i) K_x : Y → ${H}$ is a Hilbert-Schmidt operator for all x ∈ X and $\kappa :=\sqrt{\underset{x\in X}{sup}Tr(K_x^{*}{K}_x)}<\infty$.

(ii) The real function from X × X to ℝ, defined by (x, t) ↦ 〈K_tv, K_xw〉_${H}$, is measurable ∀v, w ∈ Y.

By the representation theorem [22], the solution of the penalized regularization problem (5) will be of the form:

$f_{\text{z},\lambda }={\displaystyle \sum _{i=1}^m}{K}_{x_i}{c}_i,\mathrm{\text{ for }}(c_1,\dots ,c_m)\in Y^m.$

Definition 2.2. let ${H}$ be a separable Hilbert space and ${\left\{e_k\right\}}_{k=1}^{\infty }$ be an orthonormal basis of ${H}$. Then for any positive operator $A\in {L}({H})$ we define $Tr(A)={\displaystyle \sum _{k=1}^{\infty }}\langle A{e}_k,e_k\rangle$. It is well-known that the number Tr(A) is independent of the choice of the orthonormal basis.

Definition 2.3. An operator $A\in {L}({H})$ is called Hilbert-Schmidt operator if Tr(A*A) < ∞. The family of all Hilbert-Schmidt operators is denoted by ${{L}}_2({H})$. For $A\in {{L}}_2({H})$, we define $Tr(A)={\displaystyle \sum _{k=1}^{\infty }}\langle A{e}_k,e_k\rangle$ for an orthonormal basis ${\left\{e_k\right\}}_{k=1}^{\infty }$ of ${H}$.

It is well-known that ${{L}}_2({H})$ is the separable Hilbert space with the inner product,

${\langle A,B\rangle }_{{{L}}_2({H})}=Tr(B^{*}A)$

and its norm satisfies

$\left|\right|A|{|}_{{L}({H})}\le |\left|A\right|{|}_{{{L}}_2({H})}\le Tr(\left|A\right|),$

where $\left|A\right|=\sqrt{A^{*}A}$ and $\left|\right|·|{|}_{{L}({H})}$ is the operator norm (For more details see [24]).

For the positive trace class operator $K_x{K}_x^{*}$, we have

$\left|\right|K_x{K}_x^{*}|{|}_{{L}({H})}\le |\left|K_x{K}_x^{*}\right|{|}_{{{L}}_2({H})}\le Tr(K_x{K}_x^{*})\le {\kappa }^2.$

Given the ordered set $\text{x}=(x_1,\dots ,x_m)\in X^m$, the sampling operator $S_{\text{x}}:{H}\to Y^m$ is defined by S_x(f) = (f(x₁), …, f(x_m)) and its adjoint $S_{\text{x}}^{*}:Y^m\to {H}$ is given by $S_{\text{x}}^{*}\text{y}=\frac{1}{m}{\displaystyle \sum _{i=1}^m}{K}_{x_i}{y}_i,\forall \text{y}=(y_1,\dots ,y_m)\in Y^m.$

The regularization scheme (5) can be expressed as

$\begin{array}{rc}{f}_{\text{z},\lambda }=\underset{f\in {H}}{\mathrm{\text{argmin}}}\left\{\left|\right|S_{\text{x}}f-\text{y}|{|}_m^2+\lambda |\left|f\right|{|}_{{H}}^2\right\},& \left(6\right)\end{array}$

where $\left|\right|\text{y}|{|}_m^2=\frac{1}{m}{\displaystyle \sum _{i=1}^m}|\left|y_i\right|{|}_Y^2$.

We obtain the explicit expression of f_{z, λ} by taking the functional derivative of above expression over RKHS ${H}$.

Theorem 2.1. For the positive choice of λ, the functional (6) has unique minimizer:

$\begin{array}{rc}{f}_{\text{z},\lambda }={(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1}{S}_{\text{x}}^{*}\text{y}.& \left(7\right)\end{array}$

Define f_λ as the minimizer of the optimization functional,

$\begin{array}{rc}{f}_{\lambda }:=\underset{f\in {H}}{\mathrm{\text{argmin}}}\left\{{\displaystyle {\int }_Z}||f(x)-y|{|}_Y^{2}d\rho (x,y)+\lambda ||f|{|}_{{H}}^2\right\}.& \left(8\right)\end{array}$

Using the fact ${E}(f)=\left|\right|L_K^{1/2}(f-f_{{H}})|{|}_{{H}}^2+{E}(f_{{H}})$, we get the expression of f_λ,

$\begin{array}{rc}{f}_{\lambda }={(L_K+\lambda I)}^{-1}{L}_K{f}_{{H}},& \left(9\right)\end{array}$

where the integral operator $L_K:{{L}}_{{\rho }_X}^2\to {{L}}_{{\rho }_X}^2$ is a self-adjoint, non-negative, compact operator, defined as

$L_K(f)(x):={\displaystyle {\int }_X}K(x,t)f(t)d{\rho }_X(t),x\in X.$

The integral operator L_K can also be defined as a self-adjoint operator on ${H}$. We use the same notation L_K for both the operators defined on different domains. It is well-known that $L_K^{1/2}$ is an isometry from the space of square integrable functions to reproducing kernel Hilbert space.

In order to achieve the uniform convergence rates for learning algorithms we need some prior assumptions on the probability measure ρ. Following the notion of Bauer et al. [4] and Caponnetto and De Vito [12], we consider the class of probability measures ${{P}}_{\varphi }$ which satisfies the assumptions:

(i) For the probability measure ρ on X × Y,

$\begin{array}{rc}{\displaystyle {\int }_Z}\left|\right|y|{|}_Y^{2}d\rho (x,y)<\infty .& \left(10\right)\end{array}$

(ii) The minimizer of the generalization error f_${H}$ (4) over the hypothesis space ${H}$ exists.

(iii) There exist some constants M, Σ such that for almost all x ∈ X,

$\begin{array}{rc}{\displaystyle {\int }_Y}\left(e^{\left|\right|y-f_{{H}}(x)|{|}_Y/M}-\frac{\left|\right|y-f_{{H}}(x)|{|}_Y}{M}-1\right)d\rho (y|x)\le \frac{{\Sigma }^2}{2M^2}.& \left(11\right)\end{array}$

(iv) The target function f_${H}$ belongs to the class Ω_{ϕ, R} with

$\begin{array}{rc}{\Omega }_{\varphi ,R}:=\left\{f\in {H}:f=\varphi (L_K)g\mathrm{\text{ and }}\left|\right|g|{|}_{{H}}\le R\right\},& \left(12\right)\end{array}$

where ϕ is a continuous increasing index function defined on the interval [0, κ²] with the assumption ϕ(0) = 0. This condition is usually referred to as general source condition [17].

In addition, we consider the set of probability measures ${{P}}_{\varphi ,b}$ which satisfies the conditions (i), (ii), (iii), (iv) and the eigenvalues t_n's of the integral operator L_K follow the polynomial decay: For fixed positive constants α, β and b > 1,

$\begin{array}{rc}\alpha n^{-b}\le t_n\le \beta n^{-b}\forall n\in \mathbb{N}.& \left(13\right)\end{array}$

Under the polynomial decay of the eigenvalues the effective dimension ${N}(\lambda )$, to measure the complexity of RKHS, can be estimated from Proposition 3 [12] as follows,

$\begin{array}{rc}{N}(\lambda ):=Tr({(L_K+\lambda I)}^{-1}{L}_K)\le \frac{\beta b}{b-1}{\lambda }^{-1/b},\mathrm{\text{for }}b>1& \left(14\right)\end{array}$

and without the polynomial decay condition (13), we have

${N}(\lambda )\le \left|\right|{(L_K+\lambda I)}^{-1}|{|}_{{L}({H})}Tr(L_K)\le \frac{{\kappa }^2}{\lambda }.$

We discuss the convergence issues for the learning algorithms (z → f_z ∈ ${H}$) in probabilistic sense by exponential tail inequalities such that

$Pro{b}_{\text{z}}\left\{||f_{\text{z}}-f_{\rho }|{|}_{\rho }\le \epsilon (m)log\left(\frac{1}{\eta }\right)\right\}\ge 1-\eta$

for all 0 < η ≤ 1 and ε(m) is a positive decreasing function of m. Using these probabilistic estimates we can obtain error estimates in expectation by integration of tail inequalities:

$\begin{array}{l}{E}_{\text{z}}(||f_{\text{z}}-f_{\rho }|{|}_{\rho })={\displaystyle \underset{0}{\overset{\infty }{\int }}}Pro{b}_{\text{z}}(||f_{\text{z}}-f_{\rho }|{|}_{\rho }>t)dt\\ \le {\displaystyle \underset{0}{\overset{\infty }{\int }}}exp(-\frac{t}{\epsilon (m)})dt=\epsilon (m),\end{array}$

where $\left|\right|f|{|}_{\rho }=|\left|f\right|{|}_{{{L}}_{{\rho }_X}^2}={\left\{{\int }_X\left|\right|f(x)|{|}_Y^{2}d{\rho }_X(x)\right\}}^{1/2}$ and $E_{\text{z}}(\xi )={\int }_{Z^m}\xi d\rho (z_1)\dots d\rho (z_m)$.

3. Convergence Analysis

In this section, we analyze the convergence issues of the learning algorithms on reproducing kernel Hilbert space under the smoothness priors in the supervised learning framework. We discuss the upper and lower convergence rates for vector-valued estimators in the standard minimax setting. Therefore, the estimates can be utilized particularly for scalar-valued functions and multi-task learning algorithms.

3.1. Upper Rates for Tikhonov Regularization

In General, we consider Tikhonov regularization in learning theory. Tikhonov regularization is briefly discussed in the literature [7, 9, 10, 25]. The error estimates for Tikhonov regularization are discussed theoretically under Hölder's source condition [12, 15, 16]. We establish the error estimates for Tikhonov regularization scheme under general source condition f_${H}$ ∈ Ω_ϕ,R for some continuous increasing index function ϕ and the polynomial decay of the eigenvalues of the integral operator L_K.

In order to estimate the error bounds, we consider the following inequality used in the papers [4, 12] which is based on the results of Pinelis and Sakhanenko [26].

Proposition 3.1. Let ξ be a random variable on the probability space $(\Omega ,{B},P)$ with values in real separable Hilbert space ${H}$. If there exist two constants Q and S satisfying

$\begin{array}{rc}E\left\{\left|\right|\xi -E(\xi )|{|}_{{H}}^n\right\}\le \frac{1}{2}n!S^2{Q}^{n-2}\forall n\ge 2,& \left(15\right)\end{array}$

then for any 0 < η < 1 and for all m ∈ ℕ,

$\begin{array}{l}Prob{\{({\omega }_1,\dots ,{\omega }_m)\in {\Omega }^m:\Vert \frac{1}{m}{\displaystyle {\displaystyle \sum }_{i=1}^m}[\xi ({\omega }_i)-E(\xi ({\omega }_i))]\Vert }_{{H}}\\ \le 2\left(\frac{Q}{m}+\frac{S}{\sqrt{m}}\right)\log \left(\frac{2}{\eta }\right)\}\ge 1-\eta .\end{array}$

In particular, the inequality (15) holds if

$\left|\right|\xi (\omega )|{|}_{{H}}\le QandE(\left|\right|\xi (\omega )|{|}_{{H}}^2)\le S^2.$

We estimate the error bounds for the regularized estimators by measuring the effect of random sampling and the complexity of f_${H}$. The quantities described in Proposition 3.2 express the probabilistic estimates of the perturbation measure due to random sampling. The expressions of Proposition 3.3 describe the complexity of the target function f_${H}$ which are usually referred to as the approximation errors. The approximation errors are independent of the samples z.

Proposition 3.2. Let z be i.i.d. samples drawn according to the probability measure ρ satisfying the assumptions (10), (11) and $\kappa =\sqrt{\underset{x\in X}{sup}Tr(K_x^{*}{K}_x)}$. Then for all 0 < η < 1, we have

$\begin{array}{cc}\begin{array}{l}\left|\right|{\left(L_K+\lambda I\right)}^{-1/2}\{S_x^{\ast }y-S_x^{\ast }{S}_x{f}_{{H}}\}|{|}_{{H}}\\ \le 2\left(\frac{\kappa M}{m\sqrt{\lambda }}+\sqrt{\frac{{\Sigma }^2{N}(\lambda )}{m}}\right)\log \left(\frac{4}{\eta }\right)\end{array}& \left(16\right)\end{array}$

and

$\begin{array}{rc}\left|\right|S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{{L}}_2({H})}\le 2\left(\frac{{\kappa }^2}{m}+\frac{{\kappa }^2}{\sqrt{m}}\right)log\left(\frac{4}{\eta }\right).& \left(17\right)\end{array}$

with the confidence 1 − η.

The proof of the first expression is the content of the step 3.2 of Theorem 4 [12] while the proof of the second expression can be obtained from Theorem 2 in De Vito et al. [25].

Proposition 3.3. Suppose f_${H}$ ∈ Ω_ϕ,R. Then,

(i) Under the assumption that $\varphi (t)\sqrt{t}$ and $\sqrt{t}/\varphi (t)$ are non-decreasing functions, we have

$\begin{array}{rc}\left|\right|f_{\lambda }-f_{{H}}|{|}_{\rho }\le R\varphi (\lambda )\sqrt{\lambda }.& \left(18\right)\end{array}$

(ii) Under the assumption that ϕ(t) and t/ϕ(t) are non-decreasing functions, we have

$\begin{array}{rc}\left|\right|f_{\lambda }-f_{{H}}|{|}_{\rho }\le R\kappa \varphi (\lambda )& \left(19\right)\end{array}$

and

$\begin{array}{rc}\left|\right|f_{\lambda }-f_{{H}}|{|}_{{H}}\le R\varphi (\lambda ).& \left(20\right)\end{array}$

Under the source condition f_${H}$ ∈ Ω_{ϕ, R}, the proposition can be proved using the ideas of Theorem 10 [4].

Theorem 3.1. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$ where ϕ is the index function satisfying the conditions that ϕ(t), t/ϕ(t) are non-decreasing functions. Then for all 0 < η < 1, with confidence 1 − η, for the regularized estimator f_z,λ (7) the following upper bound holds:

$\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}\le 2\left\{R\varphi (\lambda )+\frac{2\kappa M}{m\lambda }+\sqrt{\frac{4{\Sigma }^2{N}(\lambda )}{m\lambda }}\right\}log\left(\frac{4}{\eta }\right)$

provided that

$\begin{array}{rc}\sqrt{m}\lambda \ge 8{\kappa }^{2}log(4/\eta ).& \left(21\right)\end{array}$

Proof. The error of regularized solution f_{z, λ} can be estimated in terms of the sample error and the approximation error as follows:

$\begin{array}{rc}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}\le ||f_{\text{z},\lambda }-f_{\lambda }|{|}_{{H}}+\left|\right|f_{\lambda }-f_{{H}}|{|}_{{H}}.& \left(22\right)\end{array}$

Now f_{z, λ} − f_λ can be expressed as

$f_{\text{z},\lambda }-f_{\lambda }={(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1}\left\{S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{\lambda }-\lambda f_{\lambda }\right\}.$

Then $f_{\lambda }={(L_K+\lambda I)}^{-1}{L}_K{f}_{{H}}$ implies

$L_K{f}_{{H}}=L_K{f}_{\lambda }+\lambda f_{\lambda }.$

Therefore,

$f_{\text{z},\lambda }-f_{\lambda }={(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1}\left\{S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{\lambda }-L_K(f_{{H}}-f_{\lambda })\right\}.$

Employing RKHS-norm we get,

$\begin{array}{l}\left|\right|f_{z,\lambda }-f_{\lambda }|{|}_{{H}}\le |\left|{\left(S_x^{\ast }{S}_x+\lambda I\right)}^{-1}\right\{S_x^{\ast }y-S_x^{\ast }{S}_x{f}_{{H}}\\ +(S_x^{\ast }{S}_x-L_K)(f_{{H}}-f_{\lambda })\left\}\right|{|}_{{H}}\\ \le I_1{I}_2+I_3\left|\right|f_{\lambda }-f_{{H}}|{|}_{{H}}/\lambda ,& \left(23\right)\end{array}$

where $I_1=\left|\right|{(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1}{(L_K+\lambda I)}^{1/2}|{|}_{{L}({H})}$, $I_2=\left|\right|{(L_K+\lambda I)}^{-1/2}(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})|{|}_{{H}}$ and $I_3=\left|\right|S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{L}({H})}$.

The estimates of I₂, I₃ can be obtained from Proposition 3.2 and the only task is to bound I₁. For this we consider

$\begin{array}{l}{(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1}{(L_K+\lambda I)}^{1/2}={\{I-{(L_K+\lambda I)}^{-1}(L_K-S_{\text{x}}^{*}{S}_{\text{x}})\}}^{-1}\\ {(L_K+\lambda I)}^{-1/2}\end{array}$

which implies

$\begin{array}{rc}{I}_1\le {\displaystyle \sum _{n=0}^{\infty }}\left|\right|{(L_K+\lambda I)}^{-1}(L_K-S_{\text{x}}^{*}{S}_{\text{x}})\left|{|}_{{L}({H})}^n\right|\left|{(L_K+\lambda I)}^{-1/2}\right|{|}_{{L}({H})}& \left(24\right)\end{array}$

provided that $\left|\right|{(L_K+\lambda I)}^{-1}(L_K-S_{\text{x}}^{*}{S}_{\text{x}})|{|}_{{L}({H})}<1$. To verify this condition, we consider

$\left|\right|{(L_K+\lambda I)}^{-1}(S_{\text{x}}^{*}{S}_{\text{x}}-L_K)|{|}_{{L}({H})}\le I_3/\lambda .$

Now using Proposition 3.2 we get with confidence 1 − η/2,

$\left|\right|{(L_K+\lambda I)}^{-1}(S_{\text{x}}^{*}{S}_{\text{x}}-L_K)|{|}_{{L}({H})}\le \frac{4{\kappa }^2}{\sqrt{m}\lambda }log\left(\frac{4}{\eta }\right).$

From the condition (21) we get with confidence 1 − η/2,

$\begin{array}{rc}\left|\right|{(L_K+\lambda I)}^{-1}(S_{\text{x}}^{*}{S}_{\text{x}}-L_K)|{|}_{{L}({H})}\le \frac{1}{2}.& \left(25\right)\end{array}$

Consequently, using (25) in the inequality (24) we obtain with probability 1 − η/2,

$\begin{array}{l}{I}_1=\left|\right|{(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1}{(L_K+\lambda I)}^{1/2}|{|}_{{L}({H})}\\ \le 2\left|\right|{(L_K+\lambda I)}^{-1/2}|{|}_{{L}({H})}\le \frac{2}{\sqrt{\lambda }}.& \left(26\right)\end{array}$

From Proposition 3.2 we have with confidence 1 − η/2,

$\left|\right|S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{L}({H})}\le 2\left(\frac{{\kappa }^2}{m}+\frac{{\kappa }^2}{\sqrt{m}}\right)log\left(\frac{4}{\eta }\right).$

Again from the condition (21) we get with probability 1 − η/2,

$\begin{array}{rc}{I}_3=\left|\right|S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{L}({H})}\le \frac{\lambda }{2}.& \left(27\right)\end{array}$

Therefore, the inequality (23) together with (16), (20), (26), (27) provides the desired bound.□

The following theorem discuss the error estimates in ${L}$²-norm. The proof is similar to the above theorem.

Theorem 3.2. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$ and f_{z, λ} is the regularized solution (7) corresponding to Tikhonov regularization. Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds:

(i) Under the assumption that ϕ(t), $\sqrt{t}/\varphi (t)$ are non-decreasing functions,

$\begin{array}{l}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le 2\left\{R\varphi (\lambda )\sqrt{\lambda }+\frac{2\kappa M}{m\sqrt{\lambda }}+\sqrt{\frac{4{\Sigma }^2{N}(\lambda )}{m}}\right\}\\ log\left(\frac{4}{\eta }\right)\end{array}$

(ii) Under the assumption that ϕ(t), t/ϕ(t) are non-decreasing functions,

$\begin{array}{l}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le \left\{R(\kappa +\sqrt{\lambda })\varphi (\lambda )+\frac{4\kappa M}{m\sqrt{\lambda }}+\sqrt{\frac{16{\Sigma }^2{N}(\lambda )}{m}}\right\}\\ log\left(\frac{4}{\eta }\right)\end{array}$

provided that

$\begin{array}{rc}\sqrt{m}\lambda \ge 8{\kappa }^{2}log(4/\eta ).& \left(28\right)\end{array}$

We derive the convergence rates of Tikhonov regularizer based on data-driven strategy of the parameter choice of λ for the class of probability measure ${{P}}_{\varphi ,b}$.

Theorem 3.3. Under the same assumptions of Theorem 3.2 and hypothesis (13), the convergence of the estimator f_{z, λ} (7) to the target function f_${H}$ can be described as:

(i) If ϕ(t) and $\sqrt{t}/\varphi (t)$ are non-decreasing functions. Then under the parameter choice λ ∈ (0, 1], λ = Ψ⁻¹(m^−1/2) where $\Psi (t)=t^{\frac{1}{2}+\frac{1}{2b}}\varphi (t)$, we have

$\begin{array}{l}Pro{b}_{\text{z}}\left\{\begin{array}{c}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le C{({\text{Ψ}}^{-1}(m^{-1/2}))}^{1/2}\varphi \\ ({\text{Ψ}}^{-1}(m^{-1/2}))log\left(\frac{4}{\eta }\right)\end{array}\right\}\ge 1-\eta \end{array}$

and

$\begin{array}{l}\underset{\tau \to \infty }{\lim }\underset{m\to \infty }{\lim \sup }\underset{\rho \in P_{\varphi ,b}}{\sup }Pro{b}_z\{\left|\right|f_{z,\lambda }-f_{{H}}|{|}_{\rho }\\ >\tau {({\Psi }^{-1}(m^{-1/2}))}^{1/2}\varphi ({\Psi }^{-1}(m^{-1/2}))\}=0,\end{array}$

(ii) If ϕ(t) and t/ϕ(t) are non-decreasing functions. Then under the parameter choice λ ∈ (0, 1], λ = Θ⁻¹(m^−1/2) where $\Theta (t)=t^{\frac{1}{2b}}\varphi (t)$, we have

$Pro{b}_{\text{z}}\left\{||f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le C^{\prime }\varphi ({\Theta }^{-1}(m^{-1/2}))log\left(\frac{4}{\eta }\right)\right\}\ge 1-\eta$

and

$\begin{array}{l}\underset{\tau \to \infty }{\lim }\underset{m\to \infty }{\lim \sup }\underset{\rho \in P_{\varphi ,b}}{\sup }\text{}Pro{b}_z\text{}\{\left|\right|f_{z,\lambda }-f_{{H}}|{|}_{\rho }\\ >\tau \varphi ({\Theta }^{-1}(m^{-1/2}))\}=0.\end{array}$

Proof. (i) Let $\Psi (t)=t^{\frac{1}{2}+\frac{1}{2b}}\varphi (t)$. Then it follows,

${\displaystyle \underset{t\to 0}{lim}}\frac{\text{Ψ}(t)}{\sqrt{t}}={\displaystyle \underset{t\to 0}{lim}}\frac{t^2}{{\text{Ψ}}^{-1}(t)}=0.$

Under the parameter choice λ = Ψ⁻¹(m^−1/2) we have,

${\displaystyle \underset{m\to \infty }{lim}}m\lambda =\infty .$

Therefore, for sufficiently large m,

$\frac{1}{m\lambda }=\frac{{\lambda }^{\frac{1}{2b}}\varphi (\lambda )}{\sqrt{m\lambda }}\le {\lambda }^{\frac{1}{2b}}\varphi (\lambda ).$

Under the fact λ ≤ 1 from Theorem 3.2 and Equation (14) follows that with confidence 1 − η,

$\begin{array}{rc}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le C{({\text{Ψ}}^{-1}(m^{-1/2}))}^{1/2}\varphi ({\text{Ψ}}^{-1}(m^{-1/2}))log\left(\frac{4}{\eta }\right),& \left(29\right)\end{array}$

where $C=2R+4\kappa M+4\sqrt{\beta b{\Sigma }^2/(b-1)}$.

Now defining $\tau :=Clog\left(\frac{4}{\eta }\right)$ gives

$\eta ={\eta }_{\tau }=4e^{-\tau /C}.$

The estimate (29) can be reexpressed as

$\begin{array}{rc}Pro{b}_{\text{z}}\{||f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }>\tau {({\text{Ψ}}^{-1}(m^{-1/2}))}^{1/2}\varphi ({\text{Ψ}}^{-1}(m^{-1/2}))\}\le {\eta }_{\tau }.& \left(30\right)\end{array}$

(ii) Suppose $\Theta (t)=t^{\frac{1}{2b}}\varphi (t).$ Then the condition (28) follows that

$\sqrt{m\lambda }\ge \frac{8{\kappa }^{2}log(4/\eta )}{\sqrt{\lambda }}\ge \frac{8{\kappa }^2}{\sqrt{\lambda }}.$

Hence under the parameter choice λ ∈ (0, 1], λ = Θ⁻¹(m^−1/2) we have

$\frac{1}{m\sqrt{\lambda }}\le \frac{\sqrt{\lambda }}{8{\kappa }^2\sqrt{m}}\le \frac{{\lambda }^{\frac{1}{2}+\frac{1}{2b}}\varphi (\lambda )}{8{\kappa }^2}\le \frac{\varphi (\lambda )}{8{\kappa }^2}.$

From Theorem 3.2 and Equation (14), it follows that with confidence 1 − η,

$\begin{array}{rc}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le C^{\prime }\varphi ({\Theta }^{-1}(m^{-1/2}))log\left(\frac{4}{\eta }\right),& \left(31\right)\end{array}$

where $C^{\prime }:=R(\kappa +1)+M/2\kappa +4\sqrt{\beta b{\Sigma }^2/(b-1)}.$

Now defining $\tau :=C^{\prime }log\left(\frac{4}{\eta }\right)$ gives

$\eta ={\eta }_{\tau }=4e^{-\tau /C^{\prime }}.$

The estimate (31) can be reexpressed as

$\begin{array}{rc}Pro{b}_{\text{z}}\{||f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }>\tau \varphi ({\Theta }^{-1}(m^{-1/2}))\}\le {\eta }_{\tau }.& \left(32\right)\end{array}$

Then from Equations (30) and (32) our conclusions follow. □

Theorem 3.4. Under the same assumptions of Theorem 3.1 and hypothesis (13) with the parameter choice λ ∈ (0, 1], λ = Ψ⁻¹(m^−1/2) where $\Psi (t)=t^{\frac{1}{2}+\frac{1}{2b}}\varphi (t)$, the convergence of the estimator f_{z, λ} (7) to the target function f_${H}$ can be described as

$Pro{b}_{\text{z}}\left\{||f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}\le C\varphi ({\text{Ψ}}^{-1}(m^{-1/2}))log\left(\frac{4}{\eta }\right)\right\}\ge 1-\eta$

and

$\begin{array}{l}{\displaystyle \underset{\tau \to \infty }{lim}}{\displaystyle \underset{m\to \infty }{limsup}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}Pro{b}_{\text{z}}\left\{||f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}>\tau \varphi ({\text{Ψ}}^{-1}(m^{-1/2}))\right\}\\ =0.\end{array}$

The proof of the theorem follows the same steps as of Theorem 3.3 (i). We obtain the following corollary as a consequence of Theorem 3.3, 3.4.

Corollary 3.1. Under the same assumptions of Theorem 3.3, 3.4 for Tikhonov regularization with Hölder's source condition $f_{{H}}\in {\Omega }_{\varphi ,R},\varphi \left(t\right)=t^r$, for all 0 < η < 1, with confidence 1 − η, for the parameter choice $\lambda =m^{-\frac{b}{2br+b+1}}$, we have

$\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}\le C{m}^{-\frac{br}{2br+b+1}}log\left(\frac{4}{\eta }\right)\mathit{\text{for }}0\le r\le 1,$

$\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le C{m}^{-\frac{2br+b}{4br+2b+2}}log\left(\frac{4}{\eta }\right)\mathit{\text{for }}0\le r\le \frac{1}{2}$

and for the parameter choice $\lambda =m^{-\frac{b}{2br+1}}$, we have

$\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le C^{\prime }{m}^{-\frac{br}{2br+1}}log\left(\frac{4}{\eta }\right)\mathit{\text{for }}0\le r\le 1.$

3.2. Upper Rates for General Regularization Schemes

Bauer et al. [4] discussed the error estimates for general regularization schemes under general source condition. Here we study the convergence issues for general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator L_K. We define the regularization in learning theory framework similar to considered for ill-posed inverse problems (See Section 3.1 [4]).

Definition 3.1. A family of functions $g_{\lambda }:\left[0,{\kappa }^2\right]\to ℝ$, 0 < λ ≤ κ², is said to be the regularization if it satisfies the following conditions:

• $\exists D:\underset{\sigma \in (0,{\kappa }^2]}{sup}\left|\sigma g_{\lambda }(\sigma )\right|\le D$.

• $\exists B:\underset{\sigma \in (0,{\kappa }^2]}{sup}\left|g_{\lambda }(\sigma )\right|\le \frac{B}{\lambda }$.

• $\exists \gamma :\underset{\sigma \in (0,{\kappa }^2]}{sup}|1-g_{\lambda }(\sigma )\sigma |\le \gamma$.

• The maximal p satisfying the condition:

${\displaystyle \underset{\sigma \in (0,{\kappa }^2]}{sup}}|1-g_{\lambda }(\sigma )\sigma |{\sigma }^p\le {\gamma }_p{\lambda }^p$

is called the qualification of the regularization g_λ, where γ_p does not depend on λ.

The properties of general regularization are satisfied by the large class of learning algorithms which are essentially all the linear regularization schemes. We refer to Section 2.2 [10] for brief discussion of the regularization schemes. Here we consider general regularized solution corresponding to the above regularization:

$\begin{array}{ll}{f}_{\text{z},\lambda }=g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})S_{\text{x}}^{*}\text{y}.& (33)\end{array}$

Here we are discussing the connection between the qualification of the regularization and general source condition [17].

Definition 3.2. The qualification p covers the index function ϕ if the function $t\to \frac{t^p}{\varphi (t)}$ on t ∈ (0, κ²] is non-decreasing.

The following result is a restatement of Proposition 3 [17].

Proposition 3.4. Suppose ϕ is a non-decreasing index function and the qualification of the regularization g_λ covers ϕ. Then

${\displaystyle \underset{\sigma \in (0,{\kappa }^2]}{sup}}|1-g_{\lambda }(\sigma )\sigma |\varphi (\sigma )\le c_g\varphi (\lambda ),c_g=max(\gamma ,{\gamma }_p).$

Generally, the index function ϕ is not stable with respect to perturbation in the integral operator L_K. In practice, we are only accessible to the perturbed empirical operator $S_{\text{x}}^{*}{S}_{\text{x}}$ but the source condition can be expressed in terms of L_K only. So we want to control the difference $\varphi (L_K)-\varphi (S_{\text{x}}^{*}{S}_{\text{x}})$. In order to obtain the error estimates for general regularization, we further restrict the index functions to operator monotone functions which is defined as

Definition 3.3. A function ϕ₁:[0, d] → [0, ∞) is said to be operator monotone index function if ϕ₁(0) = 0 and for every non-negative pair of self-adjoint operators A, B such that ||A||, ||B|| ≤ d and A ≤ B we have ϕ₁(A) ≤ ϕ₁(B).

We consider the class of operator monotone index functions:

$\begin{array}{l}{{F}}_{\mu }=\{{\varphi }_1:[0,{\kappa }^2]\to [0,\infty ) \text{operator monotone},\\ {\varphi }_1(0)=0,{\varphi }_1({\kappa }^2)\le \mu \}.\end{array}$

For the above class of operator monotone functions from Theorem 1 [4], given ϕ₁ ∈ F_μ there exists c_ϕ1 such that

$||{\varphi }_1(S_{\text{x}}^{*}{S}_{\text{x}})-{\varphi }_1(L_K)|{|}_{{L}({H})}\le c_{{\varphi }_1}{\varphi }_1(||S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{L}({H})}).$

Here we observe that the rate of convergence of ${\varphi }_1(S_{\text{x}}^{*}{S}_{\text{x}})$ to ϕ₁(L_K) is slower than the convergence rate of $S_{\text{x}}^{*}{S}_{\text{x}}$ to L_K. Therefore, we consider the following class of index functions:

$\begin{array}{l}{F}=\{\varphi ={\varphi }_2{\varphi }_1:{\varphi }_1\in {{F}}_{\mu },{\varphi }_2:[0,{\kappa }^2]\\ \to [0,\infty )\text{non-decreasing Lipschitz},{\varphi }_2(0)=0\}.\end{array}$

The splitting of ϕ = ϕ₂ϕ₁ is not unique. So we can take ϕ₂ as a Lipschitz function with Lipschitz constant 1. Now using Corollary 1.2.2 [27] we get

$\left|\right|{\varphi }_2(S_{\text{x}}^{*}{S}_{\text{x}})-{\varphi }_2(L_K)|{|}_{{{L}}_2({H})}\le ||S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{{L}}_2({H})}.$

General source condition f_${H}$ ∈ Ω_ϕ,R corresponding to index class functions ${F}$ covers wide range of source conditions as Hölder's source condition ϕ(t) = t^r, logarithm source condition $\varphi (t)=t^p{log}^{-\nu }(\frac{1}{t})$. Following the analysis of Bauer et al. [4] we develop the error estimates of general regularization for the index class function ${F}$ under the suitable priors on the probability measure ρ.

Theorem 3.5. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$. Suppose f_z,λ is the regularized solution (33) corresponding to general regularization and the qualification of the regularization covers ϕ. Then for all 0 < η < 1, with confidence 1 − η, the following upper bound holds:

$\begin{array}{l}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}\le \left\{\begin{array}{c}R{c}_g(1+c_{{\varphi }_1})\varphi (\lambda )+\frac{4R\mu \gamma {\kappa }^2}{\sqrt{m}}+\frac{2\sqrt{2}{\nu }_1\kappa M}{m\lambda }\\ +\sqrt{\frac{8{\nu }_1^2{\Sigma }^2{N}(\lambda )}{m\lambda }}\end{array}\right\}\\ log\left(\frac{4}{\eta }\right)\end{array}$

provided that

$\begin{array}{ll}\sqrt{m}\lambda \ge 8{\kappa }^{2}log(4/\eta ).& (34)\end{array}$

Proof. We consider the error expression for general regularized solution (33),

$\begin{array}{ll}{f}_{\text{z},\lambda }-f_{{H}}=g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})-r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}},& (35)\end{array}$

where r_λ(σ) = 1 − g_λ(σ)σ.

Now the first term can be expressed as

$\begin{array}{l}{g}_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})=g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{1/2}\\ {(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1/2}{(L_K+\lambda I)}^{1/2}\\ {(L_K+\lambda I)}^{-1/2}(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}}).\end{array}$

On applying RKHS-norm we get,

$\begin{array}{l}||g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})|{|}_{{H}}\le I_2{I}_5||g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})\\ {(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{1/2}|{|}_{{L}({H})},& (36)\end{array}$

where $I_2=||{(L_K+\lambda I)}^{-1/2}(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})|{|}_{{H}}$ and $I_5=||{(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1/2}{(L_K+\lambda I)}^{1/2}|{|}_{{L}({H})}$.

The estimate of I₂ can be obtained from the first estimate of Proposition 3.2 and from the second estimate of Proposition 3.2 with the condition (34) we obtain with probability 1 − η/2,

$\begin{array}{l}\left|\right|{(L_K+\lambda I)}^{-1/2}(L_K-S_{\text{x}}^{*}{S}_{\text{x}}){(L_K+\lambda I)}^{-1/2}|{|}_{{L}({H})}\\ \le \frac{1}{\lambda }\left|\right|S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{L}({H})}\le \frac{4{\kappa }^2}{\sqrt{m}\lambda }log\left(\frac{4}{\eta }\right)\le \frac{1}{2}.\end{array}$

which implies that with confidence 1 − η/2,

$\begin{array}{l}{I}_5=\left|\right|{\left(S_x^{\ast }{S}_x+\lambda I\right)}^{-1/2}{\left(L_K+\lambda I\right)}^{1/2}|{|}_{{L}({H})}\\ =\left|\right|{\left(L_K+\lambda I\right)}^{1/2}{\left(S_x^{\ast }{S}_x+\lambda I\right)}^{-1}{\left(L_K+\lambda I\right)}^{1/2}|{|}_{{L}({H})}^{1/2}\\ =\left|\right|\{I-{\left(L_K+\lambda I\right)}^{-1/2}(L_K-S_x^{\ast }{S}_x)\\ {\left(L_K+\lambda I\right)}^{-1/2}{\}}^{-1}|{|}_{{L}({H})}^{1/2}\\ \le \sqrt{2}.& (37)\end{array}$

From the properties of the regularization we have,

$\begin{array}{l}\left|\right|g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){(S_{\text{x}}^{*}{S}_{\text{x}})}^{1/2}|{|}_{{L}({H})}\le {\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma )\sqrt{\sigma }|\\ ={\left({\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma )\sigma |{\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma )|\right)}^{1/2}\le \sqrt{\frac{BD}{\lambda }}.& (38)\end{array}$

Hence it follows,

$\begin{array}{l}\left|\right|g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{1/2}|{|}_{{L}({H})}\le {\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma ){(\sigma +\lambda )}^{1/2}|\\ \le {\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}\left|g_{\lambda }(\sigma )\sqrt{\sigma }\right|+\sqrt{\lambda }{\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}\left|g_{\lambda }(\sigma )\right|\le \frac{{\nu }_1}{\sqrt{\lambda }},& (39)\end{array}$

where ${\nu }_1:=B+\sqrt{BD}$.

Therefore, using (16), (37) and (39) in Equation (36) we conclude that with probability 1 − η,

$\begin{array}{lll}\left|\right|g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})|{|}_{{H}}& \le & 2\sqrt{2}{\nu }_1\left\{\frac{\kappa M}{m\lambda }+\sqrt{\frac{{\Sigma }^2{N}(\lambda )}{m\lambda }}\right\}\\ log\left(\frac{4}{\eta }\right).& (40)\end{array}$

Now we consider the second term,

$\begin{array}{l}{r}_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}=r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})\varphi (L_K)v=r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})\varphi (S_{\text{x}}^{*}{S}_{\text{x}})v\\ +r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){\varphi }_2(S_{\text{x}}^{*}{S}_{\text{x}})({\varphi }_1(L_K)-{\varphi }_1(S_{\text{x}}^{*}{S}_{\text{x}}))v\\ +r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})({\varphi }_2(L_K)-{\varphi }_2(S_{\text{x}}^{*}{S}_{\text{x}})){\varphi }_1(L_K)v.\end{array}$

Employing RKHS-norm we get

$\begin{array}{l}\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}|{|}_{{H}}\le R{c}_g\varphi (\lambda )+R{c}_g{c}_{{\varphi }_1}{\varphi }_2(\lambda ){\varphi }_1\\ (||L_K-S_{\text{x}}^{*}{S}_{\text{x}}|{|}_{{L}({H})})+R\mu \gamma \left|\right|L_K-S_{\text{x}}^{*}{S}_{\text{x}}|{|}_{{{L}}_2({H})}.\end{array}$

Here we used the fact that if the qualification of the regularization covers ϕ = ϕ₁ϕ₂, then the qualification also covers ϕ₁ and ϕ₂ both separately.

From Equations (17) and (34) we have with probability 1 − η/2,

$\begin{array}{ll}||S_{\text{x}}^{*}{S}_{\text{x}}-L_K|{|}_{{L}({H})}\le \frac{4{\kappa }^2}{\sqrt{m}}log\left(\frac{4}{\eta }\right)\le \lambda /2.& (41)\end{array}$

Therefore, with probability 1 − η/2,

$\begin{array}{ll}\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}|{|}_{{H}}\le R{c}_g(1+c_{{\varphi }_1})\varphi (\lambda )+\frac{4R\mu \gamma {\kappa }^2}{\sqrt{m}}log\left(\frac{4}{\eta }\right).& (42)\end{array}$

Combining the bounds (40) and (42) we get the desired result.

                                      □

Theorem 3.6. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi }$ and f_z,λ is the regularized solution (33) corresponding to general regularization. Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds:

(i) If the qualification of the regularization covers ϕ,

$\begin{array}{l}||f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le \left\{\begin{array}{c}R{c}_g(1+c_{{\varphi }_1})(\kappa +\sqrt{\lambda })\varphi (\lambda )\\ +\frac{4R\mu \gamma {\kappa }^2(\kappa +\sqrt{\lambda })}{\sqrt{m}}+\frac{2\sqrt{2}{\nu }_2\kappa M}{m\sqrt{\lambda }}\\ +\sqrt{\frac{8{\nu }_2^2{\Sigma }^2{N}(\lambda )}{m}}\hfill \end{array}\right\}log\left(\frac{4}{\eta }\right),\\ \end{array}$

(ii) If the qualification of the regularization covers $\varphi (t)\sqrt{t}$,

$\begin{array}{l}\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le \left\{\begin{array}{c}2R{c}_g(1+c_{{\varphi }_1})\varphi (\lambda )\sqrt{\lambda }+\frac{4R\mu (\gamma +c_g){\kappa }^2\sqrt{\lambda }}{\sqrt{m}}\\ +\frac{2\sqrt{2}{\nu }_2\kappa M}{m\sqrt{\lambda }}+\sqrt{\frac{8{\nu }_2^2{\Sigma }^2{N}(\lambda )}{m}}\hfill \end{array}\right\}\\ log\left(\frac{4}{\eta }\right)\end{array}$

provided that

$\begin{array}{ll}\sqrt{m}\lambda \ge 8{\kappa }^{2}log(4/\eta ).& (43)\end{array}$

Proof. Here we establish ${L}$²-norm estimate for the error expression:

$\begin{array}{l}{f}_{\text{z},\lambda }-f_{{H}}=g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})-r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}.\end{array}$

On applying ${L}$²-norm in the first term we get,

$\begin{array}{l}\left|\right|g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})|{|}_{\rho }\le I_2{I}_5||L_K^{1/2}{g}_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})\\ {(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{1/2}|{|}_{{L}({H})},& (44)\end{array}$

where $I_2=\left|\right|{(L_K+\lambda I)}^{-1/2}(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})|{|}_{{H}}$ and $I_5=\left|\right|{(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{-1/2}{(L_K+\lambda I)}^{1/2}|{|}_{{L}({H})}$.

The estimates of I₂ and I₅ can be obtained from Proposition 3.2 and Theorem 3.5 respectively. Now we consider

$\begin{array}{l}||L_K^{1/2}{g}_{\lambda }(S_x^{\ast }{S}_x){\left(S_x^{\ast }{S}_x+\lambda I\right)}^{1/2}|{|}_{{L}({H})}\le ||L_K^{1/2}\\ -{\left(S_x^{\ast }{S}_x\right)}^{1/2}|{|}_{{L}({H})}||g_{\lambda }(S_x^{\ast }{S}_x){\left(S_x^{\ast }{S}_x+\lambda I\right)}^{1/2}\\ |{|}_{{L}({H})}+||{\left(S_x^{\ast }{S}_x\right)}^{1/2}{g}_{\lambda }(S_x^{\ast }{S}_x)\\ {\left(S_x^{\ast }{S}_x+\lambda I\right)}^{1/2}|{|}_{{L}({H})}.\end{array}$

Since $\varphi (t)=\sqrt{t}$ is operator monotone function. Therefore, from Equation (41) with probability 1 − η/2, we get

$\left|\right|L_K^{1/2}-{(S_{\text{x}}^{*}{S}_{\text{x}})}^{1/2}|{|}_{{L}({H})}\le {(||L_K-S_{\text{x}}^{*}{S}_{\text{x}}|{|}_{{L}({H})})}^{1/2}\le \sqrt{\lambda }.$

Then using the properties of the regularization and Equation (38) we conclude that with probability 1 − η/2,

$\begin{array}{l}\left|\right|L_K^{1/2}{g}_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){(S_{\text{x}}^{*}{S}_{\text{x}}+\lambda I)}^{1/2}|{|}_{{L}({H})}\\ \le \sqrt{\lambda }{\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma ){(\sigma +\lambda )}^{1/2}|+{\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma ){({\sigma }^2+\lambda \sigma )}^{1/2}|\\ \le {\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma )\sigma |+\lambda {\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma )|+2\sqrt{\lambda }{\displaystyle \underset{0<\sigma \le {\kappa }^2}{sup}}|g_{\lambda }(\sigma )\sqrt{\sigma }|\\ \le B+D+2\sqrt{BD}={\nu }_2\text{(let)}.& (45)\end{array}$

From Equations (44) with Equations (16), (37), and (45) we obtain with probability 1 − η,

$\begin{array}{l}\left|\right|g_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})(S_{\text{x}}^{*}\text{y}-S_{\text{x}}^{*}{S}_{\text{x}}{f}_{{H}})|{|}_{\rho }\le 2\sqrt{2}{\nu }_2\left\{\frac{\kappa M}{m\sqrt{\lambda }}+\sqrt{\frac{{\Sigma }^2{N}(\lambda )}{m}}\right\}\\ log\left(\frac{4}{\eta }\right).& (46)\end{array}$

The second term can be expressed as

$\begin{array}{l}\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}|{|}_{\rho }\le ||L_K^{1/2}-{(S_{\text{x}}^{*}{S}_{\text{x}})}^{1/2}|{|}_{{L}({H})}\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}|{|}_{{H}}\\ +\left|\right|{(S_{\text{x}}^{*}{S}_{\text{x}})}^{1/2}{r}_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}|{|}_{{H}}\\ \le \left|\right|L_K-S_{\text{x}}^{*}{S}_{\text{x}}\left|{|}_{{L}({H})}^{1/2}\right|\left|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}\right|{|}_{{H}}\\ +\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){(S_{\text{x}}^{*}{S}_{\text{x}})}^{1/2}\varphi (S_{\text{x}}^{*}{S}_{\text{x}})v|{|}_{{H}}\\ +\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){(S_{\text{x}}^{*}{S}_{\text{x}})}^{1/2}{\varphi }_2(S_{\text{x}}^{*}{S}_{\text{x}})({\varphi }_1(S_{\text{x}}^{*}{S}_{\text{x}})-{\varphi }_1(L_K))v|{|}_{{H}}\\ +\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}}){(S_{\text{x}}^{*}{S}_{\text{x}})}^{1/2}({\varphi }_2(S_{\text{x}}^{*}{S}_{\text{x}})-{\varphi }_2(L_K)){\varphi }_1(L_K)v|{|}_{{H}}.\end{array}$

Here two cases arises:

Case 1. If the qualification of the regularization covers ϕ. Then we get with confidence 1 − η/2,

$\begin{array}{l}\left|\right|r_{\lambda }(S_x^{\ast }{S}_x)f_{{H}}|{|}_{\rho }\le (\kappa +\sqrt{\lambda })(R{c}_g(1+c_{{\varphi }_1})\varphi (\lambda )\\ +R\mu \gamma \left|\right|S_x^{\ast }{S}_x-L_K|{|}_{{{L}}_2({H})}).\end{array}$

Therefore, using Equation (17) we obtain with probability 1 − η/2,

$\begin{array}{l}\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}|{|}_{\rho }\le (\kappa +\sqrt{\lambda })\\ \left(R{c}_g(1+c_{{\varphi }_1})\varphi (\lambda )+\frac{4R\mu \gamma {\kappa }^2}{\sqrt{m}}log\left(\frac{4}{\eta }\right)\right).& (47)\end{array}$

Case 2. If the qualification of the regularization covers $\varphi (t)\sqrt{t}$, we get with probability 1 − η/2,

$\begin{array}{l}\left|\right|r_{\lambda }(S_{\text{x}}^{*}{S}_{\text{x}})f_{{H}}|{|}_{\rho }\le 2R{c}_g(1+c_{{\varphi }_1})\varphi (\lambda )\sqrt{\lambda }\\ +4R\mu (\gamma +c_g){\kappa }^2\sqrt{\frac{\lambda }{m}}log\left(\frac{4}{\eta }\right).& (48)\end{array}$

Combining the error estimates (46), (47) and (48) we get the desired results.

                                           □

We discuss the convergence rates of general regularizer based on data-driven strategy of the parameter choice of λ for the class of probability measure ${{P}}_{\varphi ,b}$. The proof of Theorem 3.7, 3.8 are similar to Theorem 3.3.

Theorem 3.7. Under the same assumptions of Theorem 3.5 and hypothesis (13) with the parameter choice λ ∈ (0, 1], λ = Ψ⁻¹(m^−1/2) where $\Psi (t)=t^{\frac{1}{2}+\frac{1}{2b}}\varphi (t)$, the convergence of the estimator f_z,λ (33) to the target function f_${H}$ can be described as

$Pro{b}_{\text{z}}\left\{||f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}\le \tilde{C}\varphi ({\text{Ψ}}^{-1}(m^{-1/2}))log\left(\frac{4}{\eta }\right)\right\}\ge 1-\eta ,$

where $\tilde{C}=R{c}_g(1+c_{{\varphi }_1})+4R\mu \gamma {\kappa }^2+2\sqrt{2}{\nu }_1\kappa M+\sqrt{8\beta b{\nu }_1^2{\Sigma }^2/(b-1)}$ and

$\begin{array}{l}{\displaystyle \underset{\tau \to \infty }{lim}}{\displaystyle \underset{m\to \infty }{lim sup}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}Pro{b}_{\text{z}}\left\{\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}>\tau \varphi ({\text{Ψ}}^{-1}(m^{-1/2}))\right\}\\ =0.\end{array}$

Theorem 3.8. Under the same assumptions of Theorem 3.6 and hypothesis (13), the convergence of the estimator f_z,λ (33) to the target function f_${H}$ can be described as

(i) If the qualification of the regularization covers ϕ. Then under the parameter choice λ ∈ (0, 1], λ = Θ⁻¹(m^−1/2) where $\Theta (t)=t^{\frac{1}{2b}}\varphi (t)$, we have

$Pro{b}_{\text{z}}\left\{\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le {\tilde{C}}_1\varphi ({\Theta }^{-1}(m^{-1/2}))\text{log}\left(\frac{4}{\eta }\right)\right\}\ge 1-\eta$

where ${\tilde{C}}_1=R{c}_g(1+c_{{\varphi }_1})(\kappa +1)+4R\mu \gamma {\kappa }^2(\kappa +1)+{\nu }_{2}M/2\sqrt{2}\kappa +\sqrt{8\beta b{\nu }_2^2{\Sigma }^2/(b-1)}$ and

$\begin{array}{l}{\displaystyle \underset{\tau \to \infty }{lim}}{\displaystyle \underset{m\to \infty }{\text{lim sup}}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}Pro{b}_{\text{z}}\left\{\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }>\tau \varphi ({\text{Θ}}^{-1}(m^{-1/2}))\right\}\\ =0,\end{array}$

(ii) If the qualification of the regularization covers $\varphi (t)\sqrt{t}$. Then under the parameter choice λ ∈ (0, 1], λ = Ψ⁻¹(m^−1/2) where $\Psi (t)=t^{\frac{1}{2}+\frac{1}{2b}}\varphi (t)$, we have

$\begin{array}{l}Pro{b}_{\text{z}}\begin{array}{c}\left\{\right||f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le \widetilde{C_2}{({\text{Ψ}}^{-1}(m^{-1/2}))}^{1/2}\varphi \\ ({\text{Ψ}}^{-1}(m^{-1/2}))log\left(\frac{4}{\eta }\right)\}\ge 1-\eta \end{array}\end{array}$

where ${\tilde{C}}_2=2R{c}_g(1+c_{{\varphi }_1})+4R\mu (\gamma +c_g){\kappa }^2+2\sqrt{2}{\nu }_2\kappa M+\sqrt{8\beta b{\nu }_2^2{\Sigma }^2/(b-1)}$ and

$\begin{array}{l}Pro{b}_z\{||f_{z,\lambda }-f_{{H}}|{|}_{\rho }\le {\tilde{C}}_2{({\Psi }^{-1}(m^{-1/2}))}^{1/2}\varphi ({\Psi }^{-1}(m^{-1/2}))\\ \log \left(\frac{4}{\eta }\right)\text{}\}\ge \text{}1-\eta ,\end{array}$

We obtain the following corollary as a consequence of Theorem 3.7, 3.8.

Corollary 3.2. Under the same assumptions of Theorem 3.7, 3.8 for general regularization of the qualification p with Hölder's source condition $f_{{H}}\in {\Omega }_{\varphi ,R},\varphi (t)=t^r$, for all 0 < η < 1, with confidence 1 − η, for the parameter choice $\lambda =m^{-\frac{b}{2br+b+1}}$, we have

$\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{{H}}\le \tilde{C}{m}^{-\frac{br}{2br+b+1}}log\left(\frac{4}{\eta }\right)\text{for }0\le r\le p,$

$\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le {\tilde{C}}_2{m}^{-\frac{2br+b}{4br+2b+2}}log\left(\frac{4}{\eta }\right)\text{for }0\le r\le p-\frac{1}{2}$

and for the parameter choice $\lambda =m^{-\frac{b}{2br+1}}$, we have

$\left|\right|f_{\text{z},\lambda }-f_{{H}}|{|}_{\rho }\le {\tilde{C}}_1{m}^{-\frac{br}{2br+1}}log\left(\frac{4}{\eta }\right)\text{for }0\le r\le p.$

Remark 3.1. It is important to observe from Corollary 3.1, 3.2 that using the concept of operator monotonicity of index function we are able to achieve the same error estimates for general regularization as of Tikhonov regularization up to a constant multiple.

Remark 3.2. (Related work) Corollary 3.1 provides the order of convergence same as of Theorem 1 [12] for Tikhonov regularization under the Hölder's source condition f_${H}$ ∈ Ω_{ϕ, R} for $\varphi (t)=t^r(\frac{1}{2}\le r\le 1)$ and the polynomial decay of the eigenvalues (13). Blanchard and Mücke [18] addressed the convergence rates for inverse statistical learning problem for general regularization under the Hölder's source condition with the assumption f_ρ ∈ ${H}$. In particular, the upper convergence rates discussed in Blanchard and Mücke [18] agree with Corollary 3.2 for considered learning problem which is referred as direct learning problem in Blanchard and Mücke[18]. Under the fact ${N}(\lambda )\le \frac{{\kappa }^2}{\lambda }$ from Theorem 3.5, 3.6 we obtain the similar estimates as of Theorem 10 [4] for general regularization schemes without the polynomial decay condition of the eigenvalues (13).

Remark 3.3. For the real valued functions and multi-task algorithms (the output space Y ⊂ ℝ^m for some m ∈ ℕ) we can obtain the error estimates from our analysis without imposing any condition on the conditional probability measure (11) for the bounded output space Y.

Remark 3.4. We can address the convergence issues of binary classification problem [28] using our error estimates as similar to discussed in Section 3.3 [4] and Section 5 [16].

3.3. Lower Rates for General Learning Algorithms

In this section, we discuss the estimates of minimum possible error over a subclass of the probability measures ${{P}}_{\varphi ,b}$ parameterized by suitable functions f ∈ ${H}$. Throughout this section we assume that Y is finite-dimensional.

Let ${\left\{v_j\right\}}_{j=1}^d$ be a basis of Y and f ∈ Ω_{ϕ, R}. Then we parameterize the probability measure based on the function f,

$\begin{array}{ll}{\rho }_f(x,y):=\frac{1}{2dL}{\displaystyle \sum _{j=1}^d}(a_j(x){\delta }_{y+dL{v}_j}+b_j(x){\delta }_{y-dL{v}_j})\nu (x),& (49)\end{array}$

where a_j(x) = L − 〈f,K_xv_j〉_${H}$, b_j(x) = L + 〈f,K_xv_j〉_${H}$, L = 4κϕ(κ²)R and δ_ξ denotes the Dirac measure with unit mass at ξ. It is easy to observe that the marginal distribution of ρ_f over X is ν and the regression function for the probability measure ρ_f is f (see Proposition 4 [12]). In addition to this, for the conditional probability measure ρ_f(y|x) we have,

$\begin{array}{l}{\displaystyle {\int }_Y}\left(e^{\left|\right|y-f(x)|{|}_Y/M}-\frac{\left|\right|y-f(x)|{|}_Y}{M}-1\right)d{\rho }_f(y|x)\\ \le (d^2{L}^2-\left|\right|f(x)|{|}_Y^2){\displaystyle \sum _{i=2}^{\infty }}\frac{{(dL+\left|\right|f(x)|{|}_Y)}^{i-2}}{M^{i}i!}\le \frac{{\Sigma }^2}{2M^2}\end{array}$

provided that

$dL+L/4\le M\text{ and }\sqrt{2}dL\le \Sigma .$

We assume that the eigenvalues of the integral operator L_K follow the polynomial decay (13) for the marginal probability measure ν. Then we conclude that the probability measure ρ_f parameterized by f belongs to the class ${{P}}_{\varphi ,b}$.

The concept of information theory such as the Kullback-Leibler information and Fano inequalities (Lemma 3.3 [29]) are the main ingredients in the analysis of lower bounds. In the literature [12, 29], the closeness of probability measures is described through Kullback-Leibler information: Given two probability measures ρ₁ and ρ₂, it is defined as

${K}({\rho }_1,{\rho }_2):={\displaystyle {\int }_Z}log(g(z))d{\rho }_1(z),$

where g is the density of ρ₁ with respect to ρ₂, that is, ${\rho }_1(E)={\int }_{E}g(z)d{\rho }_2(z)$ for all measurable sets E.

Following the analysis of Caponnetto and De Vito [12] and DeVore et al. [29] we establish the lower rates of accuracy that can be attained by any learning algorithm.

To estimate the lower rates of learning algorithms, we generate N_ε-functions belonging to Ω_ϕ,R for given ε > 0 such that (53), (54) holds. Then we construct the probability measures ${\rho }_i\in {{P}}_{\varphi ,b}$ from Equation (49), parameterized by these functions f_i's (1 ≤ i ≤ N_ε). On applying Lemma 3.3 [29], we obtain the lower convergence rates using Kullback-Leibler information.

Theorem 3.9. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi ,b}$ under the hypothesis dim(Y) = d < ∞. Then for any learning algorithm (z → f_z ∈ ${H}$) there exists a probability measure ${\rho }_{*}\in {{P}}_{\varphi ,b}$ and f_ρ* ∈ ${H}$ such that for all 0 < ε < ε_o, f_z can be approximated as

$Pro{b}_{\text{z}}\left\{||f_{\text{z}}-f_{{\rho }_{*}}|{|}_{{H}}>\epsilon /2\right\}\ge min\left\{\frac{1}{1+e^{-{\ell }_{\epsilon }/24}},\vartheta e^{\left(\frac{{\ell }_{\epsilon }}{48}-\frac{cm{\epsilon }^2}{{\ell }_{\epsilon }^b}\right)}\right\}$

where ϑ = e^−3/e, $c=\frac{64\beta }{15(b-1)d{L}^2}(1-\frac{1}{2^{b-1}})$ and ${\ell }_{\epsilon }=\lfloor \frac{1}{2}{\left(\frac{\alpha }{{\varphi }^{-1}(\epsilon /R)}\right)}^{1/b}\rfloor$.

Proof. For given ε > 0, we define

$g={\displaystyle \sum _{n=\ell +1}^{2\ell }}\frac{\epsilon {\sigma }^{n-\ell }{e}_n}{\sqrt{\ell }\varphi \left(t_n\right)},$

where σ = (σ¹, …, σ^ℓ) ∈ {−1, +1}^ℓ, t_n's are the eigenvalues of the integral operator L_K, e_n's are the eigenvectors of the integral operator L_K and the orthonormal basis of RKHS ${H}$. Under the decay condition on the eigenvalues $\alpha \le n^b{t}_n$, we get

$\left|\right|g|{|}_{{H}}^2={\displaystyle \sum _{n=\ell +1}^{2\ell }}\frac{{\epsilon }^2}{\ell {\varphi }^2\left(t_n\right)}\le {\displaystyle \sum _{n=\ell +1}^{2\ell }}\frac{{\epsilon }^2}{\ell {\varphi }^2\left(\frac{\alpha }{n^b}\right)}\le \frac{{\epsilon }^2}{{\varphi }^2\left(\frac{\alpha }{2^b{\ell }^b}\right)}.$

Hence f = ϕ(L_K)g ∈ Ω_ϕ,R provided that ||g||_${H}$ ≤ R or equivalently,

$\begin{array}{ll}\ell \le \frac{1}{2}{\left(\frac{\alpha }{{\varphi }^{-1}\left(\epsilon /R\right)}\right)}^{1/b}.& (50)\end{array}$

For ℓ = ${\ell }_{\epsilon }=\lfloor \frac{1}{2}{\left(\frac{\alpha }{{\varphi }^{-1}(\epsilon /R)}\right)}^{1/b}\rfloor$, choose ε_o such that ℓ_εo > 16. Then according to Proposition 6 [12], for every positive ε < ε_o (ℓ_ε > ℓ_εo) there exists N_ε ∈ ℕ and ${\sigma }_1,\dots ,{\sigma }_{N_{\epsilon }}\in {\left\{-1,+1\right\}}^{{\ell }_{\epsilon }}$ such that

$\begin{array}{ll}{\displaystyle \sum _{n=1}^{{\ell }_{\epsilon }}}{\left({\sigma }_i^n-{\sigma }_j^n\right)}^2\ge {\ell }_{\epsilon },\mathrm{\text{for all }}1\le i,j\le N_{\epsilon },i\ne j& (51)\end{array}$

and

$\begin{array}{ll}{N}_{\epsilon }\ge e^{{\ell }_{\epsilon }/24}.& (52)\end{array}$

Now we suppose f_i = ϕ(L_K)g_i and for ε > 0,

$g_i={\displaystyle \sum _{n={\ell }_{\epsilon }+1}^{2{\ell }_{\epsilon }}}\frac{\epsilon {\sigma }_i^{n-{\ell }_{\epsilon }}{e}_n}{\sqrt{{\ell }_{\epsilon }}\varphi \left(t_n\right)},\mathrm{\text{for }}i=1,\dots ,N_{\epsilon },$

where ${\sigma }_i=({\sigma }_i^1,\dots ,{\sigma }_i^{{\ell }_{\epsilon }})\in {\left\{-1,+1\right\}}^{{\ell }_{\epsilon }}$. Then from Equation (51) we get,

$\begin{array}{ll}\epsilon \le \left|\right|f_i-f_j|{|}_{{H}},\mathrm{\text{for all }}1\le i,j\le N_{\epsilon },i\ne j.& (53)\end{array}$

For 1 ≤ i, j ≤ N_ε, we have

$\begin{array}{l}\left|\right|f_i-f_j|{|}_{{{L}}_{\nu }^2\left(X\right)}^2\le {\displaystyle \sum _{n={\ell }_{\epsilon }+1}^{2{\ell }_{\epsilon }}}\frac{\beta {\epsilon }^2{\left({\sigma }_i^{n-{\ell }_{\epsilon }}-{\sigma }_j^{n-{\ell }_{\epsilon }}\right)}^2}{{\ell }_{\epsilon }{n}^b}\le {\displaystyle \sum _{n={\ell }_{\epsilon }+1}^{2{\ell }_{\epsilon }}}\frac{4\beta {\epsilon }^2}{{\ell }_{\epsilon }{n}^b}\\ \le \frac{4\beta {\epsilon }^2}{{\ell }_{\epsilon }}{\displaystyle {\int }_{{\ell }_{\epsilon }}^{2{\ell }_{\epsilon }}}\frac{1}{x^b}dx=c^{\prime }\frac{{\epsilon }^2}{{\ell }_{\epsilon }^b},& (54)\end{array}$

where $c^{\prime }=\frac{4\beta }{\left(b-1\right)}\left(1-\frac{1}{2^{b-1}}\right)$.

We define the sets,

$A_i=\left\{\text{z}:||f_{\text{z}}-f_i|{|}_{{H}}<\frac{\epsilon }{2}\right\},\mathrm{\text{for }}1\le i\le N_{\epsilon }.$

It is clear from Equation (53) that A_i's are disjoint sets. On applying Lemma 3.3 [29] with probability measures ${\rho }_{f_i}^m,1\le i\le N_{\epsilon }$, we obtain that either

$\begin{array}{ll}p:={\displaystyle \underset{1\le i\le N_{\epsilon }}{max}}{\rho }_{f_i}^m\left(A_i^c\right)\ge \frac{N_{\epsilon }}{N_{\epsilon }+1}& (55)\end{array}$

or

$\begin{array}{ll}{\displaystyle \underset{1\le j\le N_{\epsilon }}{min}}\frac{1}{N_{\epsilon }}{\displaystyle \sum _{i=1,i\ne j}^{N_{\epsilon }}}{K}\left({\rho }_{f_i}^m,{\rho }_{f_j}^m\right)\ge {\text{Ψ}}_{N_{\epsilon }}\left(p\right),& (56)\end{array}$

where ${\text{Ψ}}_{N_{\epsilon }}(p)=log(N_{\epsilon })+(1-p)log(\frac{1-p}{p})-plog(\frac{N_{\epsilon }-p}{p})$. Further,

$\begin{array}{l}{\text{Ψ}}_{N_{\epsilon }}\left(p\right)\ge \left(1-p\right)log\left(N_{\epsilon }\right)+\left(1-p\right)log\left(1-p\right)-log\left(p\right)\\ +2plog\left(p\right)\ge -log\left(p\right)+log\left(\sqrt{N_{\epsilon }}\right)-3/e.& (57)\end{array}$

Since minimum value of x log(x) is −1/e on [0, 1].

For the joint probability measures ${\rho }_{f_i}^m$, ${\rho }_{f_j}^m$ $({\rho }_{f_i},{\rho }_{f_j}\in {{P}}_{\varphi ,b},1\le i,j\le N_{\epsilon })$ from Proposition 4 [12] and the Equation (54) we get,

$\begin{array}{ll}{K}({\rho }_{f_i}^m,{\rho }_{f_j}^m)=m{K}\left({\rho }_{f_i},{\rho }_{f_j}\right)\le \frac{16m}{15d{L}^2}\left|\right|f_i-f_j|{|}_{{{L}}_{\nu }^2\left(X\right)}^2\le \frac{cm{\epsilon }^2}{{\ell }_{\epsilon }^b},& (58)\end{array}$

where c = 16c′/15dL².

Therefore, Equations (55), (56), together with Equations (57) and (58) implies

$\begin{array}{l}p:={\displaystyle \underset{1\le i\le N_{\epsilon }}{max}}\left(Prob\left\{\text{z}:||f_{\text{z}}-f_i|{|}_{{H}}>\frac{\epsilon }{2}\right\}\right)\\ \ge min\left\{\frac{N_{\epsilon }}{N_{\epsilon }+1},\sqrt{N_{\epsilon }}{e}^{-\frac{3}{e}-\frac{cm{\epsilon }^2}{{\ell }_{\epsilon }^b}}\right\}.\end{array}$

From Equation (52) for the probability measure ρ_* such that $p={\rho }_{*}^m(A_i^c)$ follows the result.                                                                           □

The lower estimates in ${L}$²-norm can be obtained similar to above theorem.

Theorem 3.10. Let z be i.i.d. samples drawn according to the probability measure $\rho \in {{P}}_{\varphi ,b}$ under the hypothesis dim(Y) = d < ∞. Then for any learning algorithm (z → f_z ∈ ${H}$) there exists a probability measure ${\rho }_{*}\in {{P}}_{\varphi ,b}$ and f_ρ* ∈ ${H}$ such that for all 0 < ε < ε_o, f_z can be approximated as

$\begin{array}{l}Pro{b}_{\text{z}}\left\{||f_{\text{z}}-f_{{\rho }_{*}}|{|}_{{{L}}_{\nu }^2\left(X\right)}>\epsilon /2\right\}\\ \ge min\left\{\frac{1}{1+e^{-{\ell }_{\epsilon }/24}},\vartheta e^{\left(\frac{{\ell }_{\epsilon }}{48}-\frac{64m{\epsilon }^2}{15d{L}^2}\right)}\right\}\end{array}$

where ϑ = e^−3/e, ${\ell }_{\epsilon }=\lfloor {\left(\frac{\alpha }{{\Psi }^{-1}(\epsilon /R)}\right)}^{1/b}\rfloor$ and $\psi (t)=\sqrt{t}\varphi (t)$.

Theorem 3.11. Under the same assumptions of Theorem 3.10 for ψ(t) = t^1/2ϕ(t) and $\text{Ψ}(t)=t^{\frac{1}{2}+\frac{1}{2b}}\varphi (t)$, the estimator f_z corresponding to any learning algorithm converges to the regression function f_ρ with the following lower rate:

$\begin{array}{l}\underset{\tau \to 0}{\lim }\underset{m\to \infty }{\lim \inf }\underset{l\in \mathcal{A}}{\inf }\underset{\rho \in {{P}}_{\varphi ,b}}{\sup }Pro{b}_z\{||f_z^l-f_{\rho }|{|}_{{{L}}_{\nu }^2(X)}>\tau \psi \\ \left({\Psi }^{-1}(m^{-1/2})\right)\}=1,\end{array}$

where ${A}$ denotes the set of all learning algorithms $l:\text{z}\to f_{\text{z}}^l$.

Proof. Under the condition ${\ell }_{\epsilon }=\lfloor {\left(\frac{\alpha }{{\Psi }^{-1}(\epsilon /R)}\right)}^{1/b}\rfloor$ from Theorem 3.10 we get,

$\begin{array}{l}Pro{b}_{\text{z}}\left\{||f_{\text{z}}-f_{{\rho }_{*}}|{|}_{{{L}}_{\nu }^2\left(X\right)}>\frac{\epsilon }{2}\right\}\\ \ge min\left\{\frac{1}{1+e^{-{\ell }_{\epsilon }/24}},\vartheta e^{-\frac{1}{48}}{e}^{\left\{\frac{1}{48}{\left(\frac{\alpha }{{\psi }^{-1}\left(\epsilon /R\right)}\right)}^{1/b}-\frac{64m{\epsilon }^2}{15d{L}^2}\right\}}\right\}.\end{array}$

Choosing ${\epsilon }_m=\tau R\psi ({\text{Ψ}}^{-1}(m^{-1/2}))$, we obtain

$\begin{array}{l}Pro{b}_{\text{z}}\left\{||f_{\text{z}}-f_{{\rho }_{*}}|{|}_{{{L}}_{\nu }^2\left(X\right)}>\tau \frac{R}{2}\psi \left({\text{Ψ}}^{-1}\left(m^{-1/2}\right)\right)\right\}\\ \ge min\left\{\frac{1}{1+e^{-{\ell }_{\epsilon }/24}},\vartheta e^{-\frac{1}{48}}{e}^{c{\left({\text{Ψ}}^{-1}\left(m^{-1/2}\right)\right)}^{-1/b}}\right\},\end{array}$

where $c=\left(\frac{{\alpha }^{1/b}}{48}-\frac{64R^2{\tau }^2}{15d{L}^2}\right)>0$ for $0<\tau <min\left(\frac{\sqrt{5d}L{\alpha }^{\frac{1}{2b}}}{32R},1\right)$.

Now as m goes to ∞, ε → 0 and ℓ_ε → ∞. Therefore, for c > 0 we conclude that

$\begin{array}{l}{\displaystyle \underset{m\to \infty }{lim inf}}{\displaystyle \underset{l\in {A}}{inf}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}Pro{b}_{\text{z}}\left\{||f_{\text{z}}^l-f_{\rho }|{|}_{{{L}}_{\nu }^2\left(X\right)}>\tau \frac{R}{2}\psi \left({\text{Ψ}}^{-1}\left(m^{-1/2}\right)\right)\right\}=1.\end{array}$

                                                                           □

Choosing ${\epsilon }_m=\tau R\varphi ({\text{Ψ}}^{-1}(m^{-1/2}))$ we get the following convergence rate from Theorem 3.9.

Theorem 3.12. Under the same assumptions of Theorem 3.10 for $\Psi (t)=t^{\frac{1}{2}+\frac{1}{2b}}\varphi (t)$, the estimator f_z corresponding to any learning algorithm converges to the regression function f_ρ with the following lower rate:

$\begin{array}{l}{\displaystyle \underset{\tau \to 0}{lim}}{\displaystyle \underset{m\to \infty }{lim inf}}{\displaystyle \underset{l\in {A}}{inf}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}Pro{b}_{\text{z}}\left\{||f_{\text{z}}^l-f_{\rho }|{|}_{{H}}>\tau \varphi \left({\text{Ψ}}^{-1}\left(m^{-1/2}\right)\right)\right\}=1.\end{array}$

We obtain the following corollary as a consequence of Theorem 3.11, 3.12.

Corollary 3.3. For any learning algorithm under Hölder's source condition $f_{\rho }\in {\Omega }_{\varphi ,R},\varphi (t)=t^r$ and the polynomial decay condition (13) for b > 1, the lower convergence rates can be described as

$\begin{array}{l}{\displaystyle \underset{\tau \to 0}{lim}}{\displaystyle \underset{m\to \infty }{lim inf}}{\displaystyle \underset{l\in {A}}{inf}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}Pro{b}_{\text{z}}\left\{||f_{\text{z}}^l-f_{\rho }|{|}_{{{L}}_{\nu }^2\left(X\right)}>\tau m^{-\frac{2br+b}{4br+2b+2}}\right\}=1\end{array}$

and

${\displaystyle \underset{\tau \to 0}{lim}}{\displaystyle \underset{m\to \infty }{lim inf}}{\displaystyle \underset{l\in {A}}{inf}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}Pro{b}_{\text{z}}\left\{||f_{\text{z}}^l-f_{\rho }|{|}_{{H}}>\tau m^{-\frac{br}{2br+b+1}}\right\}=1.$

If the minimax lower rate coincides with the upper convergence rate for λ = λ_m. Then the choice of parameter is said to be optimal. For the parameter choice λ = Ψ⁻¹(m^−1/2), Theorem 3.3 and Theorem 3.8 share the upper convergence rate with the lower convergence rate of Theorem 3.11 in ${L}$²-norm. For the same parameter choice, Theorem 3.4 and Theorem 3.7 share the upper convergence rate with the lower convergence rate of Theorem 3.12 in RKHS-norm. Therefore, the choice of the parameter is optimal.

It is important to observe that we get the same convergence rates for b = 1.

3.4. Individual Lower Rates

In this section, we discuss the individual minimax lower rates that describe the behavior of the error for the class of probability measure ${{P}}_{\varphi ,b}$ as the sample size m grows.

Definition 3.4. A sequence of positive numbers a_n (n ∈ ℕ) is called the individual lower rate of convergence for the class of probability measure ${P}$, if

${\displaystyle \underset{l\in {A}}{inf}}{\displaystyle \underset{\rho \in {P}}{sup}}{\displaystyle \underset{m\to \infty }{lim sup}}\left(\frac{E_{\text{z}}\left(||f_{\text{z}}^l-f_{{H}}|{|}^2\right)}{a_m}\right)>0,$

where ${A}$ denotes the set of all learning algorithms $l:\text{z}\mapsto f_{\text{z}}^l$.

Theorem 3.13. Let z be i.i.d. samples drawn according to the probability measure ${{P}}_{\varphi ,b}$ where ϕ is the index function satisfying the conditions that $\varphi (t)/t^{r_1}$, $t^{r_2}/\varphi (t)$ are non-decreasing functions and dim(Y) = d < ∞. Then for every ε > 0, the following lower bound holds:

${\displaystyle \underset{l\in {A}}{inf}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}{\displaystyle \underset{m\to \infty }{lim sup}}\left(\frac{E_{\text{z}}\left(||f_{\text{z}}^l-f_{{H}}|{|}_{{{L}}_{\nu }^2\left(X\right)}^2\right)}{m^{-\left(b{c}_2+\epsilon \right)/\left(b{c}_1+\epsilon +1\right)}}\right)>0,$

where c₁ = 2r₁ + 1 and c₂ = 2r₂ + 1.

We consider the class of probability measures such that the target function f_${H}$ is parameterized by $\text{s}={(s_n)}_{n=1}^{\infty }\in {\left\{-1,+1\right\}}^{\infty }$. Suppose for ε > 0,

$g={\displaystyle \sum _{n=1}^{\infty }}{s}_{n}R\sqrt{\frac{\epsilon }{\epsilon +1}\frac{\alpha }{n^b{t}_n}}\left(\frac{\varphi \left(\alpha /n^b\right)}{\varphi \left(t_n\right)}\right)n^{-\left(\epsilon +1\right)/2}{e}_n,$

where $\text{s}={(s_n)}_{n=1}^{\infty }\in {\left\{-1,+1\right\}}^{\infty }$, t_n's are the eigenvalues of the integral operator L_K, e_n's are the eigenvectors of the integral operator L_K and the orthonormal basis of RKHS ${H}$. Then the target function f_${H}$ = ϕ(L_K)g satisfies the general source condition. We assume that the conditional probability measure ρ(y|x) follows the normal distribution centered at f_${H}$ and the marginal probability measure ρ_X = ν. Now we can derive the individual lower rates over the considered class of probability measures from the ideas of the literature [12, 30].

Theorem 3.14. Let z be i.i.d. samples drawn according to the probability measure ${{P}}_{\varphi ,b}$ where ϕ is the index function satisfying the conditions that $\varphi (t)/t^{r_1}$, $t^{r_2}/\varphi (t)$ are non-decreasing functions and dim(Y) = d < ∞. Then for every ε > 0, the following lower bound holds:

${\displaystyle \underset{l\in {A}}{inf}}{\displaystyle \underset{\rho \in {{P}}_{\varphi ,b}}{sup}}{\displaystyle \underset{m\to \infty }{lim sup}}\left(\frac{E_{\text{z}}\left(||f_{\text{z}}^l-f_{{H}}|{|}_{{H}}^2\right)}{m^{-\left(b{c}_2-b+\epsilon \right)/\left(b{c}_1+\epsilon +1\right)}}\right)>0.$

4. Conclusion

In our analysis we derive the upper and lower convergence rates over the wide class of probability measures considering general source condition in vector-valued setting. In particular, our minimax rates can be used for the scalar-valued functions and multi-task learning problems. The lower convergence rates coincide with the upper convergence rates for the optimal parameter choice based on smoothness parameters b, ϕ. We can also develop various parameter choice rules such as balancing principle [31], quasi-optimality principle [32], discrepancy principle [33] for the regularized solutions provided in our analysis.

Author Contributions

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors are grateful to the reviewers for their helpful comments and pointing out a subtle error that led to improve the quality of the paper.

References

1. Cucker F, Smale, S. On the mathematical foundations of learning. Bull Am Math Soc. (2002) 39:1–49. doi: 10.1090/S0273-0979-01-00923-5

CrossRef Full Text | Google Scholar

2. Evgeniou T, Pontil, M Poggio, T. Regularization networks and support vector machines. Adv Comput Math. (2000) 13:1–50. doi: 10.1023/A:1018946025316

CrossRef Full Text | Google Scholar

3. Vapnik VN, Vapnik V. Statistical Learning Theory. New York, NY: Wiley (1998).

PubMed Abstract | Google Scholar

4. Bauer F, Pereverzev S, Rosasco L. On regularization algorithms in learning theory. J Complex. (2007) 23:52–72. doi: 10.1016/j.jco.2006.07.001

CrossRef Full Text | Google Scholar

5. Engl HW, Hanke M, Neubauer A Regularization of Inverse Problems. Dordrecht: Kluwer Academic Publishers Group (1996).

Google Scholar

6. Gerfo LL, Rosasco L, Odone F, De Vito E, Verri A. Spectral algorithms for supervised learning. Neural Comput. (2008) 20:1873–97. doi: 10.1162/neco.2008.05-07-517

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Tikhonov AN, Arsenin VY. Solutions of Ill-Posed Problems. Washington, DC: W. H. Winston (1977).

Google Scholar

8. Bousquet O, Boucheron S, Lugosi G. Introduction to statistical learning theory. In: Bousquet O, von Luxburg U, Ratsch G editors. Advanced Lectures on Machine Learning, Volume 3176 of Lecture Notes in Computer Science. Berlin; Heidelberg: Springer (2004). pp. 169–207.

Google Scholar

9. Cucker F, Zhou DX. Learning Theory: An Approximation Theory Viewpoint. Cambridge, UK: Cambridge Monographs on Applied and Computational Mathematics, Cambridge University Press (2007).

Google Scholar

10. Lu S, Pereverzev S. Regularization Theory for Ill-posed Problems: Selected Topics, Berlin: DeGruyter (2013).

Google Scholar

11. Abhishake Sivananthan S. Multi-penalty regularization in learning theory. J Complex. (2016) 36:141–65. doi: 10.1016/j.jco.2016.05.003

CrossRef Full Text | Google Scholar

12. Caponnetto A, De Vito E. Optimal rates for the regularized least-squares algorithm. Found Comput Math. (2007) 7:331–68. doi: 10.1007/s10208-006-0196-8

CrossRef Full Text | Google Scholar

13. Smale S, Zhou DX. Estimating the approximation error in learning theory. Anal Appl. (2003) 1:17–41. doi: 10.1142/S0219530503000089

CrossRef Full Text | Google Scholar

14. Smale S, Zhou DX. Shannon sampling and function reconstruction from point values. Bull Am Math Soc. (2004) 41:279–306. doi: 10.1090/S0273-0979-04-01025-0

CrossRef Full Text | Google Scholar

15. Smale S, Zhou DX. Shannon sampling II: connections to learning theory. Appl Comput Harmon Anal. (2005) 19:285–302. doi: 10.1016/j.acha.2005.03.001

CrossRef Full Text | Google Scholar

16. Smale S, Zhou DX. Learning theory estimates via integral operators and their approximations. Constr Approx. (2007) 26:153–72. doi: 10.1007/s00365-006-0659-y

CrossRef Full Text | Google Scholar

17. Mathé P, Pereverzev SV. Geometry of linear ill-posed problems in variable Hilbert scales. Inverse Probl. (2003) 19:789–803. doi: 10.1088/0266-5611/19/3/319

CrossRef Full Text | Google Scholar

18. Blanchard G, Mücke N. Optimal rates for regularization of statistical inverse learning problems. arXiv:1604.04054 (2016).

Google Scholar

19. Mendelson S. On the performance of kernel classes. J Mach Learn Res. (2003) 4:759–71.

Google Scholar

20. Zhang T. Effective dimension and generalization of kernel learning. In: Thrun S, Becker S, Obermayer K. editors. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, (2003). pp. 454–61.

Google Scholar

21. Akhiezer NI, Glazman IM. Theory of Linear Operators in Hilbert Space, Translated from the Russian and with a preface by Merlynd Nestell. New York, NY: Dover Publications Inc (1993).

Google Scholar

22. Micchelli CA, Pontil M. On learning vector-valued functions. Neural Comput. (2005) 17:177–204. doi: 10.1162/0899766052530802

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Aronszajn N. Theory of reproducing kernels. Trans Am Math Soc. (1950) 68:337–404. doi: 10.1090/S0002-9947-1950-0051437-7

CrossRef Full Text | Google Scholar

24. Reed M, Simon B. Functional Analysis, Vol. 1, San Diego, CA: Academic Press (1980).

25. De Vito E, Rosasco L, Caponnetto A, De Giovannini U, Odone F. Learning from examples as an inverse problem. J Mach Learn Res. (2005) 6:883–904.

Google Scholar

26. Pinelis IF, Sakhanenko AI. Remarks on inequalities for the probabilities of large deviations. Theory Prob Appl. (1985) 30:127–31. doi: 10.1137/1130013

CrossRef Full Text | Google Scholar

27. Peller VV. Multiple operator integrals in perturbation theory. Bull Math Sci. (2016) 6:15–88. doi: 10.1007/s13373-015-0073-y

CrossRef Full Text | Google Scholar

28. Boucheron S, Bousquet O, Lugosi G. Theory of classification: a survey of some recent advances. ESAIM: Prob Stat. (2005) 9:323–75. doi: 10.1051/ps:2005018

CrossRef Full Text | Google Scholar

29. DeVore R, Kerkyacharian G, Picard D, Temlyakov V. Approximation methods for supervised learning. Found Comput Math. (2006) 6:3–58. doi: 10.1007/s10208-004-0158-6

CrossRef Full Text | Google Scholar

30. Györfi L, Kohler M, Krzyzak A, Walk H. A Distribution-Free Theory of Nonparametric Regression. New York, NY: Springer Series in Statistics, Springer-Verlag (2002).

Google Scholar

31. De Vito E, Pereverzyev S, Rosasco L. Adaptive kernel methods using the balancing principle. Found Comput Math. (2010) 10:455–79. doi: 10.1007/s10208-010-9064-2

CrossRef Full Text | Google Scholar

32. Bauer F, Reiss M. Regularization independent of the noise level: an analysis of quasi-optimality. Inverse Prob. (2008) 24:055009. doi: 10.1088/0266-5611/24/5/055009

CrossRef Full Text | Google Scholar

33. Lu S, Pereverzev SV, Tautenhahn U. A model function method in regularized total least squares. Appl Anal. (2010) 89:1693–703. doi: 10.1080/00036811.2010.492502

CrossRef Full Text | Google Scholar