Skip to main content

ORIGINAL RESEARCH article

Front. Appl. Math. Stat., 27 March 2017
Sec. Mathematics of Computation and Data Science

Optimal Rates for the Regularized Learning Algorithms under General Source Condition

  • Department of Mathematics, Indian Institute of Technology Delhi, New Delhi, India

We consider the learning algorithms under general source condition with the polynomial decay of the eigenvalues of the integral operator in vector-valued function setting. We discuss the upper convergence rates of Tikhonov regularizer under general source condition corresponding to increasing monotone index function. The convergence issues are studied for general regularization schemes by using the concept of operator monotone index functions in minimax setting. Further we also address the minimum possible error for any learning algorithm.

1. Introduction

Learning theory [13] aims to learn the relation between the inputs and outputs based on finite random samples. We require some underlying space to search the relation function. From the experiences we have some idea about the underlying space which is called hypothesis space. Learning algorithms tries to infer the best estimator over the hypothesis space such that f(x) gives the maximum information of the output variable y for any unseen input x. The given samples {xi,yi}i=1m are not exact in the sense that for underlying relation function f(xi) ≠ yi but f(xi) ≈ yi. We assume that the uncertainty follows the probability distribution ρ on the sample space X × Y and the underlying function (called the regression function) for the probability distribution ρ is given by

fρ(x)=Yydρ(y|x),  xX,

where ρ(y|x) is the conditional probability measure for given x. The problem of obtaining estimator from examples is ill-posed. Therefore, we apply the regularization schemes [47] to stabilize the problem. Various regularization schemes are studied for inverse problems. In the context of learning theory [2, 3, 810], the square loss-regularization (Tikhonov regularization) is widely considered to obtain the regularized estimator [9, 1116]. Gerfo et al. [6] introduced general regularization in the learning theory and provided the error bounds under Hölder's source condition [5]. Bauer et al. [4] discussed the convergence issues for general regularization under general source condition [17] by removing the Lipschitz condition on the regularization considered in Gerfo et al. [6]. Caponnetto and De Vito [12] discussed the square-loss regularization under the polynomial decay of the eigenvalues of the integral operator LK with Hölder's source condition. For the inverse statistical learning problem, Blanchard and Mücke [18] analyzed the convergence rates for general regularization scheme under Hölder's source condition in scalar-valued function setting. Here we are discussing the convergence issues of general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator in vector-valued framework. We present the minimax upper convergence rates for Tikhonov regularization under general source condition Ωϕ, R, for a monotone increasing index function ϕ. For general regularization the minimax rates are obtained using the operator monotone index function ϕ. The concept of effective dimension [19, 20] is exploited to achieve the convergence rates. In the choice of regularization parameters, the effective dimension plays the important role. We also discuss the lower convergence rates for any learning algorithm under the smoothness conditions. We present the results in vector-valued function setting. Therefore, in particular they can be applied to multi-task learning problems.

The structure of the paper is as follows. In the second section, we introduce some basic assumptions and notations for supervised learning problems. In Section 3, we present the upper and lower convergence rates under the smoothness conditions in minimax setting.

2. Learning From Examples: Notations and Assumptions

In the learning theory framework [2, 3, 810], the sample space Z = X × Y consists of two spaces: The input space X (locally compact second countable Hausdorff space) and the output space (Y, 〈·, ·〉Y) (the real separable Hilbert space). The input space X and the output space Y are related by some unknown probability distribution ρ on Z. The probability measure can be split as ρ(x, y) = ρ(y|xX(x), where ρ(y|x) is the conditional probability measure of y given x and ρX is the marginal probability measure on X. The only available information is the random i.i.d. samples z = ((x1, y1), …, (xm, ym)) drawn according to the probability measure ρ. Given the training set z, learning theory aims to develop an algorithm which provides an estimator fz : XY such that fz(x) predicts the output variable y for any given input x. The goodness of the estimator can be measured by the generalization error of a function f which can be defined as

E(f):=Eρ(f)=ZV(f(x),y)dρ(x,y),    (1)

where V : Y × Y → ℝ is the loss function. The minimizer of E(f) for the square loss function V(f(x),y)=||f(x)-y||Y2 is given by

fρ(x):=Yydρ(y|x),    (2)

where fρ is called the regression function. The regression function fρ belongs to the space of square integrable functions provided that

Z||y||Y2 dρ(x,y)<.    (3)

We search the minimizer of the generalization error over a hypothesis space H,

fH:=argminfH{Z||f(x)-y||Y2dρ(x,y)},    (4)

where fH is called the target function. In case fρH, fH becomes the regression function fρ.

Because of inaccessibility of the probability distribution ρ, we minimize the regularized empirical estimate of the generalization error over the hypothesis space H,

fz,λ:=argminfH{1mi=1m||f(xi)-yi||Y2+λ||f||H2},    (5)

where λ is the positive regularization parameter. The regularization schemes [47, 10] are used to incorporate various features in the solution such as boundedness, monotonicity and smoothness. In order to optimize the vector-valued regularization functional, one of the main problems is to choose the appropriate hypothesis space which is assumed to be a source to provide the estimator.

2.1. Reproducing Kernel Hilbert Space as a Hypothesis Space

Definition 2.1. (Vector-valued reproducing kernel Hilbert space) For non-empty set X and the real Hilbert space (Y, 〈·, ·〉Y), the Hilbert space (H, 〈·, ·〉H) of functions from X to Y is called reproducing kernel Hilbert space if for any x ∈ X and y ∈ Y the linear functional which maps f ∈ H toy, f(x)〉Y is continuous.

By Riesz lemma [21], for every xX and yY there exists a linear operator Kx : YH such that

y,f(x)Y=Kxy,fH,      fH.

Therefore, the adjoint operator Kx*:HY is given by Kx*f=f(x). Through the linear operator Kx : YH we define the linear operator K(x, t) : YY,

K(x,t)y:=Kty(x).

From Proposition 2.1 [22], the linear operator K(x,t)L(Y) (the set of bounded linear operators on Y), K(x, t) = K(t, x)* and K(x, x) is non-negative bounded linear operator. For any m ∈ ℕ, {xi: 1 ≤ im} ∈ X, {yi: 1 ≤ im} ∈ Y, we have that i,j=1myi,K(xi,xj)yj0. The operator valued function K:X×XL(Y) is called the kernel.

There is one to one correspondence between the kernels and reproducing kernel Hilbert spaces [22, 23]. So a reproducing kernel Hilbert space H corresponding to a kernel K can be denoted as HK and the norm in the space H can be denoted as ||·||HK. In the following article, we suppress K by simply using H for reproducing kernel Hilbert space and ||·||H for its norm.

Throughout the paper we assume the reproducing kernel Hilbert space H is separable such that

(i) Kx : YH is a Hilbert-Schmidt operator for all xX and κ:=supxXTr(Kx*Kx)<.

(ii) The real function from X × X to ℝ, defined by (x, t) ↦ 〈Ktv, KxwH, is measurable ∀v, wY.

By the representation theorem [22], the solution of the penalized regularization problem (5) will be of the form:

fz,λ=i=1mKxici, for (c1,,cm)Ym.

Definition 2.2. let H be a separable Hilbert space and {ek}k=1 be an orthonormal basis of H. Then for any positive operator AL(H) we define Tr(A)=k=1Aek,ek. It is well-known that the number Tr(A) is independent of the choice of the orthonormal basis.

Definition 2.3. An operator AL(H) is called Hilbert-Schmidt operator if Tr(A*A) < ∞. The family of all Hilbert-Schmidt operators is denoted by L2(H). For AL2(H), we define Tr(A)=k=1Aek,ek for an orthonormal basis {ek}k=1 of H.

It is well-known that L2(H) is the separable Hilbert space with the inner product,

A,BL2(H)=Tr(B*A)

and its norm satisfies

||A||L(H)||A||L2(H)Tr(|A|),

where |A|=A*A and ||·||L(H) is the operator norm (For more details see [24]).

For the positive trace class operator KxKx*, we have

||KxKx*||L(H)||KxKx*||L2(H)Tr(KxKx*)κ2.

Given the ordered set x=(x1,,xm)Xm, the sampling operator Sx:HYm is defined by Sx(f) = (f(x1), …, f(xm)) and its adjoint Sx*:YmH is given by Sx*y=1mi=1mKxiyi,  y=(y1,,ym)Ym.

The regularization scheme (5) can be expressed as

fz,λ=argminfH{||Sxf-y||m2+λ||f||H2},    (6)

where ||y||m2=1mi=1m||yi||Y2.

We obtain the explicit expression of fz, λ by taking the functional derivative of above expression over RKHS H.

Theorem 2.1. For the positive choice of λ, the functional (6) has unique minimizer:

fz,λ=(Sx*Sx+λI)-1Sx*y.    (7)

Define fλ as the minimizer of the optimization functional,

fλ:=argminfH{Z||f(x)-y||Y2dρ(x,y)+λ||f||H2}.    (8)

Using the fact E(f)=||LK1/2(f-fH)||H2+E(fH), we get the expression of fλ,

fλ=(LK+λI)-1LKfH,    (9)

where the integral operator LK:LρX2LρX2 is a self-adjoint, non-negative, compact operator, defined as

LK(f)(x):=XK(x,t)f(t)dρX(t),  xX.

The integral operator LK can also be defined as a self-adjoint operator on H. We use the same notation LK for both the operators defined on different domains. It is well-known that LK1/2 is an isometry from the space of square integrable functions to reproducing kernel Hilbert space.

In order to achieve the uniform convergence rates for learning algorithms we need some prior assumptions on the probability measure ρ. Following the notion of Bauer et al. [4] and Caponnetto and De Vito [12], we consider the class of probability measures Pϕ which satisfies the assumptions:

(i) For the probability measure ρ on X × Y,

Z||y||Y2 dρ(x,y)<.    (10)

(ii) The minimizer of the generalization error fH (4) over the hypothesis space H exists.

(iii) There exist some constants M, Σ such that for almost all xX,

Y(e||y-fH(x)||Y/M-||y-fH(x)||YM-1)dρ(y|x)Σ22M2.    (11)

(iv) The target function fH belongs to the class Ωϕ, R with

Ωϕ,R:={fH:f=ϕ(LK)g and ||g||HR},    (12)

where ϕ is a continuous increasing index function defined on the interval [0, κ2] with the assumption ϕ(0) = 0. This condition is usually referred to as general source condition [17].

In addition, we consider the set of probability measures Pϕ,b which satisfies the conditions (i), (ii), (iii), (iv) and the eigenvalues tn's of the integral operator LK follow the polynomial decay: For fixed positive constants α, β and b > 1,

αn-btnβn-b  n.    (13)

Under the polynomial decay of the eigenvalues the effective dimension N(λ), to measure the complexity of RKHS, can be estimated from Proposition 3 [12] as follows,

N(λ):=Tr((LK+λI)-1LK)βbb-1λ-1/b,for b>1    (14)

and without the polynomial decay condition (13), we have

N(λ)||(LK+λI)-1||L(H)Tr(LK)κ2λ.

We discuss the convergence issues for the learning algorithms (zfzH) in probabilistic sense by exponential tail inequalities such that

Probz{||fz-fρ||ρε(m)log(1η)}1-η

for all 0 < η ≤ 1 and ε(m) is a positive decreasing function of m. Using these probabilistic estimates we can obtain error estimates in expectation by integration of tail inequalities:

Ez(||fz-fρ||ρ)=0Probz(||fz-fρ||ρ>t)dt                             0exp(-tε(m))dt=ε(m),

where ||f||ρ=||f||LρX2={X||f(x)||Y2dρX(x)}1/2 and Ez(ξ)=Zmξdρ(z1)dρ(zm).

3. Convergence Analysis

In this section, we analyze the convergence issues of the learning algorithms on reproducing kernel Hilbert space under the smoothness priors in the supervised learning framework. We discuss the upper and lower convergence rates for vector-valued estimators in the standard minimax setting. Therefore, the estimates can be utilized particularly for scalar-valued functions and multi-task learning algorithms.

3.1. Upper Rates for Tikhonov Regularization

In General, we consider Tikhonov regularization in learning theory. Tikhonov regularization is briefly discussed in the literature [7, 9, 10, 25]. The error estimates for Tikhonov regularization are discussed theoretically under Hölder's source condition [12, 15, 16]. We establish the error estimates for Tikhonov regularization scheme under general source condition fH ∈ Ωϕ,R for some continuous increasing index function ϕ and the polynomial decay of the eigenvalues of the integral operator LK.

In order to estimate the error bounds, we consider the following inequality used in the papers [4, 12] which is based on the results of Pinelis and Sakhanenko [26].

Proposition 3.1. Let ξ be a random variable on the probability space (Ω,B,P) with values in real separable Hilbert space H. If there exist two constants Q and S satisfying

E{||ξ-E(ξ)||Hn}12n!S2Qn-2   n2,    (15)

then for any 0 < η < 1 and for all m ∈ ℕ,

Prob{(ω1,,ωm)Ωm:1mi=1m[ξ(ωi)E(ξ(ωi))]H2(Qm+Sm)log(2η)}1η.

In particular, the inequality (15) holds if

||ξ(ω)||HQ and E(||ξ(ω)||H2)S2.

We estimate the error bounds for the regularized estimators by measuring the effect of random sampling and the complexity of fH. The quantities described in Proposition 3.2 express the probabilistic estimates of the perturbation measure due to random sampling. The expressions of Proposition 3.3 describe the complexity of the target function fH which are usually referred to as the approximation errors. The approximation errors are independent of the samples z.

Proposition 3.2. Let z be i.i.d. samples drawn according to the probability measure ρ satisfying the assumptions (10), (11) and κ=supxXTr(Kx*Kx). Then for all 0 < η < 1, we have

||(LK+λI)1/2{SxySxSxfH}||H2(κMmλ+Σ2N(λ)m)log(4η)(16)

and

||Sx*Sx-LK||L2(H)2(κ2m+κ2m)log(4η).    (17)

with the confidence 1 − η.

The proof of the first expression is the content of the step 3.2 of Theorem 4 [12] while the proof of the second expression can be obtained from Theorem 2 in De Vito et al. [25].

Proposition 3.3. Suppose fH ∈ Ωϕ,R. Then,

(i) Under the assumption that ϕ(t)t and t/ϕ(t) are non-decreasing functions, we have

||fλ-fH||ρRϕ(λ)λ.    (18)

(ii) Under the assumption that ϕ(t) and t/ϕ(t) are non-decreasing functions, we have

||fλ-fH||ρRκϕ(λ)    (19)

and

||fλ-fH||HRϕ(λ).    (20)

Under the source condition fH ∈ Ωϕ, R, the proposition can be proved using the ideas of Theorem 10 [4].

Theorem 3.1. Let z be i.i.d. samples drawn according to the probability measure ρPϕ where ϕ is the index function satisfying the conditions that ϕ(t), t/ϕ(t) are non-decreasing functions. Then for all 0 < η < 1, with confidence 1 − η, for the regularized estimator fz (7) the following upper bound holds:

||fz,λ-fH||H2{Rϕ(λ)+2κMmλ+4Σ2N(λ)mλ}log(4η)

provided that

mλ8κ2log(4/η).    (21)

Proof. The error of regularized solution fz, λ can be estimated in terms of the sample error and the approximation error as follows:

||fz,λ-fH||H||fz,λ-fλ||H+||fλ-fH||H.    (22)

Now fz, λfλ can be expressed as

fz,λ-fλ=(Sx*Sx+λI)-1{Sx*y-Sx*Sxfλ-λfλ}.

Then fλ=(LK+λI)-1LKfH implies

LKfH=LKfλ+λfλ.

Therefore,

fz,λ-fλ=(Sx*Sx+λI)-1{Sx*y-Sx*Sxfλ-LK(fH-fλ)}.

Employing RKHS-norm we get,

||fz,λfλ||H||(SxSx+λI)1{SxySxSxfH+(SxSxLK)(fHfλ)}||HI1I2+I3||fλfH||H/λ,    (23)

where I1=||(Sx*Sx+λI)-1(LK+λI)1/2||L(H), I2=||(LK+λI)-1/2(Sx*y-Sx*SxfH)||H and I3=||Sx*Sx-LK||L(H).

The estimates of I2, I3 can be obtained from Proposition 3.2 and the only task is to bound I1. For this we consider

(Sx*Sx+λI)-1(LK+λI)1/2={I-(LK+λI)-1(LK-Sx*Sx)}-1                                                      (LK+λI)-1/2

which implies

I1n=0||(LK+λI)-1(LK-Sx*Sx)||L(H)n||(LK+λI)-1/2||L(H)    (24)

provided that ||(LK+λI)-1(LK-Sx*Sx)||L(H)<1. To verify this condition, we consider

||(LK+λI)-1(Sx*Sx-LK)||L(H)I3/λ.

Now using Proposition 3.2 we get with confidence 1 − η/2,

||(LK+λI)-1(Sx*Sx-LK)||L(H)4κ2mλlog(4η).

From the condition (21) we get with confidence 1 − η/2,

||(LK+λI)-1(Sx*Sx-LK)||L(H)12.    (25)

Consequently, using (25) in the inequality (24) we obtain with probability 1 − η/2,

I1=||(Sx*Sx+λI)-1(LK+λI)1/2||L(H)         2||(LK+λI)-1/2||L(H)2λ.    (26)

From Proposition 3.2 we have with confidence 1 − η/2,

||Sx*Sx-LK||L(H)2(κ2m+κ2m)log(4η).

Again from the condition (21) we get with probability 1 − η/2,

I3=||Sx*Sx-LK||L(H)λ2.    (27)

Therefore, the inequality (23) together with (16), (20), (26), (27) provides the desired bound.

The following theorem discuss the error estimates in L2-norm. The proof is similar to the above theorem.

Theorem 3.2. Let z be i.i.d. samples drawn according to the probability measure ρPϕ and fz, λ is the regularized solution (7) corresponding to Tikhonov regularization. Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds:

(i) Under the assumption that ϕ(t), t/ϕ(t) are non-decreasing functions,

||fz,λ-fH||ρ2{Rϕ(λ)λ+2κMmλ+4Σ2N(λ)m}                                  log(4η)

(ii) Under the assumption that ϕ(t), t/ϕ(t) are non-decreasing functions,

||fz,λ-fH||ρ{R(κ+λ)ϕ(λ)+4κMmλ+16Σ2N(λ)m}                              log(4η)

provided that

mλ8κ2log(4/η).    (28)

We derive the convergence rates of Tikhonov regularizer based on data-driven strategy of the parameter choice of λ for the class of probability measure Pϕ,b.

Theorem 3.3. Under the same assumptions of Theorem 3.2 and hypothesis (13), the convergence of the estimator fz, λ (7) to the target function fH can be described as:

(i) If ϕ(t) and t/ϕ(t) are non-decreasing functions. Then under the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where Ψ(t)=t12+12bϕ(t), we have

Probz{||fz,λ-fH||ρC(Ψ-1(m-1/2))1/2ϕ  (Ψ-1(m-1/2))log(4η)}1-η

and

limτlim supmsupρPϕ,bProbz{||fz,λfH||ρ>τ(Ψ1(m1/2))1/2ϕ(Ψ1(m1/2))}=0,

(ii) If ϕ(t) and t/ϕ(t) are non-decreasing functions. Then under the parameter choice λ ∈ (0, 1], λ = Θ−1(m−1/2) where Θ(t)=t12bϕ(t), we have

Probz{||fz,λ-fH||ρCϕ(Θ-1(m-1/2))log(4η)}1-η

and

limτlim supmsupρPϕ,bProbz{||fz,λfH||ρ                             >τϕ(Θ1(m1/2))}=0.

Proof. (i) Let Ψ(t)=t12+12bϕ(t). Then it follows,

limt0Ψ(t)t=limt0t2Ψ-1(t)=0.

Under the parameter choice λ = Ψ−1(m−1/2) we have,

limmmλ=.

Therefore, for sufficiently large m,

1mλ=λ12bϕ(λ)mλλ12bϕ(λ).

Under the fact λ ≤ 1 from Theorem 3.2 and Equation (14) follows that with confidence 1 − η,

||fz,λ-fH||ρC(Ψ-1(m-1/2))1/2ϕ(Ψ-1(m-1/2))log(4η),    (29)

where C=2R+4κM+4βbΣ2/(b-1).

Now defining τ:=Clog(4η) gives

η=ητ=4e-τ/C.

The estimate (29) can be reexpressed as

Probz{||fz,λ-fH||ρ>τ(Ψ-1(m-1/2))1/2ϕ(Ψ-1(m-1/2))}ητ.    (30)

(ii) Suppose Θ(t)=t12bϕ(t). Then the condition (28) follows that

mλ8κ2log(4/η)λ8κ2λ.

Hence under the parameter choice λ ∈ (0, 1], λ = Θ−1(m−1/2) we have

1mλλ8κ2mλ12+12bϕ(λ)8κ2ϕ(λ)8κ2.

From Theorem 3.2 and Equation (14), it follows that with confidence 1 − η,

||fz,λ-fH||ρCϕ(Θ-1(m-1/2))log(4η),    (31)

where C:=R(κ+1)+M/2κ+4βbΣ2/(b-1).

Now defining τ:=Clog(4η) gives

η=ητ=4e-τ/C.

The estimate (31) can be reexpressed as

Probz{||fz,λ-fH||ρ>τϕ(Θ-1(m-1/2))}ητ.    (32)

Then from Equations (30) and (32) our conclusions follow.

Theorem 3.4. Under the same assumptions of Theorem 3.1 and hypothesis (13) with the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where Ψ(t)=t12+12bϕ(t), the convergence of the estimator fz, λ (7) to the target function fH can be described as

Probz{||fz,λ-fH||HCϕ(Ψ-1(m-1/2))log(4η)}1-η

and

limτlim supmsupρPϕ,bProbz{||fz,λ-fH||H>τϕ(Ψ-1(m-1/2))}                                       =0.

The proof of the theorem follows the same steps as of Theorem 3.3 (i). We obtain the following corollary as a consequence of Theorem 3.3, 3.4.

Corollary 3.1. Under the same assumptions of Theorem 3.3, 3.4 for Tikhonov regularization with Hölder's source condition fHΩϕ,R, ϕ(t)=tr, for all 0 < η < 1, with confidence 1 − η, for the parameter choice λ=m-b2br+b+1, we have

||fz,λ-fH||HCm-br2br+b+1log(4η)for 0r1,
||fz,λ-fH||ρCm-2br+b4br+2b+2log(4η)for 0r12

and for the parameter choice λ=m-b2br+1, we have

||fz,λ-fH||ρCm-br2br+1log(4η)for 0r1.

3.2. Upper Rates for General Regularization Schemes

Bauer et al. [4] discussed the error estimates for general regularization schemes under general source condition. Here we study the convergence issues for general regularization schemes under general source condition and the polynomial decay of the eigenvalues of the integral operator LK. We define the regularization in learning theory framework similar to considered for ill-posed inverse problems (See Section 3.1 [4]).

Definition 3.1. A family of functions gλ:[0,κ2], 0 < λ ≤ κ2, is said to be the regularization if it satisfies the following conditions:

D:supσ(0,κ2]|σgλ(σ)|D.

B:supσ(0,κ2]|gλ(σ)|Bλ.

γ:supσ(0,κ2]|1-gλ(σ)σ|γ.

The maximal p satisfying the condition:

supσ(0,κ2]|1-gλ(σ)σ|σpγpλp

is called the qualification of the regularization gλ, where γp does not depend on λ.

The properties of general regularization are satisfied by the large class of learning algorithms which are essentially all the linear regularization schemes. We refer to Section 2.2 [10] for brief discussion of the regularization schemes. Here we consider general regularized solution corresponding to the above regularization:

fz,λ=gλ(Sx*Sx)Sx*y.    (33)

Here we are discussing the connection between the qualification of the regularization and general source condition [17].

Definition 3.2. The qualification p covers the index function ϕ if the function ttpϕ(t) on t ∈ (0, κ2] is non-decreasing.

The following result is a restatement of Proposition 3 [17].

Proposition 3.4. Suppose ϕ is a non-decreasing index function and the qualification of the regularization gλ covers ϕ. Then

supσ(0,κ2]|1-gλ(σ)σ|ϕ(σ)cgϕ(λ),  cg=max(γ,γp).

Generally, the index function ϕ is not stable with respect to perturbation in the integral operator LK. In practice, we are only accessible to the perturbed empirical operator Sx*Sx but the source condition can be expressed in terms of LK only. So we want to control the difference ϕ(LK)-ϕ(Sx*Sx). In order to obtain the error estimates for general regularization, we further restrict the index functions to operator monotone functions which is defined as

Definition 3.3. A function ϕ1:[0, d] → [0, ∞) is said to be operator monotone index function if ϕ1(0) = 0 and for every non-negative pair of self-adjoint operators A, B such that ||A||, ||B|| ≤ d and A ≤ B we have ϕ1(A) ≤ ϕ1(B).

We consider the class of operator monotone index functions:

Fμ={ϕ1:[0,κ2][0,) operator monotone,      ϕ1(0)=0,ϕ1(κ2)μ}.

For the above class of operator monotone functions from Theorem 1 [4], given ϕ1Fμ there exists cϕ1 such that

||ϕ1(Sx*Sx)-ϕ1(LK)||L(H)cϕ1ϕ1(||Sx*Sx-LK||L(H)).

Here we observe that the rate of convergence of ϕ1(Sx*Sx) to ϕ1(LK) is slower than the convergence rate of Sx*Sx to LK. Therefore, we consider the following class of index functions:

F={ϕ=ϕ2ϕ1:ϕ1Fμ,ϕ2:[0,κ2][0,)non-decreasing Lipschitz,ϕ2(0)=0}.

The splitting of ϕ = ϕ2ϕ1 is not unique. So we can take ϕ2 as a Lipschitz function with Lipschitz constant 1. Now using Corollary 1.2.2 [27] we get

||ϕ2(Sx*Sx)-ϕ2(LK)||L2(H)||Sx*Sx-LK||L2(H).

General source condition fH ∈ Ωϕ,R corresponding to index class functions F covers wide range of source conditions as Hölder's source condition ϕ(t) = tr, logarithm source condition ϕ(t)=tplog-ν(1t). Following the analysis of Bauer et al. [4] we develop the error estimates of general regularization for the index class function F under the suitable priors on the probability measure ρ.

Theorem 3.5. Let z be i.i.d. samples drawn according to the probability measure ρPϕ. Suppose fz is the regularized solution (33) corresponding to general regularization and the qualification of the regularization covers ϕ. Then for all 0 < η < 1, with confidence 1 − η, the following upper bound holds:

||fz,λ-fH||H{Rcg(1+cϕ1)ϕ(λ)+4Rμγκ2m+22ν1κMmλ+8ν12Σ2N(λ)mλ}                               log(4η)

provided that

mλ8κ2log(4/η).    (34)

Proof. We consider the error expression for general regularized solution (33),

fz,λ-fH=gλ(Sx*Sx)(Sx*y-Sx*SxfH)-rλ(Sx*Sx)fH,    (35)

where rλ(σ) = 1 − gλ(σ)σ.

Now the first term can be expressed as

gλ(Sx*Sx)(Sx*y-Sx*SxfH)=gλ(Sx*Sx)(Sx*Sx+λI)1/2                                                      (Sx*Sx+λI)-1/2(LK+λI)1/2                                                      (LK+λI)-1/2(Sx*y-Sx*SxfH).

On applying RKHS-norm we get,

||gλ(Sx*Sx)(Sx*y-Sx*SxfH)||HI2I5||gλ(Sx*Sx)                                                                 (Sx*Sx+λI)1/2||L(H),    (36)

where I2=||(LK+λI)-1/2(Sx*y-Sx*SxfH)||H and I5=||(Sx*Sx+λI)-1/2(LK+λI)1/2||L(H).

The estimate of I2 can be obtained from the first estimate of Proposition 3.2 and from the second estimate of Proposition 3.2 with the condition (34) we obtain with probability 1 − η/2,

||(LK+λI)-1/2(LK-Sx*Sx)(LK+λI)-1/2||L(H)1λ||Sx*Sx-LK||L(H)4κ2mλlog(4η)12.

which implies that with confidence 1 − η/2,

I5=||(SxSx+λI)1/2(LK+λI)1/2||L(H)     =||(LK+λI)1/2(SxSx+λI)1(LK+λI)1/2||L(H)1/2     =||{I(LK+λI)1/2(LKSxSx)        (LK+λI)1/2}1||L(H)1/2        2.    (37)

From the properties of the regularization we have,

||gλ(Sx*Sx)(Sx*Sx)1/2||L(H)sup0<σκ2|gλ(σ)σ|=(sup0<σκ2|gλ(σ)σ|sup0<σκ2|gλ(σ)|)1/2BDλ.    (38)

Hence it follows,

||gλ(Sx*Sx)(Sx*Sx+λI)1/2||L(H)sup0<σκ2|gλ(σ)(σ+λ)1/2|sup0<σκ2|gλ(σ)σ|+λsup0<σκ2|gλ(σ)|ν1λ,    (39)

where ν1:=B+BD.

Therefore, using (16), (37) and (39) in Equation (36) we conclude that with probability 1 − η,

||gλ(Sx*Sx)(Sx*y-Sx*SxfH)||H22ν1{κMmλ+Σ2N(λ)mλ}log(4η).    (40)

Now we consider the second term,

rλ(Sx*Sx)fH=rλ(Sx*Sx)ϕ(LK)v=rλ(Sx*Sx)ϕ(Sx*Sx)v                          +rλ(Sx*Sx)ϕ2(Sx*Sx)(ϕ1(LK)-ϕ1(Sx*Sx))v                          +rλ(Sx*Sx)(ϕ2(LK)-ϕ2(Sx*Sx))ϕ1(LK)v.

Employing RKHS-norm we get

||rλ(Sx*Sx)fH||HRcgϕ(λ)+Rcgcϕ1ϕ2(λ)ϕ1(||LK-Sx*Sx||L(H))+Rμγ||LK-Sx*Sx||L2(H).

Here we used the fact that if the qualification of the regularization covers ϕ = ϕ1ϕ2, then the qualification also covers ϕ1 and ϕ2 both separately.

From Equations (17) and (34) we have with probability 1 − η/2,

||Sx*Sx-LK||L(H)4κ2mlog(4η)λ/2.    (41)

Therefore, with probability 1 − η/2,

||rλ(Sx*Sx)fH||HRcg(1+cϕ1)ϕ(λ)+4Rμγκ2mlog(4η).    (42)

Combining the bounds (40) and (42) we get the desired result.

                                      □

Theorem 3.6. Let z be i.i.d. samples drawn according to the probability measure ρPϕ and fz is the regularized solution (33) corresponding to general regularization. Then for all 0 < η < 1, with confidence 1 − η, the following upper bounds holds:

(i) If the qualification of the regularization covers ϕ,

||fz,λ-fH||ρ{Rcg(1+cϕ1)(κ+λ)ϕ(λ)+4Rμγκ2(κ+λ)m+22ν2κMmλ+8ν22Σ2N(λ)m}log(4η),

(ii) If the qualification of the regularization covers ϕ(t)t,

||fz,λ-fH||ρ{2Rcg(1+cϕ1)ϕ(λ)λ+4Rμ(γ+cg)κ2λm+22ν2κMmλ+8ν22Σ2N(λ)m}                           log(4η)

provided that

mλ8κ2log(4/η).    (43)

Proof. Here we establish L2-norm estimate for the error expression:

fz,λ-fH=gλ(Sx*Sx)(Sx*y-Sx*SxfH)-rλ(Sx*Sx)fH.

On applying L2-norm in the first term we get,

||gλ(Sx*Sx)(Sx*y-Sx*SxfH)||ρI2I5||LK1/2gλ(Sx*Sx)                                                      (Sx*Sx+λI)1/2||L(H),    (44)

where I2=||(LK+λI)-1/2(Sx*y-Sx*SxfH)||H and I5=||(Sx*Sx+λI)-1/2(LK+λI)1/2||L(H).

The estimates of I2 and I5 can be obtained from Proposition 3.2 and Theorem 3.5 respectively. Now we consider

||LK1/2gλ(SxSx)(SxSx+λI)1/2||L(H)||LK1/2                                             (SxSx)1/2||L(H)||gλ(SxSx)(SxSx+λI)1/2                                              ||L(H)+||(SxSx)1/2gλ(SxSx)                                              (SxSx+λI)1/2||L(H).

Since ϕ(t)=t is operator monotone function. Therefore, from Equation (41) with probability 1 − η/2, we get

||LK1/2-(Sx*Sx)1/2||L(H)(||LK-Sx*Sx||L(H))1/2λ.

Then using the properties of the regularization and Equation (38) we conclude that with probability 1 − η/2,

||LK1/2gλ(Sx*Sx)(Sx*Sx+λI)1/2||L(H)λsup0<σκ2|gλ(σ)(σ+λ)1/2|+sup0<σκ2|gλ(σ)(σ2+λσ)1/2|sup0<σκ2|gλ(σ)σ|+λsup0<σκ2|gλ(σ)|+2λsup0<σκ2|gλ(σ)σ|B+D+2BD=ν2(let).    (45)

From Equations (44) with Equations (16), (37), and (45) we obtain with probability 1 − η,

||gλ(Sx*Sx)(Sx*y-Sx*SxfH)||ρ22ν2{κMmλ+Σ2N(λ)m}                                                                log(4η).    (46)

The second term can be expressed as

||rλ(Sx*Sx)fH||ρ||LK1/2-(Sx*Sx)1/2||L(H)||rλ(Sx*Sx)fH||H    +||(Sx*Sx)1/2rλ(Sx*Sx)fH||H||LK-Sx*Sx||L(H)1/2||rλ(Sx*Sx)fH||H    +||rλ(Sx*Sx)(Sx*Sx)1/2ϕ(Sx*Sx)v||H    +||rλ(Sx*Sx)(Sx*Sx)1/2ϕ2(Sx*Sx)(ϕ1(Sx*Sx)-ϕ1(LK))v||H    +||rλ(Sx*Sx)(Sx*Sx)1/2(ϕ2(Sx*Sx)-ϕ2(LK))ϕ1(LK)v||H.

Here two cases arises:

Case 1. If the qualification of the regularization covers ϕ. Then we get with confidence 1 − η/2,

||rλ(SxSx)fH||ρ(κ+λ)(Rcg(1+cϕ1)ϕ(λ)+Rμγ||SxSxLK||L2(H)).

Therefore, using Equation (17) we obtain with probability 1 − η/2,

||rλ(Sx*Sx)fH||ρ(κ+λ)(Rcg(1+cϕ1)ϕ(λ)+4Rμγκ2mlog(4η)).    (47)

Case 2. If the qualification of the regularization covers ϕ(t)t, we get with probability 1 − η/2,

||rλ(Sx*Sx)fH||ρ2Rcg(1+cϕ1)ϕ(λ)λ                                  +4Rμ(γ+cg)κ2λmlog(4η).    (48)

Combining the error estimates (46), (47) and (48) we get the desired results.

                                           □

We discuss the convergence rates of general regularizer based on data-driven strategy of the parameter choice of λ for the class of probability measure Pϕ,b. The proof of Theorem 3.7, 3.8 are similar to Theorem 3.3.

Theorem 3.7. Under the same assumptions of Theorem 3.5 and hypothesis (13) with the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where Ψ(t)=t12+12bϕ(t), the convergence of the estimator fz (33) to the target function fH can be described as

Probz{||fz,λ-fH||HC~ϕ(Ψ-1(m-1/2))log(4η)}1-η,

where C~=Rcg(1+cϕ1)+4Rμγκ2+22ν1κM+8βbν12Σ2/(b-1) and

limτlim supmsupρPϕ,bProbz{||fz,λ-fH||H>τϕ(Ψ-1(m-1/2))}                                                                                    =0.

Theorem 3.8. Under the same assumptions of Theorem 3.6 and hypothesis (13), the convergence of the estimator fz (33) to the target function fH can be described as

(i) If the qualification of the regularization covers ϕ. Then under the parameter choice λ ∈ (0, 1], λ = Θ−1(m−1/2) where Θ(t)=t12bϕ(t), we have

Probz{||fz,λ-fH||ρC~1ϕ(Θ-1(m-1/2))log(4η)}1-η

where C~1=Rcg(1+cϕ1)(κ+1)+4Rμγκ2(κ+1)+ν2M/22κ+8βbν22Σ2/(b-1) and

limτlim supmsupρPϕ,bProbz{||fz,λ-fH||ρ>τϕ(Θ-1(m-1/2))}                                                                        =0,

(ii) If the qualification of the regularization covers ϕ(t)t. Then under the parameter choice λ ∈ (0, 1], λ = Ψ−1(m−1/2) where Ψ(t)=t12+12bϕ(t), we have

Probz{||fz,λ-fH||ρC2~(Ψ-1(m-1/2))1/2ϕ  (Ψ-1(m-1/2))log(4η)}1-η

where C~2=2Rcg(1+cϕ1)+4Rμ(γ+cg)κ2+22ν2κM+8βbν22Σ2/(b-1) and

Probz{||fz,λfH||ρC˜2(Ψ1(m1/2))1/2ϕ(Ψ1(m1/2))log(4η)}1η,

We obtain the following corollary as a consequence of Theorem 3.7, 3.8.

Corollary 3.2. Under the same assumptions of Theorem 3.7, 3.8 for general regularization of the qualification p with Hölder's source condition fHΩϕ,R, ϕ(t)=tr, for all 0 < η < 1, with confidence 1 − η, for the parameter choice λ=m-b2br+b+1, we have

||fz,λ-fH||HC~m-br2br+b+1log(4η)for 0rp,
||fz,λ-fH||ρC~2m-2br+b4br+2b+2log(4η)for 0rp-12

and for the parameter choice λ=m-b2br+1, we have

||fz,λ-fH||ρC~1m-br2br+1log(4η)for 0rp.

Remark 3.1. It is important to observe from Corollary 3.1, 3.2 that using the concept of operator monotonicity of index function we are able to achieve the same error estimates for general regularization as of Tikhonov regularization up to a constant multiple.

Remark 3.2. (Related work) Corollary 3.1 provides the order of convergence same as of Theorem 1 [12] for Tikhonov regularization under the Hölder's source condition fH ∈ Ωϕ, R for ϕ(t)=tr (12r1) and the polynomial decay of the eigenvalues (13). Blanchard and Mücke [18] addressed the convergence rates for inverse statistical learning problem for general regularization under the Hölder's source condition with the assumption fρH. In particular, the upper convergence rates discussed in Blanchard and Mücke [18] agree with Corollary 3.2 for considered learning problem which is referred as direct learning problem in Blanchard and Mücke[18]. Under the fact N(λ)κ2λ from Theorem 3.5, 3.6 we obtain the similar estimates as of Theorem 10 [4] for general regularization schemes without the polynomial decay condition of the eigenvalues (13).

Remark 3.3. For the real valued functions and multi-task algorithms (the output space Y ⊂ ℝm for some m ∈ ℕ) we can obtain the error estimates from our analysis without imposing any condition on the conditional probability measure (11) for the bounded output space Y.

Remark 3.4. We can address the convergence issues of binary classification problem [28] using our error estimates as similar to discussed in Section 3.3 [4] and Section 5 [16].

3.3. Lower Rates for General Learning Algorithms

In this section, we discuss the estimates of minimum possible error over a subclass of the probability measures Pϕ,b parameterized by suitable functions fH. Throughout this section we assume that Y is finite-dimensional.

Let {vj}j=1d be a basis of Y and f ∈ Ωϕ, R. Then we parameterize the probability measure based on the function f,

ρf(x,y):=12dLj=1d(aj(x)δy+dLvj+bj(x)δy-dLvj)ν(x),    (49)

where aj(x) = L − 〈f,KxvjH, bj(x) = L + 〈f,KxvjH, L = 4κϕ(κ2)R and δξ denotes the Dirac measure with unit mass at ξ. It is easy to observe that the marginal distribution of ρf over X is ν and the regression function for the probability measure ρf is f (see Proposition 4 [12]). In addition to this, for the conditional probability measure ρf(y|x) we have,

Y(e||y-f(x)||Y/M-||y-f(x)||YM-1)dρf(y|x)(d2L2-||f(x)||Y2)i=2(dL+||f(x)||Y)i-2Mii!Σ22M2

provided that

dL+L/4M and 2dLΣ.

We assume that the eigenvalues of the integral operator LK follow the polynomial decay (13) for the marginal probability measure ν. Then we conclude that the probability measure ρf parameterized by f belongs to the class Pϕ,b.

The concept of information theory such as the Kullback-Leibler information and Fano inequalities (Lemma 3.3 [29]) are the main ingredients in the analysis of lower bounds. In the literature [12, 29], the closeness of probability measures is described through Kullback-Leibler information: Given two probability measures ρ1 and ρ2, it is defined as

K(ρ1,ρ2):=Zlog(g(z))dρ1(z),

where g is the density of ρ1 with respect to ρ2, that is, ρ1(E)=Eg(z)dρ2(z) for all measurable sets E.

Following the analysis of Caponnetto and De Vito [12] and DeVore et al. [29] we establish the lower rates of accuracy that can be attained by any learning algorithm.

To estimate the lower rates of learning algorithms, we generate Nε-functions belonging to Ωϕ,R for given ε > 0 such that (53), (54) holds. Then we construct the probability measures ρiPϕ,b from Equation (49), parameterized by these functions fi's (1 ≤ iNε). On applying Lemma 3.3 [29], we obtain the lower convergence rates using Kullback-Leibler information.

Theorem 3.9. Let z be i.i.d. samples drawn according to the probability measure ρPϕ,b under the hypothesis dim(Y) = d < ∞. Then for any learning algorithm (zfzH) there exists a probability measure ρ*Pϕ,b and fρ*H such that for all 0 < ε < εo, fz can be approximated as

Probz{||fz-fρ*||H>ε/2}min{11+e-ε/24,ϑe(ε48-cmε2εb)}

where ϑ = e−3/e, c=64β15(b-1)dL2(1-12b-1) and ε=12(αϕ1(ε/R))1/b.

Proof. For given ε > 0, we define

g=n=+12εσn-enϕ(tn),

where σ = (σ1, …, σ) ∈ {−1, +1}, tn's are the eigenvalues of the integral operator LK, en's are the eigenvectors of the integral operator LK and the orthonormal basis of RKHS H. Under the decay condition on the eigenvalues αnbtn, we get

||g||H2=n=+12ε2ϕ2(tn)n=+12ε2ϕ2(αnb)ε2ϕ2(α2bb).

Hence f = ϕ(LK)g ∈ Ωϕ,R provided that ||g||HR or equivalently,

12(αϕ-1(ε/R))1/b.    (50)

For ℓ = ε=12(αϕ1(ε/R))1/b, choose εo such that ℓεo > 16. Then according to Proposition 6 [12], for every positive ε < εo (ℓε > ℓεo) there exists Nε ∈ ℕ and σ1,,σNε{-1,+1}ε such that

n=1ε(σin-σjn)2ε,for all 1i,jNε, ij    (51)

and

Nεeε/24.    (52)

Now we suppose fi = ϕ(LK)gi and for ε > 0,

gi=n=ε+12εεσin-εenεϕ(tn),for i=1,,Nε,

where σi=(σi1,,σiε){-1,+1}ε. Then from Equation (51) we get,

ε||fi-fj||H,for all 1i,jNε, ij.    (53)

For 1 ≤ i, jNε, we have

||fi-fj||Lν2(X)2n=ε+12εβε2(σin-ε-σjn-ε)2εnbn=ε+12ε4βε2εnb                                  4βε2εε2ε1xbdx=cε2εb,    (54)

where c=4β(b-1)(1-12b-1).

We define the sets,

Ai={z:||fz-fi||H<ε2},for 1iNε.

It is clear from Equation (53) that Ai's are disjoint sets. On applying Lemma 3.3 [29] with probability measures ρfim, 1iNε, we obtain that either

p:=max1iNερfim(Aic)NεNε+1    (55)

or

min1jNε1Nεi=1,ijNεK(ρfim,ρfjm)ΨNε(p),    (56)

where ΨNε(p)=log(Nε)+(1-p)log(1-pp)-plog(Nε-pp). Further,

ΨNε(p)(1-p)log(Nε)+(1-p)log(1-p)-log(p)                    +2plog(p)-log(p)+log(Nε)-3/e.    (57)

Since minimum value of x log(x) is −1/e on [0, 1].

For the joint probability measures ρfim, ρfjm (ρfi,ρfjPϕ,b, 1i,jNε) from Proposition 4 [12] and the Equation (54) we get,

K(ρfim,ρfjm)=mK(ρfi,ρfj)16m15dL2||fi-fj||Lν2(X)2cmε2εb,    (58)

where c = 16c′/15dL2.

Therefore, Equations (55), (56), together with Equations (57) and (58) implies

p:=max1iNε(Prob{z:||fz-fi||H>ε2})         min{NεNε+1,Nεe-3e-cmε2εb}.

From Equation (52) for the probability measure ρ* such that p=ρ*m(Aic) follows the result.                                                                           □

The lower estimates in L2-norm can be obtained similar to above theorem.

Theorem 3.10. Let z be i.i.d. samples drawn according to the probability measure ρPϕ,b under the hypothesis dim(Y) = d < ∞. Then for any learning algorithm (zfzH) there exists a probability measure ρ*Pϕ,b and fρ*H such that for all 0 < ε < εo, fz can be approximated as

Probz{||fz-fρ*||Lν2(X)>ε/2}min{11+e-ε/24,ϑe(ε48-64mε215dL2)}

where ϑ = e−3/e, ε=(αΨ1(ε/R))1/b and ψ(t)=tϕ(t).

Theorem 3.11. Under the same assumptions of Theorem 3.10 for ψ(t) = t1/2ϕ(t) and Ψ(t)=t12+12bϕ(t), the estimator fz corresponding to any learning algorithm converges to the regression function fρ with the following lower rate:

limτ0lim infminflAsupρPϕ,bProbz{||fzlfρ||Lν2(X)>τψ                                                                                     (Ψ1(m1/2))}=1,

where A denotes the set of all learning algorithms l:zfzl.

Proof. Under the condition ε=(αΨ1(ε/R))1/b from Theorem 3.10 we get,

                   Probz{||fz-fρ*||Lν2(X)>ε2}min{11+e-ε/24,ϑe-148e{148(αψ-1(ε/R))1/b-64mε215dL2}}.

Choosing εm=τRψ(Ψ-1(m-1/2)), we obtain

Probz{||fz-fρ*||Lν2(X)>τR2ψ(Ψ-1(m-1/2))}   min{11+e-ε/24,ϑe-148ec(Ψ-1(m-1/2))-1/b},

where c=(α1/b48-64R2τ215dL2)>0 for 0<τ<min(5dLα12b32R,1).

Now as m goes to ∞, ε → 0 and ℓε → ∞. Therefore, for c > 0 we conclude that

lim infminflAsupρPϕ,bProbz{||fzl-fρ||Lν2(X)>τR2ψ(Ψ-1(m-1/2))}=1.

                                                                           □

Choosing εm=τRϕ(Ψ-1(m-1/2)) we get the following convergence rate from Theorem 3.9.

Theorem 3.12. Under the same assumptions of Theorem 3.10 for Ψ(t)=t12+12bϕ(t), the estimator fz corresponding to any learning algorithm converges to the regression function fρ with the following lower rate:

limτ0lim infminflAsupρPϕ,bProbz{||fzl-fρ||H>τϕ(Ψ-1(m-1/2))}=1.

We obtain the following corollary as a consequence of Theorem 3.11, 3.12.

Corollary 3.3. For any learning algorithm under Hölder's source condition fρΩϕ,R, ϕ(t)=tr and the polynomial decay condition (13) for b > 1, the lower convergence rates can be described as

limτ0lim infminflAsupρPϕ,bProbz{||fzl-fρ||Lν2(X)>τm-2br+b4br+2b+2}=1

and

limτ0lim infminflAsupρPϕ,bProbz{||fzl-fρ||H>τm-br2br+b+1}=1.

If the minimax lower rate coincides with the upper convergence rate for λ = λm. Then the choice of parameter is said to be optimal. For the parameter choice λ = Ψ−1(m−1/2), Theorem 3.3 and Theorem 3.8 share the upper convergence rate with the lower convergence rate of Theorem 3.11 in L2-norm. For the same parameter choice, Theorem 3.4 and Theorem 3.7 share the upper convergence rate with the lower convergence rate of Theorem 3.12 in RKHS-norm. Therefore, the choice of the parameter is optimal.

It is important to observe that we get the same convergence rates for b = 1.

3.4. Individual Lower Rates

In this section, we discuss the individual minimax lower rates that describe the behavior of the error for the class of probability measure Pϕ,b as the sample size m grows.

Definition 3.4. A sequence of positive numbers an (n ∈ ℕ) is called the individual lower rate of convergence for the class of probability measure P, if

inflAsupρPlim supm(Ez(||fzl-fH||2)am)>0,

where A denotes the set of all learning algorithms l:zfzl.

Theorem 3.13. Let z be i.i.d. samples drawn according to the probability measure Pϕ,b where ϕ is the index function satisfying the conditions that ϕ(t)/tr1, tr2/ϕ(t) are non-decreasing functions and dim(Y) = d < ∞. Then for every ε > 0, the following lower bound holds:

inflAsupρPϕ,blim supm(Ez(||fzl-fH||Lν2(X)2)m-(bc2+ε)/(bc1+ε+1))>0,

where c1 = 2r1 + 1 and c2 = 2r2 + 1.

We consider the class of probability measures such that the target function fH is parameterized by s=(sn)n=1{-1,+1}. Suppose for ε > 0,

g=n=1snRεε+1αnbtn(ϕ(α/nb)ϕ(tn))n-(ε+1)/2en,

where s=(sn)n=1{-1,+1}, tn's are the eigenvalues of the integral operator LK, en's are the eigenvectors of the integral operator LK and the orthonormal basis of RKHS H. Then the target function fH = ϕ(LK)g satisfies the general source condition. We assume that the conditional probability measure ρ(y|x) follows the normal distribution centered at fH and the marginal probability measure ρX = ν. Now we can derive the individual lower rates over the considered class of probability measures from the ideas of the literature [12, 30].

Theorem 3.14. Let z be i.i.d. samples drawn according to the probability measure Pϕ,b where ϕ is the index function satisfying the conditions that ϕ(t)/tr1, tr2/ϕ(t) are non-decreasing functions and dim(Y) = d < ∞. Then for every ε > 0, the following lower bound holds:

inflAsupρPϕ,blim supm(Ez(||fzl-fH||H2)m-(bc2-b+ε)/(bc1+ε+1))>0.

4. Conclusion

In our analysis we derive the upper and lower convergence rates over the wide class of probability measures considering general source condition in vector-valued setting. In particular, our minimax rates can be used for the scalar-valued functions and multi-task learning problems. The lower convergence rates coincide with the upper convergence rates for the optimal parameter choice based on smoothness parameters b, ϕ. We can also develop various parameter choice rules such as balancing principle [31], quasi-optimality principle [32], discrepancy principle [33] for the regularized solutions provided in our analysis.

Author Contributions

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

The authors are grateful to the reviewers for their helpful comments and pointing out a subtle error that led to improve the quality of the paper.

References

1. Cucker F, Smale, S. On the mathematical foundations of learning. Bull Am Math Soc. (2002) 39:1–49. doi: 10.1090/S0273-0979-01-00923-5

CrossRef Full Text | Google Scholar

2. Evgeniou T, Pontil, M Poggio, T. Regularization networks and support vector machines. Adv Comput Math. (2000) 13:1–50. doi: 10.1023/A:1018946025316

CrossRef Full Text | Google Scholar

3. Vapnik VN, Vapnik V. Statistical Learning Theory. New York, NY: Wiley (1998).

PubMed Abstract | Google Scholar

4. Bauer F, Pereverzev S, Rosasco L. On regularization algorithms in learning theory. J Complex. (2007) 23:52–72. doi: 10.1016/j.jco.2006.07.001

CrossRef Full Text | Google Scholar

5. Engl HW, Hanke M, Neubauer A Regularization of Inverse Problems. Dordrecht: Kluwer Academic Publishers Group (1996).

Google Scholar

6. Gerfo LL, Rosasco L, Odone F, De Vito E, Verri A. Spectral algorithms for supervised learning. Neural Comput. (2008) 20:1873–97. doi: 10.1162/neco.2008.05-07-517

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Tikhonov AN, Arsenin VY. Solutions of Ill-Posed Problems. Washington, DC: W. H. Winston (1977).

Google Scholar

8. Bousquet O, Boucheron S, Lugosi G. Introduction to statistical learning theory. In: Bousquet O, von Luxburg U, Ratsch G editors. Advanced Lectures on Machine Learning, Volume 3176 of Lecture Notes in Computer Science. Berlin; Heidelberg: Springer (2004). pp. 169–207.

Google Scholar

9. Cucker F, Zhou DX. Learning Theory: An Approximation Theory Viewpoint. Cambridge, UK: Cambridge Monographs on Applied and Computational Mathematics, Cambridge University Press (2007).

Google Scholar

10. Lu S, Pereverzev S. Regularization Theory for Ill-posed Problems: Selected Topics, Berlin: DeGruyter (2013).

Google Scholar

11. Abhishake Sivananthan S. Multi-penalty regularization in learning theory. J Complex. (2016) 36:141–65. doi: 10.1016/j.jco.2016.05.003

CrossRef Full Text | Google Scholar

12. Caponnetto A, De Vito E. Optimal rates for the regularized least-squares algorithm. Found Comput Math. (2007) 7:331–68. doi: 10.1007/s10208-006-0196-8

CrossRef Full Text | Google Scholar

13. Smale S, Zhou DX. Estimating the approximation error in learning theory. Anal Appl. (2003) 1:17–41. doi: 10.1142/S0219530503000089

CrossRef Full Text | Google Scholar

14. Smale S, Zhou DX. Shannon sampling and function reconstruction from point values. Bull Am Math Soc. (2004) 41:279–306. doi: 10.1090/S0273-0979-04-01025-0

CrossRef Full Text | Google Scholar

15. Smale S, Zhou DX. Shannon sampling II: connections to learning theory. Appl Comput Harmon Anal. (2005) 19:285–302. doi: 10.1016/j.acha.2005.03.001

CrossRef Full Text | Google Scholar

16. Smale S, Zhou DX. Learning theory estimates via integral operators and their approximations. Constr Approx. (2007) 26:153–72. doi: 10.1007/s00365-006-0659-y

CrossRef Full Text | Google Scholar

17. Mathé P, Pereverzev SV. Geometry of linear ill-posed problems in variable Hilbert scales. Inverse Probl. (2003) 19:789–803. doi: 10.1088/0266-5611/19/3/319

CrossRef Full Text | Google Scholar

18. Blanchard G, Mücke N. Optimal rates for regularization of statistical inverse learning problems. arXiv:1604.04054 (2016).

Google Scholar

19. Mendelson S. On the performance of kernel classes. J Mach Learn Res. (2003) 4:759–71.

Google Scholar

20. Zhang T. Effective dimension and generalization of kernel learning. In: Thrun S, Becker S, Obermayer K. editors. Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, (2003). pp. 454–61.

Google Scholar

21. Akhiezer NI, Glazman IM. Theory of Linear Operators in Hilbert Space, Translated from the Russian and with a preface by Merlynd Nestell. New York, NY: Dover Publications Inc (1993).

Google Scholar

22. Micchelli CA, Pontil M. On learning vector-valued functions. Neural Comput. (2005) 17:177–204. doi: 10.1162/0899766052530802

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Aronszajn N. Theory of reproducing kernels. Trans Am Math Soc. (1950) 68:337–404. doi: 10.1090/S0002-9947-1950-0051437-7

CrossRef Full Text | Google Scholar

24. Reed M, Simon B. Functional Analysis, Vol. 1, San Diego, CA: Academic Press (1980).

25. De Vito E, Rosasco L, Caponnetto A, De Giovannini U, Odone F. Learning from examples as an inverse problem. J Mach Learn Res. (2005) 6:883–904.

Google Scholar

26. Pinelis IF, Sakhanenko AI. Remarks on inequalities for the probabilities of large deviations. Theory Prob Appl. (1985) 30:127–31. doi: 10.1137/1130013

CrossRef Full Text | Google Scholar

27. Peller VV. Multiple operator integrals in perturbation theory. Bull Math Sci. (2016) 6:15–88. doi: 10.1007/s13373-015-0073-y

CrossRef Full Text | Google Scholar

28. Boucheron S, Bousquet O, Lugosi G. Theory of classification: a survey of some recent advances. ESAIM: Prob Stat. (2005) 9:323–75. doi: 10.1051/ps:2005018

CrossRef Full Text | Google Scholar

29. DeVore R, Kerkyacharian G, Picard D, Temlyakov V. Approximation methods for supervised learning. Found Comput Math. (2006) 6:3–58. doi: 10.1007/s10208-004-0158-6

CrossRef Full Text | Google Scholar

30. Györfi L, Kohler M, Krzyzak A, Walk H. A Distribution-Free Theory of Nonparametric Regression. New York, NY: Springer Series in Statistics, Springer-Verlag (2002).

Google Scholar

31. De Vito E, Pereverzyev S, Rosasco L. Adaptive kernel methods using the balancing principle. Found Comput Math. (2010) 10:455–79. doi: 10.1007/s10208-010-9064-2

CrossRef Full Text | Google Scholar

32. Bauer F, Reiss M. Regularization independent of the noise level: an analysis of quasi-optimality. Inverse Prob. (2008) 24:055009. doi: 10.1088/0266-5611/24/5/055009

CrossRef Full Text | Google Scholar

33. Lu S, Pereverzev SV, Tautenhahn U. A model function method in regularized total least squares. Appl Anal. (2010) 89:1693–703. doi: 10.1080/00036811.2010.492502

CrossRef Full Text | Google Scholar

Keywords: learning theory, general source condition, vector-valued RKHS, error estimate, optimal rates

Mathematics Subject Classification 2010: 68T05, 68Q32

Citation: Rastogi A and Sampath S (2017) Optimal Rates for the Regularized Learning Algorithms under General Source Condition. Front. Appl. Math. Stat. 3:3. doi: 10.3389/fams.2017.00003

Received: 02 November 2016; Accepted: 09 March 2017;
Published: 27 March 2017.

Edited by:

Yiming Ying, University at Albany, SUNY, USA

Reviewed by:

Xin Guo, The Hong Kong Polytechnic University, Hong Kong
Ernesto De Vito, University of Genoa, Italy

Copyright © 2017 Rastogi and Sampath. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Abhishake Rastogi, abhishekrastogi2012@gmail.com

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.