format: text

text:
# Statistical Mechanics of Soft Margin Classifiers

## I Introduction

Neural networks are models of learning systems composed of interconnected units that, besides their biological relevance, have been shown to be very useful for classification tasks. The weights of the connections are adjusted through a process called learning using a set of $`M`$ examples. It is assumed that these are labeled following an underlying rule, usually called teacher. The purpose of learning is not only to classify correctly the examples of the training set, but also to generalize correctly on new inputs. To this aim, the network has to infer the teacher’s rule. The quality of this inference is measured through the generalization error $`ϵ_g`$, which is the probability of misclassification of a new, randomly selected, input pattern. As $`ϵ_g`$ is not a quantity available for the training process, learning is usually performed through the minimization of a function of the training patterns. The tools of Statistical Mechanics allow to study the properties of such learning systems, providing a deep understanding of their typical behaviour . In particular, it has been shown that the minimization of the training error, that is, the fraction of training patterns misclassified by the network, does not necessarily provide the best generalizer . This is why other cost functions, based on geometrical properties like the distance of the patterns to the discriminating surface, or on probabilistic error measures like the likelihood, are used for training.

The simplest instance of a neural network, the perceptron, is a single binary unit whose output is the sign of the weighted sum of its inputs. It can only perform linear separations of the patterns. If the classification task requires more complex discriminating surfaces, these may be implemented using feedforward networks with a layer of hidden units whose number is a priori unknown. The cost functions used to tackle this problem usually have several minima, and determining the lowest one is one of the main difficulties of learning with multilayer neural networks. This is also a problem for the theoretical analysis, as the typical properties of such networks depend crucially on the structure of the minima in the weights’ space.

Recently, a new learning scheme has been proposed, which strives to get rid of the problem raised by the multiple minima. The obtained classifiers are called Support Vector Machines (SVMs) . Instead of directly looking for a complicated discriminating surface in input space, the patterns are first mapped to a high dimensional feature space, where the rule to be learned is (hopefully) linearly separable. If this is the case, a simple perceptron can be trained to find the separation in feature space. Denoting the weights by $`𝐰\mathrm{}^N`$, the perceptron’s output to an input $`𝐱\mathrm{}^N`$ is given by $`\sigma =\mathrm{sign}(𝐰𝐱+b)`$ where $`b`$ is a bias and the dot represents the inner product in $`\mathrm{}^N`$. Thus, the patterns belonging to different classes are separated by a hyperplane orthogonal to $`𝐰`$ at distance $`|b|/𝐰`$ from the origin, with $`𝐰=\sqrt{𝐰𝐰}`$. The SVM’s solution is the Maximal Stability Perceptron (MSP)  in feature space, also called maximal margin hyperplane. This is the hyperplane at maximal distance $`\kappa _{max}`$ from the closest patterns in the training set. Two different formulations of this problem in terms of cost functions have been proposed in the literature. In the first one , the cost function counts not only the number of misclassified patterns, but also the number of correctly classified ones that lie at a distance smaller than $`\kappa `$ from the separating hyperplane:

$$E_{\mathrm{MSP}}(\kappa )=\underset{\mu =1}{\overset{M}{}}\mathrm{\Theta }(\kappa 𝐰h_\mu ),$$
(1)
where $`\mathrm{\Theta }`$ is the Heaviside function, and

$$h_\mu \tau _\mu (𝐰𝐱_\mu +b),$$
(2)
is called aligned field of the training pattern $`𝐱_\mu `$, $`\tau _\mu \{1,1\}`$ being its class. If the $`M`$ $`N`$-dimensional patterns are correctly classified, the aligned fields are all positive. The SVM solution has $`𝐰`$ and $`b`$ corresponding to $`\kappa _{max}`$, the largest possible value of $`\kappa `$ such that $`E_{\mathrm{MSP}}(\kappa _{max})=0`$. If the training set is not linearly separable, $`\kappa _{max}`$ becomes negative. Notice that there are no constraints on the norm of $`𝐰`$, that can be freely chosen.

If the norm of the weight vector is chosen so that the aligned field of the closest pattern be $`1`$, this leads to an equivalent formulation of the problem , in which the function to be minimized is:

$$E_{\mathrm{SVM}}=\frac{1}{2}𝐰𝐰,$$
(3)
subject to the conditions

$$h_\mu 1,\mu =1,\mathrm{},M.$$
(4)
Clearly, the constraints (4) can only be satisfied if it is possible to classify correctly all the examples. In that case, there are no training patterns in a strip of width $`1/𝐰`$ on both sides of the hyperplane, meaning that in the error-free regime $`1/𝐰\kappa _{max}`$. An interesting property of the SVM solution is that the weight vector and the bias can be written as a linear combination of a sub-set of training patterns, the Support Vectors, having $`h_\mu =1`$.

The minimization of (1) with $`\kappa =\kappa _{max}`$ is equivalent to that of (3) with condition (4) only if the training set is linearly separable. If errors cannot be avoided, the equivalence breaks down, as in one hand (1) has either negative $`\kappa _{max}`$, or several minima if $`\kappa _{max}0`$ is imposed, and on the other hand the constraints (4) cannot be satisfied. This is why the second formulation has been generalized  through the introduction of a new set of variables $`\zeta _\mu 0`$, called slacks, which are a measure of the “amount of violation” of the constraints. An increasing function of these is included in the cost function (3) and the hard margin conditions (4) are modified to allow some patterns to be closer to the hyperplane than $`1/𝐰`$. The new problem amounts to minimize:

$$E_{C,k}=\frac{1}{2}𝐰𝐰+C\underset{\mu =1}{\overset{M}{}}\zeta _{\mu }^{}{}_{}{}^{k},$$
(5)
subject to the following conditions for $`\mu =1,\mathrm{},M`$

$`h_\mu `$ $``$ $`1\zeta _\mu ,`$ (7)
$`\zeta _\mu `$ $``$ $`0.`$ (8)
The coefficient $`C`$ in (5) is a hyperparameter that allows to control the trade-off between the error term, defined by the slacks, and the regularization term, proportional to the squared weights. As will be shown in section IV, it may be selected to optimize the generalization performance. The exponent $`k`$ in (5) modulates the relative cost of errors, depending on their distance to the hyperplane. Patterns in a strip of width $`1/𝐰`$ at each side of the hyperplane, whether correctly or incorrectly classified, as well as those incorrectly classified outside of this strip, have $`\zeta _\mu >0`$. $`1/𝐰`$ is called soft margin, and the resulting classifier soft margin SVM or soft margin classifier (SMC).

As the cost (5) is a quadratic function for $`k=1`$ and $`k=2`$, and the domain of minimization defined by (7) and (8) is convex, the minimum is unique  . This remarkable property makes the new formulation attractive for applications, as it allows to get rid of the multiple minima appearing in other learning schemes. Like in the hard margin formulation, the solution $`\{𝐰,b\}`$ can be expressed as a linear combination of the support vectors, which now include the patterns with positive slacks. The corresponding coefficients may be obtained by solving the dual problem (see for example ) which, for $`k=1`$ or $`k=2`$ has a particularly simple expression . Several efficient methods are known for solving this kind of problems, and this is one of the reasons why these classifiers are so widely used lately.

In this paper we study the typical properties of the SMCs obtained by solving equation (5) subject to the conditions (7) and (8), with the methods of Statistical Mechanics, using the replica approach. It has been shown  that the statistical properties of SVMs in high dimensional feature spaces  can be well approximated by considering a simple perceptron learning anisotropically distributed patterns. The amount of anisotropy depends on the normalization of the mapping from the input to the feature space. In this paper we restrict to an isotropic pattern distribution, which corresponds to a non-normalized mapping.

The learning properties of a perceptron learning an isotropic input pattern distribution have been extensively studied , mainly for linearly separable, i.e. realizable, tasks. In this case the hypothesis of replica symmetry is generally correct, allowing for a full analytical statistical mechanics calculation. In particular, the behaviour of the generalization error $`ϵ_g`$ in the limit of very large $`\alpha M/N`$ has a universal power law decay $`ϵ_g\alpha ^\nu `$ with $`\nu =1`$. Its prefactor allows to characterize the convergence to perfect learning of different learning algorithms. If the rule to be inferred cannot be generalized without errors, the task is called unrealizable. In this case the replica symmetric solution, although generally unstable, is believed to provide a good approximation of some learning properties. However, in the case of a linearly separable rule learned with noisy training patterns, which is thus unrealizable, the replica symmetric approximation gives an exponent $`\nu =1/2`$  whereas one step of replica symmetry breaking shows  that this exponent is modified to $`\nu =2/3`$. As this is but an approximation to the full replica symmetry breaking scheme  at zero temperature, it is not clear whether this exponent is correct. The same exponent has been found in the case of a quadratic hard margin SVM learning a linearly separable task, that is, a rule simpler than those implementable with the student’s architecture . Another case of interest is that of inconsistent learning , which refers to realizable tasks learned with algorithms that do not strive to minimize the number of training errors. In this case, the exponent within the replica symmetric approximation was found to be $`\nu =1/2`$ .

As the soft margin problem has a unique minimum for $`k=1`$ and $`k=2`$, even if the task is unrealizable, the replica symmetry hypothesis should be always correct, providing a framework for the study of complex classification tasks even when the mismatch between the student and the teacher hinders error-free learning.

In this paper we present the statistical properties of SMCs learning several kinds of realizable and unrealizable rules. The model and the statistical mechanics approach are presented in section II. The theoretical properties of SMCs with exponents $`k=1`$ and $`k=2`$ in the cost function (5) are obtained as a function of the training set size $`\alpha M/N`$ in the thermodynamic limit $`N,M\mathrm{}`$. Several teacher rules are considered in section III. One of our most striking results is that the generalization error for large $`\alpha `$ exhibits a very rich variety of asymptotic behaviours, depending on the type of rule to be inferred. In particular, even if the task is realizable, the soft margin algorithm is inconsistent unless $`C\mathrm{}`$. For finite $`C`$, we find that the fraction of training errors at finite $`\alpha `$ is finite, and the generalization error vanishes asymptotically with $`\alpha `$ following a $`\nu =2/3`$ power law. In the unrealizable tasks considered, $`ϵ_g`$ converges to an asymptotic finite value either exponentially or with a power law with $`\nu =1/2`$ . The usual exponent $`\nu =1`$ only arises for error-free learning of a realizable task. In section IV we derive the best generalization performances of SMCs through the determination of the value $`C_{opt}(\alpha )`$ that minimizes the generalization error. Finally we present a summary of our results in section V, together with some open questions. Most details of the proofs are left to the Appendix.

## II Statistical mechanics approach

We consider a student perceptron of weight vector $`𝐰=(w_1,\mathrm{},w_N)`$, without threshold. That is, we set $`b=0`$ in (7). Given any $`N`$-dimensional input vector $`𝐱`$, the classifier’s output is $`\sigma =\mathrm{sign}(𝐰𝐱)`$: all the points lying on the same side of a hyperplane orthogonal to $`𝐰`$ containing the origin are given the same class. We assume that the perceptron learns the classification with the soft margin algorithm, using a set $`_M=\{(𝐱_\mu ,\tau _\mu )\}_{\mu =1,\mathrm{},M}`$ of $`M`$ examples or training patterns. These consist of input vectors $`𝐱_\mu `$ drawn from an isotropic gaussian distribution of variance $`1/\sqrt{N}`$,

$$P(𝐱)=\frac{e^{N𝐱^2/2}}{(2\pi /N)^{N/2}};$$
(9)
and labels $`\tau _\mu \{1,1\}`$ that represent the corresponding classes. The classification tasks considered in this paper are given by the following teacher’s rule:

$$\tau =\mathrm{sign}\left(𝒫(𝐰_0𝐱)\right),$$
(10)
where $`𝐰_0`$ is referred to as the teacher’s vector hereafter, and $`𝒫(z)`$ is a polynomial of $`z`$. Each of its zeros $`z_i`$  defines a discriminating hyperplane at a distance $`|z_i|/𝐰_0`$ from the origin. Rules of the kind (10) partition the input space in as many different regions as the number of zeros of the polynomial plus one, separated by parallel hyperplanes normal to the teacher’s vector $`𝐰_0`$. Patterns in successive regions belong alternatively to class $`+1`$ or $`1`$. As only the zeros of the function $`𝒫(z)`$ matter, there is no loss of generality in our assumption that $`𝒫(z)`$ is a polynomial. We assume $`𝐰_0=\sqrt{N}`$, which is equivalent to imposing the unit of distance. Notice that the only rule realizable for the student perceptron considered in this paper is that of the linear teacher $`𝒫(z)=z`$.

In the following we study the properties of the solution to the soft margin problem using the by now standard tools of Statistical Mechanics . That is, we assume that the ensemble of classifiers follows a Gibbs distribution defined by the energy function (5), at a fictitious temperature $`1/\beta `$, and we take the zero temperature limit. The constraints (7) and (8) play the role of infinite potential walls. Notice that the phase space in the present case has dimension $`\mathrm{}^{N+M}`$, as not only the weights $`𝐰`$ but also the slacks $`\{\zeta _\mu \}_{\mu =1,\mathrm{},M}`$, have to be learned. The partition function is:

$$Z_{C,k}(\beta ;_M,𝒫)=\mathrm{exp}\left(\beta E_{C,k}(𝐰,\{\zeta _\mu \})\right)\underset{\mu =1}{\overset{M}{}}\mathrm{\Theta }\left(\tau _\mu 𝐰𝐱_\mu (1\zeta _\mu )\right)\mathrm{\Theta }(\zeta _\mu )d𝐰d\zeta _\mu .$$
(11)
The inverse temperature $`\beta `$ has obviously no physical meaning whatsoever; it is only introduced in order to study the properties of the SMC which, being the single minimum of the energy function, is selected in the limit $`\beta \mathrm{}`$. We assume that the number of training examples scales with the input space dimension, $`M=\alpha N`$, and take the thermodynamic limit $`N\mathrm{}`$, $`M\mathrm{}`$ with $`\alpha M/N`$ constant. The free energy per input space dimension averaged over all the possible training sets of $`M`$ patterns, $`f_{C,k}(\beta ;𝒫)`$, is calculated with the replica method, that uses the identity

$$f_{C,k}(\beta ;𝒫)=\underset{N\mathrm{}}{lim}\frac{1}{N\beta }\overline{\mathrm{ln}Z_{C,k}(\beta ;_M,𝒫)}=\underset{N\mathrm{}}{lim}\frac{1}{N\beta }\underset{n0}{lim}\frac{\mathrm{ln}\overline{Z_{C,k}^n(\beta ;_M,𝒫)}}{n},$$
(12)
where the overline represents the average over the pattern distribution (9), with labels given by (10). $`Z^n`$ is the partition function of $`n`$ independent replicas of the problem, that become coupled after taking the average. The typical properties of the classifier are obtained by taking the limit $`\beta \mathrm{}`$. The free energy (12) turns out to be a function of the following order parameters:

$`Q_a`$ $`=`$ $`{\displaystyle \frac{\overline{𝐰_a𝐰_a}}{N}},`$ (14)
$`q_{ab}`$ $`=`$ $`{\displaystyle \frac{\overline{𝐰_a𝐰_b}}{N}},`$ (15)
$`\stackrel{~}{R_a}`$ $`=`$ $`{\displaystyle \frac{\overline{𝐰_a𝐰_0}}{N}},`$ (16)
where the brackets represent the phase space average and $`a`$ and $`b`$ are replica indices. The norm of the perceptron’s weight vector, $`Q_a`$, is one of the order parameters because in the soft margin problem the weights are not normalized as usually. $`q_{ab}`$ is the overlap between two different weight vectors at temperature $`\beta ^1`$, and $`\stackrel{~}{R}_a`$ is the overlap of the perceptron’s solution and the teacher’s vector.

As for $`k=1`$ and $`k=2`$ the energy in (5) is a quadratic function in a convex domain, it has a single minimum , irrespective of the kind of rule that is being learned. Therefore, we may safely assume that all the replicas are equivalent, even in the case of learning unrealizable rules. We obtain thus the typical properties for cases where, using other more usual cost functions like the number of training errors, full replica symmetry breaking would be required . The excellent agreement of the theoretical predictions and the numerical simulations presented in the following section is a further justification of our hypothesis of replica symmetry. Thus, we set $`Q_aQ`$, $`q_{ab}q`$ and $`\stackrel{~}{R_a}\stackrel{~}{R}`$, and we define the normalized overlap $`R\stackrel{~}{R}/\sqrt{Q}`$, that only depends on the angle between $`𝐰`$ and $`𝐰_0`$.

Due to the unicity of the soft margin solution, only one point in phase space has non vanishing probability in the limit $`\beta \mathrm{}`$, so that $`qQ`$. It is convenient to introduce a new parameter, $`x\beta (Qq)`$, which reflects how fast the fluctuations around the minimum of (5) vanish as $`\beta \mathrm{}`$. In this limit we obtain the typical free energy of the SMC learning a rule defined by the polynomial $`𝒫`$,

$$f_{C,k}(𝒫)=\mathrm{extr}_{\{Q,R,x\}}\left(G_0(Q,R,x)\alpha G_{C,k}(Q,R,x;𝒫)\right),$$
(17)
where

$$G_0(Q,R,x)=\frac{Q}{2x}(1R^2x),$$
(18)
is an entropic term. The dependence on the rule to be learned is embodied in the second term of (17) through $`𝒫(z)`$, and on the learning algorithm through $`k`$ and $`C`$. Integrating out the slack variables in the limit $`\beta \mathrm{}`$ through a saddle point approximation, we get

$$G_{C,k}(Q,R,x;𝒫)=_{\mathrm{}}^{\mathrm{}}Dy_{\varphi (y;Q,R,𝒫)}^{\mathrm{}}Dt\underset{\zeta }{\mathrm{min}}W(\zeta ;y,t,Q,R,x,𝒫),$$
(19)
where $`Dtdt\mathrm{exp}(t^2/2)/\sqrt{2\pi }`$,

$$\varphi (y;Q,R,𝒫)=\frac{yR\mathrm{sign}(𝒫(y))1/\sqrt{Q}}{\sqrt{1R^2}},$$
(20)
and

$$W(\zeta ;y,t,Q,R,x,𝒫)=C\zeta ^k+\frac{\left(\zeta \sqrt{Q(1R^2)}(t\varphi (y;Q,R,𝒫))\right)^2}{2x}.$$
(21)
In (19), due to the saddle point approximation, $`W(\zeta ;y,t,Q,R,x,𝒫)`$ has to be taken at its minimum $`\zeta (t,y)[0,\sqrt{Q(1R^2)}\varphi (y;Q,R,𝒫)]`$ for each couple $`(y,t)`$. It is easy to see that there is a unique local minimum inside this interval for $`k>1`$. For $`k=1`$, $`W`$ is a quadratic function of $`\zeta `$, whose global minimum falls inside the allowed interval only for a finite range of values of $`t`$. Outside this range, the minimum lies at the boundary $`\zeta =0`$. As a consequence, for $`k=1`$ the inner integral in $`G_{C,k}`$ splits into two parts. The results for $`k=1`$ and $`k=2`$ are respectively:

$`G_{C,1}(Q,R,x;𝒫)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\frac{xC1}{\sqrt{Q}}}}Dt{\displaystyle \frac{(t\sqrt{Q}+1)^2}{2x}}g(t;R,𝒫)+{\displaystyle _{\frac{xC1}{\sqrt{Q}}}^{\mathrm{}}}DtC(t\sqrt{Q}+1{\displaystyle \frac{xC}{2}})g(t;R,𝒫)`$ (22)
$`G_{C,2}(Q,R,x;𝒫)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\mathrm{}}}Dt{\displaystyle \frac{C(t\sqrt{Q}+1)^2}{1+2xC}}g(t;R,𝒫)`$ (23)
with

$$g(t;R,𝒫)=\frac{dy}{\sqrt{2\pi (1R^2)}}\mathrm{exp}\left(\frac{\left(y\mathrm{sign}(P(y))+tR\right)^2}{2(1R^2)}\right).$$
(24)
Deriving the free energy (17) with respect to $`Q`$, $`R`$ and $`x`$ gives three coupled equations for the order parameters. These in turn, determine the properties of the SMC. The explicit expression of the saddle point equations for $`k=1`$ and $`k=2`$ is left to the Appendix, where we also derive some general properties of the learning curves described in the next sections.

The generalization error $`ϵ_g`$, which is the probability of misclassification of any pattern drawn with probability (9), is a geometric property that depends only on $`R`$ and the rule to be learnt. In the case of rules of type (10), it is straightforward to obtain

$$ϵ_g=DtH\left(\frac{tR\mathrm{sign}(𝒫(t))}{\sqrt{1R^2}}\right)$$
(25)
where $`H(x)=_x^{\mathrm{}}Dt`$. In the particular case of a linearly separable rule $`𝒫(z)=z`$, (25) reduces to the usual expression $`ϵ_g=\mathrm{arccos}(R)/\pi `$.

The distribution of stabilities $`\gamma _\mu h_\mu /𝐰`$ of the training patterns, $`\rho (\gamma )`$, is given by

$`\rho (\gamma )`$ $`=`$ $`\mathrm{\Theta }(\gamma {\displaystyle \frac{1}{\sqrt{Q}}}){\displaystyle \frac{e^{\gamma ^2/2}}{\sqrt{2\pi }}}g(\gamma ;R,𝒫)+\mathrm{\Theta }({\displaystyle \frac{1}{\sqrt{Q}}}\gamma ){\displaystyle \frac{e^{(\gamma kxC)^2/2}}{\sqrt{2\pi }}}g(\gamma +kxC;R,𝒫)`$ (26)
$`+`$ $`\delta (\gamma {\displaystyle \frac{1}{\sqrt{Q}}})({\displaystyle }Dt(H\left({\displaystyle \frac{tR\mathrm{sign}(𝒫(t))\gamma }{\sqrt{1R^2}}}\right)H\left({\displaystyle \frac{tR\mathrm{sign}(𝒫(t))\gamma +kxC/\sqrt{Q}}{\sqrt{1R^2}}}\right)).`$ (27)
The training error $`ϵ_t`$ is the average fraction of classification error on the training patterns. Integrating (26) over the negative stabilities we obtain

$$ϵ_t=DtH\left(\frac{tR\mathrm{sign}(𝒫(t))+kxC/\sqrt{Q}}{\sqrt{1R^2}}\right).$$
(28)
As expected, the training error is always strictly smaller than the generalization error. Both converge to the same limit for $`\alpha \mathrm{}`$.

## III Learning curves

In this section we present the learning curves, namely the training error $`ϵ_t(\alpha )`$ and the generalization error $`ϵ_g(\alpha )`$ of the SMCs for different teacher rules. We include in the figures the learning curves of the corresponding hard margin SVMs, or MSP, determined within the hypothesis of replica symmetry. In the case of unrealizable rules it is well known that the replica symmetry is broken for $`\alpha `$ larger than $`\alpha _{MSP}`$, the fraction of training patterns at which the hard margin $`\kappa _{max}`$, positive for $`\alpha <\alpha _{MSP}`$, vanishes. The results of computer simulations drawn on the same figures have been obtained by solving numerically the dual problem  using the Quadratic Optimizer for Pattern Recognition program , that we adapted to the case without threshold treated in this paper. The average has been taken over as many training sets as necessary (typically $`500`$ for small $`\alpha `$ and $`50`$ for big $`\alpha `$) to ensure that the error bars are smaller than the symbols. These simulations are in excellent agreement with the theoretical predictions.

### A The linear rule

Introducing the expression $`𝒫(z)=z`$ corresponding to a linearly separable teacher’s rule in (24), we obtain:

$$g(t;R,𝒫)=2H\left(\frac{Rt}{\sqrt{1R^2}}\right)$$
(29)
The training and generalization errors, obtained after solving the extremum equations for different values of the hyperparameter $`C`$, are plotted against $`\alpha `$ on Figures 1 and 2 for $`k=1`$ and $`k=2`$ respectively. The generalization error of the hard margin classifier, solution of (3) with conditions (4), and that of the optimal bayesian generalizer , which are both error-free solutions, are included on the figures for comparison. Despite the fact that the task is realizable by the student perceptron, the training error for finite $`C`$ is finite. It goes through a maximum and vanishes asymptotically in the limit $`\alpha \mathrm{}`$. As expected, both for $`k=1`$ and $`k=2`$ at any $`\alpha `$, $`ϵ_t`$ is larger the smaller the value of $`C`$, which controls the relative importance of the error term in the cost function (5). We can also see from the figures that, given $`C`$, the machine with $`k=2`$ performs better than the one with $`k=1`$. On increasing $`C`$, the learning curves approach those of the MSP. In fact, by taking the limit $`C\mathrm{}`$ in our saddle point equations we get exactly the equations of the MSP for every value of $`\alpha `$, independently of the power $`k`$. This is not surprising, as in this limit the error term dominates completely the soft margin cost function (5), which can only be minimized if all the slack variables, and consequently the training error, vanish. This is possible because the rule is realizable. It is well known that the generalization error of the MSP is larger than that of the bayesian generalizer even asymptotically, as for $`\alpha \mathrm{}`$ both algorithms have $`ϵ_ga/\alpha `$, but $`a=0.5005`$ in the case of the MSP , whereas $`a=0.442`$ for the bayesian perceptron .

The obtained behaviour of the learning curves at finite $`C`$ is reminiscent of that arising with other learning algorithms having a hyperparameter. In the inconsistent algorithms studied by Meir and Fontanari , patterns closer to the hyperplane than a finite imposed distance $`\kappa >\kappa _{max}`$ contribute to the cost, linearly in the case of the perceptron algorithm and quadratically in the case of the relaxation one. In the algorithm Minimerror  the hyperparameter is equivalent to a learning temperature. By training with these algorithms, as well as with the SMC studied here, the generalization error can be made smaller than that of the MSP by choosing appropriate values for the hyperparameters, at the price of learning with errors. The reason is that, in contrast with the MSP, the bayesian solution presents a finite fraction of training patterns at any distance of the hyperplane . Thus, solutions with a small controlled fraction of training errors may be closer to the optimal bayesian hyperplane than the MSP, which has no patterns at distances smaller than $`\kappa _{max}`$.

Unlike the generalization error of the inconsistent learning algorithms, that vanishes asymptotically like $`ϵ_g1/\sqrt{\alpha }`$ , SMCs with finite $`C`$ present a faster power law decay:

$$ϵ_g\frac{ϵ_0}{C^{\frac{1}{6}}}\frac{1}{\alpha ^{\frac{2}{3}}},$$
(30)
where the constant $`ϵ_0`$ is larger for $`k=1`$ than for $`k=2`$. In the limit $`C\mathrm{}`$ eq. (30) no longer holds, and the well known decay $`ϵ_g\alpha ^1`$ characteristic of error-free trained perceptrons learning realizable tasks is recovered.

Independently of the value of $`C`$, both the regularization term, proportional to $`Q`$, and the slacks term diverge like $`\alpha ^{2/3}`$ for $`\alpha \mathrm{}`$. In fact, this divergence arises because we divided the free energy in (12) by $`N`$, instead of dividing by $`N(1+\alpha )`$, which gives the energy per degree of freedom. In the large $`\alpha `$ limit, this converges to $`0`$ as it should, like $`\alpha ^{1/3}`$. The separable case is the only one where the error term in the cost function presents the same asymptotic behaviour as the regularization term. In this limit, the soft margin $`1/\sqrt{Q}`$ vanishes like $`\alpha ^{1/3}`$, in contrast with the hard margin behaviour, $`\kappa _{max}\alpha ^1`$ .

### B The shifted linear rule

Next we analyze the case of a linear teacher with a bias $`\delta >0`$. The corresponding polynomial has a single root: $`𝒫(z)=z\delta `$. This teacher separates linearly the examples with a hyperplane at a distance $`\delta `$ from the origin. As the student perceptron has no bias ($`b=0`$), zero generalization error cannot be achieved: this rule is unrealizable. The lowest value of $`ϵ_g`$, obtained by taking the asymptotic limit $`R1`$ in (25), is $`ϵ_g^{\mathrm{}}=0.5H\left(\delta \right)`$.

The function $`g`$ defined by (24) is:

$$g(t;R,𝒫)=H\left(\frac{Rt+\delta }{\sqrt{1R^2}}\right)+H\left(\frac{Rt\delta }{\sqrt{1R^2}}\right).$$
(31)
Learning curves for different values of $`C`$ are represented as a function of $`\alpha `$ on Figure 3, for the particular value $`\delta =0.3`$. The training error of the MSP is zero up to $`\alpha _{MSP}`$, at which the maximal stability $`\kappa _{max}`$ vanishes. $`\alpha _{MSP}`$ is a decreasing function of $`\delta `$. It diverges at $`\delta =0`$, as the problem becomes separable, and tends to the perceptron’s capacity $`\alpha _c=2`$ in the infinite $`\delta `$ limit. $`\alpha _{MSP}`$ cannot be smaller than $`\alpha _c`$ since in the thermodynamic limit any training set can be learned without errors for $`\alpha <\alpha _c`$ . Within the replica symmetry hypothesis, the MSP’s training error $`ϵ_t`$ displays a discontinuous transition at $`\alpha =\alpha _{MSP}`$. For $`\alpha >\alpha _{MSP}`$, $`ϵ_tϵ_g`$. The generalization error does not present any singularity at $`\alpha =\alpha _{MSP}`$. As already mentioned, $`\kappa _{max}`$ becomes negative for $`\alpha >\alpha _{MSP}`$, and the cost function $`E_{MSP}`$ (1) is likely to present several disconnected minima. Thus, the hypothesis of replica symmetry used to draw the MSP’s learning curves in Figure 3 is most probably wrong.

If we take the limit $`C\mathrm{}`$ in our equations, we get those corresponding to the MSP only for $`\alpha <\alpha _{MSP}`$. At $`\alpha _{MSP}`$, the training error of the SMC starts increasing and the generalization error curve detaches down from that of the MSP, both through a second order phase transition. The learning curves obtained in the limit $`C\mathrm{}`$ are different for $`k=1`$ and $`k=2`$, in contrast with the realizable rule considered before, in which they converge to that of the MSP irrespective of the value of $`k`$.

For finite values of $`C`$ the transition at $`\alpha _{MSP}`$ becomes a crossover both for $`ϵ_t`$ and $`ϵ_g`$, at values of $`\alpha <\alpha _{MSP}`$ that decrease on decreasing $`C`$. The training error for all $`\alpha `$ is larger than that for infinite $`C`$, both for $`k=1`$ and $`k=2`$. The generalization errors for different values of $`C`$ cross each other as a function of $`\alpha `$. The envelope of the curves $`ϵ_g(\alpha )`$ corresponds to the lowest possible value of $`ϵ_g`$ reachable by the corresponding SMCs. It depends on the exponent $`k`$. Notice that for large enough values of $`\alpha `$ the replica symmetric approximation to the MSP’s generalization error is smaller than that of the optimal soft margin solutions, and seems to provide a lower bound to $`ϵ_g`$ for the SMCs.

The convergence of the generalization error to its asymptotic limit, for all values of $`C`$, is exponentially fast with $`\alpha `$:

$$ϵ_gϵ_g^{\mathrm{}}\mathrm{exp}(\frac{\alpha }{a_k})$$
(32)
The decay constant $`a_k`$ does not depend on $`C`$. A stronger exponential drop of the generalization error, with $`\alpha ^2`$ in the exponent, has been found for SVMs learning “easy” teacher rules. These not only are realizable, but present a gap in the patterns distribution close to the discriminating surface. In contrast, here the student’s hyperplane is surrounded by unlearnable patterns. The student cannot get rid of the errors by decreasing the soft margin, like with the linear rule. On increasing $`\alpha `$, $`Q`$ converges to a constant that depends on $`k`$ and $`\delta `$ while the error term in (5) increases with $`\alpha `$. For large enough $`\alpha `$, the cost function is mainly dominated by the error term, and then $`C`$ only plays the role of an irrelevant multiplicative constant. This is why the convergence rate to the asymptotic value of the generalization error does no depend on $`C`$.

Similar results are obtained for $`k=2`$, as is shown on Figure 4.

### C Sandwich Rule

Consider now rules of the form $`𝒫(z)=z(z\delta )`$, where the polynomial defining the teacher’s output has two roots. The corresponding discriminating surfaces are two parallel hyperplanes, one containing the origin and the other at a distance $`\delta /\sqrt{N}`$ of it. The patterns that lying between the hyperplanes belong to class $`+1`$, the others to class $`1`$. Thus, not only these are unrealizable rules, but the classification errors will necessarily correspond to patterns at large distance of the student’s hyperplane.

As with all the unrealizable rules, the training error of the MSP within the replica symmetric approximation presents a discontinuity at $`\alpha _{MSP}`$ where $`\kappa _{max}`$ vanishes. Here $`\alpha _{MSP}`$ is an increasing function of $`\delta `$, starting at $`\alpha _{MSP}=2`$ for $`\delta =0`$, which corresponds to the most difficult learning task and diverging for $`\delta \mathrm{}`$. The generalization error starts decreasing at small $`\alpha `$, reaches a minimum beyond $`\alpha _{MSP}`$ and then starts to increase, and tends asymptotically to $`ϵ_g=1/2`$ for $`\alpha \mathrm{}`$. Notice however that for $`\alpha >\alpha _{MSP}`$ the replica symmetry is most probably broken.

The properties of the SMC are obtained by replacing

$$g(t;R,𝒫)=2H\left(\frac{Rt}{\sqrt{1R^2}}\right)+H\left(\frac{\delta Rt}{\sqrt{1R^2}}\right)H\left(\frac{Rt+\delta }{\sqrt{1R^2}}\right),$$
(33)
in the saddle point equations (37-39) of the Appendix.

The learning curves for different values of the hyperparameter $`C`$, corresponding to a width $`\delta =2`$, are represented on Figures 5 and 6 for $`k=1`$ and $`k=2`$ respectively. Given $`C`$, for large enough $`\alpha `$, the training error curves $`ϵ_t(\alpha )`$ for $`k=1`$ are below those for $`k=2`$. This is so because the unavoidable errors, which are very far from the hyperplane, are more heavily penalized if $`k=2`$. Thus, the SMC tries to learn these examples even if this increases the overall number of errors. As a result, learnable patterns close to the hyperplane, that have small slacks, are incorrectly classified. This can be checked up by taking a look at the distribution of stabilities, Figure 7.

Like with the previous shifted linear rule, the norm of the student’s weight vector $`Q`$ tends to a constant value and therefore, the error term dominates the cost function in the asymptotic limit $`\alpha \mathrm{}`$. However, instead of the exponential convergence, the generalization error decays asymptotically to $`ϵ_g^{\mathrm{}}=H\left(\delta \right)`$ like $`\alpha ^{\frac{1}{2}}`$. The reason of this difference is discussed in section V.

### D The Reversed Wedge

Teachers defined by third order polynomials like $`𝒫(z)=z(z\delta )(z+\delta )`$ with $`\delta >0`$, correspond to the so called Reversed Wedge  rules. Patterns $`𝐱_\mu `$ with $`𝐰_0𝐱_\mu (\mathrm{},\delta )(0,\delta )`$ belong to class $`1`$, those outside this subspace to class $`+1`$. The generalization properties of a perceptron learning a reverse wedge teacher have been addressed in , and within the on-line paradigm, using Hebb’s learning rule in .

The behaviour of the replica symmetric approximation to the MSP is as described in the previous section, but here $`\alpha _{MSP}`$ diverges both in the limits of vanishing and infinite wedge width $`\delta `$, for which the problem becomes separable, and has a minimum at $`\delta _c=\sqrt{2\mathrm{ln}2}`$ . At this value of $`\delta `$ the patterns stability distribution along the teacher’s weight $`𝐰_\mathrm{𝟎}`$ has zero mean. Correspondingly, learning becomes impossible for the MSP, as is discussed later. Thus, for $`\delta _c`$, $`R=0`$ for every value of $`\alpha `$, and $`\alpha _{MSP}=2`$ is equal to the perceptron’s capacity.

The properties of the SMCs are deduced after insertion of

$$g(t;R,𝒫)=2H\left(\frac{Rt\delta }{\sqrt{1R^2}}\right)+H\left(\frac{Rt+\delta }{\sqrt{1R^2}}\right)H\left(\frac{Rt}{\sqrt{1R^2}}\right)$$
(34)
into the saddle point equations.

In contrast with the problems considered before, the generalization error of a perceptron learning the reversed wedge rule is a monotonic function of $`R`$ only if $`\delta >\delta _c`$ . For $`0<\delta <\delta _c`$, $`ϵ_g(R)`$ presents a relative minimum at $`R_{min}>0`$ and a corresponding maximum at $`R_{min}`$. The relative minimum is the global one only for $`0<\delta <\delta _{}0.570185`$. At $`\delta _{}`$ the global minimum jumps to $`R=1`$, and for $`\delta _{}<\delta <\delta _c`$ the generalization error takes its smallest value at $`R=1`$. At $`\delta =\delta _c`$ the relative extrema collapse at the inflexion point $`R_{min}=0`$, and for larger values of $`\delta `$ the generalization error becomes a monotonic increasing function of $`R`$. This behaviour is represented on Figure 8.

For the values of $`k`$ investigated, $`R`$ has two distinct behaviors as a function of $`\alpha `$, depending on the wedge’s width $`\delta `$. If $`\delta <\delta _c`$, the teacher’s average stability is positive, and $`R(\alpha )`$ is a monotonic continuous function growing from $`0`$ to its asymptotic value $`+1`$. In this range of small wedges, the soft margin learning algorithm does not converge to the minimal value of the generalization error in the limit of infinite $`\alpha `$, as is the case in the other tasks considered before. In fact it “overshoots”, in the sense that $`R(\alpha )`$ continues to grow beyond the value that optimizes the generalization performance. Correspondingly, $`ϵ_g(\alpha )`$ goes through a minimum at finite $`\alpha `$ but, as $`R`$ increases with $`\alpha `$, it converges to a larger value, $`ϵ_g^{\mathrm{}}ϵ_g(R=1)`$. The learning curves of Figure 9 are an example of this behaviour. Notice that for $`\delta _{}<\delta <\delta _c`$ this value of $`ϵ_g^{\mathrm{}}`$ corresponds the largest value of the student’s generalization error. Moreover, for $`0.67449<\delta <\delta _c`$ the asymptotic behaviour is even worse than a random guess, because $`ϵ_g(R=1)>0.5`$.

At $`\delta =\delta _c`$ there is an abrupt change of the learning behaviour, as beyond this wedge’s width the average teacher’s stability is negative, and $`R`$ becomes a decreasing function of $`\alpha `$. Correspondingly, the soft margin solution converges to the optimal generalizer in the limit $`\alpha \mathrm{}`$. This corresponds to $`R=1`$, because for large $`\delta `$, most of the patterns lie in inside the reversed wedge, so that the student’s weight vector tends to orient antiparallel with the teacher’s vector $`𝐰_0`$, in order to classify correctly most of the examples. Learning curves for $`\delta =2>\delta _c`$ obtained with exponent $`k=1`$ for the slacks exponent in the cost function are represented on Figure 10.

As for the sandwich rule, the generalization error decays as $`\alpha ^{\frac{1}{2}}`$ to the corresponding asymptotic values, $`ϵ_g^{\mathrm{}}=12H(\delta )`$ for $`R1`$, and $`ϵ_g^{\mathrm{}}=2H(\delta )`$ for $`R1`$. The same asymptotic behaviors for $`ϵ_g`$ and $`R`$, but with different prefactors, were obtained by Inoue et al.  for the online Hebbian learning scenario.

The asymptotic value of $`Q`$ tends to zero as $`\delta `$ tends to $`\delta _c`$. In the two limiting cases $`\delta \mathrm{}`$ and $`\delta 0`$, the task becomes linearly separable and correspondingly $`Q\mathrm{}`$.

For the particular case of $`\delta =\delta _c`$, the only solution of the saddle point equations is $`R=0`$ for every value of $`\alpha `$. This “no learning” regime is discussed in section V.

## IV Optimization of the hyperparameter

The figures of the preceding section show that the behavior of the generalization error of the SMC is not monotonic with $`C`$. It can be seen that there is an optimal value $`C_{opt}(\alpha )`$ that allows to obtain the minimum generalization error for each $`\alpha `$. Obviously, $`C_{opt}`$ cannot be calculated using the training examples alone, so that in the applications it can only be estimated. Several methods for doing this have been proposed recently . Here we determine the statistical properties of the optimal SMC, thus providing reference curves against which results obtained using the different estimators may be tested.

As $`ϵ_g`$ depends implicitly on $`C`$ through $`R`$, in the cases where $`ϵ_g`$ is a monotonic function of $`R`$, its minimum is obtained by looking for the extremum of $`R`$ with respect to $`C`$, at fixed $`\alpha `$, $`C_{opt}(\alpha )`$. To this end, the three saddle point equations (37-39) of the Appendix, together with their derivatives with respect to $`C`$, constitute a system of 6 coupled equations for the variables $`Q`$, $`R`$, $`x^{}xC`$, $`Q/C`$, $`R/C`$ and $`x^{}/C`$. Setting the extremum condition $`R/C=0`$, the equations obtained by derivation of (38) and (39) form a homogeneous system for $`Q/C`$ and $`x^{}/C`$. The only nontrivial solution is obtained by setting the determinant of this system to zero, which gives

$$\frac{^2f}{QR}\frac{^2f}{x^2}\frac{^2f}{Rx}\frac{^2f}{Qx}=0.$$
(35)
where $`f`$ stands for the free energy (17). Solving the system given by equation (35) together with the three original saddle point equations for $`Q`$, $`R`$, $`x^{}`$ and $`C`$, we get $`C_{opt}`$ in the cases where $`ϵ_g`$ is a monotonic function of $`R`$.

In the other cases, as happens with the Reverse Wedge rule, determining $`C_{opt}`$ is less straightforward because the minimum of $`ϵ_g`$ may be reached for a value $`R^{}`$ (different from $`\pm 1`$) such that $`ϵ_g/R(R^{})=0`$, with $`R/C0`$. In that case, $`C_{opt}`$ is the one that gives $`R(C_{opt})=R^{}`$, and has to be determined numerically.

The optimal generalization curves for the different rules considered in this paper are represented on the figures of the preceding section. Notice that for $`\alpha <\alpha _{MSP}`$, the MSP is not optimal for any value of $`\alpha `$, as it is obtained in the limit $`C\mathrm{}`$. In the case of the realizable linear separation, the optimal generalization error of the SMC vanishes asymptotically as $`0.488\alpha ^1`$ for $`k=1`$, and as $`0.449\alpha ^1`$ for $`k=2`$. The latter is very close to that of the bayesian perceptron, $`0.442\alpha ^1`$, but the curves are also very close for finite values of $`\alpha `$, as can be seen on figure 2. Notice that the asymptotic decay of $`ϵ_g`$ for the SMC is faster than that of the MSP, even for $`k=1`$. This is an interesting result, as it shows that, even when a hard margin solution exists, learning with a soft margin machine allows to obtain better classifiers.

For the non separable cases, even if $`C_{opt}`$ allows to obtain the best performances at finite $`\alpha `$, since the asymptotic behavior of $`ϵ_g`$ is independent of $`C`$, all the learning curves, including the optimal one, tend to a value that only depends on the rule and on $`k`$, as shown in the corresponding sections.

The evolution of $`C_{opt}`$ with $`\alpha `$ can be seen on Figures 12 to LABEL:fig.12. The behaviour of the curves is qualitatively similar for the shifted linear rule and the reversed wedge with small $`\delta `$ on one hand, and for the sandwich rule and the reversed wedge with large $`\delta `$ on the other. The divergences of $`C_{opt}`$ are related to the presence of errors with unbounded slack values. For $`\alpha `$ beyond the divergence, $`C_{opt}=\mathrm{}`$.

## V Discussion

In the preceding sections we presented the learning curves of a SMC learning a variety of rules, characterized by an anisotropy axis parallel to the teacher’s vector $`𝐰_0`$. Some of the obtained results, and in particular the asymptotic behaviour in the $`\alpha \mathrm{}`$ limit, can be generalized to other teacher rules (Proofs are detailed in the Appendix). As shown by Reimann and Van den Broeck , it is useful to characterize the teacher rules by the average patterns’ stability of a perceptron aligned with the teacher’s vector,

$$\gamma =𝑑\gamma \rho (\gamma )=Dzz\mathrm{sign}(𝒫(z)),$$
(36)
where the second equality in (36) stems from our assumption (9) that the patterns’ distribution is a gaussian.

In the Appendix we show that in the limit $`\alpha \mathrm{}`$, both for $`k=1`$ and $`k=2`$, $`R`$ converges asymptotically either to $`1`$ or to $`1`$, that is, the student perceptron gets either completely aligned or completely anti-aligned with teacher’s vector. Furthermore, for non separable rules, $`1R^21/\alpha `$. In this limit of $`R\pm 1`$ we find $`ϵ_gϵ_g^{\mathrm{}}=Dz\theta (z𝒫(z))`$, irrespective of the teacher’s rule. The convergence law to this asymptotic value depends on whether the polynomial $`𝒫(z)`$ defining the rule in (10) has or not a root $`z_i=0`$. If $`0`$ is not a root of $`𝒫(z)`$, $`𝒫(0)0`$ and $`ϵ_gϵ_g^{\mathrm{}}\mathrm{exp}(\epsilon /(1R^2))`$ with $`\epsilon `$ a constant, whereas if $`0`$ is a root, then the decay follows the law $`ϵ_gϵ_g^{\mathrm{}}\sqrt{1R^2}`$.

Thus, for the unrealizable rules that have $`0`$ as one of the roots of $`𝒫`$, the generalization error decays to the asymptotic value as $`ϵ_gϵ_g^{\mathrm{}}\alpha ^{\frac{1}{2}}`$. A similar result has been obtained by Amari et al.  within the annealed approximation for the case of a deterministic machine learning a noisy teacher, and by other authors for hebbian learning of unrealizable tasks . The same power law has been obtained by Meir and Fontanari  for a realizable problem learned with inconsistent algorithms, within the approximation of replica symmetry, which is probably not valid for large values of $`\alpha `$. Indeed, the soft margin algorithm with finite $`C`$ is also inconsistent when the rule is the linear separation considered in section III A, and in that case we obtain a different power law decay.

In the case of a linearly separable rule, the SMC with $`C_{opt}`$ has $`ϵ_g1/\alpha `$, like the MSP, which corresponds to $`C=\mathrm{}`$. However, at fixed finite values of $`C`$ the decay is slower, like $`1/\alpha ^{2/3}`$. The same exponent has been obtained for a perceptron learning a separable rule using noisy examples with one step of replica symmetry breaking . Within the replica symmetric approximation to the same problem the exponent is $`1/2`$ instead of $`2/3`$ .

In the cases where $`0`$ is not a root of $`𝒫(z)`$, like for the shifted linear rule, the decay is exponential, $`ϵ_gϵ_g^{\mathrm{}}\mathrm{exp}(\epsilon \alpha )`$. A similar behaviour was found in  for a perceptron with linear output and binary weights, trying to learn examples given by a teaches with the same structure, where the generalization error vanishes exponentially.

The presence or the absence of a root $`z_i=0`$ induces different asymptotic behaviors because if $`0`$ is a root, then a student perceptron aligned with the teacher has $`|R|=1`$ and can perfectly separate the patterns closest to the hyperplane. In that case, any small misalignement modifies the classification induced by the student, thus strongly modifying the error term in the cost function. On the other hand, if $`0`$ is not a root, the student’s hyperplane is immersed in a sea of patterns of the same class. Small tilts of the hyperplane do not change significantly the classification nor the slacks term in the cost.

It is interesting to notice that the figures of the learning curves as well as those of $`C_{opt}`$ show an analogy between the behaviour for the SMCs with bounded slacks, like in the case of the shifted linear rule and that of the reversed wedge when $`\delta <\delta _c`$, and between those with unbounded slacks, as is the case with the sandwich rule and the reversed wedge when $`\delta >\delta _c`$. For this last type of rules, $`C_{opt}`$ diverges beyond some finite $`\alpha `$.

Consider now the small $`\alpha `$ limit. As shown in the Appendix, $`R\gamma \sqrt{\alpha }`$ and so, $`ϵ_g1/2\gamma ^2\sqrt{\alpha }`$. Thus, irrespective of the rule considered, when the fraction of training examples is small, the SMC generalizes better than by random guessing. This is not necessarily the case for larger values of $`\alpha `$.

If we put $`R=0`$ in the equations, and solve for $`\alpha `$, the only possible solution when $`\gamma 0`$, is $`\alpha =0`$. Thus, $`R0`$ for all $`\alpha `$, and has the sign of $`\gamma `$ unless it has discontinuous changes of sign. Notice that, given the asymptotic behaviours just mentioned, if $`R`$ is discontinuous it can only have an even number of changes of sign. A similar result has already been obtained in a broader frame . From the behaviour of $`R`$ in the small $`\alpha `$ limit, it can be seen that the problem gets very difficult to learn for rules with $`\gamma `$ close to $`0`$. In fact, in the very special case of $`\gamma =0`$, $`R=0`$ is a solution of the saddle point equations for every value of $`\alpha `$. If this is the only solution, the machine cannot learn at all, as is the case for the reverse wedge rule when $`\delta =\delta _c`$. This behaviour is similar to the one of retarded learning, found in problems of unsupervised learning with quadratic cost functions . In that case, it has been shown that learning is still possible, provided that the cost function is capable to extract the information about the anisotropy of the distribution of stabilities, contained in its higher order moments . Notice that this is not the case for the cost functions for the SMCs considered in this paper.

## VI Conclusion

The properties of the recently proposed Support Vector Machines have been studied theoretically in two situations of interest, namely for the cases where the student has either the same structure as the teacher, or it is more complex than it. In both situations the rule to be learned is realizable, and interesting properties of hard margin SVMs, like the existence of hierarchical generalization, could be analyzed within the replica symmetry hypothesis .

In the present paper we addressed the situation where the task is more complex than the learning machine. In this case the cost function for the SVMs is modified. It allows to obtain a Soft Margin Classifier that results from a trade-off, controlled by a single parameter $`C`$, between increasing the margin and minimizing the number of training errors. As the cost function is quadratic and the domain of solutions is convex, we obtain the typical learning curves for a variety of unrealizable tasks using the replica symmetry hypothesis. We considered problems characterized by a single symmetry-breaking direction $`𝐰_\mathrm{𝟎}`$, along which the patterns have alternating positive or negative class label. We have shown that the convergence of the corresponding learning curves to the asymptotic value follows either a power law or an exponential, depending on the position of the singularities of the teacher’s rule.

Even if the student is well adapted to the task’s complexity, the SMC may generalize better than the error-free hard margin SVM, provided the hyperparameter $`C`$ in the cost function is correctly tuned. It can even attain almost Bayesian performance.

We showed that the prefactors of the different asymptotic behaviours are proportional to the average stability of the teachers rule, $`\gamma `$. When this vanishes, the SMC with cost function (5) cannot learn, and the overlap between the student and the teacher directions is $`R=0`$. We considered two exponents for the error term in the cost function, $`k=1`$ and $`k=2`$. It would be interesting to study the properties of SMCs trained using exponents $`k>2`$ in the cost function, as we expect that these should detect the difference of the odd moments of the patterns distribution in the directions parallel and orthogonal to $`𝐰_0`$.

Another interesting question is whether the hierarchical learning of hard margin SVMs exists also with SMCs. To tackle this question, pattern distributions with two different anisotropies have to be considered.

## VII Acknowledgments

We thank Alex Smola for providing us with the Quadratic Optimizer for Pattern Recognition program . SR-G acknowledges economic support from the EU-research contract ARG/B7-3011/94/97. It is a pleasure to acknowledge support from the Zentrum für interdisziplinäre Forschung in Bielefeld, where this work was finished in the frame of the Research Group ”The Sciences of Complexity: From Mathematics to Technology to a Sustainable World”.

## VIII Appendix

The saddle point equations for the cases $`k=1`$ and $`k=2`$ are:

$`1R^2x`$ $`=`$ $`\alpha I_1(xC,\sqrt{Q},R;k),`$ (37)
$`R`$ $`=`$ $`\alpha I_2(xC,\sqrt{Q},R;k),`$ (38)
$`1R^2`$ $`=`$ $`\alpha I_3(xC,\sqrt{Q},R;k),,`$ (39)
with, for the case $`k=1`$

$`I_1(xC,Q,R;1)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\frac{xC1}{\sqrt{Q}}}}Dtt(t+{\displaystyle \frac{1}{\sqrt{Q}}})g(R,t,P)+{\displaystyle _{\frac{xC1}{\sqrt{Q}}}^{\mathrm{}}}Dt{\displaystyle \frac{txC}{\sqrt{Q}}}g(R,t,P),`$ (40)
$`I_2(xC,Q,R;1)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\frac{xC1}{\sqrt{Q}}}}Dt{\displaystyle \frac{1}{2}}(t+{\displaystyle \frac{1}{\sqrt{Q}}})^2{\displaystyle \frac{g(R,t,P)}{R}}+{\displaystyle _{\frac{xC1}{\sqrt{Q}}}^{\mathrm{}}}Dt{\displaystyle \frac{xC}{\sqrt{Q}}}(t+{\displaystyle \frac{2xC}{2\sqrt{Q}}}){\displaystyle \frac{g(R,t,P)}{R}},`$ (41)
$`I_3(xC,Q,R;1)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\frac{xC1}{\sqrt{Q}}}}Dt(t+{\displaystyle \frac{1}{\sqrt{Q}}})^2g(R,t,P)+{\displaystyle _{\frac{xC1}{\sqrt{Q}}}^{\mathrm{}}}Dt{\displaystyle \frac{(xC)^2}{Q}}g(R,t,P)`$ (42)
and, for the case $`k=2`$,

$`I_1(xC,Q,R;2)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\mathrm{}}}Dt{\displaystyle \frac{2xCt}{1+2xC}}(t+{\displaystyle \frac{1}{\sqrt{Q}}})g(R,t,P)`$ (43)
$`I_2(xC,Q,R;2)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\mathrm{}}}Dt{\displaystyle \frac{xC}{1+2xC}}(t+{\displaystyle \frac{1}{\sqrt{Q}}})^2{\displaystyle \frac{g(R,t,P)}{R}}`$ (44)
$`I_3(xC,Q,R;2)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\mathrm{}}}Dt{\displaystyle \frac{(2xC)^2}{(1+2xC)^2}}(t+{\displaystyle \frac{1}{\sqrt{Q}}})^2g(R,t,P)`$ (45)
From (45) it can be seen that, for $`k=2`$, $`x`$ must vanish in the infinite $`\alpha `$ limit in order to make $`I3`$ vanish. Notice that the function $`g(R,T,P)`$ is always nonnegative (24). For the case $`k=1`$ the analysis of (42) shows that $`x`$ must either vanish or tend to a positive constant with $`q`$ tending to infinity. This last case can be ruled out by noticing that it is inconsistent with the vanishing of $`I2`$ (notice that (41),as well as (44) can be solved analitically).

To show that $`R`$ can only tend to $`1`$ or $`1`$ in the infinite $`\alpha `$ limit, it is useful to rewrite I1 and I2, which in the case $`k=1`$ are

$`I_1(xC,Q,R;1)`$ $`=`$ $`{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\frac{xC1}{\sqrt{Q}}}}Dtg(R,t,P)+RI_2(xC,Q,R;2)`$ (46)
$`I_2(xC,Q,R;1)`$ $`=`$ $`{\displaystyle \underset{i=1}{\overset{N}{}}}\tau (x_i^+){\displaystyle \frac{e^{x_i^2}}{\sqrt{2\pi }}}\{{\displaystyle _{\frac{\frac{1}{\sqrt{Q}}x_iR}{\sqrt{1R^2}}}^{\frac{\frac{xC1}{\sqrt{Q}}x_iR}{\sqrt{1R^2}}}}Dt(t\sqrt{1R^2}+{\displaystyle \frac{1}{\sqrt{Q}}}+x_iR)`$ (47)
$`+`$ $`{\displaystyle \frac{xC}{\sqrt{Q}}}{\displaystyle _{\frac{\frac{xC1}{\sqrt{Q}}x_iR}{\sqrt{1R^2}}}^{\mathrm{}}}Dt+(x_ix_i)\}`$ (48)
and for $`k=2`$,

$`I_1(xC,Q,R;2)`$ $`=`$ $`{\displaystyle \frac{2xC}{1+2xC}}{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\mathrm{}}}Dtg(R,t,P)+RI_2(xC,Q,R;2)`$ (49)
$`I_2(xC,Q,R;2)`$ $`=`$ $`{\displaystyle \frac{2xC}{1+2xC}}{\displaystyle \underset{i=1}{\overset{N}{}}}\tau (x_i^+){\displaystyle \frac{e^{x_i^2}}{\sqrt{2\pi }}}\{{\displaystyle _{\frac{\frac{1}{\sqrt{Q}}x_iR}{\sqrt{1R^2}}}^{\mathrm{}}}Dt(t\sqrt{1R^2}`$ (50)
$`+`$ $`{\displaystyle \frac{1}{\sqrt{Q}}}+x_iR)+(x_ix_i)\}.`$ (51)
Let us suppose that $`R`$ tends to a constant different from $`1`$ and $`1`$ as $`\alpha `$ tends to infinity. It can be seen that in that case $`I1`$, $`I2`$ and $`I3`$ must vanish at the same rate. If we consider teachers with at least one positive root , i. e. unrealizable teachers, it can be seen that the integral in I3 (the second one for the case k=1) never vanishes. Thus, $`I3`$ must vanish as $`(x/\sqrt{Q})^2`$ for k=1 and as $`x^2`$ for k=2, if $`Q`$ tends to a constant or to infinity. But equations (46) and (49) show that $`I1`$ and $`I2`$ cannot vanish at the same rate as $`I3`$ because the first term on the right handside vanishes as $`x/\sqrt{Q}`$ for $`k=1`$ and as $`x`$ for $`k=2`$. If $`Q`$ tends to $`0`$ then I3 must vanish as $`(x/\sqrt{Q})^2`$ for both cases. But then $`I2`$ cannot vanish at the same rate, because equations (47) and (50) show that $`I2`$ must vanish as $`xC\gamma /\sqrt{Q}`$, unless $`\gamma =0`$ (this case will be analyzed below). Therefore, $`R`$ tends either to $`1`$ or to $`1`$ for all teachers with $`\gamma 0`$.

By putting $`R=0`$ in the equations one can easily (notice that $`g(0,t,P)1`$) see that if $`\gamma 0`$, it can only be a solution for $`\alpha =0`$. On the other hand, for $`\gamma =0`$, $`R=0`$ is a solution for every value of $`\alpha `$, i. e. learning is impossible for this kind of teacher.

It is also possible to find the condition that makes $`R`$ go to each one of its limiting values (1 or -1). From what has been said before regarding $`I3`$ it can be seen that it vanishes as $`x^2`$, and so, $`1R^2\alpha x^2`$. Using this, and equation (37) it is evident that $`I1`$ must vanish faster than $`x`$. But, in the infinite $`\alpha `$ limit, $`I1`$ is written, to first order,

$`I_1(xC,Q,R;1)`$ $``$ $`{\displaystyle \frac{x}{\sqrt{Q}}}\{sign(R)\gamma +{\displaystyle _{\mathrm{}}^{\frac{1}{\sqrt{Q}}}}Dttg(\pm 1,t,P)\}`$ (52)
$`I_1(xC,Q,R;2)`$ $``$ $`x\{sign(R){\displaystyle \frac{\gamma }{\sqrt{Q}}}{\displaystyle _{\frac{1}{\sqrt{Q}}}^{\mathrm{}}}Dtg(\pm 1,t,P)\}`$ (53)
$``$ $`sign(R){\displaystyle \underset{i/|x_i|>\frac{1}{\sqrt{Q}}}{}}\tau (x_i^+)(|x_i|{\displaystyle \frac{1}{\sqrt{Q}}}){\displaystyle \frac{e^{x_i^2}}{\sqrt{2\pi }}}\}`$ (54)
Thus, the term within brackets must vanish. For (52) it is evident that this can only happen if $`Rsign(\gamma )`$. The same can be shown for (53), with a bit of algebra. The asymptotic value of $`Q`$ can be obtained by imposing the vanishing of the above mentioned terms.

To see the rate of decay of $`1R^2`$, notice that, from (39) and from the fact (shown above) that $`I3x^2`$, one gets that $`1R^2\alpha x^2`$. But, using the fact that $`I1`$ must decay faster than $`x`$, equations (47) and (50) impose that $`I2x`$. This, together with (38), gives that $`x1/\alpha `$. Therefore, $`1R^21/\alpha `$.


text:
# Superconducting condensation energy and neutron resonance in high-𝑇_𝑐 superconductors

## I INTRODUCTION

One of the important aspects to understand the mechanism of superconductivity is to investigate the origin of the condensation energy, which is the free energy difference between the normal and the superconducting(SC) states. In conventional BCS superconductors, the condensation energy is accounted for by the change in the ion kinetic energy between the normal and the SC phases, as shown by Chester . In high-$`T_c`$ superconductors, it is now widely accepted that the strong electronic interactions rather than the electron-phonon interactions may be responsible for the superconductivity. However, no consensus has been achieved on the detail pairing mechanism, consequently on the origin of the condensation energy .

In analogy with the phonon-mediated electron pairing, Scalapino and White  proposed that if the pairing is mediated by the antiferromagnetic (AF) exchange interaction, the condensation energy will be proportional to the change in the AF exchange energy between the normal and the SC states. This gives a direct connection between the condensation energy and the dynamical spin susceptibility $`\chi `$ (or the spin structure factor $`S(𝐪,\omega )=\mathrm{Im}\chi (𝐪,\omega )/[1+\mathrm{exp}(\omega /T)]`$) as ,

$`E_NE_S`$ $`=`$ $`{\displaystyle \frac{3}{2}}J{\displaystyle \underset{q}{}}{\displaystyle _0^{\mathrm{}}}{\displaystyle \frac{d\omega }{\pi }}[\mathrm{Im}\chi ^N(𝐪,\omega )\mathrm{Im}\chi ^S(𝐪,\omega )]`$ (2)
$`\times [\mathrm{cos}(q_x)+\mathrm{cos}(q_y)]{\displaystyle \frac{1}{1\mathrm{exp}(\omega /T)}}.`$
The dynamical spin susceptibility can be probed directly by neutron scattering experiments. The most prominent feature in these experiments when the temperature decreases below the SC transition is the appearance of a sharp resonance peak at the AF wavevector $`𝐐=(\pi ,\pi )`$ ($`\pi `$ resonance) in the SC state . From this observation, Demler and Zhang  further argued that the appearance of the neutron resonance will cause the antiferromagnetic exchange energy to be lower than that in the normal state. Without resorting to any microscopic models, they estimated the exchange energy to be around 0.016$`J`$ based on the experimental data for the spin structure factor and the $`q`$-width of the resonance peak. The estimated condensation energy is roughly of the same magnitude as that obtained by experiments  and therefore accounts for a large part of the condensation energy. The above estimate is based on Eq.(2) without including the kinetic energy. Furthermore, it is limited to zero temperature and with the assumption that the spin susceptibility in the normal state has negligible contribution to Eq.(1). Its validity thus relies on further tests against experimental data. In particular, as pointed out at the end of Ref. , a useful test of this idea is to compare the theoretical estimate with experiments at finite temperatures. Experimentally, because the net difference of the AF exchange energy is small, it will require extremely careful measurements to check Eq.(2). Only recently, a study of the temperature dependence is performed. In Ref. , the neutron scattering measurement shows a similar temperature dependence between the condensation energy from the AF exchange interactions and the resonance peak intensity for optimal doped YBa2Cu3O6+x. In view of the experimental situation, it would be useful to carry out calculations at finite temperatures based on an existing high-$`T_c`$ model. Recently, the $`tt^{}J`$ model has been shown to produce the resonance peak , hence it would be interesting to verify its prediction for the condensation energy.

In this paper, we examine the temperature and doping dependences of the condensation energy in a microscopic model – the two-dimensional (2D) $`tt^{}J`$ model. To avoid the complications due to the appearance of the pseudo-gap in the normal state, we shall confine our calculations in the optimal and overdoped regions. Theoretically, the condensation energy can be calculated from the difference in the thermodynamic potential between the normal and the SC states because the difference in free energy in the SC state is equal to the difference in thermodynamical potential . In order to include the $`\pi `$ resonance peak, we calculate the thermodynamical potential in the linked cluster expansion via a renormalized random-phase approximation (RPA) as defined in Ref. . This approach has been previously tested in different contexts  and has been shown to capture many important features of the $`\pi `$ resonance peak. Here we shall further investigate how it would predict for the temperature dependence of the condensation energy. We show that the condensation energy coming from the AF exchange correction is nearly the same as Eq.(2) due to the presence of the neutron resonance peak. Furthermore, the condensation energy from the AF exchange correction has the similar temerature dependence as the resonance peak intensity for both optimally doped and overdoped systems. This is consistent with the experimental results observed in the optimally doped sample . However, the total condensation energy after including the mean-field free energy difference between the normal and SC states exhibits a more rapid drop in its temperature dependence than that calculated with Eq.(2). We also investigate the doping dependence of the total condensation energy and find it in agreement with the experiment. In view of the appearance of the pseudo-gap in the normal state for underdoped cuprates  where a part of the condensation energy has gained above $`T_c`$ , this result indicates that the condensation energy for the optimally doped and overdoped cuprates should show more rapid temperature drop than that for the underdoped cuprates.

This paper is organized as follows. In Sec.II, we introduce the model and define notations. The condensation energy is derived using the linked cluster expansion. In Sec.III, we calculate the temperature and doping dependences of the condensation energy and compare them with that calculated based on Eq.(2). Finally, we give a concluding remark in Sec.IV.

## II THERMODYNAMIC POTENTIAL AND CONDENSATION ENERGY

We start with the 2D $`tt^{}J`$ model which reads,

$`H`$ $`=`$ $`{\displaystyle \underset{<ij>,\sigma }{}}tc_{i\sigma }^{}c_{j\sigma }h.c.{\displaystyle \underset{<ij>^{},\sigma }{}}t^{}c_{i\sigma }^{}c_{j\sigma }h.c.`$ (4)
$`+J{\displaystyle \underset{<ij>}{}}𝐒_i𝐒_j,`$
where, $`<ij>`$ denotes the nearest-neighbor (n.n.) bond, $`<ij>^{}`$ the next n.n. bond and $`𝐒_i=\frac{1}{2}c_{i\alpha }^{}\sigma _{\alpha \beta }c_{i\beta }`$. In the slave-boson method, the physical electron operators $`c_{i\sigma }`$ are expressed by slave bosons $`b_i`$ carrying the charge and fermions $`f_{i\sigma }`$ representing the spin; $`c_{i\sigma }=b_i^+f_{i\sigma }`$. In the SC state, we consider the order parameters $`\mathrm{\Delta }_{ij}=<f_if_jf_if_j>=\pm \mathrm{\Delta }_0`$ with the $`d`$-wave symmetry and $`\chi _{ij}=_\sigma <f_{i\sigma }^+f_{j\sigma }>=\chi _0`$, in which bosons condense $`b_i<b_i>=\sqrt{\delta }`$ ($`\delta `$ is the hole concentration) . Then, the mean-field Hamiltonian in the SC state is,

$`H_m`$ $`=`$ $`{\displaystyle \underset{k\sigma }{}}ϵ_kf_{k\sigma }^{}f_{k\sigma }{\displaystyle \underset{k}{}}\mathrm{\Delta }_k(f_k^{}f_k^{}+h.c.)`$ (6)
$`+2NJ^{}(\chi _0^2+\mathrm{\Delta }_0^2),`$
where $`ϵ_k=2(\delta t+J^{}\chi _0)[\mathrm{cos}(k_x)+\mathrm{cos}(k_y)]4\delta t^{}\mathrm{cos}(k_x)\mathrm{cos}(k_y)\mu `$ is the dispersion for fermions, and $`\mathrm{\Delta }_k=2J^{}\mathrm{\Delta }_0[\mathrm{cos}(k_x)\mathrm{cos}(k_y)]`$, with $`J^{}=3J/8`$. In the optimal and overdoped regions, the mean-field Hamiltonian for the normal state is obtained by setting the SC gap $`\mathrm{\Delta }_0=0`$ . The mean-field parameters $`\chi _0`$, $`\mathrm{\Delta }_0`$ ($`\mathrm{\Delta }_0`$=0 in the normal state) and the chemical potential $`\mu `$ for different doping $`\delta `$ are obtained from a self-consistent calculation .

It has been shown  that the spin susceptibility at the mean-field level represented by the first fermionic bubble in Fig.1 fails to describe some important features observed by inelastic neutron scattering experiments, such as the resonance peak . The underlying reason is because the mean-field theory underestimates the spin fluctuation near $`(\pi ,\pi )`$. Therefore, it is necessary to go beyond the mean field approximation. Formally, this can be done by perturbing around the mean-field Hamiltonian, i.e., we write $`H=H_m+H^{}`$, and treat $`H^{}`$ as a perturbation. In principle, all the fluctuations are included. However, different selection of subset diagrams may result in different kinds of fluctuations. For the spin fluctuation, the usual random phase approximation selects a series of ring diagrams as shown in Fig.1. The resulting spin susceptibility can be resummed as

$$\chi ^+(𝐪,\omega )=\frac{\chi _0^+(𝐪,\omega )}{1+\eta J\gamma _q\chi _0^+(𝐪,\omega )}.$$
(7)
where $`\gamma _q=\mathrm{cos}(q_x)+\mathrm{cos}(q_y)`$, $`\chi _0`$ is the unperturbed spin susceptibility which comes from the fermionic bubble. The parameter $`\eta `$ is formally introduced to keep track of the renormalization of vertex and its value is one in the usual RPA approach. However, the spin susceptibility calculated using Eq.(7) with $`\eta =1`$ leads to an AF instability for doping $`\delta \delta _c0.22`$, which is much larger than the experimental data around $`\delta _c=0.02`$. It indicates that there exists other fluctuations which acts as to suppress this overestimation. In the renormalized RPA approach, one models the suppression of $`\eta `$ by treating $`\eta `$ as a phenomenological parameter, whose value is determined by setting the AF instability at the observed value $`\delta =0.02`$. For the material parameters we use, we found $`\eta =0.34`$ for the SC state and $`\eta =0.15`$ for the normal state. Using $`\eta =0.34`$ for the SC state, Brinckmann and Lee have explained the resonance peak and the incommensuration structure observed in neutron scatterings, and some of the present authors have quantitatively accounted for the doping dependences of the peak/dip/hump structure in the angle-resolved photoemission spectra and the tunneling spectra in a NS junction.

The above considerations can be carried over to the calculation of the thermodynamic potential. We consider the same set of ring diagrams in the linked cluster expansion of the thermodynamic potential as shown in Fig.2, and find it is given by

$$\mathrm{\Delta }\mathrm{\Omega }=\frac{1}{\beta }\underset{i\omega _n}{}\underset{q}{}_0^1𝑑\nu \frac{J\gamma _q\chi _0^+(𝐪,i\omega _n)}{1+\nu \eta J\gamma _q\chi _0^+(𝐪,i\omega _n)}$$
(8)
Here, we have used the renormalized strength $`\eta J`$ for the vertex. The parameter $`\nu `$ is used to keep track of the number of times that the potential appears in the perturbation expansion. Since $`\nu `$ enters in the same way as a coupling constant, the integration can be taken outside of the summation over $`i\omega _n`$. Performing the summation over $`i\omega _n`$, we get

$$\mathrm{\Delta }\mathrm{\Omega }=J\underset{q}{}_0^1𝑑\nu _0^{\mathrm{}}\frac{d\omega }{\pi }\mathrm{Im}\chi ^+(\nu ,𝐪,\omega )\gamma _q\mathrm{coth}(\frac{\beta \omega }{2})$$
(9)
with

$$\chi ^+(\nu ,𝐪,\omega )=\frac{\chi _0^+(𝐪,\omega )}{1+\nu \eta J\gamma _q\chi _0^+(𝐪,\omega )}$$
(10)
In obtaining Eq.(10), we have made use of the symmetry $`\mathrm{Im}\chi (𝐪,\omega )=\mathrm{Im}\chi (𝐪,\omega )`$.

The condensation energy $`H_c^2v/8\pi `$ per unit cell is equal to the difference of the thermodynamic potential(free energy) between the normal and the SC states. In our approach, it includes the contribution from the AF exchange correction $`\mathrm{\Delta }\mathrm{\Omega }^{}`$ and that from the mean-field Hamiltonian. The free energy at the mean-field level in the SC state can be easily calculated from Eq.(6) as,

$`F_0^S`$ $`=`$ $`{\displaystyle \frac{2}{\beta }}{\displaystyle \underset{k}{}}\mathrm{ln}(1+e^{\beta E_k})+{\displaystyle \underset{k}{}}(ϵ_kE_k)`$ (12)
$`+2NJ^{}(\mathrm{\Delta }_0^2+\chi _0^2)`$
with $`E_k=\sqrt{ϵ_k^2+\mathrm{\Delta }_k^2}`$. Its counterpart in the normal state $`F_0^N`$ can be obtained from Eq.(8) by setting $`\mathrm{\Delta }_0=0`$. To include the contribution from $`\chi `$, we note that the mean-field free energy already contains the Hartree-Fock contribution, i.e., the one-loop term in Fig.2. Therefore, we should substract it to avoid double counting and arrive at,

$`{\displaystyle \frac{H_c^2v}{8\pi }}=`$ (13)
$`{\displaystyle \frac{3J}{2}}{\displaystyle \underset{q}{}}{\displaystyle _0^1}𝑑\nu {\displaystyle _0^{\mathrm{}}}{\displaystyle \frac{d\omega }{\pi }}[\mathrm{Im}\chi ^N(\nu ,𝐪,\omega )\mathrm{Im}\chi ^S(\nu ,𝐪,\omega )]`$ (14)
$`\times \gamma _q\mathrm{coth}({\displaystyle \frac{\omega }{2T}}){\displaystyle \frac{3J}{2}}{\displaystyle \underset{q}{}}{\displaystyle _0^{\mathrm{}}}{\displaystyle \frac{d\omega }{\pi }}[\mathrm{Im}\chi _0^N(𝐪,\omega )\mathrm{Im}\chi _0^S(𝐪,\omega )]`$ (15)
$`\times \gamma _q\mathrm{coth}({\displaystyle \frac{\omega }{2T}})+F_0^NF_0^S.`$ (16)
where, we have abbreviated $`\chi ^+`$ as $`\chi `$. The first term of the right hand side of Eq.(9) comes from the AF fluctuation correction as shown in Fig.2, while the second term is the Hartree-Fock contribution which should be substracted from the first term. Finally, $`F_0^NF_0^S`$ is the free energy difference at the mean-field level.

Before performing the numerical calculation, we would like to point out that all the relevant energies such as the AF exchange energy, the kinetic energy and the energy for the formation of spin gap are contained in Eq.(16). Comparing the AF exchange energy in Eq.(16) to that in Eq.(2), we find two differences, a) there is an additional integral over $`\nu `$ in Eq.(16) and b) the temperature factors $`\mathrm{coth}(\omega /2T)`$ and $`1/[1\mathrm{exp}(\omega /T)]`$ are different. At zero temperature or if the main contribution to the integral over $`\omega `$ is dominated by those satisfying $`\omega /T1`$, the temperature factors are identical. When $`\nu =1`$, Eq.(10) is the same as the renormalized spin susceptibility Eq.(7). Therefore, if the integral over $`\nu `$ is dominated by that part around $`\nu 1`$ and the integral over $`\omega `$ is dominated by the spectral weight around $`\omega /T1`$, Eq.(16) can be reduced to Eq.(2). We will discuss this issue in detail in the first part of the following section.

## III NUMERICAL CALCULATIONS

In this section, we carry out the calculations of the condensation energy based on Eq.(16) and Eq.(2). The purpose of this section is twofold. First, we will show that when a resonance peak occurs in the spin excitation spectrum, the result obtained by Scalapino and White Eq.(2) is almost identical to the condensation energy calculated from the perturbation diagrams shown in Fig.2. Second, we will investigate the temperature and doping dependences of the condensation energy based on Eq.(16), and compare them with the experiments.

Numerical calculations are performed by dividing the Brillouin zone into $`128\times 128`$ lattices with $`t=2J`$ and $`t^{}=0.45t`$ as usual. A quasi-particle damping $`\mathrm{\Gamma }=0.02J`$ is used. We find that the results for the condensation energy (including its magnitudes) nearly does not change when $`\mathrm{\Gamma }`$ is changed around $`0.02J`$, though the spin susceptibility indeed follows the change of $`\mathrm{\Gamma }`$. The reason is because the change of $`\mathrm{\Gamma }`$ induces nearly identical changes for the spin susceptibility in both the normal and the SC states.

The temperature dependences of the difference in the AF exchange energy calculated with Eq.(2) are presented in Fig.3 for doping densities $`\delta `$=0.16, 0.18 and 0.20, respectively. The results for the first term of the right hand side of Eq.(16) with the same parameters are shown in Fig.4. In all cases, both results are only weakly $`T`$-dependent at low temperatures, but fall off rapidly as $`T`$ is increased towards $`T=0.08J`$. Comparing Figs.3 and 4, one can see that the overall temperature dependences for both cases are remarkably similar for the doping range investigated. Meanwhile, both have a similar magnitude, though the latter is slightly smaller than the former. Thus, we may conclude that the first term of Eq.(16) and Eq.(2) may describe the same physics. Therefore, it provides a thermodynamical extension to the work by Scalapino and White. As noted above, the most possible reason for the same temperature dependence is that the integral over $`\nu `$ is dominated by that part around $`\nu =1`$ and the integral over frequency is dominated by that part satisfying $`\omega /T1`$. To see the variation of the integrand in Eq.(16) with $`\nu `$, we show in Fig.5 the results for Im$`\chi ^+(\nu ,𝐐,\omega )`$, which is given by Eq.(10), at temperature $`T=0.005t`$ and doping $`\delta =0.16`$ for several values of $`\nu `$ (1, 0.9 and 0.5). When $`\nu =1`$, the integrand $`\chi ^+(\nu ,𝐪,\omega )`$ is identical to the spin susceptibility Eq.(7). The most remarkable change in the spin susceptibility across the SC transition is the emergence of the resonance peak at momentum $`𝐐=(\pi ,\pi )`$ below $`T_c`$, which is shown as the solid line in Fig.5. In the framework of the $`d`$-wave BCS theory, the origin of the neutron resonance peak has been attributed to a collective spin excitation mode , which corresponds to $`1+\eta J\gamma _Q\mathrm{Re}\chi _0^+(𝐐,\omega )=0`$ and Im$`\chi _0^+(𝐐,\omega )`$ approaches to zero. It is caused by the step-like rise of the imaginary part of the unperturbed spin susceptibility Im$`\chi _0`$ at its threshold as shown in the inset of Fig.5, where the solid line denotes its real part, the dashed line its imaginary part and the dotted line represents $`1/\nu \eta J\gamma _Q`$ with $`\nu =1`$. According to the Kramers-Kroenig relation, a logarithmic singularity in its real part Re$`\chi _0`$ occurs due to this step-like rise, which exhibits as a peak in the numerical calculations on finite lattices. This enhancement in Re$`\chi _0`$ shifts downward the position of the resonance mode, which is the cross between the dotted line and the solid line in the inset of Fig.5, into the SC gap where no damping for spin excitations is expected. When $`\nu `$ decreases, the dotted line in the inset of Fig.5 will rise, so the position of the cross will move to higher frequencies above the threshold where the dampings become nonzero. As $`\nu `$ decreases further, there is no cross anymore. However, if $`1+\eta J\gamma _Q\mathrm{Re}\chi _0(𝐐,\omega )`$ is small, a peak still occurs. But the intensity of the peak will decrease, such as the dashed line for $`\nu =0.9`$ in Fig.5. This may be regarded as the proximity effect of the resonance mode. As $`\nu `$ moves far away from 1, no peak exists and only a broad hump appears such as the dotted line in Fig.5 for $`\nu =0.5`$. From the figure, one can see that the integrated spectral weight over $`\omega `$ in the SC state decreases with the decrease of $`\nu `$. However, little change occurs in its normal state results (the extrapolated value). Therefore, it is the existence of the resonance peak that the integral over $`\nu `$ is dominated by that part around $`\nu =1`$. Because the resonance peak is around $`\omega =0.52J`$( 0.53$`J`$ and 0.52$`J`$ for $`\delta =0.18`$ and $`\delta =0.20`$, respectively), so the temperature factors $`\mathrm{coth}(\omega /T)`$ and $`1/[1\mathrm{exp}(\omega /T)]`$ are essentially equal to 1 even for the highest temperature $`T=0.08J`$ considered here. As for the momentum distribution, we wish to point out two facts: a) the resonance peak appears at $`𝐐=(\pi ,\pi )`$ and drops rapidly as $`𝐪`$ moves away from $`𝐐`$; b) the summation over $`q`$ is weighted by a factor $`\gamma _q=\mathrm{cos}(q_x)+\mathrm{cos}(q_y)`$ being maximum at $`𝐐`$. Obviously, both help to filter out the contributions around the resonance peak. Therefore, the very similar temperature dependence of the results calculated with Eq.(2) and with the first term of Eq.(16) suggests that a large part of the condensation energy due to the AF exchange interactions comes from the spectral weight around the resonance mode, as first argued by Demler and Zhang.

Because the integral over $`\nu `$ in Eq.(16) is dominated by the contributions around $`\nu =1`$ and only in this range of $`\nu `$ is the integrand Im$`\chi ^+(\nu ,𝐐,\omega )`$ approximately identical to the spin susceptibility Eq.(7), it leads the magnitudes of the condensation energy calculated with the first term of Eq.(16) to be smaller than those calculated with Eq.(2), as one can see from a comparison of Figs.3 and 4. In Fig.4, we also show the temperature dependences of the resonance peak intensities for doping $`\delta =`$0.16(squares), 0.18(circles) and 0.20(triangles). For all doping levels from the optimal to overdoped regimes, we find that the peak intensity and the condensation energy due to the AF exchange interactions follow the same temperature dependence. This result is in good agreement with the recent experiment on YBa2Cu3O6.93  which is an optimally doped system, though the result for the overdoped system waits for future experiment testing. This coincidence further supports the above conclusion that the contribution to the condensation energy from the AF exchange corrections arises mainly from spin excitations around the resonance peak.

The total condensation energy based on the renormalized RPA approach is plotted as a function of temperature in Fig.6 for different dopings $`\delta =0.16,0.18`$ and 0.20. From a comparison with Fig.3, we find that the results calculated here exhibit different temperature dependence, i.e., they decrease more rapidly with the rise of temperatures. This difference is due to the rapid variation with temperature of the contribution from the mean-field part. Meanwhile, their magnitudes are only about half of those calculated with Eq.(2). We note that the condensation energy for doping $`\delta =0.16`$(nearly optimally doped) at zero temperature is around $`0.023J`$, which is in reasonably agreement with the estimated value $`0.016J`$ based on the experimental data for the spin structure factor and the $`q`$-width of the resonance peak in YBa2Cu3O7  and also with the experimental data $`E_c=6`$K($`0.23J30`$K if $`J=1300`$K is taken) in the specific-heat measurement. Thus, our investigation based on the renormalized RPA approach to the $`tt^{}J`$ model gives a quantitative account for the condensation energy.

The doping dependence of the condensation energy calculated with Eq.(16) is shown in Fig.7. An obvious feature is that the condensation energy decreases as the doping density rises from the optimally doped to the overdoped regime. This is related to the doping dependence of the maximum SC gap $`\mathrm{\Delta }_0`$ at the mean-field level which shows the same trend, for example $`\mathrm{\Delta }_0`$=0.335J, 0.322J and 0.311J for $`\delta =0.16,0.18`$ and 0.2 at $`T=0.005J`$, respectively. Moreover, an approximate linear variation of the condensation energy with doping is found. This result is consistent with the experimental data determined from the heat capacity measurements. However, we note that the condensation energies shown in Fig.6 do not approach to zero when the temperature is as high as $`0.08J`$ which may correspond to 104K if we choose $`J=1300`$K. This is the fault of the slave-boson mean-field calculation which over-estimates $`T_c`$ so that $`\mathrm{\Delta }_0`$ does not approach to zero even at $`T=0.08J`$. Therefore, even though our approach gives good quantitative results near zero temperature, it may break down at temperatures near true transition temperatures.

## IV CONCLUDING REMARKS

In summary, based on the slave-boson theory of the two-dimensional $`tt^{}J`$ model, we calculate the superconducting condensation energy for optimally doped and overdoped high-$`T_c`$ cuprates at finite temperatures using a renormalized random phase approximation. It is composed of the contributions from the mean-field Hamiltonian and that from the antiferromagnetic spin fluctuation.

We show that the contribution from the antiferromagnetic spin fluctuation is essentially the same as the argument by Scalapino and White, which relates the condensation energy to the difference of the antiferromagnetic exchange energy between the normal state and the superconducting state, due to the existance of the neutron resonance mode. However, both the temperature dependence and the magnitudes of the total condensation energy after including the mean-field part are different from those due to the AF exchange energy. Our result for the total condensation energy gives a quantitative account for the observed condensation energy near zero temperature. We also calculate the doping dependence of the condensation energy and find it in agreement with experiments.

Taking into account of our previous studies of the relationship of the resonance neutron peak to the angle-resolved photoemission spectra  and the tunneling spectra, we believe that the resonant spin collective mode plays an important role in determining many physical properties in the superconducting state of high-$`T_c`$ cuprates. A particular interesting conclusion from this work is that the renormalized random phase approximation seems to have already captured the main features of the resonance mode. However, it is introduced phenomenologically and its derivation is left for future work.

## ACKNOWLEDGMENTS

JXL was supported by the National Nature Science Foundation of China. CYM thanks the support from NSC of Taiwan under Grant No. NSC 89-2112-M-007-091 and TKL for NSC 89-2112-M-001-103. CDG thanks the support by the Ministry of Science and Technology of China(NKBRSF-G19990646).

## FIGURE CAPTIONS

Fig.1 Feynman diagram for the renormalized spin susceptibility calculated in the random phase approximation(RPA). The solid lines represent the fermionic Green’s functions and the dashed line the antiferromagnetic coupling constant $`J\gamma _q`$.

Fig.2 Feynman diagram for the contribution to the thermodynamic potential from antiferromagnetic fluctuations. The solid lines represent the fermionic Green’s functions and the dashed line the antiferromagnetic coupling constant $`J\gamma _q`$.

Fig.3 Temperature dependences of the difference in antiferromagnetic exchange energy between the normal and the superconducting states Eq.(2) for hole concentrations $`\delta =`$0.16, 0.18 and 0.20, which is argued by Scalapino and white  to be the condensation energy.

Fig.4 Temperature dependences of the condensation energy due to the antiferromagnetic fluctuation correction \[the first term in the right hand of Eq.(9)\] for hole concentrations $`\delta =`$0.16, 0.18 and 0.20. The solid symbols denote the intensities of the resonance peakes at different temperatures (squares for $`\delta =0.16`$, circles for $`\delta =0.18`$ and triangles for $`\delta =0.20`$). The intensity is scaled in order to compare it with the condensation energy.

Fig.5 Frequency dependences of Im$`\chi ^+(\nu ,𝐐,\omega )`$ at temperature $`T=0.005J`$, $`𝐐=(\pi ,\pi )`$ and doping $`\delta =0.16`$ for $`\nu =`$1, 0.9 and 0.5, respectively. Inset shows the bare spin susceptibility Im$`\chi _0(𝐐,\omega )`$. The solid line denotes its real part and the dashed line its imaginary part. The dotted line represents $`1/\eta J\gamma _𝐐`$(see text). Similar results are obtained for other dopings expected that the positions of the resonance peak are shifted, so are not shown here.

Fig.6 Temperature dependences of the total condensation energy calculated with Eq.(16) for hole concentrations $`\delta =`$0.16, 0.18 and 0.20. They show more rapid drops in comparison to Fig.3.

Fig.7 Doping dependence of the total condensation energy calculated with Eq.(16) at $`T=0.005J`$.