text:
# Rotating models for evolved low-mass stars

## 1. Abundance anomalies in RGB stars due to in situ processes

### 1.1. Observational data

According to the classical stellar evolution theory111By this we refer to the modeling of non-rotating, non-magnetic stars, in which convection is the only mixing process considered., the only opportunity for low mass stars (LMS) to modify their surface abundances happens on their way to the RGB when they do undergo the so-called first dredge-up (1DUP; Iben 1965). During this event their expanding convective envelope deepens in mass, leading to the dilution of the surface pristine material within regions that have undergone partial nuclear processing on the earlier main sequence. Qualitatively, this leads to the decrease of the surface abundances of the fragile LiBeB elements and of 12C, while those of 3He, 13C and 14N increase. Due to too low temperatures inside main sequence LMS, the abundances of O and heavier elements remain unchanged subsequent to the 1DUP. Quantitatively, these abundance variations depend on the stellar mass and metallicity (e.g., Sweigart, Greggio & Renzini, 1989; Charbonnel 1994; Boothroyd & Sackmann 1999). After the 1DUP, the convective envelope withdraws while the hydrogen burning shell (HBS) moves outwards in mass. No more variations of the surface abundances are then expected until the star reaches the asymptotic giant branch.

The 1DUP predictions agree with the observations on the lower part of the RGB222One has of course to take into account possible variations of the surface Li abundance occurring already on the main sequence. This discussion is however out of the scope of the present paper (see Charbonnel, Deliyannis & Pinsonneault 2000).. However observational evidences have accumulated, that we list below, of a second and distinct mixing episode that occurs in low mass stars after the end of the 1DUP, and more precisely at the RGB bump.

The determination of the carbon isotopic ratio (12C/13C, hereafter cir) for RGB stars in open clusters with various turnoff masses (Gilroy 1989) provided the first pertinent clue on this process. It was indeed shown that bright RGB stars with initial masses lower than $``$ 2-2.5M exhibit cir considerably lower than predicted by the 1DUP. Thanks to data in stars sampling the RGB of M67 (Gilroy & Brown 1991), it clearly appeared that observations deviated from classical predictions just at the so-called RGB bump (Charbonnel 1994). Field and globular cluster (GC) stars behave similarly333The literature on the various abundance anomalies within GCs is so large that we chose to quote here only the most recent papers to illustrate our discussion. See Palacios (2002) for complete references.. In the former case this could be established thanks to the determination of the cir in large samples of stars for which Hipparcos parallaxes allowed the precise determination of their evolutionary status (Charbonnel, Brown & Wallerstein 1998; Gratton et al. 2000). We knew for a long time that the brightest RGB stars in GCs presented cir close to the equilibrium value. But it is only this year that the region around the bump could be probed for two GCs : In NGC 6528 and M4, the cir drops below the 1DUP predictions just at the RGB bump (Shetrone 2002). The disagreement between observations and classical predictions is stronger for lower stellar mass and metallicity.

The extra-mixing process affects the surface abundances of other chemical elements : Li also decreases at the RGB bump, both in the field and in GCs (Pilachowski, Sneden & Booth 1993; Grundahl et al. 2002). In field stars, C decreases while N increases for RGB stars brighter than the bump (Gratton et al. 2000) confirming the CN processing of the stellar envelope. The lowering of the C abundance along the RGB is also seen in GCs (e.g., Bellman et al. 2001 and references therein), though in this case the picture is more confused because of a probable non negligible dispersion of the initial \[C/Fe\].

An in situ mechanism has also frequently been invoked to explain the abundance anomalies of heavier elements in GC stars, and in particular the O-Na anticorrelation. This pattern has been observed in the brightest GC RGB stars for a long time (see references in Ivans et al. 1999 and Ramirez & Cohen 2002). However it is only thanks to 8-10m-class telescopes that O and Na abundances could be determined for less evolved stars, and in particular for turnoff stars in a couple of GCs (Gratton et al. 2001; Thévenin et al. 2001; Ramirez & Cohen 2002). There, the O-Na anticorrelation extends to the main sequence. This result is crucial. Indeed, the ON- and NeNa-cycles do not occur in main sequence LMS, as the involved reactions require too high temperatures which are only reached when LMS are on the RGB. Thus the fact that the O-Na anticorrelation already exists on the main sequence clearly proves that it is not produced by in situ processes, but by external causes, the discussion of which is out of the scope of this paper (see Charbonnel 2002b).

Recently we have searched for an eventual evolution of the O and Na abundances with luminosity along the RGB for field stars. We gathered from the literature a large sample of field (625) stars with Hipparcos parallaxes over a large range of metallicity (\[Fe/H\] between -2.5 and 0) for which we homogeneously redetermined the abundances of a large number of elements. No evolutionary effect could be found neither for O nor Na (Palacios et al. 2002a; Charbonnel et al. 2003), in agreement with the most recent GC data mentioned above.

Last but not least, the recent determination of the oxygen isotopic ratios in a couple of RGB stars with low cir (Balachandran & Carr 2002) confirmed this result. In these objects indeed, both the 16O/17O and 16O/18O ratios appear to be high, in agreement with extensive CN-processing but no dredge-up of ON-cycle material in RGB stars (see the arrow in Fig.1).

### 1.2. Origin of the extra-mixing and its inhibition by $`\mu `$-gradients

The observations summarized previously provide definitive clues on a second and distinct mixing episode that occurs in LMS when they reach the RGB bump. This process appears to be universal : it affects more than 95$`\%`$ of the LMS (Charbonnel & do Nascimento 1998), whether they belong to the field, to open or globular clusters. Its signatures in terms of abundance anomalies are clear: the Li and the 12C abundances as well as the 12C/13C ratio drop, while the 14N abundance and the 16O/18O ratio increase. On the other hand 16O, 17O, Na and heavier elements remain unaffected. As we shall see in §3, this sequence may start with a very brief episode of Li enrichment.

These coherent data clearly indicate that during the extra-mixing episode at the RGB bump, the stellar convective envelope is connected with the radiative regions where CN-burning occurs, but not with the ON- and NeNa-processed layers (see Fig.1). As we shall discuss later on, the important increase of the molecular weight (or $`\mu `$) gradients when one approaches the HBS certainly shields these deeper regions from the extra-mixing.

$`\mu `$-gradients are also clearly responsible for the fact that the extra-mixing starts to be efficient only around the bump. Indeed during the 1DUP a $`\mu `$-discontinuity is built at the region of the deepest penetration of the convective envelope. This $`\mu `$-barrier then inhibits any mixing between the convective envelope and the HBS. At the RGB bump, the external regions of the HBS pass through the $`\mu `$-discontinuity. Due to the composition changes in the burning region, the stellar structure has to readjust, and the stellar luminosity then slightly decreases before increasing again. LMS stars spend a non negligible part of their RGB lifetime in this region; this explains why, for GCs for example, the bump clearly appears as a peak in the differential luminosity function or as a change in the slope of the cumulative luminosity function (e.g., Zoccali et al. 1999). After this evolutionary point, the $`\mu `$-gradients between the base of the convective envelope and the HBS are much smoother, allowing the occurrence of some extra-mixing in this radiative region (Sweigart & Mengel 1979; Charbonnel 1994, 1995; Charbonnel et al. 1998).

All these observational facts represent strong clues on the origin of the extra-mixing process which is certainly rotationally-induced.

## 2. Rotation-induced mixing and abundance anomalies in RGB stars444We do not discuss here the numerous parametric models that have been constructed to reproduce the abundance anomalies in RGB stars. We focus only on the physical models where the extra-mixing is assumed to be induced by rotation.

### 2.1. From meridional circulation …

Sweigart & Mengel (1979, hereafter SM79) investigated the possibility that meridional circulation might lead to the mixing of CNO-processed material in RGB stars. Though the physics of the rotation-induced mixing invoked was incomplete, this pioneering work has magnificently settled the basis of a complex problem which is not definitively solved yet.

SM79 discussed in great details the problem of $`\mu `$-gradients which were known to tend to inhibit meridional circulation (Mestel 1953, 1957). For the mixing to be efficient, they underlined the necessity for the radiative zone separating the convective envelope from the HBS not to have a significant gradient in $`\mu `$. As we said in §1.2, this condition is fulfilled only after the RGB bump, and, above this evolutionary point, only in the most external part of the HBS. This last point can be easily understood by looking at the abundance profiles inside a RGB star at the bump (Fig.1). When one moves inward the star from the base of the convective envelope ($`\delta `$M=1) toward the HBS (XH=0 at $`\delta `$M=0), one encounters first a shell in which C is burned into N, then a C-depleted region and still deeper a shell in which O burns into N. As underlined by SM79, the H abundance, and thus $`\mu `$, remain essentially constant across the C shell : the negligible $`\mu `$-gradients can not chocke the circulation currents in this region. However, the $`\mu `$-gradients become appreciable when one approaches the O shell due to the greater amount of H burning close to ON equilibrium. There, it is no longer clear whether the circulation currents can overcome the $`\mu `$-gradients, especially for the more metal-rich stars in which the C and O shells are shifted inward toward the HBS. Theoretical estimates of the $`\mu `$-gradients which are required to stop the meridional circulation were too uncertain for definitive conclusions to be put forward, though the metallicity dependance was soundly addressed.

The other crucial point made by SM79 concerned the importance of the angular momentum distribution within the deep convective envelope of the RGB star on the resulting CNO processing. Indeed, the angular velocity of a radiative layer near the HBS depends on how much angular momentum is left behind as the convective envelope retreats outward. SM79 investigated the two extreme cases, namely a constant specific angular momentum within the convective envelope, and a convective envelope rotating as a solid body. Substantial CNO processing of the envelope could be obtained with plausible main sequence angular velocity only if the inner part of the convective envelope was allowed to depart from solid body rotation. As we shall see in §2.3, our lack of knowledge of the distribution of angular momentum within the RGB convective envelope still remains nowadays one of the weakest points of the global picture.

### 2.2. … over a more complete description of rotation-induced mixing

Rotational transport processes can not be simply reduced to meridional circulation. Once established indeed, this large scale circulation generates in turn advection of momentum and thus favours the development of various hydrodynamical instabilities666We refer to the review by Suzanne Talon in these proceedings for a complete description of the transport processes inside rotating stars. Zahn (1992) proposed a description of the interaction between meridional circulation and shear turbulence, pushing forward the idea of chocking the meridional circulation by $`\mu `$-gradients. Following these developments, Charbonnel (1995, hereafter C95) re-investigated the influence of such a process in RGB stars. This framework is appealing because it takes advantage of some particularities of the non-homologous RGB evolution. In particular, some mixing is expected to take place wherever the rotation profile presents steep vertical gradients, as well as near nuclear burning shells. Moreover due to the stabilizing effect of the composition gradients, the mixing is expected to change the surface abundances on the RGB only after the bump.

Using a simplified version of Zahn’s description, C95 showed that the rotation-induced mixing can indeed account for the observed behavior of carbon isotopic ratios and for the Li abundances in Population II low mass giants. Simultaneously, when this extra-mixing begins to act, 3He is rapidly transported down to the regions where it burns by the 3He($`\alpha ,\gamma )^7`$Be reaction. This leads to a decrease of the surface value of 3He/H and of the final 3He yields from LMS, solving the long-standing problem of the galactic evolution of 3He (e.g. Charbonnel 2002a). In these exploratory computations however, the transport of angular momentum by the hydrodynamical processes was not treated self-consistently, although it is of utmost importance in the understanding of the rotation-induced mixing.

### 2.3.  … to a self-consistent treatment of the transport of angular momentum and of the chemicals

Maeder & Meynet (2000) nicely review the recent developments on the rotation-induced mixing in stars. They clearly state the fact that “the study of rotational mixing requires both an understanding of angular momentum evolution of stars and an understanding of the degree of mixing for a given angular momentum distribution. Angular momentum transport is the crucial ingredient for determining the extent of rotationally induced mixing”. Let us now discuss the results obtained for RGB stars when the transport of angular momentum and of the chemicals as described by Zahn (1992), and latter on by Talon et al. (1997), Maeder & Zahn (1998), and Palacios et al. (2002b) is taken into account.

Two independant studies have been carried out using basically the same physical description of rotational mixing : Denissenkov & Tout (2000, hereafter DT00) and Palacios et al. (2003, hereafter PCTF03). They differ on some details of the physical assumptions like the choice of the shear instability criteria and of the expression of the diffusion coefficient associated with vertical turbulence. These differences actually appear to be minor. Both studies obtain indeed very similar total diffusion coefficients for Pop II RGB stars at the bump, the vertical diffusion coefficient (turbulence) dominating over the effective diffusion coefficient (meridional circulation).

More importantly, however, a drastic assumption differentiates the work by DT00 from that of PCTF03 : DT00 indeed compute the transport of angular momentum and of the chemicals in a post-processing way; namely, they use the internal structure of a few standard RGB models to estimate the transport coefficients and the subsequent expected changes in the surface abundances. In this approach, there is no feedback of the mixing on the stellar structure and evolution which are computed independently in the standard way.

On the other hand, PCTF03 do treat the transport of both angular momentum and of the chemicals inside their stellar evolution code, thereby following the “non-standard” evolution of the star from the zero age main sequence on. As a result in this case, the mixing has a feedback on the star since its early evolution. On the RGB in particular, the mixing distorts the stationary profiles of the chemicals deep inside the star even if the surface abundances are not yet affected; it alters for example the positions of the C and O depleted regions. In turn, the mixed star is allowed to react to the mixing that it undergoes, thereby influencing the mixing itself.

This difference has a strong influence on the final predictions. In DT00 indeed, the transport is applied on standard profiles, and surface variations of O and Na are expected in RGB stars. In PCTF03 however, the mixing is applied on self-consistently distorted profiles; as a result, only variations of the C isotopes and of N are predicted at the surface of the star. These latter predictions are in better agreement with the observations described in §1.

### 2.4. Clues from horizontal branch stars

Let us note an important point. In all these studies (SM79, DT00, PCTF03), notable changes of the surface abundances could be reached only under the assumption of constant specific angular momentum within the convective envelope (see §2.1). Indeed in the case of solid body rotation prohibitively high values of the main sequence surface rotation velocities are required to produce enough mixing. The same result has been obtained by Chanamé, Pinsonneault & Terndrup (these proceedings) in their simplified rotating Pop I RGB model.

Unfortunately, the distribution of angular momentum within the very deep convective envelope of an RGB star is far from being understood. More than 20 years after SM79, one is still reduced to use the abundance anomalies on the RGB as evidences for some departure from solid-body rotation (see also Kumar, Narayan & Loeb 1995). This assumption is actually sustained by the study of horizontal branch stars which provides some intriguing clues about angular momentum evolution on the RGB.

Peterson (1983) first discovered that some blue horizontal branch stars are rapid rotators. Pinsonneault et al. (1991) noted that the combination of RGB mass loss, high horizontal branch rotation rate, and low main sequence rotation required strong differential rotation with depth in giants. If the convection zone of RGB stars had solid body rotation, differential rotation with depth in their MS precursors was also required. Behr et al. (2000) found a break in the rotational properties of horizontal branch stars, in the sense that very blue horizontal branch stars both exhibited the surface signature of atomic diffusion and rotated more slowly than slightly cooler stars. Sills & Pinsonneault (2000) interpreted this as an evidence that mean molecular weight gradients caused by atomic diffusion inhibit angular momentum transport in hot horizontal branch stars. This is an independent test of the impact of composition gradients in a different evolutionary phase (for discussions of the effects on the main sequence see for example Vauclair 1999; Théado & Vauclair 2001; Palacios et al. 2002b).

In addition, Sills & Pinsonneault (2000) found that uniform rotation at the main sequence turnoff was only compatible with rapid horizontal branch rotation under the following conditions: (1) turnoff rotation of order 4 km/s rather than the 1 km/s inferred from an extrapolation of the Population I angular momentum loss law to Population II stars; (2) constant specific angular momentum in the convective envelopes of giants; (3) strong differential rotation with depth in the radiative cores of giants.

All of these are different from the expectations from main sequence angular momentum evolution models, and they are an indication that further theoretical work is needed in physical models of RGB rotational mixing. It is encouraging, however, that all of the above properties favor more vigorous rotational mixing on the RGB than would be expected from the opposite conclusions.

## 3. The Li flash

A few ($`1\%`$) RGB stars present Li-overabundances (see also Drake et al. and Konstantinova-Antova in these proceedings). The fact that these so-called super Li-rich giants are all located at the RGB bump lead Charbonnel & Balachandran (2000, hereafter CB00) to conclude that the Li-rich phase was a precursor to the extra-mixing process described previously. This brought very tight constraints on the underlying physics.

Denissenkov & Weiss (2000, hereafter DW00) investigated the effect of rotation-induced mixing on the surface lithium abundance using the transport coefficients DR derived by DT00 (see §2.3). They found that the corresponding values of DR ($`10^8`$ to 1010cm2sec-1) were too low to lead to lithium enrichment and invoked the engulfing of a planet in order to trigger and dope the transport process. While this combined scenario is rather attractive, it is harmed by the fact that it can happen at any time on the RGB while the super-Li rich giants with solar metallicity discovered up to now do stand by the bump (CB00). However what was not investigated by DT00 and DW00 was the structural response of the star to the extra-mixing. Indeed, as already discussed in §2.3, their computations of the transport coefficients and of the abundance variations were done in a post-processing approach where standard stellar structures were used as background models.

Palacios, Charbonnel & Forestini (2001) proposed that the structural response to the mixing could actually be the cause of the increase of DR which is necessary to produce super-Li rich giants. The mixing sequence is the following. At the RGB bump the external wing of the 7Be peak crosses the molecular weight discontinuity before the 13C peak does, and is then connected with the convective envelope by the extra-mixing process. 7Be produced via 3He($`\alpha ,\gamma )^7`$Be starts to diffuse outwards. However, due to the relatively low DR given above, the transported 7Be decays in regions where 7Li is rapidly destroyed by proton capture. A lithium burning shell appears. 7Li(p,$`\alpha )\alpha `$ becomes the dominant reaction leading to an increase of the local temperature and of both the local and total luminosities. Palacios et al. suggested that the meridional circulation and the corresponding transport coefficient then increase due to the $`ϵ_{\mathrm{nuc}}(^7`$Li+p) burst, allowing for lithium enrichment of the convective envelope. A lithium flash terminates this phase777Our scenario is different from the previous works which explored the possible occurrence of shell flashes on the RGB and which considered only the reactions of the CNO cycle (Bolton & Eggleton 1973; Dearborn et al. 1975; Von Rudloff et al. 1988). It also differs from the model developed by Fujimoto et al. (1999) to explain Mg and Al anomalies in globular cluster red giants. These authors indeed proposed that a flash is triggered off deeper in the HBS due to the inward mixing of H down into the degenerate He core.. As the convective instability develops around the Li-burning shell, it erases the molecular weight gradient and the mixing is free to proceed deeper. Both the lithium abundance and the carbon isotopic ratio then drop at the stellar surface. When the flash is quenched, the external convective envelope deepens again. We suggest that this creates a new $`\mu `$-barrier, similar to that built up during the 1DUP, which then inhibits again the extra-mixing during the subsequent evolution.

During the whole sequence described above the temporary increase of the stellar luminosity causes an enhanced mass loss rate which naturally accounts for the dust shell suggested by the far-infrared color excesses measured for some lithium-rich objects from IRAS fluxes (de la Reza et al. 1996, 1997). Let us note that the contribution of these stars to the lithium enrichment of the Galaxy should be very modest.

## 4. Conclusions

During the last three decades, an incredible amount of work has been devoted to the understanding of the chemical anomalies exhibited by evolved LMS. On the observational side, crucial breakthroughs were made possible recently thanks to the advent of 8-10m-class telescopes. The signatures of the extra-mixing process that occurs at the RGB bump are now very clear.

On the other hand, recent theoretical models include a sophisticated description of the mixing processes induced by rotation. Those which consistently couple the transport of angular momentum and of the chemicals with the structural evolution of the star predict surface abundance variations on the RGB in agreement with the observational data. However, a couple of assumptions remain to be tested. One of the main uncertainties concerns the distribution of angular momentum within the deep convective envelope. A final answer to this problem certainly necessitates 3D hydrodynamical simulations. Another important point which remains to be investigated is the reaction of the meridional circulation and of the various instabilities to a major and local release of nuclear energy.

Rotation-induced mixing processes modify all the stellar outputs, and in particular the final yields from LMS (i.e., 3He is not over-produced anymore by rotating LMS). Their impacts on the stellar lifetimes and on the further evolution (luminosity of the RGB tip, morphology of the horizontal branch, AGB phase, …) have to be investigated now in a systematic way. This is of crucial importance if one wants to correctly understand the role of LMS in the evolution (chemical and spectro-photometric) of stellar clusters and of galaxies in general.

#### Acknowledgments.

We thank the french Programme National de Physique Stellaire and Programme National Galaxies for their support on this work. AP aknowledges financial support from the IAU.

## References

Balachandran, S., Carr, J. 2002, in: C.Charbonnel, D.Schaerer, & G.Meynet (eds.), CNO in the Universe, ASP Conference Series, in press

Behr, B.B., Cohen, J.G., McCarthy, J.K. 2000, ApJ, 531, L37

Bellman, S., et al., 2001, PASP, 113, 326

Bolton, A.J.C., Eggleton, P.P., 1973, A&A, 24, 429

Boothroyd, A.I., Sackmann, I.J., 1999, ApJ, 510, 232

Charbonnel, C., 1994, A&A, 282, 811

Charbonnel, C., 1995, ApJ, 543, L41

Charbonnel, C., 2002a, Nature, 415, 27

Charbonnel, C., 2002b, in: C.Charbonnel, D.Schaerer, & G.Meynet (eds.), CNO in the Universe, ASP Conference Series, in press

Charbonnel, C., Balachandran, S., 2000, A&A, 359, 563, CB00

Charbonnel, C., Brown, J.A., Wallerstein, G., 1998, A&A, 332, 204

Charbonnel, C., do Nascimento, J., 1998, A&A, 336, 915

Charbonnel, C., Deliyannis, C., Pinsonneault, M., 2000, in: Da Silva et al. (eds), The Light Elements and their Evolution, Proc. IAU Symp. 198, 87

Charbonnel, C., Palacios, A., Thévenin, F., Bolmont, J., 2003, in preparation

Dearborn, D.D., Bolton, A.J.C., Eggleton, P.P., 1975, MNRAS, 170, 7

de la Reza, R., Drake, N.A., da Silva, L., 1996, ApJ, 456, L115

de la Reza, R., Drake, N.A., da Silva, L., Torres, C.A.O., Martin, E.L., 1997, ApJ, 482, L77

Denissenkov, P.A., Tout, C.A., 2000, MNRAS, 316, 395

Denissenkov, P.A., Weiss, A., 2000, A&A, 358, L49, DW00

Fujimoto, M.Y., Aikawa, M., Kato, K., 1999, ApJ, 519, 733

Gilroy, K.K., 1989, ApJ, 347, 835

Gilroy, K.K., Brown, J., 1991, ApJ, 371, 578

Gratton, R.G., Bonifacio, P., Bragaglia, A., et al., 2001, A&A, 369, 87

Gratton, R.G., Sneden, C., Carreta, E., Bragaglia, A., 2000, A&A, 354, 169

Grundahl, F., Briley, M., Nissen, P.E., Feltzing, S., 2002, A&A, 336, 915

Iben, I. Jr, 1965, ApJ, 142, 1447

Ivans, I.I., Sneden, C., Kraft, R.P., et al., 1999, AJ, 118, 1273

Kumar, P., Narayan, R., Loeb, A., 1995, ApJ, 453, 480

Maeder, A., Meynet, G., 2000, ARA&A, 38, 143

Maeder, A., Zahn, J.P., 1998, A&A, 334, 1000

Mestel, L., 1953, MNRAS, 113, 716

Mestel, L., 1957, ApJ, 126, 550

Palacios, A., 2002, PhD thesis, UPS Toulouse III

Palacios, A., Charbonnel, C., Bolmont, J., Thévenin, F., 2002a, Ap&SS, 281, 213

Palacios, A., Charbonnel, C., Forestini, M., 2001, A&A, 375, L9

Palacios, A., Talon, S., Charbonnel,C., Forestini, M., 2002b, A&A, in press

Palacios, A., Charbonnel, C., Talon, S., Forestini, M., 2003, A&A, in preparation

Peterson, R.C., 1983, ApJ, 275, 737

Pilachowski, C.A., Sneden, C., Booth, J., 1993, ApJ, 407, 699

Pinsonneault, M.H., Deliyannis, C.P., Demarque, P., 1991, ApJ, 367, 239

Ramirez, S.V., Cohen, J.G., 2002, AJ, 123, 3277

Sills, A., Pinsonneault, M.H., 2000, ApJ, 544, 944

Shetrone, M., 2002, in: C.Charbonnel, D.Schaerer, & G.Meynet (eds.), CNO in the Universe, ASP Conference Series, in press

Sweigart, A.V., Greggio, L., Renzini, A., 1989, ApJ, 69, 911

Sweigart, A.V., Mengel, J.G. 1979, ApJ, 229, 624, SM79

Talon, S., Zahn, J.P., Maeder, A., Meynet, G., 1997, A&A, 322, 209

Thévenin, F., Charbonnel, C., de Freitas Pacheco, J.A., et al., 2001, A&A, 373, 905

Vauclair, S., 1999, A&A, 351, 973

Vauclair, S., Théado, S., 2001, A&A, 375, 70

Von Rudloff, I.R., VandenBerg, D.A., Hartwick, F.D.A., 1988, ApJ, 324, 840

Zahn, J.P., 1992, A&A, 265, 115

Zoccali, M., Cassisi, S., Piotto, G., Bono, G., Salaris, M., 1999, ApJ, 518, L49


format: text

text:
# Affine Maximal Hypersurfaces

## 1. Introduction

The concept of affine maximal surface in affine geometry corresponds to that of minimal surface in Euclidean geometry. The affine Bernstein problem and affine Plateau problem, as proposed in , are two fundamental problems for affine maximal surfaces. We shall describe some recent advances, mostly obtained by Neil Trudinger and myself \[21-24\], on these two problems.

Given an immersed hypersurface $`^{n+1}`$, one defines the affine metric (also called the Berwald-Blaschke metric) by $`g=|K|^{1/(n+2)}II`$, where $`K`$ is the Gauss curvature, $`II`$ is the second fundamental form of $``$. In order that the metric is positive definite, the hypersurface will always be assumed to be locally uniformly convex, namely it has positive principal curvatures. From the affine metric one has the affine area functional,

$$A()=_{}K^{1/(n+2)},$$
$`1.1`$
which can also be written as

$$A(u)=_\mathrm{\Omega }[\text{det}D^2u]^{1/(n+2)}$$
$`1.2`$
if $``$ is given as the graph of a convex function $`u`$ over a domain $`\mathrm{\Omega }^n`$. The affine metric and affine surface area are invariant under unimodular affine transformations.

A locally uniformly convex hypersurface is called affine maximal if it is stationary for the functional $`A`$ under interior convex perturbation. A convex function is called an affine maximal function if its graph is affine maximal. Traditionally such hypersurfaces were called affine minimal . Calabi suggested using the terminology affine maximal as the second variation of the affine area functional is negative . If the hypersurface is a graph of convex function $`u`$, then $`u`$ satisfies the affine maximal surface equation (the Euler-Lagrange equation of the functional $`A`$),

$$L[u]:=U^{ij}w_{ij}=0,$$
$`1.3`$
where $`[U^{ij}]`$ is the cofactor of the Hessian matrix $`D^2u`$,

$$w=[\text{det}D^2u]^{(n+1)/(n+2)},$$
$`1.4`$
and the subscripts $`i,j`$ denote partial derivatives with respect to the variables $`x_i,x_j`$. Note that for any given $`i`$ or $`j`$, $`U^{ij}`$, as a vector field in $`\mathrm{\Omega }`$, is divergence free. The equation (1.3) is a nonlinear fourth order partial differential equation, which can also be written in the short form

$$\mathrm{\Delta }_gh=0,$$
$`1.5`$
where $`h=(\text{det}D^2u)^{1/(n+2)}`$, and $`\mathrm{\Delta }_g`$ denotes the Laplace-Beltrami operator with respect to $`g`$.

The quantity

$$H_A()=\frac{1}{n+1}L(u)$$

is called the affine mean curvature of $``$, and is also invariant under unimodular affine transformations. In particular it is invariant if one rotates the coordinates or adds a linear function to $`u`$. The affine mean curvature of the unit sphere is $`n`$.

The affine Bernstein problem concerns the uniqueness of entire convex solutions to the affine maximal surface equation, and asks whether an entire convex solution of (1.3) is a quadratic polynomial. The Chern conjecture  asserts this is true in dimension two. Geometrically, and more generally, it can be stated as that a Euclidean complete, affine maximal, locally uniformly convex surface in 3-space must be an elliptic paraboloid. Calabi proved the assertion assuming in addition that the surface is affine complete , see also . A problem raised by Calabi, called the Calabi conjecture in , is whether affine completeness alone is enough for the Bernstein theorem. The Chern conjecture was proved true in  (see Theorem 3.1 below). The Calabi conjecture was resolved in , as a byproduct of our fundamental result that affine completeness implies Euclidean completeness for locally uniformly convex hypersurfaces of dimensions larger than one (Theorem 3.2). See also  for a different proof of the Calabi conjecture.

The affine Plateau problem deals with the existence and regularity of affine maximal hypersurfaces with prescribed boundary of which the normal bundles on the boundary coincide with that of a given locally uniformly convex hypersurface. The affine Plateau problem, which had not been studied before, is more complicated when compared with the affine Bernstein problem in 3-space. The first boundary value problem, namely prescribing the solution and its gradient on the boundary, is a special case of the affine Plateau problem. We need to impose two boundary conditions as the affine maximal surface equation is a fourth order equation. We will formulate the Plateau problem as a variational maximization problem and prove the existence and regularity of maximizers to the problem in 3-space  (Theorem 5.1). For the existence we need a uniform cone property of locally convex hypersurfaces, proved in , which also led us to the proof of the conjecture by Spruck in  (Theorem 4.1), concerning the existence of locally convex hypersurfaces of constant Gauss curvature.

Equation (1.3) can be decomposed as a system of two second order partial differential equations, one of which is a linearized Monge-Ampère equation and the other is a Monge-Ampère equation, see (2.6) and (2.7) below. This formulation enables us to establish the regularity for equation (1.3) (Theorem 2.1), using the regularity theory for Monge-Ampère type equations . A crucial assumption in Theorem 2.1 is the strict convexity of solutions, which is the key issue for both the affine Bernstein and affine Plateau problems. We succeeded in proving the necessary convexity estimates only in dimension two.

## 2. A priori estimates

Instead of the homogeneous equation (1.3), we consider here the non- homogeneous (prescribed affine mean curvature) equation

$$L(u)=f\text{in}\mathrm{\Omega },$$
$`2.1`$
where $`f`$ is a bounded measurable function, and $`\mathrm{\Omega }`$ is a normalized convex domain in $`^n`$. A convex domain is called normalized if its minimum ellipsoid, that is the ellipsoid with minimum volume among all ellipsoids containing the domain, is a unit ball.

Let $`u`$ be a smooth, locally uniformly convex solution of (2.1) which vanishes on $`\mathrm{\Omega }`$. First we need positive upper and lower bounds for the determinant $`\text{det}D^2u`$. For the upper bound we have, by constructing appropriate auxiliary function, for any subdomain $`\mathrm{\Omega }^{}\mathrm{\Omega }`$, the estimate

$$\underset{x\mathrm{\Omega }^{}}{sup}\text{det}D^2u(x)C,$$
$`2.2`$
where $`C`$ depends only on $`n`$, $`\text{dist}(\mathrm{\Omega }^{},\mathrm{\Omega })`$, $`sup_\mathrm{\Omega }|Du|`$, $`sup_\mathrm{\Omega }f`$, and $`sup_\mathrm{\Omega }|u|`$.

For the lower bound we need a key assumption, namely a control on the strict convexity of solutions, which can be measured by introducing the modulus of convexity. Let $`v`$ be a convex function in $`\mathrm{\Omega }`$. For any $`y\mathrm{\Omega }`$, $`h>0`$, denote

$$S_{h,v}(y)=\{x\mathrm{\Omega }|v(x)=v(y)+Dv(y)(xy)+h\}.$$

The modulus of convexity of $`v`$ is a nonnegative function, defined by

$$\rho _v(r)=\underset{y\mathrm{\Omega }}{inf}\rho _{v,y}(r),r>0,$$

where

$$\rho _{v,y}(r)=sup\{h0|S_{h,v}(y)B_r(y)\}$$

if there exists $`h0`$ such that $`S_{h,v}(y)B_r(y)`$, otherwise we define $`\rho _{v,y}(r)=0`$. We have $`\rho _v(r)>0`$ for all $`r>0`$ if $`v`$ is strictly convex.

Let $`u`$ be a smooth, locally uniformly convex solution of (2.1). Then we have the following lower bound estimate, for any $`\mathrm{\Omega }^{}\mathrm{\Omega }`$,

$$\underset{x\mathrm{\Omega }^{}}{inf}\text{det}D^2u(x)C,$$
$`2.3`$
where $`C`$ depends on $`n`$, $`\text{dist}(\mathrm{\Omega }^{},\mathrm{\Omega })`$, $`sup_\mathrm{\Omega }|Du|`$, $`inf_\mathrm{\Omega }f`$, and $`\rho _u`$. The proof again can be achieved by introducing an appropriate auxiliary function.

From the a priori estimates (2.2) and (2.3) we then have

###### Theorem 2.1

Let $`uC^4(\mathrm{\Omega })C^0(\overline{\mathrm{\Omega }})`$ be a locally uniformly convex solution of (2.1). Then for any subdomain $`\mathrm{\Omega }^{}\mathrm{\Omega }`$, we have:

(i) $`W^{4,p}`$ estimate,

$$u_{W^{4,p}(\mathrm{\Omega }^{})}C,$$
$`2.4`$
where $`p[1,\mathrm{})`$, $`C`$ depends on $`n,p,sup_\mathrm{\Omega }|f|`$, $`\text{dist}(\mathrm{\Omega }^{},\mathrm{\Omega })`$, $`sup_\mathrm{\Omega }|u|`$, and $`\rho _u`$.

(ii) Schauder estimate,

$$u_{C^{4,\alpha }(\mathrm{\Omega }^{})}C,$$
$`2.5`$
where $`\alpha (0,1)`$, $`C`$ depends on $`n,\alpha ,f_{C^\alpha (\overline{\mathrm{\Omega }})}`$, $`\text{dist}(\mathrm{\Omega }^{},\mathrm{\Omega })`$, $`sup_\mathrm{\Omega }|u|`$, and $`\rho _u`$.

Note that the gradient of $`u`$ is locally controlled by $`\rho _u`$, the modulus of convexity of $`u`$. To prove Theorem 2.1, we write (2.1) as a second order partial differential system

$`U^{ij}w_{ij}`$ $`=f\text{in}\mathrm{\Omega },`$ $`2.6`$$`2.7`$
$`\text{det}D^2u`$ $`=w^{(n+2)/(n+1)}\text{in}\mathrm{\Omega },`$
where (2.6) is regarded as a second order elliptic equation for $`w`$. By (2.2) and (2.3), and the Hölder continuity of linearized Monge-Ampère equation , we have the interior a priori Hölder estimate for $`w`$. We note that the Hölder continuity in  is proved for the homogeneous equation, but the argument there can be easily carried over to the non-homogeneous case under (2.2) and (2.3). By the interior Schauder estimate for the Monge-Ampère equation , we obtain the interior a priori $`C^{2,\alpha }`$ estimate for $`u`$. It follows that (2.6) is a linear uniformly elliptic equation with Hölder coefficients. Hence Theorem 2.1 follows.

The control on strict convexity is a key condition in Theorem 2.1. One cannot expect the strict convexity of solutions when $`n3`$. Indeed, there are convex solutions to the Monge-Ampère equation

$$\text{det}D^2u=1$$
$`2.8`$
which are not strictly convex, and so not smooth . Note that any non-smooth convex solution of (2.8) can be approximated by smooth ones, and a smooth solution of (2.8) is obviously a solution of (2.1), with $`f=0`$.

An interesting problem is to find appropriate conditions to estimate the strict convexity of solutions of (2.1). For the affine Bernstein problem it suffices to prove convexity estimate for solutions vanishing on the boundary. We succeeded only in dimension two, see §5.

## 3. The affine Bernstein problem

We say a hypersurface $``$, immersed in $`^{n+1}`$, is Euclidean complete if it is complete under the metric induced from the standard Euclidean metric.

###### Theorem 3.1

A Euclidean complete, affine maximal, locally uniformly convex surface in $`^3`$ is an elliptic paraboloid.

Theorem 3.1 extends Jorgens’ theorem , which asserts that an entire convex solution of (2.8) in $`^2`$ must be a quadratic function. Jorgens’ theorem also leads to the Bernstein theorem for minimal surfaces in dimension two . Jorgens’ theorem was extended to higher dimensions by Calabi  for $`2n5`$ and Pogorelov  for $`n2`$. See also . Observe that the Chern conjecture follows from Theorem 3.1 immediately.

The proof of Theorem 3.1 uses the affine invariance of equation (1.3) and the a priori estimates in §2. First note that a Euclidean complete locally uniformly convex hypersurface must be a graph. Suppose the surface in Theorem 3.1 is the graph of a nonnegative convex function $`u`$ with $`u(0)=0`$. For any constant $`h>1`$, let $`T_h`$ be the linear transformation which normalizes the section $`S_{h,u}^0=\{u<h\}`$, and let $`v_h(x)=h^1u(T_h^1(x))`$. By the convexity estimate in dimension two, the modulus of convexity of $`v_h`$ is independent of $`h`$. Hence there is a uniform positive distance from the origin to the boundary $`T_h(S_{h,u}^0)`$. By Theorem 2.1, we infer that the largest eigenvalue of $`T_h`$ is controlled by the least one of $`T_h`$, which implies that $`u`$ is defined in the entire $`^2`$. By Theorem 2.1 again, $`D^3v_h(0)`$ is bounded. Hence for any given $`x^2`$,

$$|D^2u(x)D^2u(0)|Ch^{1/2}0$$

as $`h0`$, namely $`D^2u(x)=D^2u(0)`$.

Note that the dimension two restriction is used only for the strict convexity estimate. The affine Bernstein problem was investigated by Calabi in a number of papers . Using the result that a nonnegative harmonic function (i.e. $`h`$ in (1.5)) defined on a complete manifold with nonnegative Ricci curvature must be a constant, he proved that, among others, the Bernstein theorem in dimension two, under the additional hypothesis that the surface is also complete under the affine metric.

Instead of the Euclidean completeness as in the Chern conjecture, Calabi asks whether affine completeness alone is sufficient for the Bernstein theorem. This question was recently answered affirmatively in . See also  for a different treatment based on the result in . In  we proved a much stronger result. That is

###### Theorem 3.2

An affine complete, locally uniformly convex hypersurface in $`^{n+1}`$, $`n2`$, is also Euclidean complete.

The converse of Theorem 3.2 is not true , nor is it for $`n=1`$. For the proof, which uses the Legendre transform and Lemma 4.1 below, we refer the reader to  for details.

## 4. Locally convex hypersurfaces with boundary

In this section we present some results in , which guarantee the sub-convergence of bounded sequences of locally convex hypersurfaces with prescribed boundary.

Recall that a hypersurface $`^{n+1}`$ (not necessarily smooth) is called locally convex if it is a locally convex immersion of a manifold $`𝒩`$ and there is a continuous vector field on the convex side of $``$, transversal to $``$ everywhere. Let $`T`$ denote the immersion, namely $`=T(𝒩)`$. For any given point $`x`$, $`T^1(x)`$ may contains more than one point. To avoid confusion when referring to a point $`x`$ we understand a pair $`(x,p)`$ for some point $`p𝒩`$ such that $`x=T(p)`$. We say $`\omega _x`$ is a neighborhood of $`x`$ if it is the image of a neighborhood of $`p`$ in $`𝒩`$. The $`r`$-neighborhood of $`x`$, $`\omega _r(x)`$, is the connected component of $`B_r(x)`$ containing the point $`x`$. In  we proved the following fundamental lemma for locally convex hypersurfaces.

###### Lemma 4.1

Let $``$ be a compact, locally convex hypersurface in $`^{n+1}`$, $`n>1`$. Suppose the boundary $``$ lies in the hyperplane $`\{x_{n+1}=0\}`$. Then any connected component of $`\{x_{n+1}<0\}`$ is convex.

A locally convex hypersurface $``$ is called convex if it lies on the boundary of the convex closure of $``$ itself. From Lemma 4.1 it follows that a (Euclidean) complete locally convex hypersurface with at least one strictly convex point is convex, and that a closed, locally convex hypersurface is convex. Lemma 4.1 also plays a key role in the proof of Theorem 3.2.

An application we will use here is the uniform cone property for locally convex hypersurfaces. Let $`𝒞_{x,\xi ,r,\alpha }`$ denote the cone

$$𝒞_{x,\xi ,r,\alpha }=\{y^{n+1}||yx|<r,yx,\xi \mathrm{cos}\alpha |yx|\}.$$

We say that $`𝒞_{x,\xi ,r,\alpha }`$ is an inner contact cone of $``$ at $`x`$ if this cone lies on the concave side of $`\omega _r(x)`$. We say $``$ satisfies the uniform cone condition with radius $`r`$ and aperture $`\alpha `$ if $``$ has an inner contact cone at all points with the same $`r`$ and $`\alpha `$.

###### Lemma 4.2

Let $`B_R(0)`$ be a locally convex hypersurface with boundary $``$. Suppose $``$ can be extended to $`\stackrel{~}{}`$ such that $``$ lies in the interior of $`\stackrel{~}{}`$ and $`\stackrel{~}{}`$ is locally strictly convex. Then there exist $`r,\alpha >0`$ depending only on $`n`$, $`R`$, and the extended part $`\stackrel{~}{}`$, such that the $`r`$-neighborhood $`\omega _r(x)`$ is convex for any $`x`$, and $``$ satisfies the uniform cone condition with radius $`r`$ and aperture $`\alpha `$.

In  we have shown that if $``$ is smooth and $``$ is smooth and locally uniformly convex near $``$, then $``$ can be extended to $`\stackrel{~}{}`$ as required in Lemma 4.2. The main point of Lemma 4.2 is that $`r`$ and $`\alpha `$ depend only on $`n,R`$ and the extended part $`\stackrel{~}{}`$. Therefore it holds with the same $`r`$ and $`\alpha `$ for a family of locally convex hypersurfaces, which includes all locally uniformly convex hypersurfaces with boundary $``$, contained in $`B_R(0)`$, such that its Gauss mapping image coincides with that of $``$. For any sequence of locally convex hypersurfaces in this family, the uniform cone property implies the sequence converges subsequently and no singularity develops in the limit hypersurface. This property is the key for the existence proof of maximizers to the affine Plateau problem. It also plays a key role for our resolution of the Plateau problem for prescribed constant Gauss curvature (as conjectured in ), see . We state the result as follows.

###### Theorem 4.1

Let $`\mathrm{\Gamma }=(\mathrm{\Gamma }_1,\mathrm{},\mathrm{\Gamma }_n)^{n+1}`$ be a smooth disjoint collection of closed co-dimension two embedded submanifolds. Suppose $`\mathrm{\Gamma }`$ bounds a locally strictly convex hypersurface $`S`$ with Gauss curvature $`K(S)>K_0>0`$. Then $`\mathrm{\Gamma }`$ bounds a smooth, locally uniformly convex hypersurface of Gauss curvature $`K_0`$.

If $`S`$ is a (multi-valued) radial graph over a domain in $`S^n`$ which does not contain any hemi-spheres, Theorem 4.1 was established in . Theorem 4.1 has been extended to more general curvature functions in .

## 5. The affine Plateau problem

First we formulate the affine Plateau problem as a variational maximization problem. Let $`_0`$ be a compact, connected, locally uniformly convex hypersurface in $`^{n+1}`$ with smooth boundary $`\mathrm{\Gamma }=_0`$. Let $`S[_0]`$ denote the set of locally uniformly convex hypersurfaces $``$ with boundary $`\mathrm{\Gamma }`$ such that the image of the Gauss mapping of $``$ coincides with that of $`_0`$. Then any two hypersurfaces in $`S[]`$ are diffeomorphic. Let $`\overline{S}[_0]`$ denote the set of locally convex hypersurfaces which can be approximated by smooth ones in $`S[_0]`$. Our variational affine Plateau problem is to find a smooth maximizer to

$$\underset{\overline{S}[_0]}{sup}A().$$
$`5.1`$
To study (5.1) we need to extend the definition of the affine area functional to non-smooth convex hypersurfaces. Different but equivalent definitions can be found in . Here we adopt a new definition introduced in , which is also more straightforward. Observe that the Gauss curvature $`K`$ can be extended to a measure on a non-smooth convex hypersurface, and the measure can be decomposed as the sum of a singular part and a regular part, $`K=K_s+K_r`$, where the singular part $`K_s`$ is a measure supported on a set of Lebesgue measure zero, and the regular part $`K_r`$ can be represented by an integrable function. We extend the definition of affine area functional (1.1) to

$$A()=_{}K_r^{1/(n+2)}.$$
$`5.2`$
The affine area functional is upper semi-continuous . See also  for different proofs.

A necessary condition for the affine Plateau problem is that the Gauss mapping image of $`_0`$ cannot contain any semi-spheres. Indeed if $``$ is affine maximal such that its Gauss mapping image contains, say, the south hemi-sphere, then the pre-image of the south hemi-sphere is a graph of a convex function $`u`$ over a domain $`\mathrm{\Omega }`$ such that $`|Du(x)|\mathrm{}`$ as $`x\mathrm{\Omega }`$. Then necessarily $`\text{det}D^2u=\mathrm{}`$ and so $`w=0`$ on $`\mathrm{\Omega }`$. It follows that $`w0`$ in $`\mathrm{\Omega }`$, a contradiction.

###### Theorem 5.1

Let $`_0`$ be a compact, connected, locally uniformly convex hypersurface in $`^3`$ with smooth boundary $`\mathrm{\Gamma }=_0`$. Suppose the image of the Gauss mapping of $`_0`$ does not contain any semi-spheres. Then there is a smooth maximizer to (5.1).

To prove the existence we observe that by the necessary condition, there exists a positive constant $`R`$ such that $`B_R(0)`$ for any $`\overline{S}[_0]`$. Hence by the uniform cone property, Lemma 4.2, any maximizing sequence in $`\overline{S}[_0]`$ is sub-convergent. The existence of maximizers then follows from the upper semi-continuity of the affine area functional. Note that the existence is true for all dimensions.

To prove the regularity we need to show that

(i) $``$ can be approximated by smooth affine maximal surfaces; and

(ii) $``$ is strictly convex.

The purpose of (i) is such that the a priori estimate in Section 2 is applicable. Note that (i) also implies the Bernstein Theorem 3.1 holds for non-smooth affine maximal surfaces.

By the penalty method we proved (i) for all dimensions, using the following classical solvability of the second boundary value problem for the affine maximal surface equation.

###### Theorem 5.2

Consider the problem

$`L(u)`$ $`=f(x,u)\text{in}\mathrm{\Omega },`$ $`5.3`$
$`u`$ $`=\phi \text{on}\mathrm{\Omega },`$
$`w`$ $`=\psi \text{on}\mathrm{\Omega },`$
where $`w`$ is given in (1.4), $`\mathrm{\Omega }`$ is a uniformly convex domain with $`C^{4,\alpha }`$ boundary, $`0<\alpha <1`$, $`f`$ is Hölder continuous, non-decreasing in $`u`$, $`\phi ,\psi C^{4,\alpha }(\overline{\mathrm{\Omega }})`$, and $`\psi `$ is positive. Then there is a unique uniformly convex solution $`uC^{4,\alpha }(\overline{\mathrm{\Omega }})`$ to the above problem.

To prove Theorem 5.2 we first prove that $`u`$ satisfies (2.2) and (2.3), and that $`w`$ is Lipschitz continuous on the boundary, namely $`|w(x)w(y)|C|xy|`$ for any $`x\mathrm{\Omega }`$, $`y\mathrm{\Omega }`$. Theorem 5.2 is then reduced to the boundary $`C^{2,\alpha }`$ estimate for the Monge-Ampère equation. The proof for the boundary $`C^{2,\alpha }`$ estimate involves a delicate iteration scheme. We refer to  for details.

The interior $`C^{2,\alpha }`$ estimate for the Monge-Ampère equation was proved by Caffarelli , using a perturbation argument. The boundary $`C^{2,\alpha }`$ estimate, which also uses a similar perturbation argument, contains substantial new difficulty, as that the sections

$$S_{h,v}^0(y)=\{x\mathrm{\Omega }|v(x)<v(y)+Dv(y)(xy)+h\}$$

can be normalized for the interior estimate but not for the boundary estimate. We need to prove that $`S_{h,v}^0(y)`$ has a good shape for sufficiently small $`h>0`$ and $`y\mathrm{\Omega }`$.

Finally we would like to mention our idea of proving the strict convexity, namely (ii) above. Note that for both the affine Bernstein and affine Plateau problem, the dimension two assumption is only used for the proof of the strict convexity. To prove the strict convexity we suppose to the contrary that $``$ contains a line segment. Let $`P`$ be a tangent plane of $``$ which contains the line segment. Then the contact set $`F`$, namely the connected set of $`P`$ containing the line segment, is a convex set. If $`F`$ has an extreme point which is an interior point of $``$, by rescaling and choosing appropriate coordinates we obtain a sequence of affine maximal functions which converges to a convex function $`v`$, such that $`v(0)=0`$ and $`v(x)>0`$ for $`x0`$ in an appropriate coordinate system, and $`v`$ is not $`C^1`$ at the origin $`0`$. In dimension two this means $`\text{det}D^2v`$ is unbounded near $`0`$, which is in contradiction with the estimate (2.2).

If all extreme points of $`F`$ are boundary points of $``$, we use the Legendre transform to get a new convex function which is a maximizer of a variational problem similar to (5.1), and satisfies the properties as $`v`$ above, which also leads to a contradiction.

## 6. Remarks

We proved the affine Bernstein problem in dimension two. In high dimensions ($`n10`$) a counter-example was given in , where we proved that the function

$$u(x)=(|x^{}|^9+x_{10}^2)^{1/2}$$
$`6.1`$
is affine maximal, where $`x^{}=(x_1,\mathrm{},x_9)`$.

The function $`u`$ in (6.1) contains a singular point, namely the origin. The graph of $`u`$ is indeed an affine cone, that is all the level sets $`S_{h,u}=\{u=h\}`$ are affine self-similar, in the sense that there is an affine transformation $`T_h`$ such that $`T_h(S_{h,u})=S_{1,u}`$. The above counter-example shows that there is an affine cone in dimensions $`n10`$ which is affine maximal but is not an elliptic paraboloid. We have not been successful in finding smooth counter-examples. Little is known for dimensions $`3n9`$.

For the affine Plateau problem, an interesting problem is whether the maximizer satisfies the boundary conditions. If $`_0`$ is the graph of a smooth, uniformly convex function $`\phi `$, defined in a bounded domain $`\mathrm{\Omega }^n`$, the Plateau problem becomes the first boundary value problem, that is equation (1.3) subject to the boundary conditions:

$`u`$ $`=\phi \text{on}\mathrm{\Omega },`$ $`6.2`$$`6.3`$
$`Du`$ $`=D\phi \text{on}\mathrm{\Omega }.`$
In this case we also proved the uniqueness of maximizers of (5.1). Obviously the maximizer $`u`$ satisfies (6.2). Whether $`u`$ satisfies (6.3) is still unknown. Recall that the Dirichlet problem of the minimal surface equation is solvable for any smooth boundary values if and only if the boundary is mean convex. Therefore an additional condition may be necessary in order that (6.3) is fulfilled.

## References

 W. Blaschke, Vorlesungen über Differential geometrie, Berlin, 1923.

 L.A. Caffarelli, Interior $`W^{2,p}`$ estimates for solutions of Monge-Ampère equations, Ann. Math., 131 (1990), 135–150.

 L.A. Caffarelli and C.E. Gutiérrez, Properties of the solutions of the linearized Monge-Ampère equations, Amer. J. Math., 119(1997), 423–465.

 E. Calabi, Improper affine hypersurfaces of convex type and a generalization of a theorem by K. Jörgens, Michigan Math. J., 5(1958), 105–126.

 E. Calabi, Hypersurfaces with maximal affinely invariant area, Amer. J. Math. 104(1982), 91–126.

 E. Calabi, Convex affine maximal surfaces, Results in Math., 13(1988), 199–223.

 E. Calabi, Affine differential geometry and holomorphic curves, Lecture Notes Math. 1422(1990), 15–21.

 S.Y. Cheng and S.T. Yau, Complete affine hypersurfaces, I. The completeness of affine metrics, Comm. Pure Appl. Math., 39(1986), 839–866.

 S.S. Chern, Affine minimal hypersurfaces, in minimal submanifolds and geodesics, Proc. Japan-United States Sem., Tokyo, 1977, 17–30.

 B. Guan and J. Spruck, Boundary value problems on $`S^n`$ for surfaces of constant Gauss curvature. Ann. of Math., 138(1993), 601–624.

 K. Jorgens, Über die Lösungen der Differentialgleichung $`rts^2=1`$, Math. Ann. 127(1954), 130–134.

 K. Nomizu and T. Sasaki, Affine differential geometry, Cambridge, 1994.

 K. Leichtweiss, Affine geometry of convex bodies, Johann Ambrosius Barth Verlag, Heidelberg, 1998.

 A.M. Li and F. Jia, The Calabi conjecture on affine maximal surfaces, Result. Math., 40(2001), 265–272.

 E. Lutwak, Extended affine surface area, Adv. Math., 85(1991), 39–68.

 A. Martinez and F. Milan, On the affine Bernstein problem, Geom. Dedicata, 37(1991), 295–302.

 A.V. Pogorelov, The muitidimensional Minkowski problems, J. Wiley, New York, 1978.

 W.M. Sheng, J. Urbas, and X.-J. Wang, Interior curvature bounds for a class of curvature equations, preprint.

 U. Simon, Affine differential geometry, in Handbook of differential geometry, North-Holland, Amsterdam, 2000, 905–961.

 J. Spruck, Fully nonlinear elliptic equations and applications in geometry, Proc. International Congress Math., Birkhäuser, Basel, 1995, 1145–1152.

 N.S. Trudinger and X.-J. Wang, The Bernstein problem for affine maximal hypersurfaces, Invent. Math., 140 (2000), 399–422.

 N.S. Trudinger and X.-J. Wang, Affine complete locally convex hypersurfaces, Invent. Math., to appear.

 N.S. Trudinger and X.-J. Wang, On locally convex hypersurfaces with boundary, J. Reine Angew. Math., to appear.

 N.S. Trudinger and X.-J. Wang, The Plateau problem for affine maximal hypersurfaces, Preprint.


text:
# Collective Modes of an Anisotropic Quark-Gluon Plasma

## I Introduction

In the ongoing ultra-relativistic heavy collision experiments at the Relativistic Heavy Ion Collider (RHIC) and the upcoming ones at the Large Hadron Collider (LHC) the behavior of nuclear matter under extreme conditions will be studied. The hope of these experiments is to create temperatures which are high enough for nuclear matter to undergo a phase transition to a quark gluon plasma (QGP). The quark gluon plasma, if generated, is expected to expand, cool, and then hadronize in the final stage of its evolution. In this context, an outstanding question faced by experimentalists and theorists is whether or not the system will “thermalize” fast enough to allow a thermodynamic description of the system during the central part of its evolution.

In this paper we study the role of the collective modes of finite-temperature QCD in the thermalization, particularly the isotropization, of a finite-temperature QGP with anisotropic momentum-space distribution functions. This question has been addressed in previous papers in which the existence of instabilities of a QGP were studied. In Refs. SM:1993 ; SM:1994 ; SM:1997  Mrówczyński discussed the existence of instabilities to chromomagnetic fluctuations with a particular orientation of the chromoelectric field and wave-vector. In those papers Mrówczyński showed that there existed an instability which was the equivalent of the Weibel or filamentation instability in electrodynamics Weibel:1959 . Weibel showed in his original paper that, within electrodynamics, unstable transverse modes exist in plasmas with anisotropic momentum distributions and he also derived their rate of growth in linear response theory. These types of instabilities are potentially very important to QGP evolution at RHIC or LHC due to the large amount of momentum-space anisotropy in the gluon distribution functions at $`\tau 1\mathrm{fm}/\mathrm{c}`$.

Mrówczyński and Randrup have recently performed phenomenological estimates of the growth rate of the instabilities for two types of anisotropic distribution functions RM:2003 . They found that the degree of amplification of the Weibel instability is not expected to dominate the dynamics of a QGP instead being comparable to the contribution from elastic Boltzmann collisions. However, they did point out that since a large number of the unstable modes could be excited then it is possible that their combined effect on the overall dynamics could be significant. In this paper we perform a detailed study of the hard-thermal-loop resummed gluon self-energy including a complete tensor decomposition of the self-energy and identification of all stable and unstable collective modes.

In Sec. II we set up the framework used to obtain the hard-thermal-loop self-energy in a system with an anisotropic momentum space distribution. In Sec. III we present a tensor decomposition of the self-energy and dielectric tensors. In Sec. IV we work out the details of the tensor decomposition and give expressions for the self-energy “structure functions.” In Sec. V we discuss the static limit of the various self-energy structure functions. In Sec. VI we use the tensor decomposition of the dielectric tensor to determine dispersion relations for all stable and unstable modes. In Sec. VII we present analytic expressions for the self-energy structure functions in the small-anisotropy limit. Finally, in Sec. VIII we present conclusions and outlook for application of the results found here. We provide a summary of our notational conventions and expressions for the various self-energy structure functions in two appendices.

## II Hard-Thermal-Loop Self-Energy

We begin by repeating some of the steps necessary to derive the hard-thermal-loop resummed gluon self-energy within semi-classical transport theory SM:1993 ; SM:1994 ; SM:1997 . Within this approach partons are described by their phase-space densities and their time evolution is given by Vlasov-type transport equations EH:1989 ; BI:2002 . In this paper we will concentrate on the physics at the soft scale, $`kgTT`$, which is the first scale at which collective motion appears. At this scale the magnitude of the field fluctuations is $`A\sqrt{g}T`$ and derivatives are of the scale $`_xgT`$. With this power-counting a systematic truncation of the terms contributing to the transport equations for soft momenta can be realized.

At leading order in the coupling constant the color current, $`J^\mu `$, induced by a soft gauge field, $`A^\mu `$, with four-momentum $`K=(\omega ,𝐤)`$ can be obtained by performing a covariant gradient expansion of the quark and gluon Wigner functions in mean-field approximation. The result is

$$J_{\mathrm{ind}}^{\mu ,a}\left(X\right)=g\frac{d^3p}{\left(2\pi \right)^3}V^\mu (2N_c\delta N^a(p,X)+N_f(\delta n_+^a(p,X)\delta n_{}^a(p,X)),$$
(1)
where $`V^\mu =(1,𝐤/\omega )`$ is the gauge field four-velocity, $`\delta N^a(p,X)`$ is the fluctuating part of the gluon density, and $`\delta n_+^a(p,X)`$ and $`\delta n_{}^a(p,X)`$ are the fluctuating parts of the quark and anti-quark densities, respectively. Note that $`\delta N^a`$ transforms as a vector in the adjoint representation ($`\delta N\delta N^aT^a`$) and $`\delta n_\pm ^a`$ transforms as a vector in the fundamental representation ($`\delta n_\pm \delta n_\pm ^at^a`$).

The quark and gluon density matrices above satisfy the following transport equations

$`[VD_X,\delta n_\pm (p,X)]`$ $`=`$ $`gV_\mu F^{\mu \nu }(p,X)_\nu n_\pm \left(𝐩\right),`$ (2)
$`[VD_X,\delta N(p,X)]`$ $`=`$ $`gV_\mu F^{\mu \nu }(p,X)_\nu N\left(𝐩\right),`$ (3)
where $`D_X=_X+igA\left(X\right)`$ is the covariant derivative.

Solving the transport equations (2) and (3) for the fluctuations $`\delta N`$ and $`\delta n_\pm `$ gives the induced current via Eq. (1)

$`J_{\mathrm{ind}}^\mu \left(X\right)=g^2{\displaystyle \frac{d^3p}{\left(2\pi \right)^3}V^\mu V^\alpha _{\left(p\right)}^\beta f\left(𝐩\right)𝑑\tau U(X,XV\tau )F_{\alpha \beta }\left(XV\tau \right)U(XV\tau ,X)},`$ (4)
where $`U(X,Y)`$ is a gauge parallel transporter defined by the path-ordered integral

$$U(X,Y)=𝒫\mathrm{exp}\left[ig_X^Y𝑑Z_\mu A^\mu \left(Z\right)\right],$$
(5)
$`F_{\alpha \beta }=_\alpha A_\beta _\beta A_\alpha ig[A_\mu ,A_\nu ]`$ is the gluon field strength tensor, and

$$f\left(𝐩\right)=2N_cN\left(𝐩\right)+N_f\left(n_+\left(𝐩\right)+n_{}\left(𝐩\right)\right).$$
(6)
Neglecting terms of subleading order in $`g`$ (implying $`U1`$ and $`F_{\alpha \beta }_\alpha A_\beta _\beta A_\alpha `$) and performing a Fourier transform of the induced current to momentum space we obtain

$$J_{\mathrm{ind}}^\mu \left(K\right)=g^2\frac{d^3p}{\left(2\pi \right)^3}V^\mu _{\left(p\right)}^\beta f\left(𝐩\right)\left(g_{\gamma \beta }\frac{V_\gamma K_\beta }{KV+iϵ}\right)A^\gamma \left(K\right),$$
(7)
where $`ϵ`$ is a small parameter that has to be sent to zero in the end.

From this expression of the induced current the self-energy is obtained via

$$\mathrm{\Pi }^{\mu \nu }\left(K\right)=\frac{\delta J_{\mathrm{ind}}^\mu \left(K\right)}{\delta A_\nu \left(K\right)},$$
(8)
which gives

$$\mathrm{\Pi }^{\mu \nu }\left(K\right)=g^2\frac{d^3p}{\left(2\pi \right)^3}V^\mu _{\left(p\right)}^\beta f\left(𝐩\right)\left(g_{\nu \beta }\frac{V_\nu K_\beta }{KV+iϵ}\right).$$
(9)
This tensor is symmetric, $`\mathrm{\Pi }^{\mu \nu }\left(K\right)=\mathrm{\Pi }^{\nu \mu }\left(K\right)`$, and transverse, $`K^\mu \mathrm{\Pi }^{\mu \nu }\left(K\right)=0`$. Note that the same result can be obtained using diagrammatic methods if one assumes that the distribution function is symmetric under $`𝐩𝐩`$ MT:2000 .

In the linear approximation the equation of motion for the gauge fields can be obtained by expressing the induced current in terms of the self-energy

$$J_{\mathrm{ind}}^\mu \left(K\right)=\mathrm{\Pi }^{\mu \nu }\left(K\right)A^\nu \left(K\right),$$
(10)
and plugging this into Maxwell’s equation

$$iK_\mu F^{\mu \nu }\left(K\right)=J_{\mathrm{ind}}^\nu \left(K\right)+J_{\mathrm{ext}}^\nu \left(K\right),$$
(11)
to obtain

$$\left[K^2g^{\mu \nu }K^\mu K^\nu +\mathrm{\Pi }^{\mu \nu }\left(K\right)\right]A_\nu \left(K\right)=J_{\mathrm{ext}}^\nu \left(K\right),$$
(12)
where $`J_{\mathrm{ext}}^\nu `$ is an external current. Using the gauge invariance of the self-energy we can write this in terms of a physical electric field by specifying a particular gauge. In the temporal axial gauge defined by $`A_0=0`$ we obtain

$$\left[\left(k^2\omega ^2\right)\delta ^{ij}k^ik^j+\mathrm{\Pi }^{ij}\left(K\right)\right]E^j\left(K\right)=\left(\mathrm{\Delta }^1\left(K\right)\right)^{ij}E^j\left(K\right)=i\omega J_{\mathrm{ext}}^i\left(K\right).$$
(13)
Inverting the propagator allows us to determine the response of the system to the external source

$$E^i\left(K\right)=i\omega \mathrm{\Delta }^{ij}\left(K\right)J_{\mathrm{ext}}^j\left(K\right).$$
(14)
The dispersion relations for the collective modes can be obtained by finding the poles in the propagator $`\mathrm{\Delta }^{ij}\left(K\right)`$.

## III Tensor Decomposition

In this section we develop a tensor basis for an anisotropic system in which there is only one preferred direction. As mentioned above the self-energy is symmetric and transverse. As a result not all components of $`\mathrm{\Pi }^{\mu \nu }`$ are independent and we can restrict our considerations to the spatial part of $`\mathrm{\Pi }^{\mu \nu }`$, denoted $`\mathrm{\Pi }^{ij}`$. We therefore need to construct a basis for a symmetric 3-tensor that – apart from the momentum $`k^i`$ – also depends on a fixed anisotropy three-vector $`n^i`$, with $`n^2=1`$. Following Ref. KKR:1991  we first define the projection operator

$$A^{ij}=\delta ^{ij}k^ik^j/k^2,$$
(15)
and use it to construct $`\stackrel{~}{n}^i=A^{ij}n^j`$ which obeys $`\stackrel{~}{n}k=0`$. With this we can construct the remaining three tensors

$$B^{ij}=k^ik^j/k^2$$
(16)
$$C^{ij}=\stackrel{~}{n}^i\stackrel{~}{n}^j/\stackrel{~}{n}^2$$
(17)
$$D^{ij}=k^i\stackrel{~}{n}^j+k^j\stackrel{~}{n}^i.$$
(18)
Any symmetric 3-tensor $`𝐓`$ can now be decomposed into the basis spanned by the four tensors $`𝐀,𝐁,𝐂,`$ and $`𝐃`$

$$𝐓=a𝐀+b𝐁+c𝐂+d𝐃.$$
(19)
Furthermore, the inverse of any such tensor is then given as

$$𝐓^1=a^1𝐀+\frac{\left(a+c\right)𝐁a^1\left(bc\stackrel{~}{n}^2k^2d^2\right)𝐂d𝐃}{b\left(a+c\right)\stackrel{~}{n}^2k^2d^2}.$$
(20)
## IV Self-energy Structure Functions

The spacelike components of the self-energy tensor can be written as

$$\mathrm{\Pi }^{ij}\left(K\right)=g^2\frac{d^3p}{\left(2\pi \right)^3}v^i^lf\left(𝐩\right)\left(\delta ^{jl}+\frac{v^jk^l}{KV+iϵ}\right).$$
(21)
At this point the distribution function $`f\left(𝐩\right)`$ is completely arbitrary. In order to proceed we need to specify a form for the distribution function. In what follows we will assume that $`f\left(𝐩\right)`$ can be obtained from an arbitrary isotropic distribution function by the rescaling of only one direction in momentum space. In practice this means that, given any isotropic distribution function $`f_{\mathrm{iso}}\left(𝐩^2\right)`$, we can construct an anisotropic version by changing the argument

$$f\left(𝐩\right)=f_{\mathrm{iso}}\left(𝐩^2+\xi \left(𝐩\widehat{𝐧}\right)^2\right),$$
(22)
where $`\widehat{𝐧}`$ is the direction of the anisotropy and $`\xi >1`$ is a adjustable anisotropy parameter. Note that $`\xi >0`$ corresponds to a contraction of the distribution in the $`\widehat{𝐧}`$ direction whereas $`1<\xi <0`$ corresponds to a stretching of the distribution in the $`\widehat{𝐧}`$ direction. This assumption allows us to simplify (21) by performing a change of variables to $`\stackrel{~}{p}`$

$$\stackrel{~}{p}^2=p^2\left(1+\xi \left(𝐯𝐧\right)^2\right).$$
(23)
After making this change of variables it is possible to integrate out the $`\left|\stackrel{~}{p}\right|`$-dependence giving

$$\mathrm{\Pi }^{ij}\left(K\right)=m_D^2\frac{d\mathrm{\Omega }}{4\pi }v^i\frac{v^l+\xi (𝐯.𝐧)n^l}{(1+\xi (𝐯.𝐧)^2)^2}\left(\delta ^{jl}+\frac{v^jk^l}{KV+iϵ}\right),$$
(24)
where

$$m_D^2=\frac{g^2}{2\pi ^2}_0^{\mathrm{}}𝑑pp^2\frac{df_{\mathrm{iso}}\left(p^2\right)}{dp}.$$
(25)
We can then decompose the self-energy into four structure functions

$$\mathrm{\Pi }^{ij}=\alpha A^{ij}+\beta B^{ij}+\gamma C^{ij}+\delta D^{ij},$$
(26)
which are determined by taking the following contractions:

$`k^i\mathrm{\Pi }^{ij}k^j`$ $`=`$ $`k^2\beta ,`$
$`\stackrel{~}{n}^i\mathrm{\Pi }^{ij}k^j`$ $`=`$ $`\stackrel{~}{n}^2k^2\delta ,`$
$`\stackrel{~}{n}^i\mathrm{\Pi }^{ij}\stackrel{~}{n}^j`$ $`=`$ $`\stackrel{~}{n}^2\left(\alpha +\gamma \right),`$
$`\mathrm{Tr}\mathrm{\Pi }^{ij}`$ $`=`$ $`2\alpha +\beta +\gamma .`$ (27)
In Appendix B we collect the resulting integral expressions for the structure functions. All four structure functions depend on $`m_D`$, $`\omega `$, $`k`$, $`\xi `$, and $`\widehat{𝐤}\widehat{𝐧}=\mathrm{cos}\theta _n`$. In the limit $`\xi 0`$ the structure functions $`\alpha `$ and $`\beta `$ reduce to the isotropic hard-thermal-loop self-energies and $`\gamma `$ and $`\delta `$ vanish

$`\alpha (K,0)`$ $`=`$ $`\mathrm{\Pi }_T\left(K\right),`$
$`\beta (K,0)`$ $`=`$ $`{\displaystyle \frac{\omega ^2}{k^2}}\mathrm{\Pi }_L\left(K\right),`$
$`\gamma (K,0)`$ $`=`$ $`0,`$
$`\delta (K,0)`$ $`=`$ $`0,`$ (28)
with

$`\mathrm{\Pi }_T\left(K\right)`$ $`=`$ $`{\displaystyle \frac{m_D^2}{2}}{\displaystyle \frac{\omega ^2}{k^2}}\left[1{\displaystyle \frac{\omega ^2k^2}{2\omega k}}\mathrm{log}{\displaystyle \frac{\omega +k}{\omega k}}\right],`$ (29)
$`\mathrm{\Pi }_L\left(K\right)`$ $`=`$ $`m_D^2\left[{\displaystyle \frac{\omega }{2k}}\mathrm{log}{\displaystyle \frac{\omega +k}{\omega k}}1\right].`$ (30)
For finite $`\xi `$ the analytic structure of the structure functions is the same as in the isotropic case. There is a cut in the complex $`\omega `$ plane which we can chose to run along the real $`\omega `$ axis from $`k<\omega <k`$. For real-valued $`\omega `$ the structure functions are complex for all $`\omega <k`$ and real for $`\omega >k`$. For imaginary-valued $`\omega `$ all four structure functions are real-valued. In Fig. 1 we plot the structure function $`\alpha `$ for real and imaginary values of $`\omega `$, $`\xi =\{0,1,10\}`$, and $`\theta _n=\pi /4`$.

With these structure functions in hand we can construct the propagator $`\mathrm{\Delta }^{ij}\left(K\right)`$ using the expressions from the previous section. Writing $`𝚫^1\left(K\right)`$ in terms of our tensor basis

$$𝚫^1\left(K\right)=\left(k^2\omega ^2+\alpha \right)𝐀+\left(\beta \omega ^2\right)𝐁+\gamma 𝐂+\delta 𝐃$$
(31)
and applying the inversion formula (20) we obtain an expression for the propagator

$$𝚫\left(K\right)=\mathrm{\Delta }_A𝐀+\left(k^2\omega ^2+\alpha +\gamma \right)\mathrm{\Delta }_G𝐁+\left[\left(\beta \omega ^2\right)\mathrm{\Delta }_G\mathrm{\Delta }_A\right]𝐂\delta \mathrm{\Delta }_G𝐃,$$
(32)
with

$`\mathrm{\Delta }_A^1\left(K\right)`$ $`=`$ $`k^2\omega ^2+\alpha ,`$ (33)
$`\mathrm{\Delta }_G^1\left(K\right)`$ $`=`$ $`\left(k^2\omega ^2+\alpha +\gamma \right)\left(\beta \omega ^2\right)k^2\stackrel{~}{n}^2\delta ^2.`$ (34)
Note that we can reorganize $`\mathrm{\Delta }\left(K\right)`$ and write it as

$$\mathrm{\Delta }\left(K\right)=\mathrm{\Delta }_A\left[𝐀𝐂\right]+\mathrm{\Delta }_G\left[\left(k^2\omega ^2+\alpha +\gamma \right)𝐁+\left(\beta \omega ^2\right)𝐂\delta 𝐃\right].$$
(35)
## V Static Limit

In order to see how the momentum-space anisotropy in the distribution functions affects the response to static electric and magnetic fluctuations we examine the limit $`\omega 0`$ of the propagators (33) and (34). Approaching along the real $`\omega `$ axis we find that to leading order $`\alpha \gamma O\left(\omega ^0\right)`$, $`\beta O\left(\omega ^2\right)`$, and $`\delta O\left(i\omega \right).`$111Identical results can be obtained by coming in along the imaginary axis with a suitable redefinition of $`m_\delta ^2`$. We can therefore define four mass scales

$`m_\alpha ^2`$ $`=`$ $`\underset{\omega 0}{lim}\alpha ,`$
$`m_\beta ^2`$ $`=`$ $`\underset{\omega 0}{lim}{\displaystyle \frac{k^2}{\omega ^2}}\beta ,`$
$`m_\gamma ^2`$ $`=`$ $`\underset{\omega 0}{lim}\gamma ,`$
$`m_\delta ^2`$ $`=`$ $`\underset{\omega 0}{lim}{\displaystyle \frac{\stackrel{~}{n}k^2}{\omega }}\mathrm{Im}\delta .`$ (36)
Writing the static limit of the propagators (33) and (34) in terms of these masses gives

$`\mathrm{\Delta }_A^1`$ $`=`$ $`k^2+m_\alpha ^2`$ (37)
$`\mathrm{\Delta }_G^1`$ $`=`$ $`{\displaystyle \frac{\omega ^2}{k^2}}\left[\left(k^2+m_\alpha ^2+m_\gamma ^2\right)\left(k^2+m_\beta ^2\right)m_\delta ^4\right].`$ (38)
$`\mathrm{\Delta }_G^1`$ can be factorized into

$$\mathrm{\Delta }_G^1=\frac{\omega ^2}{k^2}\left(k^2+m_+^2\right)\left(k^2+m_{}^2\right),$$
(39)
where

$$2m_\pm ^2=M^2\pm \sqrt{M^44\left(m_\beta ^2\left(m_\alpha ^2+m_\gamma ^2\right)m_\delta ^4\right)},$$
(40)
with

$$M^2=m_\alpha ^2+m_\beta ^2+m_\gamma ^2.$$
(41)
In the isotropic limit, $`\xi 0`$, $`m_\alpha ^2=m_\gamma ^2=m_\delta ^2=m_{}^2=0`$ and $`m_+^2=m_D^2`$.

For finite $`\xi `$ it is possible to evaluate all four masses defined above analytically. The results for $`m_\alpha `$ and $`m_\beta `$ are listed in Appendix B. In Fig. 2 we plot the angular dependence of $`m_\alpha ^2`$, $`m_+^2`$, and $`m_{}^2`$ at fixed $`\xi =10`$ and $`\xi =0.9`$. In the case $`\xi >0`$ (Fig. 2a) we see that for small $`\theta _n`$ the scale $`m_+^2m_D^2`$ and for $`\theta _n`$ near $`\pi /2`$, $`m_+^2m_D^2`$. For small $`\theta _n`$ the scales $`m_\alpha ^2`$ and $`m_{}^2`$ are negative. The fact that $`m_\alpha `$ and $`m_{}`$ are non-vanishing is in agreement with the findings of Cooper et al. CKN:2002 ; however, they neglected to consider the fact that these masses might be negative and would therefore not correspond to screening of the magnetic interaction. The fact that these quantities are negative indicates that for $`\xi >0`$ the system possesses an instability to transverse and “mixed” external perturbations associated with $`m_\alpha ^2`$ and $`m_{}^2`$, respectively. The transverse instability is present for any $`\theta _n\pi /2`$ while the mixed instability is only present for $`\theta _n<\theta _c^{\mathrm{mixed}}`$ with $`\theta _c^{\mathrm{mixed}}`$ depending on the value of $`\xi `$. In the case $`\xi <0`$ (Fig. 2b) we see that for small $`\theta _n`$ the scale $`m_+^2m_D^2`$ and for $`\theta _n`$ near $`\pi /2`$, $`m_+^2m_D^2`$. For $`\theta _n\pi /4`$ the scale $`m_{}^2`$ is negative again signaling the presence of an instability in the system. In the next section we will discuss these instabilities in more detail.

## VI Collective modes

A similar factorization of $`\mathrm{\Delta }_G^1`$ can be achieved in the non-static case allowing us to determine the dispersion relations for all of the collective modes in the system.

### VI.1 Stable Modes

First, let’s consider the stable collective modes which have poles at real-valued $`\omega >k`$. In this case we factorize $`\mathrm{\Delta }_G^1`$ as

$$\mathrm{\Delta }_G^1=\left(\omega ^2\mathrm{\Omega }_+^2\right)\left(\omega ^2\mathrm{\Omega }_{}^2\right),$$
(42)
where

$$2\mathrm{\Omega }_\pm ^2=\overline{\mathrm{\Omega }}^2\pm \sqrt{\overline{\mathrm{\Omega }}^44\left(\left(\alpha +\gamma +k^2\right)\beta k^2\stackrel{~}{n}^2\delta ^2\right)},$$
(43)
and

$$\overline{\mathrm{\Omega }}^2=\alpha +\beta +\gamma +k^2.$$
(44)
Note that the quantity under the square root in (43) can be written as $`\left(\alpha \beta +\gamma +k^2\right)^2+4k^2\stackrel{~}{n}^2\delta ^2`$ which is always positive for real $`\omega >k`$. Therefore there are at most two stable modes coming from $`\mathrm{\Delta }_G`$.

The remaining stable collective mode comes from the zero of $`\mathrm{\Delta }_A^1`$. The dispersion relations for all of the collective modes can be determined by finding the solutions to

$`\omega _\pm ^2`$ $`=`$ $`\mathrm{\Omega }_\pm ^2\left(\omega _\pm \right),`$ (45)
$`\omega _\alpha ^2`$ $`=`$ $`k^2+\alpha \left(\omega _\alpha \right).`$ (46)
In the isotropic limit (28) $`\omega _\alpha =\omega _+=\omega _T`$ and $`\omega _{}=\omega _L`$. For finite $`\xi `$ there are three stable quasiparticle modes with dispersion relations which depend on the angle of propagation with respect to the anisotropy vector, $`\theta _n`$. In Fig. 3 we plot the dispersion relations for all three modes for $`m_D=\sqrt{3}`$, $`\xi =10`$, and $`\theta _n=\{0,\pi /4,\pi /2\}`$.

### VI.2 Unstable Modes

For non-zero $`\xi `$ the propagator also has poles along the imaginary $`\omega `$ axis.222We have checked for poles at complex $`\omega `$ numerically but found none. The dispersion relation for these modes can be determined by taking $`\omega i\mathrm{\Gamma }`$ with $`\mathrm{\Gamma }`$ real-valued and solving for $`\mathrm{\Gamma }\left(k\right)`$. In this case we factorize the inverse propagator as

$$\mathrm{\Delta }_G^1=\left(\mathrm{\Gamma }^2+\mathrm{\Omega }_+^2\right)\left(\mathrm{\Gamma }^2+\mathrm{\Omega }_{}^2\right),$$
(47)
where $`\mathrm{\Omega }_\pm `$ on the right hand side are evaluated at $`\omega =i\mathrm{\Gamma }`$. However, in contrast to the stable modes there is at most one solution in this case since numerically we find that $`\mathrm{\Omega }_+^2>0`$ for all $`\mathrm{\Gamma }>0`$.

For $`\xi >0`$ there is also an unstable mode present in $`\mathrm{\Delta }_A`$ so that in this case there are two unstable modes in the system which can be found by solving

$`\mathrm{\Gamma }_{}^2`$ $`=`$ $`\mathrm{\Omega }_{}^2\left(i\mathrm{\Gamma }_{}\right),`$ (48)
$`\mathrm{\Gamma }_\alpha ^2`$ $`=`$ $`k^2\alpha \left(i\mathrm{\Gamma }_\alpha \right).`$ (49)
Note that in both cases there are two solutions corresponding to modes with positive and negative growth rates. One of these corresponds to an exponentially growing solution and the other an exponentially decaying one. In Fig. 4a we plot $`\mathrm{\Gamma }_\alpha \left(k\right)`$ and $`\mathrm{\Gamma }_{}\left(k\right)`$ with $`\xi =10`$ and $`\theta _n=\pi /8`$. For $`\xi <0`$ there is no longer an unstable mode coming from $`\mathrm{\Delta }_A`$ and there is therefore only one unstable mode coming from $`\mathrm{\Gamma }_{}`$. In Fig. 4b we plot $`\mathrm{\Gamma }_{}\left(k\right)`$ with $`\xi =0.9`$ and $`\theta _n=\pi /2`$.

## VII Small $`\xi `$ expansion

In the small-$`\xi `$ limit it is possible to obtain analytic expressions for all of the structure functions order-by-order in $`\xi `$. To linear order in $`\xi `$

$`\alpha `$ $`=`$ $`\mathrm{\Pi }_T\left(z\right)+\xi [{\displaystyle \frac{z^2}{12}}(3+5\mathrm{cos}2\theta _n)m_D^2{\displaystyle \frac{1}{6}}(1+\mathrm{cos}2\theta _n)m_D^2`$
$`+{\displaystyle \frac{1}{4}}\mathrm{\Pi }_T\left(z\right)((1+3\mathrm{cos}2\theta _n)z^2(3+5\mathrm{cos}2\theta _n))],`$
$`z^2\beta `$ $`=`$ $`\mathrm{\Pi }_L\left(z\right)+\xi \left[{\displaystyle \frac{1}{6}}\left(1+3\mathrm{cos}2\theta _n\right)m_D^2+\mathrm{\Pi }_L\left(z\right)\left(\mathrm{cos}2\theta _n{\displaystyle \frac{z^2}{2}}\left(1+3\mathrm{cos}2\theta _n\right)\right)\right],`$
$`\gamma `$ $`=`$ $`{\displaystyle \frac{\xi }{3}}\left(3\mathrm{\Pi }_T\left(z\right)m_D^2\right)\left(z^21\right)\mathrm{sin}^2\theta _n,`$
$`\delta `$ $`=`$ $`{\displaystyle \frac{\xi }{3k}}\left(4z^2m_D^2+3\mathrm{\Pi }_T\left(z\right)\left(14z^2\right)\right)\mathrm{cos}\theta _n,`$ (50)
where $`z=\omega /k`$.

### VII.1 Static Limit

Using the linear expansions and the fact that in the static limit $`\mathrm{\Pi }_Lm_D^2`$ and $`\mathrm{\Pi }_Ti\pi \omega /\left(4k\right)`$ we can write for the masses (36)

$`\widehat{m}_\alpha ^2`$ $`=`$ $`{\displaystyle \frac{\xi }{6}}\left(1+\mathrm{cos}2\theta _n\right),`$
$`\widehat{m}_\beta ^2`$ $`=`$ $`1+{\displaystyle \frac{\xi }{6}}\left(3\mathrm{cos}2\theta _n1\right),`$
$`\widehat{m}_\gamma ^2`$ $`=`$ $`{\displaystyle \frac{\xi }{3}}\mathrm{sin}^2\theta _n,`$
$`\widehat{m}_\delta ^2`$ $`=`$ $`\xi {\displaystyle \frac{\pi }{4}}\mathrm{sin}\theta _n\mathrm{cos}\theta _n,`$ (51)
where $`\widehat{m}^2=m^2/m_D^2`$. Using these we can obtain small-$`\xi `$ expressions for $`m_\pm `$ defined in (40)

$`\widehat{m}_+^2`$ $`=`$ $`1+{\displaystyle \frac{\xi }{6}}\left(3\mathrm{cos}2\theta _n1\right),`$
$`\widehat{m}_{}^2`$ $`=`$ $`{\displaystyle \frac{\xi }{3}}\mathrm{cos}2\theta _n.`$ (52)
### VII.2 Collective Modes

Since $`\delta `$ is $`O\left(\xi \right)`$ it can be ignored in the expansion of (43) so that to linear order in $`\xi `$ the collective modes satisfy

$`\mathrm{\Delta }_A^1`$ $`=`$ $`k^2\omega ^2+\alpha =0`$
$`\mathrm{\Delta }_G^1`$ $`=`$ $`\left(k^2\omega ^2+\alpha +\gamma \right)\left(\beta \omega ^2\right)=0,`$ (53)
where $`\alpha `$, $`\beta `$, and $`\gamma `$ are given by (50). Note again that there is only one unstable mode coming from $`\mathrm{\Delta }_G^1`$ since $`\beta \left(i\mathrm{\Gamma }\right)>0`$ for all $`\mathrm{\Gamma }>0`$.

## VIII Conclusions

In this paper we have derived a tensor basis for the gluon self-energy in a high-temperature quark gluon plasma with an anisotropic momentum-space distribution. We then restricted the distribution function by requiring that it could be obtained from an isotropic distribution function by the rescaling of one direction specified by an anisotropy vector, $`\widehat{𝐧}`$, and strength, $`\xi `$. Positive values of $`\xi `$ correspond to a contraction of the isotropic distribution function along $`\widehat{𝐧}`$ while negative values of $`\xi `$ correspond to a stretching along $`\widehat{𝐧}`$. Within this framework we could derive analytic forms for all of the structure functions associated with the tensor basis. Using these expressions we were then able to identify and determine the dispersion relations for the collective modes for both positive and negative $`\xi `$. We found that for $`\xi >0`$ there were at most three stable and two unstable modes with dispersion relations which depended on the angle between the wave vector, $`𝐤`$, and the anisotropy vector. For $`\xi <0`$ we found that there were also three stable modes but only one unstable mode. Additionally, we obtained analytic expressions for the structure functions in the limit of small $`\xi `$. These results should provide a reference point for the systematic study of the isotropization of a relativistic plasma.

The study of Mrówczyński and Randrup suggests that during heavy-ion collisions the rate of isotropization via collective modes is comparable with collisions and therefore cannot be ignored RM:2003 . In this paper we have made no attempt to discuss the phenomenological rate for instability growth because there are a number of questions which would need to be addressed prior to making any definitive statements about the role of instabilities in plasma evolution and their expected contribution to observables. This is because we have only derived the self-energy in a linear expansion in the fluctuations and to leading-order in the coupling constant. Assuming that there is truly exponential growth of the fields in the direction of the anisotropy this means that the linear approximation will break down very quickly. In practice the non-linear terms in the transport equations will become important and regulate the growth of the modes which have become unstable.

Within electrodynamics the coupling constant is small and it is possible to experimentally study the Weibel instability CLF:2002 . However, with QCD the story is dramatically different since for experimentally realizable situations the coupling constant is large and the non-linear effects due to gluon self-interaction become important much sooner than any non-linear effects would for QED. Nevertheless, this does not diminish from the fact that these unstable modes exist and will therefore have a role to play in plasma evolution. In order to assess this role, however, detailed studies of the time evolution of anisotropic quark-gluon plasmas will need to be performed.

## Acknowledgments

M.S. and P.R. would like to thank S. Mrówczyński and A. Rebhan for discussions. M.S. was supported by an Austrian Science Fund (FWF) Lise Meitner fellowship M689. P.R. was supported by the Austrian Science Fund Project No. P14632.

## Appendix A Notation and Conventions

We summarize here the notation and conventions which we use in the main body of the text.

* Natural units: $`\mathrm{}=c=1`$
* Metric: $`g^{\mu \nu }=g_{\mu \nu }=\mathrm{diag}(1,1,1,1)`$
* 4-vectors: Indicated by Greek indices, e.g. $`K^\mu =(\omega ,𝐤)`$.
* 3-vectors: Indicated by lowercase Latin characters. Upper Latin indices like $`i,j,k`$ use a Euclidean 3-metric, e.g. $`𝐤=k^i`$, $`k^ik^i=𝐤^2`$.
* Fourier transform:

$`j^\mu \left(K\right)`$ $`=`$ $`{\displaystyle d^4Xe^{iKX}j^\mu \left(X\right)}`$
$`j^\mu \left(X\right)`$ $`=`$ $`{\displaystyle \frac{d^4K}{\left(2\pi \right)^4}e^{iKX}j^\mu \left(K\right)}`$

## Appendix B Analytic expressions for structure functions

In this appendix we collect the integral and analytic expressions for the structure functions $`\alpha `$, $`\beta `$, $`\gamma `$, and $`\delta `$. Choosing $`𝐧=\widehat{𝐳}`$ and $`𝐤`$ to lie in the $`xz`$ plane ($`k_x/k_z=\mathrm{tan}\theta _n`$) we have $`𝐯𝐧=\mathrm{cos}\theta `$ and $`𝐯𝐤=k_x\mathrm{cos}\varphi \mathrm{sin}\theta +k_z\mathrm{cos}\theta `$. Using this parameterization the $`\varphi `$ integration in all four structure functions defined by the contractions in Eq. (27) can be performed analytically.

$`\alpha (K,\xi )`$ $`=`$ $`{\displaystyle \frac{m_D^2}{k^2\stackrel{~}{n}^2}}{\displaystyle }{\displaystyle \frac{d\left(\mathrm{cos}\theta \right)}{2}}{\displaystyle \frac{\omega +\xi k_z\mathrm{cos}\theta }{\left(1+\xi \left(\mathrm{cos}\theta \right)^2\right)^2}}[\omega k_z\mathrm{cos}\theta `$ (B.54)
$`+k^2(s^2(\mathrm{cos}\theta {\displaystyle \frac{\omega k_z}{k^2}})^2)R(\omega k_z\mathrm{cos}\theta ,k_x\mathrm{sin}\theta )],`$
$`\beta (K,\xi )`$ $`=`$ $`{\displaystyle \frac{m_D^2\omega ^2}{k^2}}{\displaystyle \frac{d\left(\mathrm{cos}\theta \right)}{2}\frac{1}{\left(1+\xi \left(\mathrm{cos}\theta \right)^2\right)^2}\left[1\left(\omega +\xi k_z\mathrm{cos}\theta \right)R(\omega k_z\mathrm{cos}\theta ,k_x\mathrm{sin}\theta )\right]},`$ (B.55)
$`\gamma (K,\xi )`$ $`=`$ $`m_D^2{\displaystyle }{\displaystyle \frac{d\left(\mathrm{cos}\theta \right)}{2}}{\displaystyle \frac{1}{k^2\left(1+\xi \mathrm{cos}^2\theta \right)^2}}[\omega ^2+\xi k^2\mathrm{cos}^2\theta 2{\displaystyle \frac{k^2}{k_x^2}}(\omega ^2\xi k_z^2\mathrm{cos}^2\theta )`$ (B.56)
$`+{\displaystyle \frac{\left(\omega +\xi k_z\mathrm{cos}\theta \right)k^4}{k_x^2}}(2(\mathrm{cos}\theta {\displaystyle \frac{\omega k_z}{k^2}})^2s^2)R(\omega k_z\mathrm{cos}\theta ,k_x\mathrm{sin}\theta )],`$
$`\delta (K,\xi )`$ $`=`$ $`{\displaystyle \frac{m_D^2\omega }{k^4\stackrel{~}{n}^2}}{\displaystyle \frac{d\left(\mathrm{cos}\theta \right)}{2}\frac{\omega +\xi k_z\mathrm{cos}\theta }{\left(1+\xi \mathrm{cos}^2\theta \right)^2}\left(k_z+\left(k^2\mathrm{cos}\theta \omega k_z\right)R(\omega k_z\mathrm{cos}\theta ,k_x\mathrm{sin}\theta )\right)},`$ (B.57)
where $`s^2=\left(1\omega ^2/k^2\right)\left(k_x^2/k^2\right)`$ and

$$R(a,b)=_0^{2\pi }\frac{d\varphi }{2\pi }\frac{1}{ab\mathrm{cos}\varphi +iϵ}=\frac{1}{\sqrt{a+b+iϵ}\sqrt{ab+iϵ}}.$$
(B.58)
When $`a`$ and $`b`$ are real-valued $`R`$ can be simplified to

$$R(a,b)=\frac{\mathrm{sgn}\left(\mathrm{a}\right)\mathrm{\Theta }\left(\mathrm{a}^2\mathrm{b}^2\right)}{\sqrt{a^2b^2}}\frac{i\mathrm{\Theta }\left(b^2a^2\right)}{\sqrt{b^2a^2}},$$
(B.59)
with $`\mathrm{\Theta }\left(x\right)`$ being the usual step-function. Note that the remaining integration over $`\theta `$ can also be done analytically but the results are rather unwieldy so we do not list them here.

## Static Limit

In the limit $`\omega 0`$ it is possible to obtain analytic expressions for all four structure functions. The results for $`m_\alpha `$ and $`m_\beta `$ defined in Eq. (36) are

$`m_\alpha ^2`$ $`=`$ $`{\displaystyle \frac{m_D^2}{2k_x^2\sqrt{\xi }}}\left(k_z^2\mathrm{arctan}\sqrt{\xi }{\displaystyle \frac{\mathrm{k}_\mathrm{z}\mathrm{k}^2}{\sqrt{\mathrm{k}^2+\xi \mathrm{k}_\mathrm{x}^2}}}\mathrm{arctan}{\displaystyle \frac{\sqrt{\xi }\mathrm{k}_\mathrm{z}}{\sqrt{\mathrm{k}^2+\xi \mathrm{k}_\mathrm{x}^2}}}\right),`$ (B.60)
$`m_\beta ^2`$ $`=`$ $`m_D^2{\displaystyle \frac{\left(\sqrt{\xi }+\left(1+\xi \right)\mathrm{arctan}\sqrt{\xi }\right)\left(\mathrm{k}^2+\xi \mathrm{k}_\mathrm{x}^2\right)+\xi \mathrm{k}_\mathrm{z}\left(\mathrm{k}_\mathrm{z}\sqrt{\xi }+\frac{\mathrm{k}^2\left(1+\xi \right)}{\sqrt{\mathrm{k}^2+\xi \mathrm{k}_\mathrm{x}^2}}\mathrm{arctan}\frac{\sqrt{\xi }\mathrm{k}_\mathrm{z}}{\sqrt{\mathrm{k}^2+\xi \mathrm{k}_\mathrm{x}^2}}\right)}{2\sqrt{\xi }\left(1+\xi \right)\left(k^2+\xi k_x^2\right)}},`$ (B.61)
with similar results for $`m_\gamma ^2`$ and $`m_\delta ^2`$.