format: text

text:
# 1 From Maurer-Cartan equations to Moment maps

## 1 From Maurer-Cartan equations to Moment maps

### 1.1

Let $`\mathrm{𝕜}`$ denote the field of either real or complex numbers. Let $`𝔊=𝔊_+𝔊_{}`$ be a differential $`/2`$-graded Lie (super)-algebra (DGLA) over $`\mathrm{𝕜}`$. Write $`d:𝔊_\pm 𝔊_{}`$ for the differential, and $`H_{\text{}}(𝔊,d):=\mathrm{Ker}d/\mathrm{Im}d`$ for the corresponding homology space, which is again $`/2`$-graded Lie super-algebra.

Depending on the problem, the space $`𝔊`$ may be either finite or infinite dimensional. To fix ideas, we shall assume below that $`𝔊`$ has finite dimension over $``$; moreover, it will be assumed that $`𝔊_+=\mathrm{Lie}G_+`$ is the Lie algebra of a complex connected simply-connected linear algebraic group $`G_+`$. These assumptions, though certainly too restrictive, will allow us to make the main ideas more clear. In reality, the algebra $`𝔊`$ itself is typically infinite-dimensional while the homology algebra, $`H_{\text{}}(𝔊,d),`$ is typically finite-dimensional. In such cases, the space $`𝔊`$ usually comes equipped with a natural topology. Various analytic issues (e.g., the closedness of the kernel and image of the differential $`d`$) that arise in such a topological framework require a considerable amount of machinery and are beyond the scope of this short paper.

### 1.2

The algebraic group $`G_+`$ acts on its Lie algebra $`𝔊_+`$ via the adjoint action. Fix an $`\mathrm{𝙰𝚍}^{}G_+`$-orbit $`𝕆𝔊_+`$, and consider the following (locally closed) subscheme in $`𝔊_{}`$:

$$\mathrm{𝖬𝖢}(𝔊,𝕆):=\left\{x𝔊_{}\right|dx+\frac{1}{2}[x,x]𝕆\}.$$

The equation $`dx+\frac{1}{2}[x,x]=0`$ is known as the Maurer-Cartan equation, so we call $`\mathrm{𝖬𝖢}(𝔊,𝕆)`$ the Maurer-Cartan scheme 111It is actually a DG scheme, cf. \[CFK\], the fact that will be exploited later. associated to an orbit $`𝕆`$. If $`𝕆=\{0\}`$ is a one-point orbit, then our scheme $`\mathrm{𝖬𝖢}(𝔊,𝕆)`$ reduces to the zero-scheme of the above-mentioned standard Maurer-Cartan equation.

Given $`a𝔊_+`$, let $`\xi _a`$ denote an affine-linear algebraic vector field on $`𝔊_{}`$ whose value at the point $`x𝔊_{}`$ is $`\xi _a(x):=[a,x]da`$. The lemma below is well-known, see e.g. \[GM, sect. 1.3\].

###### Lemma 1.2.1.

$`(𝗂)`$The map $`a\xi _a`$ is a Lie algebra homomorphism.

$`(\mathrm{𝗂𝗂})`$For any orbit $`𝕆𝔊_+`$ and any $`a𝔊_+,`$ the vector field $`\xi _a`$ is tangent to the Maurer-Cartan scheme $`\mathrm{𝖬𝖢}(𝔊,𝕆)𝔊_{}`$.

It follows from part (i) of the Lemma that, exponentiating the vector fields $`\xi _a,a𝔊_+`$, one obtains an action of the group $`G_+`$ on $`𝔊_{}`$ by affine-linear transformations. This $`G_+`$-action on $`𝔊_{}`$ is known as the gauge action (it should not be confused with the ordinary $`\mathrm{𝙰𝚍}^{}G_+`$-action on $`𝔊_{}`$). We observe also that (by the Lemma) the map $`a\xi _a`$ intertwines the $`\mathrm{𝙰𝚍}^{}G_+`$-action on $`𝔊_+`$ with $`G_+`$-action on vector fields induced by the gauge action on $`𝔊_{}`$. Further, part (ii) of the Lemma implies that the Maurer-Cartan scheme $`\mathrm{𝖬𝖢}(𝔊,𝕆)`$ is stable under the gauge action of $`G_+`$.

###### Definition 1.2.2.

We write $`(𝔊,𝕆):=\mathrm{𝖬𝖢}(𝔊,𝕆)/G_+`$ for the stack-quotient of $`\mathrm{𝖬𝖢}(𝔊,𝕆)`$ by the gauge $`G_+`$-action, cf. \[LMB\],\[To\], and call $`(𝔊,𝕆)`$ the modular stack attached to the orbit $`𝕆𝔊_+`$.

###### Remark 1.2.3.

Usually, one has a natural $``$-grading $`𝔊=_i𝔊_i`$, making $`𝔊`$ a DGLA with differential $`𝔊_{\text{}}𝔊_{+1}`$. We then put $`𝔊_+=_i𝔊_{2i}`$ and $`𝔊_{}=_i𝔊_{2i+1}`$. In such a case, one introduces a smaller Lie group $`G_0G_+`$ corresponding to the Lie subalgebra $`𝔊_0𝔊_+`$. This group acts naturally on $`𝔊_2`$, and for any $`G_0`$-orbit $`𝕆_2𝔊_2`$ we may form the corresponding $`𝔊_+`$-orbit $`𝕆=𝔊_+(𝕆_2)`$. Define

$$\mathrm{𝖬𝖢}_1(𝔊,𝕆):=\left\{x𝔊_1\right|dx+\frac{1}{2}[x,x]𝕆_2\}.$$

It is clear that, for $`x𝔊_1`$ we have: $`dx+\frac{1}{2}[x,x]𝕆_2`$ if and only if $`dx+\frac{1}{2}[x,x]𝕆`$. This way the modular stack $`_1(𝔊,𝕆):=\mathrm{𝖬𝖢}_1(𝔊,𝕆)/G_0`$ becomes a (locally-closed) substack in $`(𝔊,𝕆)`$.

### 1.3

Let $`E=_iE^i`$ be a DG vector space with differential $`d:E^{\text{}}E^{\text{}+1}`$, and $`(,):E\times E`$ a $``$-bilinear form such that for any homogeneous $`x,yE`$ we have

$$(dx,y)+(1)^{\mathrm{deg}x}(x,dy)=0,$$
(1.3.1)
and moreover $`E^iE^j`$ whenever $`i+j0.`$

Put $`E_i^{}:=(E^i)^{},`$ the dual of $`E^i,`$ and let $`E^{}:=_iE_i^{}.`$ Dualizing the map $`d`$ makes $`E^{}`$ a DG vector space. The assignment $`x(x,)`$ gives rise to a morphism of DG vector spaces $`\kappa :EE^{}`$. We say that the form $`(,)`$ is non-degenerate provided the morphism $`\kappa `$ induces an isomorphism $`H^{\text{}}(\kappa ):H^{\text{}}(E)\stackrel{_{}}{}H^{\text{}}(E^{})`$ on cohomology.

### 1.4

Now, let $`𝔊`$ be a DGLA, as in §§1.1-1.2, and let $`\beta (,)`$ be an even nondegenerate invariant bilinear form on $`𝔊`$, that is, a $``$-bilinear form $`𝔊\times 𝔊`$ that restricts to a symmetric, resp. skew-symmetric, form on $`𝔊_+`$, resp. on $`𝔊_{},`$ such that $`𝔊_+𝔊_{}`$, (1.3.1) holds and for any homogeneous $`x,y,z𝔊,`$ one has:

$$\beta ([x,y],z)=\beta (x,[y,z]).$$
(1.4.1)
###### Theorem 1.4.2.

Let $`𝔊`$ be a DGLA with an even nondegenerate invariant bilinear form $`\beta `$, and $`𝕆𝔊_+`$ an $`\mathrm{𝙰𝚍}^{}G_+`$-orbit. Then, for any $`x(𝔊,𝕆)`$, the form $`\beta `$ induces a non-degenerate 2-form on $`𝖳_x(𝔊,𝕆)`$, the tangent space (at $`x`$) to the modular stack.222See explanation in §1.6 below. These 2-forms give rise to a symplectic structure on the stack $`(𝔊,𝕆)`$.

Thus, a DGLA with an even nondegenerate invariant bilinear form gives rise to symplectic stacks.

###### Remark 1.4.3.

Morally, the main message of this paper is that all natural symplectic structures on moduli spaces ‘arising in nature’ come from an appropriate (even) nondegenerate invariant bilinear form on the DGLA that controls the moduli problem in question.

We will now show that the construction of the modular stack $`(𝔊,𝕆)`$ is a special case of the Hamiltonian reduction. That will yield the proof of Theorem 1.4.2.

### 1.5 Hamiltonian reduction construction.

In the setup of Theorem 1.4.2, let $`\omega :=\beta |_𝔊_{}`$ and consider $`𝔊_{}`$ as a symplectic vector space equipped with the symplectic form $`\omega `$. Consider also the gauge action of the group $`G_+`$ on $`𝔊_{}`$. This is a symplectic action, and we claim that it has moment map

$$\mathrm{\Phi }:𝔊_{}𝔊_+^{}\stackrel{\beta }{}𝔊_+:xdx+\frac{1}{2}[x,x].$$

To this end, note that the differential of the map $`\mathrm{\Phi }`$ at the point $`x𝔊_{}`$ is given by

$$\mathrm{\Phi }_x^{}:y\mathrm{\Phi }_x^{}(y)=dy+[x,y]y𝔊_{}T_x𝔊_{}.$$

A direct computation yields $`\mathrm{\Phi }_x^{}(\xi _a(x))=[a,\mathrm{\Phi }(x)]a𝔊_+.`$ It follows that $`\mathrm{\Phi }`$ is $`G_+`$-equivariant. Next, by (1.4.1), we have

$$\omega (\xi _a(x),y)=\beta (a,\mathrm{\Phi }_x^{}(y))a𝔊_+,x,y𝔊_{}.$$
(1.5.1)
It follows from the last equation combined with the $`G_+`$-equivariance of $`\mathrm{\Phi }`$ that the $`G_+`$-action is Hamiltonian with moment map $`\mathrm{\Phi }`$. Thus, $`(𝔊,𝕆)`$ is precisely the Hamiltonian reduction of $`𝔊_{}`$ with respect to the orbit $`𝕆𝔊_+𝔊_+^{}`$, see \[AM\].

### 1.6

In this subsection we explain the meaning of the word ‘nondegenerate’ in Theorem 1.4.2. We will see that this is closely related to the ‘self-dual nature’ of the Hamiltonian reduction construction.

In general, let $`(X,\omega )`$ be a smooth symplectic variety equipped with a Hamiltonian action of an algebraic group $`G`$. Let $`𝔤:=\mathrm{Lie}G`$, fix a coadjoint orbit $`𝕆𝔤^{}`$, and write $`\mathrm{\Phi }:X𝔤^{}`$ for the moment map.

We consider the scheme $`\mathrm{\Sigma }:=\mathrm{\Phi }^1(𝕆)`$. If $`\mathrm{\Phi }^{}`$, the differential of $`\mathrm{\Phi }`$, is surjective at any point $`xX`$ such that $`\mathrm{\Phi }(x)𝕆`$, then $`\mathrm{\Sigma }`$ is a smooth (locally closed) subscheme of $`X`$. In general, if $`\mathrm{\Phi }^{}`$ is not necessarily surjective, it is natural to view $`\mathrm{\Sigma }`$ as a DG scheme, which we denote $`\mathrm{\Sigma }_{_{\mathrm{DG}}}`$. The tangent space to this DG scheme at a closed point $`x\mathrm{\Sigma }_{_{\mathrm{DG}}}`$ is a DG vector space $`𝖳_x\mathrm{\Sigma }_{_{\mathrm{DG}}}=T_x^0\mathrm{\Sigma }_{_{\mathrm{DG}}}T_x^1\mathrm{\Sigma }_{_{\mathrm{DG}}},`$ (concentrated in degrees $`0`$ and $`1`$), where

$`T_x^0\mathrm{\Sigma }_{_{\mathrm{DG}}}:=T_xX,`$ $`T_x^1\mathrm{\Sigma }_{_{\mathrm{DG}}}:=𝔤^{},`$ (1.6.1)
$`\text{with differential}T_x^0\mathrm{\Sigma }_{_{\mathrm{DG}}}=T_xX\stackrel{d}{}𝔤^{}=T_x^1\mathrm{\Sigma }_{_{\mathrm{DG}}},`$
given by the map $`\mathrm{\Phi }_x^{}:T_xX𝔤^{}`$. We write $`H^{\text{}}(𝖳_x\mathrm{\Sigma }_{_{\mathrm{DG}}})`$ for the cohomology groups of the two-term complex above. We have $`H^0(𝖳_x\mathrm{\Sigma }_{_{\mathrm{DG}}})=\mathrm{Ker}\mathrm{\Phi }_x^{}`$, is the Zariski tangent space to $`\mathrm{\Sigma }_{_{\mathrm{DG}}}`$ (viewed as an ordinary scheme $`\mathrm{\Sigma }`$), and $`H^1(𝖳_x\mathrm{\Sigma }_{_{\mathrm{DG}}})=𝔤^{}/\mathrm{Im}\mathrm{\Phi }_x^{}`$, is the space measuring the failure of the differential of $`\mathrm{\Phi }`$ to be surjective.

Next, we perform the Hamiltonian reduction and consider the quotient $`\mathrm{\Sigma }/G`$ (where $`\mathrm{\Sigma }`$ is an ordinary scheme, not a DG scheme). If the $`G`$-action on $`\mathrm{\Sigma }`$ is free, then this quotient is a well-defined scheme again. In general, for a not necessarily free action, it is natural to consider $`:=\mathrm{\Sigma }/G`$ as a stack, cf. \[LMB\],\[To\]. Given $`x\mathrm{\Sigma }`$, write $`\overline{x}`$ for the corresponding point in $``$. Then, the tangent space at $`\overline{x}`$ to the stack $`=\mathrm{\Sigma }/G`$ is a DG vector space $`𝖳_{\overline{x}}=T_{\overline{x}}^1T_{\overline{x}}^0`$ (concentrated in degrees $`1`$ and $`0`$), where

$`T_{\overline{x}}^1:=𝔤,`$ $`T_{\overline{x}}^0:=T_x\mathrm{\Sigma },`$ (1.6.2)
$`\text{with differential}T_{\overline{x}}^1=𝔤\stackrel{d}{}T_x\mathrm{\Sigma }=T_{\overline{x}}^0,`$
given by the derivative (at $`1G`$) of the action-map $`gg(x)`$. Then, for the cohomology groups, we have $`H^0(𝖳_{\overline{x}})=T_x\mathrm{\Sigma }/𝔤x,`$ is the normal space to the $`G`$-orbit through $`x\mathrm{\Sigma }`$, and $`H^1(𝖳_{\overline{x}})=𝔤^x`$ is the Lie algebra of the isotropy group of the point $`x`$, that measures the failure of the $`𝔤`$-action on $`\mathrm{\Sigma }`$ to be infinitesimally-free.

Now, if we view (as has been explained earlier) $`\mathrm{\Sigma }_{_{\mathrm{DG}}}:=\mathrm{\Phi }^1(𝕆)`$ as a DG scheme rather than an ordinary scheme, then the quotient $`_{_{\mathrm{DG}}}:=\mathrm{\Sigma }_{_{\mathrm{DG}}}/G`$ becomes a DG stack rather than an ordinary stack. To get the tanget space of this DG stack, we must combine formulas (1.6.1) and (1.6.2) together. Thus, the tangent space to $`_{_{\mathrm{DG}}}`$ is a DG vector space $`𝖳_{\overline{x}}_{_{\mathrm{DG}}}=T_{\overline{x}}^1_{_{\mathrm{DG}}}T_{\overline{x}}^0_{_{\mathrm{DG}}}T_{\overline{x}}^1_{_{\mathrm{DG}}},`$ concentrated in degrees $`1,0,1`$, such that the corresponding 3-term complex $`T_{\overline{x}}^1_{_{\mathrm{DG}}}T_{\overline{x}}^0_{_{\mathrm{DG}}}T_{\overline{x}}^1_{_{\mathrm{DG}}}`$ reads:

$$𝖳_{\overline{x}}_{_{\mathrm{DG}}}:\text{}$$
(1.6.3)
An important feature of (1.6.3) is that this complex is self-dual. In more detail, the definition of moment map implies that the map $`\mathrm{\Phi }_x^{}:T_xX𝔤^{}`$ is obtained by composing the isomorphism $`\omega :T_xX\stackrel{_{}}{}T_x^{}X,`$ induced by the symplectic form $`\omega ,`$ with the adjoint $`a^{}:T_x^{}X𝔤^{}`$ of the action-map $`a:𝔤T_xX`$, that is, one has:

$$\mathrm{\Phi }_x^{}=a^{}{}_{^{^{}}}{}^{}\omega :T_xX\stackrel{_{}}{}T_x^{}X𝔤^{}.$$

This shows that the DG vector space $`(𝖳_{\overline{x}}_{_{\mathrm{DG}}})^{}`$ is canonically isomorphic to $`𝖳_{\overline{x}}_{_{\mathrm{DG}}}`$. The isomorphism $`𝖳_{\overline{x}}_{_{\mathrm{DG}}}\stackrel{_{}}{}(𝖳_{\overline{x}}_{_{\mathrm{DG}}})^{}`$ induces, of course, an isomorphism of cohomology groups, and the non-degeneracy of symplectic form on the DG stack $`_{_{\mathrm{DG}}}`$ follows.

### 1.7 Example: moduli of $`G`$-local systems.

A typical (infinite-dimensional) example of the Theorem above is the moduli space of bundles with flat connection on a compact $`C^{\mathrm{}}`$-manifold $`X`$.

In more details, let $`G`$ be a Lie group with Lie algebra $`𝔤`$, and $`P`$ a principal $`G`$-bundle on $`X`$. Let $`𝔤__P`$ denote the associated vector bundle (with fiber $`𝔤`$) corresponding to the adjoint representation $`\mathrm{𝙰𝚍}^{}:G\mathrm{𝖦𝖫}(𝔤)`$. Thus, $`𝔤__P`$ is a bundle of Lie algebras.

Let $`\mathrm{\Omega }^i(X,𝔤__P)`$ be the vector space of $`C^{\mathrm{}}`$-differential $`i`$-forms on $`X`$ with values in $`𝔤__P`$. The Lie bracket on $`𝔤__P`$ combined with wedge-product of differential forms makes the graded space $`𝔊_P:=_i\mathrm{\Omega }^i(X,𝔤__P)`$ a Lie super-algebra.

We are interested in the moduli space (or stack) $`(P)`$ of flat $`C^{\mathrm{}}`$-connections on $`P`$, modulo gauge equivalence. So, assume that $`(P)`$ is non-empty and choose some flat $`C^{\mathrm{}}`$-connection $``$ on $`P`$. The connection induces a differential $`:\mathrm{\Omega }^{\text{}}(X,𝔤__P)\mathrm{\Omega }^{\text{}+1}(X,𝔤__P)`$, thus gives $`𝔊_P`$ the structure of a DGLA.

Any connection on $`P`$ can be written in the form $`^{}=+\gamma `$, for some $`\gamma \mathrm{\Omega }^1(X,𝔤__P)`$. The curvature of $`^{}`$ is $`^{}{}_{^{^{}}}{}^{}_{}^{}=\gamma +\frac{1}{2}[\gamma ,\gamma ].`$ Thus, $`^{}`$ is flat if and only if $`\gamma `$ satisfies the Maurer-Cartan equation $`\gamma +\frac{1}{2}[\gamma ,\gamma ]=0`$. Thus, in the notation of Remark 1.2.3 (with $`d:=`$), we have $`(P)\mathrm{𝖬𝖢}_1(𝔊_P,\{0\})/G_0\left(=_1(𝔊_P,\{0\})\right),`$ where $`G_0:=\mathrm{𝙰𝚞𝚝}(P)`$ is the infinite-dimensional group of gauge transformations.

To proceed further, we assume $`X`$ to be compact oriented, and assume also that there is a non-degenerate invariant symmetric bilinear form $`,:𝔤\times 𝔤`$. The form $`,`$ induces a nondegenerate pairing $`,__P:𝔤__P\times 𝔤__PC^{\mathrm{}}(X)`$. It is straightforward to verify that the following formula ($`i=0,\mathrm{},n=dim_{}X`$):

$$\beta :\text{}$$

gives a nondegenerate symmetric (even) bilinear form $`\beta :𝔊_P\times 𝔊_P.`$ The form $`\beta `$ is invariant, provided the connection $``$ was chosen so that the paring $`,__P`$ is $``$-horizontal, i.e., such that $`x,y__P+x,y__P=0,`$ for any sections $`x,y𝔤__P`$, c.f. (1.3.1).

Now if $`dim_{}X=2`$, then $`\mathrm{\Omega }^i(X,𝔤__P)=0`$ for $`i>2`$, and therefore we have $`\mathrm{𝖬𝖢}(𝔊_P,\{0\})=\mathrm{𝖬𝖢}_1(𝔊_P,\{0\})`$. Hence the construction of §1.5 applied to the DGLA $`𝔊=𝔊_P`$ gives a symplectic structure on the stack $`(P)=\mathrm{𝖬𝖢}_1(𝔊_P,\{0\})/G_0=\mathrm{𝖬𝖢}(𝔊_P,\{0\})/G=(𝔊_P,\{0\})`$.

For $`dim_{}X>2,`$ our construction only gives a symplectic structure on $`(𝔊_P,\{0\})(P)`$. However, it is known (see e.g., \[Kar\]) that if the Hard Lefschetz theorem holds for $`X`$, then $`(P)`$ is in effect a symplectic substack in $`(𝔊_P,\{0\})`$.

###### Remark 1.7.1.

For a trivial $`G`$-bundle $`P`$, the space $`(P)`$ may be identified with $`\mathrm{𝙷𝚘𝚖}(\pi _1(X),G)/\mathrm{𝙰𝚍}^{}G`$, the moduli space of $`G`$-local systems on $`X`$. The symplectic structure on this space has been studied by many authors, see e.g. \[Go\],\[We\], \[Kar\].

## 2 $`L_{\mathrm{}}`$-algebra version

### 2.1

Let $`𝔊=𝔊_+𝔊_{}`$ be a $`/2`$-graded $`L_{\mathrm{}}`$-algebra, see e.g. \[LM\], \[MSS\]. Let $`\beta `$ be an even nondegenerate invariant bilinear form on $`𝔊`$. Thus, for any homogeneous $`x_1,\mathrm{},x_{n+1}𝔊`$, one has:

$$\beta ([x_1,\mathrm{},x_n],x_{n+1})=(1)^{n(\mathrm{deg}x_1+1)}\beta (x_1,[x_2,\mathrm{},x_{n+1}]).$$

Define

$$\mathrm{\Phi }:𝔊_{}𝔊_+:xdx+\frac{1}{2!}[x,x]+\frac{1}{3!}[x,x,x]+\mathrm{}.$$

The differential of the map $`\mathrm{\Phi }`$ at the point $`x𝔊_{}`$ is given by

$$\mathrm{\Phi }_x^{}:y\mathrm{\Phi }_x^{}(y)=dy+[x,y]+\frac{1}{2!}[x,x,y]+\mathrm{},y𝔊_{}.$$

For any $`a𝔊_+`$ and $`x𝔊_{}`$, let

$$\xi _a(x):=\left(da+[x,a]+\frac{1}{2!}[x,x,a]+\mathrm{}\right).$$
(2.1.1)
Observe that we have

$$\beta (\xi _a(x),y)=\beta (a,\mathrm{\Phi }_x^{}(y))a𝔊_+,x,y𝔊_{}.$$
(2.1.2)
By \[La, Appendix B\], we also have

$$\mathrm{\Phi }_x^{}(\xi _a(x))=[a,\mathrm{\Phi }(x)]+[a,\mathrm{\Phi }(x),x]+\frac{1}{2!}[a,\mathrm{\Phi }(x),x,x]+\mathrm{}a𝔊_+,x𝔊_{}.$$
(2.1.3)
$``$  We say that two elements $`x_0,x_1𝔊_{}`$ are *gauge equivalent* if there exists a path $`a(t)𝔊_+`$ and a path $`x(t)𝔊_{}`$ such that $`x^{}(t)=\xi _{a(t)}(x(t))`$ and $`x(0)=x_0,x(1)=x_1`$.

$``$  We say that two elements $`b_0,b_1𝔊_+`$ are *adjoint equivalent* if there exists two paths $`a(t),b(t)𝔊_+`$ and a path $`x(t)𝔊_{}`$ such that $`b(0)=b_0,b(1)=b_1`$ and

$$b^{}(t)=[a(t),b(t)]+[a(t),b(t),x(t)]+\frac{1}{2!}[a(t),b(t),x(t),x(t)]+\mathrm{}.$$
(2.1.4)
###### Remark 2.1.5.

If $`𝔊=_i𝔊_i`$ is a $``$-graded $`L_{\mathrm{}}`$-algebra, then we say that $`b_0,b_1𝔊_2`$ are adjoint equivalent if there exists a path $`a(t)𝔊_0`$, a path $`b(t)𝔊_2`$ and a path $`x(t)𝔊_1`$ such that $`b(0)=b_0,b(1)=b_1`$ and (2.1.4) is satisfied.

### 2.2

If $`𝕆𝔊_+`$ is an adjoint equivalence class, then we define

$$\mathrm{𝖬𝖢}(𝔊,𝕆):=\{x𝔊_{}|\mathrm{\Phi }(x)𝕆\},(𝔊,𝕆):=\mathrm{𝖬𝖢}(𝔊,𝕆)/\text{gauge equivalence}.$$

###### Remark 2.2.1.

This definition of the objects $`\mathrm{𝖬𝖢}(𝔊,𝕆)`$ and $`(𝔊,𝕆)`$ involves solving differential equations, as well as formulas (2.1.1)-(2.1.3) which contain infinite series. One way to make sense of the definition above is to use a topological setting and to prove the convergence of all the series that arise.

If $`𝕆=\{0\}`$, then there is an alternative, purely algebraic, approach based on the language of formal schemes. Specifically, let $`\widehat{𝔊}_{}`$ be the formal completion of the vector space $`𝔊_{}`$ at the origin. The coordinate ring of $`\widehat{𝔊}_{}`$ is the ring $`[[𝔊_{}]]`$ of formal power series, so that the series for $`\mathrm{\Phi }`$ gives a well-defined morphism $`\widehat{𝔊}_{}𝔊_+`$. Thus, $`\mathrm{\Phi }^1(0)\widehat{𝔊}_{}`$ is a well-defined closed subscheme. Further, gauge equivalence gives a well-defined pro-algebraic groupoid acting on this subscheme, and one puts $`\widehat{}(𝔊,\{0\}):=\mathrm{\Phi }^1(0)/\text{gauge equivalence}`$, a pro-algebraic stack.

One can extend the concept of Hamiltonian reduction from Hamiltonian group-actions to Hamiltonian groupoid-actions. This way, using formula (2.1.2), one derives the following

###### Theorem 2.2.2.

Let $`𝔊=𝔊_+𝔊_{}`$ be an $`L_{\mathrm{}}`$-algebra with an invariant nondegenerate even bilinear form $`\beta `$, and $`𝕆𝔊_+`$ an adjoint equivalence class. Then, for any $`x(𝔊,𝕆)`$, the form $`\beta `$ induces a non-degenerate 2-form on $`𝖳_x(𝔊,𝕆)`$, the tangent space (at $`x`$) to the modular stack. These 2-forms give rise to a symplectic structure on $`(𝔊,𝕆)`$.

## 3 From moment map to Maurer-Cartan equations

### 3.1

Let $`(V,\omega )`$ be a finite dimensional symplectic vector space, and $`[V]`$ the algebra of polynomial functions on $`V`$ viewed as a Poisson algebra with Poisson bracket $`\{,\}`$ corresponding to the symplectic structure.

Fix a finite dimensional Lie algebra $`𝔤`$. Let $`H:𝔤^{>0}[V]=_{i1}^i[V],aH_a,`$ be a Lie algebra homomorphism. We get, tautologically, a (non-linear) Hamiltonian $`𝔤`$-action on the vector space $`V`$ with polynomial moment map $`\mathrm{\Phi }:V𝔤^{},\mathrm{\Phi }(v):aH_a(v).`$

### 3.2

Introduce a $``$-graded vector space

$$𝔊=𝔤V𝔤^{},𝔊_0:=𝔤,𝔊_1:=V,𝔊_2:=𝔤^{},$$

and write $`𝔊_+:=𝔊_0𝔊_2`$ and $`𝔊_{}:=𝔊_1`$. The canonical pairing $`𝔤\times 𝔤^{}`$ gives a non-degenerate symmetric bilinear form on $`𝔊_+`$. Combined with the symplectic form $`\omega `$ on $`V`$ this gives an even non-degenerate bilinear form $`\beta `$ on the super-space $`𝔊=𝔊_+𝔊_{}`$.

Further, let $`\mathrm{\Phi }=\mathrm{\Phi }_1+\mathrm{\Phi }_2+\mathrm{}:V𝔤^{}`$ be an expansion of the moment map $`\mathrm{\Phi }`$ into homogeneous components (by assumption there is no constant term). For each $`i1`$, the map $`\mathrm{\Phi }_i`$ gives rise, via the canonical isomorphism $`^i[V](\mathrm{𝚂𝚢𝚖}_{}^{}{}_{}{}^{i}V)^{},`$ to a linear map $`\stackrel{~}{\mathrm{\Phi }}_i:\mathrm{𝚂𝚢𝚖}_{}^{}{}_{}{}^{i}V𝔤^{},`$ such that $`\stackrel{~}{\mathrm{\Phi }}_i(y^i)=\mathrm{\Phi }_i(y),yV`$.

We now define the following maps:

$``$  $`𝔊_0𝔊𝔊`$ given, for any $`u=xv\lambda 𝔤V𝔤^{},`$ by $`au[a,u]:=\mathrm{𝚊𝚍}^{}a(x)a(v)\mathrm{𝚊𝚍}_{}^{}{}_{}{}^{}a(\lambda );`$

$``$  $`\mathrm{𝚂𝚢𝚖}_{}^{}{}_{}{}^{i}(𝔊_{})𝔊_2𝔊_+,y_1y_2\mathrm{}y_i[y_1,y_2,\mathrm{},y_i]:=\stackrel{~}{\mathrm{\Phi }}_i(y_1y_2\mathrm{}y_i),`$ for each $`i2;`$

Further, let $`d:=\mathrm{\Phi }_1:𝔊_{}𝔊_+`$, and define $`d:𝔊_+𝔊_{}`$ to be the zero-map.

###### Proposition 3.2.1.

The above defined maps give $`𝔊`$ an $`L_{\mathrm{}}`$-structure (with all brackets that were not specified above being set equal to zero), such that $`\beta `$ becomes an invariant form.

###### Proof.

Straightforward computation. $`\mathrm{}`$

We observe that two elements in $`𝔊_2=𝔤^{}`$ are adjoint equivalent (in the sense of §2) if and only if they belong to the same coadjoint orbit of the group $`G=G_0`$ (corresponding to the Lie algebra $`𝔤`$) acting on $`𝔤^{}`$. Given such an orbit $`𝕆𝔊_2=𝔤^{}`$, we see that $`\mathrm{𝖬𝖢}(𝔊,𝕆)=\mathrm{\Phi }^1(𝕆)`$. Hence, for the Maurer-Cartan stack we get $`(𝔊,𝕆)=\mathrm{\Phi }^1(𝕆)/\mathrm{𝙰𝚍}^{}G_0,`$ is the standard Hamiltonian reduction of $`V`$ over $`𝕆`$.

Remarks. $`(𝗂)`$Note that the construction above may be thought of as a non-linear version of the construction of the tangent space as a DG vector space, given in §1.5.

$`(\mathrm{𝗂𝗂})`$In the special case $`H:𝔤^2[V]`$ (quadratic Hamiltonians), we have $`\mathrm{\Phi }=\mathrm{\Phi }_2`$, and the $`L_{\mathrm{}}`$-structure above reduces to an ordinary Lie super-algebra structure on $`𝔊=𝔤V𝔤^{}.`$ The symmetric bracket $`𝔊_{}\times 𝔊_{}𝔊_+`$ is in this case provided by the map $`\mathrm{\Phi }`$ viewed as a linear map: $`\mathrm{𝚂𝚢𝚖}_{}^{}{}_{}{}^{2}(𝔊_{})=\mathrm{𝚂𝚢𝚖}_{}^{}{}_{}{}^{2}(V)𝔤^{}=𝔊_2.`$ 333We are grateful to E. Getzler for pointing out a similarity between our construction and the results of Goldman and Millson, c.f. \[GM2, p. 499\]. This special case was also implicit in \[Ko\]. Notice that a Lie algebra homomorphism $`𝔤\mathrm{𝚂𝚢𝚖}_{}^{}{}_{}{}^{2}V𝔰𝔭(V)`$ has a natural extension to a Lie algebra homomorphism $`\nu :𝔊_+=𝔤𝔤^{}𝔰𝔭(V)`$ by sending $`𝔤^{}`$ to zero. We observe that $`\nu `$ is of super Lie type in Kostant’s terminology, i.e. the condition on the Casimir in \[Ko, Theorem 0.1\] holds (trivially) for this $`\nu `$.

$`(\mathrm{𝗂𝗂𝗂})`$J. Stasheff informed us that it is possible to define Hamiltonian reductions with respect to (infinitesimal) actions of an $`L_{\mathrm{}}`$-algebra, and to extend the construction of §3 to such an $`L_{\mathrm{}}`$-setup. He also pointed out to us the relationship of our construction to the classical BRST complex. Namely, an $`L_{\mathrm{}}`$-structure on $`𝔊`$ is equivalent to a square zero derivation on the free super-commutative algebra $`S^{>0}(𝔊[1]^{})`$ generated by $`𝔊[1]^{}`$, see e.g. \[LM\]. Here, $`𝔊[1]`$ is the super vector space with $`𝔊[1]_+=𝔊_{}`$ and $`𝔊[1]_{}=𝔊_+`$. For the $`L_{\mathrm{}}`$-algebra $`𝔊`$ in Proposition 3.2.1, we thus obtained a differential on $`S^{>0}(𝔊[1]^{})`$ which turns out to be the classical BRST operator on $`𝔤^{}𝔤\mathrm{𝚂𝚢𝚖}^{}V^{}`$ defined in \[KS, p. 57\]. To see this, it suffices to note that the generators of $`S^{>0}(𝔊[1]^{})`$ are

$$𝔊[1]_{}^{}=𝔤^{}𝔤,𝔊[1]_+^{}=V^{},$$

and the differential is defined on these generators by taking the sum of the maps

$$H:𝔤\mathrm{𝚂𝚢𝚖}_{}^{}{}_{}{}^{>0}V^{}\text{and}J:𝔊[1]^{}𝔤^{}𝔊[1]^{},$$

where $`J`$ is obtained from dualizing the map $`𝔊_0𝔊𝔊:au[a,u]`$.

Department of Mathematics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA;
x wlgan@math.mit.edu

Department of Mathematics, University of Chicago, Chicago, IL 60637, USA;
x ginzburg@math.uchicago.edu


text:
# Holomorphic Equivariant Cohomology via a Transversal Holomorphic Vector Field Partially supported by the NNSF of China (10271059) and the K. C. Wong Education Foundation

## 1 Introduction

For a compact complex manifold $`X`$, let $`T_𝐂X`$ denote the complexification of the real tangent bundle $`T_𝐑X`$ of $`X`$. Then $`T_𝐂X`$ splits canonically as $`T_𝐂X=TX\overline{TX}`$, where $`TX`$ and $`\overline{TX}`$ are the holomorphic and the anti-holomorphic tangent bundle of $`X`$, respectively. Let $`\xi X`$ be a holomorphic vector bundle over $`X`$. There exists a natural $`𝐙`$-grading in $`\mathrm{\Lambda }^{}(T_𝐂^{}X)\xi `$ defined by the following decomposition:

$$\mathrm{\Lambda }^{}(T_𝐂^{}X)\xi =\underset{dimXrdimX}{}\mathrm{\Lambda }^{(r)}(T_𝐂^{}X)\xi ,$$
$`(1.1)`$
where

$$\mathrm{\Lambda }^{(r)}(T_𝐂^{}X)\xi =\underset{qp=r}{}\mathrm{\Lambda }^q(\overline{T^{}X})(\mathrm{\Lambda }^p(T^{}X)\xi ).$$
$`(1.2)`$
We will use $`\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi `$ to denote $`\mathrm{\Lambda }^{}(T_𝐂^{}X)\xi `$ with this $`𝐙`$-grading. Let $`\mathrm{\Omega }^{(r)}(X,\xi )`$ (resp. $`\mathrm{\Omega }^{()}(X,\xi )`$) be the complex vector space of smooth sections of $`\mathrm{\Lambda }^{(r)}(T_𝐂^{}X)\xi `$ (resp. $`\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi `$). Then $`(\mathrm{\Omega }^{()}(X,\xi )=_r\mathrm{\Omega }^{(r)}(X,\xi ),\overline{}^X)`$ is a complex and the cohomology groups $`H^{(r)}(X,\xi )`$ associated to this complex are direct sums:

$$H^{(r)}(X,\xi )=\underset{qp=r}{}H^{p,q}(X,\xi ),$$
$`(1.3)`$
where $`H^{p,q}(X,\xi )`$ are the usual Dolbeault cohomology groups of $`X`$ with coefficients in the holomorphic vector bundle $`\xi `$.

For any holomorphic vector field $`v`$ on $`X`$, set

$$\overline{}_v^X=\overline{}^X+i(v):\mathrm{\Omega }^{()}(X,\xi )\mathrm{\Omega }^{(+1)}(X,\xi ),$$
$`(1.4)`$
where $`i(v)`$ is the standard contraction operator defined by $`v`$. The consideration of operators $`\overline{}^X`$ and $`i(v)`$ together goes back to \[Bot\]. Clearly, $`(\mathrm{\Omega }^{()}(X,\xi ),\overline{}_v^X)`$ is also a $`𝐙`$-graded complex. Denote the cohomology groups associated to this complex by $`H_v^{(r)}(X,\xi )`$, which are called the holomorphic equivariant cohomology groups of $`X`$ via $`v`$ with coefficients in $`\xi `$ (cf. \[L\]).

In \[CL\], Carrell and Lieberman discussed the relation between Dolbeault cohomology of a connected compact K$`\ddot{\mathrm{a}}`$hler manifold $`X`$ and the zero point set $`Y`$ of a holomorphic vector field $`v`$ on $`X`$ by using Deligne degeneracy criterion and proved that $`H^{(r)}(X,𝐂)`$ vanished for all $`|r|>dimY`$ if $`Y\mathrm{}`$. Since $`dimH_v^{()}(X,𝐂)=dimH^{()}(X,𝐂)`$ in this case (see \[L, Theorem 1.3\] and \[CL\]), the corresponding vanishing results for $`H_v^{()}(X,𝐂)`$ are also valid. For a general compact complex manifold $`X`$ and a transversal holomorphic vector field $`v`$ on $`X`$ (see a definition in \[Bis1, Sect.5.1\]), Liu in \[L\] proved that $`dimH^{()}(Y,𝐂)dimH_v^{()}(X,𝐂)`$ by constructing an injective homomorphism $`\alpha _r:H^{(r)}(Y,𝐂)H_v^{()}(X,𝐂)`$. Moreover, Liu got a counting formula for $`dimH_v^{(0)}(X,𝐂)`$ in term of the multiplicities of the zero points of $`v`$ if the zero points of $`v`$ are discrete. Under the assumption that the zero point of $`v`$ is nodegenerate, motivated by Witten’s deformation idea (\[W\]), he also sketched an analytic proof of his formula in \[L, Sect.7\] by examining the behavior of a natural deformation $`D_T^X`$ of the Riemann-Roch operator on $`X`$ as $`T\mathrm{}`$.

A more general result in this direction is due to Bismut. By using the technique of spectral sequences in \[Bis1, Theorem 5.1\], Bismut proves that, when $`v`$ is a transversal holomorphic vector field on $`X`$ with a zero point set $`Y`$, the embedding $`j:YX`$ induces naturally a quasi-isomorphism $`j^{}:(\mathrm{\Omega }^{()}(X,𝐂),\overline{}_v^X)(\mathrm{\Omega }^{()}(Y,𝐂),\overline{}^Y)`$.

In this paper, we will give an analytic proof of the following fairly straightforward generalization of the Bismut’s theorem \[Bis1, Theorem 5.1\]:

Theorem 1.1 Let $`v`$ be a transversal holomorphic vector field on a compact complex manifold $`X`$ with a zero point set $`Y`$. Then

$$j^{}:(\mathrm{\Omega }^{()}(X,\xi ),\overline{}_v^X)(\mathrm{\Omega }^{()}(Y,\xi |_Y),\overline{}^Y)$$
$`(1.5)`$
is a quasi-isomorphism.

Following Witten’s deformation idea (\[W\]) as in \[L, Sect.7\], we will also work with a deformation $`D_T^X`$ of the twisted Riemann-Roch operator on $`X`$ by $`\xi `$ but the whole proof now is heavily based on the analytic localization techniques developed by Bismut and Lebeau (cf. \[BL\]), since the analysis involved here is much complicated than the situation in \[L, Sect.7\]. A key point in our proof is to express $`D_T^X`$ and to trivialize the bundle on which $`D_T^X`$ acts by using the Bismut connection $`^B`$ (see \[Bis2, II. b)\]). We should point out that if the trivialization is made by a lifting of the holomorphic Hermitian connection $`^{TX}`$ on $`TX`$, then an extra term coming from the torsion of $`^{TX}`$ will enter to the final operator on $`Y`$ in the Bismut-Lebeau localization process of $`D_T^X`$ as $`T\mathrm{}`$. But it is not clear to us that the extra term is zero for a general complex manifold $`X`$. Consequently, we could only obtain an equality related to the involved operators at the index level.

## 2 A deformed twisted Riemann-Roch operator $`D_T^X`$ and its local behavior near $`Y`$

This section is divided into three parts. In a) we introduce a deformation $`D_T^X`$ of the twisted Riemann-Roch operator by $`\xi `$ via $`v`$, which has been used in \[L, Sect.7\] in the case of $`\xi =𝐂`$. In b) we recall the definition of the Bismut connection $`^B`$ (cf. \[Bis2, II. b)\]) and express $`D_T^X`$ in this connection by a direct application of \[Bis2, Theorem 2.2\]. In c) we study the local behavior of the deformed operator $`D_T^X`$ near the submanifold $`Y`$ following \[BL, Sect.8\], in which the Bismut connection will play an essential role.

a) A deformed twisted Riemann-Roch operator $`D_T^X`$

Let $`X`$ be a compact complex manifold of $`𝐂`$-dimension $`n`$. For any $`T𝐑`$, we consider the following deformed operator

$$\overline{}_T^X=\overline{}+Ti(v):\mathrm{\Omega }^{()}(X,\xi )\mathrm{\Omega }^{(+1)}(X,\xi )$$
$`(2.1)`$
and the deformed complex $`(\mathrm{\Omega }^{()}(X,\xi ),\overline{}_T^X)`$. One verifies easily as the proof of \[L, Lemma 1.1\] that the cohomologies associated to this deformed complex do not depend on $`T0`$.

Let $`g^{TX}`$ (resp. $`g^\xi `$) be a Hermitian metric on $`X`$ (resp. $`\xi `$). By the standard procedure there is an induced Hermitian metric $`,_{\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi }`$ on $`\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi `$. Let $`d\upsilon _X`$ denote the Riemannian volume element of $`(X,g^{TX})`$. Then for $`s_1,s_2\mathrm{\Omega }^{()}(X,\xi )`$,

$$s_1,s_2=\frac{1}{(2\pi )^{dim_𝐂X}}_Xs_1,s_2_{\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi }𝑑\upsilon _X$$
$`(2.2)`$
defines an $`L^2`$-Hermitian inner product on $`\mathrm{\Omega }^{()}(X,\xi )`$. Let $`\overline{}^X`$ and $`\overline{}_T^X`$ denote the formal adjoint operators of $`\overline{}^X`$ and $`\overline{}_T^X`$, respectively. For any $`UT_𝐂X`$, let $`U^{}`$ be an element in $`T_𝐂^{}X`$ defined by $`g^{TX}(U,)`$. Clearly, $`\overline{v}^{}`$ is a (1,0)-form on $`X`$ and $`\overline{v}^{}`$ is the dual operator of $`i(v)`$. Moreover, we have

$$\overline{}_T^X=\overline{}^X+T\overline{v}^{}:\mathrm{\Omega }^{()}(X,\xi )\mathrm{\Omega }^{(1)}(X,\xi ).$$
$`(2.3)`$
Set

$$D^X=\sqrt{2}(\overline{}^X+\overline{}^X),D_T^X=\sqrt{2}(\overline{}_T^X+\overline{}_T^X).$$
$`(2.4)`$
Clearly, $`D^X`$ is the usual twisted Riemann-Roch operator by the holomorphic bundle $`\xi `$ and $`D_T^X`$ is a deformation of $`D^X`$ and interchanges $`\mathrm{\Omega }^{(\mathrm{even})}(X,\xi )`$ and $`\mathrm{\Omega }^{(\mathrm{odd})}(X,\xi )`$. From Hodge theory we have the following isomorphisms

$$\mathrm{ker}(D_T^X)^2|_{\mathrm{\Omega }^{(r)}(X,\xi )}H_v^{(r)}(X,\xi ).$$
$`(2.5)`$
Lemma 2.1 For any open neighborhood $`𝒰`$ of $`Y`$, there exist constants $`a>0`$, $`b>0`$ and $`T_0>0`$ such that for any $`s\mathrm{\Omega }^{()}(X,\xi )`$ with $`\mathrm{Supp}sX𝒰`$ and any $`TT_0`$, one has the following estimate for Sobolev norms,

$$D_T^Xs_0^2a(s_1^2+(Tb)s_0^2).$$
$`(2.6)`$
Proof. An easy computation shows that

$$D_T^{X,2}=D^{X,2}+2T^2|v|^2+2T((\overline{}^X\overline{v}^{})+i(\overline{}^X\overline{v}^{})),$$
$`(2.7)`$
where $`i(\overline{}^X\overline{v}^{})`$ denotes the adjoint operator of $`(\overline{}^X\overline{v}^{})`$. Note that $`((\overline{}^X\overline{v}^{})+i(\overline{}^X\overline{v}^{}))`$ is a zero order operator and $`v0`$ on $`X𝒰`$, the lemma follows from the well-known Garding’s inequality directly. $`\mathrm{}`$

By Lemma 2.1 and Hodge theory, we can study $`H_v^{()}(X,\xi )`$ through the behavior of the operator $`D_T^X`$ near $`Y`$ for large $`T`$. Also by Lemma 2.1, it is an easy observation that, when $`Y=\mathrm{}`$, the cohomology group $`H_v^{()}(X,\xi )`$ vanishes. In the following we will always assume that $`v`$ is transversal and $`Y\mathrm{}`$. Note that generally $`Y`$ consists of some connected components $`Y_k`$ with different $`𝐂`$-dimensions $`l_k`$. When no confusion arises, we always drop the subscripts and simply denote them by $`Y`$ and $`l`$, respectively.

b) An expression of $`D_T^X`$ in the Bismut connection

We first recall the definition of the Bismut connection in \[Bis2, II. b)\]. For a complex manifold $`X`$ with a Hermitian metric $`g^{TX}`$, let $`^{TX}`$ be the holomorphic Hermitian connection on $`TX`$. Note that $`^{TX}`$ induces naturally an Euclidean connection on $`T_𝐑X`$ which preserves the complex structure of $`T_𝐑X`$. Let $`T_X`$ denote the torsion tensor of the connection $`^{TX}`$. Let $`B_X`$ be the antisymmetrization of the tensor $`(U,V,W)\frac{1}{4}T_X(U,V),W`$ and let $`S^{B_X}`$ denote the one form with values in antisymmetric elements of $`End(T_𝐑X)`$ which is such that

$$S^{B_X}(U)V,W=2B_X(U,V,W),$$
$`(2.8)`$
where $`U,V,WT_𝐑X`$. Let $`^{L^X}`$ be the Levi-Civita connection on $`T_𝐑X`$. Set

$$S_X=^{TX}^{L^X}.$$
$`(2.9)`$
Then $`S_X`$ is also a one form with values in antisymmetric elements of $`\mathrm{End}(T_𝐑X)`$. The important thing here is that $`S^{B_X}S_X`$ preserves the complex structure of $`T_𝐑X`$ (cf. \[Bis2, (2.38)\]). Now the Bismut connection $`^{B_X}`$ on $`T_𝐑X`$ is defined by (cf. \[Bis2, (2.37)\])

$$^{B_X}=^{TX}+(S^{B_X}S_X)=^{L_X}+S^{B_X}.$$
$`(2.10)`$
The Bismut connection $`^{B_X}`$ also preserves the complex structure of $`T_𝐑X`$ and so induces naturally a unitary connection on $`TX`$ and a unitary connection on $`\overline{TX}`$, which are still denoted by $`^{B_X}`$. Note that when $`X`$ is K$`\ddot{\mathrm{a}}`$hler, the Bismut connection $`^{B_X}`$ coincides with the holomorphic Hermitian connection $`^{TX}`$. There is a unitary connection on $`\mathrm{\Lambda }^{}(\overline{T^{}X})`$ lifted canonically from $`^{B_X}`$, which we still call the Bismut connection and denote by the same notation $`^{B_X}`$.

Let $`Y`$ be a complex submanifold of $`X`$. Let $`\pi :NY`$ be the normal bundle of $`Y`$ in $`X`$. We identify $`N`$ with the sub-bundle of $`TX|_Y`$ orthogonal to $`TY`$ with respect to the restriction metric $`g^{TX|_Y}`$ on $`TX|_Y`$ by $`g^{TX}`$. So we have the identification of $`C^{\mathrm{}}`$ bundles $`TX|_Y=TYN`$. Let $`g^{TY}`$ (resp. $`g^N`$) be the induced metric on $`TY`$ (resp. $`N`$) from $`g^{TX|_Y}`$. Let $`P^{T_𝐑Y}`$, $`P^{N_𝐑}`$ be the orthogonal projection operators from $`T_𝐑X|_Y`$ onto $`T_𝐑Y`$ and $`N_𝐑`$ respectively. Let $`j`$ denote the embedding of $`Y`$ into $`X`$. Then $`j^{}^{B_X}`$ is a connection on $`T_𝐑X|_Y`$ preserving the metric $`g^{TX|_Y}`$ and the complex structure of $`T_𝐑X|_Y`$. Moreover, $`P^{T_𝐑Y}(j^{}^{B_X})P^{T_𝐑Y}`$ is exactly the Bismut connection on $`T_𝐑Y`$ associated to the induce metric $`g^{TY}`$, and $`P^{N_𝐑}(j^{}^{B_X})P^{N_𝐑}`$ is a connection on $`N_𝐑`$ preserving the metric $`g^N`$ and the complex structure of $`N_𝐑`$. Set

$$^{B_X,}=^{B_Y}\left(P^{N_𝐑}(j^{}^{B_X})P^{N_𝐑}\right),$$
$`(2.11)`$
$$𝐀=j^{}^{B_X}^{B_X,}.$$
$`(2.12)`$
Clearly, $`^{B_X,}`$ is also a connection on $`T_𝐑X|_Y`$ preserving the metric $`g^{TX|_Y}`$ and the complex structure of $`T_𝐑X|_Y`$, and $`𝐀`$ is the second fundamental form of the Bismut connection $`^{B_X}`$. We still use the same notation $`^{B_X,}`$ to denote its restriction on $`\overline{TX}|_Y`$ as well as its lifting on $`\mathrm{\Lambda }^{}(\overline{T^{}X}|_Y)`$.

Now we return to our situation and express the deformed twisted Riemann-Roch operator $`D_T^X`$ in the Bismut connection by applying \[Bis2, Theorem 2.2\]. To do this we still need a holomorphic Hermitian connection on the bundle $`\mathrm{\Lambda }^{}(T^{}X)\xi `$. Since our problem does not depend on the metrics, we can and will choose a special metric on the holomorphic bundle $`\mathrm{\Lambda }(T^{}X)`$ to simplify the analysis.

Let $`𝐋_v:TX|_YTX|_Y`$ be the holomorphic Lie homomorphism defined by $`𝐋_v(u)=[v,u]`$ for any $`uTX|_Y`$. Denote $`𝐋_v(TX|_Y)`$ by $`\stackrel{~}{N}`$. Since $`v`$ is transversal, $`TX|_Y`$ splits holomorphically into $`TY\stackrel{~}{N}`$ and $`𝐋_v`$ induces an isomorphism from $`N`$ to $`\stackrel{~}{N}`$, which we still denote by $`𝐋_v`$. We introduce a new Hermitian metric $`\stackrel{~}{g}^{\stackrel{~}{N}}`$ on $`\stackrel{~}{N}`$ by requiring that $`𝐋_v:N\stackrel{~}{N}`$ is unitary fiberwisely. Consequently, we get a new Hermitian metric $`\stackrel{~}{g}^{TX|_Y}=g^{TY}\stackrel{~}{g}^{\stackrel{~}{N}}`$ on $`TX|_Y`$. We can and we will extend $`\stackrel{~}{g}^{TX|_Y}`$ to a Hermitian metric $`\stackrel{~}{g}^{TX}`$ on $`TX`$. We will denote $`TX`$ (resp. $`TX|_Y`$) with the metric $`\stackrel{~}{g}^{TX}`$ (resp. $`\stackrel{~}{g}^{TX|_Y}`$) by $`\stackrel{~}{TX}`$ (resp. $`\stackrel{~}{TX}|_Y`$) to distinguish the same bundle with different metrics. Let $`^{\stackrel{~}{TX}}`$ (resp. $`^{\stackrel{~}{TX}|_Y}`$, resp. $`^{\stackrel{~}{N}}`$) be the holomorphic Hermitian connection on $`\stackrel{~}{TX}`$ (resp. $`\stackrel{~}{TX}|_Y`$, resp. $`\stackrel{~}{N}`$). We have the following standard fact:

$$j^{}^{\stackrel{~}{TX}}=^{\stackrel{~}{TX}|_Y}=^{TY}^{\stackrel{~}{N}}.$$
$`(2.13)`$
We lift the holomorphic Hermitian connection $`^{\stackrel{~}{TX}}`$ (resp. $`^{TY}`$) to the holomorphic Hermitian connection $`^{\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})}`$ (resp. $`^{\mathrm{\Lambda }^{}(T^{}Y)}`$) on $`\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})`$ (resp. $`\mathrm{\Lambda }^{}(T^{}Y)`$).

Let $`g^\xi `$ be a Hermitian metric on $`\xi `$ and let $`^\xi `$ be the holomorphic Hermitian connection on $`\xi `$. Set $`^{\xi |_Y}=j^{}^\xi `$, which is the holomorphic Hermitian connection on $`\xi |_Y`$. So $`^{\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})\xi }=^{\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})}1+1^\xi `$ (resp. $`^{\mathrm{\Lambda }^{}(T^{}Y)\xi |_Y}=^{\mathrm{\Lambda }^{}(T^{}Y)}1+1^{\xi |_Y}`$) is a holomorphic Hermitian connection on $`\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})\xi `$ (resp. $`\mathrm{\Lambda }^{}(T^{}Y)\xi |_Y`$). Therefore,

$$^{B,X}=^{B_X}1+1^{\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})\xi },$$
$`(2.14)`$
$$^{B,Y}=^{B_Y}1+1^{\mathrm{\Lambda }^{}(T^{}Y)\xi |_Y}$$
$`(2.15)`$
are unitary connections on the Hermitian vector bundle $`\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi `$, $`\mathrm{\Lambda }^{()}(T_𝐂^{}Y)\xi |_Y`$, respectively.

For $`U(TX,g^{TX})`$, set

$$c(U)=\sqrt{2}U^{},c(\overline{U})=\sqrt{2}i(\overline{U});$$
$`(2.16)`$
and for $`U(\stackrel{~}{TX},\stackrel{~}{g}^{TX})`$, set

$$\widehat{c}(U)=\sqrt{2}i(U),\widehat{c}(\overline{U})=\sqrt{2}\overline{U}^{}.$$
$`(2.17)`$
We extend the map $`c`$ (resp. $`\widehat{c}`$) by $`𝐂`$ linearity into the Clifford action of $`T_𝐂X`$ (resp. $`\stackrel{~}{T_𝐂X}`$) on $`\mathrm{\Lambda }^{}(\overline{T^{}X})`$ (resp. $`\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})`$).

Let $`\{e_1,\mathrm{},e_{2n}\}`$ be an orthonormal basis $`T_𝐑X`$. Set

$$c(B_X)=\frac{1}{6}\underset{i,j,k=1}{\overset{2n}{}}B_X(e_i,e_j,e_k)c(e_i)c(e_j)c(e_k).$$
$`(2.18)`$
Note that

$$\frac{1}{4}\underset{i,j,k=1}{\overset{2n}{}}S^{B_X}(e_i)e_j,e_kc(e_i)c(e_j)c(e_k)=3c(B_X).$$
$`(2.19)`$
Now recall the definition (2.10) and apply \[Bis2, Theorem 2.2\] directly, we obtain the following expressions of $`D^Y`$ and $`D_T^X`$ with respect to the orthonormal basis $`\{e_1^{},\mathrm{},e_{2l}^{}\}`$ for $`T_𝐑Y`$ and $`\{e_1,\mathrm{},e_{2n}\}`$ for $`T_𝐑X`$, respectively:

$$D^Y=\underset{i=1}{\overset{2l}{}}c(e_i^{})_{e_i^{}}^{B,Y}+2c(B_Y),$$
$`(2.20)`$
$$D_T^X=\underset{i=1}{\overset{2n}{}}c(e_i)_{e_i}^{B,X}+2c(B_X)+\sqrt{1}T(\widehat{c}(v)+\widehat{c}(\overline{v})).$$
$`(2.21)`$
c) The local behavior of the deformed operator $`D_T^X`$ near $`Y`$

For $`yY`$ and $`ZN_{𝐑,y}`$, let $`t𝐑x_t=\mathrm{exp}_y^X(tZ)X`$ be the geodesic in $`X`$ with respect to the Levi-Civita connection $`^{L^X}`$, such that $`x_0=y`$, $`dx/dt|_{t=0}=Z`$. For $`ϵ>0`$, let $`B_ϵ=\{ZN_𝐑|Z|<ϵ\}.`$ Since $`X`$ and $`Y`$ are compact, there exists an $`ϵ_0>0`$ such that for $`0<ϵ<ϵ_0`$, the map $`(y,Z)N_𝐑\mathrm{exp}_y^X(Z)X`$ is a diffeomorphism from $`B_ϵ`$ to a tubular neighborhood $`𝒰_ϵ`$ of $`Y`$ in $`X`$. From now on, we will identify $`B_ϵ`$ with $`𝒰_ϵ`$ and use the notation $`x=(y,Z)`$ instead of $`x=\mathrm{exp}_y^X(Z)`$.

We will make use of the trivialization of $`(\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_{𝒰_{ϵ_0}}`$ by the parallel transport of $`(\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y`$ with respect to the connection $`^{B,X}`$ along the geodesic $`t(y,tZ)`$. The key point here is that this trivialization preserves the metric and the $`𝐙`$-grading since the Bismut connection is a unitary connection and preserves the complex structure of $`T_𝐑X`$. By using the trivialization of $`\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi `$ over $`𝒰_{ϵ_0}`$, we can and will make the identification of $`(\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_{𝒰_{ϵ_0}}`$ with $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)|_{B_{ϵ_0}}`$ and so we can consider $`^{B,X}`$ as a unitary connection on the Hermitian vector bundle $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)|_{B_{ϵ_0}}`$ with the obviously induced metric. Note that there exists another unitary connection $`^{B,X,}`$ on $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)`$ defined by

$$^{B,X,}=\pi ^{}\left(^{B_X,}1+1j^{}^{\mathrm{\Lambda }^{}(\stackrel{~}{T^{}X})\xi }\right).$$
$`(2.22)`$
Let $`d\upsilon _Y`$ (resp. $`d\upsilon _N`$) denote the Riemannian volume element of $`(Y,g^{TY})`$ (resp. the fibres of $`(N,g^N)`$). We define a smooth positive function $`k(y,Z)`$ on $`B_{ϵ_0}`$ by the equation $`d\upsilon _X(y,Z)=k(y,Z)d\upsilon _Y(y)d\upsilon _{N_y}(Z)`$ and an $`L^2`$-Hermitian inner product on $`𝐄`$ by

$$f,g=\frac{1}{(2\pi )^{dim_𝐂X}}_Y_{N_{𝐑,y}}f,g(y,Z)𝑑\upsilon _N(Z)𝑑\upsilon _Y(y),$$
$`(2.23)`$
for any $`f,g𝐄`$ with compact support, where $`𝐄`$ denotes the set of smooth sections of $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)`$ on $`N_𝐑`$. Clearly, $`k(y)=k(y,0)=1`$ on $`Y`$ and $`k(y,Z)`$ has a positive lower bound on $`𝒰_{ϵ_0/2}`$. If $`f𝐄`$ has compact support in $`B_{ϵ_0}`$, we can identify $`f`$ with an element in $`\mathrm{\Omega }^{()}(X,\xi )`$ which has compact support in $`𝒰_{ϵ_0}`$.

Let $`TN_𝐑=T^HN_𝐑N_𝐑`$ be the splitting of $`TN_𝐑`$ induced by the Euclidean connection $`^{L^N}=P^{N_𝐑}(j^{}^{L^X})P^{N_𝐑}`$ on $`N_𝐑`$, where $`T^HN_𝐑`$ denotes the horizontal part of $`TN_𝐑`$. If $`UT_𝐑Y`$, let $`U^HT^HN_𝐑`$ denote the horizontal lift of $`U`$ in $`T^HN_𝐑`$, so that $`\pi _{}U^H=U`$. Let

$$\{e_1,\mathrm{},e_{2l},e_{2l+1},\mathrm{},e_{2n}\}$$
$`(2.24)`$
be an orthonormal basis of $`T_𝐑X|_Y`$ with $`\{e_1,\mathrm{},e_{2l}\}`$ an orthonormal basis of $`T_𝐑Y`$ and $`\{e_{2l+1},\mathrm{},e_{2n}\}`$ an orthonormal basis of $`N_𝐑`$.

Definition 2.2 Let $`D^H`$, $`D^N`$ be the operators acting on $`𝐄`$

$$D^H=\underset{i=1}{\overset{2l}{}}c(e_i)_{e_i^H}^{B,X,}+2c(B_Y),D^N=\underset{\alpha =2l+1}{\overset{2n}{}}c(e_\alpha )_{e_\alpha }^{B,X,}.$$
$`(2.25)`$
Clearly, $`D^N`$ acts along the fibres $`N_{𝐑,y}`$ as the operator $`\sqrt{2}(\overline{}^{N_y}+\overline{}^{N_y})`$. Note that $`D^H`$, $`D^N`$ are self-adjoint with respect to the Hermitian inner product (2.23).

Now we turn to Taylor expansions of $`v`$ near $`Y`$ along the geodesic $`(y,tZ)`$ for $`yY`$ and $`ZN_{𝐑,y}`$. Let $`\{w_1,\mathrm{},w_l,w_{l+1},\mathrm{},w_n\}`$ be a unitary basis for $`TX|_Y`$ and let $`(z^{l+1},\mathrm{},z^n)`$ denote the associated holomorphic coordinate system on $`N_y`$ with $`w_\alpha =\sqrt{2}/z^\alpha `$ for $`l+1\alpha n`$. Note that $`𝐋_v:N\stackrel{~}{N}`$ is unitary fiberwisely. Set

$$\stackrel{~}{w}_\alpha =𝐋_v(w_\alpha ),\stackrel{~}{\overline{w}}_\alpha =𝐋_v(\overline{w}_\alpha ),l+1\alpha n.$$
$`(2.26)`$
Hence, $`\{w_1,\mathrm{},w_l,\stackrel{~}{w}_{l+1},\mathrm{},\stackrel{~}{w}_n\}`$ is a unitary basis for $`\stackrel{~}{TX}|_Y`$. We use $`\stackrel{~}{w}^\tau `$ to denote the parallel transport of $`\stackrel{~}{w}`$ with respect to the holomorphic Hermitian connection $`^{\stackrel{~}{TX}}`$ along the geodesic $`(y,tZ)`$. We write $`v`$ on $`𝒰_ϵ`$ as

$$v(y,Z)=\frac{1}{\sqrt{2}}\left(\underset{i=1}{\overset{l}{}}v^i(y,Z)w_i^\tau +\underset{\alpha =l+1}{\overset{n}{}}v^\alpha (y,Z)\stackrel{~}{w}_\alpha ^\tau \right)$$
$`(2.27)`$
for some smooth functions $`v^i`$ and $`v^\alpha `$. Set

$$v_{Y,1}=\frac{1}{\sqrt{2}}\underset{i=1}{\overset{l}{}}\underset{\alpha =l+1}{\overset{n}{}}\frac{v^i}{z^\alpha }(y)z^\alpha w_i,v_{N,1}=\frac{1}{\sqrt{2}}\underset{\alpha ,\beta =l+1}{\overset{n}{}}\frac{v^\alpha }{z^\beta }(y)z^\beta \stackrel{~}{w}_\alpha ,$$
$`(2.28)`$
$$v_{Y,2}=\frac{1}{2\sqrt{2}}\underset{i=1}{\overset{l}{}}\underset{\alpha ,\beta =l+1}{\overset{n}{}}\frac{^2v^i}{z^\alpha z^\beta }(y)z^\alpha z^\beta w_i,v_{N,2}=\frac{1}{2\sqrt{2}}\underset{\alpha ,\beta ,\gamma =l+1}{\overset{n}{}}\frac{^2v^\alpha }{z^\beta z^\gamma }(y)z^\beta z^\gamma \stackrel{~}{w}_\alpha .$$
$`(2.29)`$
Since $`v`$ is transversal, we get by the definition (2.26),

$$v_{Y,1}=0,v_{N,1}=\frac{1}{\sqrt{2}}\underset{\alpha =l+1}{\overset{n}{}}z^\alpha \stackrel{~}{w}_\alpha ,$$
$`(2.30)`$
and so

$$v(y,Z)=v_{N,1}(y,Z)+v_{Y,2}(y,Z)+v_{N,2}(y,Z)+O(|Z|^3).$$
$`(2.31)`$
Define

$$D_T^N=D^N+\sqrt{1}T(\widehat{c}(v_{N,1})+\widehat{c}(\overline{v}_{N,1})).$$
$`(2.32)`$
A direct and easy computation shows that

$$(D_T^N)^2=4\underset{\alpha =l+1}{\overset{n}{}}\frac{^2}{z^\alpha \overline{z}^\alpha }+T^2|Z|^2\sqrt{1}T\underset{\alpha =l+1}{\overset{n}{}}(c(\overline{w}_\alpha )\widehat{c}(\stackrel{~}{w}_\alpha )+c(w_\alpha )\widehat{c}(\overline{\stackrel{~}{w}}_\alpha )).$$
$`(2.33)`$
Set

$$\theta =\underset{\alpha =l+1}{\overset{n}{}}w_\alpha ^{}\overline{\stackrel{~}{w}}_\alpha ^{}.$$
$`(2.34)`$
Clearly, $`\theta `$ is a well-defined smooth section of $`\mathrm{\Lambda }^{()}(\overline{N}^{}\stackrel{~}{N}^{})`$ over $`Y`$ of the degree 0. Now we have the following analogue of \[BL, Proposition 7.3\]:

Lemma 2.3 Take $`T>0`$. Then for any $`yY`$, the operator $`(D_T^N)^2`$ acting on $`\mathrm{\Gamma }\left(\pi ^{}\mathrm{\Lambda }^{()}(\overline{N}_y^{}\stackrel{~}{N}_y)\right)`$ over $`N_y`$ is nonnegative with the kernel $`𝐂\{\beta _y\}`$, where

$$\beta _y=\mathrm{exp}(\theta _y\frac{T}{2}|Z|^2),|\mathrm{exp}\theta _y|_{\mathrm{\Lambda }^{()}(\overline{N}_y^{}\stackrel{~}{N}_y)}=2^{(nl)/2}.$$
$`(2.35)`$
Moreover, the nonzero eigenvalues of $`(D_T^N)^2`$ are all $`TA`$ for some positive constant $`A`$ which can be chosen to be independent of $`y`$.

Proof. The proof of the lemma is standard (cf. \[BL, Sect.7, (7.10)–(7.13)\]; also cf. \[Z1, Chapter 4, Sect. 4.5\]). $`\mathrm{}`$

For any $`yY`$, $`ZN_{𝐑,y}`$, let $`\tau U`$ denote the parallel transport of $`UT_{𝐑,y}X`$ with respect to the Levi-Civita connection $`^{L^X}`$ along the geodesic $`(y,tZ)`$. Note that we have identified the bundle $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)|_{B_{ϵ_0}}`$ with the bundle $`(\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_{𝒰_{ϵ_0}}`$ by trivializing the later bundle along the geodesic $`(y,tZ)`$ by using the connection $`^{B,X}`$. The Clifford action of $`c((\tau U)(y,tZ))`$ on $`\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi `$ is generally not constant along the geodesic $`(y,tZ)`$. This is different from the situation in \[BL, Sect.8\], where the connection on the related bundle is the lifting of the Levi-Civita connection on $`TX`$ since the manifold $`X`$ is K$`\ddot{\mathrm{a}}`$hler. Hence, to obtain an analogue of \[BL, Theorem 8.18\], we need to work out the difference between $`c((\tau U)(y,Z))`$ and the constant Clifford action $`c(U)`$ on $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)|_{B_{ϵ_0}}`$. Since $`^{B_X}=^{L^X}+S^{B_X}`$ is unitary, we know that

$$[_Z^{B,X},c((\tau U)(y,tZ))]_{|_{t=0}}=c(_Z^{B_X}(\tau U)(y,tZ))_{|_{t=0}}=c(S^{B_X}(Z)U),$$
$`(2.36)`$
thus

$$c((\tau U)(y,Z))=c(U)+c(S^{B_X}(Z)U)+O(|Z|^2).$$
$`(2.37)`$
Set with respect to the basis (2.24):

$$M=\frac{1}{2}\underset{i,j}{\overset{2l}{}}\underset{\alpha =2l+1}{\overset{2n}{}}𝐀(e_i)e_j,e_\alpha c(e_i)c(e_j)c(e_\alpha )\frac{1}{2}\underset{\alpha =2l+1}{\overset{2n}{}}(e_\alpha k)c(e_\alpha ),$$
$`(2.38)`$
$$c(B^{}(y))=\frac{1}{2}\underset{i,j=1}{\overset{2l}{}}\underset{\alpha =2l+1}{\overset{2n}{}}B_X(e_i,e_j,e_\alpha )c(e_i)c(e_j)c(e_\alpha )$$

$$+\frac{1}{6}\underset{\alpha ,\beta ,\gamma =2l+1}{\overset{2n}{}}B_X(e_\alpha ,e_\beta ,e_\gamma )c(e_\alpha )c(e_\beta )c(e_\gamma ),$$
$`(2.39)`$
$$c(B_{}^{}{}_{}{}^{}(y))=\frac{1}{2}\underset{i=1}{\overset{2l}{}}\underset{\alpha ,\beta =2l+1}{\overset{2n}{}}B_X(e_i,e_\alpha ,e_\beta )c(e_i)c(e_\alpha )c(e_\beta ).$$
$`(2.40)`$
One verifies easily that

$$c(B_X(y))=c(B_Y(y))+c(B^{}(y))+c(B_{}^{}{}_{}{}^{}(y)).$$
$`(2.41)`$
Now we have the following analogue of \[BL, Theorem 8.18\], which describes the local behavior of $`D_T^X`$ near $`Y`$. Comparing to \[BL, Theorem 8.18, (8.58)\], some new terms enter into the following theorem.

Theorem 2.4 As $`T+\mathrm{}`$, then

$$k^{1/2}D_T^Xk^{1/2}=D^H+D_T^N+M_T+c(B_{}^{}{}_{}{}^{})+T\sqrt{1}\widehat{c}(v_{Y,2}+\overline{v}_{Y,2})+𝐒+R_T,$$
$`(2.42)`$
where

$$M_T=M+c(B^{})+T\sqrt{1}\widehat{c}(v_{N,2}+\overline{v}_{N,2}),$$
$`(2.43)`$
$$𝐒=\underset{i=1}{\overset{2n}{}}c(e_i)_{P^{N_𝐑}S^{B_X}(Z)e_i}^{B,X,},$$
$`(2.44)`$
$$R_T=O(|Z|^H+|Z|^2^N+|Z|+T|Z|^3),$$
$`(2.45)`$
and $`^H`$, $`^N`$ represent horizontal and vertical differential operators, respectively.

Proof. Let $`\{\tau e_1,\mathrm{},\tau e_{2n}\}`$ be the parallel transport of the basis (2.24) with respect to the Levi-Civita connection $`^{L^X}`$ along the geodesic $`(y,tZ)`$ for $`yY`$ and $`ZN_{𝐑,y}`$. From (2.21), we have

$$D_T^X=\underset{i=1}{\overset{2n}{}}c(\tau e_i)_{\tau e_i}^{B,X}+2c(B_X)+\sqrt{1}T(\widehat{c}(v)+\widehat{c}(\overline{v})).$$
$`(2.46)`$
We identify $`\overline{TX}`$ with $`\pi ^{}(\overline{TX}|_Y)`$ over $`𝒰_{ϵ_0}`$ by trivializing $`\overline{TX}`$ with respect to the Bismut connection $`^{B_X}`$ along the geodesic $`(y,tZ)`$ and set

$$\mathrm{\Gamma }=^{B_X}^{B_X,}.$$
$`(2.47)`$
Let $`\mathrm{\Gamma }^{}`$ denote the lifting action of $`\mathrm{\Gamma }`$ on $`\pi ^{}(\mathrm{\Lambda }^{}(\overline{T^{}X})|_Y)`$. For any $`yY`$, we find by (2.12)

$$\underset{i=1}{\overset{2n}{}}c(e_i)\mathrm{\Gamma }_y^{}(e_i)=\frac{1}{2}\underset{i,j}{\overset{2l}{}}\underset{\alpha =2l+1}{\overset{2n}{}}𝐀_y(e_i)e_j,e_\alpha c(e_i)c(e_j)c(e_\alpha ).$$
$`(2.48)`$
Furthermore, recall (2.13) and then we get

$`k^{1/2}D_T^Xk^{1/2}`$ $`=`$ $`{\displaystyle \underset{i=1}{\overset{2n}{}}}c(\tau e_i)_{\tau e_i}^{B,X,}+2c(B_X)+\sqrt{1}T(\widehat{c}(v)+\widehat{c}(\overline{v}))`$
$`+{\displaystyle \underset{i=1}{\overset{2n}{}}}c(e_i)\mathrm{\Gamma }_y^{}(e_i){\displaystyle \frac{1}{2}}{\displaystyle \underset{\alpha =2l+1}{\overset{2n}{}}}(e_\alpha k)(y)c(e_\alpha )+O(|Z|).`$
Note that (2.31), (2.38), (2.41), (2.43) and (2.48), we have

$`k^{1/2}D_T^Xk^{1/2}`$ $`=`$ $`{\displaystyle \underset{i=1}{\overset{2n}{}}}c(\tau e_i)_{\tau e_i}^{B,X,}+2c(B_Y)+T\sqrt{1}\widehat{c}(v_{N,1}+\overline{v}_{N,1})`$
$`+M_T+c(B_{}^{}{}_{}{}^{})+T\sqrt{1}\widehat{c}(v_{Y,2}+\overline{v}_{Y,2})+O(|Z|+T|Z|^3).`$
By (2.37) and the expansion of $`\tau e_i`$ along $`(y,tZ)`$ in the proof of \[BL, Theorem 8.18\], especially \[BL, (8.80), (8.84)\], we have

$`{\displaystyle \underset{i=1}{\overset{2n}{}}}c(\tau e_i)_{\tau e_i}^{B,X,}`$ $`=`$ $`{\displaystyle \underset{i=1}{\overset{2l}{}}}c(e_i)_{e_i^H}^{B,X,}+{\displaystyle \underset{\alpha =2l+1}{\overset{2n}{}}}c(e_\alpha )_{e_\alpha }^{B,X,}`$
$`+{\displaystyle \underset{i=1}{\overset{2n}{}}}c(S^{B_X}(Z)e_i)_{e_i}^{B,X,}+O(|Z|^H+|Z|^2^N),`$
and then by the definition of $`D^H`$ and $`D_T^N`$,

$`k^{1/2}D_T^Xk^{1/2}`$ $`=`$ $`D^H+D_T^N+M_T+c(B_{}^{}{}_{}{}^{})+T\sqrt{1}\widehat{c}(v_{Y,2}+\overline{v}_{Y,2})`$
$`+{\displaystyle \underset{i=1}{\overset{2n}{}}}c(S^{B_X}(Z)e_i)_{e_i}^{B,X,}+O(|Z|^H+|Z|^2^N+|Z|+T|Z|^3).`$
But

$`{\displaystyle \underset{i=1}{\overset{2n}{}}}c(S^{B_X}(Z)e_i)_{e_i}^{B,X,}`$ $`=`$ $`{\displaystyle \underset{i=1}{\overset{2n}{}}}c(e_i)_{S^{B_X}(Z)e_i}^{B,X,}`$
$`=`$ $`{\displaystyle \underset{i=1}{\overset{2n}{}}}c(e_i)_{P^{N_𝐑}S^{B_X}(Z)e_i}^{B,X,}{\displaystyle \underset{i=1}{\overset{2n}{}}}c(e_i)_{P^{T_𝐑Y}S^{B_X}(Z)e_i}^{B,X,}`$
$`=`$ $`𝐒+O(|Z|^H),`$
from which we complete the proof of the theorem. $`\mathrm{}`$

## 3 The proof of Theorem 1.1

In this section, we prove Theorem 1.1 by using Bismut-Lebeau’s techniques in \[BL, Sect.9, Sect.10, a)\].

For any $`\mu 0,`$ let $`\mathrm{E}^\mu `$ (resp. $`𝐄^\mu `$, resp. $`\mathrm{F}^\mu `$) be the set of sections of $`\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi `$ on $`X`$ (resp. of $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)`$ on the total space of $`N`$, resp. of $`\mathrm{\Lambda }^{()}(T_𝐂^{}Y)\xi |_Y`$ on Y) which lie in the $`\mu `$-th Sobolev spaces. Let $`_{\mathrm{E}^\mu }`$ (resp. $`_{𝐄^\mu }`$, resp. $`_{\mathrm{F}^\mu }`$) be the Sobolev norm on $`\mathrm{E}^\mu `$ (resp. $`𝐄^\mu `$, resp. $`\mathrm{F}^\mu `$). We always assume that the norms $`_{\mathrm{E}^0}`$ (resp. $`_{𝐄^0}`$, resp. $`_{\mathrm{F}^0}`$ is the norm associated with the scalar products on the corresponding bundles).

Let $`\gamma :𝐑[0,1]`$ be a smooth even function with $`\gamma (a)=1`$ if $`|a|\frac{1}{2}`$ and $`\gamma (a)=0`$ if $`|a|1.`$ For any $`yY`$, $`ZN_y`$ and $`ϵ(0,ϵ_0)`$, where $`ϵ_0`$ is chosen as in Section 2, $`𝐜)`$, set

$$\gamma _ϵ(Z)=\gamma \left(\frac{|Z|}{ϵ}\right),\alpha _T=_{N_{𝐑,y}}\gamma _ϵ^2(Z)\mathrm{exp}(T|Z|^2)\frac{d\upsilon _{N_y}(Z)}{(2\pi )^{dim_𝐂N}}.$$
$`(3.1)`$
Clearly, $`\alpha _T`$ does not depend on $`yY`$ and $`\alpha _T=O(\frac{1}{T^{nl}})`$.

For $`\mu 0,T>0,`$ define linear maps $`I_T:\mathrm{F}^\mu 𝐄^\mu `$ and $`J_T:\mathrm{F}^\mu \mathrm{E}^\mu `$ by

$$I_Tu=\left(\frac{1}{2^{nl}\alpha _T}\right)^{1/2}\gamma _ϵ(Z)(\pi ^{}u)\beta _y,J_Tu=k^{1/2}I_Tu,u\mathrm{F}^\mu .$$
$`(3.2)`$
It is easy to see that $`I_T`$, $`J_T`$ are isometries from $`\mathrm{F}^0`$ onto their images. For $`\mu 0,T>0,`$ let $`𝐄_T^\mu `$ (resp. $`\mathrm{E}_T^\mu `$) be the image of $`\mathrm{F}^\mu `$ in $`𝐄^\mu `$ (resp. $`\mathrm{E}^\mu `$) under $`I_T`$ (resp. $`J_T`$) and let $`𝐄_T^{0,}`$ (resp. $`\mathrm{E}_T^{0,}`$) be the orthogonal complement of $`𝐄_T^0`$ (resp. $`\mathrm{E}_T^0`$) in $`𝐄^0`$ (resp. $`\mathrm{E}^0`$) and let $`p_T`$, $`p_T^{}`$ (resp. $`\overline{p}_T,\overline{p}_T^{}`$) be the orthogonal projection operators from $`𝐄^0`$ (resp. $`\mathrm{E}^0`$) onto $`𝐄_T^0,𝐄_T^{0,}`$ (resp. $`\mathrm{E}_T^0,\mathrm{E}_T^{0,}`$), respectively. Set

$$\mathrm{E}^{\mu ,}=\mathrm{E}^\mu \mathrm{E}_T^{0,}.$$
$`(3.3)`$
Then $`\mathrm{E}^0`$ splits orthogonally into

$$\mathrm{E}^0=\mathrm{E}_T^0\mathrm{E}_T^{0,}.$$
$`(3.4)`$
Since the map $`s𝐄^0k^{1/2}s\mathrm{E}^0`$ is an isometry, we see that the map $`sk^{1/2}s`$ identifies the Hilbert space $`𝐄_T^0`$ and $`\mathrm{E}_T^0.`$ Corresponding to the decomposition (3.4) we set:

$$D_{T,1}=\overline{p}_TD_T^X\overline{p}_T,D_{T,2}=\overline{p}_TD_T^X\overline{p}_T^{},D_{T,3}=\overline{p}_T^{}D_T^X\overline{p}_T,D_{T,4}=\overline{p}_T^{}D_T^X\overline{p}_T^{}.$$
$`(3.5)`$
Then

$$D_T^X=D_{T,1}+D_{T,2}+D_{T,3}+D_{T,4}.$$
$`(3.6)`$
We have the following analogue of \[BL, Theorem 9.8\].

Lemma 3.1 The following formula holds on $`\mathrm{\Gamma }(\mathrm{\Lambda }^{()}(T_𝐂^{}Y)\xi |_Y)`$ as $`T+\mathrm{}`$

$$J_T^1D_{T,1}J_T=D^Y+O(\frac{1}{\sqrt{T}}),$$
$`(3.7)`$
where $`O(\frac{1}{\sqrt{T}})`$ is a first order differential operator with smooth coefficients dominated by $`C/\sqrt{T}`$.

Proof. Note that the action of the operator $`M_T`$ on $`\pi ^{}((\mathrm{\Lambda }^{()}(T_𝐂^{}X)\xi )|_Y)`$ interchanges $`\pi ^{}(\mathrm{\Lambda }^{(\mathrm{even})}(\overline{N}^{}\stackrel{~}{N}^{}))`$ and $`\pi ^{}(\mathrm{\Lambda }^{(\mathrm{odd})}(\overline{N}^{}\stackrel{~}{N}^{}))`$, we get

$$p_TM_Tp_T=0.$$
$`(3.8)`$
Note that $`B_X`$ is antisymmetric and $`c(e_\alpha )c(e_\beta )\beta _y,\beta _y=0`$ for any $`\alpha ,\beta `$ with $`2l+1\alpha <\beta 2n`$, we obtain

$$p_Tc(B_{}^{}{}_{}{}^{})p_T=0.$$
$`(3.9)`$
Since $`B_X`$ is antisymmetric, by (2.8) we get $`S^{B_X}(Z)e_i,Z=0`$ and so

$$_{P^{N_𝐑}S^{B_X}(Z)e_i}^{B,X,}\left(\gamma _ϵ(Z)\mathrm{exp}(\theta _y\frac{T|Z|^2}{2})\right)=\mathrm{exp}(\theta _y\frac{T|Z|^2}{2})(P^{N_𝐑}S^{B_X}(Z)e_i)\gamma _ϵ(Z).$$
$`(3.10)`$
From the equality above we can prove easily the following estimate for some uniformly positive constant $`C`$ and any $`s𝐄^1`$:

$$p_T𝐒s_{𝐄^0}\frac{C}{\sqrt{T}}s_{𝐄^1}.$$
$`(3.11)`$
Since $`_𝐂e^{T|z|^2}z^2𝑑z𝑑\overline{z}=0`$ and $`(\gamma _ϵ^21)`$ vanishes on a symmetric domain containing $`0`$, we have for any $`u\mathrm{F}`$ that

$$I_T^1p_T(\widehat{c}(v_{Y,2})+\widehat{c}(\overline{v}_{Y,2}))\left(\frac{1}{2^{nl}\alpha _T}\right)^{1/2}\gamma _ϵ(Z)(\pi ^{}u)\beta _y=O(\frac{1}{T^{3/2}}).$$
$`(3.12)`$
On the other hand, note that $`\beta _y`$ is of constant length on $`Y`$, we get for $`1i2l`$ that $`_{e_i}^{B,X,}\beta ,\beta =0`$ and so

$$I_T^1p_TD^Hp_TI_T=D^Y.$$
$`(3.13)`$
One can then proceed as in \[BL, Proof of Theorem 9.8\] and use (3.8)–(3.12) to complete the proof of Lemma 3.1 easily. $`\mathrm{}`$

Note that the estimate (3.12) and proceed as the proof of Theorem 9.10, Theorem 9.11 and Theorem 9.14 in \[BL, Sect.9\], one can prove the following lemma without any new difficulty.

Lemma 3.2 There exist $`C_1>0`$, $`C_2>0`$ and $`T_0>0`$ such that for any $`TT_0`$, $`s\mathrm{E}_T^{1,}`$ and $`s^{}\mathrm{E}_T^1,`$ then

$$D_{T,2}s_{\mathrm{E}^0}C_1\left(\frac{s_{\mathrm{E}^1}}{\sqrt{T}}+s_{\mathrm{E}^0}\right),$$
$`(3.14)`$
$$D_{T,3}s^{}_{\mathrm{E}^0}C_1\left(\frac{s^{}_{\mathrm{E}^1}}{\sqrt{T}}+s^{}_{\mathrm{E}^0}\right),$$
$`(3.15)`$
$$D_{T,4}s_{\mathrm{E}^0}C_2(s_{\mathrm{E}^1}+\sqrt{T}s_{\mathrm{E}^0}).$$
$`(3.16)`$
Let $`\mathrm{𝐒𝐩𝐞𝐜}(D^Y)`$ denote the spectrum of $`D^Y`$. Choose $`c>0`$ such that $`\mathrm{𝐒𝐩𝐞𝐜}(D^Y)[2c,2c]\{0\}.`$ Let $`\delta =\{\lambda 𝐂:|\lambda |=c\}`$. Let $`\mathrm{E}_c(T)`$ denote the direct sum of the eigenspaces of $`D_T^X`$ with eigenvalues lying in $`[c,c]`$. Then $`\mathrm{E}_c(T)`$ is a finite dimensional subspace of $`\mathrm{E}^0.`$ Let $`P_{T,c}`$ denote the orthogonal projection from $`J_T(\mathrm{ker}(D^Y))`$ to $`\mathrm{E}_c(T)`$. By Lemma 3.1 and Lemma 3.2, we have the following analogue of \[BL, (9.156)\] (also see \[TZ, Proposition 4.4\] for a proof without using the norm in \[BL, Sect.9, Definition 9.17\] and the distance in \[BL, Sect.9, Definition 9.22\]):

Theorem 3.3 There exist $`c>0`$ and $`T_0>0`$ such that for any $`TT_0`$, the projection

$$P_{T,c}:J_T(\mathrm{ker}(D^Y))\mathrm{E}_c(T)$$
$`(3.17)`$
is an isomorphism.

Now to prove Theorem 1.1 we only need to prove that when $`T`$ large enough, $`D_T^X`$ has no nonzero small eigenvalues or equivalently, to prove the following equality:

$$E_c(T)=\mathrm{ker}(D_T^X).$$
$`(3.18)`$
Let $`Q`$ denote the orthogonal projection from $`\mathrm{\Omega }^{()}(Y,\xi )`$ to $`\mathrm{ker}(D^Y)`$. Then we have the following analogue of \[Z2, Theorem 1.10\] (also see \[BL, Theorem 10.1, (10.4)\]):

Theorem 3.4 There exist $`c>0`$, $`C>0`$, $`T_1>0`$ such that for any $`TT_1`$, any $`\sigma \mathrm{ker}(D^Y)`$,

$$(2^{nl}\alpha _T)^{1/2}Qj^{}P_{T,c}J_T\sigma \sigma _0\frac{C}{\sqrt{T}}\sigma _0.$$
$`(3.19)`$
Proof. The proof of \[Z2, Theorem 1.10\], which is a modified version of the proof of \[BL, Theorem 10.1, (10.4)\], is carried out here with the identity \[Z2,(1.34)\] in the proof of the \[Z2, Theorem 1.10\] replaced by the following equality

$$j^{}\frac{1}{2\pi \sqrt{1}}_\delta k^{1/2}\gamma _ϵ\frac{(\pi ^{}\sigma )\beta _y}{\lambda }𝑑\lambda =\sigma .$$
$`(3.20)`$
The proof of (3.20) is similar to that of the identity \[BL, (10.29)\]. $`\mathrm{}`$

Note that $`j^{}\beta _y=1`$ is crucial in the proof of (3.19). It is no longer true for the case of the analytic proof of Morse inequalities of Witten (\[W\]) since in that case the contribution of the bundle $`\mathrm{\Lambda }(N^{})`$ to the kernel of $`D_T^N`$ is a pure $`p`$-form around each critical point of index $`p`$ (cf. \[Z1, Chapter 5, 6\]) and its pull-back by $`j^{}`$ vanishes on $`Y`$. Consequently, $`j^{}`$ can not be a quasi-isomorphism at all in that case.

Proof of Theorem 1.1. The proof of Theorem 1.1 now is similar to that in \[Z2, Sect.1, e)\]. Note that the trick used in Zhang’s proof and so ours is inspired by Braverman (\[Br, Sect.3\]). First of all, we know from Theorem 3.3 that

$$P_{T,c}J_T:\mathrm{ker}(D^Y)\mathrm{E}_c(T)$$
$`(3.21)`$
is an isomorphism when $`T`$ is very large. Take $`\alpha \mathrm{E}_c(T)`$. Then $`\overline{}_T^X\alpha \mathrm{E}_c(T)`$. By the above discussion, there exists $`\sigma \mathrm{ker}(D^Y)`$ such that

$$\overline{}_T^X\alpha =(2^{nl}\alpha _T)^{1/2}P_{T,c}J_T\sigma .$$
$`(3.22)`$
From (3.22) and that $`j^{}`$ is a quasi-homomorphism, i.e. $`j^{}\overline{}_T^X=\overline{}^Yj^{}`$, we have

$$(2^{nl}\alpha _T)^{1/2}Qj^{}P_{T,c}J_T\sigma =Qj^{}\overline{}_T^X\alpha =Q\overline{}^Yj^{}\alpha =0.$$
$`(3.23)`$
From (3.23) and (3.19), we get

$$\sigma _0\frac{C}{\sqrt{T}}\sigma _0,$$
$`(3.24)`$
and so $`\sigma =0`$ as $`T`$ large enough. Thus, when $`T`$ is large enough, we have that

$$\overline{}_T^X|_{\mathrm{E}_c(T)}=0.$$
$`(3.25)`$
From (3.25) and Theorem 3.3, we have that when $`T`$ is large enough,

$$dim\mathrm{ker}(D_T^X)=dim\mathrm{E}_c(T)=dim\mathrm{ker}(D^Y).$$
$`(3.26)`$
Now by Theorem 3.4,

$$j^{}:\mathrm{ker}(D_T^X)\mathrm{ker}(D^Y)$$
$`(3.27)`$
is clearly an injective and so an isomorphism from (3.26). $`\mathrm{}`$

Acknowledgements The author would like to thank Professors Jean-Michel Bismut and Xiaonan Ma for many helpful discussions from which the author benefits a lot. This work was done while the author was visiting the Institut des Hautes $`\stackrel{´}{\mathrm{E}}`$tudes Scientifiques in Bures-Sur-Yvette. He would like to thank Professor Jean-Pierre Bourguignon and the IHES for their hospitality and support.

References

\[Bis1\] J. -M. Bismut, Holomorphic and de Rham torsions, preprint.

\[Bis2\] J. -M. Bismut, A local index theorem for non K$`\ddot{\mathrm{a}}`$hler manifolds, Math. Ann. 284(1989), 681-699.

\[BL\] J. -M. Bismut and G. Lebeau, Complex immersions and Quillen metrics, Publ. Math. IHES. V.74 (1991).

\[Bot\] R. Bott, A residue formula for holomorphic vector fields, J. Diff. Geom. 1(1967), 311-330.

\[Br\] M. Braverman, Cohomology of the Mumford quotient, Progress in Math. 198(2001), 47-59.

\[CL\] J. B. Carrell and D. I. Lieberman, Holomorphic vector fields and Kaehler manifolds, Invent. Math. 21 (1973), 303-309.

\[L\] K. Liu, Holomorphic equivariant cohomolgy, Math. Ann. 303(1995), 125-148.

\[TZ\] Y. Tian and W. Zhang, Quantization formula for symplectic manifolds with boundary, Geom. Funct. Anal. 9(1999), 596-640.

\[W\] E. Witten, Supersymmetry and Morse theory, J. Diff. Geom. 17(1982), 661-692.

\[Z1\] W. Zhang, Lectures on Chern-Weil Theory and Witten Deformations, Nankai Tracts in Mathematics, V.4, World Scientific, Singapore, 2001.

\[Z2\] W. Zhang, A holomorphic quantization formula in singular reduction, Commun. Contemp. Math. 1(1999), No.3, 281-293.

Feng: College of Mathematical Sciences, Nankai University, Tianjin, 300071, China;
fht@nankai.edu.cn

Current address:

Feng: Institut Des Hautes $`\stackrel{´}{\mathrm{E}}`$tudes Scientifiques, Bures-Sur-Yvette, 91440, France;
feng@ihes.fr


text:
# 1 U(1)FN charge assignment in SU(5) GUT. Here 𝑎=0 or 1 and 0≤𝑏≤𝑐≤𝑑.

Recent neutrino experiments have provided quite convincing evidence for neutrino masses and their flavour mixings. The atmospheric neutrino anomaly is explained by the $`\nu _\mu `$-$`\nu _\tau `$ oscillations with $`\delta m_{\mathrm{atm}}^22.5\times 10^3\text{eV}^2`$ and $`\mathrm{sin}^22\theta _{\mathrm{atm}}1.0`$ , while the solar neutrino problem is now solved by the so-called “large-mixing angle solution” with $`\delta m_{\mathrm{sol}}^27\times 10^5\text{eV}^2`$ and $`\mathrm{tan}^2\theta _{\mathrm{sol}}0.5`$ . These developments give us an important clue to physics beyond the Standard Model.

The most natural extension is probably to introduce right-handed neutrinos with superheavy Majorana masses, since they induce the observed suppressed masses of neutrinos through the seesaw mechanism . Further, non-equilibrium decays of right-handed neutrinos in the early Universe offer one natural way to generate the present baryon asymmetry . Then, grand unified theory (GUT) based on SO(10) gauge group  is particularly attractive, since all quarks and leptons of each family are unified into a single spinor representation together with the right-handed neutrino. For unification of gauge couplings and stabilization of the gauge hierarchy we had better incorporate supersymmetry.

The observed mixing angles among neutrino flavours are both large, which is completely different from the quark sector. This is a big challenge in constructing a realistic model of SO(10) GUT, since it describes quarks and leptons in unified way. One simple possibility has been proposed in the so-called “lopsided” models . Mass matrices for down quarks and leptons are arranged to have the lopsided structure, i.e., off-diagonal elements appear in a lopsided way such that mixings of left-handed leptons are large while those of left-handed down quarks are small, which give desired mixings of quarks and leptons in the charged current. There have been proposed so far many models to realize the lopsided mass matrices .

Such mass matrices can be based on the Froggatt-Nielsen (FN) mechanism . It has been shown in Ref.  by using SU(5) GUT that the observed mass hierarchies of quarks and charged leptons are explained by their lopsided charges under some FN flavour symmetry, and the large $`\nu _\mu `$-$`\nu _\tau `$ mixing angle in the atmospheric neutrinos is a direct consequence of such lopsided charge assignment. In this letter we would like to realize this idea in the framework of SO(10) GUT and show one possibility to yield the successful mass matrices obtained in the lopsided SU(5) model. We also discuss cosmological implication of the model, in particular, possible scenarios of leptogenesis in the inflationary Universe.

Masses of quarks and charged leptons approximately satisfy the following relations at the unification scale $`M_{GUT}2\times 10^{16}`$ GeV:

$`m_u:m_c:m_tϵ^4:ϵ^2:1,`$
$`m_e:m_\mu :m_\tau m_d:m_s:m_bϵ^3:ϵ:1,`$ (1)
where $`ϵ1/16`$. One attractive attempt to understand mass hierarchies and mixings among fermions consists in calling upon the Froggatt-Nielsen (FN) mechanism . This mechanism can be based on U(1)FN flavor symmetry, which is broken by a vacuum expectation value (vev) of a gauge singlet field $`S_{FN}`$ carrying the $`U(1)_{FN}`$ charge $`1`$.111We take here U(1)FN as the FN flavour symmetry by way of illustration. This U(1)FN can be replaced by some discrete symmetry which is anomaly-free. Matter $`f_i`$ and Higgs $`H`$ superfields are introduced with charges $`Q_i`$ ($`Q_i0`$) and 0, respectively. Then, the U(1)FN flavor symmetry allows the following Yukawa terms

$`W=c_{ij}\left({\displaystyle \frac{S_{FN}}{M_{}}}\right)^{Q_i+Q_j}Hf_if_j,`$ (2)
where $`M_{}`$ denotes the fundamental cutoff scale of the theory and we regard it as the gravitational scale. By assuming that constant coefficients $`c_{ij}`$ are of order unity, fermion mass matrices scale as $`m_{ij}ϵ^{Q_i+Q_j}`$ depending on the U(1)FN charges, where the parameter $`ϵ`$ is defined by

$`ϵ{\displaystyle \frac{S_{FN}}{M_{}}}.`$ (3)
We set $`ϵ1/16`$ from the mass relations shown in Eq. (1).

The above FN mechanism can be used to construct lopsided mass matrices for down quarks and leptons, which explain the fermion mass relations in Eq. (1) as well as the observed large neutrino mixing angles while keeping quark mixings small. First, we shall briefly review this point by using SU(5) model, which have been discussed in Refs. .

In SU(5) GUT, one family of quarks and leptons can be grouped into the SU(5) multiplets, $`\mathrm{𝟏𝟎}`$-plet, $`\mathrm{𝟓}^{}`$-plet and $`\mathrm{𝟏}`$-plet. Their Yukawa interactions are given by

$`W=h^u{}_{ij}{}^{}H_{u}^{}\mathrm{𝟏𝟎}_i\mathrm{𝟏𝟎}_j+h^d{}_{ij}{}^{}H_{d}^{}\mathrm{𝟏𝟎}_i\mathrm{𝟓}_{}^{}{}_{j}{}^{}+h^D{}_{ij}{}^{}H_{u}^{}\mathrm{𝟏}_i\mathrm{𝟓}_{}^{}{}_{j}{}^{}+h^N{}_{ij}{}^{}S\mathrm{𝟏}_i\mathrm{𝟏}_j,`$ (4)
where $`H_u`$ and $`H_d`$ denote Higgs fields of $`\mathrm{𝟓}`$-plet and $`\mathrm{𝟓}^{}`$-plet and their vevs are denoted by $`v_2`$ and $`v_1`$, respectively. Here we assumed that Majorana masses for right-handed neutrinos come from the vev of the singlet Higgs $`S`$. The hierarchies in the Yukawa couplings $`h^{u,d,D,N}`$ are explained by the FN mechanism. In Table 1 we show the U(1)FN charge assignment for matter fields . Then, we obtain mass matrices for quarks and charged leptons as

$`m^u`$ $`=`$ $`v_2h^u=v_2\left(\begin{array}{ccc}ϵ^4& ϵ^3& ϵ^2\\ ϵ^3& ϵ^2& ϵ\\ ϵ^2& ϵ& 1\end{array}\right),m^e=(m^d)^T=v_1h^d=v_1ϵ^a\left(\begin{array}{ccc}ϵ^3& ϵ^2& ϵ^2\\ ϵ^2& ϵ& ϵ\\ ϵ& 1& 1\end{array}\right).`$ (11)
It should be noted that every component in these mass matrices contains a coefficient of order unity, which are implicitly assumed here and hereafter. It is found that the “lopsided” charge assignment between $`\mathrm{𝟏𝟎}`$-plets and $`\mathrm{𝟓}^{}`$-plets leads to the lopsided mass matrices for down quarks and charged leptons (compared with the mass matrix for up quarks), which is crucial for explaining the mass relations given in Eq. (1). The mixing angles for quarks are found to be small.

On the other hand, the charge assignment in Table 1 can also explain large angles of neutrino flavour mixings. Dirac and Majorana mass matrices for neutrinos, $`m^D`$ and $`m^N`$, are given by

$`m^D=v_2h^D=v_2ϵ^a\left(\begin{array}{ccc}ϵ^{d+1}& ϵ^d& ϵ^d\\ ϵ^{c+1}& ϵ^c& ϵ^c\\ ϵ^{b+1}& ϵ^b& ϵ^b\end{array}\right),m^N=Sh^N=S\left(\begin{array}{ccc}ϵ^{2d}& 0& 0\\ 0& ϵ^{2c}& 0\\ 0& 0& ϵ^{2b}\end{array}\right),`$ (18)
where we have chosen a basis where $`m^N`$ is diagonal and real. It is seen that $`m^D`$ takes also the lopsided form. Then, the seesaw mechanism  brings us the mass matrix for three light neutrinos as

$`m^\nu =(m^D)^T{\displaystyle \frac{1}{m^N}}m^D={\displaystyle \frac{v_2^2}{S}}ϵ^{2a}\left(\begin{array}{ccc}ϵ^2& ϵ& ϵ\\ ϵ& 1& 1\\ ϵ& 1& 1\end{array}\right).`$ (22)
Notice that the dependence on the FN charges of right-handed neutrinos drops off in this expression. It is clearly seen that the angle of a $`\nu _\mu `$-$`\nu _\tau `$ mixing is large, which is observed in the atmospheric neutrino experiments. This is a direct consequence of the fact that $`\mathrm{𝟓}^{}`$-plets in the second and third families have the same FN charges in order to explain the mass hierarchies of down quarks and charged leptons . Furthermore, it have been shown in Ref.  that there is no difficulty in this mass matrix to give the large mixing angle in the solar neutrinos.

The above lopsided SU(5) model gives the successful mass matrices for quarks and leptons, to the first approximation. To explain the mass matrices more precisely, we have to include effects of SU(5) breaking. Otherwise, the unwanted SU(5) mass relations $`m_e=m_d`$ and $`m_\mu =m_s`$ are obtained. This can be achieved by introducing additional Higgs fields. However, this issue is beyond the scope of this analysis.

In SO(10) GUT, one family of quarks and leptons can be simply grouped into an irreducible spinor representation (16-plet).222 The representations with and without underline correspond to those under SO(10) and SU(5) gauge groups, respectively.  If it is the case, quarks and leptons in each family possess the same FN charge and all mass matrices of quarks and leptons have the same hierarchical structure. This means that we fail to explain the mass relations (1). To avoid this difficulty, we introduce three 16-plets $`\psi _i`$ ($`i=1,2,3`$) and also one additional 10-plet $`\eta `$ as matter fields:

$`\psi _i=(\mathrm{𝟏𝟎}_i,\mathrm{𝟓}^{}{}_{i}{}^{},\mathrm{𝟏}_i)\text{and}\eta =(\mathrm{𝟓}_4,\mathrm{𝟓}^{}{}_{4}{}^{}),`$ (23)
where we have shown the decomposition under SU(5) group. In Table 2 we show the FN charges for these matter superfields. We determine the charges of $`\psi _i`$ from the mass hierarchy of up quarks and assign zero charge to $`\eta `$. In the followings, we will exchange one combination of $`\mathrm{𝟓}_i^{}`$ in $`\psi _i`$ for $`\mathrm{𝟓}_4^{}`$ in $`\eta `$, which have been proposed in the different SO(10) model . By this exchange we can have the lopsided structure between $`\mathrm{𝟏𝟎}`$-plets and $`\mathrm{𝟓}^{}`$-plets in three families.

We introduce here the following Higgs superfields

$`H_1(\underset{¯}{\mathrm{𝟏𝟎}}),H_2(\underset{¯}{\mathrm{𝟏𝟎}}),\mathrm{\Phi }(\underset{¯}{\mathrm{𝟏𝟔}}),\mathrm{\Phi }^c(\underset{¯}{\mathrm{𝟏𝟔}}^{}),\mathrm{\Sigma }(\underset{¯}{\mathrm{𝟏𝟔}})\text{and}\mathrm{\Sigma }^c(\underset{¯}{\mathrm{𝟏𝟔}}^{}).`$ (24)
Two Higgs doublets which couple to down quarks and up quarks are assumed to be contained in $`H_1`$ and $`H_2`$, respectively.333 In the SO(10) models discussed in Refs.  one 10-plet and one 16-plet are introduced for the Higgs doublets. The considering two 10-plets for the weak Higgs doublets are a natural consequence of the model in six dimensions where SO(10) breaking is achieved by orbifold compactification . In this case, the mass splitting between the weak doublet and the color triplet Higgs fields is realized naturally.  We assume the vevs of these Higgs fields as follows:

$`H_1=v_1,H_2=v_2,\mathrm{\Phi }=\mathrm{\Phi }^c=v_\mathrm{\Phi }\text{and}\mathrm{\Sigma }=\mathrm{\Sigma }^c=v_\mathrm{\Sigma },`$ (25)
where $`v_1^2+v_2^2=(174\text{GeV})^2`$ and $`v_\mathrm{\Phi }`$ and $`v_\mathrm{\Sigma }`$ are of order of the unification scale $`M_{GUT}`$ which keep an SU(5) subgroup unbroken. All these Higgs fields carry zero charge under the U(1)FN symmetry (see, however, the discussion in the footnote 4.).

Now we are at the point to discuss mass matrices for quarks and leptons. The Yukawa interactions we shall consider here are given by the following superpotential:

$`W`$ $`=`$ $`{\displaystyle \frac{1}{2}}h_{ij}^dH_1\psi _i\psi _j+{\displaystyle \frac{1}{2}}h_{ij}^uH_2\psi _i\psi _j+{\displaystyle \frac{1}{2}}h_{ij}^n{\displaystyle \frac{\mathrm{\Phi }^c\mathrm{\Phi }^c}{M_{}}}\psi _i\psi _j`$ (26)
$`+g_i^\eta \mathrm{\Sigma }\psi _i\eta +g_i^d{\displaystyle \frac{\mathrm{\Sigma }^c}{M_{}}}H_1\psi _i\eta +g_i^u{\displaystyle \frac{\mathrm{\Sigma }^c}{M_{}}}H_2\psi _i\eta ,`$
where $`h^{d,u,n}`$ and $`g^{\eta ,d,u}`$ denote Yukawa coupling constants which are explained by the FN charges of matter fields. In this equation, the first two terms give usual Dirac masses for quarks and leptons, the third term gives Majorana masses for right-handed neutrinos, and the rest three terms denote mass mixings between matter 16-plets and additional 10-plet.

Let us first discuss the exchange of $`\mathrm{𝟓}^{}`$-plets in matter fields. The vev of the $`\mathrm{\Sigma }`$ field induces

$`W=g_i^\eta v_\mathrm{\Sigma }\mathrm{𝟓}_4\mathrm{𝟓}_{}^{}{}_{i}{}^{},`$ (27)
which gives a Dirac mass for one linear combination (denoted by $`\mathrm{𝟓}_H^{}`$) among three $`\mathrm{𝟓}_{}^{}{}_{i}{}^{}`$ in $`\psi _i`$, while $`\mathrm{𝟓}_{}^{}{}_{4}{}^{}`$ in $`\eta `$ is still massless . From the FN charge assignment in Table 2 the Yukawa couplings $`g^\eta _i`$ are given by $`g^\eta {}_{i}{}^{}=(ϵ^2,ϵ,1)`$ with $`ϵ1/16`$. Note again that coefficients of order unity are implicitly assumed. We find, then, the dominant component of $`\mathrm{𝟓}_{}^{}{}_{H}{}^{}`$ is $`\mathrm{𝟓}_{}^{}{}_{3}{}^{}`$:

$`\mathrm{𝟓}_{}^{}{}_{H}{}^{}ϵ^2\mathrm{𝟓}_{}^{}{}_{1}{}^{}+ϵ\mathrm{𝟓}_{}^{}{}_{2}{}^{}+\mathrm{𝟓}_{}^{}{}_{3}{}^{}\mathrm{𝟓}_{}^{}{}_{3}{}^{}.`$ (28)
In the following analysis we will take $`\mathrm{𝟓}_{}^{}{}_{H}{}^{}=\mathrm{𝟓}_{}^{}{}_{3}{}^{}`$ for simplicity. Since the mass of $`\mathrm{𝟓}_{}^{}{}_{H}{}^{}`$ is estimated as $`g^\eta {}_{3}{}^{}v_{\mathrm{\Sigma }}^{}v_\mathrm{\Sigma }M_{GUT}`$, three families of quarks and leptons at low energies are given by

$`\begin{array}{ccc}\mathrm{𝟏𝟎}_1(2),& \mathrm{𝟏𝟎}_2(1),& \mathrm{𝟏𝟎}_3(0),\\ \mathrm{𝟓}_{}^{}{}_{1}{}^{}(2),& \mathrm{𝟓}_{}^{}{}_{2}{}^{}(1),& \mathrm{𝟓}_{}^{}{}_{4}{}^{}(0),\\ \mathrm{𝟏}_1(2),& \mathrm{𝟏}_2(1),& \mathrm{𝟏}_3(0).\end{array}`$ (32)
Notice that there appears no other massless matter field. Here we have also shown the FN charge of each multiplet. Even after the exchange of $`\mathrm{𝟓}^{}`$-plets the FN charge assignment does not have the lopsided structure. As we will explain below, the Higgs field $`\mathrm{\Sigma }^c`$ plays an important role to generate the lopsided mass matrices.

Up quarks obtain Dirac masses from the usual Yukawa term $`W=\frac{1}{2}h^u{}_{ij}{}^{}H_{2}^{}\psi _i\psi _j`$ in the superpotential (26) and its mass matrix takes the form

$`m^u=v_2h^u=v_2\left(\begin{array}{ccc}ϵ^4& ϵ^3& ϵ^2\\ ϵ^3& ϵ^2& ϵ\\ ϵ^2& ϵ& 1\end{array}\right).`$ (36)
This is the same as in the SU(5) model (cf. Eq. (11)) and gives the approximate mass relations shown in Eq. (1). On the other hand, down quarks and charged leptons receive masses from the following terms in the superpotential (26)

$`W={\displaystyle \frac{1}{2}}h^d{}_{ij}{}^{}H_{1}^{}\psi _i\psi _j+g^d{}_{i}{}^{}{\displaystyle \frac{\mathrm{\Sigma }^c}{M_{}}}H_1\psi _i\eta =h^d{}_{ij}{}^{}H_1\mathrm{𝟏𝟎}_i\mathrm{𝟓}_{}^{}{}_{j}{}^{}+g^d{}_{i}{}^{}ϵ_{\mathrm{\Sigma }}^{}H_1\mathrm{𝟏𝟎}_i\mathrm{𝟓}_{}^{}{}_{4}{}^{}+\mathrm{},`$ (37)
where $`ϵ_\mathrm{\Sigma }v_\mathrm{\Sigma }/M_{}`$. Since $`\mathrm{𝟓}_{}^{}{}_{H}{}^{}(=\mathrm{𝟓}_{}^{}{}_{3}{}^{})`$ decouples from the low energy physics, we have the effective mass terms for down-quarks and charged leptons as

$`W=H_1(\mathrm{𝟏𝟎}_1,\mathrm{𝟏𝟎}_2,\mathrm{𝟏𝟎}_3)\left(\begin{array}{ccc}ϵ^4& ϵ^2ϵ_\mathrm{\Sigma }& ϵ^3\\ ϵ^3& ϵϵ_\mathrm{\Sigma }& ϵ^2\\ ϵ^2& ϵ_\mathrm{\Sigma }& ϵ\end{array}\right)\left(\begin{array}{c}\mathrm{𝟓}_{}^{}{}_{1}{}^{}\\ \mathrm{𝟓}_{}^{}{}_{4}{}^{}\\ \mathrm{𝟓}_{}^{}{}_{2}{}^{}\end{array}\right).`$ (44)
Mass matrices for down quarks and charged leptons are

$`m^e=(m^d)^T=v_1\left(\begin{array}{ccc}ϵ^4& ϵ^2ϵ_\mathrm{\Sigma }& ϵ^3\\ ϵ^3& ϵϵ_\mathrm{\Sigma }& ϵ^2\\ ϵ^2& ϵ_\mathrm{\Sigma }& ϵ\end{array}\right),`$ (48)
which yields the following mass relations

$`m_d:m_s:m_b=m_e:m_\mu :m_\tau ϵ^3:ϵ_\mathrm{\Sigma }:1.`$ (49)
It should be noted that, when $`ϵ_\mathrm{\Sigma }`$ is equal to $`ϵ`$, the mass matrices in Eq. (48) coincide with those in the SU(5) lopsided model with $`a=1`$ (see Eq. (11)). In the considering SO(10) model, the $`\mathrm{\Sigma }^c`$ field plays partially the same role as the FN field $`S_{FN}`$ only for $`\mathrm{𝟓}_{}^{}{}_{4}{}^{}`$, and the successful mass matrices for down quarks and leptons are obtained when we arrange the vev of $`\mathrm{\Sigma }^c`$ such that $`ϵ_\mathrm{\Sigma }ϵ`$. In this analysis we consider that $`\mathrm{𝟓}_{}^{}{}_{4}{}^{}`$ belongs to the second family as in Refs. . It can, however, belong to the third family as long as $`ϵ_\mathrm{\Sigma }ϵ`$. The ratio of vevs between Higgs doublets is estimated as $`\mathrm{tan}\beta =v_2/v_1ϵm_t/m_b8`$ at the unification scale.

We turn to discuss neutrino masses. Dirac mass terms are generated in the similar way to down quarks and charged leptons.

$`W`$ $`=`$ $`{\displaystyle \frac{1}{2}}h^u{}_{ij}{}^{}H_{2}^{}\psi _i\psi _j+g^u{}_{i}{}^{}{\displaystyle \frac{\mathrm{\Sigma }^c}{M_{}}}H_2\psi _i\eta `$ (56)
$`=`$ $`H_2(\mathrm{𝟏}_1,\mathrm{𝟏}_2,\mathrm{𝟏}_3)\left(\begin{array}{ccc}ϵ^4& ϵ^2ϵ_\mathrm{\Sigma }& ϵ^3\\ ϵ^3& ϵϵ_\mathrm{\Sigma }& ϵ^2\\ ϵ^2& ϵ_\mathrm{\Sigma }& ϵ\end{array}\right)\left(\begin{array}{c}\mathrm{𝟓}_{}^{}{}_{1}{}^{}\\ \mathrm{𝟓}_{}^{}{}_{4}{}^{}\\ \mathrm{𝟓}_{}^{}{}_{2}{}^{}\end{array}\right)+\mathrm{},`$
and hence we have

$`m^D=v_2\left(\begin{array}{ccc}ϵ^4& ϵ^2ϵ_\mathrm{\Sigma }& ϵ^3\\ ϵ^3& ϵϵ_\mathrm{\Sigma }& ϵ^2\\ ϵ^2& ϵ_\mathrm{\Sigma }& ϵ\end{array}\right).`$ (60)
It is seen that $`m^D`$ has the same lopsided structure as $`m^e`$ and $`(m^d)^T`$. Majorana masses for right-handed neutrinos are induced by the following terms, $`W=h^n{}_{ij}{}^{}\frac{\mathrm{\Phi }^c\mathrm{\Phi }^c}{M_{}}\psi _i\psi _j`$, which leads to

$`m^n={\displaystyle \frac{v_\mathrm{\Phi }^2}{M_{}}}h^n={\displaystyle \frac{v_\mathrm{\Phi }^2}{M_{}}}\left(\begin{array}{ccc}ϵ^4& ϵ^3& ϵ^2\\ ϵ^3& ϵ^2& ϵ\\ ϵ^2& ϵ& 1\end{array}\right).`$ (64)
Through the seesaw mechanism we obtain the following Majorana mass matrix for the left-handed neutrinos

$`m^\nu ={\displaystyle \frac{v_2^2M_{}}{v_\mathrm{\Phi }^2}}\left(\begin{array}{ccc}ϵ^4& ϵ^2ϵ_\mathrm{\Sigma }& ϵ^3\\ ϵ^2ϵ_\mathrm{\Sigma }& ϵ_\mathrm{\Sigma }^2& ϵϵ_\mathrm{\Sigma }\\ ϵ^3& ϵϵ_\mathrm{\Sigma }& ϵ^2\end{array}\right).`$ (68)
Just as before, when we set $`ϵ_\mathrm{\Sigma }ϵ`$, we have the same structure as Eq. (22) in the lopsided SU(5) model. In this case, we can have naturally a large mixing angle for the $`\nu _\mu `$-$`\nu _\tau `$ atmospheric neutrino oscillation, and we have no difficulty explaining a large mixing angle in solar neutrino oscillation.

We denote eigenvalues of the mass matrix (68) by $`m_i`$ ($`m_1<m_2<m_3`$). Identifying $`m_3`$ with $`\sqrt{\delta m_{\mathrm{atm}}^2}5\times 10^2\text{eV}`$ and requiring $`ϵ_\mathrm{\Sigma }ϵ`$ the vev $`v_\mathrm{\Phi }`$ is estimated as

$`v_\mathrm{\Phi }ϵ{\displaystyle \frac{v_2M_{}^{1/2}}{(\delta m_{\mathrm{atm}}^2)^{1/4}}}2\times 10^{15}\text{GeV},`$ (69)
where we have taken the cutoff scale $`M_{}`$ as the gravitational scale $`M_P2.4\times 10^{18}`$ GeV. It is seen that the scale of $`v_\mathrm{\Phi }`$ is one order below the unification scale.444If the Higgs field $`\mathrm{\Phi }^c`$ carries the FN charge $`+1`$, the vev $`v_\mathrm{\Phi }`$ is estimated as $`v_\mathrm{\Phi }4\times 10^{16}`$ GeV and is comparable to the unification scale. Further, we find from Eq. (64) Majorana masses $`M_i`$ ($`M_1<M_2<M_3`$) for right-handed neutrinos $`n_i`$ as

$`M_14\times 10^7\text{GeV},M_29\times 10^9\text{GeV}\text{and}M_32\times 10^{12}\text{GeV}.`$ (70)
Notice that, in the considering SO(10) model, the FN charges for right-handed neutrinos are fixed by those for up quarks, and hence we can predict Majorana masses uniquely.555 Even if we consider the non-zero FN charge for the Higgs field $`\mathrm{\Phi }^c`$ (see footnote 4), the prediction of Majorana masses $`M_i`$ does not change. This is completely different from the SU(5) model where they are determined by the unknown FN charges $`b`$, $`c`$ and $`d`$ (see Table 1). This point is crucial for considering possible scenarios of leptogenesis later.

We have shown that the successful mass matrices of quarks and leptons in the SU(5) model can be obtained in the framework of SO(10) GUT when $`ϵ_\mathrm{\Sigma }ϵ1/16`$. This means the vev of the Higgs field $`\mathrm{\Sigma }^c`$ should be comparable to the vev of the FN singlet field $`S_{FN}`$ as

$`v_\mathrm{\Sigma }S_{FN}ϵM_{}2\times 10^{17}\text{GeV},`$ (71)
where we have taken $`M_{}=M_P`$. It is found that there should be a new physics scale one order above the unification scale. In fact, the SO(10) GUT is realized only above this scale and we have SU(5) as an unbroken group for the scale $`v_\mathrm{\Sigma }\mu M_{GUT}`$. If the origin of the vev $`S_{FN}`$ is associated with the SO(10) breaking, this GUT breaking pattern might answer the required condition $`ϵ_\mathrm{\Sigma }ϵ`$, although we have to tune the scale correctly.666 One explanation for $`ϵ_\mathrm{\Sigma }ϵ1/16`$ might be obtained by embedding the considering SO(10) GUT model in the higher dimensional theory. We expect the cutoff scale $`M_{}`$ is given by the $`(4+d)`$-dimensional Planck scale, and hence $`M_{}=(M_P^2M_C^d)^{1/(d+2)}`$ where $`M_C`$ denotes the compactification scale of the extra $`d`$-dimensional space. We consider $`M_C`$ is comparable to the unification scale. When the SO(10) GUT is embedded in 6-dimensions ($`d=2`$), we find $`M_{}2\times 10^{17}`$ GeV, and then $`v_\mathrm{\Sigma }S_{FN}1\times 10^{16}`$ GeV. Moreover, when the FN charge for the Higgs field $`\mathrm{\Phi }^c`$ is $`+1`$ (see the footnote 4), $`v_\mathrm{\Phi }`$ is also estimated to be $`1\times 10^{16}\text{GeV}`$. This suggests that the SO(10) GUT in 6-dimensions might realize the successful lopsided mass matrices by the vevs $`S_{FN}`$, $`v_\mathrm{\Sigma }`$ and $`v_\mathrm{\Phi }`$ which are all of order of the unification scale (or $`M_C`$). It is very interesting to note that the attractive SO(10) GUT models have recently been constructed in 6-dimensions . (See also the footnote 3.)

Finally, we would like to discuss cosmological implication of the model, in particular, implication in “leptogenesis” . Non-equilibrium decays of right-handed neutrinos $`n_i`$ give an attractive mechanism to generate dynamically the observed baryon asymmetry in the present Universe. This is because these decays can generate a lepton number in the early Universe, which is partially converted into a baryon number through the electroweak sphaleron processes . The CP asymmetry by the $`n_i`$ decay can be expressed by the parameter $`ϵ_i`$ and is estimated as

$`ϵ_i`$ $``$ $`{\displaystyle \frac{\mathrm{\Gamma }(n_i\mathrm{}+H_u)\mathrm{\Gamma }(n_i\mathrm{}^{}+H_u^{})}{\mathrm{\Gamma }(n_i\mathrm{}+H_u)+\mathrm{\Gamma }(n_i\mathrm{}^{}+H_u^{})}}`$ (72)
$`=`$ $`{\displaystyle \frac{1}{8\pi v_2^2}}{\displaystyle \frac{1}{(m^Dm^D{}_{}{}^{})_{ii}}}{\displaystyle \underset{ji}{}}\text{Im}\left[\left\{(m^Dm^D{}_{}{}^{})_{ij}\right\}^2\right]f\left({\displaystyle \frac{M_j^2}{M_i^2}}\right),`$
where $`n_i`$, $`\mathrm{}`$ and $`H_u`$ denote here scalar or fermionic components of corresponding superfields ($`\mathrm{}`$ are lepton doublets at low energies) and

$`f(x)=\sqrt{x}\mathrm{ln}\left(1+{\displaystyle \frac{1}{x}}\right)+{\displaystyle \frac{2\sqrt{x}}{x1}}.`$ (73)
The SO(10) model described above can predict Majorana masses for right-handed neutrinos as shown in Eq. (70) and their mass ratios are determined by those of up quarks, which is completely different from the SU(5) model. This suggests possible scenarios of leptogenesis are restricted. The CP asymmetry $`ϵ_1`$ of the lightest right-handed neutrino $`n_1`$ is estimated from Eq. (72) as $`\left|ϵ_1\right|1\times 10^8`$, where we have assumed the CP-violating phase of order unity. It have been shown in Ref.  that the conventional thermal leptogenesis  does not work since $`ϵ_1`$ is too small to account for the present baryon asymmetry.

We, then, consider non-thermal leptogenesis via inflaton decay , where right-handed neutrinos are produced non-thermally in decays of inflaton $`\phi `$. The baryon asymmetry (the ratio of baryon number density $`n_B`$ to the entropy density $`s`$) induced by $`n_1`$ is given by777In this analysis, we neglect the sign of the produced baryon asymmetry.

$`{\displaystyle \frac{n_B}{s}}0.5Br_1\left|ϵ_1\right|{\displaystyle \frac{T_R}{M_\phi }},`$ (74)
where $`M_\phi `$, $`T_R`$ and $`Br_1`$ denote the inflaton mass, the reheating temperature, and the branching ratio of $`\phi n_1+n_1`$, respectively. In order to ensure the non-thermal production of $`n_1`$ we assume $`M_\phi >2M_1`$ and $`T_{R}^{}{}_{}{}^{<}M_1`$. In the above model we estimate as

$`{\displaystyle \frac{n_B}{s}}6\times 10^{10}Br_1\left({\displaystyle \frac{T_R}{10^7\text{GeV}}}\right)\left({\displaystyle \frac{2M_1}{M_\phi }}\right),`$ (75)
which should be compared with the observation $`(n_B/s)_{\mathrm{OBS}}(0.11)\times 10^{10}`$. It is found that the successful leptogenesis is available only for the inflation models which give

$`M_\phi 10^9\text{GeV},T_R10^7\text{GeV}\text{and}Br_11.`$ (76)
Further, as recently proposed in Ref. , decays of heavier right-handed neutrinos $`n_2`$ and $`n_3`$ can be a dominant source of the present baryon asymmetry. It is usually considered that decays of the lightest right-handed neutrino are responsible to $`(n_B/s)_{\mathrm{OBS}}`$, although decays of heavier ones also induce lepton asymmetry. This is because after the decays of $`n_2`$ and $`n_3`$ the lightest right-handed neutrino can remain in thermal equilibrium and wash out the lepton asymmetry from $`n_2`$ and $`n_3`$. This is true for the conventional thermal leptogenesis. However, in the non-thermal leptogenesis scenarios, as shown in Ref. , such dangerous wash-out effects can be killed just requiring that the reheating temperature is $`T_{R}^{}{}_{}{}^{<}M_1`$. We shall illustrate this idea in the described model, especially, leptogenesis by decays of the next-to-lightest right-handed neutrino $`n_2`$. The successful scenario requires that $`M_\phi >2M_2`$ and also $`T_{R}^{}{}_{}{}^{<}M_1`$, which means that both $`n_1`$ and $`n_2`$ are produced non-thermally in inflaton decays and induce the lepton asymmetry. We find the CP asymmetry for $`n_2`$ is $`\left|ϵ_2\right|2\times 10^6`$ and

$`{\displaystyle \frac{n_B}{s}}\left[3\times 10^{12}Br_1+6\times 10^{10}Br_2\right]\left({\displaystyle \frac{T_R}{10^7\text{GeV}}}\right)\left({\displaystyle \frac{2M_2}{M_\phi }}\right).`$ (77)
This equation shows that the dominant contribution to the baryon asymmetry comes from the decays of $`n_2`$ rather than the lightest one $`n_1`$ unless $`Br_2Br_1`$. In this case, the successful leptogenesis requires

$`M_\phi 10^{11}\text{GeV},T_R10^7\text{GeV}\text{and}Br_21.`$ (78)
We have seen that possible scenarios of leptogenesis are restricted in the SO(10) model even for the non-thermal leptogenesis via inflaton decays, and the present baryon asymmetry can suggest parameters of inflation models as shown in Eqs. (76) and (78). It is quite interesting to observe that the SO(10) model described above provides naturally the hybrid inflation and, moreover, the values given in Eq. (78) are just predicted by this inflation.888See Refs.  for the similar discussion in the different models. Let us write the superpotential which gives non-zero vev for the Higgs fields $`\mathrm{\Phi }`$ and $`\mathrm{\Phi }^c`$ as follows;

$`W=\lambda X(\mathrm{\Phi }\mathrm{\Phi }^cv_\mathrm{\Phi }^2),`$ (79)
where $`\lambda `$ is the coupling constant. This is nothing but the superpotential for the supersymmetric hybrid inflation .999 Supergravity effects are potentially dangerous since they disturb the slow-roll motion of inflation. These effects are induced from the nonrenormalizable interaction in the Kähler potential $`K=(k/4)\left|X\right|^4/M_{}^2`$. For the successful inflation the coupling $`k`$ should be negative and also $`\left|k\right|_{}^<10^3`$ when $`\lambda 10^4`$. We assume it by hand since smallness of couplings in the Käher potential cannot be explained by the FN mechanism.  The value of $`v_\mathrm{\Phi }2\times 10^{15}`$ GeV (see Eq. (69)), which is suggested from the atmospheric neutrino oscillation, gives the coupling constant of $`\lambda 10^4`$ in order to explain the COBE normalization of the cosmic density fluctuations.101010The coupling constant $`\lambda `$ of $`10^4`$ can be explained by the FN mechanism with the FN charge $`+3`$ for $`X`$.  (See the detailed analysis of the hybrid inflation in Ref. .) The inflaton mass $`M_\phi `$ is, then, estimated as $`M_\phi =\sqrt{2}\lambda v_\mathrm{\Phi }10^{11}`$ GeV, which is just the required value in Eq. (78) for the successful leptogenesis. The reheating of inflation takes place via decays through the interactions $`W=\frac{1}{2}h_{ij}^n\frac{\mathrm{\Phi }^c\mathrm{\Phi }^c}{M_{}}\psi _i\psi _j`$ in Eq. (26). Therefore, the inflaton decays mainly into pairs of right-handed (s)neutrinos and their partial widths are proportional to $`M_i^2`$.111111Although the inflaton decays also through interactions in the Kähler potential, they give negligible corrections to the total width.  We find from Eq. (70) that $`Br_21`$ is ensured for the inflaton with $`M_\phi 10^{11}`$ GeV. Further, the reheating temperature is estimated as $`T_R10^7`$ GeV. With such low reheating temperatures, we can avoid the cosmological gravitino problem .121212 We can neglect gravitinos produced non-thermally at the preheating epoch  since the coupling $`\lambda `$ is sufficiently small. Similarly, production of right-handed neutrinos by the preheating  can also be neglected.  Therefore, the hybrid inflation provided by the model itself offers the successful non-thermal leptogenesis by decays of the next-to-lightest right-handed neutrinos. The observed baryon asymmetry in the present Universe is explained by just the neutrino mass scale suggested by the atmospheric neutrino experiments.

The author would like to thank W. Buchmüller for useful discussions.