text:
# Unitary Representations and Theta Correspondence for Type I Classical Groups

## 1 Introduction

Let $`(G_1,G_2)`$ be an irreducible reductive dual pair of type I in $`Sp`$ (see  ). The dual pairs in this paper will be considered as ordered. For example, the pair $`(O(p,q),Sp_{2n}())`$ is considered different from the pair $`(Sp_{2n}(),O(p,q))`$. We will in general assume that the size of $`G_1(V_1)`$ is less or equal to the size of $`G_2(V_2)`$. In other words, $`dim_D(V_1)dim_D(V_2)`$. Let $`Mp`$ be the unique double covering of $`Sp`$. Let $`\{1,ϵ\}`$ be the preimage of the identity element in $`Sp`$. For a subgroup $`H`$ of $`Sp`$, let $`MH`$ be the preimage of $`H`$ under the double covering. Whenever we use the notation $`MH`$, $`H`$ is considered as a subgroup of certain $`Sp`$. Let $`\omega (MG_1,MG_2)`$ be a Schrödinger model of the oscillator representation of $`Mp`$. The Harish-Chandra module of $`\omega (MG_1,MG_2)`$ consists of polynomials multiplied by the Gaussian function.
Since the pair $`(G_1,G_2)`$ is ordered, we use $`\theta (MG_1,MG_2)`$ to denote the theta correspondence from $`(MG_1,\omega (MG_1,MG_2))`$ to $`(MG_2,\omega (MG_1,MG_2)`$ (see ). In this paper, whenever we talk about ”$`K`$-finite matrix coefficients” or ”$`K`$-finite vectors” of a representation $`\pi `$ of a real reductive group $`G`$, ”K” is used as a generic term for a specified maximal compact subgroup of $`G`$. Throughout this paper, we will mainly work within the category of Harish-Chandra modules. A representation of a real reductive group refers to an admissible representation unless stated otherwise. Throughout this paper, a vector in an admissible representation $`\pi `$ means that $`v`$ is in the Harish-Chandra module of $`\pi `$ which shall be evident within the context.
Let $`V`$ be a vector space of finite dimension. Let $`W`$ be a subspace of $`V`$. A direct complement of $`W`$ in $`V`$ is a subspace $`U`$ such that

$$UW=V.$$

Now suppose $`V`$ is equipped with a nondegenerate sesquilinear form $`(,)`$. The orthogonal complement of $`W`$ in $`V`$ consists of

$$\{vV(v,w)=0wW\}.$$

It is denoted by $`W^{}`$.
Let $`\pi `$ be an irreducible admissible representation of $`MG_1`$ such that $`\pi (ϵ)=1`$. $`\pi `$ is said to be in the semistable range of $`\theta (MG_1,MG_2)`$ if the function

$$(\omega (MG_1,MG_2)(\stackrel{~}{g}_1)\varphi ,\psi )(u,\pi (\stackrel{~}{g}_1)v)(\varphi ,\psi \omega (MG_1,MG_2);u,v\pi )$$

is in $`L^{1\delta }(MG_1)`$ for all sufficiently small nonnegative $`\delta `$ (i.e., $`\delta [0,c]`$ for some $`c>0`$). We denote the semistable range by $`_s(MG_1,\omega (MG_1,MG_2))`$. Suppose from now on that $`\pi `$ is in $`_s(MG_1,\omega (MG_1,MG_2))`$. For each $`\varphi ,\psi \omega (MG_1,MG_2)`$ and $`u,v\pi `$, we define an averaging integral

$$_{MG_1}(\omega (MG_1,MG_2)(\stackrel{~}{g}_1)\varphi ,\psi )(u,\pi (\stackrel{~}{g}_1)v)𝑑\stackrel{~}{g}_1$$

and denote it by $`(\varphi v,\psi u)_\pi `$. Thus $`(,)_\pi `$ becomes a real bilinear form on $`\omega (MG_1,MG_2)\pi `$. Our definition of $`(,)_\pi `$ differs slightly from the original definition of Li in . Let $`\stackrel{~}{g}_2MG_2`$ act on $`\omega (MG_1,MG_2)\pi `$ by $`\omega (MG_1,MG_2)(\stackrel{~}{g}_2)Id`$. In , we show that if $`(,)_\pi 0`$ then $`(,)_\pi `$ descends into a sesquilinear form on the $`K`$-finite dual representation of $`\theta (MG_1,MG_2)(\pi )`$. For $`\pi `$ unitary, $`(,)_\pi `$ is an invariant Hermitian form on $`\theta (MG_1,MG_2)(\pi )`$.
For $`\pi `$ unitary, a conjecture of Li says that $`(,)_\pi `$ will always be positive semidefinite (see ). If Li’s conjecture holds and $`(,)_\pi 0`$, then $`\theta (MG_1,MG_2)(\pi )`$ is unitary. In this paper, we will prove that $`(,)_\pi `$ is positive semidefinite under certain restrictions. This partly confirms the conjecture of Li. The nonvanishing of certain $`(,)_\pi `$ is proved in  and in .
We adopt the notations from , , . Let $`(G_1(V_1),G_2(V_2))`$ be a dual pair of type I. Suppose $`V_2=V_2^0V_2^{}`$ such that

1. $`(,)_2`$ restricted onto $`V_2^0`$ is nondegenerate;
2. $`V_2^{}=(V_2^0)^{}`$;
3. $`V_2^0`$ is a direct sum of two isotropic subspaces:

$$V_2^0=X_2^0Y_2^0.$$

Obviously, $`V_2^0`$ will always be of even dimension. Let $`X^0=Hom_D(V_1,X_2^0)`$. The oscillator representation $`\omega (MG_1(V_1),MG_2(V_2^0))`$ can be modeled on $`L^2(X^0)`$. The action of $`MG_1`$ on $`L^2(X^0)`$ is equivalent to the regular action of $`G_1`$ on $`L^2(X^0)`$ tensoring with a unitary character $`\xi `$ of $`MG_1`$. The generic actions of $`G_1`$ on $`X^0`$ are classified abstractly in Theorem 4.1 and Theorem 4.2.
Later in this paper, the oscillator representation $`\omega (MG_1(V_1),MG_2(V_2^0))`$ is denoted as $`\omega (M^0G_1,M^0G_2^0)`$ to indicate the fact that $`MG_1(V_1)`$ in $`(MG_1(V_1),MG(V_2))`$ might be different from $`MG_1(V_1)`$ in $`(MG_1(V_1),MG_2(V_2^0))`$. For the same reason, the oscillator representation $`\omega (MG_1(V_1),MG_2(V_2^{}))`$ is denoted by $`\omega (M^{}G_1,M^{}G_2^{})`$.

###### Theorem 1.1 (Main Theorem)

Let $`(G_1,G_2)`$ be a dual pair. Let $`\mathrm{\Xi }(g)`$ be Harish-Chandra’s basic spherical function of $`G_1`$. Suppose $`\pi `$ is an irreducible unitary representation of $`MG_1`$ such that $`\pi (ϵ)=1`$. Suppose

1. for any $`x,yG_1`$, the function $`\mathrm{\Xi }(xgy)`$ is integrable on $`G_{1}^{}{}_{\varphi }{}^{}`$ for every generic $`\varphi Hom_D(V_1,X_2^0)`$ (see Definition 4.1);
2. the tensor product $`\pi _0=\omega (M^{}G_1,M^{}G_2^{})\pi \overline{\xi }`$, considered as a representation of $`G_1`$, is weakly contained in $`L^2(G_1)`$ (see ).

Then $`(,)_\pi `$ is positive semidefinite. If $`(,)_\pi `$ does not vanish, then $`\theta (MG_1,MG_2)(\pi )`$ is unitary.

Remarks:

1. $`\omega (M^{}G_1,M^{}G_2^{}))`$, $`\pi `$ and $`\overline{\xi }`$ are all projective representations of $`G_1`$. The fact that $`\pi _0`$ becomes a unitary representation of $`G_1`$ is explained in Part II.
2. The first condition roughly requires that

$$dim_D(X_2^0)>\frac{dim_D(V_1)}{2}.$$

The precise statement depends on the groups involved. The function $`\mathrm{\Xi }(g)|_{G_{1}^{}{}_{\varphi }{}^{}}`$ is in $`L^1(G_{1}^{}{}_{\varphi }{}^{})`$ implies that $`\mathrm{\Xi }(xgy)|_{G_{1}^{}{}_{\varphi }{}^{}}`$ is in $`L^1(G_{1}^{}{}_{\varphi }{}^{})`$ for any $`x,yG_1`$ and vice versa. In fact, $`\mathrm{\Xi }(g)`$ is bounded by a multiple of $`\mathrm{\Xi }(xgy)`$ and vice versa. Furthermore, for any compact subset $`Y`$ of $`G_1`$, there exists a constant $`C`$, such that for any $`x,yY`$,

$$\mathrm{\Xi }(xgy)C\mathrm{\Xi }(g)(gG_1).$$
(1)
One can prove this by studying the compact picture of the basic spherical principle series representation (see Chapter VII.1 ). Since this remark may have already been in the literature and a proof will incur a new set of notations, we choose not to give the proof.
3. The growth of matrix coefficients of $`\omega (MG_1(V_1),MG_2(V_2^{}))`$ can be determined easily. Thus the second condition can be converted into a growth condition on the matrix coefficients of $`\pi `$ (see Corollary 5.1).
4. The condition 1 and 2 imply that $`\pi `$ is in $`_s(MG_1,\omega (MG_1,MG_2))`$. Therefore, $`(,)_\pi `$ is an invariant Hermitian form on $`\theta (MG_1,MG_2)(\pi )`$. The unitarity of $`\theta (MG_1,MG_2)(\pi )`$ follows since $`(,)_\pi `$ is positive semidefinite.

This paper is organized as follows. In Part I, we prove some positivity theorems in the sense of Godement . In Part II, we construct the dual pair $`(G_1,G_2)`$ in terms of homomorphisms and study various subgroups and liftings concerning the tensor decomposition

$$\omega (MG_1,MG_2)\omega (M^0G_1,M^0G_2^0)\omega (M^{}G_1,M^{}G_2^{}).$$

This tensor decomposition is termed as the mixed model in . The interpretation of this tensor product is not completely trivial since $`MG_1`$, $`M^0G_1`$ and $`M^{}G_1`$ may be different double coverings of $`G_1`$. In Part II, we essentially redo part of section 4 in  just to be safe. In Part III, we study $`(\omega (M^0G_1,M^0G_2^0),L^2(X^0))`$ and classify all the generic $`G_1`$-orbits in $`X^0`$. This enables us to reduce our averaging integral $`(\varphi u,\varphi u)_\pi `$ to an integral on $`G_1`$-orbits:

$$_{𝒪G_1\backslash X^0}_{G_1}_{x𝒪}\varphi (g^1x)\overline{\varphi (x)}(u,\pi _0(g)u)𝑑x𝑑gd[𝒪].$$

We study each generic orbit integral

$$_{G_1}_{x𝒪}\varphi (g^1x)\overline{\varphi (x)}(u,\pi _0(g)u)𝑑x𝑑g$$

in full generality and convert it into an integral on the isotropic group $`G_{1}^{}{}_{x}{}^{}`$

$$_{G_{1}^{}{}_{x}{}^{}}(\pi _0(g)u_0,u_0)𝑑g.$$

Next, we apply the positivity theorem (Theorem 2.3) to show that this integral is nonnegative. Thus $`(,)_\pi `$ is positive semidefinite. Finally, we take the pair $`(O(p,q),Sp_{2n}())`$ as an example and state our main theorem in terms of leading exponents of $`\pi `$.
The author wishes to thank Professors Sigurdur Helgason, Roger Howe, Jian-Shu Li, Tomasz Przebinda, Irving Segal, David Vogan Jr. and Nolan Wallach for encouragements, suggestions and helpful discussions. He also likes to thank Marie Hutchinson for helping him read through the paper of Godement ( ).

## 2 Part I: Positivity Theorems

Let $`G`$ be a real reductive Lie group. Let $`K`$ be a maximal compact subgroup of $`G`$. For any unitary representation $`(\pi ,H)`$ of $`G`$ and any $`\sigma \widehat{K}`$, let $`H_\sigma `$ be the $`K`$-isotypic subspace of $`H`$. Let $`S`$ be a subset of $`\widehat{K}`$. We denote

$$_{\sigma S}H_\sigma $$

by $`H(S)`$.

### 2.1 A generic Theorem

###### Theorem 2.1

Let $`G`$ be a real reductive Lie group. Let $`K`$ be a maximal compact subgroup of $`G`$. Let $`\mathrm{\Xi }(g)`$ be Harish-Chandra’s basic spherical function with respect to $`K`$. Let $`H`$ be a closed unimodular Lie subgroup of $`G`$. Suppose that $`\mathrm{\Xi }(g)|_H`$ is in $`L^1(H)`$. Let $`\varphi `$ be a positive definite function in $`L^{2+ϵ}(G)(S)`$ for some finite subset $`S`$ of $`\widehat{K}`$ and any $`ϵ>0`$. Then $`_H\varphi (h)𝑑h0`$.

Here $`L^{2+ϵ}(G)(S)`$ is defined with respect to the left regular action of $`G`$.
Proof: By the GNS construction, we construct a unitary representation $`(\sigma ,)`$ such that $`\varphi (g)=(\sigma (g)\eta ,\eta )`$ for some cyclic vector $`\eta `$ in $`(S)`$. Since $`\varphi `$ is a positive definite function in $`L^{2+ϵ}(G)`$ for any $`ϵ>0`$, by Theorem 1 in , $`\sigma `$ is weakly contained in $`L^2(G)`$. Thus, there exists a sequence of convex linear combinations of diagonal matrix coefficients of $`L^2(G)(S)`$

$$A_i(g)=\underset{l=1}{\overset{l_i}{}}a_i^{(l)}(L(g)u_i^{(l)},u_i^{(l)}),\underset{l=1}{\overset{l_i}{}}a_i^{(l)}=1(u_i^{(l)}L^2(G)(S),a_i^{(l)}0)$$

such that

$$A_i(g)\varphi (g)$$

uniformly on compacta. Let $`C_c(G)(S)`$ be the space of continuous and compactly supported functions in $`L^2(G)(S)`$. Since $`C_c(G)(S)`$ is dense in $`L^2(G)(S)`$, we choose $`u_i^{(l)}`$ to be in $`C_c(G)(S)`$. Notice that

$$A_i(e)=\underset{l=1}{\overset{l_i}{}}a_i^{(l)}u_i^{(l)}_{L^2}^2\varphi (e)=\eta ^2$$

Hence $`\{A_i(e)\}_{i=1}^{\mathrm{}}`$ is a bounded set. Suppose $`A_i(e)C`$. From Theorem 2 in ,

$$|(L(g)u_i^{(l)},u_i^{(l)})|u_i^{(l)}_{L^2}^2(\underset{\sigma S}{}d(\sigma ))^{\frac{1}{2}}\mathrm{\Xi }(g).$$

It follows that

$$\begin{array}{cc}\hfill |A_i(g)|=& |\underset{l=1}{\overset{l_i}{}}a_i^{(l)}(L(g)u_i^{(l)},u_i^{(l)})|\hfill \\ \hfill & \underset{l=1}{\overset{l_i}{}}a_i^{(l)}u_i^{(l)}_{L^2}^2(\underset{\sigma S}{}d(\sigma ))^{\frac{1}{2}}\mathrm{\Xi }(g)\hfill \\ \hfill & C(\underset{\sigma S}{}d(\sigma ))^{\frac{1}{2}}\mathrm{\Xi }(g).\hfill \end{array}$$
(2)
We have proved that $`\varphi (g)`$ can be approximated by positive definite functions $`A_i(g)`$ such that $`A_i(g)`$ are uniformly bounded by a fixed multiple of $`\mathrm{\Xi }(g)`$.
Now consider the restrictions of $`\varphi (g)`$ to $`H`$. From (22.2.3) in , for $`(L(g)u_i^{(l)},u_i^{(l)})`$ with $`u_i^{(l)}`$ a compactly supported continuous function,

$$_H(L(h)u_i^{(l)},u_i^{(l)})𝑑h0.$$

Thus $`_HA_i(h)𝑑h0`$. But $`A_i(g)|_H`$ are bounded by a fixed multiple of an integrable function $`\mathrm{\Xi }(g)|_H`$. By the Dominated Convergence Theorem,

$$_H\varphi (h)𝑑h=\underset{i\mathrm{}}{lim}_HA_i(h)𝑑h0.$$

Q.E.D.

### 2.2 First Variation

###### Theorem 2.2

Let $`G`$ be a real reductive Lie group. Let $`K`$ be a maximal compact subgroup of $`G`$. Let $`H`$ be a closed unimodular Lie subgroup of $`G`$. Let $`\mathrm{\Xi }(g)`$ be the basic spherical function of $`G`$ of Harish-Chandra. Suppose that $`\mathrm{\Xi }(g)|_H`$ is in $`L^1(H)`$. Suppose $`(\pi ,)`$ is an irreducible unitary representation weakly contained in $`L^2(G)`$ (see ). Let

$$v=\underset{i=1}{\overset{k}{}}_M\varphi _i(x)\pi (\gamma _i(x))u𝑑x$$

where

* $`u`$ is a $`K`$-finite vector in $``$;
* $`M`$ is a smooth manifold;
* $`\varphi _i`$ is continuous and is supported on a compact set $`X_iM`$;
* $`\gamma _i:MG`$ is smooth except a codimension $`1`$ subset and the closure of $`\gamma _i(X_i)`$ is compact.

Then

$$_H(\pi (h)v,v)𝑑h0.$$

The basic idea is to control the function $`(\pi (g)v,v)`$ by a convergent integral of left and right translations of $`\mathrm{\Xi }(g)`$.
Proof: From the proof of Theorem 2.1, we have a sequence of $`K`$-finite compactly supported continuous positive definite functions

$$A_m(g)(\pi (g)u,u)$$

uniformly on any compact subset and

$$|A_m(g)|C\mathrm{\Xi }(g).$$

This implies that

$$|A_m(xgy)|C\mathrm{\Xi }(xgy).$$

By the compactness of $`supp(\varphi _i)`$ and the unitarity of $`\pi `$,

$$(\pi (g)v,v)=\underset{i,j=1}{\overset{k}{}}_{M\times M}\varphi _i(x)\overline{\varphi _j(y)}(\pi (g\gamma _i(x))u,\pi (\gamma _j(y))u)𝑑x𝑑y.$$

Since the closure of $`\gamma _i(X_i)`$ is compact, the closure of $`\gamma _j(X_j)^1g\gamma _i(X_i)`$ is compact for every $`gG`$. By the Ineqaulity 1, for any $`m`$,

$$\begin{array}{cc}& |\underset{i,j=1}{\overset{k}{}}_{M\times M}\varphi _i(x)\overline{\varphi _j(y)}A_m(\gamma _j(y)^1g\gamma _i(x)))dxdy|\hfill \\ \hfill & C\underset{i,j=1}{\overset{k}{}}_{M\times M}|\varphi _i(x)||\varphi _j(y)|\mathrm{\Xi }(\gamma _j(y)^1g\gamma _i(x)))dxdy\hfill \\ \hfill & C_1\mathrm{\Xi }(g)\hfill \end{array}$$
(3)
for some $`C_1>0`$. Furthermore,

$$\underset{i,j=1}{\overset{k}{}}_{M\times M}\varphi _i(x)\overline{\varphi _j(y)}A_m(\gamma _j(y)^1g\gamma _i(x))𝑑x𝑑y(\pi (g)v,v)$$

pointwisely as $`m\mathrm{}`$. By the Dominated Convergence Thereom,

$$_H(\pi (h)v,v)dh=\underset{m\mathrm{}}{lim}_H\underset{i,j=1}{\overset{k}{}}_{M\times M}\varphi _i(x)\overline{\varphi _j(y)}A_m(\gamma _j(y)^1h\gamma _i(x)))dxdydh.$$

But

$$A_m(g)=\underset{l=1}{\overset{l_m}{}}a_m^{(l)}(L(g)u_m^{(l)},u_m^{(l)}).$$

For each $`l`$,

$$\begin{array}{cc}& _H\underset{i,j=1}{\overset{k}{}}_{M\times M}\varphi _i(x)\overline{\varphi _j(y)}(L(\gamma _j(y)^1h\gamma _i(x))u_m^{(l)},u_m^{(l)})𝑑x𝑑y𝑑h\hfill \\ \hfill =& _H(L(h)[\underset{i=1}{\overset{k}{}}_M\varphi _i(x)L(\gamma _i(x))u_m^{(l)}𝑑x],[\underset{i=1}{\overset{k}{}}_M\varphi _i(x)L(\gamma _i(x))u_m^{(l)}𝑑x])𝑑h\hfill \\ \hfill & 0\hfill \end{array}$$
(4)
because $`_{i=1}^k_M\varphi _i(x)L(\gamma _i(x))u_m^{(l)}𝑑x`$ is a continuous and compactly supported function on $`G`$. Hence for every $`m`$,

$$_H\underset{i,j=1}{\overset{k}{}}_{M\times M}\varphi _i(x)\overline{\varphi _j(y)}A_m(\gamma _j(y)^1h\gamma _i(x))𝑑x𝑑y𝑑h0.$$

It follows that

$$_H(\pi (h)v,v)𝑑h0.$$

Q.E.D.

### 2.3 Second Variation

###### Theorem 2.3

Let $`G`$ be a real reductive Lie group. Let $`K`$ be a maximal compact subgroup of $`G`$. Let $`H`$ be a closed unimodular Lie subgroup of $`G`$. Let $`\mathrm{\Xi }(g)`$ be the basic spherical function of $`G`$ of Harish-Chandra. Suppose that $`\mathrm{\Xi }(g)|_H`$ is in $`L^1(H)`$. Suppose $`(\pi ,)`$ is an irreducible unitary representation weakly contained in $`L^2(G)`$ (see , ). Let

$$v=\underset{i=1}{\overset{n}{}}_M\varphi _i(x)\pi (\gamma _i(x))u_i𝑑x$$

where

* $`u_i`$ are $`K`$-finite vectors in $``$;
* $`M`$ is a smooth manifold;
* $`\varphi _i`$ is continuous and is supported on a compact subset $`X_iM`$;
* $`\gamma _i:MG`$ is smooth except a codimension $`1`$ subset and the closure of $`\gamma _i(X_i)`$ is compact.

Then

$$_H(\pi (h)v,v)𝑑h0$$

The only difference from Theorem 2.2 is

$$v=\underset{i=1}{\overset{n}{}}_M\varphi _i(x)\pi (\gamma _i(x))u_i𝑑x$$

instead of

$$v=\underset{i=1}{\overset{n}{}}_M\varphi _i(x)\pi (\gamma _i(x))u𝑑x.$$

Proof: Let $`V`$ be the linear span of

$$\{\pi (k)u_ii[1,n],kK\}.$$

Since $`u_i`$ are $`K`$-finite, $`V`$ is a finite dimensional representation of $`K`$. Let $`u`$ be a $`K`$-cyclic vector in $`V`$. Let $`C(K)`$ be the space of continuous functions on $`K`$. Consider the action of $`C(K)`$ on $`u`$:

$$\pi (f)u=_Kf(k)\pi (k)u𝑑k.$$

Apparently, $`\pi (C(K))u=V`$. Let

$$u_i=_Kf_i(k)\pi (k)u𝑑k.$$

Then

$$\begin{array}{cc}\hfill v=& \underset{i=1}{\overset{n}{}}_M\varphi _i(x)\pi (\gamma _i(x))u_i𝑑x\hfill \\ \hfill =& \underset{i=1}{\overset{n}{}}_M\varphi _i(x)\pi (\gamma _i(x))_Kf_i(k)\pi (k)u𝑑k𝑑x\hfill \\ \hfill =& \underset{i=1}{\overset{n}{}}_M_K\varphi _i(x)f_i(k)\pi (\gamma _i(x)k)u𝑑x𝑑k\hfill \end{array}$$
(5)
Apply Theorem 2.2 to functions $`\varphi _i(x)f_i(k)`$ on $`M\times K`$ and

$$\gamma _i^{}:(x,k)M\times K\gamma _i(x)kG.$$

The conclusion follows immediately. Q.E.D.

###### Conjecture 1

Let $`G`$ be a real reductive group. Let $`K`$ be a maximal compact subgroup of $`G`$. Let $`\mathrm{\Xi }(g)`$ be Harish-Chandra’s basic spherical function. Let $`H`$ be a subgroup of $`G`$ such that $`\mathrm{\Xi }(g)|_H`$ is in $`L^1(H)`$. Let $`\varphi (g)`$ be a positive definite continuous function bounded by $`\mathrm{\Xi }(g)`$. Then $`_H\varphi (h)𝑑h0`$.

## 3 Part II: Dual Pairs and Mixed Model

The basic theory on the mixed model of the oscillator representation is covered in  with reference to an unpublished note of Howe. We redo part of section 4 of  with emphasis on the actions of various coverings of $`G_1`$ regarding the mixed model

$$\omega (MG_1,MG_2)\omega (M^0G_1,M^0G_2^0)\omega (M^{}G_1,M^{}G_2^{}).$$

Let $`V_1`$ be a vector space over $`D`$ equipped with a sesquilinear form $`(,)_1`$, $`V_2`$ be a vector space over $`D`$ equipped with a sesquilinear form $`(,)_2`$. Suppose one sesquilinear form is $`\mathrm{}`$-Hermitian and the other is $`\mathrm{}`$-skew Hermitian. Let $`G_i`$ be the isometry group of $`(,)_i`$. Let $`V=Hom_D(V_1,V_2)`$ be the space of $`D`$-linear homomorphisms from $`V_1`$ to $`V_2`$.

### 3.1 Setup

Let $`\varphi ,\psi V`$, $`v_1,u_1V_1`$ and $`v_2V_2`$. We define a unique $`\varphi ^{}(v_2)`$ such that

$$(\varphi ^{}(v_2),v_1)_1=(v_2,\varphi (v_1))_2.$$

It is easy to verify that $`\varphi ^{}Hom_D(V_2,V_1)`$. Thus we obtain a $``$ operation from $`V`$ to $`V^{}=Hom_D(V_2,V_1)`$. Let $`a`$. Then

$$((a\varphi )^{}(v_2),v_1)_1=(v_2,a\varphi (v_1))_2=a(v_2,\varphi (v_1))_2=a(\varphi ^{}(v_2),v_1)_1=(a\varphi ^{}(v_2),v_1)_1.$$

Therefore, the $``$-operation is real linear.
Let $`tr()`$ be the real trace of a real linear endomorphism. Since $`V`$ and $`V^{}`$ are real vector spaces, we can now define a real bilinear form $`\mathrm{\Omega }`$ on $`V`$ as follows

$$\mathrm{\Omega }(\varphi ,\psi )=tr(\psi ^{}\varphi ).$$

We observe that

$$(\psi ^{}\varphi (v_1),v_1^{})_1=(\varphi (v_1),\psi (v_1^{}))_2=\pm (\psi (v_1^{}),\varphi (v_1))_2^{\mathrm{}}=\pm (\varphi ^{}\psi (v_1^{}),v_1)_1^{\mathrm{}}=(v_1,\varphi ^{}\psi (v_1^{}))_1.$$

Define a $``$-operation on $`End_D(V_1)`$ by

$$(A^{}u_1,v_1)_1=(u_1,A(v_1))_1(AEnd_D(V_1)).$$

Then, $`(\varphi ^{}\psi )^{}=\psi ^{}\varphi `$. It follows that

$$\mathrm{\Omega }(\psi ,\varphi )=tr(\varphi ^{}\psi )=tr((\varphi ^{}\psi )^{})=tr(\psi ^{}\varphi )=\mathrm{\Omega }(\varphi ,\psi ).$$

It is easy to verify that $`\mathrm{\Omega }`$ is nondegenerate. Therefore, $`\mathrm{\Omega }`$ is a real symplectic form on $`V`$.
Next we define the action of $`G_1`$ on $`V`$ as follows

$$(g_1\varphi )(v_1)=\varphi (g_1^1v_1).$$

We observe that

$$\begin{array}{cc}& ((g_1\psi )^{}(g_1\varphi )(u_1),v_1)_1\hfill \\ \hfill =& ((g_1\varphi )(u_1),(g_1\psi )(v_1))_2\hfill \\ \hfill =& (\varphi (g_1^1u_1),\psi (g_1^1v_1))_2\hfill \\ \hfill =& (\psi ^{}\varphi (g_1^1u_1),g_1^1v_1)_1\hfill \\ \hfill =& (g_1(\psi ^{}\varphi )g_1^1u_1,v_1)_1.\hfill \end{array}$$
(6)
It follows that

$$\mathrm{\Omega }(g_1\varphi ,g_1\psi )=tr((g_1\psi )^{}(g_1\varphi ))=tr(g_1\psi ^{}\varphi g_1^1)=tr(\psi ^{}\varphi )=\mathrm{\Omega }(\varphi ,\psi ).$$

Therefore $`G_1`$ is in $`Sp(V,\mathrm{\Omega })`$. We define the action of $`G_2`$ on $`V`$ similarly by

$$(g_2\varphi )(v_1)=g_2\varphi (v_1).$$

One can verify that $`G_2`$ also preserves $`\mathrm{\Omega }`$. In addition, the action of $`G_1`$ commutes with the action of $`G_2`$.

### 3.2 Subgroups

Let $`V_2^0`$ be a $`D`$-linear subspace of $`V_2`$ such that

* $`(,)_2`$ restricted to $`V_2^0`$ is nondegenerate;
* There exist isotropic subspaces $`X_2^0`$ and $`Y_2^0`$ such that

$$X_2^0Y_2^0=V_2^0$$

Let $`V_2^{}`$ be the space of vectors perpendicular to $`V_2^0`$ with respect to $`(,)_2`$. Write

$$X^0=Hom_D(V_1,X_2^0),Y^0=Hom_D(V_1,Y_2^0),$$

$$V^{}=Hom_D(V_1,V_2^{}),V^0=Hom_D(V_1,V_2^0).$$

For any $`\varphi ,\psi X^0`$,

$$(\psi ^{}\varphi v_1,u_1)_1=(\varphi v_1,\psi u_1)_2=0(v_1,u_1V_1).$$

Thus $`\mathrm{\Omega }(\psi ,\varphi )=tr(\varphi ^{}\psi )=0`$. $`X^0`$ is an isotropic subspace of $`(V,\mathrm{\Omega })`$. For the same reason, $`Y^0`$ is also an isotropic subspace of $`(V,\mathrm{\Omega })`$. Furthermore, we have

$$V=V^{}V^0,V^0=X^0Y^0.$$

Let $`G_2^0`$ be the subgroup of $`G_2`$ such that $`G_2^0`$ restricted to $`V_2^{}`$ is trivial. Then $`G_2^0`$ is isomorphic to $`G_2(V_2^0)`$. Let $`G_2^{}`$ be the subgroup of $`G_2`$ such that $`G_2^{}`$ restricted to $`V_2^0`$ is trivial. Then $`G_2^{}`$ is isomorphic to $`G_2(V_2^{})`$.
Let $`\mathrm{\Omega }^0`$ be the restriction of $`\mathrm{\Omega }`$ on $`V^0`$. Let $`\mathrm{\Omega }^{}`$ be the restriction of $`\mathrm{\Omega }`$ on $`V^{}`$. Then $`Sp(V^0,\mathrm{\Omega }^0)`$ and $`Sp(V^{},\mathrm{\Omega }^{})`$ can be embedded into $`Sp(V,\mathrm{\Omega })`$ diagonally. Let $`GL(X^0,Y^0)`$ be the subgroup of $`Sp(V^0,\mathrm{\Omega }^0)`$ stabilizing $`X^0`$ and $`Y^0`$. Since $`G_1`$ and $`G_2^0`$ act on $`V^0`$, we obtain a dual pair

$$(G_1,G_2^0)Sp(V^0,\mathrm{\Omega }^0).$$

We denote this embedding by $`i^0`$. On the other hand, since $`G_1`$ and $`G_2^{}`$ act on $`V^{}`$, we obtain another dual pair

$$(G_1,G_2^{})Sp(V^{},\mathrm{\Omega }^{}).$$

We denote this embedding by $`i^{}`$. Now the group $`G_1`$ is embedded into $`Sp(V,\mathrm{\Omega })`$ by $`i^0\times i^{}`$. We denote this embedding by $`i`$.

### 3.3 Metaplectic Covering and Compatibility

For any symplectic group $`Sp`$, there is a unique double covering $`MSp`$. We call this the metaplectic covering. Let $`ϵ`$ be the nonidentity element in $`MSp`$ whose image is the identity element in $`Sp`$. For any subgroup $`G`$ of $`Sp`$, let $`MG`$ be the preimage of $`G`$ under the metaplectic covering. Then every $`MG`$ contains $`ϵ`$.
Let $`M^0Sp(V^0,\mathrm{\Omega }^0)`$, $`M^{}Sp(V^{},\mathrm{\Omega }^{})`$ and $`MSp(V,\mathrm{\Omega })`$ be the metaplectic coverings of $`Sp(V^0,\mathrm{\Omega }^0)`$, $`Sp(V^{},\mathrm{\Omega }^{})`$ and $`Sp(V,\mathrm{\Omega })`$ respectively. Let $`M^0`$, $`M^{}`$ and $`M`$ be the covering maps respectively. When we consider $`Sp(V^0,\mathrm{\Omega }^0)`$ as a subgroup of $`Sp(V,\mathrm{\Omega })`$, we obtain a group $`MSp(V^0,\mathrm{\Omega }^0)`$. On the other hand, $`Sp(V^0,\mathrm{\Omega }^0)`$ has its own metaplectic covering, namely, $`M^0Sp(V^0,\mathrm{\Omega }^0)`$.

###### Lemma 3.1 (compatibility)

The group $`MSp(V^0,\mathrm{\Omega }^0)`$ is isomorphic to $`M^0Sp(V^0,\mathrm{\Omega }^0)`$.

Proof: It suffices to show that $`MSp(V^0,\mathrm{\Omega }^0)`$ does not split. Suppose $`MSp(V^0,\mathrm{\Omega }^0)`$ splits. Let $`K`$ be a maximal compact subgroup of $`Sp(V,\mathrm{\Omega })`$ such that $`K^0=KSp(V^0,\mathrm{\Omega }^0)`$ is a maximal compact subgroup of $`Sp(V^0,\mathrm{\Omega }^0)`$. Then $`MK^0`$ splits. On the other hand, $`K`$ can be identified with a unitary group $`U`$. The metaplectic covering of $`U`$ can be represented by

$$\{(\xi ,g)\xi ^2=detg,gU\}$$

For the subgroup $`K^0`$, we see that $`MK^0`$ must be the nontrivial double covering of $`K^0`$. It does not split. We reach a contradiction. Q.E.D.
This Lemma basically asserts that if a smaller symplectic group is embedded in a bigger symplectic group canonically, then the metaplectic covering on the smaller group is compatible with the metaplectic covering on the bigger group. Let

$$\stackrel{~}{i^0}:(M^0G_1,M^0G_2^0)M^0Sp(V^0,\mathrm{\Omega }^0)$$

be the lifting of $`i^0`$. Let

$$\stackrel{~}{i^{}}:(M^{}G_1,M^{}G_2^{})M^{}Sp(V^{},\mathrm{\Omega }^{})$$

be the metaplectic lifting of $`i^{}`$. Let

$$\stackrel{~}{i}:(MG_1,MG_2)MSp(V,\mathrm{\Omega })$$

be the lifting of $`i`$. According to the compatibility lemma, we may consider $`M^0Sp(V^0,\mathrm{\Omega }^0)`$ and $`M^{}Sp(V^{},\mathrm{\Omega }^{})`$ as subgroups of $`MSp(V,\mathrm{\Omega })`$. These two subgroups intersect. The intersection is $`\{1,ϵ\}`$.
Consider the natural multiplication map

$$j:M^0Sp(V^0,\mathrm{\Omega }^0)\times M^{}Sp(V^{},\mathrm{\Omega }^{})MSp(V,\mathrm{\Omega }).$$

Its kernel is $`\{(1,1),(ϵ,ϵ)\}`$. If $`gG_1`$, then

$$i(g)=(i^0(g),i^{}(g))Sp(V^0,\mathrm{\Omega }^0)\times Sp(V^{},\mathrm{\Omega }^{})Sp(V,\mathrm{\Omega })$$

The covering group $`MG_1`$ is then isomorphic to the quotient

$$\{j(g^0,g^{})g^0M^0G_1,g^{}M^{}G_1,M^0(g^0)=g=M^{}(g^{})\}/\{(1,1),(ϵ,ϵ)\}.$$

###### Lemma 3.2

Each element in $`MG_1`$ can be expressed as $`j(g^0,g^{})`$ with

$$(g^0M^0G_1,g^{}M^{}G_1,M^0(g^0)=M^{}(g^{}))$$

up to a factor of

$$\{(1,1),(ϵ,ϵ)\}.$$

###### Lemma 3.3

As a group,

$$M^0G_1\{(g,g^{})M(g)=M^{}(g^{}),gMG_1,g^{}M^{}G_1\}/\{(1,1),(ϵ,ϵ)\}.$$

### 3.4 Oscillator Representation as tensor product

###### Theorem 3.1

The representation

$$\omega (M^0G_1,M^0G_2^0)\omega (M^{}G_1,M^{}G_2^{})$$

restricted to

$$\{j(g^0,g^{})g^0M^0G_1,g^{}M^{}G_1,M^0(g^0)=g=M^{}(g^{})\}$$

descends into $`\omega (MG_1,MG_2)|_{MG_1}`$.

Proof: Suppose $`gMG_1`$. Then $`g`$ can be written as

$$(g^0,g^{})g^0M^0G_1,g^{}M^{}G_1,M^0(g^0)=M^{}(g^{})$$

up to a multiplication by

$$\{(1,1),(ϵ,ϵ)\}.$$

It is easy to see that

$$\omega (MG_1,MG_2)(1,1)=id=\omega (M^0G_1,M^0G_2^0)(ϵ)\omega (M^{}G_1,M^{}G_2^{})(ϵ)$$

It follows that

$$\omega (MG_1,MG_2)(g)=\omega (M^0G_1,M^0G_2^0)(g^0)\omega (M^{}G_1,M^{}G_2^{})(g^{})$$

Our theorem is proved. Q.E.D.
Let $`\pi `$ be an irreducible unitary representation of $`MG_1`$ in the semistable range of $`\theta (MG_1,MG_2)`$ such that $`\pi (ϵ)=1`$. Identify the representation $`\omega (MG_1,MG_2)^c\pi `$ with

$$\omega (M^0G_1,M^0G_2^0)^c(\omega (M^{}G_1,M^{}G_2^{})^c\pi ).$$

From Lemma 3.3, $`g^0M^0G_1`$ can be represented by a pair $`(\stackrel{~}{g},g^{})`$ up to a multiplication of $`(ϵ,ϵ)`$. Since

$$\omega (M^{}G_1,M^{}G_2^{})^c(ϵ)\pi (ϵ)=id,$$

we can write

$$(\omega (M^{}G_1,M^{}G_2^{})^c\pi )(g^0)=\omega (M^{}G_1,M^{}G_2^{})^c(g^{})\pi (g).$$

The proof of Theorem 3.1 shows that

$$\omega (M^{}G_1,M^{}G_2^{})^c\pi $$

can be regarded as a unitary representation of $`M^0G_1`$.

### 3.5 Schrödinger Model of $`\omega (M^0G_1,M^0G_2^0)`$

Recall $`V^0=X^0Y^0`$ and both $`X^0,Y^0`$ are Lagrangian in $`(V^0,\mathrm{\Omega }^0)`$. Let $`GL(X^0,Y^0)`$ be the subgroup of $`Sp(V^0,\mathrm{\Omega }^0)`$ stabilizing $`X^0`$ and $`Y^0`$. Then

$$GL(X^0,Y^0)GL(X^0)GL(Y^0).$$

Let $`L^2(X^0)`$ be a Schrödinger model of $`\omega (M^0G_1,M^0G_2^0)`$ (see , ). The group $`M^0GL(X^0,Y^0)`$ acts on $`L^2(X^0)`$ naturally. Since $`G_1`$ is a subgroup of $`GL(X^0,Y^0)`$, an element in the group $`M^0G_1`$ can be written as

$$(\xi ,g)gG_1,\xi $$

such that the operator

$$(\omega (M^0G_1,M^0G_2^0)(\xi ,g)\varphi )(x)=\xi \varphi (g^1x)(xX^0,\varphi L^2(X^0))$$

is unitary.
Consider

$$_{M^0G_1}(\omega (M^0G_1,M^0G_2^0)(\xi ,g)\varphi ,\psi )(u,(\omega (M^{}G_1,M^{}G_2^{})^c\pi )(\xi ,g)v)𝑑g𝑑\xi $$
(7)
with $`u,v\omega (M^{}G_1,M^{}G_2^{})\pi `$. Since the group action of $`G_1`$ on $`L^2(X^0)`$ is already unitary, $`\xi `$ is a unitary character of $`M^0G_1`$. Thus $`\overline{\xi }\omega (M^0G_1,M^0G_2^0)`$ can be viewed as a unitary representation of $`G_1`$. Moreover,

$$\overline{\xi }\omega (M^0G_1,M^0G_2^0)(g,\xi )\varphi (x)=\varphi (g^1x).$$

Define

$$\pi _0=\overline{\xi }(\omega (M^{}G_1,M^{}G_2^{})^c\pi ).$$

Viewing $`(\omega (M^{}G_1,M^{}G_2^{})^c\pi )`$ as a representation of $`M^0G_1`$, $`\pi _0`$ descends into a unitary representation of $`G_1`$.
Tensor products with $`\overline{\xi }`$ here do not change the ambient spaces. However, the group actions differ by a unitary character. Now, the integral ( 7) becomes a multiple of

$$_{G_1}_{X^0}\varphi (g^1x)\overline{\psi (x)}𝑑x(u,\pi _0(g)v)𝑑g.$$
(8)
This integral can be expressed as orbital integral

$$_{G_1}_{𝒪G_1\backslash X^0}_{x𝒪}.$$

In Part III, we will classify the generic $`G_1`$-orbits in $`X^0`$ and study each generic orbital integral

$$_{G_1}_{x𝒪}\varphi (g^1x)\overline{\psi (x)}(u,\pi _0(g)v)𝑑x𝑑g.$$

## 4 Part III: Orbital Integrals

Recall that $`X^0=Hom_D(V_1,X_2^0)`$. We need to classify the orbital structure of the $`G_1`$-action on $`X^0`$. Let $`m=dim_DV_1`$ and $`dim_DX_2^0=p`$. If $`mp`$, $`(G_1,G_2)`$ is said to be in the stable range. The action of $`G_1`$ on $`X^0`$ is almost free. This case is already treated in . For $`(G_1,G_2)`$ in the stable range, our approach can be simplified and in deed coincides with Li’s approach in . From now on, assume $`mp`$. The set of nonsurjective homomorphisms from $`V_1`$ to $`X_2^0`$ is of measure zero. Hence we will focus on surjective homomorphisms in $`X^0`$. We denote the set of surjective homomorphisms by $`X_0^0`$. Let $`\varphi X_0^0`$.

### 4.1 The Isotropic subgroup $`G_{1}^{}{}_{\varphi }{}^{}`$

Let $`e_1,e_2,\mathrm{},e_m`$ be a $`D`$-linear basis for $`V_1`$, and $`f_1,f_2,\mathrm{},f_p`$ be a $`D`$-linear basis for $`X_2^0`$. Then $`\varphi `$ is uniquely determined by

$$\varphi (e_1),\varphi (e_2),\mathrm{},\varphi (e_m).$$

We will determine the ”generic” isotropic subgroups of the $`G_1`$-action on $`X_0^0`$. Suppose $`gG_1`$ stabilizes $`\varphi `$. In other words,

$$\varphi (u)=(g\varphi )(u)=\varphi (g^1u)(u).$$

This implies that $`\mathrm{ker}(\varphi )`$ is stabilized by $`g`$. Therefore $`\mathrm{ker}(\varphi )^{}`$ is also stabilized by $`g`$.

###### Lemma 4.1

Let $`gG_1`$ and $`\varphi X_0^0`$. Then $`\varphi `$ is fixed by $`g`$ if and only if any vector in $`\mathrm{ker}(\varphi )^{}`$ is fixed by $`g`$.

Proof: Suppose $`\varphi `$ is fixed by $`g`$. Let $`(v,\mathrm{ker}\varphi )_1=0`$. We choose an arbitrary $`uV_1`$. Since $`\varphi (g^1u)=\varphi (u)`$, $`g^1uu\mathrm{ker}\varphi `$. This implies that $`(v,g^1uu)_1=0`$. Thus $`(gv,u)_1=(v,u)_1`$ for every $`uV_1`$. It follows that $`gv=v`$. $`g`$ fixes every vector in $`v\mathrm{ker}\varphi ^{}`$.
Conversely, suppose $`gv=v`$ for any $`(v,\mathrm{ker}\varphi )=0`$. We choose an arbitrary $`uV_1`$. Then $`(gvv,u)_1=0`$. Hence $`(v,g^1uu)_1=0`$ for every $`v\mathrm{ker}\varphi ^{}`$. From the nondegeneracy of $`(,)_1`$,

$$g^1uu(\mathrm{ker}\varphi ^{})^{}=\mathrm{ker}\varphi $$

Therefore, $`\varphi (g^1uu)=0`$ for every $`uV_1`$. It follows that $`g\varphi =\varphi `$. Q.E.D.

###### Theorem 4.1

Let $`\varphi `$ be a surjective homomorphism from $`V_1`$ to $`X_2^0`$. Then the isotropic subgroup $`G_{1}^{}{}_{\varphi }{}^{}`$ is the subgroup that fixes all vectors in $`\mathrm{ker}(\varphi )^{}`$.

The restriction of $`(,)_1`$ onto $`\mathrm{ker}\varphi ^{}`$ contains a null space, namely,

$$W=\mathrm{ker}\varphi \mathrm{ker}\varphi ^{}.$$
(9)
$`W`$ is an isotropic subspace of $`V_1`$ and it may or may not be trivial. Let $`U`$ be a direct complement of $`W`$ in $`\mathrm{ker}\varphi ^{}`$, i.e.,

$$UW=\mathrm{ker}\varphi ^{}.$$
(10)
Then $`(,)_1`$ restricted to $`U`$ is nondegenerate. Thus $`(,)_1`$ restricted onto $`U^{}`$ is a nondegenerate sesquilinear form. Since the group $`G_{1}^{}{}_{\varphi }{}^{}`$ fixes all vectors in $`\mathrm{ker}\varphi ^{}`$ and $`U\mathrm{ker}\varphi ^{}`$, $`G_{1}^{}{}_{\varphi }{}^{}`$ can be identified with the subgroup of $`G_1(U^{}`$ that fixes all vectors in $`W`$.
From Equations 9, 10, $`\mathrm{ker}\varphi `$ is the orthogonal complement of $`W`$ in $`U^{}`$. From Equations (28),(29) in , $`G_{1}^{}{}_{\varphi }{}^{}`$ is a twisted product of $`G_1(\mathrm{ker}\varphi /W)`$ with a at most two-step nilpotent group $`N`$.

###### Theorem 4.2

For orthogonal groups, we take $`G_1=SO(p,q)`$. The isotropic subgroup $`G_{1}^{}{}_{\varphi }{}^{}`$ is a twisted product of a classical group of the same type with a at most two-step nilpotent group $`N`$. It is always unimodular.

Proof: To show that $`G_{1}^{}{}_{\varphi }{}^{}`$ is unimodular, one must show that the adjoint action of $`G_1(\mathrm{ker}\varphi /W)`$ on the Lie algebra $`𝔫`$ has determinant $`1`$. This is obvious since $`𝔫`$ as a $`G_1`$-module decomposes into direct sum of of trivial representations and the standard representations. Q.E.D.

### 4.2 Generic Element

The homomorphism $`\varphi `$ induces an isomorphism

$$[\varphi ]:V_1/\mathrm{ker}\varphi X_2^0$$

Notice that $`\mathrm{ker}\varphi `$ can be regarded as a point in the Grassmannian $`𝒢(m,mp)`$. We obtain a fibration

$$GL_p(D)X_0^0𝒢(m,mp).$$

The projection maps $`\varphi `$ to $`\mathrm{ker}\varphi `$. The fiber contains all isomorphisms from $`V_1/\mathrm{ker}\varphi `$ to $`X_2^0`$. Thus the fiber can be identified with $`GL_p(D)`$.

###### Definition 4.1

Generic elements in $`X^0`$ are those surjective $`\varphi `$ such that

1. either $`(,)_1`$ restricted on $`\mathrm{ker}(\varphi )`$ is nondegenerate;
2. or if the above case is not possible,

$$dim_D(\mathrm{ker}(\varphi )\mathrm{ker}(\varphi )^{})=1$$

Let $`X_{00}^0`$ be the subset of generic elements. The subspaces $`\mathrm{ker}(\varphi )`$ in $`𝒢(m,mp)`$ are called generic $`(mp)`$-subspaces. The set of generic $`(mp)`$-subspaces is denoted by $`𝒢_0(m,mp)`$.

Consider the following fibration

$$GL_p(D)X_{00}^0𝒢_0(m,mp).$$

Since the set $`𝒢_0(m,mp)`$ is open and dense in $`𝒢(m,mp)`$, the set $`X_{00}^0`$ is open and dense in $`X_0^0`$. Therefore, $`X_{00}^0`$ is open and dense in $`X^0`$.
First, suppose $`(,)_1`$ restricted to $`\mathrm{ker}(\varphi )`$ is nondegenerate. We must have

$$\mathrm{ker}(\varphi )\mathrm{ker}(\varphi )^{}=V_1$$

The isotropic subgroup $`G_{1}^{}{}_{\varphi }{}^{}`$ can be identified with $`G_1(\mathrm{ker}(\varphi ))`$ by restriction according to Theorem 4.1. It is a smaller group of type $`G_1`$. The group $`G_{1}^{}{}_{\varphi }{}^{}`$ is automatically unimodular.
Secondly, suppose

$$dim_D(\mathrm{ker}(\varphi )\mathrm{ker}(\varphi )^{})=1$$

Notice that this case does not occur for $`O(p,q)`$. From Theorem 4.2, $`G_{1}^{}{}_{\varphi }{}^{}`$ is a unimodular group. We obtain

###### Corallary 4.1

For any generic element $`\varphi X_{00}^0`$, the isotropy subgroup $`G_{1}^{}{}_{\varphi }{}^{}`$ is always unimodular.

### 4.3 Averaging Integral Revisited

Let $`\pi `$ be an irreducible unitary representation in the semistable range of $`\theta (MG_1,MG_2)`$. Recall that

$$\pi _0=\omega (M^{}G_1,M^{}G_2^{})^c\pi \overline{\xi }$$

is a unitary representation of $`G_1`$. Consider the integral

$$_{G_1}_{X^0}\varphi (g^1x)\overline{\psi (x)}𝑑x(u,\pi _0(g)v)𝑑g$$
(11)
where $`\varphi ,\psi `$ are $`K`$-finite vectors in $`L^2(X^0)`$ and $`u,v\pi _0`$.

###### Theorem 4.3

Let $`\pi `$ be an irreducible unitary representation in the semistable range of $`\theta (MG_1,MG_2)`$. Let $`\varphi ,\psi `$ be in the Harish-Chandra module of $`\omega (M^0G_1,M^0G_2^0)`$. Let $`u,v\pi _0`$. Then the function $`\varphi (g^1x)\overline{\psi (x)}(u,\pi _0(g)v)`$ is continuous and absolutely integrable on $`G_1\times X^0`$. Therefore we have

$$_{G_1}_{X^0}\varphi (g^1x)\overline{\psi (x)}dx(u,\pi _0(g)v)dg=_{X^0}_{G_1}(\varphi (g^1x)\overline{\psi (x)}(u,\pi _0(g)v)dgdx$$

From our discussion in Part II, the integral 11 is a form of the averaging integral under the mixed model

$$\omega (MG_1,MG_2)\omega (M^0G_1,M^0G_2^0)\omega (M^{}G_1,M^{}G_2^{}).$$

The absolute integrabilty of $`\varphi (g^1x)\overline{\psi (x)}(u,\pi _0(g)v)`$ is guaranteed by the semistable condition (see ). We skip the proof.

### 4.4 Orbital integral in General

First, let me quote a simplified version of Theorem 8.36 from .

###### Theorem 4.4

Let $`G`$ be a unimodular group and $`H`$ be a closed unimodular subgroup of $`G`$. Let $`dg`$ and $`dh`$ be their Haar measures, respectively. Then up to a scalar, there exists a unique $`G`$-invariant measure $`d[gH]`$ on $`G/H`$. Furthermore, this measure can be normalized such that for any $`L^1`$ function on $`G`$,

$$_Gf(g)𝑑g=_{G/H}_Hf(gh)𝑑hd[gH].$$

Suppose $`\tau `$ is a unitary representation of $`G`$, $`u`$ and $`v`$ are $`K`$ finite vectors in $`\tau `$.

###### Theorem 4.5

Let $`G`$ be a real reductive group, and $`M`$ be a $`G`$-homogeneous space.

* Let $`x_0`$ be a fixed base point and $`G_0`$ be the isotropy group of $`x_0`$. Suppose that $`G_0`$ is unimodular. Then $`M`$ is isomorphic to $`G/G_0`$ and possesses a $`G`$-invariant measure.
* Let $`\gamma :MG`$ be a smooth section of the principle bundle

$$B:G_0GM$$

except for a subset of at least codimension $`1`$. Assume $`\varphi (y)`$ is an absolutely integrable function on $`M`$. Then

$$v_0=_M\overline{\varphi (y)}\tau (\gamma (y)^1)v𝑑y$$

is well-defined.
* Assume $`\varphi (g^1x_0)(u,\tau (g)v)`$ is integrable as a function on $`G`$. Then we have

$$_G\varphi (gx_0)(\tau (g)u,v)𝑑g=_{G_0}(\tau (g_0)u,v_0)𝑑g_0.$$

Proof: (1) follows directly from Theorem 4.4 by identifying $`M`$ with $`G/G_0`$.
Since $`\tau `$ is unitary and $`\varphi (y)`$ is integrable, $`v_0`$ is well-defined. (2) is proved.
Notice that $`\gamma (y)G_0x_0=y`$. We compute

$$\begin{array}{cc}& _G\varphi (gx_0)(\tau (g)u,v)𝑑g\hfill \\ \hfill =& _{[g]G/G_0}_{G_0}\varphi (gg_0x_0)(\tau (gg_0)u,v)𝑑g_0d[gG_0]\hfill \\ \hfill =& _{yM}\varphi (y)_{G_0}(\tau (\gamma (y)g_0)u,v)𝑑g_0𝑑y\hfill \\ \hfill =& _M\varphi (y)_{G_0}(\tau (g_0)u,\tau (\gamma (y)^1)v)𝑑g_0𝑑y\hfill \\ \hfill =& _{G_0}(\tau (g_0)u,_M\overline{\varphi (y)}\tau (\gamma (y)^1)v𝑑y)𝑑g_0\hfill \\ \hfill =& _{G_0}(\tau (g_0)u,v_0)𝑑g_0.\hfill \end{array}$$
(12)
Q.E.D.
We can further utilize the right invariance of the Haar measure on $`G`$ by changing $`x_0`$ into an arbitrary $`xM`$.

###### Theorem 4.6

Under the same assumptions from Theorem 4.5, suppose $`\psi (x)`$ is an absolutely integrable function on $`M`$. Let

$$u_0=_M\overline{\psi (x)}\tau (\gamma (x)^1)u𝑑x$$

Suppose the function

$$\varphi (g^1x)\overline{\psi (x)}(u,\tau (g)v)$$

is in $`L^1(G\times M)`$. Then we have

$$_M_G\varphi (g^1x)\overline{\psi (x)}(u,\tau (g)v)𝑑g𝑑x=_{G_0}(\tau (g_0)u_0,v_0)𝑑g_0.$$

Proof: First of all, since $`\tau `$ is unitary and $`\psi (x)`$ is integrable, $`u_0`$ is well-defined. According to Fubini’s theorem, we can interchange the order of integrations. We obtain

$$\begin{array}{cc}& _M_G\varphi (g^1x)\overline{\psi (x)}(u,\tau (g)v)𝑑g𝑑x\hfill \\ \hfill =& _M_G\varphi (gx)\overline{\psi (x)}(\tau (g)u,v)𝑑g𝑑x\hfill \\ \hfill =& _M\overline{\psi (x)}_G\varphi (gx)(\tau (g)u,v)𝑑g𝑑x\hfill \\ \hfill =& _M\overline{\psi (x)}(_G\varphi (g\gamma (x)x_0)(\tau (g)u,v)𝑑g)𝑑x\hfill \\ \hfill =& _M\overline{\psi (x)}(_G\varphi (gx_0)(\tau (g\gamma (x)^1)u,v)𝑑g)𝑑x\text{by the right invariance of }dg\hfill \\ \hfill =& _M\overline{\psi (x)}(_G\varphi (gx_0)(\tau (g)\tau (\gamma (x)^1)u,v)𝑑g)𝑑x\hfill \\ \hfill =& _M\overline{\psi (x)}_{G_0}(\tau (g_0)\tau (\gamma (x)^1)u,v_0)𝑑g_0𝑑x\text{by Theorem }\text{4.5}\hfill \\ \hfill =& _{G_0}(\tau (g_0)(_M\overline{\psi (x)}\tau (\gamma (x)^1)u𝑑x),v_0)𝑑g_0\hfill \\ \hfill =& _{G_0}(\tau (g_0)u_0,v_0)𝑑g_0.\hfill \end{array}$$
(13)
Q.E.D.

### 4.5 Orbital Integral $`I(\varphi ,u,𝒪_x)`$

Let $`𝒪_x`$ be a generic $`G_1`$-orbit in $`X_{00}^0`$. Then $`𝒪_x`$ possesses an $`G_1`$-invariant measure. Let $`\pi `$ be a unitary representation in the semistable range of $`\theta (MG_1,MG_2)`$. Let us recall some notations and facts from Part II.

1. $`\xi `$ is a central unitary character of $`M^0G_1`$ and any element $`g^0`$ in $`M^0G_1`$ can be expressed as a pair $`(\xi ,g)`$ with $`g`$ in $`G_1`$.
2. $`\pi _0=\omega (M^{}G_1,M^{}G_2^{})^c\pi \overline{\xi }`$ is a representation of $`G_1`$.

We fix a $`K`$-finite vector $`u`$ in $`\pi \overline{\xi }`$. Suppose

$$\varphi =\underset{i=1}{\overset{s}{}}\varphi _i^0\varphi _i^{}$$

with $`\varphi _i^0\omega (M^0G_1,M^0G_2^0)`$ and $`\varphi _i^{}\omega (M^{}G_1,M^{}G_2^{})`$. Then we have

$$\begin{array}{cc}& (\varphi u,\varphi u)_\pi \hfill \\ \hfill =& _{MG_1}(\omega (MG_1,MG_2)(\stackrel{~}{g})\varphi ,\varphi )(u,\pi (\stackrel{~}{g})u)𝑑\stackrel{~}{g}\hfill \\ \hfill =& \underset{i,j}{}_{M^0G_1}(\omega (M^0G_1,M^0G_2^0)(g^0)\varphi _i^0,\varphi _j^0)(\varphi _j^{}u,(\omega (M^{}G_1,M^{}G_2^{})^c\pi )(g^0)(\varphi _i^{}u))𝑑g^0\hfill \\ \hfill =& \underset{i,j}{}_{M^0G_1}((\omega (M^0G_1,M^0G_2^0)\overline{\xi })(g^0)\varphi _i^0,\varphi _j^0)(\varphi _j^{}u,(\omega (M^{}G_1,M^{}G_2^{})^c\pi \overline{\xi })(g^0)(\varphi _i^{}u))𝑑g^0\hfill \\ \hfill =& 2\underset{i,j}{}_{G_1}_{X^0}\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}𝑑x(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))𝑑g\hfill \\ \hfill =& 2\underset{i,j}{}_{X^0}_{G_1}\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))𝑑g𝑑x\hfill \end{array}$$
(14)
First of all, due to Theorem 4.3, the above integral converges absolutely. Since $`X_{00}^0`$ is open and dense in $`X^0`$,

$$2_{X_{00}^0}_{G_1}\underset{i,j}{}\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))dgdx$$

converges absolutetly. Due to Fubini’s Theorem, for almost all the orbits $`𝒪_x`$ in $`X_{00}^0`$, the function

$$\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))i,j[1,s]$$

is absolutely integrable on $`𝒪_x\times G_1`$. Secondly, since $`\{\varphi _j^0\}_{j=1}^s`$ are rapidly decaying functions in the Schrödinger model of $`\omega (M^0G_1,M^0G_2^0))`$, $`\{\varphi _j^0\}_{j=1}^s`$ are absolutely integrable on $`X_{00}^0`$. Hence $`\{\varphi _j^0\}_{j=1}^s`$ are absolutely integrable on almost every $`G_1`$ orbit $`𝒪_x`$.
Take $`M`$ to be an $`G_1`$-orbit $`𝒪_x`$ such that

1. $`\varphi _j^0`$ is absolutely integrable on $`𝒪_x`$ for every $`j`$;
2. The function

$$\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))$$

is absolutely integrable on $`𝒪_x\times G_1`$ for every $`i,j[1,s]`$.

Denote the orbital integral

$$\underset{i,j}{}_{𝒪_x}_{G_1}\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))𝑑g𝑑x$$

by $`I(\varphi ,u,𝒪_x)`$. Take $`\tau `$ to be $`\pi _0`$. Since $`M`$ can be identified with $`G_1/G_{1}^{}{}_{x}{}^{}`$, $`G_1`$ forms a fiber bundle over $`M`$. By local triviality, we choose a smooth section $`\gamma :MG_1`$ over an open dense subset of $`M`$. Then Theorem 4.6 implies

$$\begin{array}{cc}& I(\varphi ,u,𝒪_x)\hfill \\ \hfill =& \underset{i,j}{}_{𝒪_x}_{G_1}\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))𝑑g𝑑x\hfill \\ \hfill =& \underset{i,j}{}_{G_{1}^{}{}_{x}{}^{}}(\pi _0(g_0)_{𝒪_x}\overline{\varphi _j^0(y)}\pi _0(\gamma (y)^1)(\varphi _j^{}u)𝑑y,_{𝒪_x}\overline{\varphi _i^0(y)}\pi _0(\gamma (y)^1)(\varphi _i^{}u)𝑑y)𝑑g_0\hfill \\ \hfill =& _{G_{1}^{}{}_{x}{}^{}}(\pi _0(g_0)u_0,u_0)𝑑g_0.\hfill \end{array}$$
(15)
Here

$$u_0=_{𝒪_x}\underset{i}{}\overline{\varphi _i^0(y)}\pi _0(\gamma (y)^1)(\varphi _i^{}u)dy.$$

### 4.6 Compactly Supported Continuous Functions

The theorems we have so far proved hold for compactly supported continuous (not necessarily smooth) functions $`\varphi _i^0,\psi _i^0`$ as well. In fact, any compactly supported continuous function on $`X^0`$ can be dominated by a multiple of the Gaussian function $`\mu (x)`$ on $`X^0`$. Therefore, the function

$$|\varphi _i^0(g^1x)\overline{\psi _j^0(x)}(\psi _j^{}u,\pi _0(g)(\varphi _i^{}v))|$$

is always in $`L^1(G_1\times X^0)`$. The rest of the argument from Part III goes through. Again, we obtain

###### Theorem 4.7

Let $`\pi `$ be a unitary representation in the semistable range of $`\theta (MG_1,MG_2).`$ Let $`u`$ be a $`K`$-finite vector in $`\pi \overline{\xi }`$. Let $`\varphi _i^0`$ be compactly supported continuous functions on $`X^0`$ and $`\varphi _i^{}\omega (M^{}G_1,M^{}G_2^{})`$. Write

$$\varphi =\underset{i=1}{\overset{s}{}}\varphi _i^0\varphi _i^{}.$$

Then the integral

$$(\varphi u,\varphi u)_\pi =2\underset{i,j}{}_{X_{00}^0}_{G_1}\varphi _i^0(g^1x)\overline{\varphi _j^0(x)}(\varphi _j^{}u,\pi _0(g)(\varphi _i^{}u))𝑑g𝑑x$$

is absolutely convergent. For almost every $`G_1`$-orbit $`𝒪`$ (except a subset of measure zero), $`I(\varphi ,u,𝒪)`$ converges absolutely. Fix such an orbit $`𝒪_x`$ and a base point $`x`$. Choose any smooth section $`\gamma :𝒪_xG_1`$ over an open dense subset of $`𝒪_x`$. Let

$$u_0=_{𝒪_x}\underset{i}{}\overline{\varphi _i^0(y)}\pi _0(\gamma (y)^1)(\varphi _i^{}u)dy.$$

Then

$$I(\varphi ,u,𝒪_x)=_{G_{1}^{}{}_{x}{}^{}}(\pi _0(g)u_0,u_0)𝑑g.$$

## 5 Part IV: Positivity and Unitarity

###### Lemma 5.1

Suppose $`\pi `$ is a unitary representation in $`(MG_1,\omega (MG_1,MG_2))`$. Suppose for every $`\varphi \omega (MG_1,MG_2)`$ and a fixed nonzero $`u\pi `$

$$(\varphi u,\varphi u)_\pi 0.$$

Then $`(,)_\pi `$ is positive semidefinite. If $`(,)_\pi `$ does not vanish, Then $`\theta (MG_1,MG_2)(\pi )`$ is unitary.

A similar statement can be found in .
Proof: If $`(,)_\pi `$ vanishes, the lemma holds automatically. Suppose $`(,)_\pi `$ does not vanish. Let $`_\pi `$ be the radical of $`(,)_\pi `$. The linear space

$$(𝒫u)/(_\pi (𝒫u))$$

must be nontrivial. Otherwise $`𝒫u_\pi `$. Since $`_\pi `$ is a $`(𝔤_1,MK_1)`$-module, by the $`(𝔤_1,MK_1)`$-action,

$$𝒫\pi ^c_\pi .$$

This contradicts the nonvanishing of $`(,)_\pi `$.
Observe that

$$(𝒫u)/(_\pi (𝒫u))$$

is an admissible Harish-Chandra module of $`MG_2`$. From Theorem 7.8 , it must be irreducible and equivalent to $`𝒫\pi ^c/_\pi `$. Since

$$_{MG_1}(\varphi ,\omega (g)\varphi )(\pi (g)u,u)𝑑g0$$

for a fixed $`u\pi `$ and any $`K`$-finite $`\varphi `$, $`(,)_\pi |_{𝒫u}`$ induces an invariant positive definite form on $`\theta (MG_1,MG_2)(\pi )`$. Thus $`\theta (MG_1,MG_2)(\pi )`$ must be unitary. Consequently, $`(,)_\pi `$ must be positive semidefinite. Q.E.D.

### 5.1 Proof of the Main Theorem

###### Theorem 5.1

Let $`\mathrm{\Xi }(g)`$ be Harish-Chandra’s basic spherical function of $`G_1`$. Suppose

1. $`\pi `$ is a unitary representation in the semistable range of $`\theta (MG_1,MG_2)`$.
2. For any $`x,yG_1`$, the function $`\mathrm{\Xi }(xgy)`$ is integrable on $`G_{1}^{}{}_{\varphi }{}^{}`$ for every generic $`\varphi Hom_D(V_1,X_2^0)`$ (see Definition 4.1).
3. $`\pi _0`$ is weakly contained in $`L^2(G_1)`$.

Then $`(,)_\pi `$ is positive semidefinite. If $`(,)_\pi `$ does not vanish, then $`\theta (MG_1,MG_2)(\pi )`$ is unitary.

Roughly speaking, the second condition requires $`G_{1}^{}{}_{\varphi }{}^{}`$ be half the ”size” of $`G_1`$. The first condition is redundant assuming the second and the third conditions are true. The third conditions can be converted into a growth condition on the matrix coefficients of $`\pi `$.
Proof of the Theorem: Let $`u`$ be a fixed $`K`$-finite vector in $`\pi \overline{\xi }`$. Write

$$𝒮=\{\varphi =\underset{i=1}{\overset{s}{}}\varphi _i^0\varphi _i^{}\varphi _i^0C_c^{\mathrm{}}(X^0),\varphi _i^{}\omega (M^{}G_1,M^{}G_2^{})\}.$$

Let $`\varphi 𝒮`$. Choose an arbitrary $`G_1`$-orbit $`𝒪_x`$ in $`X_{00}^0`$ such that $`I(\varphi ,u,𝒪_x)`$ converges absolutely. There is a canonical fibre bundle

$$G_{1}^{}{}_{x}{}^{}G_1𝒪_x.$$

Fix a smooth section $`\gamma :𝒪_xG_1`$ over an open dense subset of $`𝒪_x`$ such that the closure of $`\gamma (supp(\varphi _i^0))`$ is compact for every $`i`$. Let

$$u_0=_{𝒪_x}\underset{i}{}\overline{\varphi _i^0(y)}\pi _0(\gamma (y)^1)(\varphi _i^{}u)dy.$$

From Theorem 2.3, we have

$$_{G_{1}^{}{}_{x}{}^{}}(\pi _0(g)u_0,u_0)𝑑g0.$$

Combined with Theorem 4.7, we obtain

$$I(\varphi ,u,𝒪_x)0;$$

$$(\varphi u,\varphi u)_\pi =_{𝒪G_1\backslash X_{00}^0}I(\varphi ,u,𝒪)d[𝒪]0.$$

We have thus proved that the Hermitian form $`(,)_\pi `$ restricted to $`𝒮u`$ is positive semidefinite, i.e.,

$$_{MG_1}(\omega (MG_1,MG_2)(\stackrel{~}{g})\varphi ,\varphi )(u,\pi (\stackrel{~}{g})u)𝑑\stackrel{~}{g}0$$

for every $`\varphi 𝒮`$.
For an arbitrary $`K`$-finite vector $`f`$ in $`\omega (MG_1,MG_2)`$, write

$$f=\underset{k=1}{\overset{s}{}}f_k^0(x)f_k^{}(f_k^0\omega (M^0G_1,M^0G_2^0),f_k^{}\omega (M^{}G_1,M^{}G_2^{})).$$

For each $`k`$, choose a sequence $`\psi _k^{(j)}(x)C_c^{\mathrm{}}(X^0)`$ such that

$$|\psi _k^{(j)}(x)||f(x)|$$

$$\psi _k^{(j)}(x)f(x).$$

Let $`\psi ^{(j)}=_{k=1}^s\psi _k^{(j)}f_k^{}`$. Apparently, $`\psi ^{(j)}𝒮`$ and

$$(\omega (MG_1,MG_2)(\stackrel{~}{g})\psi ^{(j)},\psi ^{(j)})(u,\pi (\stackrel{~}{g})u)(\omega (MG_1,MG_2)(\stackrel{~}{g})f,f)(u,\pi (\stackrel{~}{g})u)$$

pointwise. Furthermore,

$$|(\omega (MG_1,MG_2)(\stackrel{~}{g})\psi _j,\psi _j)(u,\pi (\stackrel{~}{g})u)|\underset{k,i=1}{\overset{s}{}}|(\omega (MG_1,MG_2)(\stackrel{~}{g})|f_k^0|f_k^{},|f_i^0|f_i^{})(u,\pi (\stackrel{~}{g})u)|.$$

By the definition of semistable range, the function

$$|((\omega (MG_1,MG_2)(\stackrel{~}{g})|f_k^0|f_k^{},|f_i^0|f_i^{})(u,\pi (\stackrel{~}{g})u)|$$

is absolutely integrable on $`MG_1`$ (see ). Hence, by dominated convergence theorem,

$$(fu,fu)_\pi =\underset{j\mathrm{}}{lim}(\psi ^{(j)}u,\psi ^{(j)}u)_\pi 0$$

Therefore, the form $`(,)_\pi `$ is positive semidefinite. If $`(,)_\pi `$ does not vanish, then $`(,)_\pi `$ considered as a form on

$$\theta (MG_1,MG_2)(\pi )$$

is positive definite ( see ). We conclude that $`\theta (MG_1,MG_2)(\pi )`$ is unitary. Q.E.D.
For $`(G_1,G_2)`$ in the stable range, the generic isotropic group $`G_{1}^{}{}_{\varphi }{}^{}`$ will be trivial. In this case, if $`\pi `$ is an irreducible unitary representation of $`MG_1`$, then $`(,)_\pi `$ is positive semidefinite and nonvanishing. This result is due to Li ( ).

### 5.2 $`G_1=Sp_{2n}()`$

Take $`G_1=Sp_{2n}()`$ as an example. We can make our theorem more precise. First let me define a partial order $``$ in $`^n`$. We say that $`ab`$ if and only if

$$\underset{j=1}{\overset{k}{}}a_j\underset{j=1}{\overset{k}{}}b_j$$

for all $`k`$.

###### Corallary 5.1

Suppose $`n<pq`$. Let $`\pi `$ be an irreducible unitary representation of $`MSp_{2n}()`$. Suppose for every leading exponent (see Chapter 8.8 in ) $`v`$ of $`\pi `$ we have

$$\mathrm{}(v)(\frac{p+q}{2}n1)\rho (Sp_{2n}()).$$

Then $`(,)_\pi `$ is positive semidefinite. In addition, if $`(,)_\pi `$ is nonvanishing, then

$$\theta (MG_1,MG_2)(\pi )$$

is unitary.

Proof: Take $`V_1=^{2n}`$ and $`X_2^0=^{n+1}`$. Then $`V_2^{}`$ is a linear space equipped with a nondegenerate symmetric form of signature $`(pn1,qn1)`$. We verify the conditions in Theorem 5.1.

* For $`xHom(V_1,X_2^0)`$, the generic isotropic group $`G_{1}^{}{}_{x}{}^{}`$ is just $`Sp_{n1}()`$ for $`n`$ odd. For $`n`$ even, the generic $`G_{1}^{}{}_{x}{}^{}`$ can be identified with $`Sp_{n2}()\times N`$ where $`N^n`$. One can easily check that $`\mathrm{\Xi }(g)`$ for $`Sp_{2n}()`$ is integrable on $`G_{1}^{}{}_{x}{}^{}`$.
* Since

$$\mathrm{}(v)(\frac{p+q}{2}n1)\rho (Sp_{2n}()),$$

$`\pi _0=\omega (M^{}G_1,M^{}G_2^{})^c\pi \overline{\xi }`$ has almost square integrable matrix coefficients. According to Theorem 1 of , $`\pi _0`$ is weakly contained in $`L^2(G_1)`$.
* The first condition is really redundant. By Theorem 3.2 , matrix coefficients of $`\omega (MO(n+1,n+1),MSp_{2n}())`$ are in $`L^{2\delta }(MSp_{2n}())`$ for small $`\delta >0`$. Since $`\pi _0`$ is almost square integrable, the matrix coefficients of $`\omega (MO(p,q),MSp_{2n}())\pi `$ are in $`L^{1\delta _0}(MG_2)`$ for small $`\delta _0>0`$. Thus $`\pi `$ must be in the semistable range of $`\theta (MG_1,MG_2)`$.

We conclude that $`(,)_\pi `$ is positive semidefinite. Q.E.D.

### 5.3 $`G_1=O(p,q)`$

Similarly, we obtain

###### Corallary 5.2

Suppose $`p+q2n+1`$. Let $`\pi `$ be an irreducible unitary representation of $`MO(p,q)`$. Suppose for every leading exponent $`v`$ of $`\pi `$ we have

$$\mathrm{}(v)(n\frac{p+q}{2})\rho (O(p,q)).$$

Then $`(,)_\pi `$ is positive semidefinite. In addition, if $`(,)_\pi `$ is nonvanishing, then

$$\theta (MG_1,MG_2)(\pi )$$

is unitary.

For $`p+q`$ odd, the growth condition concerning the leading exponent $`v`$ can be strengthened to allow

$$\mathrm{}(v)(n\frac{p+q1}{2})\rho (O(p,q)).$$

The proof is omitted.


format: text

text:
# The Imprint of the Cosmic Dark Ages on the Near Infrared Background

## 1 Introduction

The cosmic infrared background (CIRB) can set important constraints on the star formation history of the universe (e.g. Hauser & Dwek 2001 and references therein). The CIRB consists of the cumulative radiation from all extragalactic sources. In the near-infrared (NIR) the CIRB is dominated by starlight, whereas in the mid- and far-infrared it results from dust emission.

Different authors had provided measures of the Near Infrared Background (NIRB) in the J, K, and L bands using the data of DIRBE instrument aboard of COBE satellite. Recently, Matsumoto et al. (2000) have accurately measured the NIRB from 1.4 to 4 $`\mu `$m, using the data taken with the Near Infrared Spectrometer. The largest uncertainty on the NIRB measurements is due to the subtraction of the interplanetary dust (IPD) scattered sunlight. In spite of this difficulty some studies have concluded that normal galaxies cannot account for the whole observed NIRB (Totani et al. 2001).

Bond, Carr & Hogan (1986) pointed out that the first population of zero-metallicity stars, the so called Population III (Pop III), might contribute to the cosmic background in the near infrared. More recently, Santos, Bromm & Kamionkowski (2001) suggested that a significant part of the unaccounted NIRB could come from Pop III stars. They considered a very top-heavy Initial Mass Function (IMF), assuming that all stars have masses $`>300M_{}`$ as recent three-dimensional numerical simulations seem to suggest (Bromm, Coppi & Larson 1999, 2002; Abel, Bryan & Norman 2000). In this paper we improve on these earlier studies in many respects. First, we confront theoretical predictions with the recent Matsumoto et al. (2000) data, which provides a much tighter test and constraint. Secondly, we consider the effect of several IMFs. Third, we use the new stellar spectra of Schaerer (2002) for zero age mean sequence (ZAMS) metal free stars, including the nebular continuous emission found to be very important for stars with strong ionizing fluxes.

We fit the estimate of the unaccounted NIRB finding limits on the star formation efficiency and on the redshift at which the formation of Pop III stars ends. We compare our results with high redshift observations and draw some results on IGM metal enrichment due to Pop III supernovae.

The paper is organized as follows: in §2 we describe the background intensity calculation, the stellar spectra used, and how we model the IGM. In §3 the available data in the NIR range are reviewed along with the number counts of normal galaxies from deep field surveys. We discuss the results of our analysis in §4; in §5 we present our conclusions.

We adopt the ‘concordance’ model values for the cosmological parameters: $`h=0.7`$, $`\mathrm{\Omega }_M=0.3`$, $`\mathrm{\Omega }_\mathrm{\Lambda }=0.7`$, $`\mathrm{\Omega }_b=0.038`$, $`\sigma _8=0.9`$, and $`\mathrm{\Gamma }=0.21`$. Here $`h`$ is the dimensionless Hubble constant, $`H_0=100h`$ km s-1 Mpc-1; $`\mathrm{\Omega }_M`$, $`\mathrm{\Omega }_\mathrm{\Lambda }`$, and $`\mathrm{\Omega }_b`$ are the total matter, cosmological constant, and baryon density in units of the critical density; $`\sigma _8`$ gives the normalization of the power spectrum on $`8h^1`$ Mpc scale and $`\mathrm{\Gamma }`$ is the shape of the power spectrum. We use the power spectrum for the fluctuations derived by Efstathiou, Bond & White (1992).

## 2 COSMIC INFRARED BACKGROUND

The mean specific intensity of the background $`I(\nu _0,z_0)`$, as seen at frequency $`\nu _0`$ by an observer at redshift $`z_0`$, is given by

$$I(\nu _0,z_0)=\frac{1}{4\pi }_{z_0}^{\mathrm{}}ϵ(\nu ,z)e^{\tau _{eff}(\nu _0,z_0,z)}\frac{\text{d}l}{\text{d}z}\text{d}z$$
(1)
\[Peebles 1993\]. Here $`ϵ(\nu ,z)`$ is the comoving specific emissivity, $`\nu =\nu _0(1+z)/(1+z_0)`$, $`\tau _{eff}(\nu _0,z_0,z)`$ is the effective optical depth at $`\nu _0`$ of the IGM between redshift $`z_0`$ and $`z`$, and $`\text{d}l/\text{d}z`$ is the proper line element

$`{\displaystyle \frac{\text{d}l}{\text{d}z}}=c[H_0(1+z)E(z)]^1,`$
where $`c`$ is the light speed and

$$E(z)=[\mathrm{\Omega }_M(1+z)^3+\mathrm{\Omega }_\mathrm{\Lambda }+(1\mathrm{\Omega }_M\mathrm{\Omega }_\mathrm{\Lambda })(1+z)^2]^{1/2}.$$
(2)
### 2.1 The Comoving Specific Emissivity

The gas initially virialized in the potential well of the parent dark matter halo, can subsequently fragment and ignite star formation only if the gas can cool efficiently and lose pressure support. For a plasma of primordial composition at temperature $`T<10^4`$ K, the typical virial temperatures of the early bound structures, the only efficient coolant is molecular hydrogen. Thus, a minimum H2 fraction is required for a gas cloud to be able to cool in a Hubble time. As the intergalactic relic H2 abundance falls short by at least two orders of magnitude such requirement, the fate of a virialized lump depends crucially on its ability to rapidly increase its H2 content during the collapse phase. Once a critical H2 fractional abundance of $`5\times 10^4`$ is achieved in an object, the lump will cool, fragment and eventually form stars. This criterion is met only by larger haloes, so that for each virialization redshift there will exist a critical mass, $`M_{min}(z)`$, for which protogalaxies with total mass $`M_h>M_{min}`$ form stars and those with mass $`M_h<M_{min}`$ do not. We adopt here $`M_{min}(z)`$ as computed by Fuller & Couchman (2000).

In absence of additional effects that could prevent or delay the collapse, we can associate to each dark matter halo with mass $`M_h>M_{min}`$ a corresponding stellar mass

$`M_{}=f_{}{\displaystyle \frac{\mathrm{\Omega }_b}{\mathrm{\Omega }_M}}M_h,`$ (3)
where $`f_{}`$ is the fraction of baryons able to cool and form stars; we will refer to this quantity as the star formation efficiency. Thus, the stellar mass per unit comoving volume at redshift $`z`$ contained in haloes with mass $`M_h>M_{min}(z)`$ is given by

$$\rho _{}(z)=_{M_{min}(z)}^{\mathrm{}}n(M_h,z)M_{}\text{d}M_h,$$
(4)
where $`n(M_h,z)`$ is the comoving number density of dark matter haloes of mass $`M_h`$ at redshift $`z`$ given by Press & Schechter (1974).

The comoving specific emissivity, in units of erg s-1 Hz-1 cm-3, is then

$$ϵ(\nu ,z)=l_\nu (z)\rho _{}(z),$$
(5)
where $`ł_\nu (z)`$ is the specific luminosity of the population \[in erg s-1 Hz-1 $`M_{}^1`$\] at redshift $`z`$ (see Sec. 2.4).

### 2.2 The Intergalactic Medium

Absorption due to intergalactic gas located in discrete systems along the line of sight can seriously distort our view of objects at cosmological distances. At wavelengths shortward of Ly$`\alpha `$ ($`\lambda <1216`$ Å) in the emitter rest frame, the source continuum intensity is attenuated by the combined blanketing of lines in the Lyman series, and strongly suppressed by the continuum absorption from neutral hydrogen shortward of the Lyman limit in the emitter rest frame. As pointed out by Haiman & Loeb (1999) no flux is transmitted at wavelengths $`\lambda _0<\lambda _\alpha (1+z_s)`$ from sources at redshift $`1+z_s>32/27(1+z_i)`$, where $`z_i=6.2`$(Gnedin 2001) is the currently favored reionization redshift.

The effective optical depth $`\tau _{eff}`$ through the IGM is defined as $`e^{\tau _{eff}}=<e^\tau >`$, where the mean is taken over all the lines of sight to the redshift of interest. For a Poisson distribution of absorbers (Madau 1991, 1992),

$$\tau _{eff}(\nu _0,z_0,z)=_{z_0}^z𝑑z^{}_0^{\mathrm{}}𝑑N_{\text{H}\mathrm{I}}\zeta (N_{\text{H}\mathrm{I}},z^{})(1e^\tau ),$$
(6)
where $`\zeta (N_{\text{H}\mathrm{I}},z^{})=d^2N/dN_{\text{H}\mathrm{I}}dz^{}`$ is the distribution of the absorbers as a function of redshift and neutral hydrogen column density, $`N_{\text{H}\mathrm{I}}`$; $`\tau (\nu ^{})`$ is the optical depth of an individual cloud for ionizing radiation at frequency $`\nu ^{}`$.

For the Lyman-$`\alpha `$ forest, in the wavelength range $`\lambda _\beta <\lambda _0/(1+z_{em})<\lambda _\alpha `$ where $`\lambda _\alpha =1216`$ Å and $`\lambda _\beta =1026`$ Å, we have \[Madau 1995\]

$`\tau _{eff}=0.0036\left({\displaystyle \frac{\lambda _0}{\lambda _\alpha }}\right)^{3.46}.`$
When $`\lambda _0/(1+z_{em})<\lambda _\beta `$, a significant contribution to the blanketing opacity comes from the higher order lines of the Lyman series. In the wavelength range $`\lambda _{i+1}<\lambda _0/(1+z)<\lambda _i`$, the total effective line-blanketing optical depth can be written as the sum of the contributions from the $`j1`$ transitions,

$$\tau _{eff}=\underset{j=2,i}{}A_j\left(\frac{\lambda _0}{\lambda _j}\right)^{3.46},$$
(7)
where $`\lambda _j`$ and the corresponding values for $`A_j`$ are given in Table 1.

Continuum absorption from neutral hydrogen along the line of sight affects photons observed at $`\lambda _0/(1+z_{em})<\lambda _L`$, where $`\lambda _L=912`$ Å is the Lyman limit. We have

$$\tau =N_{\text{H}\mathrm{I}}\sigma $$
(8)
where $`\sigma (\lambda _0,z)6.3\times 10^{18}(\lambda _0/\lambda _L)^3(1+z)^3`$ cm2 (Osterbrock 1989) is the hydrogen photoionization cross section, and $`(1+z_c)=(\lambda _0/\lambda _L)`$ for $`\lambda _0>\lambda _L`$, whereas for $`\lambda _0<\lambda _L`$, H$`\mathrm{I}`$absorbs photons all the way down to $`z_c=0`$.

For the redshift and column density distribution of absorption lines, the usual form can be adopted

$$\zeta (N_{\text{H}\mathrm{I}},z)=\left(\frac{A}{10^{17}}\right)\left(\frac{N_{\text{H}\mathrm{I}}}{10^{17}\text{ cm}\text{-2}\text{ }}\right)^\beta (1+z)^\gamma .$$
(9)
where the values of coefficients $`A`$, $`\beta `$ and $`\gamma `$ in different ranges in $`N_{\text{H}\mathrm{I}}`$ are given in Table 2, and are taken from Fardal, Giroux & Shull (1998).

### 2.3 Scattering of Ly$`\alpha `$ Photons

Ly$`\alpha `$ line photons from first galaxies are absorbed by neutral hydrogen along the line of sight (Gunn & Peterson 1965). However, they are not destroyed but scatter and diffuse in frequency to the red of the Ly$`\alpha `$ resonance owing to the Hubble expansion of the surrounding H$`\mathrm{I}`$. Eventually, when their net frequency shift is sufficiently large, they escape and travel freely toward the observer. The profile results in a strong, asymmetric Ly$`\alpha `$ emission line around 1225 Å with a scattering tail extending to long wavelengths.

The resulting scattered line profile, $`\mathrm{\Phi }(\nu ,z)`$, has been simulated by Loeb & Rybicki (1999). Here, we use the fitting analytical expression given by Santos et al. (2001)

$$\mathrm{\Phi }(\nu ,z)=\{\begin{array}{cc}\nu _{}(z)\nu ^2\mathrm{exp}\left[\frac{\nu _{}}{\nu }\right]\hfill & \text{if }\nu >0\hfill \\ 0\hfill & \text{if }\nu 0\hfill \end{array},$$
(10)
$$\nu _{}(z)=1.5\times 10^{12}\text{Hz}\left(\frac{\mathrm{\Omega }_bh^2}{0.019}\right)\left(\frac{h}{0.7}\right)^1\frac{(1+z)^3}{E(z)},$$
(11)
where $`E(z)`$ is defined in eq. (2). Note that we have allowed the IGM in the vicinity of the emitting galaxy to be overdense by a factor $`\delta 10`$, to account for the high clustering of first sources.

### 2.4 Emission from a Pop III ‘Stellar Cluster’

We calculate the emission of a Pop III ‘stellar cluster’ according to the equation:

$$l_\nu (z)=_{M_l}^{M_u}F(\nu ,M,z)\varphi (M)\text{d}M,$$
(12)
where $`\varphi (M)`$ is the IMF normalized so that $`_{M_l}^{M_u}\varphi (M)\text{d}M=1`$; $`M_l`$ and $`M_u`$ are the lower and upper mass limits. The spectrum of a star of mass $`M`$ at the rest-frame frequency $`\nu `$ and redshift $`z`$ is given by

$$F(\nu ,M,z)=l_\nu ^{star}(M)+l_\nu ^{neb}(M)+l_\nu ^{Ly\alpha }(M,z).$$
(13)
where $`l_\nu ^{star}(M)`$ is the spectrum of a Pop III star of mass $`M`$, $`l_\nu ^{neb}`$ is the emission of the nebula surrounding the star, and $`l_\nu ^{Ly\alpha }(z)`$ is the emission due to Ly$`\alpha `$ line photons from a source at redshift $`z`$ scattered by the IGM (see Sec. 2.3).

As spectra of Pop III we adopt here those modeled by Schaerer (2002) who relaxed some assumptions made in previous works (Tumlinson & Shull 2000; Tumlinson, Giroux & Shull 2001; Bromm, Kudritzki, & Loeb 2001). Schaerer (2002) presented realistic models for massive Pop III stars and stellar populations based on non-LTE atmospheres, recent stellar evolution tracks and up-to-date evolutionary synthesis models. Moreover, he included nebular continuous emission, which cannot be neglected for metal-poor objects with strong ionizing fluxes. This process increases significantly the total continuum flux at wavelengths redward of Lyman-$`\alpha `$ and leads in turn to reduced emission line equivalent widths.

We add thus the contribution of the nebular emission ($`n_e=100`$ cm-3 is assumed) as described in Schaerer (2002):

$$l_\nu ^{neb}=\frac{\gamma _{tot}}{\alpha _B}(1f_{esc})q(H)$$
(14)
(e.g. Osterbrock 1989), where $`\alpha _B`$ (in units of cm-3 s-1) is the Case B recombination coefficient for hydrogen and $`f_{esc}`$ is the ionizing photon escape fraction out of the idealized region considered here (we assume $`f_{esc}=0`$); $`q(H)=Q(H)/M`$, where $`Q(H)`$ is the ionizing photon rate (in units of s-1) for H and is given for different stellar masses in Table 3 of Schaerer (2002). The continuous emission coefficient $`\gamma _{tot}`$, including free-free and free-bound emission by H, neutral He and single ionized He, as well as the two-photon continuum of hydrogen is given by

$$\gamma _{tot}=\gamma _{HI}+\gamma _{2q}+\gamma _{HeI}\frac{n(He^+)}{n(H^+)}+\gamma _{HeII}\frac{n(He^{++})}{n(H^+)}.$$
(15)
The continuous emission coefficients $`\gamma _i`$ (in units of erg cm3 s-1 Hz-1) are taken from Aller (1987) for an electron temperature of 20000 K. The spectrum of a 1000 $`M_{}`$ star (with and without the nebular emission) and of a 5 $`M_{}`$ star are plotted in Fig. 1.

Finally, the emission of Ly$`\alpha `$ line photons is given by

$$l_\nu ^{Ly\alpha }(z)=c_{Ly\alpha }(1f_{esc})q(H)\mathrm{\Phi }(\nu _{Ly\alpha }\nu ,z),$$
(16)
where $`c_{Ly\alpha }=1.04\times 10^{11}`$ erg (Schaerer 2002). The line profile $`\mathrm{\Phi }(\nu )`$ is given in eq. (10).

#### 2.4.1 Initial Mass Function

The issue of the mass distribution of first metal free stars has been tackled via various hydrodynamical models and other studies (e.g. Abel et al. 1998, Bromm et al. 1999, 2001, 2002, Nakamura & Umemura 2001, Ripamonti et al. 2002). There seems to be an overall consensus that massive stars (up to 1000 $`M_{}`$) may form. Simulations seem to suggest that the primordial IMF might have been biased towards stellar masses $`100`$ $`M_{}`$, but other studies (e.g. Nakamura & Umemura 2001) found that the formation of stars with masses down to 1 $`M_{}`$ is not excluded.

In the local universe the IMF is usually well described by the standard Salpeter law (Salpeter 1955)

$$\varphi (M)M^{2.35}.$$
(17)
Larson (1998) suggested a different form for the high $`z`$ IMF. This has a universal Salpeter-like form at the upper end, but flattens below a characteristic mass, $`M_c`$ which may vary with time. By increasing $`M_c`$ we can mimick a top-heavy IMF of Pop III stars,

$$\varphi (M)M^1\left(1+\frac{M}{M_c}\right)^{1.35}.$$
(18)
Hernandez & Ferrara (2001) have explored the predictions of the standard hierarchical clustering scenario of galaxy formation, regarding the numbers and metallicities of Pop III stars that are likely to be found within our Galaxy today. By comparing these values with observational data, they suggested that the IMF of first stars was increasingly high-mass weighted towards high redshifts, levelling off at $`z9`$ at a characteristic stellar mass scale of 10-15 $`M_{}`$.

In view of our poor knowledge of the IMF at high redshift we consider here three different distributions of mass for the first population of stars: Salpeter, Larson with $`M_c=15`$ $`M_{}`$ (heavy IMF), and Larson with $`M_c=100`$ $`M_{}`$ (very heavy IMF). For all IMFs, $`M_l=1`$ $`M_{}`$ and $`M_u=1000`$ $`M_{}`$. In Fig. 2 are shown the resulting spectra of a Pop III cluster at redshift $`z_s`$=10 for the three different IMFs assumed.

## 3 Observational Constraints

Observations of the NIRB are seriously hampered by the strong atmospheric foreground (e.g. Mandolesi et al. 1998). On the other hand space measurements are not easy because it is difficult to subtract correctly the contribution to the extragalactic background light by interplanetary dust scattered sunlight (zodiacal light). Using the data from the instrument DIRBE aboard of the COBE satellite and from the 2MASS data, Cambresy et al. (2001) found a non zero, isotropic background in the J and K bands. With the same data but a different zodiacal light model, Wright & Johnson (2001) extended the flux estimate to the L band. Cambresy’s and Wright’s results in the J and K bands are compatible if the same zodiacal light model is applied. Matsumoto et al. (2000) provided a preliminary analysis of the the NIRS data obtaining a detection of the NIRB in the 1.4-4 $`\mu `$m range. They applied the same zodiacal light model of Cambresy et al. (2001) and obtained compatible results in the DIRBE bands. At 8000 $`\AA `$, Bernstein, Freedman & Madore (2002) observed a CIRB of $`1.76\pm 0.48\times 10^5`$ erg s-1 cm-2 sr-1, that is $``$4 times smaller than the DIRBE and NIRS measurements in the J band, showing a break around $``$1 $`\mu `$m. The integrated counts in the J and K bands from deep surveys do not account for the total NIRB (Madau & Pozzetti 2000; Totani et al. 2001). Totani et al. (2001) modeled the contribution of galaxies missed by deep galaxy surveys. They found unlikely that the contribution of all the normal galaxies to the NIRB is larger than 30%. All the available data are plotted in Fig. 3. In Table 3 the estimate of the NIRB from the DIRBE data (top panel) and the contribution of normal galaxies (bottom panel) are reported.

### 3.1 The DIRBE Data

The Diffuse Infrared Background Experiment (DIRBE) on board of the Cosmic Background Explorer (COBE, see Boggess et al. (1992)) satellite was designed to search for the Cosmic Infrared Background from 1.25 $`\mu `$m to 240 $`\mu `$m. The DIRBE instrument (Silverberg et al. 1993) was an absolute photometer which provided maps of the full sky in 10 broad bands at 1.25, 2.2, 3.5, 4.9, 12, 25, 60, 100, 140, and 240 $`\mu `$m with a starlight rejection of $`<1`$ nW m-2 sr-1 and an absolute brightness calibration uncertainty of 0.05 and 0.03 nW m-2 sr-1, at 1.25 $`\mu `$m and 2.2 $`\mu `$m respectively. A summary of the DIRBE results is provided by Hauser et al. (1998) and all the NIRB measures based on the DIRBE data are reported in Table 3 (top panel).

In the near infrared bands the dominant foreground intensities in the DIRBE data are the zodiacal light and the light from stars in the Milky Way.

A first attempt to detect the NIRB from the DIRBE data was carried out by the DIRBE group but this gave only an upper limit and failed the tests of isotropy even in limited regions of sky (Arendt et al. 1998). Dwek & Arendt (1998) made a correlation study for the K and L bands and obtained a lower limit for the L band.

Gorjian et al. (2000) removed the foreground due to Galactic stars by directly measuring all stars brighter than 9th magnitude at 2.2 and 3.5 $`\mu `$m in a 2$`{}_{}{}^{}\times 2^{}`$ dark spot near the North Galactic Pole using ground-based telescopes. They calculated the contribution of fainter stars using the statistical model of Wainscoat et al. (1992) and subtracted the zodiacal light contribution using a improvement of the Wright (1998) model. Gorjian et al. found significant positive residuals in the K and L bands which they identified as a probable detection of the NIRB. Wright & Reese (2000) obtained a consistent estimate of the NIRB in the same bands using a different approach based on a histogram fitting method to remove the stellar foreground from the DIRBE data.

Wright (2001) used the Two Micron All Sky Survey (2MASS) data (Cutri et al. 2000) to remove the contribution of Galactic stars brighter than 14th magnitude from the DIRBE maps at 1.25 and 2.2 $`\mu `$m in four dark regions in the north and south Galactic Pole caps. For the subtraction of the zodiacal light foreground the model presented in Gorjian et al. (2000) is applied. Cambresy et al. (2000) have also used the 2MASS data to remove the Galactic stars from the DIRBE data, but they modeled the zodiacal light as in Kelsall et al. (1998). Their results agree with those of Wright (2001) if the same zodiacal light model is applied.

Recently, Wright & Johnson (2001) extended the analysis of Wright (2001) to 13 fields with a wide range of ecliptic latitudes tripling the number of pixels. They obtained an estimate of the NIRB at 1.25 and 2.2 $`\mu `$m consistent with the previous results. They also found a significant residual in the L band combining the Wright (2001) and Dwek & Arendt (1998) techniques, consistent with the Wright & Reese (2000) result.

Matsumoto et al. (in preparation) observed in the J band with the Kiso Schmidt Telescope toward the DIRBE dark spot. They subtracted all detected stars brighter than 14th magnitude and estimated the contribution of fainter stars by the Cohen’s sky model (Cohen 1997). They used the Kelsall’s zodiacal light model.

### 3.2 The NIRS Data

The Near Infrared Spectrometer (NIRS)111http://www.ir.isas.ac.jp/irts/nirs/index-e.html is one of the focal instruments of the InfraRed Telescope in Space (IRTS) (Noda et al. 1994). The NIRS covers the wavelength range from 1.4 to 4.0 $`\mu `$m with a spectral resolution of 0.13 $`\mu `$m. The beam size is $`8^{}\times 20^{}`$. which is considerably smaller than that of DIRBE; about 7% of the sky is surveyed. In order to reduce the contribution from the faint stars, the sky at high latitudes ($`b>40^{}`$) is chosen.

Matsumoto et al. (2000) provided a preliminary analysis of the NIRS data, reporting detection of the Cosmic Infrared Background based upon analysis of the five days of data which were least disturbed by atmospheric, lunar, and nuclear radiation effects. The sky area analyzed included Galactic latitudes from 40 to 58, and ecliptic latitudes from 12 to 71. NIRS is able to identify stars brighter than 10.5 mag. at 2.24 $`\mu `$m, whereas to find out the contribution of fainter stars the Cohen model (1997) is used. The model of Kelsall et al. (1998) is applied to subtract the contribution of the zodiacal light foreground, interpolating between the DIRBE wavelengths to the wavelengths of the NIRS measurements. After subtraction of the IPD contribution, there remains a fairly isotropic residual emission, which they interpreted as evidence for a non-zero background. To obtain a quantitative value for the background at each wavelength they correlated their star-subtracted brightness at each point with the IPD model brightness, and used the extrapolation to zero IPD contribution as a measurement of the NIRB.

The NIRB intensities reported by Matsumoto et al. (2000) near 2.2 and 3.5 $`\mu `$m are similar to the values found by Gorjian et al. (2000). At shorter wavelengths, the NIRS results continue to rise steeply to $`6.5\times 10^5`$ erg s-1 cm-2 sr-1 at 1.4 $`\mu `$m. This is somewhat above the 95% CL upper limit at 1.25 $`\mu `$m of Wright (2001), but it is in agreement with the value obtained by Cambresy et al. (2001) using the same zodiacal light model. The NIRS result implies an integrated background energy over the 1.4-4.0 $`\mu `$m range of $`3.0\times 10^5`$ erg s-1 cm-2 sr-1.

The preliminary report by Matsumoto et al. (2000) does not provide details regarding systematic uncertainties in their results. The largest uncertainty in the reported NIRB values is attributed to the IPD model.

### 3.3 Galaxy Contribution from Deep Counts

Deep optical and NIR galaxy counts provide an estimate of the extragalactic background light (EBL) coming from normal galaxies in the universe. Madau & Pozzetti (2000) derived the contribution of known galaxies in the UBVIJHK bands from the Southern Hubble Deep Field. Although the slope of the number-magnitude relation of the faintest counts is flat enough for the count integration to converge, the recent observations of the NIRB (see previous Section) suggest that the diffuse EBL flux is considerably larger than the count integration. However, a considerable fraction of EBL from galaxies could still have been missed in deep galaxy surveys because of various selection effects.

Totani et al. (2001) using the Subaru Deep Field data (Maihara et al. 2001) have shown that more than 80%-90% of EBL from galaxies has been resolved in the K and J bands and that the contribution by missing galaxies cannot account for the discrepancy between the count integration and the NIRB observations. In Table 3 (bottom panel) we show the count integration in optical and NIR bands from Madau & Pozzetti (2000) and Totani et al. (2001), and the resolved fractions for two galaxy evolution models. The possible number evolution of galaxies is considered by a phenomenological model in which the Schechter parameters of the luminosity function have a redshift dependence as $`\varphi ^{}(1+z)^\eta `$ and $`L^{}(1+z)^\eta `$; i.e., luminosity density is conserved. In Model A an evolution with $`\eta =1`$ is considered, whereas in Model B no evolution ($`\eta =0`$) is assumed (see Totani et al. (2001) for more details). Currently no measure of the contribution of normal galaxies to the NIRB in L band is available whose accurate measurement has to await for SIRTF.

## 4 Model vs. Observations

We compare the NIR data with the predictions of our model, summarized by eq. (1). The data set is composed by the measurements of Matsumoto et al. (2000) and those obtained by the DIRBE data using the Kelsall model for the zodiacal light emission as reported in Table 3 (top panel). For each wavelength we subtracted from the data the contribution of normal galaxies using the upper (lower) limits in the J and K band, obtained by Totani et al. (2001) (last line of Table 3). As discussed already, for $`\lambda >2.2`$ $`\mu `$m no estimate of the NIRB due to normal galaxies is available. Thus in the range 2.2 to 4 $`\mu `$m we consider two extreme cases: no contribution from normal galaxies (labeled with cut) and a constant contribution (labeled with flat).

### 4.1 Model Results

The fit results are quite sensitive to the adopted IMF and to the way we model the contribution of normal galaxies for $`\lambda >2.2`$ $`\mu `$m. In Table 4 are reported the results of the fit for different IMFs, count integration model, and $`z_{end}`$. The reported errors represent the statistical uncertainty of the fit, but systematic uncertainties difficult to evaluate might be present. In Fig. 4 are plotted the values of the star formation efficiency and the corresponding $`\chi ^2`$/DOF (Degree of Freedom) as function $`z_{end}`$ for the cut galaxy count model. The $`\chi ^2`$ increases rapidly as $`z_{end}`$ becomes greater than 9, showing that in this case our model fails to account of the NIRB observation in the J band.

For a Salpeter IMF we find that a large fraction of baryons have to be turned into stars to account of the whole unexplained NIRB. This fraction goes from 37% ($`z_{end}=8`$) to 89% ($`z_{end}=10`$) if we consider the case of no contribution from normal galaxies for $`\lambda >2.2`$ $`\mu `$m, slightly dependent on the choice of the upper and lower limits of the count integration. If we consider a constant (flat model) contribution of normal galaxies up to 2.2 $`\mu `$m the value of $`f_{}`$ goes from $`0.24`$ ($`z_{end}=8`$) to $`0.58`$ ($`z_{end}=10`$). The value of $`f_{}`$ decreases considerably if we consider a top-heavy IMF. For the cut model of the normal galaxy contribution, $`f_{}`$ ranges between $`0.15`$ ($`z_{end}=8`$) and $`0.36`$ ($`z_{end}=10`$) if $`M_c=15`$ $`M_{}`$ and between $`0.1`$ and $`0.23`$ if $`M_c=100`$ $`M_{}`$, whereas for the flat model we find $`f_{}`$ values from $`0.1`$ ($`z_{end}=8`$) to $`0.23`$ ($`z_{end}=10`$) with a heavy IMF, and $`0.060.15`$ (for $`z_{end}=810`$) with a very heavy IMF.

To find the best model, we allow both $`f_{}`$ and $`z_{end}`$ to vary simultaneously and perform a two-parameter fit to the NIR data. The results are reported in Table 5. Again, quoted errors take into account for statistical uncertainties of the fit. In Fig. 5 we plot the best fit models for the cut contribution of normal galaxies. For the flat model the star formation efficiencies are significantly lower than in the cut case, but the $`\chi ^2`$/DOF remains always greater than 1.9, showing that a low contribution of normal galaxies for $`\lambda >2.2`$ $`\mu `$m is favored by our model. A very remarkable result is the constancy of $`z_{end}8.8`$ in the various cases, independent on the IMF or the galaxy contribution model: its value is strongly constrained by the position of the break around 1 $`\mu `$m. The break is principally due to the Ly$`\alpha `$ line emission redshifted into the J band, so that $`\lambda _{obs}\lambda _{\text{Ly}\alpha }/(1+z_{end})`$.

Barkana (2002) derived a value for $`f_{}`$ using the measured distribution of star formation rates in galaxies at various redshifts (Lanzetta et al. 2002) and semi-analytic models of hierarchical galaxy formation in a $`\mathrm{\Lambda }`$CDM cosmology. He found a best fit value of 2.3$`{}_{0.5}{}^{}{}_{}{}^{+0.8}`$% (2$`\sigma `$), neglecting dust obscuration, but the uncertainties on this evaluation are still large. If this value applies also to the first star formation, for a Salpeter IMF the Pop III contribution to the NIRB is not significant ($`<10`$%), whereas Pop III stars characterized by a top-heavy IMF continue to contribute substantially to the NIRB.

### 4.2 Associated Metal Enrichment

Recent theoretical analyses on the evolution of metal-free stars predict that the fate of the massive metal-free stars can be classified as follows (Heger, Woosley & Waters 2000, Chiosi 2000, Heger & Woosley 2002):

1. $`M>260`$ $`M_{}`$: the nuclear energy release from the collapse of stars in this mass range is insufficient to reverse the implosion. The final result is a very massive black hole (VMBH) locking up all heavy elements produced;
2. 130 $`M_{}`$ $`<M<260`$ $`M_{}`$: the mass regime of the pair-instability supernovae (SNγγ). Precollapse winds and pulsations should result in little mass loss, the star implodes reaching a maximum temperature that depends on its mass and then explodes, leaving no remnant. The explosion expels metals into the surrounding ambient ISM;
3. 40 $`M_{}`$ $`<M<130`$ $`M_{}`$: black hole formation is the most likely outcome, because either a successful outgoing shock fails to occur or the shock is so weak that the fall-back converts the neutron star remnant into a black hole (Fryer 1999);
4. 8 $`M_{}`$ $`<M<40`$ $`M_{}`$: SN explosion from ‘normal’ progenitors.

Stars in the mass range (i) and (iii) above fail to eject most of (or all) their heavy elements, whereas (ii) and (iv) type of stars eject metals into the IGM.

To estimate the metal production of Pop III objects we consider SN yields of Woosley & Weaver (1995) for progenitor metal-free stars between 8 and 40 $`M_{}`$ and the results of Heger & Woosley (2002) for very massive objects of pair instability SNe originating from stars in the mass range $`130M_{}`$ to 260 $`M_{}`$. We neglect the contribution from long lived intermediate mass stars ($`1<M<8`$ $`M_{}`$) as their evolution time scale is longer than the Hubble time at the relevant redshift. The density of metals ejected into the IGM in units of the critical density, $`\rho _c=3H_0^2/8\pi G`$, is obtained by

$$\mathrm{\Omega }_Z(z)=f_Z(\varphi )\mathrm{\Omega }_{}(z)$$
(19)
where $`f_Z`$ is the fraction of metal ejected from normal and pair-instability supernovae; it depends on the adopted IMF, $`\varphi `$. Finally, $`\mathrm{\Omega }_{}(z)=\rho _{}/\rho _c`$, where $`\rho _{}`$ is given in eq. (4).

We calculate the IGM metallicity by imposing that the whole unaccounted NIRB is due to Pop III stars. We use the best results for the efficiency reported in Fig. 5 for the cut model of the galaxy counts. We assume here that all the metal escape into the IGM and that they are smoothly distributed. Fig. 6 shows the metallicity of the IGM (in solar units) as a function of redshift.

We compare our results with the estimated value of metals in the Ly$`\alpha `$ forest ($`10^{14.5}<N_{\text{H}\mathrm{I}}<10^{16.5}`$ cm-2) at $`z3`$ (Songaila & Cowie 1996; Davé et al. 1998) and in the ‘true’ IGM ($`N_{\text{H}\mathrm{I}}<10^{14}`$ cm-2) at $`z5`$ (Songaila 2002). Identification of C IV, Si IV and O VI absorption lines which correspond to Ly$`\alpha `$ absorption lines in the spectra of high redshift quasars has revealed that the low density IGM has been enriched up to $`Z_{IGM}10^{2.5\pm 0.5}Z_{}`$. For lower column density clouds (the ‘true’ IGM) Songaila (2002) found a lower limit on the IGM metallicity at $`z5`$ of $`10^{3.5}Z_{}`$.

With the assumed star formation efficiency ($`f_{}=0.534`$ for a Salpeter; $`f_{}=0.214`$ for a heavy IMF; $`f_{}=0.142`$ for a very heavy IMF) the IGM would be enriched to the observed metallicity already at $`z=1518`$ for the Ly$`\alpha `$ forest value and at $`z=2025`$ for the ‘true’ IGM lower limit (Fig. 6). As seen from Fig. 6, for $`z8.8`$ the mean IGM metallicity exceeds by one order of magnitude that observed in the Ly$`\alpha `$ forest.

As an alternative, we are thus forced to consider the analogous case in which only pair-instability SNe eject metals into the IGM (Fig. 7). For a Salpeter IMF the obtained metallicity is now consistent with the limits of the Ly$`\alpha `$ forest, whereas for a heavy IMF this value is slightly exceeded. For top heavy IMFs, for which the contribution of normal SNe is not so important, the results change only slightly.

In order to further reduce these metallicity values, we have to consider lower star formation efficiencies. Consequently only a fraction of the NIRB can be due to Pop III stars. By imposing that Pop III stars do not eject more metals than the observed ones, we find self consistent limits on the star formation efficiency and on the contribution of the first population of stars to the NIRB; Table 6 summarizes the results. Apart for the case of a Salpeter IMF in which only pair-instability SNe contribute to the metal enrichment of the IGM, the star formation efficiency has to be small ($`<10`$%) and the fraction of NIRB due to Pop III stars is $`<20`$%.

The value obtained by Songaila (2002) for the ‘true’ IGM at $`z5`$ allows us to set a lower limit on the star formation efficiency and the contribution of Pop III stars to the NIRB. The efficiency has to be greater than few$`\times 10^4`$ to enrich the IGM to $`Z10^{3.5}Z_{}`$; thus, the Pop III star contribution to the NIRB is at least of the order of 0.4-1.7%. This fraction increases up to 8% if we consider the case in which only pair-instability SNe contribute to the metal enrichment of the universe.

## 5 Discussion

Pop III stars can account for the entire NIRB excess if the high redshift star formation efficiency is $`f_{}=10\%50\%`$, depending on the IMF and on the unknown ‘normal’ galaxy contribution in the L band. Our best results are obtained by considering a value $`z_{end}8.8`$ for the redshift at which the formation of Pop III stars ends, almost independently of the IMF and of the ‘normal’ galaxy contribution model. A hard upper limit $`z_{end}9`$ is set by the J band data. The above efficiency values are not at odd with expectations for objects forming in the Dark Ages (Madau, Ferrara & Rees 2001), although they seem to be higher than those found at lower redshifts.

In addition, the values of $`f_{}`$ found should be considered as upper limits since the unaccounted NIRB level could be lower due to a different zodiacal light subtraction. To quantify this uncertainty we have performed fits using the Wright zodiacal light model. In this case the DIRBE data alone are considered, as NIRS data are only available for the Kelsall model. We find that required star formation efficiencies decrease to $`0.150.41`$ ($`z_{end}=810`$) with a Salpeter IMF, $`0.060.16`$ with a heavy IMF, and $`0.040.1`$ with a heavy IMF; at $`z_{end}=8.8`$ the resulting star formation efficiencies are 0.23 (Salpeter IMF), 0.09 (heavy IMF), and 0.06 (very heavy IMF). However, the errors are doubled with respect to the case of Kelsall model for the zodiacal light emission. Also, the $`\chi ^2`$ analysis does not allow to evaluate a unique redshift at which Pop III formation ends. This is due to the fact that the break around 1 $`\mu `$m is not as sharp as in the previous case, as already mentioned.

As a consistency test, we have calculated the associated IGM metal enrichment. If the whole unaccounted NIRB is due to Pop III stars, then we predict that the low density (‘true’) IGM would be enriched at the observed mean value already by $`z=1520`$, and that the Ly$`\alpha `$ forest data imply that either a Salpeter or a heavy IMF with characteristic mass $`M_c=15M_{}`$ are acceptable, provided pair-instability SNe are the only source of heavy elements. This conclusion can be modified by the two following occurrences: (i) only stars with masses in excess of 260 $`M_{}`$ are formed, that lock their nucleosynthetic products into the VMBH remnant, effectively breaking the light-metal production link (Schneider et al. 2002) or (ii) metals are very inhomogeneously mixed in the IGM, with filling factors much smaller than unity (Scannapieco, Ferrara & Madau 2002) at these high redshifts. In both cases, the limits obtained from metallicity arguments are much weaker if not purely indicative.

To exemplify case (i), let us consider an IMF of the form $`\varphi (M)=\delta (1000M_{})`$. By imposing that the whole NIRB is due to Pop III stars ($`f_{}=0.04\pm 0.01`$ and $`z_{end}=8.83\pm 0.01`$) and assuming the cut model for the galaxy counts, we obtain a density of VMBH $`\mathrm{\Omega }_{VMBH}(z=8.8)7\times 10^4`$. This value is two orders of magnitude greater than the measured mass density in super massive black holes (SMBHs) found from the demography of nuclei of nearby galaxies (Magorrian et al. 1998; Gebhardt et al. 2001), $`\mathrm{\Omega }_{SMBH}=10^4h\mathrm{\Omega }_b=2.66\times 10^6`$ (Merritt & Ferrarese 2001), but it is unlikely that all VMBHs merge into SMBHs (Schneider et al. 2002; Volonteri, Haardt & Madau 2002). Hence it is not clear if this scenario can represent a viable solution.

In the second case (ii), the measured metallicity might not be representative of the actual cosmic metal density and might result in a large overestimate of the amount of IGM heavy elements. In addition, note that above estimates assume that all the produced heavy elements can escape into the IGM. If the metal escape fraction is equal or less than $`10`$%, the whole unaccounted NIRB could easily be due to Pop III stars with no conflict with the observed IGM metallicity limits.

As a final remark, we point out that additional sources might contribute to the NIRB, as accretion onto black holes (Carr 1994) or decay of massive primordial particles (Bond, Carr & Hogan 1986).

## Acknowledgements

We thank D. Schaerer for providing Pop III spectra and V. Bromm and L. Danese for constructive and stimulating comments. This work was partially supported (AF) by the Research and Training Network ‘The Physics of the Intergalactic Medium’ set up by the European Community under the contract HPRN-CT2000-00126 RG29185.