text:
# Tessellations of homogeneous spaces of classical groups of real rank two

## 1. Introduction

###### 1.1 Definition (\[KN, pp. 43–44\]).

A group $`\mathrm{\Gamma }`$ of homeomorphisms of a topological space $`M`$ acts *properly discontinuously* on $`M`$ if, for every compact subset $`C`$ of $`M`$,

$`\{\gamma \mathrm{\Gamma }C\gamma C\mathrm{}\}`$ is finite.
Classically, a discrete group $`\mathrm{\Gamma }`$ of isometries of a Riemannian manifold $`M`$ is a crystallographic group if $`\mathrm{\Gamma }`$ acts properly discontinuously on $`M`$, and the quotient $`\mathrm{\Gamma }\backslash M`$ is compact. The $`\mathrm{\Gamma }`$-translates of any fundamental domain for $`\mathrm{\Gamma }\backslash M`$ form a tessellation of $`M`$.

These notions generalize to any homogeneous space, even without an invariant metric.

###### 1.2 Definition.

Let

* $`G`$ be a Lie group and
* $`H`$ be a closed subgroup of $`G`$.

A discrete subgroup $`\mathrm{\Gamma }`$ of $`G`$ is a *crystallographic group* for $`G/H`$ if

1. $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H`$; and
2. $`\mathrm{\Gamma }\backslash G/H`$ is compact.

We say that $`G/H`$ has a *tessellation* if there exists a crystallographic group $`\mathrm{\Gamma }`$ for $`G/H`$.

Crystallographic groups and the corresponding tessellations have been studied for many groups $`G`$. (A brief recent introduction to the subject is given in \[Kb7\].) The classical Bieberbach Theorems \[Cha, Chap. 1\] deal with the case where $`G`$ is the group of isometries of Euclidean space $`^n=G/H`$. As another example, the Auslander Conjecture \[Abe, AMS, FG, Mr1, Tom\] asserts that if $`G`$ is the group of all affine transformations of $`^n`$, then every crystallographic group has a solvable subgroup of finite index. In addition, the case where $`G`$ is solvable has been discussed in \[Wit\].

In this paper, we focus on the case where $`G`$ is a simple Lie group, such as $`\mathrm{SL}(n,)`$, $`\mathrm{SO}(m,n)`$, or $`\mathrm{SU}(m,n)`$.

###### 1.3 Standing assumptions.

Throughout this paper:

1. $`G`$ is a linear, semisimple Lie group with only finitely many connected components; and
2. $`H`$ is a closed subgroup of $`G`$ with only finitely many connected components.

###### 1.4 Remark.

Because $`H/H^{}`$ is finite (hence compact), it is easy to see that $`G/H`$ has a tessellation if and only if $`G/H^{}`$ has a tessellation. Also, if $`G/H`$ has a tessellation, then $`G^{}/H^{}`$ has a tessellation. Furthermore, the converse holds in many situations. (See §2D for a discussion of this issue.) Thus, there is usually no harm in assuming that both $`G`$ and $`H`$ are connected; we will feel free to do so whenever it is convenient. On the other hand, because $`\mathrm{SO}(m,n)`$ is usually not connected (it usually has two components \[Hel, Lem. 10.2.4, p. 451\]), it would be somewhat awkward to make this a blanket assumption.

###### 1.5 Example.

There are two classical cases in which $`G/H`$ is well known to have a tessellation.

1. If $`G/H`$ is compact, then we may let $`\mathrm{\Gamma }=e`$ (or any finite subgroup of $`G`$).
2. If $`H`$ is compact, then we may let $`\mathrm{\Gamma }`$ be any cocompact lattice in $`G`$. (A. Borel \[Br1\] proved that every connected, simple Lie group has a cocompact lattice.)

Thus, the existence of a tessellation is an interesting question only when neither $`H`$ nor $`G/H`$ is compact. (In this case, any crystallographic group $`\mathrm{\Gamma }`$ must be infinite, and cannot be a lattice in $`G`$.)

Given $`G`$ (satisfying 1.3(1)), we would like to find all the subgroups $`H`$ (satisfying 1.3(2)), such that $`G/H`$ has a tessellation. This seems to be a difficult problem in general. (See the surveys \[Kb5\] and \[Lab\] for a discussion of the many partial results that have been obtained, mainly under the additional assumption that $`H`$ is reductive.) However, it can be solved in certain cases of low real rank. In particular, as we will now briefly explain, the problem is very easy if $`\text{-rank}G=0`$ or $`1`$. Most of this paper is devoted to solving the problem for certain cases where $`\text{-rank}G=2`$.

If $`\text{-rank}G=0`$ (that is, if $`G`$ is compact), then $`G/H`$ must be compact (and $`H`$ must also be compact), so $`G/H`$ has a tessellation, but this is not interesting. If $`\text{-rank}G=1`$, then there are some interesting homogeneous spaces, but it turns out that none of them have tessellations.

###### 1.6 Example.

$`G=\mathrm{SL}(2,)`$ is transitive on $`^2\{0\}`$, so $`^2\{0\}`$ is a homogeneous space for $`G`$. It does not have a tessellation, for reasons that we now explain.

Let $`C`$ be the unit circle, so $`C`$ is a compact subset of $`^2\{0\}`$.

We claim that $`CgC\mathrm{}`$, for every $`gG`$ (cf. Figure 1.1). To see this, note that, because $`detg=1`$, the ellipse bounded by $`gC`$ has the same area as the disk bounded by $`C`$, so $`gC`$ cannot be contained in the interior of the disk bounded by $`C`$, and cannot contain $`C`$ in its interior. Thus, $`gC`$ must be partly inside $`C`$ and partly outside, so $`gC`$ must cross $`C`$, as claimed.

Let $`\mathrm{\Gamma }`$ be any discrete subgroup of $`G`$. The preceding paragraph implies that $`C\gamma C\mathrm{}`$, for every $`\gamma \mathrm{\Gamma }`$. If $`\mathrm{\Gamma }`$ acts properly discontinuously on $`^2\{0\}`$, then, because $`C`$ is compact, this implies that $`\mathrm{\Gamma }`$ is finite. So the quotient $`\mathrm{\Gamma }\backslash (^2\{0\})`$ is not compact. Therefore $`\mathrm{\Gamma }`$ is not a crystallographic group. We have shown that no subgroup of $`G`$ is a crystallographic group, so we conclude that $`^2\{0\}`$ does not have a tessellation.

This example illustrates the *Calabi-Markus Phenomenon*: if there is a compact subset $`C`$ of $`G/H`$, such that $`CgC\mathrm{}`$, for every $`gG`$, then no infinite subgroup of $`G`$ acts properly discontinuously on $`G/H`$ (see 2.8). Thus, $`G/H`$ does not have a tessellation, unless $`G/H`$ is compact (see 2.9).

We will see in Section 2 that the following proposition can be proved quite easily from basic properties of the Cartan projection.

###### 2.16 Proposition ((cf. \[Kb3, Lem. 3.2\])).

If $`\text{-rank}G=1`$, and $`H`$ is not compact, then there is a compact subset $`C`$ of $`G/H`$, such that $`CgC\mathrm{}`$, for every $`gG`$.

###### 1.7 Corollary ((Kulkarni)).

If $`\text{-rank}G=1`$, and neither $`H`$ nor $`G/H`$ is compact, then $`G/H`$ does not have a tessellation.

We now consider groups of real rank two. The obvious example is $`\mathrm{SL}(3,)`$, but, in this case, once again, none of the interesting homogeneous spaces have tessellations. Moreover, the same is true when real numbers are replaced by complex numbers or quaternions. The case where $`dimH=1`$ relies on beautiful methods of Y. Benoist and F. Labourie \[BL\] or G. A. Margulis \[Mr2\], which we describe in Section 5.

###### 1.8 Theorem ((Benoist, Margulis, Oh-Witte, see §6)).

If

$`G=\mathrm{SL}(3,)`$, $`\mathrm{SL}(3,)`$, or $`\mathrm{SL}(3,)`$,
and neither $`H`$ nor $`G/H`$ is compact, then $`G/H`$ does not have a tessellation.

It is important to note that some interesting homogeneous spaces do have tessellations.

###### 1.9 Example.

Suppose $`G=L\times H`$, and let $`\mathrm{\Gamma }`$ be a cocompact lattice in $`L`$. Then $`\mathrm{\Gamma }`$ acts properly discontinuously on $`LG/H`$, and $`\mathrm{\Gamma }\backslash G/H\mathrm{\Gamma }\backslash L`$ is compact. So $`G/H`$ has a tessellation.

The following easy lemma generalizes this example to the situation where $`G`$ is a more general product of $`L`$ and $`H`$, not necessarily a direct product.

###### 9.1 Lemma.

Let $`H`$ and $`L`$ be closed subgroups of $`G`$, such that

* $`G=LH`$,
* $`LH`$ is compact; and
* $`L`$ has a cocompact lattice $`\mathrm{\Gamma }`$.

Then $`G/H`$ has a tessellation. (Namely, $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$.)

For $`G=\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$, this lemma leads to some interesting examples found by R. Kulkarni \[Kul, Thm. 6.1\] and T. Kobayashi \[Kb1, Prop. 4.9\].

###### 1.10 Example ((Kulkarni, Kobayashi)).

There are natural embeddings

$`\mathrm{SO}(1,n)\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(1,n)\mathrm{SU}(2,n)`$.
Furthermore, identifying $`^{1+m}`$ with $`^{2+2m}`$ yields an embedding

$$\mathrm{SU}(1,m)\mathrm{SO}(2,2m).$$

Similarly, identifying $`^{1+m}`$ with $`^{2+2m}`$ yields an embedding

$$\mathrm{Sp}(1,m)\mathrm{SU}(2,2m).$$

Thus, we may think of $`\mathrm{SO}(1,2m)`$ and $`\mathrm{SU}(1,m)`$ as subgroups of $`\mathrm{SO}(2,2m)`$; and we may think of $`\mathrm{SU}(1,2m)`$ and $`\mathrm{Sp}(1,m)`$ as subgroups of $`\mathrm{SU}(2,2m)`$.

With the above understanding, we see that $`\mathrm{SO}(1,2m)`$ is the stabilizer of a vector of norm $`+1`$. Since $`\mathrm{SU}(1,m)`$ is transitive on the set of such vectors, we have

$$\mathrm{SO}(2,2m)=\mathrm{SO}(1,2m)\mathrm{SU}(1,m).$$

Similarly,

$$\mathrm{SU}(2,2m)=\mathrm{SU}(1,2m)\mathrm{Sp}(1,m).$$

Then Lemma 9.1 implies that each of the following four homogeneous spaces has a tessellation:

* $`\mathrm{SO}(2,2m)/\mathrm{SO}(1,2m)`$,
* $`\mathrm{SO}(2,2m)/\mathrm{SU}(1,m)`$,
* $`\mathrm{SU}(2,2m)/\mathrm{SU}(1,2m)`$, and
* $`\mathrm{SU}(2,2m)/\mathrm{Sp}(1,m)`$.

###### 1.11 Remark.

When discussing $`\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$, we always assume $`n>2`$. This causes no harm, because $`\mathrm{SO}(2,2)`$ is locally isomorphic to $`\mathrm{SL}(2,)\times \mathrm{SL}(2,)`$ \[Hel, (x), p. 520\], and $`\mathrm{SU}(2,2)`$ is locally isomorphic to $`\mathrm{SO}(2,4)`$ \[Hel, (vi), p. 519\].

When $`n`$ is even, H. Oh and D. Witte \[OW3\] provided a complete description of all the (closed, connected) subgroups $`H`$, such that $`\mathrm{SO}(2,n)/H`$ has a tessellation, but their classification is not quite complete when $`n`$ is odd.

In this paper, we extend the work of Oh and Witte to obtain analogous results for homogeneous spaces of $`G=\mathrm{SU}(2,n)`$. We also give a much shorter proof of the main results of \[OW3\]. The same techniques should yield significant results for homogeneous spaces of the other simple groups of real rank two, although the calculations seem to be difficult. On the other hand, the groups of higher real rank require different ideas.

Once one knows that a tessellation of $`G/H`$ exists, it would be interesting to find *all* of the crystallographic groups for $`G/H`$ and, for each crystallographic group, describe the possible tessellations. These are much more delicate questions, which we do not address at all. (W. Goldman \[Gol\], F. Salein \[Sal\], T. Kobayashi \[Kb6\], and A. Zeghib \[Zeg\] have interesting results in some special cases.)

In the remainder of this introduction, we state the specific results for homogeneous spaces of $`\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(2,n)`$.

###### 1.12 Notation ((Iwasawa \[Iwa, p. 533\])).

For any connected Lie group $`H`$, let

$$d(H)=dimHdimK_H,$$

where $`K_H`$ is any maximal compact subgroup of $`H`$. This is well defined, because all the maximal compact subgroups of $`H`$ are conjugate \[Hc2, Thm. 15.3.1(iii), pp. 180–181\].

###### 1.13 Example.

If $`H`$ is semisimple, we have the Iwasawa decomposition $`H=K_HA_HN_H`$ \[Hel, Thm. 6.5.1, pp. 270–271\], from which it is obvious that $`d(H)=dim(A_HN_H)`$.

This yields the following calculations (see 7.5 and 7.15)):

* $`d\left(\mathrm{SO}(1,n)\right)=n`$.
* $`d\left(\mathrm{SO}(2,n)\right)=2n`$.
* $`d\left(\mathrm{SU}(1,n)\right)=2n`$.
* $`d\left(\mathrm{SU}(2,n)\right)=4n`$.
* $`d\left(\mathrm{Sp}(1,n)\right)=4n`$.

###### 1.14 Remark.

If $`HAN`$ (for some Iwasawa decomposition $`G=KAN`$ of $`G`$), then $`d(H)=dimH`$ (see 3.18 and 3.15(3)).

###### 1.15 Notation (\[Kb4, Defn. 2.1.1\]).

For subgroups $`H_1`$ and $`H_2`$ of $`G`$, we write $`H_1H_2`$ if there is a compact subset $`C`$ of $`G`$, such that $`H_1CH_2C`$ and $`H_2CH_1C`$.

###### 1.16 Remark.

Note that $`d`$ is not invariant under the equivalence relation $``$. For example, the Cartan decomposition $`G=KAK`$ implies that $`GA`$, but we have $`d(A)=dimAdim(AN)=d(G)`$.

The following two theorems state a version of the main results for even $`n`$.

###### 11.5 Theorem ((Oh-Witte \[OW3\])).

Assume $`G=\mathrm{SO}(2,2m)`$, and let $`H`$ be a closed, connected, subgroup of $`G`$, such that neither $`H`$ nor $`G/H`$ is compact.

The homogeneous space $`G/H`$ has a tessellation if and only if

1. $`d(H)=2m`$; and
2. either $`H\mathrm{SO}(1,2m)`$ or $`H\mathrm{SU}(1,m)`$.

###### 11.5′′ Theorem.

Assume $`G=\mathrm{SU}(2,2m)`$, and let $`H`$ be a closed, connected, subgroup of $`G`$, such that neither $`H`$ nor $`G/H`$ is compact.

The homogeneous space $`G/H`$ has a tessellation if and only if

1. $`d(H)=4m`$; and
2. either $`H\mathrm{SU}(1,2m)`$ or $`H\mathrm{Sp}(1,m)`$.

The subgroups $`H`$ that arise in Theorems 11.5 and 11.5′′ can also be described more explicitly (cf. 11.2 and 11.2′′ below).

T. Kobayashi \[Kb6, 1.4\] conjectured that if $`H`$ is reductive and it is impossible to construct a tessellation of $`G/H`$ by using a generalization of Lemma 9.1 (see 9.1), then $`G/H`$ does not have a tessellation. The following lists three special cases of this general conjecture.

###### 1.17 Conjecture.

The homogeneous spaces

1. $`\mathrm{SO}(2,2m+1)/\mathrm{SU}(1,m)`$,
2. $`\mathrm{SU}(2,2m+1)/\mathrm{Sp}(1,m)`$, and
3. $`\mathrm{SU}(2,2m+1)/\mathrm{SU}(1,2m+1)`$

do not have tessellations.

If this conjecture is true, then, for odd $`n`$, there is no interesting example of a homogeneous space of $`\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$ that has a tessellation.

###### 11.1 Theorem ((Oh-Witte \[OW3, Thm. 1.7\], Iozzi-Witte)).

Assume

$`G=\mathrm{SO}(2,2m+1)`$ or $`\mathrm{SU}(2,2m+1)`$,
and let $`H`$ be any closed, connected subgroup of $`G`$, such that neither $`H`$ nor $`G/H`$ is compact.

If Conjecture 1.17 is true, then $`G/H`$ does not have a tessellation.

The proof of Theorem 11.1 assumes the following special case proved by R. Kulkarni \[Kul, Cor. 2.10\]. In short, Kulkarni noted that the Euler characteristic of $`\mathrm{\Gamma }\backslash G/H`$ must both vanish (because the Euler characteristic of $`G/H`$ vanishes) and not vanish (by the Gauss-Bonnet Theorem). (Other results in the same spirit, obtaining a contradiction from the study of characteristic classes of $`\mathrm{\Gamma }\backslash G/H`$, appear in \[KO\].)

###### 1.18 Theorem ((Kulkarni)).

If $`n`$ is odd, then $`\mathrm{SO}(2,n)/\mathrm{SO}(1,n)`$ does not have a tessellation.

Let us give a more explicit description of the closed, connected subgroups $`H`$ of $`\mathrm{SO}(2,2m)`$ or $`\mathrm{SU}(2,2m)`$, such that $`G/H`$ has a tessellation. This shows that if $`n`$ is even, then the Kulkarni-Kobayashi examples (1.10) and certain deformations are essentially the only interesting homogeneous spaces of $`\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$ that have tessellations.

###### 1.19 Notation (\[Hel, Thm. 6.5.1, pp. 270–271\], \[Hc2, p. 180\]).

Fix an Iwasawa decomposition $`G=KAN`$. Thus,

* $`K`$ is a maximal compact subgroup,
* $`A`$ is the identity component of a maximal split torus, and
* $`N`$ is a maximal unipotent subgroup.

The following two results are stated only for subgroups of $`AN`$, because the general case reduces to this (see 3.5). The reason is basically that $`H`$ contains a connected, cocompact subgroup that is conjugate to a subgroup of $`AN`$. (Clearly, if $`H^{}`$ is any cocompact subgroup of $`H`$, then $`G/H`$ has a tessellation if and only if $`G/H^{}`$ has a tessellation.) This is not quite true in general, but the following lemma provides a satisfactory substitute, by showing that it becomes true after enlarging $`H`$ by a compact amount.

###### 3.5 Lemma.

After replacing $`H`$ by a conjugate subgroup, there is a closed, connected subgroup $`H^{}`$ of $`G`$, such that $`H^{}/H`$ and $`H^{}/(ANH^{})^{}`$ are compact, where $`(ANH^{})^{}`$ denotes the identity component of $`ANH^{}`$.

###### 11.2 Theorem ((Oh-Witte \[OW3, Thm. 1.7\])).

Assume $`G=\mathrm{SO}(2,2m)`$, and let $`H`$ be a closed, connected, nontrivial, proper subgroup of $`AN`$.

The homogeneous space $`G/H`$ has a tessellation if and only if $`H`$ is conjugate to a subgroup $`H^{}`$, such that either

1. $`H^{}=\mathrm{SO}(1,2m)AN`$; or
2. $`H^{}`$ belongs to a certain family $`\{H_B\}`$ of deformations of $`\mathrm{SU}(1,m)AN`$, described explicitly in Theorem 9.7 (with $`𝔽=`$).

###### 11.2′′ Theorem.

Assume $`G=\mathrm{SU}(2,2m)`$, and let $`H`$ be a closed, connected, nontrivial, proper subgroup of $`AN`$.

The homogeneous space $`G/H`$ has a tessellation if and only if $`H`$ is conjugate to a subgroup $`H^{}`$, such that either

1. $`H^{}`$ belongs to a certain family $`\{H_{[c]}\}`$ of deformations of $`\mathrm{SU}(1,2m)AN`$, described explicitly in Theorem 9.14; or
2. $`H^{}`$ belongs to a certain family $`\{H_B\}`$ of deformations of $`\mathrm{Sp}(1,m)AN`$, described explicitly in Theorem 9.7 (with $`𝔽=`$).

The proof of Theorem 11.2 (or 11.5) in \[OW3\] requires a list \[OW1\] of all the homogeneous spaces of $`\mathrm{SO}(2,n)`$ that admit a proper action of a noncompact subgroup of $`\mathrm{SU}(2,n)`$. (The list was obtained by very tedious case-by-case analysis. It was extended to homogeneous spaces of $`\mathrm{SU}(2,n)`$ in \[IW\].) The following proposition (1.20) provides an *a priori* lower bound on $`dimH`$, and it turns out that the classification of the interesting subgroups of large dimension can be achieved fairly easily (see §10). This is the main reason that we are able to give reasonably short complete proofs of Theorems 11.5, 11.5′′, 11.1, 11.2, and 11.2′′.

###### 1.20 Proposition ((see 4.12, 8.21, and 7.15)).

Suppose $`G=\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$, and let $`H`$ be a closed, connected, nontrivial subgroup of $`AN`$. If $`G/H`$ has a tessellation, then

$$dimH\{\begin{array}{cc}n\hfill & \text{if }G=\mathrm{SO}(2,n)\text{ and }n\text{ is even};\hfill \\ n1\hfill & \text{if }G=\mathrm{SO}(2,n)\text{ and }n\text{ is odd};\hfill \\ 2n\hfill & \text{if }G=\mathrm{SU}(2,n)\text{ and }n\text{ is even};\hfill \\ 2n2\hfill & \text{if }G=\mathrm{SU}(2,n)\text{ and }n\text{ is odd}.\hfill \end{array}$$

###### 1.21 Acknowledgment.

This research was partially supported by a grant from the National Science Foundation (DMS-9801136). Much of the work was carried out during productive visits to the Isaac Newton Institute for Mathematical Sciences (Cambridge, U.K.); we would like to thank the Newton Institute for the financial support that made the visits possible. A.I. would like to thank the mathematics department of Oklahoma State University for their warm and generous hospitality and Marc Burger both for pointing out a mistake in the original statement and proof of Theorem 11.2 and for many enlightening conversations.

## 2. Cartan projection and Cartan-decomposition subgroups

The main problem in this paper is to determine whether or not a homogeneous space $`G/H`$ has a tessellation. This requires some method to determine whether or not a given discrete subgroup $`\mathrm{\Gamma }`$ of $`G`$ acts properly discontinuously on $`G/H`$. Y. Benoist and T. Kobayashi (independently) demonstrated that the Cartan projection $`\mu `$ is an effective tool to study this question. It is the foundation of almost all of our work in later sections.

In this section, we introduce the Cartan projection, and describe some of its basic properties. First, however, we recall the notion of a proper action (a generalization of properly discontinuous actions) and of a Cartan-decomposition subgroup. At the end of the section, we use the Cartan projection to briefly discuss the question of when there is a loss of generality in assuming that $`G`$ is connected.

### 2A. Proper actions

###### 2.1 Definition ((\[Kb5, Defn. 2.11\], \[Pal, Defn. 1.2.2, (6)\]).

A topological group $`L`$ of homeomorphisms of a topological space $`M`$ acts *properly* on $`M`$ if, for every compact subset $`C`$ of $`M`$,

$`\{gLCgC\mathrm{}\}`$ is compact.
###### 2.2 Remark.

It is important to note that a *discrete* group of homeomorphisms of $`M`$ acts properly on $`M`$ if and only if it acts properly discontinuously on $`M`$.

For the special case where $`M=G/H`$ is a homogeneous space, the following lemma restates the definition of a proper action in more group-theoretic terms.

###### 2.3 Lemma (\[Kb4, Obs. 2.1.3\]).

A closed subgroup $`L`$ of $`G`$ acts properly on $`G/H`$ if and only if, for every compact subset $`C`$ of $`G`$, the intersection $`L(CHC)`$ is compact.

###### Proof.

If $`C`$ is any compact subset of $`G`$, then $`\overline{C}=CH/H`$ is a compact subset of $`G/H`$; furthermore, any compact subset of $`G/H`$ is contained in one of the form $`\overline{C}`$. We have

$`\{gL\overline{C}g\overline{C}\mathrm{}\}`$ $`=\{gL(CH)(gCH)\mathrm{}\}`$
$`=\{gLg(CH)(CH)^1\}`$
$`=L(CHC^1).`$
This has the following well-known, easy consequence.

###### 2.4 Corollary ((cf. \[Kb4, Lem. 2.2(2)\])).

Suppose $`H`$, $`H_1`$, $`L`$, and $`L_1`$ are closed subgroups of $`G`$. If

* $`L`$ acts properly on $`G/H`$, and
* there is a compact subset $`C`$ of $`G`$, such that $`H_1CHC`$ and $`L_1CLC`$,

then $`L_1`$ acts properly on $`G/H_1`$.

### 2B. Cartan-decomposition subgroups

The following definition describes the subgroups to which the Calabi-Markus Phenomenon applies (cf. Example 1.6).

###### 2.5 Definition.

We say that $`H`$ is a *Cartan-decomposition subgroup* of $`G`$ if $`HG`$ (see Notation 1.15).

###### 2.6 Remark.

From the Cartan decomposition $`G=KAK`$, we know that $`A`$ is a Cartan-decomposition subgroup.

###### 2.7 Remark.

Any conjugate of a Cartan-decomposition subgroup is a Cartan-decomposition subgroup.

###### 2.8 Lemma ((Calabi-Markus Phenomenon, cf. \[Kul, pf. of Thm. A.1.2\])).

If $`H`$ is a Cartan-decomposition subgroup of $`G`$, and $`\mathrm{\Gamma }`$ is a discrete subgroup of $`G`$ that acts properly discontinuously on $`G/H`$, then $`\mathrm{\Gamma }`$ is finite.

###### Proof.

Because $`H`$ is a Cartan-decomposition subgroup, there is a compact subset $`C`$ of $`G`$, such that $`CHC=G`$. However, from Lemma 2.3, we know that $`\mathrm{\Gamma }(CHC)`$ is finite. Therefore

$$\mathrm{\Gamma }=\mathrm{\Gamma }G=\mathrm{\Gamma }(CHC)$$

is finite. ∎

The following well-known, easy fact is a direct consequence of the Calabi-Markus Phenomenon. It is an important first step toward determining which homogeneous spaces have tessellations.

###### 2.9 Corollary.

If $`H`$ is a Cartan-decomposition subgroup of $`G`$, such that $`G/H`$ is not compact, then $`G/H`$ does not have a tessellation.

### 2C. The Cartan projection

###### 2.10 Notation (\[Hel, §9.1, p. 402\]).

* If $`G`$ is connected, let $`A^+`$ be the (closed) positive Weyl chamber of $`A`$ in which the roots occurring in the Lie algebra of $`N`$ are positive (cf. 1.19). Thus, $`A^+`$ is a fundamental domain for the action of the (real) Weyl group of $`G`$ on $`A`$.
* In the general case, let $`A^+`$ be a closed, convex fundamental domain for the action of the (real) Weyl group of $`G`$ on $`A`$, such that $`A^+`$ is contained in the (closed) positive Weyl chamber of $`A`$ in which the roots occurring in the Lie algebra of $`N`$ are positive.

###### 2.11 Definition (\[Hel, Thm. 9.1.1, p. 402\], \[Ben, Kb5\]).

For each element $`g`$ of $`G`$, the Cartan decomposition $`G=KA^+K`$ implies that there is an element $`a`$ of $`A^+`$ with $`gKaK`$. In fact, the element $`a`$ is unique, so there is a well-defined function $`\mu :GA^+`$ given by $`gK\mu (g)K`$.

We remark that the function $`\mu `$ is continuous and proper (that is, the inverse image of any compact set is compact).

The following crucial result of Y. Benoist provides a uniform estimate on the variation of $`\mu `$ over disks of bounded radius. (A related result was proved, independently and simultaneously, by T. Kobayashi \[Kb4, Thm. 3.4\].) The proof is both elementary and elegant. However, it requires a bit of notation, so we postpone it to §8B (and, for concreteness, we will assume that $`G`$ is either $`\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$ in the proof).

###### 2.12 Proposition ((Benoist \[Ben, Prop. 5.1\])).

For any compact subset $`C`$ of $`G`$, there is a compact subset $`C^{}`$ of $`A`$, such that $`\mu (CgC)\mu (g)C^{}`$, for all $`gG`$.

###### 2.13 Notation.

For subsets $`U`$ and $`V`$ of $`A^+`$, we write $`UV`$ if there is a compact subset $`C`$ of $`A`$, such that $`UVC`$ and $`VUC`$.

###### 2.14 Corollary ((Benoist \[Ben, Prop. 5.1\], Kobayashi \[Kb4, Thm. 1.1\])).

For any subgroups $`H_1`$ and $`H_2`$ of $`G`$, we have $`H_1H_2`$ if and only if $`\mu (H_1)\mu (H_2)`$.

###### Proof.

($``$) Let $`C`$ be a compact subset of $`G`$, such that $`H_1CH_2C`$ and $`H_2CH_1C`$. Choose a corresponding compact subset $`C^{}`$ of $`A`$, as in Proposition 2.12. Then

$$\mu (H_1)\mu (CH_2C)\mu (H_2)C^{}$$

and, similarly, $`\mu (H_2)\mu (H_1)C^{}`$.

($``$) Let $`C`$ be a compact subset of $`A`$, such that $`\mu (H_1)\mu (H_2)C`$ and $`\mu (H_2)\mu (H_1)C`$. Then

$$H_1K\mu (H_1)KK(\mu (H_2)C)KK\left((KH_2K)C\right)K$$

and, similarly, $`H_2KH_1(KCK)`$. ∎

The special case where $`H_2=G`$ (and $`H_1`$ is closed and almost connected) can be restated as follows.

###### 2.15 Corollary ((Benoist, Kobayashi)).

$`H`$ is a Cartan-decomposition subgroup of $`G`$ if and only if $`\mu (H)A^+`$.

###### 2.16 Corollary ((cf. \[Kb3, Lem. 3.2\])).

Assume that $`\text{-rank}G=1`$. The subgroup $`H`$ is a Cartan-decomposition subgroup of $`G`$ if and only if $`H`$ is noncompact.

###### Proof.

($``$) We have $`\mu (e)=e`$, and, because $`\mu `$ is a proper map, we have $`\mu (h)\mathrm{}`$ as $`h\mathrm{}`$ in $`H`$. Because $`\text{-rank}G=1`$, we know that $`A^+`$ is homeomorphic to the half-line $`[0,\mathrm{})`$ (with the point $`e`$ in $`A^+`$ corresponding to the endpoint $`0`$ of the half-line), so, by continuity, it must be the case that $`\mu (H)=A^+`$. Then Corollary 2.15 implies that $`H`$ is a Cartan-decomposition subgroup, but we provide the following direct proof that avoids any appeal to Proposition 2.12.

From the definition of $`\mu `$, we have $`KHK=K\mu (H)K`$. Therefore

$$KHK=K\mu (H)K=KA^+K=G,$$

so $`H`$ is a Cartan-decomposition subgroup (by taking $`C=K`$ in Definition 1.15). ∎

By using Lemma 2.3, the proof of Corollary 2.14 also establishes the following.

###### 2.17 Corollary ((Benoist \[Ben, Prop. 1.5\], Kobayashi \[Kb4, Cor. 3.5\])).

Suppose $`H`$ and $`L`$ are closed subgroups of $`G`$. The subgroup $`L`$ acts properly on $`G/H`$ if and only if $`\mu (L)\mu (H)C`$ is compact, for every compact subset $`C`$ of $`A`$.

### 2D. Disconnected groups

As was mentioned in Remark 1.4, we may assume, without loss of generality, that $`H`$ is connected. However, it may not be possible to assume that $`G`$ is connected, because, although there are no known examples, it is possible that the following question has an affirmative answer.

###### 2.18 Question.

Does there exist a homogeneous space $`G/H`$ (satisfying Assumption 1.3), such that $`G^{}/H^{}`$ has a tessellation, but $`G/H`$ does not have a tessellation?

If $`\mathrm{\Gamma }`$ is a crystallographic group for $`G^{}/H^{}`$, then it is easy to see that $`\mathrm{\Gamma }\backslash G/H`$ is compact. However, the following example shows that $`\mathrm{\Gamma }`$ may not act properly discontinuously on $`G/H`$.

###### 2.19 Example.

Let

* $`L=H=\mathrm{SL}(2,)`$,
* $`\sigma `$ be the automorphism of $`L\times H`$ that interchanges the two factors (that is, $`\sigma (x,y)=(y,x)`$),
* $`G=(L\times H)\sigma `$ (semidirect product), and
* $`\mathrm{\Gamma }`$ be a cocompact lattice in $`L`$ (cf. 1.5(2)).

Then $`H=H^{}`$, and $`\mathrm{\Gamma }`$ is a crystallographic group for $`G^{}/H=(L\times H)/H`$ (see Example 1.9).

However, $`\mathrm{\Gamma }L=\sigma ^1H\sigma `$, so $`\mathrm{\Gamma }`$ does not act properly on $`G/H`$ (see 2.3 with $`C=\{\sigma ,\sigma ^1\}`$).

Even so, $`G/H`$ does have a tessellation, because the diagonal embedding

$$\mathrm{\Delta }(\mathrm{\Gamma })=\{(\gamma ,\gamma )L\times H\gamma \mathrm{\Gamma }\}$$

is a crystallographic group for $`G/H`$. Thus, this example does not provide an answer to Question 2.18.

In this example, $`\sigma `$ represents an element of the Weyl group of $`G`$ that does not belong to the Weyl group of $`G^{}`$. The following proposition shows that this is a crucial ingredient in the construction.

###### 2.20 Proposition.

Let $`\mathrm{\Gamma }`$ be a crystallographic group for $`G^{}/H^{}`$.

If the (real) Weyl group of $`G`$ is same as the (real) Weyl group of $`G^{}`$, then $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$.

###### Proof.

By assumption, we may choose the same fundamental domain $`A^+`$ for the Weyl groups of $`G`$ and $`G^{}`$. Let $`\mu :GA^+`$ and $`\mu ^{}:G^{}A^+`$ be the Cartan projections; then $`\mu ^{}`$ is the restriction of $`\mu `$ to $`G^{}`$. For simplicity, assume, without loss of generality, that $`HG^{}`$ (for example, assume $`H`$ is connected). Then, for any compact subset $`C`$ of $`A`$, we have

$$\mu (\mathrm{\Gamma })\mu (H)C=\mu ^{}(\mathrm{\Gamma })\mu ^{}(H)C$$

is finite (see 2.17). Thus, $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H`$ (see 2.17), as desired. ∎

A. Borel and J. Tits \[BT, Cor. 14.6, p. 147\] proved that if $`G`$ is Zariski connected, then every element of the Weyl group of $`G`$ has a representative in $`G^{}`$. Also, any element of the Weyl group must act as an automorphism of the root system. Thus, we have the following corollary.

###### 2.21 Corollary.

Let $`\mathrm{\Gamma }`$ be a crystallographic group for $`G^{}/H^{}`$.

If either

* $`G`$ is Zariski connected, or
* every automorphism of the real root system of $`G^{}`$ belongs to the Weyl group of the root system,

then $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$.

###### 2.22 Example.

1. If $`G=\mathrm{SO}(2,n)`$, then $`G`$ is Zariski connected (because $`\mathrm{SO}(n+2,)`$ is connected \[GdW, Thm. 2.1.9, p. 60\]), so $`G/H`$ has a tessellation if and only if $`G^{}/H^{}`$ has a tessellation.
2. More generally, if $`G^{}=\mathrm{SO}(2,n)^{}`$ or $`\mathrm{SU}(2,n)`$ (with $`n3`$), then every automorphism of the real root system of $`G^{}`$ belongs to the Weyl group of the root system (cf. Figure 7.1), so $`G/H`$ has a tessellation if and only if $`G^{}/H^{}`$ has a tessellation.

###### 2.23 Example.

If $`G=\mathrm{SL}(3,)\sigma `$, where $`\sigma `$ is the Cartan involution of $`\mathrm{SL}(3,)`$, then $`\sigma `$ represents an element of the Weyl group of $`G`$ that does not belong to the Weyl group of $`G^{}`$, so the proposition does not apply to $`G`$. However, this does not matter: if neither $`H`$ nor $`G/H`$ is compact, then Theorem 1.8 implies that $`G^{}/H^{}`$ has no tessellations, so $`G/H`$ has no tessellations either.

## 3. Preliminaries on subgroups of $`AN`$

This section recalls a technical result that often allows us to assume that $`H`$ is a subgroup of $`AN`$. It also recalls some basic topological properties of such subgroups, and also recalls a simple observation relating these subgroups to the root spaces of the Lie algebra $`𝔤`$.

### 3A. Reduction to subgroups of $`AN`$

###### 3.1 Definition (\[Hel, Thm. 9.7.2, p. 431\]).

An element $`g`$ of $`G`$ is:

* *hyperbolic* if $`g`$ is conjugate to an element of $`A`$;
* *unipotent* if $`g`$ is conjugate to an element of $`N`$;
* *elliptic* if $`g`$ is conjugate to an element of $`K`$.

###### 3.2 Lemma ((Real Jordan Decomposition \[Hel, Lem. 9.7.1, p. 430\])).

Each $`gG`$ has a unique decomposition in the form $`g=auc`$, such that

* $`a`$ is hyperbolic, $`u`$ is unipotent, and $`c`$ is elliptic; and
* $`a`$, $`u`$, and $`c`$ all commute with each other.

###### 3.3 Remark.

If $`g=auc`$ is the Real Jordan Decomposition of some element $`g`$ of $`G`$, then $`a`$, $`u`$, and $`c`$ commute, not only with each other, but also with any element of $`G`$ that commutes with $`g`$. This is because the Real Jordan Decomposition of $`h^1gh`$ is

$$h^1gh=(h^1ah)(h^1uh)(h^1ch):$$

if $`h^1gh=g`$, then the uniqueness of the Real Jordan Decomposition of $`g`$ implies $`h^1ah=a`$, $`h^1uh=u`$, and $`h^1gh=c`$.

The following observation is a generalization of the fact that a collection of commuting triangularizable matrices can be simultaneously triangularized.

###### 3.4 Lemma ((cf. pf. of \[Hm1, Thm. 17.6\])).

If $`H`$ is abelian (or, more generally, solvable), and is generated by hyperbolic and/or unipotent elements, then $`H`$ is conjugate to a subgroup of $`AN`$.

Because of the following result, we usually assume $`HAN`$ (by replacing $`H`$ with a conjugate of $`H^{}`$).

###### 3.5 Lemma ((cf. \[OW1, Lem. 2.9\])).

If $`H`$ is connected, then there is a closed, connected subgroup $`H^{}`$ of $`G`$ and a compact, connected subgroup $`C`$ of $`G`$, such that

1. $`H^{}`$ is conjugate to a subgroup of $`AN`$;
2. $`CH=CH^{}`$ is a subgroup of $`G`$; and
3. $`d(H^{})=d(H)`$ (see Notation 1.12).

Moreover, it is easy to see from (2) that the homogeneous space $`G/H`$ has a tessellation if and only if $`G/H^{}`$ has a tessellation.

###### Idea of proof..

First, let us note that every connected subgroup of $`AN`$ is closed (see 3.15(1) and 3.18), so we do not need to show that $`H^{}`$ is closed.

Second, let us note that (3) is a consequence of (1) and (2). To see this, let $`K^{}`$ be a maximal compact subgroup of $`CH`$ that contains $`C`$. Then a standard argument shows that $`K^{}H`$ is a maximal compact subgroup of $`H`$. (Because all maximal compact subgroups of $`CH`$ are conjugate, there is some $`gCH`$, such that $`(g^1K^{}g)H`$ is a maximal compact subgroup of $`H`$ that contains $`K^{}H`$. Since $`CK^{}`$, we know that $`C`$ normalizes $`K^{}`$, so we may assume $`gH`$; thus, $`g`$ normalizes $`H`$. Then $`g^1(K^{}H)g=(g^1K^{}g)H`$ contains $`K^{}H`$. Because $`K^{}H`$ is compact, this implies that $`g`$ normalizes $`K^{}H`$. So $`K^{}H=(g^1K^{}g)H`$ is a maximal compact subgroup of $`H`$.) Therefore

$$dim(K^{}H/K^{})=dim\left(H/(K^{}H)\right)=dimHdim(K^{}H)=d(H).$$

Similarly, $`dim(K^{}H^{}/K)=d(H^{})`$. Since $`K^{}H=CH=CH^{}=K^{}H^{}`$, we conclude that $`d(H^{})=d(H)`$, as desired.

Case 1 . Assume $`H`$ is semisimple.  We have an Iwasawa decomposition $`H=K_HA_HN_H`$; let $`H^{}=A_HN_H`$ and $`C=K_H`$.

Case 2 . Assume $`H=\{h^t\}`$ is a one-parameter subgroup.  Let

* $`h^t=a^tu^tc^t`$ be the Real Jordan Decomposition of $`h^t`$ (see 3.2);
* $`H^{}=\{a^tu^t\}`$; and
* $`C=\overline{\{c^t\}}`$ be the closure of $`\{c^t\}`$.

(Lemma 3.4 implies that $`H^{}`$ is conjugate to a subgroup of $`AN`$.)

Case 3 . Assume $`H`$ is abelian.  We may write $`H`$ as a product of one-parameter subgroups:

$$H=\{h_1^{t_1}h_2^{t_2}\mathrm{}h_r^{t_r}t_1,\mathrm{},t_r\}.$$

Let $`h_j^t=a_j^tu_j^tc_j^t`$ be the Real Jordan Decomposition of $`h_j^t`$ (see 3.2). Note that $`a_j^{t_j}`$, $`u_j^{t_j}`$, and $`c_j^{t_j}`$ commute, not only with each other, but also with every $`a_k^{t_k}`$, $`u_k^{t_k}`$, and $`c_k^{t_k}`$ (see 3.3). Let

$$H^{}=\{(a_1^{t_1}u_1^{t_1})(a_2^{t_2}u_2^{t_2})\mathrm{}(a_k^{t_r}u_k^{t_r})t_1,\mathrm{},t_r\},$$

and let $`C=\overline{\{c_1^t\}}\mathrm{}\overline{\{c_1^t\}}`$. (Lemma 3.4 implies that $`H^{}`$ is conjugate to a subgroup of $`AN`$.)

Case 4 . The general case.  From the Levi decomposition \[Jac, p. 91\], we know that there is a connected, semisimple subgroup $`L`$ of $`H`$ and a connected, solvable, normal subgroup $`R`$ of $`H`$, such that $`H=LR`$ (and $`LR`$ is finite). Let $`U=[H,R]`$, so $`U`$ is a connected, normal subgroup of $`H`$, and $`U`$ is conjugate to a subgroup of $`N`$ (cf. \[Jac, Cor. 2.7.1, p. 51\]). By modding out $`U`$, we (essentially) reduce to the direct product of Cases 3A and 3A. ∎

###### 3.6 Remark.

For $`H`$ and $`H^{}`$ as in Lemma 3.5, Proposition 3.20 (and 3.18) implies that if $`H^{}AN`$, then $`AN/H^{}`$ is not compact; also, Proposition 3.15(3) (and 3.18) implies that if $`H^{}e`$, then $`H^{}`$ is not compact. Therefore:

* $`H^{}=AN`$ if and only if $`G/H`$ is compact; and
* $`H^{}=e`$ if and only if $`H`$ is compact.

Thus, if neither $`H`$ nor $`G/H`$ is compact, then $`H^{}`$ is a nontrivial, proper subgroup of $`AN`$.

### 3B. Topology of solvable groups and their homogeneous spaces

Everything is this subsection is well known, though somewhat scattered in the literature. The main results are Propositions 3.15 and 3.20, which, together with Corollary 3.18, show that connected subgroups of $`AN`$ and their homogeneous spaces are very well behaved topologically. Corollary 3.19, on the homology of very simple quotient spaces, is also used in later sections.

We begin with the easy case of abelian groups. This lemma generalizes almost verbatim to solvable groups (see 3.15), but the proof in that generality is not as trivial.

###### 3.7 Lemma (\[Var, Thm. 3.6.2, p. 196\]).

Let $`R`$ be a $`1`$-connected, abelian Lie group.

1. If $`H`$ is a connected subgroup of $`R`$, then $`H`$ is closed, simply connected, and isomorphic to $`^k`$, for some $`k`$.
2. If $`H`$ and $`L`$ are connected subgroups of $`R`$, then $`HL`$ is connected.
3. If $`C`$ is a compact subgroup of $`R`$, then $`C`$ is trivial.

###### Proof.

Because $`R`$ is abelian and 1-connected, the exponential map is a Lie group isomorphism from the additive group of the Lie algebra $`𝔯`$ onto $`R`$.

(1) Let $`k=dimH`$. Because the exponential map is a Lie group isomorphism (hence a diffeomorphism), and because $`𝔥`$ is a closed $`k`$-submanifold of $`𝔯`$, we know that $`\mathrm{exp}(𝔥)`$ is a closed $`k`$-submanifold of $`R`$. Of course, $`\mathrm{exp}(𝔥)`$ is contained in $`H`$, which is also a $`k`$-submanifold of $`R`$. Because the dimensions are the same, we know that $`\mathrm{exp}(𝔥)`$ is open in $`H`$. Also, because $`\mathrm{exp}(𝔥)`$ is closed in $`R`$, we know that $`\mathrm{exp}(𝔥)`$ is closed in $`H`$. Therefore

(3.8)
$$\mathrm{exp}(𝔥)=H$$

(because $`H`$ is connected). Finally, we know that $`\mathrm{exp}|_𝔥`$ is a diffeomorphism from its domain $`𝔥^k`$ onto its image $`H`$.

(2) From (3.8), we have $`\mathrm{exp}(𝔥)=H`$ and, similarly, $`\mathrm{exp}(𝔩)=L`$. Also, because $`\mathrm{exp}`$ is bijective, we have $`\mathrm{exp}𝔥\mathrm{exp}𝔩=\mathrm{exp}(𝔥𝔩)`$. Therefore

$$HL=\mathrm{exp}𝔥\mathrm{exp}𝔩=\mathrm{exp}(𝔥𝔩)$$

is connected.

(3) Because $`^k`$ is not compact (for $`k>0`$), we know, from 3.7(1), that $`C^{}`$ is trivial; so $`C`$ is finite. Since $`R(𝔯,+)^d`$ has no elements of finite order, we conclude that $`C`$ is trivial. ∎

As is usual in the theory of solvable groups, the main results of this section are proved by induction, based on modding out some normal subgroup $`L`$. To be effective, this method requires an understanding of the quotient space $`R/L`$. The information we need (even if $`L`$ is not normal) comes from the following elementary observation, because $`R`$ is a principal $`L`$-bundle over $`R/L`$.

###### 3.9 Lemma.

Let $`P`$ be a principal $`H`$-bundle over a manifold $`M`$.

1. If $`H`$ is diffeomorphic to $`^n`$, then

1. $`P`$ is $`H`$-equivariantly diffeomorphic to $`M\times H`$, so
    2. $`P`$ is homotopy equivalent to $`M`$.
2. If $`M`$ is diffeomorphic to $`^n`$, then

1. $`P`$ is $`H`$-equivariantly diffeomorphic to $`M\times H`$, so
    2. $`P`$ is homotopy equivalent to $`H`$.

###### Proof.

Any principal bundle with a section is trivial \[Hus, Cor. 4.8.3, p. 48\]. If either the fiber or the base is contractible, then there is no obstruction to constructing a section \[Hus, Thm. 2.7.1(H1), p. 21\], so $`P`$ is trivial: $`PM\times H`$. (The diffeomorphism can be taken to be $`H`$-equivariant, with respect to the natural $`H`$-action on $`M\times H`$, given by $`(m,h)h^{}=(m,hh^{})`$.) Then the conclusions on homotopy equivalence follow from the fact that $`^n`$ is contractible (that is, homotopically trivial). ∎

We recall the long exact sequence of the fibration $`HRR/H`$:

###### 3.10 Lemma (\[Whi, Cor. IV.8.6, p. 187\]).

Let $`H`$ be a closed subgroup of a Lie group $`R`$. There is a (natural) long exact sequence of homotopy groups:

$$\mathrm{}\pi _1(H)\pi _1(R)\pi _1(R/H)\pi _0(H)\pi _0(R)\pi _0(R/H)0.$$

###### 3.11 Corollary.

Let $`H`$ be a closed subgroup of a $`1`$-connected Lie group $`R`$. The homogeneous space $`R/H`$ is simply connected if and only if $`H`$ is connected.

###### Proof.

Because $`R`$ is 1-connected, we have $`\pi _1(R)=\pi _0(R)=0`$, so, from (3.10), we know that the sequence

$$0\pi _1(R/H)\pi _0(H)0$$

is exact. Thus, $`\pi _1(R/H)\pi _0(H)`$, so the desired conclusion is immediate. ∎

As a step toward Proposition 3.15, we prove two special cases that describe the topology of normal subgroups.

###### 3.12 Lemma.

If $`R`$ is a $`1`$-connected, solvable Lie group, then $`R`$ is diffeomorphic to $`^d`$, for some $`d`$.

###### Proof.

We may assume the group $`R`$ is nonabelian (otherwise, the desired conclusion is given by Lemma 3.7(1)). Then, because $`R`$ is solvable, there is a nontrivial, connected, proper, closed, normal subgroup $`L`$ of $`R`$. Since $`R/L`$ is simply connected (see 3.11), and $`dim(R/L)<dimR`$, we may assume, by induction on $`dimR`$, that $`R/L`$ is diffeomorphic to some $`^{d_1}`$. Therefore

1. $`R`$ is diffeomorphic to $`(R/L)\times L`$ and
2. $`L`$ is homotopy equivalent to $`R`$

(see 3.9(2)). Because $`R`$ is $`1`$-connected, (b) implies that $`L`$ is $`1`$-connected; hence, $`L`$ is a 1-connected, solvable Lie group, so we may assume, by induction on $`dimR`$, that $`L`$ is diffeomorphic to some $`^{d_2}`$. Thus, (a) implies that $`R`$ is diffeomorphic to $`^{d_1}\times ^{d_2}^{d_1+d_2}`$, as desired. ∎

###### 3.13 Corollary ((of proof)).

If $`R`$ is a $`1`$-connected, solvable Lie group, then every connected, closed, normal subgroup of $`R`$ is $`1`$-connected.

The following proposition is a nearly complete generalization of Lemma 3.7 to the class of solvable groups. There are two exceptions:

1. Of course, subgroups of a solvable group may not be abelian, so the conclusion in 3.7(1) that $`H`$ is isomorphic to some $`^k`$ must be weakened to the conclusion that $`H`$ is diffeomorphic to some $`^k`$.
2. The intersection of connected subgroups is not always connected (see 3.14), so we add the restriction that $`L`$ is normal to 3.7(2). (We remark that no such restriction is necessary if $`RAN`$, because the exponential map is a diffeomorphism from $`𝔯`$ onto $`R`$ in this case \[Dix, Sai\].)

###### 3.14 Example.

Let

$$R=\left\{\left(\begin{array}{ccc}e^{2\pi it}& x+iy& 0\\ 0& 1& 0\\ 0& 0& e^t\end{array}\right)|t,x,y\right\},g^t=\left(\begin{array}{ccc}e^{2\pi it}& 0& 0\\ 0& 1& 0\\ 0& 0& e^t\end{array}\right),$$

$$u=\left(\begin{array}{ccc}1& 1& 0\\ 0& 1& 0\\ 0& 0& 0\end{array}\right),h^t=u^1g^tu=\left(\begin{array}{ccc}e^{2\pi it}& e^{2\pi it}1& 0\\ 0& 1& 0\\ 0& 0& e^t\end{array}\right).$$

Then $`R`$, being diffeomorphic to $`^3`$, is 1-connected; and $`\{g^t\}`$ and $`\{h^t\}`$ are connected subgroups. But

$$\{g^t\}\{h^t\}=\left\{\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& e^n\end{array}\right)|n\right\}$$

is not connected.

###### 3.15 Proposition (\[Hc2, Thms. 12.2.2 and 12.2.3, pp. 137–138\]).

Let $`R`$ be a $`1`$-connected, solvable Lie group.

1. If $`H`$ is a connected subgroup of $`R`$, then $`H`$ is closed, simply connected, and diffeomorphic to some $`^d`$.
2. If $`H`$ and $`L`$ are connected subgroups of $`R`$, and $`L`$ is normal, then $`HL`$ is connected.
3. If $`C`$ is a compact subgroup of $`R`$, then $`C`$ is trivial.

###### Proof.

(2) We may assume $`L`$ is nontrivial, so $`dim(R/L)<dimR`$. Thus, by induction on $`dimR`$, using (1), we may assume that $`HL/L`$ is a closed, simply connected subgroup of $`R/L`$. Then, since $`H/(HL)`$ is homeomorphic to $`HL/L`$, we see that

(3.16)
$$H/(HL)\text{ is simply connected},$$

so Lemma 3.11 implies that $`HL`$ is connected.

(1) Because $`R`$ is solvable, there is a connected, closed, proper, normal subgroup $`L`$ of $`R`$, such that $`R/L`$ is abelian. We know that $`L`$ is 1-connected (see 3.13), so, by induction on $`dimR`$, we may assume that every connected subgroup of $`L`$ is closed and simply connected. From (2), we know that $`HL`$ is connected, so we conclude that $`HL`$ is closed, and

(3.17)
$$\pi _1(HL)=0.$$

From (3.10) (with $`H`$ in the place of $`R`$, and $`L`$ in the place of $`H`$), together with (3.16) and (3.17), we conclude that $`\pi _1(H)=0`$; that is, $`H`$ is simply connected. So (3.12) implies $`H`$ is diffeomorphic to some $`^d`$.

Because both $`HL/L`$ and $`HL`$ are closed, it is not difficult to see that $`H`$ is closed.

(3) Because $`R`$ is solvable, there is a connected, closed, proper, normal subgroup $`L`$ of $`R`$, such that $`R/L`$ is abelian. We know that $`R/L`$ is 1-connected (see 3.11), so $`R/L`$ has no nontrivial, compact subgroups (see 3.7(3)); thus, we must have $`CL`$. Therefore, $`C`$ is a compact subgroup of $`L`$. Then, since $`L`$ is 1-connected (see 3.13), we may conclude, by induction on $`dimR`$, that $`C`$ is trivial. ∎

###### 3.18 Corollary.

$`AN`$ is a $`1`$-connected, solvable Lie group.

###### Proof.

Because $`G`$ is linear, it is a subgroup of some $`\mathrm{GL}(n,)`$. Replacing $`G`$ by a conjugate, we may assume that $`AN`$ is contained in the group $`B`$ of upper triangular matrices with positive diagonal entries (cf. 3.4). The matrix entries provide an obvious diffeomorphism from $`B`$ onto $`(^+)^n\times ^{n(n1)/2}^{n(n+1)/2}`$, so $`B`$ is 1-connected. Thus, Proposition 3.15(1) implies that $`AN`$ is simply connected. ∎

The following observation will be used in Sections 4 and 9.

###### 3.19 Corollary.

Let $`F`$ be a connected subgroup of $`AN`$, and suppose we have a proper, $`C^{\mathrm{}}`$ action of $`F`$ on a manifold $`M`$. Then $`M`$ and $`M/F`$ have the same homology.

###### Proof.

Because the action is proper, we know that the stabilizer of each point of $`M`$ is compact. However, $`F`$ has no nontrivial compact subgroups (see 3.15(3)). Thus, the action is free.

Because the action is free, proper, and $`C^{\mathrm{}}`$, it is easy to see that the manifold $`M`$ is a principal fiber bundle over the quotient $`M/F`$ \[Pal, Thm. 1.1.3\]. Furthermore, the fiber $`F`$ of the bundle is contractible (see 3.15(1)), so Lemma 3.9(1) implies that $`M`$ homotopy equivalent to $`M/F`$. Therefore, the spaces $`M`$ and $`M/F`$ have the same homology. ∎

For the special case where $`M/F`$ is a homogeneous space of a solvable group, the following more detailed result describes the topology of $`M/F`$, not just its homology.

###### 3.20 Proposition ((Mostow \[Mos, Prop. 11.2\])).

If $`H`$ is any connected subgroup of a $`1`$-connected, solvable Lie group $`R`$, then $`R/H`$ is diffeomorphic to the Euclidean space $`^d`$, for some $`d`$.

###### Proof.

Because $`R`$ is solvable, it has a nontrivial, connected, closed, abelian, normal subgroup $`L`$. Since $`L`$ is abelian and $`HL`$ is connected (see 3.15(2)), we know that $`L/(HL)`$ is a 1-connected abelian group (see 3.11), so it is isomorphic to some $`^{d_1}`$ (see 3.7(1)).

We know $`H`$ is closed (see 3.15(1)). Also, since $`L`$ is nontrivial, we have $`dim(R/L)<dimR`$, so we may assume, by induction on $`dimR`$, that

$$R/(HL)(R/L)/(HL/L)$$

is diffeomorphic to some $`^{d_2}`$.

Now $`R`$ is a principal $`HL`$-bundle over $`R/(HL)`$. Because $`R/(HL)^{d_2}`$, this bundle is trivial (see 3.9(2)): $`R`$ is $`HL`$-equivariantly diffeomorphic to $`R/(HL)\times HL`$. Then

$$R/HR/(HL)\times HL/HR/(HL)\times L/(HL)^{d_2}\times ^{d_1}=^{d_1+d_2},$$

as desired. ∎

### 3C. $`T`$-invariant subspaces of $`𝔞+𝔫`$

The following well-known observation puts an important restriction on the subspaces of $`𝔞+𝔫`$ that are normalized by a torus. It is an ingredient in our case-by-case analysis of all possible subgroups of $`AN`$ in Sections 10 and 11.

###### 3.21 Lemma.

Let

* $`\mathrm{\Phi }^+`$ be the set of weights of $`A`$ on $`𝔫`$ (in other words, the set of all positive real roots of $`G`$);
* $`T`$ be a subgroup of $`A`$;
* $`\omega \mathrm{\Phi }^+\{0\}`$;
* $`𝔫^{=\omega }=_{\sigma |_T=\omega |_T}𝔫_\sigma `$, where the sum is over all $`\sigma \mathrm{\Phi }^+\{0\}`$, such that the restriction of $`\sigma `$ to $`T`$ is the same as the restriction of $`\omega `$ to $`T`$;
* $`𝔫^\omega =_{\sigma |_T\omega |_T}𝔫_\sigma `$, where the sum is over all $`\sigma \mathrm{\Phi }^+\{0\}`$, such that the restriction of $`\sigma `$ to $`T`$ is not the same as the restriction of $`\omega `$ to $`T`$.

If $`𝔲`$ is any $``$-subspace of $`𝔞+𝔫`$ normalized by $`T`$, then $`𝔲=(𝔲𝔫^{=\omega })(𝔲𝔫^\omega )`$.

###### Proof.

Since $`TA`$, we know that the elements of $`\mathrm{Ad}_GT`$ are simultaneously diagonalizable (over $``$), so their restrictions to the invariant subspace $`𝔲`$ are also simultaneously diagonalizable (cf. \[ZS, Thms. 26 and 27 in §3.12, pp. 167–168\]). Thus, $`𝔲`$ is a direct sum of weight spaces:

$$𝔲=\underset{\psi \mathrm{\Psi }}{}𝔲_\psi .$$

For each weight $`\psi `$ of $`T`$ on $`𝔲`$, we have

$$𝔲_\psi =𝔲𝔫_\psi =𝔲𝔫^{=\psi },$$

so

$$𝔲_{\omega |_T}=𝔲𝔫^{=\omega }$$

and

$$\underset{\psi \omega |_T}{}𝔲_\psi =𝔲𝔫^\omega .$$

The conclusion follows. ∎

## 4. Lower bound on the dimension of $`H`$

In this section, we prove Corollary 4.12, an *a priori* lower bound on $`dimH`$. On the way, we recall a result of T. Kobayashi that will also be used several times in later sections, and we establish that crystallographic groups have only one end.

### 4A. T. Kobayashi’s Dimension Theorem

The following theorem is essentially due to T. Kobayashi. (Kobayashi assumed that $`H`$ is reductive, but H. Oh and D. Witte \[OW3, Thm. 3.4\] pointed out that, by using Lemma 3.5, this restriction can be eliminated.) The proof here is based on Kobayashi’s original argument and the modifications of Oh-Witte, but uses less sophisticated topology. Namely, instead of group cohomology and the spectral sequence of a covering space, we use only some basic properties of homology groups of manifolds (including Lemma 3.19). These comments also apply to Theorem 9.1.

###### 4.1 Theorem ((Kobayashi, cf. \[Kb2, Thm. 1.5\], \[Kb1, Thm. 4.7\])).

Let $`H`$ and $`H_1`$ be closed, connected subgroups of $`G`$, and assume there is a crystallographis group $`\mathrm{\Gamma }`$ for $`G/H`$, such that $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H_1`$. Then:

1. We have $`d(H_1)d(H)`$.
2. If $`d(H_1)d(H)`$, then $`\mathrm{\Gamma }\backslash G/H_1`$ is compact, so $`G/H_1`$ has a tessellation.

###### Proof.

By Lemma 3.5, we may assume $`H,H_1AN`$. (So $`d(H)=dimH`$ and $`d(H_1)=dimH_1`$ (see 1.14).)

From Lemma 3.19, we know that $`\mathrm{\Gamma }\backslash G`$ and $`\mathrm{\Gamma }\backslash G/H_1`$ have the same homology. Therefore

$$\mathrm{max}\{k_k(\mathrm{\Gamma }\backslash G)0\}=\mathrm{max}\{k_k(\mathrm{\Gamma }\backslash G/H_1)0\}dimG/H_1,$$

with equality if and only if $`\mathrm{\Gamma }\backslash G/H_1`$ is compact \[Dol, Cor. 8.3.4, p. 260\]. Similarly, we have

$$\mathrm{max}\{k_k(\mathrm{\Gamma }\backslash G)0\}=dimG/H.$$

Combining these two statements, we conclude (1) that $`dimG/HdimG/H_1`$ and, furthermore, (2) that equality holds if and only if $`\mathrm{\Gamma }\backslash G/H_1`$ is compact. ∎

###### 4.2 Corollary ((Kobayashi)).

Let $`H`$ and $`H_1`$ be closed, connected subgroups of $`G`$, such that $`d(H_1)>d(H)`$. If there is a compact subset $`C`$ of $`A`$, such that $`\mu (H_1)\mu (H)C`$, then $`G/H`$ does not have a tessellation.

###### Proof.

Suppose $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$. (This will lead to a contradiction.) Because $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H`$, the assumption on $`\mu (H_1)`$ implies that $`\mathrm{\Gamma }`$ also acts properly discontinuously on $`G/H_1`$ (cf. 2.17). So Theorem 4.1(1) yields a contradiction. ∎

### 4B. Crystallographic groups have only one end

It is easy to see that crystallographic groups are finitely generated; we now show that they have only one end (see 4.9).

###### 4.3 Definition ((cf. \[Gro, $`0.2.A_2^{}`$, p. 4\])).

Let $`F`$ be a finite generating set for an (infinite) group $`\mathrm{\Gamma }`$. We say that $`\mathrm{\Gamma }`$ has *only one end* if, for every partition $`\mathrm{\Gamma }=A_1A_2C`$ of $`\mathrm{\Gamma }`$ into three disjoint sets $`A_1`$, $`A_2`$, and $`C`$, such that $`A_1`$ and $`A_2`$ are infinite, but $`C`$ is finite, there exists $`\gamma A_1`$ and $`fFF^1`$, such that $`\gamma fA_2`$. (This does not depend on the choice of the generating set $`F`$.)

The following observation is a straightforward reformulation of Definition 4.3 (obtained by letting $`A_2=\mathrm{\Gamma }(A_1C^{})`$ and $`C=C^{}A_1`$).

###### 4.4 Lemma.

Let $`F`$ be a finite generating set for an infinite group $`\mathrm{\Gamma }`$. If $`A_1`$ and $`C^{}`$ are subsets of $`\mathrm{\Gamma }`$, such that

* $`A_1`$ is infinite,
* $`C^{}`$ is finite, and
* $`A_1fA_1C^{}`$, for every $`fFF^1`$,

then the complement $`\mathrm{\Gamma }A_1`$ is finite.

###### 4.5 Remark ((cf. \[Coh, pp. 25–26, p. 32, and Prop. 2.14\])).

Definition 4.3 is often stated in the language of Cayley graphs: The *Cayley graph* of $`\mathrm{\Gamma }`$, with respect to the generating set $`F`$, is the graph $`\mathrm{Cay}(\mathrm{\Gamma };F)`$ whose vertex set $`V`$ and edge set $`E`$ are given by:

$`V`$ $`=\mathrm{\Gamma };`$
$`E`$ $`=\{(\gamma ,\gamma f)\gamma \mathrm{\Gamma },fFF^1\}.`$
The group $`\mathrm{\Gamma }`$ has only one end if and only if, for every finite subset $`C`$ of $`\mathrm{\Gamma }`$, the graph $`\mathrm{Cay}(\mathrm{\Gamma };F)C`$ has only one infinite component.

The following lemma is not difficult, but, unfortunately, we do not have a proof that is both short and elementary.

###### 4.6 Lemma ((see proof of Lemma 10.11(1))).

If $`HN=AN`$, then, for some $`xN`$, the conjugate $`x^1Hx`$ is normalized by $`A`$.

###### 4.7 Corollary.

If $`d(G)d(H)1`$, and $`G/H`$ is not compact, then $`G/H`$ does not have a tessellation.

###### Proof.

It suffices to show that $`H`$ is a Cartan-decomposition subgroup of $`G`$ (see 2.9).

We may assume, without loss of generality, that $`HAN`$ (see 3.5); then

$$dimH+1=d(H)+1d(G)=dim(AN)$$

(see 1.14 and 1.13). A theorem of B. Kostant \[Kos, Thm. 5.1\] implies that $`N`$ is a Cartan-decomposition subgroup, so we may assume $`NH`$; then $`dim(HN)dimN1`$. Therefore

$$dim(HN)=dimH+dimNdim(HN)dimH+1dim(AN).$$

Hence $`HN=AN`$, so, from Lemma 4.6, we see that, after replacing $`H`$ by a conjugate subgroup, we may assume that $`H`$ is normalized by $`A`$. Then, letting $`\omega =0`$ and $`T=A`$ in (3.21), we see that $`𝔥=(𝔥𝔞)+(𝔥𝔫)`$. Since $`HN=AN`$, we have $`𝔥+𝔫=𝔞+𝔫`$, so this implies that $`𝔞𝔥`$; therefore $`H`$ contains $`A`$. Since $`A`$ is a Cartan-decomposition subgroup (see 2.6), this implies $`H`$ is a Cartan-decomposition subgroup, as desired. ∎

###### 4.8 Definition ((cf. \[Gro, $`0.2.A_2^{}`$, p. 4\])).

A topological space $`M`$ is *connected at $`\mathrm{}`$* if every compact subset $`𝒞`$ is contained in a compact subset $`𝒞^{}`$, such that the complement $`M𝒞^{}`$ is connected.

###### 4.9 Proposition.

If $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$, then $`\mathrm{\Gamma }`$ is finitely generated and has only one end.

###### Proof.

Assume, without loss of generality, that $`HAN`$ (see 3.5). Then $`H`$ is torsion free, so $`\mathrm{\Gamma }`$ must act freely on $`G/H`$; therefore $`\mathrm{\Gamma }\backslash G/H`$ is a compact manifold (rather than an orbifold). Because $`\mathrm{\Gamma }`$ is essentially the fundamental group of $`\mathrm{\Gamma }\backslash G/H`$ (specifically, $`\mathrm{\Gamma }\pi _1(\mathrm{\Gamma }\backslash G/H)/\pi _1(G/H)`$), and the fundamental group of any compact manifold is finitely generated \[Rag, Thm. 6.16, p. 95\], we know that $`\mathrm{\Gamma }`$ is finitely generated.

From the Iwasawa decomposition $`G=KAN`$, we see that $`G/H`$ is homeomorphic to $`K\times (AN/H)`$, and Proposition 3.20 asserts that $`AN/H`$ is homeomorphic to $`^d`$, for some $`d`$. Obviously, we must have $`d=dim(AN)dimH`$, and we may assume $`G/H`$ is not compact (otherwise, $`\mathrm{\Gamma }`$ is finite, so the desired conclusion is obvious), so Corollary 4.7 implies that $`d>1`$. Thus, we conclude that $`G/H`$ is connected at $`\mathrm{}`$.

To complete the proof, we use a standard argument (cf. \[Gro, $`0.2.C_1`$, p. 5\]) to show that, because $`G/H`$ is connected at $`\mathrm{}`$ and $`\mathrm{\Gamma }\backslash G/H`$ is compact, the group $`\mathrm{\Gamma }`$ has only one end. To begin, note that there is a compact subset $`𝒞`$ of $`G/H`$, such that $`\mathrm{\Gamma }𝒞=G/H`$. Let

$$F_0=\{f\mathrm{\Gamma }𝒞f𝒞\mathrm{}\}$$

(cf. \[PR, (ii), p. 195\]). Because $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H`$, we know that $`F_0`$ is finite; let $`F`$ be a finite generating set for $`\mathrm{\Gamma }`$, such that $`F_0F`$.

Suppose $`\mathrm{\Gamma }=A_1A_2C`$, with $`|A_1|=|A_2|=\mathrm{}`$ and $`|C|<|\mathrm{}|`$. (We wish to show there exist $`\gamma A_1`$ and $`fF`$, such that $`\gamma fA_2`$; this establishes that $`\mathrm{\Gamma }`$ has only one end.) Because $`G/H`$ is connected at $`\mathrm{}`$, there is a compact subset $`𝒞^{}`$ of $`G/H`$, containing $`C𝒞`$, such that $`(G/H)𝒞^{}`$ is connected. Because $`C𝒞𝒞^{}`$, we have

$$(G/H)𝒞^{}=(\mathrm{\Gamma }𝒞)𝒞^{}A_1𝒞A_2𝒞.$$

Because $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H`$, we know $`A_1𝒞`$ and $`A_2𝒞`$ are closed (and neither is contained in $`𝒞^{}`$), so connectivity implies that $`A_1𝒞A_2𝒞\mathrm{}`$: there exist $`\gamma A_1`$ and $`\gamma ^{}A_2`$, such that $`\gamma 𝒞\gamma ^{}𝒞\mathrm{}`$. Let $`f=\gamma ^1\gamma ^{}`$; then $`\gamma A_1`$, $`\gamma f=\gamma ^{}A_2`$, and

$$𝒞f𝒞=\gamma ^1(\gamma 𝒞\gamma ^{}𝒞)\mathrm{},$$

so $`fF_0F`$, as desired. ∎

### 4C. Walls of $`A^+`$ and a lower bound on $`d(H)`$

###### 4.10 Proposition.

Assume $`\text{-rank}G=2`$. Let

* $`L_1`$ and $`L_2`$ be the two walls of $`A^+`$, and
* $`\mathrm{\Gamma }`$ be a crystallographic group for $`G/H`$.

If $`H`$ is not compact, then there exists $`k\{1,2\}`$, such that, for every compact subset $`C`$ of $`A`$, the intersection $`\mu (\mathrm{\Gamma })L_kC`$ is finite.

###### Proof (cf. Figure 4.1).

Suppose there is a compact subset $`C`$ of $`A`$, such that each of $`\mu (\mathrm{\Gamma })L_1C`$ and $`\mu (\mathrm{\Gamma })L_2C`$ is infinite. (This will lead to a contradiction.) Let $`F`$ be a (symmetric) finite generating set for $`\mathrm{\Gamma }`$ (see 4.9). We may assume $`C`$ is so large that $`\mu (\gamma F)\mu (\gamma )C`$ for every $`\gamma \mathrm{\Gamma }`$ (see 2.12). We may also assume that $`C`$ is convex and symmetric.

Because $`\mathrm{\Gamma }`$ acts properly on $`G/H`$, there is a compact subset $`𝒞`$ of $`A`$, such that $`\mu (H)C\mu (\mathrm{\Gamma })𝒞`$ (see 2.17). Furthermore, we may assume that $`\mu (L_1)C\mu (L_2)C𝒞`$.

Let

* $`M=_{\gamma \mathrm{\Gamma }}\mu (\gamma )C𝒞`$,
* $`M_1`$ be the union of all the connected components of $`M`$ that contain a point of $`L_1`$, and
* $`A_1=\mathrm{\Gamma }\mu ^1(M_1)`$.

Then $`A_1`$ is infinite (because $`\mu (\mathrm{\Gamma })L_1C`$ is infinite). Also, for any $`\gamma A_1`$ and $`fF`$, we have $`\mu (\gamma f)\mu (\gamma )C`$, so $`\gamma fA_1\mu ^1(𝒞)`$. Since $`\mathrm{\Gamma }`$ has only one end (see 4.9), this implies $`\mathrm{\Gamma }A_1`$ is finite (see 4.4). Because $`\mu (\mathrm{\Gamma })L_2C`$ is infinite, we conclude that $`M_1L_2\mathrm{}`$.

Because $`\mu (H)`$ separates $`L_1`$ from $`L_2`$, and every connected component of $`M_1`$ contains a point of $`L_1`$, we conclude that $`\mu (H)M_1\mathrm{}`$. This contradicts the fact that $`\mu (H)C\mu (\mathrm{\Gamma })𝒞`$. ∎

###### 4.11 Corollary.

Assume $`\text{-rank}G=2`$. Let

1. $`L_1`$ and $`L_2`$ be the two walls of $`A^+`$; and
2. $`H_1`$ and $`H_2`$ be closed, connected, nontrivial subgroups of $`G`$,

such that

$$\mu (H_k)L_k$$

for $`k=1,2`$. If $`H`$ is not compact, then any crystallographic group for $`G/H`$ acts properly discontinuously on either $`G/H_1`$ or $`G/H_2`$.

###### Proof.

Suppose $`\mathrm{\Gamma }`$ acts properly discontinuously on neither $`G/H_1`$ nor $`G/H_2`$. (This will lead to a contradiction.) From Proposition 2.17, we know there is a compact subset $`C`$ of $`A`$, such that each of $`\mu (\mathrm{\Gamma })\mu (H_1)C`$ and $`\mu (\mathrm{\Gamma })\mu (H_2)C`$ is infinite. Then, since $`\mu (H_k)L_k`$, we may assume (by enlarging $`C`$) that each of $`\mu (\mathrm{\Gamma })L_1C`$ and $`\mu (\mathrm{\Gamma })L_2C`$ is infinite. This contradicts the conclusion of Proposition 4.10. ∎

###### 4.12 Corollary.

Assume $`\text{-rank}G=2`$. Let

1. $`L_1`$ and $`L_2`$ be the two walls of $`A^+`$; and
2. $`H_1`$ and $`H_2`$ be closed, connected, nontrivial subgroups of $`G`$;

such that

$$\mu (H_k)L_k$$

for $`k=1,2`$. If $`G/H`$ has a tessellation, and $`H`$ is not compact, then

$$d(H)\mathrm{min}\{d(H_1),d(H_2)\}.$$

###### Proof.

The desired conclusion is obtained by combining Corollary 4.11 with Theorem 4.1(1). ∎

###### 4.13 Remark.

For $`G=\mathrm{SL}(3,)`$, there does not exist a connected subgroup $`H_k`$, such that $`\mu (H_k)L_k`$ (cf. 6.3). Thus, Corollary 4.12 does not provide a lower bound on $`d(H)`$ in this case.

## 5. One-dimensional subgroups

Although the following conjecture does not seem to have been stated previously in the literature, it is perhaps implicit in \[OW3\].

###### 5.1 Conjecture.

If $`d(H)=1`$, then $`G/H`$ does not have a tessellation.

In this section, we establish that the conjecture is valid in two cases: if either $`\text{-rank}G2`$ (see 5.14) or $`G`$ is almost simple (see 5.30). Each of these illustrates a general theorem: for groups of real rank two, the conjecture follows from a theorem of Y. Benoist and F. Labourie that is based on differential geometry; H. Oh and D. Witte observed that, for simple groups, the conjecture follows from a theorem of G. A. Margulis that is based on unitary representation theory.

The following example is the only case of Conjecture 5.1 that is needed in later sections. (It is used in the proof of Theorem 1.8.) Because $`\text{-rank}\left(\mathrm{SL}(3,𝔽)\right)=2`$ and $`\mathrm{SL}(3,𝔽)`$ is almost simple, this example is covered both by the theorem of Benoist-Labourie and by the theorem of Margulis, but it would be interesting to have an easy proof.

###### 5.2 Proposition ((see 5.14 or 5.30)).

Assume $`G=\mathrm{SL}(3,𝔽)`$, for $`𝔽=`$, $``$, or $``$, and let

(5.3)
$$H_1=\left\{\left(\begin{array}{ccc}e^t& 0& 0\\ 0& e^t& 0\\ 0& 0& e^{2t}\end{array}\right)|t\right\}G.$$

Then $`G/H_1`$ does not have a tessellation.

Let us begin with an easy observation.

###### 5.4 Lemma.

If $`d(H)=1`$ and $`\text{-rank}G=1`$, then $`G/H`$ does not have a tessellation.

###### Proof.

We may assume $`HAN`$ (see 3.5). From (2.16), we know that $`H`$ is a Cartan-decomposition subgroup, so Lemma 2.9 implies that $`G/H`$ must be compact; thus, the trivial group $`e`$ is a crystallographic group for $`G/H`$. However, since

$$d(G)=dimA+dimN1+1>1=d(H)$$

(see 1.13), and $`e`$ acts properly discontinuously on $`G/G`$, this contradicts Theorem 4.1(1). ∎

###### 5.5 Remark.

The proof of Lemma 5.4 shows that the dimension of every connected, cocompact subgroup of $`G`$ is at least $`d(G)`$. This is a result of M. Goto and H.–C. Wang \[GtW, (1.2), p. 263\].

### 5A. The geometric method of Y. Benoist and F. Labourie

###### 5.6 Theorem ((Benoist-Labourie \[BL, Cor. 3\])).

If $`H`$ is reductive and contains an element of $`A`$ in its center, then $`G/H`$ does not have a tessellation.

To illustrate the idea behind Theorem 5.6, we give a direct proof of the following special case (under an additional technical assumption (see 5.11)), which is sufficient for our needs. (Note that Condition (5.11) is satisfied for the subgroup $`H_1`$ of Proposition 5.2.) Benoist and Labourie prove the general case by using a slightly different 1-form in place of the form $`\omega `$ that we define in Step 5A.

###### 5.7 Corollary ((Benoist-Labourie)).

If $`H`$ is a one-dimensional subgroup of $`A`$, then $`G/H`$ does not have a tessellation.

###### Proof \[BL\].

Suppose $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$. (This will lead to a contradiction.) By passing to finite-index subgroups, we may assume that $`G`$ and $`H`$ are connected.

We will construct a volume form $`\nu `$ on $`\mathrm{\Gamma }\backslash G/H`$ that is exact: $`\nu =d\xi `$. The integral of $`\nu `$ over $`\mathrm{\Gamma }\backslash G/H`$ is the volume of $`\mathrm{\Gamma }\backslash G/H`$, which is obviously not $`0`$, but Stokes’ Theorem implies that the integral of any exact form over a closed manifold is $`0`$. This is a contradiction.

Step 1 . Construction of a certain $`2`$-form $`\stackrel{˘}{\mathrm{\Omega }}`$ on $`\mathrm{\Gamma }\backslash G/H`$.  Let $`𝔪=𝔥^{}`$ be the orthogonal complement to $`𝔥`$, under the Killing form. Then $`𝔪`$ is an $`(\mathrm{Ad}_GH)`$-invariant subspace of $`𝔤`$, such that $`𝔤=𝔥+𝔪`$ and $`𝔥𝔪=0`$. Let $`\omega _e:𝔤𝔥`$ be the projection with kernel $`𝔪`$, and let $`\omega `$ be the corresponding left-invariant $`𝔥`$-valued 1-form on $`G`$.

The space $`G`$ is a homogeneous principal $`H`$-bundle over $`G/H`$. It is well known \[KN, Thm. II.11.1, p. 103\] that $`\omega `$ is the connection form of a $`G`$-invariant connection on this bundle, and that the curvature form $`\mathrm{\Omega }`$ of this connection is given by

(5.8) $`\mathrm{\Omega }(X,Y)=\frac{1}{2}\omega \left([X,Y]\right)`$ for $`X,Y𝔪`$.
Also, because $`H`$ is abelian, the Structure Equation \[KN, Thm. II.5.2, p. 77\] implies

(5.9) $`\mathrm{\Omega }(X,Y)=d\omega (X,Y)`$ for $`X,Y𝔤`$.
By identifying the $`1`$-dimensional Lie algebra $`𝔥`$ with $``$, we may think of $`\omega `$ and $`\mathrm{\Omega }`$ as ordinary (that is, $``$-valued) differential forms.

Because $`\omega `$ and (hence) $`\mathrm{\Omega }`$ are left-invariant, they determine well-defined forms $`\overline{\omega }`$ and $`\overline{\mathrm{\Omega }}`$ on $`\mathrm{\Gamma }\backslash G`$. (Note that $`\overline{\omega }`$ is a connection form on the principal $`H`$-bundle $`\mathrm{\Gamma }\backslash G`$ over $`\mathrm{\Gamma }\backslash G/H`$, and the curvature form of this connection is $`\overline{\mathrm{\Omega }}`$.) Because $`H`$ is abelian, we have $`\overline{\mathrm{\Omega }}_{gh}=\overline{\mathrm{\Omega }}_g`$ for all $`hH`$ (cf. \[KN, Prop. II.5.1(c), p. 76\]), so the horizontal form $`\overline{\mathrm{\Omega }}`$ determines a well-defined form $`\stackrel{˘}{\mathrm{\Omega }}`$ on the base space $`\mathrm{\Gamma }\backslash G/H`$.

Step 2 . Construction, for a certain $`s`$, of a certain $`s`$-form $`\stackrel{˘}{\mu }`$ on $`\mathrm{\Gamma }\backslash G/H`$.  Identifying $`𝔥`$ with $``$ provides an ordering on the weights of $`𝔥`$. Let

* $`𝔪_0=𝔠_𝔤(𝔥)𝔪`$ be the $`0`$ weight space of $`\mathrm{ad}_𝔤𝔥`$ on $`𝔪`$;
* $`𝔪_+`$ be the sum of the positive weight spaces of $`\mathrm{ad}_𝔤𝔥`$ on $`𝔪`$;
* $`𝔪_{}`$ be the sum of the negative weight spaces of $`\mathrm{ad}_𝔤𝔥`$ on $`𝔪`$;
* $`s=dim𝔪_0`$; and
* $`\mu `$ be a nontrivial left-invariant $`s`$-form on $`G`$, such that

(5.10)
$$\mu (𝔥𝔪_+𝔪_{},,\mathrm{},)=0.$$

Because $`\mu `$ is left-invariant, and $`\mu (𝔥,,\mathrm{},)=0`$, the form $`\mu `$ determines a well-defined differential form $`\stackrel{˘}{\mu }`$ on $`\mathrm{\Gamma }\backslash G/H`$. (We remark that, because $`𝔤=𝔥𝔪_+𝔪_0𝔪_{}`$, Condition (5.10) implies that the form $`\mu `$ is unique up to a scalar multiple.)

Step 3 . For a certain $`r`$, the wedge product $`\stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^r`$ is a volume form on $`\mathrm{\Gamma }\backslash G/H`$.  Let $`r=dim𝔪_+`$. It suffices to show that the restriction of $`\mathrm{\Omega }`$ to $`𝔪_+𝔪_{}`$ is a symplectic form. It is obviously skew, so we need only show that it is nondegenerate. Thus, letting

$`𝔷`$ $`=\{X𝔪_+𝔪_{}\mathrm{\Omega }(X,𝔪_+𝔪_{})=0)`$
$`=\{X𝔪_+𝔪_{}\omega \left([X,𝔪_+𝔪_{}]\right)=0\}`$ (5.8)
$`=\{X𝔪_+𝔪_{}[X,𝔪_+𝔪_{}]𝔪\}`$ (definition of $`\omega `$)
$`=\{X𝔪_+𝔪_{}𝔥[X,𝔪_+𝔪_{}]_{\text{Killing}}=0\}`$ $`\text{(}𝔪=𝔥^{}\text{)},`$
we wish to show $`𝔷=0`$.

There is no harm in passing to the complexification $`𝔤^{}`$ of $`𝔤`$. Let $`𝔱^{}`$ be a Cartan subalgebra of $`𝔤^{}`$ that contains $`𝔥`$. Because $`𝔱^{}`$ preserves the Killing form, centralizes $`𝔥`$, and normalizes $`𝔪_+𝔪_{}`$, we know that $`𝔷^{}`$ is $`(\mathrm{ad}_𝔤^{}𝔱^{})`$-invariant; thus, $`𝔷^{}`$ is a sum of root spaces.

Suppose there exists a nonzero element $`X`$ of $`𝔷^{}`$, such that $`X`$ belongs to some root space $`𝔤_\alpha ^{}`$. (This will lead to a contradiction.) There exists $`Y𝔤_\alpha ^{}`$, such that

$$t[X,Y]_{\text{Killing}}=\alpha (t)$$

for all $`t𝔱^{}`$ \[Hm2, Prop. 8.3(c), p. 37\]. Because $`𝔤_\alpha ^{}𝔪_+𝔪_{}`$, we have $`\alpha (𝔥)0`$. Then $`(\alpha )(𝔥)0`$, so $`𝔤_\alpha ^{}𝔪_+𝔪_{}`$. We now know that $`X𝔷^{}`$ and $`Y𝔪_+𝔪_{}`$, so $`𝔥[X,Y]_{\text{Killing}}=0`$. We therefore conclude, from the definition of $`Y`$, that $`\alpha (𝔥)=0`$. This is a contradiction.

Step 4 . The form $`\stackrel{˘}{\mathrm{\Omega }}`$ is exact: we may write $`\stackrel{˘}{\mathrm{\Omega }}=d\stackrel{˘}{\varphi }`$.  Let $`\stackrel{˘}{\omega }_0`$ be the connection form of a flat connection on $`\mathrm{\Gamma }\backslash G`$ over $`\mathrm{\Gamma }\backslash G/H`$. (Since the principal bundle is trivial (see 3.9), it is obvious that there is a flat connection.) For any vector field $`X`$ on $`\mathrm{\Gamma }\backslash G/H`$, let $`\stackrel{~}{X}`$ be the lift of $`X`$ to a vector field on $`\mathrm{\Gamma }\backslash G`$ that is horizontal with respect to the flat connection $`\stackrel{˘}{\omega }_0`$.

Since $`H`$ is abelian, there is a well-defined 1-form $`\stackrel{˘}{\varphi }`$ on $`\mathrm{\Gamma }\backslash G/H`$ given by

$$\stackrel{˘}{\varphi }(X)=\overline{\omega }(\stackrel{~}{X}).$$

Then

$`d\stackrel{˘}{\varphi }(X,Y)`$ $`={\displaystyle \frac{1}{2}}\left(X\left(\stackrel{˘}{\varphi }(Y)\right)Y\left(\stackrel{˘}{\varphi }(X)\right)\stackrel{˘}{\varphi }\left([X,Y]\right)\right)`$ (definition of $`d`$)
$`={\displaystyle \frac{1}{2}}\left(\stackrel{~}{X}\left(\overline{\omega }(\stackrel{~}{Y})\right)\stackrel{~}{Y}\left(\overline{\omega }(\stackrel{~}{X})\right)\overline{\omega }\left(\stackrel{~}{[X,Y]}\right)\right)`$ (definition of $`\stackrel{˘}{\varphi }`$)
$`={\displaystyle \frac{1}{2}}\left(\stackrel{~}{X}\left(\overline{\omega }(\stackrel{~}{Y})\right)\stackrel{~}{Y}\left(\overline{\omega }(\stackrel{~}{X})\right)\overline{\omega }\left([\stackrel{~}{X},\stackrel{~}{Y}]\right)\right)`$ ($`\stackrel{˘}{\omega }_0`$ is flat)
$`=d\overline{\omega }(\stackrel{~}{X},\stackrel{~}{Y})`$ (definition of $`d`$)
$`=\overline{\mathrm{\Omega }}(\stackrel{~}{X},\stackrel{~}{Y})`$ (5.9)
$`=\stackrel{˘}{\mathrm{\Omega }}(X,Y)`$ $`\text{(definition of }\stackrel{˘}{\mathrm{\Omega }}\text{)}.`$
Assumption. For simplicity, assume that

(5.11) every hyperbolic element of the center of $`𝔠_𝔤(𝔥)`$ is contained in $`𝔥`$.
Step 5 . We have $`d\stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^{(r1)}=0`$.  Let

* $`X_1,\mathrm{},X_r`$ be a basis of $`𝔪_+`$, and
* $`Y_1,\mathrm{},Y_r`$ be the dual basis of $`𝔪_{}`$, with respect to the symplectic form $`\mathrm{\Omega }`$ on $`𝔪_+𝔪_{}`$.

Thus, $`\mathrm{\Omega }_e(X_j,Y_k)=\delta _{j,k}`$.

Let $`Z_1,\mathrm{},Z_s`$ be a basis of $`𝔪_0`$, write

$$[X_j,Y_k]=\underset{\mathrm{}}{}a_{j,k}^{\mathrm{}}Z_{\mathrm{}}(mod𝔥+𝔪_++𝔪_{}),$$

and define

$$W=\underset{j}{}[X_j,Y_j]=\underset{j,\mathrm{}}{}a_{j,j}^{\mathrm{}}Z_{\mathrm{}}.$$

Substep 5.1 . $`W`$ is independent of the choice of the basis $`\{X_j\}`$ of $`𝔪_+`$ (with the understanding that $`\{Y_j\}`$ must be the dual basis of $`𝔪_{}`$).  Let

(5.12)
$$X_j^{}=\{\begin{array}{cc}aX_1+bX_2\hfill & \text{if }j=1\hfill \\ X_j\hfill & \text{if }j2\hfill \end{array}$$

for some $`a,b`$ with $`a0`$. Then

$$Y_j^{}=\{\begin{array}{cc}(1/a)Y_1\hfill & \text{if }j=1\hfill \\ Y_2(b/a)Y_1\hfill & \text{if }j=2\hfill \\ Y_j\hfill & \text{if }j3,\hfill \end{array}$$

so

$`W^{}`$ $`=[X_1^{},Y_1^{}]+[X_2^{},Y_2^{}]+{\displaystyle \underset{j3}{}}[X_j^{},Y_j^{}]`$
$`=\left([X_1,Y_1]+(b/a)[X_2,Y_1]\right)+\left([X_2,Y_2](b/a)[X_2,Y_1]\right)+{\displaystyle \underset{j3}{}}[X_j,Y_j]`$
$`={\displaystyle \underset{j}{}}[X_j,Y_j]`$
$`=W.`$
Since $`X_1,\mathrm{},X_r`$ can be transformed into any other basis of $`𝔪_+`$ by a sequence of elementary operations as in (5.12), we conclude that $`W`$ is independent of the choice of basis, as desired.

Substep 5.2 . We have $`W𝔥`$.  Substep 5A implies that $`W`$ is centralized by $`C_G(H)`$, so $`W`$ is in the center of $`𝔠_𝔤(𝔥)`$.

Let $`\sigma `$ be a Cartan involution of $`G`$ with $`\sigma (h)=h`$ for $`h𝔥`$. Substep 5A implies $`\sigma (W)=W`$. (From Substep 5A, we see that $`W`$ depends only on $`𝔥`$ and the chosen identification of $`𝔥`$ with $``$; $`\sigma `$ reverses the choice of identification.)

Thus, $`W`$ is a hyperbolic element of the center of $`𝔠_𝔤(𝔥)`$. By Assumption 5.11, this implies $`W𝔥`$, as desired.

Substep 5.3 . Completion of Step 5A.  Let

$$X_1^{},\mathrm{},X_r^{},Y_1^{},\mathrm{},Y_r^{},Z_1^{},\mathrm{},Z_s^{}$$

be the basis of $`(𝔤/𝔥)^{}`$ dual to

$$X_1,\mathrm{},X_r,Y_1,\mathrm{},Y_r,Z_1,\mathrm{},Z_s.$$

We may assume $`\mu =Z_1^{}\mathrm{}Z_s^{}`$. Then, because $`[𝔪_0+𝔪_{},𝔪_{}]𝔪_{}`$ and $`[𝔪_0+𝔪_+,𝔪_+]𝔪_+`$, we have

$$dZ_{\mathrm{}}^{}(𝔪_0+𝔪_{},𝔪_{})=0=dZ_{\mathrm{}}^{}(𝔪_0+𝔪_+,𝔪_+),$$

so

$$dZ_{\mathrm{}}^{}=\underset{j,k}{}a_{j,k}^{\mathrm{}}X_j^{}Y_k^{}(mod𝔪_0^{}𝔪_0^{}).$$

Therefore

$`(s+1)d\stackrel{˘}{\mu }`$ $`={\displaystyle \underset{\mathrm{}}{}}(1)^\mathrm{}1dZ_{\mathrm{}}^{}Z_1^{}\mathrm{}\widehat{Z_{\mathrm{}}^{}}\mathrm{}Z_s^{}`$
$`={\displaystyle \underset{j,k,\mathrm{}}{}}(1)^{\mathrm{}}a_{j,k}^{\mathrm{}}X_j^{}Y_k^{}Z_1^{}\mathrm{}\widehat{Z_{\mathrm{}}^{}}\mathrm{}Z_s^{}.`$
From the choice of $`Y_1,\mathrm{},Y_r`$, we have $`\mathrm{\Omega }=2_{j=1}^rX_j^{}Y_j^{}`$, so

$$\mathrm{\Omega }^{(r1)}=2^{r1}(r1)!\underset{j=1}{\overset{r}{}}X_1^{}Y_1^{}\mathrm{}\widehat{X_j^{}}\widehat{Y_j^{}}\mathrm{}X_r^{}Y_r^{}$$

and

$$\mathrm{\Omega }^r=2^rr!X_1^{}Y_1^{}\mathrm{}X_r^{}Y_r^{}.$$

Hence

$`(s+1)d\stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^{(r1)}`$ $`={\displaystyle \frac{1}{2r}}{\displaystyle \underset{j,\mathrm{}}{}}(1)^{\mathrm{}}a_{j,j}^{\mathrm{}}Z_1^{}\mathrm{}\widehat{Z_{\mathrm{}}^{}}\mathrm{}Z_s^{}\stackrel{˘}{\mathrm{\Omega }}^r`$
$`={\displaystyle \frac{s+2r}{2r}}\iota _W(\stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^r).`$
Since $`W𝔥`$, we have $`\iota _W(\mu _e\mathrm{\Omega }_e^r)=0`$, so the desired conclusion follows.

Step 6 . $`\stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^r`$ is exact.  We have

$`d\left(\stackrel{˘}{\mu }\stackrel{˘}{\varphi }\stackrel{˘}{\mathrm{\Omega }}^{(r1)}\right)`$ $`=\pm d\stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^{(r1)}\stackrel{˘}{\varphi }\pm \stackrel{˘}{\mu }d\stackrel{˘}{\varphi }\stackrel{˘}{\mathrm{\Omega }}^{(r1)}\pm \stackrel{˘}{\mu }\stackrel{˘}{\varphi }d\stackrel{˘}{\mathrm{\Omega }}^{(r1)}`$
$`=0\stackrel{˘}{\varphi }\pm \stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}\stackrel{˘}{\mathrm{\Omega }}^{(r1)}\pm \stackrel{˘}{\mu }\stackrel{˘}{\varphi }0`$
$`=\pm \stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^r.`$
Step 7 . A contradiction.  From Step 5A, we know that

$$_{\mathrm{\Gamma }\backslash G/H}\stackrel{˘}{\mu }\stackrel{˘}{\mathrm{\Omega }}^r0.$$

On the other hand, Step 5A implies that this integral is zero. This is a contradiction. ∎

###### 5.13 Notation (\[Ben, p. 320\]).

Let $`\tau `$ be the opposition involution in $`A^+`$; that is, for $`aA^+`$, $`\tau (a)=\mu (a^1)`$ is the unique element of $`A^+`$ that is conjugate (under an element of the Weyl group) to $`a^1`$. Thus, for all $`hG`$, we have

$$\mu (h^1)=\tau \left(\mu (h)\right).$$

See (6.1) for an explicit description of the opposition involution in $`G=\mathrm{SL}(3,)`$. For some groups, such as $`G=\mathrm{SO}(2,n)`$, we have $`\mu (h^1)=\mu (h)`$ for all $`hG`$ (see 8.14); in such a case, the opposition involution is simply the identity map on $`A^+`$.

###### 5.14 Corollary.

If $`d(H)=1`$ and $`\text{-rank}G2`$, then $`G/H`$ does not have a tessellation.

###### Proof.

Suppose $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$. (This will lead to a contradiction.)

From (5.4), we know $`\text{-rank}G=2`$. Let $`L_1`$ and $`L_2`$ be the two walls of $`A^+`$ and, for $`k\{1,2\}`$, let $`H_k=L_kL_k^1`$. Because $`L_k`$ is a ray (that is, a one-parameter semigroup), it is clear that $`H_k`$ is a subgroup of $`A`$.

From Proposition 4.10, we know that there is some $`k\{1,2\}`$, such that

$`\mu (\mathrm{\Gamma })L_kC`$ is finite,
for every compact subset $`C`$ of $`A`$. Since $`\mathrm{\Gamma }=\mathrm{\Gamma }^1`$, we have $`\tau \left(\mu (\mathrm{\Gamma })\right)=\mu (\mathrm{\Gamma })`$, so this implies that

$`\mu (\mathrm{\Gamma })\tau (L_k)C`$ is finite,
for every compact subset $`C`$ of $`A`$. Also, because $`L_kA^+`$, we have $`\mu (L_k)=L_k`$, so

$$\mu (H_k)=\mu (L_kL_k^1)=\mu (L_k)\tau \left(\mu (L_k)\right)=L_k\tau (L_k).$$

Therefore

$`\mu (\mathrm{\Gamma })\mu (H_k)C=(\mu (\mathrm{\Gamma })L_kC)(\mu (\mathrm{\Gamma })\tau (L_k)C)`$ is finite,
for every compact subset $`C`$ of $`A`$. Hence, Corollary 2.17 implies that $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H_k`$. Then, because $`d(H)=1=d(H_k)`$ (see 1.14), Theorem 4.1(2) implies that $`G/H_k`$ has a tessellation. This contradicts Corollary 5.7. ∎

### 5B. The representation-theoretic method of G. A. Margulis

###### 5.15 Definition (\[Mr2, Defn. 2.2, Rmk. 2.2\]).

The subgroup $`H`$ is *tempered* in $`G`$ if there exists a (positive) function $`fL^1(H)`$ (with respect to a left-invariant Haar measure on $`H`$), such that, for every unitary representation $`\pi `$ of $`G`$, either

* $`|\pi (h)\varphi \psi |f(h)\varphi \psi `$ for all $`hH`$ and all $`K`$-fixed vectors $`\varphi `$ and $`\psi `$; or
* some nonzero vector is fixed by every element of $`\pi (G)`$.

For many examples of tempered subgroups of simple Lie groups, see \[Oh\].

###### 5.16 Theorem ((Margulis \[Mr2, Thm. 3.1\])).

If $`H`$ is noncompact and tempered, then $`G/H`$ does not have a tessellation.

###### Proof \[Mr2\].

Suppose $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$. (This will lead to a contradiction.) To simplify the notation somewhat (and because this is the only case we need), let us assume $`H=\{h^t\}`$ is a one-parameter subgroup of $`G`$.

Because $`G`$ and $`\mathrm{\Gamma }`$ are unimodular (recall that $`G`$ is semisimple (see 1.3) and $`\mathrm{\Gamma }`$ is discrete), there is a $`G`$-invariant measure (in fact, a $`G`$-invariant volume form) on the homogeneous space $`\mathrm{\Gamma }\backslash G`$ \[Rag, Lem 1.4, p. 18\]. Thus, the natural representation $`\pi `$ of $`G`$ on $`L^2(\mathrm{\Gamma }\backslash G)`$, defined by

$`\left(\pi (g)\varphi \right)(x)=\varphi (xg^1)`$, for $`\varphi L^2(\mathrm{\Gamma }\backslash G)`$, $`gG`$, $`x\mathrm{\Gamma }\backslash G`$,
is unitary.

Because $`H`$ is noncompact, and acts properly on $`\mathrm{\Gamma }\backslash G`$, we know that any compact subset of $`\mathrm{\Gamma }\backslash G`$ has infinitely many pairwise-disjoint translates (all of the same measure), so we see that

(5.17)
$$\mathrm{\Gamma }\backslash G\text{ is not compact, and has infinite volume}.$$

Therefore, $`\pi `$ has no nonzero $`G`$-invariant vectors, so, because $`H`$ is tempered, we know that there is some $`fL^1()`$, such that

(5.18)
$$f(t)\varphi _2\psi _2\left|_{\mathrm{\Gamma }\backslash G}\varphi (xh^t)\psi (x)𝑑x\right|$$

for all $`t`$ and all $`K`$-invariant $`\varphi ,\psi L^2(\mathrm{\Gamma }\backslash G)`$.

Because $`\mathrm{\Gamma }\backslash G/H`$ is compact, there is a compact subset $`C`$ of $`G`$, such that $`\mathrm{\Gamma }CH=G`$; let $`\overline{C}`$ be the image of $`C`$ in $`\mathrm{\Gamma }\backslash G`$. From the choice of $`C`$, we know, for each $`x\mathrm{\Gamma }\backslash G`$, that there is some $`T_x`$, such that

(5.19)
$$xh^{T_x}\overline{C}.$$

Because $`_{t=0}^1\overline{C}h^tK`$ is a compact subset of $`\mathrm{\Gamma }\backslash G`$, there is a positive, continuous function $`\varphi `$ on $`\mathrm{\Gamma }\backslash G`$ with compact support, such that

(5.20) $`\varphi (xh^t)1`$ for all $`x\overline{C}`$ and all $`t[0,1]`$,
and, by averaging over $`K`$, we may assume that $`\varphi `$ is $`K`$-invariant.

Fix some large $`T^+`$. Because $`_{t=(T+1)}^{T+1}\overline{C}h^tK`$ is compact, and $`\mathrm{\Gamma }\backslash G`$ has infinite volume (see 5.17), there is some $`K`$-invariant continuous function $`\psi _T`$ on $`\mathrm{\Gamma }\backslash G`$, such that $`\psi _T_2=1`$,

(5.21)
$$0\psi _T(x)1\text{ for all }x\mathrm{\Gamma }\backslash G,$$

and

(5.22)
$$|T_x|>T+1\text{ for all }x\text{ in the support of }\psi _T.$$

We have

$`\varphi _2{\displaystyle _{|t|>T}}f(t)𝑑t`$ $`{\displaystyle _{\mathrm{\Gamma }\backslash G}}{\displaystyle _{|t|>T}}\varphi (xh^t)\psi _T(x)𝑑t𝑑x`$ ((5.18) and $`\psi _T_2=1`$)
$`{\displaystyle _{\mathrm{\Gamma }\backslash G}}{\displaystyle _0^1}\varphi (xh^{T_x+t})\psi _T(x)𝑑t𝑑x`$ (5.22)
$`{\displaystyle _{\mathrm{\Gamma }\backslash G}}\psi _T(x)𝑑x`$ ((5.19) and (5.20))
$`1`$ $`\text{((}\text{5.21}\text{) and }\psi _T_2=1\text{)}.`$
However, because $`fL^1()`$, we know that $`lim_T\mathrm{}_{|t|>T}f(t)𝑑t=0`$. This is a contradiction. ∎

We state the following well-known result of representation theory without proof. As is explained in \[KS, §3, p. 140\], it can be obtained by combining work of R. Howe \[How, Cor. 7.2 and §7\] and M. Cowling \[Cow, Thm. 2.4.2\]. (The assumption that $`\text{-rank}G2`$ can be relaxed: it suffices to assume that $`G`$ is not locally isomorphic to $`\mathrm{SO}(1,n)`$ or $`\mathrm{SU}(1,n)`$.)

Fix any matrix norm $``$ on $`G`$; for example, we may let $`g=\mathrm{max}_{j,k}|g_{j,k}|`$.

###### 5.23 Theorem ((Cowling, Howe)).

If $`G`$ is almost simple, and $`\text{-rank}G2`$, then there are constants $`C>0`$ and $`p>0`$ such that, for every unitary representation $`\pi `$ of $`G`$, either

1. $`|\pi (g)\varphi \psi |C\varphi \psi g^p`$ for all $`gG`$ and all $`\pi (K)`$-fixed vectors $`\varphi `$ and $`\psi `$; or
2. some nonzero vector is fixed by every element of $`\pi (G)`$.

Although we cannot prove Theorem 5.23 here, we present an elementary proof of the following related result, which, unfortunately, is qualitative, rather than quantitative. On the other hand, this simple result applies to all vectors, not only the $`K`$-fixed vectors, and it applies to all semisimple groups, including $`\mathrm{SO}(1,n)`$ and $`\mathrm{SU}(1,n)`$. It was first proved by R. Howe and C. Moore \[HM, Thm. 5.1\] and (independently) R. Zimmer \[Zm1, Thm. 5.2\].

###### 5.24 Theorem (\[Zm2, Thm. 2.2.20, p. 23\]).

If

* $`G`$ is connected and almost simple;
* $`\pi `$ is a unitary representation of $`G`$ on a Hilbert space $``$, such that no nonzero vector is fixed by $`\pi (G)`$; and
* $`\{g_j\}`$ is a sequence of elements of $`G`$, such that $`g_j\mathrm{}`$,

then $`\pi (g_j)\varphi \psi 0`$, for every $`\varphi ,\psi `$.

###### Proof (Ellis-Nerurkar \[EN\]).

Case 1 . Assume $`\{g_j\}A`$.  By passing to a subsequence, we may assume $`\pi (g_j)`$ converges weakly, to some operator $`E`$; that is,

$$\pi (g_j)\varphi \psi E\varphi \psi \text{ for every }\varphi ,\psi .$$

Let

(5.25)
$$U=\{vGg_j^1vg_je\}\text{ and }U^{}=\{uGg_jug_j^1e\}.$$

For $`uU^{}`$, we have

$$E\pi (u)\varphi \psi =lim\pi (g_ju)\varphi \psi =lim\pi (g_jug_j^1)\pi (g_j)\varphi \psi =lim\pi (g_j)\varphi \psi =E\varphi \psi ,$$

so $`E\pi (u)=E`$. Therefore $`E\left((^U^{})^{}\right)=0`$.

We have

$$E^{}\varphi \psi =\varphi E\psi =lim\varphi \pi (g_j)\psi =lim\pi (g_j^1)\varphi \psi ,$$

so the same argument, with $`E^{}`$ in the place of $`E`$ and $`g_j^1`$ in the place of $`g_j`$, shows that $`E^{}\left((^U)^{}\right)=0`$.

Because $`\pi `$ is unitary, we know that $`\pi (g_j)`$ is normal (that is, commutes with its adjoint) for every $`j`$; thus, the limit $`E`$ is also normal: we have $`E^{}E=EE^{}`$. Therefore

$$E\varphi ^2=E\varphi E\varphi =(E^{}E)\varphi \varphi =(EE^{})\varphi \varphi =E^{}\varphi E^{}\varphi =E^{}\varphi ^2,$$

so $`\mathrm{ker}E=\mathrm{ker}E^{}`$.

Thus,

$$\mathrm{ker}E=\mathrm{ker}E+\mathrm{ker}E^{}(^U^{})^{}+(^U)^{}=(^U^{}^U)^{}=(^{U,U^{}})^{}.$$

By passing to a subsequence of $`\{g_j\}`$, we may assume $`U,U^{}=G`$ (see 5.27). Then $`^{U,U^{}}=^G=0`$, so $`\mathrm{ker}E0^{}=`$. Hence, for all $`\varphi ,\psi `$, we have

$$lim\pi (g_j)\varphi \psi =E\varphi \psi =0\psi =0,$$

as desired.

Case 2 . The general case.  From the Cartan Decomposition $`G=KAK`$, we may write $`g_j=c_ja_jc_j^{}`$, with $`c_j,c_j^{}K`$ and $`a_jA`$. Because $`K`$ is compact, we may assume, by passing to a subsequence, that $`\{c_j\}`$ and $`\{c_j^{}\}`$ converge: say, $`c_jc`$ and $`c_j^{}c^{}`$. Then

$`lim\pi (g_j)\varphi \psi `$ $`=lim\pi (c_ja_jc_j^{})\varphi \psi `$
$`=lim\pi (a_j)\pi (c_j^{})\varphi \pi (c_j)^1\psi `$
$`=lim\pi (a_j)\left(\pi (c^{})\varphi \right)|\pi (c)^1\psi `$
$`=0,`$
by Case 5B. ∎

The following example illustrates Lemma 5.27.

###### 5.26 Example.

Let $`G=\mathrm{SL}(3,)`$, define $`H_1`$ as in Proposition 5.2, and suppose $`\{g_j\}`$ is some sequence of elements of $`H_1`$, such that $`g_j\mathrm{}`$. We may write

$$g_j=\left(\begin{array}{ccc}e^{t_j}& 0& 0\\ 0& e^{t_j}& 0\\ 0& 0& e^{2t_j}\end{array}\right),$$

where $`t_j`$. By passing to a subsequence, we may assume that either $`t_j\mathrm{}`$ or $`t_j\mathrm{}`$. If $`t_j\mathrm{}`$, then, in the notation of (5.25), we have

$$U=\left\{\left(\begin{array}{ccc}1& 0& \\ 0& 1& \\ 0& 0& 1\end{array}\right)\right\}\text{ and }U^{}=\left\{\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ & & 1\end{array}\right)\right\};$$

if $`t_j\mathrm{}`$, then $`U`$ and $`U^{}`$ are interchanged. Thus, in either case, $`𝔲`$ is the sum of two root spaces of $`𝔤`$, and $`𝔲^{}`$ is the sum of the two opposite root spaces. It is not difficult to see that $`[𝔲,𝔲^{}]`$ is the sum of $`𝔞`$ and the remaining two root spaces. Therefore, we have $`𝔲,𝔲^{}=𝔤`$, so $`U,U^{}=G`$.

###### 5.27 Lemma.

If $`G`$ and $`\{g_j\}`$ are as in Theorem 5.24, and $`\{g_j\}A`$, then, after replacing $`\{g_j\}`$ by a subsequence, we have $`U,U^{}=G`$, where $`U`$ and $`U^{}`$ are defined in (5.25).

###### Proof.

By passing to a subsequence, we may assume $`\{g_j\}`$ is contained in a single Weyl chamber, which we may take to be $`A^+`$. Then, by passing to a subsequence yet again, we may assume, for every positive real root $`\alpha `$, that either $`\alpha (g_j)\mathrm{}`$ or $`\alpha (g_j)`$ is bounded. Let

* $`\mathrm{\Phi }^+`$ be the set of positive real roots;
* $`\mathrm{\Delta }`$ be the set of positive simple real roots;
* $`\mathrm{\Psi }=\{\alpha \mathrm{\Phi }^+\alpha (g_j)\text{ is bounded}\}`$; and
* $`T=_{\psi \mathrm{\Psi }}\mathrm{ker}\psi =_{\psi \mathrm{\Psi }\mathrm{\Delta }}\mathrm{ker}\psi `$.

There is a compact subset $`C`$ of $`A`$, such that $`\{g_j\}CT`$, so, because $`g_j\mathrm{}`$, we know that $`T`$ is not trivial.

For each real root $`\alpha `$, let $`𝔫_\alpha `$ be the corresponding root subspace of $`𝔤`$. Then

$$𝔲=\underset{\alpha \mathrm{\Phi }^+\mathrm{\Psi }}{}𝔫_\alpha \text{ and }𝔲^{}=\underset{\alpha \mathrm{\Phi }^+\mathrm{\Psi }}{}𝔫_\alpha .$$

Now, for $`\alpha \mathrm{\Phi }^+`$, we have $`\alpha \mathrm{\Psi }`$ if and only if $`\alpha `$ is in the linear span of $`\mathrm{\Psi }\mathrm{\Delta }`$. Thus, we see that $`𝔲`$ is precisely the unipotent radical of the standard parabolic subalgebra $`𝔭=𝔠_𝔤(T)+𝔞+𝔫`$ corresponding to the set $`\mathrm{\Psi }\mathrm{\Delta }`$ of simple roots \[BT, 4.2, pp. 85–86\].

Similarly, $`𝔲^{}`$ is the unipotent radical of the opposite parabolic algebra $`𝔭^{}=𝔠_𝔤(T)+𝔞+𝔫^{}`$. Because $`G`$ is simple, the unipotent radicals of opposite parabolics generate $`𝔤`$ \[BT, Prop. 4.11, p. 89\], so $`U,U^{}=G`$, as desired. ∎

###### 5.28 Corollary ((of Theorem 5.23)).

Assume $`G`$ is simple, and $`\text{-rank}G2`$. If $`H`$ is a one-parameter subgroup of $`AN`$, then either

1. $`H`$ is tempered; or
2. $`HN`$.

###### Proof \[OW3, Prop. 3.7\].

Write $`H=\{h^t\}`$. From the Real Jordan Decomposition (3.2), we may assume, after replacing $`H`$ by a conjugate subgroup, that $`h^t=a^tu^t`$, where $`a^tA`$ is a hyperbolic one-parameter subgroup, and $`u^tN`$ is a unipotent one-parameter subgroup, such that $`a^t`$ and $`u^t`$ commute with each other.

We may assume $`HN`$, so $`a^t`$ is nontrivial. Since the growth of the hyperbolic one-parameter subgroup $`a^t`$ is exponential, while that of the unipotent one-parameter subgroup $`u^t`$ is polynomial, there is some $`ϵ>0`$, such that

$$h^t=a^tu^t>\sqrt{a^t}>e^{ϵ|t|}$$

for large $`t`$. Since the function $`C/e^{pϵ|t|}`$ is in $`L^1()`$, it follows from Theorem 5.23 that $`H`$ is tempered, as desired. ∎

###### 5.29 Lemma.

If $`d(H)=1`$ and $`HN`$, then $`G/H`$ does not have a tessellation.

###### Proof \[OW3, Prop. 3.7\].

We have $`dimH=d(H)=1`$ (see 1.14), so $`H`$ is a connected, one-dimensional, unipotent subgroup. Hence, the Jacobson-Morosov Lemma \[Hel, Thm. 9.7.4, p. 432\] implies that there exists a connected, closed subgroup $`H_1`$ of $`G`$, such that $`H_1`$ contains $`H`$, and $`H_1`$ is locally isomorphic to $`\mathrm{SL}(2,)`$. Then $`H`$ is a Cartan-decomposition subgroup of $`H_1`$ (see 2.16), so there is a compact subset $`C`$ of $`A`$, such that $`\mu (H_1)\mu (H)C`$ (see 2.12). Also, we have $`d(H_1)=2>1=d(H)`$. Therefore, Theorem 4.2 applies. ∎

###### 5.30 Corollary ((Oh-Witte \[OW3, Prop. 3.7\])).

If $`d(H)=1`$ and $`G`$ is simple, then $`G/H`$ does not have a tessellation.

###### Proof.

We may assume $`HAN`$ (see 3.5), so $`dimH=d(H)=1`$ (see 1.14).

* If $`\text{-rank}G<2`$, then Lemma 5.4 applies.
* If $`HN`$, then Lemma 5.29 applies.
* If $`\text{-rank}G2`$ and $`HN`$, then Corollary 5.28 implies that $`H`$ is tempered, so Theorem 5.16 applies.

## 6. Homogeneous spaces of $`\mathrm{SL}(3,)`$, $`\mathrm{SL}(3,)`$, and $`\mathrm{SL}(3,)`$

Y. Benoist \[Ben, Cor. 1\] and G.A. Margulis (unpublished) proved (independently) that $`\mathrm{SL}(3,)/\mathrm{SL}(2,)`$ does not have a tessellation. Using Benoist’s method, H. Oh and D. Witte \[OW3, Prop. 1.10\] generalized this result by replacing $`\mathrm{SL}(2,)`$ with any closed, connected subgroup $`H`$, such that neither $`H`$ nor $`\mathrm{SL}(3,)/H`$ is compact. The same argument applies even if $``$ replaced with either $``$ or $``$. However, the proof of Benoist (which applies in a more general context) relies on a somewhat lengthy argument to establish one particular lemma. Here, we adapt Benoist’s method to obtain a short proof of Theorem 1.8 that avoids any appeal to the lemma.

###### 6.1 Notation.

Assume $`G=\mathrm{SL}(3,𝔽)`$, for $`𝔽=`$, $``$, or $``$.

* Let $`\tau `$ be the opposition involution in $`A^+`$ (see 5.13);
* Let $`B^+=\{aA^+\tau (a)=a\}`$.

More concretely, we have

$$A^+=\left\{\left(\begin{array}{ccc}a_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)|\begin{array}{c}a_1,a_2,a_3^+,\\ a_1a_2a_3=1,\\ a_1a_2a_3\end{array}\right\};$$

$$\tau \left(\begin{array}{ccc}a_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)=\left(\begin{array}{ccc}a_3^1& 0& 0\\ 0& a_2^1& 0\\ 0& 0& a_1^1\end{array}\right);$$

$$B^+=\left\{\left(\begin{array}{ccc}a& 0& 0\\ 0& 1& 0\\ 0& 0& a^1\end{array}\right)|\begin{array}{c}\\ a1\end{array}\right\}.$$

###### 6.2 Lemma.

If $`G=\mathrm{SL}(3,𝔽)`$ and $`d(H)=1`$, then $`G/H`$ does not have a tessellation.

###### Proof.

Since $`\text{-rank}G=2`$, the desired conclusion follows from Corollary 5.14; since $`G`$ is simple, it also follows from Corollary 5.30. However, we give a proof that requires only the special case described in Proposition 5.2, rather than the full strength of (5.14) or (5.30).

Suppose $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$. (This will lead to a contradiction.) Let $`L_1`$ and $`L_2`$ be the two walls of $`A^+`$. From Proposition 4.10, we know that there exists $`k\{1,2\}`$, such that $`\mu (\mathrm{\Gamma })CL_k`$ is finite, for every compact subset $`C`$ of $`A`$.

Because $`\mathrm{\Gamma }^1=\mathrm{\Gamma }`$, we have $`\tau \left(\mu (\mathrm{\Gamma })\right)=\mu (\mathrm{\Gamma })`$. On the other hand, $`\tau `$ interchanges $`L_1`$ and $`L_2`$. Thus, the preceding paragraph implies that $`\mu (\mathrm{\Gamma })C(L_1L_2)`$ is finite, for every compact subset $`C`$ of $`A`$.

For $`H_1`$ as in (5.3), we have $`\mu (H_1)=L_1L_2`$, so the conclusion of the preceding paragraph implies that $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H_1`$ (see 2.17). Now Theorem 4.1(2) implies $`\mathrm{\Gamma }\backslash G/H_1`$ is compact; thus, $`G/H_1`$ has a tessellation. This contradicts Proposition 5.2. ∎

For completeness, we include the proof of the following simple proposition.

###### 6.3 Proposition (\[OW3, Prop. 7.3\]).

Assume $`G=\mathrm{SL}(3,𝔽)`$. If $`H`$ is a closed, connected subgroup of $`AN`$ with $`dimH2`$, then $`B^+\mu (H)`$.

###### Proof (cf. Figure 6.2 and proof of 8.19).

Since $`HAN`$ and $`dimH2`$, it is easy to construct a continuous, proper map $`\mathrm{\Phi }:[0,1]\times ^+H`$ such that $`\mathrm{\Phi }(1,t)=\mathrm{\Phi }(0,t)^1`$, for all $`t^+`$ (cf. Figure 6.1). For example, choose two linearly independent elements $`u`$ and $`v`$ of $`𝔥`$, and define

$$\mathrm{\Phi }(s,t)=\mathrm{exp}\left(t\mathrm{cos}(\pi s)u+t\mathrm{sin}(\pi s)v\right).$$

If we identify $`A`$ with its Lie algebra $`𝔞`$, then $`A^+`$ is a convex cone in $`𝔞`$ and the opposition involution $`\tau `$ is the reflection in $`A^+`$ across the ray $`B^+`$. Thus, for any $`aA^+`$, the points $`a`$ and $`\tau (a)`$ are on opposite sides of $`B^+`$, so any continuous curve in $`A^+`$ from $`a`$ to $`\tau (a)`$ must intersect $`B^+`$. In particular, for each $`t^+`$, the curve

$$\{\mu \left(\mathrm{\Phi }(s,t)\right)0s1\}$$

from $`\mu \left(\mathrm{\Phi }(0,t)\right)`$ to $`\mu \left(\mathrm{\Phi }(1,t)\right)`$ must intersect $`B^+`$. Thus, we see, from an elementary continuity argument, that $`\mu \left[\mathrm{\Phi }\left([0,1]\times ^+\right)\right]`$ contains $`B^+`$. Therefore, $`B^+`$ is contained in $`\mu (H)`$. ∎

###### Proof of Theorem 1.8 (cf. Figure 6.3 and proof of 4.12).

Suppose $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$. (This will lead to a contradiction.) We may assume $`HAN`$ (see 3.5).

Let $`F`$ be a (symmetric) finite generating set for $`\mathrm{\Gamma }`$, and choose a compact, convex, symmetric subset $`C`$ of $`A`$ so large that

$$\mu (\gamma F)\mu (\gamma )C$$

for every $`\gamma \mathrm{\Gamma }`$ (see 2.12).

From Lemma 6.2, we know that $`dimH2`$, so Proposition 6.3 implies that $`B^+\mu (H)`$. Then, because $`\mathrm{\Gamma }`$ acts properly on $`G/H`$, we conclude that $`\mu (\mathrm{\Gamma })B^+C`$ is finite (see 2.17). Since $`\mu `$ is a proper map, this implies that $`\mathrm{\Gamma }\mu ^1(B^+C)`$ is finite.

Let $`A_1`$ and $`A_2`$ be the two components of $`A^+B^+`$. Because $`\mathrm{\Gamma }^1=\mathrm{\Gamma }`$, we know that $`\tau \left(\mu (\mathrm{\Gamma })\right)=\mu (\mathrm{\Gamma })`$. Then, because $`\tau `$ interchanges $`A_1`$ and $`A_2`$, we conclude that $`\tau \left(\mu (\mathrm{\Gamma })A_1\right)=\mu (\mathrm{\Gamma })A_2`$. Therefore, $`\mu (\mathrm{\Gamma })A_1`$ and $`\mu (\mathrm{\Gamma })A_2`$ have the same cardinality, so they must both be infinite. So

each of $`\mathrm{\Gamma }\mu ^1(A_1)`$ and $`\mathrm{\Gamma }\mu ^1(A_2)`$ is infinite.
Because $`\mathrm{\Gamma }`$ has only one end (see 4.9), this implies there exist

$$\gamma \left(\mathrm{\Gamma }\mu ^1(A_1)\right)\mu ^1(B^+C),$$

such that

(6.4)
$$\gamma f\left(\mathrm{\Gamma }\mu ^1(A_2)\right)\mu ^1(B^+C),$$

for some $`fF`$. Then $`\mu (\gamma )A_1`$, $`\mu (\gamma f)A_2`$, and

$$\mu (\gamma f)\mu (\gamma F)\mu (\gamma )C.$$

Using the fact that $`C`$ is symmetric and the fact that $`C`$ contains the identity element $`e`$, we conclude that

$$\mu (\gamma )\left(\mu (\gamma f)C\right)A_1\text{ and }\mu (\gamma f)\left(\mu (\gamma f)C\right)A_2;$$

therefore $`\mu (\gamma f)C`$ intersects both $`A_1`$ and $`A_2`$. Since $`B^+`$ separates $`A_1`$ from $`A_2`$, and $`C`$ is connected, this implies that $`\mu (\gamma f)C`$ intersects $`B^+`$; hence $`\mu (\gamma f)B^+C`$. This contradicts the fact that $`\gamma f\mu ^1(B^+C)`$ (see 6.4). ∎

## 7. Explicit coordinates on $`𝔰𝔬(2,n)`$ and $`𝔰𝔲(2,n)`$

From this point on, we focus almost entirely on $`\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(2,n)`$. (The only exception is that some of the examples constructed in Section 9 are for other groups.) In this section, we define the group $`\mathrm{SU}(2,n;𝔽)`$, which allows us to provide a fairly unified treatment of $`\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(2,n)`$ in later sections.

### 7A. The group $`\mathrm{SU}(2,n;𝔽)`$

###### 7.1 Notation.

* We use $`𝔽`$ to denote either $``$ or $``$.
* Let $`q=dim_{}𝔽`$, so $`q\{1,2\}`$.
* We use $`𝔽_{\text{imag}}`$ to denote the purely imaginary elements of $`𝔽`$, so

$$𝔽_{\text{imag}}=\{\begin{array}{cc}0\hfill & \text{if }𝔽=\hfill \\ i\hfill & \text{if }𝔽=.\hfill \end{array}$$
* For $`\varphi 𝔽`$, there exist unique $`\mathrm{Re}\varphi `$ and $`\mathrm{Im}\varphi 𝔽_{\text{imag}}`$, such that $`\varphi =\mathrm{Re}\varphi +\mathrm{Im}\varphi `$. (Warning: in our notation, the imaginary part of $`a+bi`$ is $`bi`$, *not* $`b`$.)
* For $`\varphi 𝔽`$, we use $`\overline{\varphi }`$ to denote the conjugate $`\mathrm{Re}\varphi \mathrm{Im}\varphi `$ of $`\varphi `$. (If $`𝔽=`$, then $`\overline{\varphi }=\varphi `$.)
* For a row vector $`x𝔽^{n2}`$, or, more generally, for any matrix $`x`$ with entries in $`𝔽`$, we use $`x^{}`$ to denote the conjugate-transpose of $`x`$.

###### 7.2 Notation.

For

$$J=\left(\begin{array}{ccccccc}0& 0& 0& \mathrm{}& 0& 0& 1\\ 0& 0& 0& \mathrm{}& 0& 1& 0\\ 0& 0& & & & 0& 0\\ \mathrm{}& \mathrm{}& & \mathrm{Id}& & \mathrm{}& \mathrm{}\\ 0& 0& & & & 0& 0\\ 0& 1& 0& \mathrm{}& 0& 0& 0\\ 1& 0& 0& \mathrm{}& 0& 0& 0\end{array}\right)\mathrm{SL}(n+2,𝔽),$$

we define

$$\mathrm{SU}(2,n;𝔽)=\{g\mathrm{SL}(n+2,𝔽)gJg^{}=J\}$$

and

$$𝔰𝔲(2,n;𝔽)=\{u𝔰𝔩(n+2,𝔽)uJ+Ju^{}=0\}.$$

Then:

* $`\mathrm{SU}(2,n;)`$ is a realization of $`\mathrm{SO}(2,n)`$,
* $`\mathrm{SU}(2,n;)`$ is a realization of $`\mathrm{SU}(2,n)`$, and
* $`𝔰𝔲(2,n;𝔽)`$ is the Lie algebra of $`\mathrm{SU}(2,n;𝔽)`$.

We choose

* $`A`$ to consist of the diagonal matrices in $`\mathrm{SU}(2,n;𝔽)`$ that have nonnegative real entries,
* $`N`$ to consist of the upper-triangular matrices in $`\mathrm{SU}(2,n;𝔽)`$ with only $`1`$’s on the diagonal, and
* $`K=\mathrm{SU}(2,n;𝔽)\mathrm{SU}(n+2)`$.

A straightforward matrix calculation shows that the Lie algebra of $`AN`$ is

(7.3)
$$𝔞+𝔫=\left\{\left(\begin{array}{ccccc}t_1& \varphi & x& \eta & 𝗑\\ 0& t_2& y& 𝗒& \overline{\eta }\\ 0& 0& 0& y^{}& x^{}\\ 0& 0& 0& t_2& \overline{\varphi }\\ 0& 0& 0& 0& t_1\end{array}\right)|\begin{array}{c}t_1,t_2,\\ \varphi ,\eta 𝔽,\\ x,y𝔽^{n2},\\ 𝗑,𝗒𝔽_{\text{imag}}\end{array}\right\}.$$

###### 7.4 Remark.

From (7.3), we see that the first two rows of any element of $`𝔞+𝔫`$ are sufficient to determine the entire matrix. In fact, it is also not necessary to specify the last entry of the second row of the matrix.

###### 7.5 Remark.

From (1.13) and (7.3), we see that $`d\left(\mathrm{SU}(2,n;𝔽)\right)=dim(𝔞+𝔫)=2qn`$.

###### 7.6 Notation.

Because $`N`$ is simply connected and nilpotent, the exponential map is a diffeomorphism from $`𝔫`$ to $`N`$ (indeed, its inverse, the logarithm map, is a polynomial \[Hc1, Thm. 8.1.1, p. 107\], so each element of $`N`$ has a unique representation in the form $`\mathrm{exp}u`$ with $`u𝔫`$. Thus, each element $`h`$ of $`N`$ determines corresponding values of $`\varphi `$, $`x`$, $`y`$, $`\eta `$, $`𝗑`$ and $`𝗒`$ (with $`t_1=t_2=0`$). We write

$$\varphi _h,x_h,y_h,\eta _h,𝗑_h,𝗒_h$$

for these values.

###### 7.7 Notation.

We let $`\alpha `$ and $`\beta `$ be the simple real roots of $`\mathrm{SU}(2,n;𝔽)`$, defined by

$$\alpha (a)=a_{1,1}/a_{2,2}\text{ and }\beta (a)=a_{2,2},$$

for a (diagonal) element $`a`$ of $`A`$. Thus, the positive real roots (see Figure 7.1) are

$$\{\begin{array}{cc}\alpha ,\beta ,\alpha +\beta ,\alpha +2\beta ,\hfill & \text{if }𝔽=\hfill \\ \alpha ,\beta ,\alpha +\beta ,\alpha +2\beta ,2\beta ,2\alpha +2\beta \hfill & \text{if }𝔽=.\hfill \end{array}$$

Concretely:

* the root space $`𝔫_\alpha `$ is the $`\varphi `$-subspace in $`𝔫`$,
* the root space $`𝔫_\beta `$ is the $`y`$-subspace in $`𝔫`$,
* the root space $`𝔫_{\alpha +\beta }`$ is the $`x`$-subspace in $`𝔫`$,
* the root space $`𝔫_{\alpha +2\beta }`$ is the $`\eta `$-subspace in $`𝔫`$,
* the root space $`𝔫_{2\beta }`$ is the $`𝗒`$-subspace in $`𝔫`$ (this is 0 if $`𝔽=`$), and
* the root space $`𝔫_{2\alpha +2\beta }`$ is the $`𝗑`$-subspace in $`𝔫`$ (this is 0 if $`𝔽=`$).

###### 7.8 Definition.

Let

$`𝔡`$ $`=𝔫_{\alpha +2\beta }+𝔫_{2\alpha +2\beta }+𝔫_{2\beta }`$
$`=\{z𝔫\varphi _z=0,x_z=y_z=0\}`$
$`=\left\{\left(\begin{array}{ccccc}0& 0& 0& \eta & 𝗑\\ & 0& 0& 𝗒& \overline{\eta }\\ & & \mathrm{}\end{array}\right)|\begin{array}{c}\eta 𝔽,\\ 𝗑,𝗒𝔽_{\text{imag}}\end{array}\right\}.`$
and, for a given Lie algebra $`𝔥𝔫`$,

$$𝔡_𝔥=𝔡𝔥.$$

Note that if $`\varphi _u=0`$ for every $`u𝔥`$, then $`[𝔥,𝔥]𝔡_𝔥`$ and $`𝔡_𝔥`$ is contained in the center of $`𝔥`$ (cf. 7.22).

###### 7.9 Remark.

By definition (see 2.10), we have

$$A^+=\{aA\alpha (a)1,\beta (a)1\}.$$

Therefore, from the definition of $`\alpha `$ and $`\beta `$ (see 7.7), we see that

(7.10)
$$A^+=\{aAa_{1,1}a_{2,2}1\}.$$

###### 7.11 Remark.

For $`𝔽=`$, the division algebra of real quaternions, the group $`\mathrm{SU}(2,n;)`$ is a realization of $`\mathrm{Sp}(2,n)`$. Most of the work in this paper carries over, but the upper bound on $`dimH`$ given in Theorem 10.19 is not sharp in this case (and it does not seem to be easy to improve this result to obtain a sharp bound). Thus, we have not obtained any interesting conclusions about the nonexistence of tessellations of homogeneous spaces of $`\mathrm{Sp}(2,n)`$.

### 7B. The subgroups $`\mathrm{SU}(1,n;𝔽)`$ and $`\mathrm{Sp}(1,m;𝔽)`$

We now describe how the four important families of homogeneous spaces of Example 1.10 are realized in terms of $`\mathrm{SU}(2,n;𝔽)`$.

###### 7.12 Definition.

Let

* $`\mathrm{SU}(1,n;)=\mathrm{SO}(1,n)`$;
* $`\mathrm{Sp}(1,n;)=\mathrm{SU}(1,n)`$;
* $`\mathrm{SU}(1,n;)=\mathrm{SU}(1,n)`$; and
* $`\mathrm{Sp}(1,n;)=\mathrm{Sp}(1,n)`$.

Then, for an appropriate choice of the embeddings in Example 1.10, we have

(7.13)
$$𝔰𝔲(1,n;𝔽)(𝔞+𝔫)=\left\{\left(\begin{array}{ccccc}t& \varphi & x& \varphi & 𝗑\\ 0& 0& 0& 0& \overline{\varphi }\\ & & \mathrm{}\end{array}\right)|\begin{array}{c}t,\\ \varphi 𝔽,\\ x𝔽^{n2},\\ 𝗑𝔽_{\text{imag}}\end{array}\right\}$$

and (if $`2mn`$) we have

(7.14)
$$\begin{array}{c}𝔰𝔭(1,m;𝔽)(𝔞+𝔫)=\hfill \\ \hfill \left\{\left(\begin{array}{cccccccccccccc}t& 0& x_1& x_2& x_3& x_4& \mathrm{}& x_{2m3}& x_{2m2}& 0& \mathrm{}& 0& \eta & 𝗑\\ 0& t& \overline{x_2}& \overline{x_1}& \overline{x_4}& \overline{x_3}& \mathrm{}& \overline{x_{2m2}}& \overline{x_{2m3}}& 0& \mathrm{}& 0& 𝗑& \overline{\eta }\\ & & & & & & & \mathrm{}\end{array}\right)|\begin{array}{c}t,\\ x_j𝔽,\\ \eta 𝔽,\\ 𝗑𝔽_{\text{imag}}\end{array}\right\}.\end{array}$$

###### 7.15 Remark.

From (1.13), (7.13) and (7.14), we see that

* $`d\left(\mathrm{SU}(1,n;𝔽)\right)=dim\left(𝔰𝔲(1,n;𝔽)(𝔞+𝔫)\right)=qn`$ and
* $`d\left(\mathrm{Sp}(1,m;𝔽)\right)=dim\left(𝔰𝔭(1,m;𝔽)(𝔞+𝔫)\right)=2qm`$.

### 7C. Formulas for exponentials and brackets

The arguments in later sections often require the calculation of $`\mathrm{exp}u`$, for some $`u𝔫`$, or of $`[u,v]`$, for some $`u,v𝔫`$. We now provide these calculations for the reader’s convenience.

###### 7.16 Remark.

For

$$u=\left(\begin{array}{ccccc}0& \varphi & x& \eta & 𝗑\\ 0& 0& y& 𝗒& \overline{\eta }\\ 0& 0& 0& y^{}& x^{}\\ 0& 0& 0& 0& \overline{\varphi }\\ 0& 0& 0& 0& 0\end{array}\right)𝔫,$$

we have

(7.17)
$$\mathrm{exp}(u)=\left(\begin{array}{ccccc}1& \varphi & x+\frac{1}{2}\varphi y& \eta \frac{1}{2}xy^{}+\frac{1}{2}\varphi 𝗒\frac{1}{6}\varphi |y|^2& \genfrac{}{}{0.0pt}{}{\frac{1}{2}|x|^2\mathrm{Re}(\varphi \overline{\eta })+\frac{1}{24}|\varphi |^2|y|^2}{+\left(𝗑\frac{1}{6}\varphi 𝗒\overline{\varphi }+\frac{1}{3}\mathrm{Im}(xy^{}\overline{\varphi })\right)}\\ 0& 1& y& 𝗒\frac{1}{2}|y|^2& \overline{\eta }\frac{1}{2}yx^{}\frac{1}{2}𝗒\overline{\varphi }+\frac{1}{6}|y|^2\overline{\varphi }\\ 0& 0& \mathrm{Id}& y^{}& x^{}+\frac{1}{2}y^{}\overline{\varphi }\\ 0& 0& 0& 1& \varphi ^{}\\ 0& 0& 0& 0& 1\end{array}\right).$$

When $`\varphi =0`$, this simplifies to:

(7.18)
$$\mathrm{exp}(u)=\left(\begin{array}{ccccc}1& 0& x& \eta \frac{1}{2}xy^{}& 𝗑\frac{1}{2}|x|^2\\ 0& 1& y& 𝗒\frac{1}{2}|y|^2& \overline{\eta }\frac{1}{2}yx^{}\\ 0& 0& \mathrm{Id}& y^{}& x^{}\\ 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 1\end{array}\right).$$

Similarly, when $`y=0`$, we have

(7.19)
$$\mathrm{exp}(u)=\left(\begin{array}{ccccc}1& \varphi & x& \eta +\frac{1}{2}\varphi 𝗒& \genfrac{}{}{0.0pt}{}{\frac{1}{2}|x|^2\mathrm{Re}(\varphi \overline{\eta })}{+\left(𝗑\frac{1}{6}\varphi 𝗒\overline{\varphi }\right)}\\ 0& 1& 0& 𝗒& \overline{\eta }\frac{1}{2}𝗒\overline{\varphi }\\ 0& 0& \mathrm{Id}& 0& x^{}\\ 0& 0& 0& 1& \varphi ^{}\\ 0& 0& 0& 0& 1\end{array}\right).$$

###### 7.20 Remark.

For

(7.21)
$$u=\left(\begin{array}{ccccc}0& \varphi & x& \eta & 𝗑\\ & 0& y& 𝗒& \overline{\eta }\\ & & \mathrm{}\end{array}\right)\text{ and }\stackrel{~}{u}=\left(\begin{array}{ccccc}0& \stackrel{~}{\varphi }& \stackrel{~}{x}& \stackrel{~}{\eta }& \stackrel{~}{𝗑}\\ & 0& \stackrel{~}{y}& \stackrel{~}{𝗒}& \overline{\stackrel{~}{\eta }}\\ & & \mathrm{}\end{array}\right),$$

we have

(7.22)
$$[u,\stackrel{~}{u}]=\left(\begin{array}{ccccc}0& 0& \varphi \stackrel{~}{y}\stackrel{~}{\varphi }y& x\stackrel{~}{y}^{}+\stackrel{~}{x}y^{}+\varphi \stackrel{~}{𝗒}\stackrel{~}{\varphi }𝗒& 2\mathrm{Im}(x\stackrel{~}{x}^{}+\varphi \overline{\stackrel{~}{\eta }}\stackrel{~}{\varphi }\overline{\eta })\\ & 0& 0& 2\mathrm{Im}(y\stackrel{~}{y}^{})& \stackrel{~}{y}x^{}y\stackrel{~}{x}^{}+\stackrel{~}{𝗒}\overline{\varphi }𝗒\overline{\stackrel{~}{\varphi }}\\ & & & \mathrm{}\end{array}\right).$$

###### 7.23 Remark (\[Var, Eq. (2.13.8), p. 104\]).

For $`u,v𝔥`$, we have

$$\mathrm{exp}(v)u\mathrm{exp}(v)=u+[u,v]+\frac{1}{2}[[u,v],v]+\frac{1}{3!}[[[u,v],v],v]+\mathrm{}.$$

Combining this with (7.22) allows us to calculate the effect of conjugating by an element of $`N`$.

For example, suppose $`u𝔫`$, with $`\varphi _u=0`$ and $`y_u=0`$, and suppose $`v𝔫_{\alpha +\beta }`$. We see, from (7.22), that $`\varphi _{[u,v]}=0`$ and that $`x_{[u,v]}=y_{[u,v]}=0`$, so $`[[u,v],v]=0`$ (see 7.22). Therefore

$$\mathrm{exp}(v)u\mathrm{exp}(v)=u+[u,v].$$

## 8. Calculating the Cartan projection

Y. Benoist \[Ben, Lem. 2.4\] showed that calculating values of the Cartan projection $`\mu `$ is no more difficult than calculating the norm of a matrix (see 8.11). In this section, we describe this elegant method and some of its consequences, in the special case $`G=\mathrm{SU}(2,n;𝔽)`$.

###### 8.1 Standing assumptions.

Throughout this section, we assume $`G=\mathrm{SU}(2,n;𝔽)`$.

### 8A. The basic definitions

###### 8.2 Notation.

We employ the usual Big Oh and little oh notation: for functions $`f_1,f_2`$ on a subset $`X`$ of $`G`$, we say

$`f_1=O(f_2)`$ for $`hX`$
if there is a constant $`C`$, such that, for all $`hX`$ with $`h`$ large, we have $`f_1(h)Cf_2(h)`$. (The values of each $`f_j`$ are assumed to belong to some finite-dimensional normed vector space, typically either $``$ or a space of complex matrices. Which particular norm is used does not matter, because all norms are equivalent up to a bounded factor.) We say

$`f_1=o(f_2)`$ for $`hX`$
if $`f_1(h)/f_2(h)0`$ as $`h\mathrm{}`$. (We use $`h\mathrm{}`$ to mean $`h\mathrm{}`$.) Also, we write

$$f_1f_2$$

if $`f_1=O(f_2)`$ and $`f_2=O(f_1)`$.

We use the following norm on $`\mathrm{SU}(2,n;𝔽)`$, because it is easy to calculate. The reader is free to make a different choice, at the expense of changing $`=`$ to $``$ in a few of the calculations.

###### 8.3 Definition.

For $`h\mathrm{SU}(2,n;𝔽)`$, we define $`h`$ to be the maximum absolute value among the matrix entries of $`h`$. That is,

$$h=\underset{1j,kn+2}{\mathrm{max}}|h_{j,k}|.$$

###### 8.4 Definition.

Define $`\rho :\mathrm{SU}(2,n;𝔽)\mathrm{GL}(𝔽^{n+2}𝔽^{n+2})`$ by $`\rho (h)=hh`$, so $`\rho `$ is the second exterior power of the standard representation of $`\mathrm{SU}(2,n;𝔽)`$. Thus, we may define $`\rho (h)`$ to be the maximum absolute value among the determinants of all the $`2\times 2`$ submatrices of the matrix $`h`$. That is,

$$\rho (h)=\underset{1j,k,\mathrm{},mn+2}{\mathrm{max}}\left|det\left(\begin{array}{cc}h_{j,k}& h_{j,\mathrm{}}\\ h_{m,k}& h_{m,\mathrm{}}\end{array}\right)\right|.$$

From (7.17), (7.18), and (7.19), it is clear that the $`2\times 2`$ minor in the top right corner is often larger than the other $`2\times 2`$ minors, so we give it a special name.

###### 8.5 Definition.

For $`h\mathrm{Mat}_{n+2}(𝔽)`$, define

$$\mathrm{\Delta }(h)=det\left(\begin{array}{cc}h_{1,n+1}& h_{1,n+2}\\ h_{2,n+1}& h_{2,n+2}\end{array}\right).$$

### 8B. Y. Benoist’s method for using matrix norms to calculate $`\mu `$

###### 8.6 Lemma.

For $`aA^+`$, we have $`a=a_{1,1}`$ and $`\rho (a)=a_{1,1}a_{2,2}`$.

###### Proof.

From (7.3), we see that

(8.7)
$$a_{j,j}=\{\begin{array}{cc}1\hfill & \text{if }3jn\hfill \\ 1/a_{2,2}\hfill & \text{if }j=n+1\hfill \\ 1/a_{1,1}\hfill & \text{if }j=n+2.\hfill \end{array}$$

Thus, from (7.10), we see that

$$a_{1,1}a_{2,2}a_{j,j}$$

for $`j3`$ (and, since $`a`$ is diagonal, we have $`a_{j,k}=0`$ for $`jk`$). Therefore, the desired conclusions follow from the definitions of $`a`$ and $`\rho (a)`$. ∎

###### 8.8 Proposition ((Benoist, cf. \[Ben, Lem. 2.4\])).

We have

(8.9)
$$\mu (h)h,$$

(8.10)
$$\rho \left(\mu (h)\right)\rho (h),$$

(8.11) $`\mu (h)_{1,1}h`$, and $`\mu (h)_{2,2}\rho (h)/h`$,
for $`h\mathrm{SU}(2,n;𝔽)`$.

###### Proof.

Choose $`k_1,k_2K`$, such that $`\mu (h)=k_1hk_2`$. Because $`xy=O\left(xy\right)`$ for $`x,y\mathrm{SU}(2,n;𝔽)`$, and $`\mathrm{max}_{kK}k<\mathrm{}`$ (since $`K`$ is compact), we have

$$\mu (h)=k_1hk_2=O\left(h\right)$$

and

$$h=k_1^1\mu (h)k_2^1=O\left(\mu (h)\right),$$

so (8.9) holds. Similarly, we have

$$\rho \left(\mu (h)\right)=\rho (k_1)\rho (h)\rho (k_2)\rho (h),$$

so (8.10) holds.

For $`aA^+`$, we know, from (8.6), that $`a_{1,1}=a`$ and $`a_{2,2}=\rho (a)/a_{1,1}`$. Thus, letting $`a=\mu (h)`$, and using (8.9) and (8.10), we see that

$$\mu (h)_{1,1}=\mu (h)h$$

and

$$\mu (h)_{2,2}=\frac{\rho \left(\mu (h)\right)}{\mu (h)_{1,1}}\frac{\rho (h)}{h},$$

as desired. ∎

###### 8.12 Remark.

Proposition 8.8 generalizes to any reductive group $`G`$ \[Ben, Lem. 2.3\]. However, one may need to use a different representation in the place of $`\rho `$. In fact, if $`\text{-rank}G=r`$, then $`r`$ representations of $`G`$ are needed; for $`G=\mathrm{SU}(2,n;𝔽)`$, we have $`\text{-rank}G=2`$, and the two representations we use are $`\rho `$ and the identity representation $`I(h)=h`$.

###### 8.13 Corollary.

Let $`g_n\mathrm{}`$ and $`h_n\mathrm{}`$ be two sequences of elements of $`\mathrm{SU}(2,n;𝔽)`$. We have

$`g_nh_n`$ and $`\rho (g_n)\rho (h_n)`$
if and only if

$$\text{there is a compact subset }C\text{ of }A\text{, such that, for all }n^+\text{, we have }\mu (g_n)\mu (h_n)C.$$

###### Proof.

($``$) Let $`a=\mu (h_n)^1\mu (g_n)`$. From (8.11), we see that $`\mu (g_n)_{j,j}\mu (h_n)_{j,j}`$ for $`j\{1,2\}`$, so, using (8.7), we have

$$a_{j,j}=\frac{\mu (g_n)_{j,j}}{\mu (h_n)_{j,j}}=\{\begin{array}{cc}O(1)\hfill & \text{if }1j2\text{;}\hfill \\ 1/1=1\hfill & \text{if }3jn\text{;}\hfill \\ \mu (h_n)_{2,2}/\mu (g_n)_{2,2}=O(1)\hfill & \text{if }j=n+1\text{;}\hfill \\ \mu (h_n)_{1,1}/\mu (g_n)_{1,1}=O(1)\hfill & \text{if }j=n+2.\hfill \end{array}$$

Therefore $`a=O(1)`$, as desired.

($``$) Because $`C`$ is compact, we have

$`\mu (g_n)\mu (h_n)`$ and $`\rho \left(\mu (g_n)\right)\rho \left(\mu (h_n)\right)`$
(cf. proof of (8.9) and (8.10)). Then the desired conclusions follow from (8.9) and (8.10). ∎

###### Proof of Proposition 2.12 for $`G=\mathrm{SU}(2,n;𝔽)`$.

Because $`C`$ is compact, we have $`g^{}g`$ and $`\rho (g^{})\rho (g)`$ for any $`g^{}CgC`$ (cf. proof of (8.9) and (8.10)). Thus, the desired conclusion follows from Corollary 8.13. ∎

Because of Proposition 8.8, we will often need to calculate $`h`$ and $`\rho (h)`$. The following observation and its corollary sometimes simplifies the work, by allowing us to replace $`h`$ with $`h^1`$.

###### 8.14 Lemma.

We have $`\mu (h^1)=\mu (h)`$ for $`h\mathrm{SU}(2,n;𝔽)`$.

###### Proof.

Define $`J`$ as in (7.2), and choose $`k_1,k_2K`$, such that $`\mu (h)=k_1hk_2`$. For any $`aA^+`$, we see, using (7.3) or (8.7), that $`Ja^1J=a`$, so

$$(Jk_2^1)h^1(k_1^1J)=J\mu (h)^1J=\mu (h).$$

Note that $`detJ=1`$. Also, we have $`J^2=\mathrm{Id}`$ and $`J^{}=J`$, so it is obvious that $`JJJ^{}=J`$ and $`JJ^{}=\mathrm{Id}`$. Therefore

$$J\mathrm{SU}(2,n;𝔽)\mathrm{SU}(n+2)=K.$$

Thus, from the definition of $`\mu `$, we conclude that $`\mu (h^1)=\mu (h)`$, as desired. ∎

The following corollary is obtained by combining Lemma 8.14 with Corollary 8.13.

###### 8.15 Corollary.

We have $`h^1h`$ and $`\rho (h^1)\rho (h)`$ for $`h\mathrm{SU}(2,n;𝔽)`$.

### 8C. The walls of $`A^+`$

###### 8.16 Notation.

For $`k\{1,2\}`$, set

(8.17)
$$L_k=\{aA^+a_{2,2}=a_{1,1}^{k1}\}.$$

From (7.10), we see that $`L_1`$ and $`L_2`$ are the two walls of $`A^+`$. From (8.6), we have

(8.18)
$$\rho (a)a^k\text{ for }aL_k.$$

We reproduce the proof of the following result, because it is both short and instructive. (Although we have no need for it here, let us point out that the converse of this proposition also holds, and that there is no need to assume $`HAN`$.) Because of this proposition (and Corollary 2.9), Section 10 will study the existence of curves $`h^t`$, such that $`h^th^t^k`$, for $`k\{1,2\}`$.

###### 8.19 Proposition ((Oh-Witte \[OW1, Prop. 3.24\])).

Let $`H`$ be a closed, connected subgroup of $`AN`$ in $`\mathrm{SU}(2,n;𝔽)`$. If, for each $`k\{1,2\}`$, there is a continuous curve $`h^t`$ in $`H`$, such that $`\rho (h^t)h^t^k\mathrm{}`$ as $`t\mathrm{}`$, then $`H`$ is a Cartan-decomposition subgroup.

###### Proof (cf. Figure 8.1 and proof of Prop. 6.3).

By hypothesis, there is a continuous, proper map $`\mathrm{\Phi }:\{1,2\}\times ^+H`$, such that $`\rho \left(\mathrm{\Phi }(k,t)\right)\mathrm{\Phi }(k,t)^k`$. Because $`HAN`$, we know that $`H`$ is homeomorphic to some Euclidean space $`^m`$ (see 3.15(1)).

Suppose, for the moment, that $`dimH=1`$. (This will lead to a contradiction.) We know that $`\rho (h)h`$ for $`h\mathrm{\Phi }(1,^+)`$. Because $`h^1h`$ and $`\rho (h^1)\rho (h)`$ (see 8.15), we must also have $`\rho (h)h`$ for $`h\mathrm{\Phi }(1,^+)^1`$. There is no harm in assuming $`\varphi (1,0)=\mathrm{Id}`$; then $`\mathrm{\Phi }(1,^+)\mathrm{\Phi }(1,^+)^1=H`$ (because $`dimH=1`$), so we conclude that $`\rho (h)h`$ for all $`hH`$. This contradicts the fact that $`\rho (h)h^2`$ for $`h\mathrm{\Phi }(2,^+)`$.

We may now assume $`dimH2`$. Then, because $`H`$ is homeomorphic to $`^m`$, it is easy to extend $`\mathrm{\Phi }`$ to a continuous and proper map $`\mathrm{\Phi }^{}:[1,2]\times ^+H`$. From (8.18) and (8.13), we know that the curve $`\mu \left(\mathrm{\Phi }^{}(k,t)\right)`$ stays within a bounded distance from the wall $`L_k`$; say $`\mathrm{dist}[\left(\mathrm{\Phi }^{}(k,t)\right),L_k]<C`$ for all $`t`$. We may assume $`C`$ is large enough that $`\mathrm{dist}(\mathrm{\Phi }^{}(s,1),e)<C`$ for all $`s[1,2]`$. Then an elementary homotopy argument shows that $`\mu \left[\mathrm{\Phi }^{}\left([1,2]\times ^+\right)\right]`$ contains

$$\{aA^+\mathrm{dist}(a,L_1L_2)>C\},$$

so $`\mu \left[\mathrm{\Phi }^{}\left([1,2]\times ^+\right)\right]A^+`$. Because $`\mu (H)\mu \left[\mathrm{\Phi }^{}\left([1,2]\times ^+\right)\right]`$, we conclude from Theorem 2.15 that $`H`$ is a Cartan-decomposition subgroup. ∎

###### 8.20 Remark.

When $`\text{-rank}G=1`$, the Weyl chamber $`A^+`$ has only one point at infinity. Thus, if $`H`$ is any noncompact subgroup, then the closure of $`\mu (H)`$ must contain this point at infinity. This is why it is easy to prove that any noncompact subgroup of $`G`$ is a Cartan-decomposition subgroup (see 2.16).

The idea of Proposition 8.19 is that if $`\text{-rank}G=2`$, then the points at $`\mathrm{}`$ of the Weyl chamber $`A^+`$ form a closed interval. If the closure of $`\mu (H)`$ contains the two endpoints of this interval, then, by continuity, it must also contain all the points in between.

Unfortunately, we have no good substitute for this proposition when $`\text{-rank}G>2`$. The points at $`\mathrm{}`$ of $`A^+`$ form a closed disk (topologically speaking). It is easy to define a map $`f`$ from one disk to another, such that the image of $`f`$ contains the entire boundary sphere, but does not contain the interior of the disk. Thus, it does not suffice to show only that the closure of $`\mu (H)`$ contains the boundary of the disk at $`\mathrm{}`$; rather, one needs additional homotopical information to guarantee that no interior points are missed.

###### 8.21 Lemma.

Let $`G=\mathrm{SU}(2,n;𝔽)`$, and fix some $`mn/2`$. Then $`\mu \left(\mathrm{SU}(1,n;𝔽)\right)`$ and $`\mu \left(\mathrm{Sp}(1,m;𝔽)\right)`$ are the two walls of $`A^+`$.

We have

1. $`\rho (h)h`$ for $`h\mathrm{SU}(1,n;𝔽)`$; and
2. $`\rho (h)h^2`$ for $`h\mathrm{Sp}(1,m;𝔽)`$.

###### Proof.

Let $`H=\mathrm{SU}(1,n;𝔽)`$ or $`\mathrm{Sp}(1,m;𝔽)`$. Then $`HK`$ is a maximal compact subgroup of $`H`$. From the Cartan decomposition

$$H=(KH)(AH)(KH),$$

and the definition of $`\mu `$, we conclude that $`\mu (H)=\mu (AH)`$. In the notation of (8.17), we see (from Definition 7.12) that $`HA=L_kL_k^1`$, where

$$k=\{\begin{array}{cc}1\hfill & \text{if }H=\mathrm{SU}(1,n;𝔽);\hfill \\ 2\hfill & \text{if }H=\mathrm{Sp}(1,m;𝔽).\hfill \end{array}$$

Then, since $`\mu (a^1)=\mu (a)`$ (see 8.14) and $`\mu (a)=a`$ for $`aA^+`$, we conclude that $`\mu (H)=L_k`$ is a wall of $`A^+`$. Furthermore, we have $`\rho (a)a^k`$ for $`a\mu (H)`$ (see 8.18), so $`\rho (h)h^k`$ for $`hH`$ (see 8.8). ∎

###### 8.22 Corollary.

If there is a continuous curve $`h^t\mathrm{}`$ in $`H`$, such that $`\rho (h^t)h^t`$, then there is a compact subset $`C`$ of $`G`$, such that $`\mathrm{SU}(1,n;𝔽)CHC`$.

###### Proof.

For any (large) $`g\mathrm{SU}(1,n;𝔽)`$, we see from continuity (more precisely, from the Intermediate Value Theorem) that there exists $`t^+`$, such that

$$h^t=g.$$

Then, by assumption and from 8.21(1), we have

$$\rho (h^t)h^tg\rho (g),$$

so there is a compact subset $`C^{}`$ of $`A`$, such that

$$\mu \left(\mathrm{SU}(1,n;𝔽)\right)\{\mu (h^t)t^+\}C^{}\mu (H)C^{}$$

(see 8.13). Therefore

$$\mathrm{SU}(1,n;𝔽)K\mu \left(\mathrm{SU}(1,n;𝔽)\right)KK\mu (H)C^{}KK(KHK)C^{}K,$$

as desired. ∎

The following corollary can be proved by a similar argument. (Recall that the equivalence relation $``$ is defined in (1.15).)

###### 8.23 Corollary.

Assume $`H`$ is not compact.

1. We have $`H\mathrm{SU}(1,n;𝔽)`$ if and only if $`\rho (h)h`$ for $`hH`$.
2. We have $`H\mathrm{Sp}(1,m;𝔽)`$ if and only if $`\rho (h)h^2`$ for $`hH`$.

Because of Proposition 8.19, we will often want to show that a curve $`h^t`$ satisfies $`\rho (h^t)h^k`$, for some $`k\{1,2\}`$. The following lemma does half of the work.

###### 8.24 Lemma.

Let $`X`$ be a subset of $`\mathrm{SU}(2,n;𝔽)`$.

1. If $`\rho (h)=O(h)`$ for $`hX`$, then $`\rho (h)h`$ for $`hX`$.
2. If $`h^2=O\left(\rho (h)\right)`$ for $`hX`$, then $`\rho (h)h^2`$ for $`hX`$.

###### Proof.

From (8.6) and (7.10), we have

$$a=a_{1,1}a_{1,1}a_{2,2}=\rho (a)$$

and

$$\rho (a)=a_{1,1}a_{2,2}a_{1,1}^2=a^2$$

for $`aA^+`$. Thus, letting $`a=\mu (h)`$, and using (8.9) and (8.10), we have:

$$h\mu (h)\rho \left(\mu (h)\right)\rho (h)$$

and

$$\rho (h)\rho \left(\mu (h)\right)\mu (h)^2h^2,$$

so $`h=O\left(\rho (h)\right)`$ and $`\rho (h)=O\left(h^2\right)`$. The desired conclusions follow. ∎

For convenience, we record the following simple observation. (For the proof, cf. the proof of (8.9) and (8.10).)

###### 8.25 Lemma.

Let

* $`k\{1,2\}`$,
* $`gG`$, and
* $`h^t\mathrm{}`$ be a continuous curve in $`H`$.

If $`\rho (h^t)h^t^k`$, then $`\rho (g^1h^tg)g^1h^tg^k`$.

### 8D. Homogeneous functions of the same degree

The following well-known, elementary observation is used frequently in the later sections.

###### 8.26 Lemma.

Let $`V^{}`$ be a subspace of a finite-dimensional real vector space $`V`$, and let $`f_1:VW_1`$ and $`f_2:VW_2`$ be linear transformations.

1. If $`f_1^1(0)V^{}=\{0\}`$ (or, more generally, if $`f_1^1(0)V^{}f_2^1(0)`$), then there is a linear transformation $`f:W_1W_2`$, such that $`f_2(v)=f\left(f_1(v)\right)`$ for all $`vV^{}`$. Therefore $`f_2=O(f_1)`$ on $`V^{}`$.
2. If $`f_1^1(0)V^{}=f_2^1(0)V^{}`$, then $`f_1f_2`$ on $`V^{}`$.

###### Proof.

(1) By passing to a subspace, we may assume $`V^{}=V`$. Then, by modding out $`f_1^1(0)`$, we may assume $`f_1`$ is an isomorphism onto its image. Define $`f^{}:f_1(V)W_2`$ by $`f^{}(w)=f_2\left(f_1^1(w)\right)`$, and let $`f:W_1W_2`$ be any extension of $`f^{}`$.

For $`vV^{}`$, we have

$$f_2(v)=f\left(f_1(v)\right)ff_1(v),$$

so $`f_2=O(f_1)`$.

(2) From (1), we have $`f_2=O(f_1)`$ and $`f_1=O(f_2)`$, so $`f_1f_2`$. ∎

###### 8.27 Example.

Let $`𝔥`$ be a real Lie subalgebra of $`𝔞+𝔫`$, and assume there does not exist a nonzero element $`u`$ of $`𝔥`$, such that $`x_u=0`$ and $`y_u=0`$. Then there exist $``$-linear transformations $`R,S:𝔽^{n2}𝔽`$, such that $`\eta _u=R(x_u)+S(y_u)`$ for all $`u𝔥`$. (Similarly, $`\varphi _u`$, $`𝗑_u`$, and $`𝗒_u`$ are also functions of $`(x_u,y_u)`$.) Furthermore, we have $`u|x_u|+|y_u|`$.

The following well-known result is a generalization of the fact that all norms on a finite-dimensional vector space are equivalent up to a bounded factor.

###### 8.28 Lemma.

If $`V`$ is any finite-dimensional real vector space, and $`f_1,f_2:V`$ are two continuous, homogeneous functions of the same degree, such that $`f_1^1(0)=f_2^1(0)=\{0\}`$, then $`f_1f_2`$.

###### Proof.

By continuity, the function $`f_1/f_2`$ attains a non-zero minimum and a finite maximum on the unit sphere. Because $`f_1/f_2`$ is homogeneous of degree zero, these values bound $`f_1/f_2`$ on all of $`V\{0\}`$. ∎

## 9. Existence of tessellations

In this section, we show how to construct several families of homogeneous spaces that have tessellations. All of these examples are based on a method of T. Kobayashi (see 9.1) that generalizes Example 1.10.

### 9A. The general Kulkarni-Kobayashi construction

As explained in the comments before Theorem 4.1, the following theorem is essentially due to T. Kobayashi.

###### 9.1 Theorem ((Kobayashi, cf. \[Kb1, Thm. 4.7\])).

If

* $`H`$ and $`L`$ are closed subgroups of $`G`$, with only finitely many connected components;
* $`L`$ acts properly on $`G/H`$;
* $`d(L)+d(H)=d(G)`$; and
* there is a cocompact lattice $`\mathrm{\Gamma }`$ in $`L`$,

then $`G/H`$ has a tessellation. (Namely, $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$.)

###### Proof.

Because $`\mathrm{\Gamma }`$ is a closed subgroup of $`L`$, we know that it acts properly on $`G/H`$ (see 2.4). Thus, it suffices to show that $`\mathrm{\Gamma }\backslash G/H`$ is compact.

From Lemma 3.5, we see that there is no harm in assuming $`HAN`$, and that there is a closed, connected subgroup $`L^{}`$ of $`G`$, such that

* $`L^{}`$ is conjugate to a subgroup of $`AN`$,
* $`d(L^{})+d(H)=d(G)`$, and
* $`L^{}C=LC`$, for some compact subset $`C`$ of $`G`$.

(Unfortunately, we cannot assume $`LAN`$: we may not be able to replace $`L`$ with $`L^{}`$, because there may not be a cocompact lattice in $`L^{}`$. For example, there is not lattice in $`AN`$, because any group with a lattice must be unimodular \[Rag, Rem. 1.9, p. 21\].)

It suffices to show that $`L^{}\backslash G/H`$ is compact. (Because $`L^{}LC`$ is compact, and $`\mathrm{\Gamma }\backslash L`$ is compact, this implies that $`\mathrm{\Gamma }\backslash G/H`$ is compact, as desired.)

We know that $`L^{}`$ acts properly on $`G/H`$ (see 2.4), so $`L^{}\times H`$ acts properly on $`G`$, with quotient $`L^{}\backslash G/H`$. Therefore, Lemma 3.19 implies that $`L^{}\backslash G/H`$ has the same homology as $`G`$; in particular,

$$_{dimK}(L^{}\backslash G/H)_{dimK}(G).$$

From the Iwasawa decomposition $`G=KAN`$, and because $`AN`$ is homeomorphic to $`^{d(G)}`$ (see 3.18 and 3.12), we know that $`G`$ is homeomorphic to $`K\times ^{d(G)}`$. Since $`^{d(G)}`$ is contractible, this implies that $`G`$ is homotopy equivalent to $`K`$, so $`G`$ and $`K`$ have the same homology; in particular,

$$_{dimK}(G)=_{dimK}(K)0.$$

Since

$`dim(L^{}\backslash G/H)`$ $`=dimGdimL^{}dimH`$
$`=dimG\left(d(L^{})+d(H)\right)`$
$`=dimGd(G)`$
$`=dim(KAN)dim(AN)`$
$`=dimK,`$
this implies that the top-dimensional homology of the manifold $`L^{}\backslash G/H`$ is nontrivial. Therefore $`L^{}\backslash G/H`$ is compact \[Dol, Cor. 8.3.4\], as desired. ∎

Our results for $`G=\mathrm{SU}(2,2m;𝔽)`$ are based on the following special case of the theorem. The converse of this corollary is proved in Section 11 (see 11.5).

Recall the equivalence relation $``$, introduced in Notation 1.15.

###### 9.2 Corollary ((Kobayashi \[Kb1, Prop. 4.9\])).

Let $`H`$ be a closed, connected subgroup of $`G=\mathrm{SU}(2,2m;𝔽)`$. If

* $`d(H)=2qm`$; and
* either $`H\mathrm{SU}(1,2m;𝔽)`$ or $`H\mathrm{Sp}(1,m;𝔽)`$,

then $`G/H`$ has a tessellation.

###### Proof.

Let $`L_+=\mathrm{SU}(1,2m;𝔽)`$ and $`L_{}=\mathrm{Sp}(1,m;𝔽)`$. By assumption, we have $`HL_\epsilon `$, for some $`\epsilon \{+,\}`$; let $`L=L_\epsilon `$. Because $`\mu (L_+)`$ and $`\mu (L_{})`$ are the two walls of $`A^+`$ (see 8.21), we know that $`L=L_\epsilon `$ acts properly on $`G/L_\epsilon `$ (see 2.17); since $`HL_\epsilon `$, this implies that $`L`$ acts properly on $`G/H`$ (see 2.4). Also, we have

$$d(L)+d(H)=2qm+2qm=d(G),$$

(see 7.15 and 1.13), and there is a cocompact lattice in $`L`$ (cf. 1.5(2)). Thus, the desired conclusion follows from Theorem 9.1. ∎

### 9B. Deformations of $`\mathrm{SO}(2,2m)/\mathrm{SU}(1,m)`$ and $`\mathrm{SU}(2,2m)/\mathrm{Sp}(1,m)`$

The homogeneous spaces described here were found by H. Oh and D. Witte \[OW2, Thms. 4.1 and 4.6\], \[OW3, Thm. 1.5\].

###### 9.3 Notation.

For any $``$-linear $`B:𝔽^{n2}𝔽^{n2}`$, we define

$$𝔥_B=\left\{\left(\begin{array}{ccccc}t& 0& x& \eta & 𝗑\\ & t& xB& 𝗑& \overline{\eta }\\ & & \mathrm{}\end{array}\right)|\begin{array}{c}t,\\ x𝔽^{n2},\\ \eta 𝔽,\\ 𝗑𝔽_{\text{imag}}\end{array}\right\}𝔞+𝔫.$$

We write $`xB`$, rather than $`Bx`$, because $`x`$ is a row vector.

It is easy to see, using, for instance, the formula for the bracket in (7.22), that if

(9.4)
$$\mathrm{Im}\left((vB)(wB)^{}\right)=\mathrm{Im}(vw^{})\text{ for every }v,w𝔽^{n2},$$

then $`𝔥_B`$ is a real Lie subalgebra of $`𝔞+𝔫`$; we let $`H_B`$ denote the corresponding connected Lie subgroup of $`AN`$.

From (1.14), we have

(9.5)
$$\begin{array}{cc}\hfill d(H_B)& =dim𝔥_B\hfill \\ & =dim+dim𝔽^{n2}+dim𝔽+dim𝔽_{\text{imag}}\hfill \\ & =1+q(n2)+q+(q1)\hfill \\ & =qn.\hfill \end{array}$$

###### 9.6 Remark.

Assume $`n=2m`$. By comparing (7.14) with (9.3), we see that there is a $``$-linear map $`B_0:𝔽^{2m2}𝔽^{2m2}`$, such that $`𝔰𝔭(1,m;𝔽)(𝔞+𝔫)=H_{B_0}`$ (and $`B_0`$ satisfies (9.4)). Thus, in general, $`H_B`$ is a deformation of $`𝔰𝔭(1,m;𝔽)(𝔞+𝔫)`$.

###### 9.7 Theorem ((Oh-Witte \[OW2, Thms. 4.1 and 4.6\])).

Let $`B:𝔽^{2m2}𝔽^{2m2}`$ be $``$-linear. If

* Condition (9.4) holds, and
* $`xB𝔽x`$, for every nonzero $`x𝔽^{2m2}`$,

then

1. $`\rho (h)h^2`$ for $`hH_B`$; and
2. $`\mathrm{SU}(2,2m;𝔽)/H_B`$ has a tessellation.

###### Proof.

(1) Given $`hH_B`$, write $`h=au`$, with $`aA`$ and $`uN`$. We may assume that $`a_{1,1}1`$ (by replacing $`h`$ with $`h^1`$ if necessary (see 8.15)). It suffices to show $`h^2=O\left(a_{1,1}a_{2,2}+|\mathrm{\Delta }(h)|\right)`$ (for then $`h^2=O\left(\rho (h)\right)`$, so Lemma 8.24(2) applies).

Case 1 . Assume $`a`$ is trivial.  From (7.18) and (9.3), we see that

$$h=O\left(|x_h|^2+|\eta _h|+|𝗑_h|\right),$$

so

$$h^2=O\left(|x_h|^4+|\eta _h|^2+|𝗑_h|^2\right).$$

From (7.18) and (8.5), we have

$$\mathrm{Re}\mathrm{\Delta }(h)=\frac{1}{4}\left(|x_h|^2|y_h|^2|xy^{}|^2\right)+\left(|\eta _h|^2+𝗑_h𝗒_h\right).$$

From (9.9), we see that $`|x|^2|xB|^2|x(xB)^{}|^2>0`$ for every nonzero $`x𝔽^{2m2}`$, so Lemma 8.28 implies

$$|x_h|^4|x_h|^2|y_h|^2|x_hy_h^{}|^2.$$

Also, because $`𝗒_h=𝗑_h`$ (and $`𝗑_h𝔽_{\text{imag}}`$), we have

$$|\eta _h|^2+𝗑_h𝗒_h=|\eta _h|^2+|𝗑_h|^20.$$

Thus,

$$\mathrm{Re}\mathrm{\Delta }(h)|x_h|^4+\left(|\eta _h|^2+|𝗑_h|^2\right),$$

so $`h^2=O\left(\mathrm{Re}\mathrm{\Delta }(h)\right)=O\left(\mathrm{\Delta }(h)\right)`$, as desired.

Case 2 . The general case.  From Case 9B, we know $`u^2=O\left(1+|\mathrm{\Delta }(u)|\right)`$. Then, because $`hau=a_{1,1}u`$, we have

$$h^2a_{1,1}^2u^2=O\left(a_{1,1}^2(1+|\mathrm{\Delta }(u)|)\right)=O\left(a_{1,1}^2+a_{1,1}^2|\mathrm{\Delta }(u)|\right).$$

Then, since $`a_{1,1}=a_{2,2}`$ and $`\mathrm{\Delta }(h)=a_{1,1}a_{2,2}\mathrm{\Delta }(u)`$, we conclude that $`h^2=O\left(a_{1,1}a_{2,2}+|\mathrm{\Delta }(h)|\right)`$, as desired.

(2) From (1) and 8.23(2), we see that $`H_B\mathrm{Sp}(1,m;𝔽)`$. Then, because $`d(H_B)=q(2m)`$ (see 9.5), Theorem 9.2 implies that $`\mathrm{SU}(2,2m;𝔽)/H_B`$ has a tessellation. ∎

###### 9.8 Lemma.

Let $`B:𝔽^{n2}𝔽^{n2}`$ be $``$-linear. Condition (9.4) holds if and only if either

1. $`𝔽=`$; or
2. $`𝔽=`$ and $`B^{}\mathrm{Sp}(2n4;)`$, where $`xB^{}=\overline{xB}`$ and we use the natural identification of $`^{n2}`$ with $`^{2n4}`$.

###### Proof.

Case 1 . Assume $`𝔽=`$.  Because $`\mathrm{Im}z=0`$ for every $`z=0`$, it is obvious that (9.4) holds.

Case 2 . Assume $`𝔽=`$.  If (9.4) holds, then

$`\mathrm{Im}\left((vB^{})(wB^{})^{}\right)`$ $`=\mathrm{Im}\left((\overline{vB})(\overline{wB})^{}\right)`$
$`=\mathrm{Im}\overline{\left((vB)(wB)^{}\right)}`$
$`=\mathrm{Im}\left((vB)(wB)^{}\right)`$
$`=\left(\mathrm{Im}(vw^{})\right)`$
$`=\mathrm{Im}(vw^{}),`$
so $`B^{}`$ is symplectic. The argument is reversible. ∎

###### 9.9 Remark.

* For $`𝔽=`$, the assumption that $`xB𝔽x`$ simply requires that $`B`$ have no real eigenvalues.
* For $`𝔽=`$, we do not know a good description of the linear transformations $`B`$ that satisfy $`xB𝔽x`$, although it is easy to see that this is an open set (and not dense). A family of examples was constructed by H. Oh and D. Witte (see 9.10 below).
* If $`n`$ is odd, then there does not exist $`B:𝔽^{n2}𝔽^{n2}`$ satisfying the assumption that $`xB𝔽x`$. For $`𝔽=`$, this is simply the elementary fact that a linear transformation on an odd-dimensional real vector space must have a real eigenvalue. For $`𝔽=`$, see Step 10B of the proof of Proposition 10.17.
* If $`n`$ is even, then, by varying $`B`$, one can obtain uncountably many pairwise non-conjugate subgroups $`H_B`$, such that $`\mathrm{SU}(2,n;𝔽)/H_B`$ has a tessellation. For $`𝔽=`$, this is proved in \[OW3, Thm. 1.5\]). For $`𝔽=`$, a similar argument can be applied to the examples constructed in (9.10) below.

###### 9.10 Example ((Oh-Witte \[OW2, Thm. 4.6(1)\])).

Assume $`n`$ is even, let $`B^{}\mathrm{SO}(n2;)`$, such that $`B^{}`$ has no real eigenvalue, and define an $``$-linear map $`B:^{n2}^{n2}`$ by $`xB=\overline{x}B^{}`$. Let us verify that $`B`$ satisfies the conditions of Theorem 9.7 (for $`𝔽=`$).

Let $`x_1,x_2,y_1,y_2^{n2}`$. From the definition of $`B`$, and because $`B^{}\mathrm{SO}(n2;)`$, we have

$`\mathrm{Im}\left(\left((x_1+ix_2)B\right)\right((y_1+iy_2)B)^{})`$ $`=\mathrm{Im}\left(\left((x_1ix_2)B^{}\right)\right((y_1iy_2)B^{})^{})`$
$`=i\left((x_1B^{})(y_2B^{})^{}(x_2B^{})(y_1B^{})^{}\right)`$
$`=i(x_1y_2^{}x_2y_1^{})`$
$`=\mathrm{Im}\left((x_1+ix_2)(y_1+iy_2)^{}\right).`$
Suppose $`Bx=\lambda x`$, for some $`\lambda `$. Because $`B\mathrm{SO}(2n4;)`$, we must have $`|\lambda |=1`$. Then

$$B^{}(x+\overline{\lambda x})=B^{}x+\overline{\lambda }B^{}\overline{x}=\overline{B^{}\overline{x}}+\overline{\lambda }Bx=\overline{Bx}+\overline{\lambda }(\lambda x)=\overline{\lambda x}+x.$$

Because $`B^{}`$ has no real eigenvalues, we know that $`1`$ is not an eigenvalue of $`B^{}`$, so we conclude that $`x+\overline{\lambda x}=0`$. Similarly, because $`1`$ is not an eigenvalue of $`B^{}`$, we see that $`x\overline{\lambda x}=0`$. Therefore

$$x=\frac{1}{2}\left((x+\overline{\lambda x})+(x\overline{\lambda x})\right)=\frac{1}{2}(0+0)=0.$$

### 9C. Deformations of $`\mathrm{SU}(2,2m)/\mathrm{SU}(1,2m)`$

These examples are new for $`𝔽=`$, but provide nothing interesting for $`𝔽=`$ (see 9.13(1)).

###### 9.11 Notation.

For $`c(0,1]`$, we define

$$𝔥_{[c]}=\left\{\left(\begin{array}{ccccc}t& \varphi & x& \mathrm{Re}\varphi +c\mathrm{Im}\varphi & 𝗑\\ & 0& 0& 0& \\ & & \mathrm{}\end{array}\right)|\begin{array}{c}t,\\ \varphi 𝔽,\\ x𝔽^{n2},\\ 𝗑𝔽_{\text{imag}}\end{array}\right\}.$$

It is easy to see, using, for instance, the formula for the bracket in (7.22), that $`𝔥_{[c]}`$ is a real Lie subalgebra of $`𝔞+𝔫`$ (even without the assumption that $`0<c1`$); we let $`H_{[c]}`$ be the corresponding connected Lie subgroup of $`AN`$.

From (1.14), we have

(9.12)
$$\begin{array}{cc}\hfill d(H_{[c]})& =dim𝔥_{[c]}\hfill \\ & =dim+dim𝔽+dim𝔽^{n2}+dim𝔽_{\text{imag}}\hfill \\ & =1+q+q(n2)+(q1)\hfill \\ & =qn.\hfill \end{array}$$

###### 9.13 Remark.

Let $`𝔰𝔲(1,n;𝔽)`$ be embedded into $`𝔰𝔲(2,n;𝔽)`$ as in 7.13.

1. If $`𝔽=`$, then $`c`$ is irrelevant in the definition of $`𝔥_{[c]}`$ (because $`\mathrm{Im}\varphi =0`$); therefore $`𝔥_{[c]}=𝔰𝔲(1,n;)(𝔞+𝔫)`$.
2. If $`𝔽=`$, then $`𝔥_{[1]}=𝔰𝔲(1,n;)(𝔞+𝔫)`$.

Thus, in general, $`𝔥_{[c]}`$ is either $`𝔰𝔲(1,n;𝔽)(𝔞+𝔫)`$ or a deformation of it.

###### 9.14 Theorem.

Assume $`𝔽=`$, and $`n=2m`$ is even. If $`c(0,1]`$, then

1. $`\rho (h)h`$ for $`hH_{[c]}`$; and
2. $`\mathrm{SU}(2,2m;𝔽)/H_{[c]}`$ has a tessellation.

###### Proof.

(1) Given $`hH_{[c]}`$, it suffices to show that $`\rho (h)=O(h)`$ (see 8.24(1)). Write $`h=au`$, with $`aA`$ and $`uN`$. We may assume that $`a_{1,1}1`$ (by replacing $`h`$ with $`h^1`$ if necessary (see 8.15)).

Let $`Q:^{n2}`$ be the real quadratic form

$$Q(\varphi ,x,\eta )=|x|^2+2\mathrm{Re}(\varphi \overline{\eta }),$$

and let $`V`$ be the $``$-subspace of $`^{n2}`$ defined by

$$V=\left\{(\varphi ,x,\eta )|\begin{array}{c}\varphi ,\\ x^{n2},\\ \eta =\mathrm{Re}\varphi +c\mathrm{Im}\varphi \end{array}\right\}.$$

Step 1 . For $`vV`$, we have $`Q(v)|\varphi |^2+|x|^2`$.  For $`(\varphi ,x,\eta )V\{0\}`$, we have

$`Q(\varphi ,x,\eta )`$ $`=|x|^2+2\mathrm{Re}(\varphi \overline{\eta })`$
$`=|x|^2+2\mathrm{Re}\left(\varphi (\overline{\mathrm{Re}\varphi +c\mathrm{Im}\varphi })\right)`$
$`=|x|^2+2(\mathrm{Re}\varphi )^22c(\mathrm{Im}\varphi )^2`$
$`>0`$
(because $`c>0`$ and $`\mathrm{Im}\varphi `$ is purely imaginary). Thus, the restriction of $`Q`$ to $`V`$ is positive definite, so the desired conclusion follows from Lemma 8.28.

Step 2 . We have $`u_{1,n+2}\left(|\varphi _u|^2+|x_u|^2\right)+|𝗑_u|`$.  From (7.19) (with $`𝗒=0`$), we have

$$\mathrm{Re}u_{1,n+2}=\left(\frac{1}{2}|x_u|^2+\mathrm{Re}(\varphi _u\overline{\eta _u})\right)|x_u|^2+2\mathrm{Re}(\varphi _u\overline{\eta _u})$$

and

$$\mathrm{Im}u_{1,n+2}=𝗑_u.$$

Then, from Step 9C, we see that $`\mathrm{Re}u_{1,n+2}|\varphi _u|^2+|x_u|^2`$, so

$$u_{1,n+2}|\mathrm{Re}u_{1,n+2}|+|\mathrm{Im}u_{1,n+2}|\left(|\varphi _u|^2+|x_u|^2\right)+|𝗑_u|,$$

as desired.

Step 3 . Completion of the proof.  From Step 9C, we have

$$h_{1,n+2}=a_{1,1}u_{1,n+2}a_{1,1}\left(|x_u|+|\varphi _u|\right)^2+a_{1,1}|𝗑_u|.$$

Also, from (7.19), we have

$$h_{jk}=\{\begin{array}{cc}O(1)\hfill & \text{if }j1\text{ and }kn+2\hfill \\ O\left(a_1(|\varphi _u|+|x_u|)\right)\hfill & \text{if }j=1\text{ and }kn+2\hfill \\ O\left(|\varphi _u|+|x_u|\right)\hfill & \text{if }j1\text{ and }k=n+2.\hfill \end{array}$$

Thus, it is easy to see that

$$\rho (h)=O\left(a_{1,1}|𝗑_u|+a_{1,1}(|\varphi _u|+|x_u|)^2\right)=O(h_{1,n+2})=O(h),$$

so the desired conclusion follows from Lemma 8.24(1).

(2) From (1) and 8.23(1), we see that $`H_{[c]}\mathrm{SU}(1,n)`$. Then, because $`d(H_{[c]})=2n`$ (see 9.12), Theorem 9.2 implies that $`\mathrm{SU}(2,2m;𝔽)/H_{[c]}`$ has a tessellation. ∎

###### 9.15 Remark.

Proposition 11.6 shows that if $`𝔽=`$, then $`H_{[c]}`$ is not conjugate to $`H_{[c^{}]}`$ unless $`c=c^{}`$ (for $`c,c^{}(0,1]`$). Thus, Theorem 9.14(2) implies that, by varying $`c`$, one obtains uncountably many nonconjugate subgroups $`H_{[c]}`$, such that $`\mathrm{SU}(2,2m)/H_{[c]}`$ has a tessellation.

### 9D. The product of two rank-one groups

###### 9.16 Proposition.

Let $`G=G_1\times G_2`$ be the direct product of two connected, linear, almost simple Lie groups $`G_1`$ and $`G_2`$ of real rank one, with finite center, and let $`H`$ be a nontrivial, closed, connected, proper subgroup of $`AN`$.

The homogeneous space $`G/H`$ has a tessellation if and only if, perhaps after interchanging $`G_1`$ and $`G_2`$, there is a continuous homomorphism $`\sigma :ANG_1ANG_2`$, such that

$$H=\{(h,\sigma (h))hANG_1\}.$$

###### Proof.

($``$) We may assume $`d(G_1)d(G_2)`$ (by interchanging $`G_1`$ and $`G_2`$ if necessary).

Case 1 . Assume $`HG_1e`$ and $`HG_2e`$.  For $`j=1,2`$, we know that $`HG_j`$ is not compact (see 3.15(3)), so Corollary 2.16 implies that there is a compact subset $`C_j`$ of $`G_j`$, such that $`C_j(HG_j)C_j=G_j`$. Then, letting $`C=C_1C_2`$, we have $`CHC=G`$, so Proposition 2.9 implies that $`G/H`$ does not have a tessellation. This is a contradiction.

Case 2 . Assume $`HG_1e`$ and $`HG_2=e`$.  From Corollaries 2.16 and 2.15, we know that there is a compact subset $`C`$ of $`AG_1`$, such that $`\mu (G_1)\mu (H)C`$. Therefore, Corollary 4.2 (with $`G_1`$ in the place of $`H_1`$) implies $`d(H)d(G_1)=dim(G_1AN)`$. Then, because $`HG_2=e`$ (and $`HAN`$), we conclude that $`H`$ is the graph of a homomorphism from $`G_1AN`$ to $`G_2AN`$, as desired.

Case 3 . Assume $`HG_1=e`$.  From Corollary 4.12, we know that $`dimHd(G_2)`$. Then, since $`HG_1=e`$, we conclude that $`H`$ is the graph of a homomorphism from $`G_2AN`$ to $`G_1AN`$. Interchanging $`G_1`$ and $`G_2`$ yields the desired conclusion.

($``$) We verify the hypotheses of Theorem 9.1, with $`G_2`$ in the role of $`L`$.

Let $`\overline{H}`$ be the image of $`H`$ under the natural homomorphism $`GG/G_2`$. Because $`HAN`$, we know that $`\overline{H}`$ is closed (see 3.15(1)). It is well known (and follows easily from (2.3)) that any closed subgroup acts properly on the ambient group, so this implies that $`\overline{H}`$ acts properly on $`G/G_2`$. From the definition of $`H`$, we have $`HG_2=e`$, so we conclude that $`H\overline{H}`$ acts properly on $`G/G_2`$; equivalently, $`G_2`$ acts properly on $`G/H`$ (cf. 2.3).

Because $`AN=(ANG_1)\times (ANG_2)`$, we have $`d(G)=d(G_1)+d(G_2)`$. Also, we have $`d(H)=dimH`$ (see 1.14) and, from the definition of $`H`$, we have $`dimH=dim(ANG_1)=d(G_1)`$. Therefore

$$d(H)+d(G_2)=d(G_1)+d(G_2)=d(G).$$

There is a cocompact lattice in $`G_2`$ (cf. 1.5(2)).

So Theorem 9.1 implies that $`G/H`$ has a tessellation. ∎

### 9E. T. Kobayashi’s examples of higher real rank

T. Kobayashi observed that, besides the examples with $`G=\mathrm{SO}(2,2n)`$ or $`\mathrm{SU}(2,2n)`$ (see 1.10), Theorem 9.1 can also be used to construct tessellations of some homogeneous spaces $`G/H`$ in which $`G`$ and $`H`$ are simple Lie groups with $`\text{-rank}G>2`$. He found one pair of infinite families, and several isolated examples.

###### 9.17 Theorem ((Kobayashi \[Kb5, Cor. 5.6\])).

Each of the following homogeneous spaces has a tessellation:

1. $`\mathrm{SO}(4,4n)/\mathrm{Sp}(1,n)`$;
2. $`\mathrm{SO}(4,4n)/\mathrm{SO}(3,4n)`$;
3. $`\mathrm{SO}(8,8)/\mathrm{SO}(8,7)`$;
4. $`\mathrm{SO}(8,8)/\mathrm{Spin}(8,1)`$;
5. $`\mathrm{SO}(4,4)/\mathrm{SO}(4,1)`$;
6. $`\mathrm{SO}(4,4)/\mathrm{Spin}(4,3)`$;
7. $`\mathrm{SO}(4,3)/\mathrm{SO}(4,1)`$;
8. $`\mathrm{SO}(4,3)/G_{2(2)}`$.

It would be very interesting to find other examples of simple Lie groups $`G`$ with reductive subgroups $`H`$ and $`L`$ that satisfy the hypotheses of Theorem 9.1.

###### 9.18 Remark.

Let $`G=\mathrm{SO}(4,4n)`$ and $`H^{}=\mathrm{Sp}(1,n)AN`$. From 9.17(1), we know that $`G/H`$ has a tessellation. H. Oh and D. Witte \[OW2, Thm. 4.6(2)\] pointed out that the deformations $`G/H_B`$ (where $`H_B`$ is as in Theorem 9.7, with $`𝔽=`$) also have tessellations, but it is not known whether there are other deformations of $`G/H^{}`$ that also have tessellations.

It does not seem to be known whether the other examples in Theorem 9.17 lead to nontrivial deformations, after intersecting $`H`$ with $`AN`$.

## 10. Large subgroups of $`\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(2,n)`$

This section presents a short proof of the results we need from \[OW1\] and \[IW\]. Those papers provide an approximate calculation of $`\mu (H)`$, for every closed, connected subgroup $`H`$ of $`\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$, respectively, but here we consider only subgroups of large dimension. Also, we do not need a complete description of the entire set $`\mu (H)`$; we are only interested in whether or not there is a curve $`h^t`$, such that $`\rho (h^t)h^t^k`$, for some $`k\{1,2\}`$. The main results of this section are Theorem 10.19 (for $`k=1`$) and Theorem 10.26 (for $`k=2`$). They give a sharp upper bound on $`d(H)`$, for subgroups $`H`$ that fail to contain such a curve, and, if $`n`$ is even, also provide a fairly explicit description of all the subgroups of $`AN`$ that attain the bound.

Because of the limited scope of this section, the proof here is shorter than the previous work, and we are able to give a fairly unified treatment of the two groups $`\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(2,n)`$. The arguments are elementary, but they involve case-by-case analysis and a lot of details, so they are not pleasant to read.

###### 10.1 Standing assumptions.

Throughout this section:

1. We use the notation of §7. (In particular, $`𝔽=`$ or $``$, and $`q=dim_{}𝔽`$.)
2. $`G=\mathrm{SU}(2,n;𝔽)`$.
3. $`n3`$.
4. $`H`$ is a closed, connected subgroup of $`AN`$ that is compatible with $`A`$ (see 10.2), so $`dimH=d(H)`$ (see 1.14).
5. $`U=HN`$. (Note that $`U`$ is connected (see 3.15(2)).)
6. $`𝔲_{\varphi =0}=\{u𝔲\varphi _u=0\}`$.
7. We use the notation of §8. (In particular, $`\rho (h)`$ is defined in (8.4) and $`\mathrm{\Delta }(h)`$ is defined in (8.5).)
8. Except in Subsection 10A, $`H`$ is compatible with $`A`$ (see 10.2).

### 10A. Subgroups compatible with $`A`$

Recall that the Real Jordan Decomposition of an element of $`G`$ is defined in (3.2); any element $`g`$ of $`AN`$ has a Real Jordan Decomposition $`g=au`$ (with $`c`$ trivial). If $`a`$ is an element of $`A`$, rather than only conjugate to an element of $`A`$, we could say that $`g`$ is “compatible with $`A`$.” We now define a similar, useful notion for subgroups of $`AN`$. Lemma 10.4 shows there is usually no loss of generality in assuming that $`H`$ is compatible with $`A`$, and Lemma 10.9 shows that the compatible subgroups can be described fairly explicitly.

###### 10.2 Definition (\[OW1, Defn. 2.2\]).

Let us say that $`H`$ is *compatible* with $`A`$ if $`HTUC_N(T)`$, where $`T=A(HN)`$, $`U=HN`$, and $`C_N(T)`$ denotes the centralizer of $`T`$ in $`N`$.

In preparation for the proofs of the main results, let us state a lemma that records a few of the nice properties of Jordan components.

###### 10.3 Lemma ((cf. \[Hm1, Lem. 15.3, p. 99\])).

Let $`g=auc`$ be the Real Jordan Decomposition of an element $`g`$ of $`G`$. Then

1. the Real Jordan Decomposition of $`\mathrm{Ad}g`$ is $`\mathrm{Ad}g=(\mathrm{Ad}a)(\mathrm{Ad}u)(\mathrm{Ad}c)`$; and
2. $`a`$, $`u`$, and $`c`$ all belong to the Zariski closure of $`g`$.

Therefore:

1. $`a`$, $`u`$, and $`c`$ each normalize any connected subgroup of $`G`$ that is normalized by $`g`$; and
2. if $`(\mathrm{Ad}g)vv+W`$, for some $`v𝔤`$ and some $`(\mathrm{Ad}g)`$-invariant subspace $`W`$ of $`𝔤`$, then

$$(\mathrm{Ad}a)v\text{}(\mathrm{Ad}u)v\text{, and }(\mathrm{Ad}c)v\text{ all belong to }v+W.$$

###### 10.4 Lemma (\[OW1, Lem. 2.3\]).

$`H`$ is conjugate, via an element of $`N`$, to a subgroup that is compatible with $`A`$.

###### Idea of proof.

Case 1 . Assume, for the Real Jordan Decomposition $`h=au`$ of each element $`h`$ of $`H`$, that $`a`$ and $`u`$ belong to $`H`$.  Let $`T`$ be a maximal split torus of $`H`$. (Recall that a split torus is a subgroup consisting entirely of hyperbolic elements.) Then $`T`$ is contained in some maximal split torus of $`G`$, that is, in some subgroup of $`G`$ conjugate to $`A`$; replacing $`H`$ by a conjugate, we may assume $`TA`$. In other words, we now know that $`HA`$ is a maximal split torus of $`H`$.

Given $`hH`$, we have the Real Jordan Decomposition $`h=au`$. By assumption, $`aH`$; thus, $`a`$ belongs to some maximal split torus $`T^{}`$ of $`H`$. A fundamental result of the theory of solvable algebraic groups implies that all maximal split tori of $`H`$ are conjugate via an element of $`HN`$ \[BT, Thm. 4.21\], so there is some $`xHN`$, such that $`x^1axA`$. Then $`T,x^1ax`$, being a subgroup of $`A`$, is a split torus. Thus, the maximality of $`T`$ implies that $`x^1axT`$; let $`t=x^1ax`$. Then

$$h=au=xtx^1u=t(t^1xt)x^1uT(HN).$$

Since $`hH`$ is arbitrary, we conclude that

(10.5)
$$H=T(HN),$$

so $`H`$ is compatible with $`A`$.

Case 2 . The general case.  Let

$$\overline{H}=a,uau=uaH\text{}a\text{ hyperbolic, }u\text{ unipotent}$$

be the subgroup of $`AN`$ generated by the Jordan Components of the elements of $`H`$. (Of course, since every element of $`H`$ has a Jordan Decomposition, we have $`H\overline{H}`$.) Then Case 10A applies to $`\overline{H}`$, so, replacing $`H`$ by a conjugate, we may assume $`\overline{H}=\overline{T}\overline{U}`$, where $`\overline{T}=\overline{H}A`$ and $`\overline{U}=\overline{H}N`$ (see 10.5).

Because $`\overline{H}`$ normalizes $`H`$ (see 10.3(a)), we know that $`\mathrm{Ad}_Gh`$ acts as the identity on $`\overline{𝔥}/𝔥`$, for all $`hH`$. Hence, Lemma 10.3(b) implies that $`\mathrm{Ad}_G\overline{h}`$ acts as the identity on $`\overline{𝔥}/𝔥`$, for all $`\overline{h}\overline{H}`$; therefore $`[\overline{H},\overline{H}]H`$. Also, we have $`[\overline{H},\overline{H}][AN,AN]N`$. Thus, letting $`U=HN`$, we have

(10.6)
$$[\overline{H},\overline{H}]HN=U.$$

Because $`\overline{T}A`$ and $`\overline{𝔲}`$ is $`(\mathrm{Ad}_G(T))`$-invariant, the adjoint action of $`\overline{T}`$ on $`\overline{𝔲}`$ is completely reducible, so (10.6) implies that there is a subspace $`𝔠`$ of $`\overline{𝔲}`$, such that $`[\overline{T},𝔠]=0`$ and $`𝔲+𝔠=\overline{𝔲}`$. Therefore, $`UC_{\overline{U}}(\overline{T})=\overline{U}`$, so

(10.7)
$$\overline{H}=\overline{T}\overline{U}=\overline{T}UC_{\overline{U}}(\overline{T})\overline{T}UC_N(\overline{T}).$$

Let $`\pi :ANA`$ be the projection with kernel $`N`$, and let $`T=\pi (H)`$. Then

$$T=\pi (H)\pi (\overline{H})=\overline{T},$$

so $`C_N(T)C_N(\overline{T})`$. For any $`hH`$, we know, from (10.7), that there exist $`t\overline{T}`$, $`uU`$ and $`cC_N(\overline{T})`$, such that $`h=tuc`$. Because $`ucN`$, we must have $`t=\pi (h)T`$ and, because $`C_N(T)C_N(\overline{T})`$, we have $`cC_N(T)`$. Therefore, $`hTUC_N(T)`$. We conclude that $`HTUC_N(T)`$, so $`H`$ is compatible with $`A`$. ∎

The preceding proposition shows that $`H`$ is conjugate to a subgroup $`H^{}`$ that is compatible with $`A`$. The subgroup $`H^{}`$ is usually not unique, however. The following lemma provides one way to change $`H^{}`$, often to an even better subgroup.

###### 10.8 Lemma.

Assume that $`H`$ is compatible with $`A`$, and let $`T=A(HN)`$. If $`uC_N(T)`$, then $`u^1Hu`$ is compatible with $`A`$.

###### Proof.

Let $`H^{}=u^1Hu`$, $`T^{}=A(H^{}N)`$, and $`U^{}=H^{}N`$. Because $`u`$ centralizes $`T`$, we have

$$u^1Tu=T.$$

Also, because $`uN`$, and $`N`$ is normal, we have $`u^1HuN=HN`$, so

$$u^1Tu=T=A(HN)=A(u^1HuN)=A(H^{}N)=T^{}.$$

Since $`uN`$, we have $`u^1Nu=N`$, so

$$u^1Uu=u^1(HN)u=(u^1Hu)(u^1Nu)=H^{}N=U^{},$$

and

$$u^1C_N(T)u=C_{u^1Nu}(u^1Tu)=C_N(T^{}).$$

Thus,

$$H^{}=u^1Huu^1TUC_N(T)u=\left(u^1Tu\right)\left(u^1Uu\right)\left(u^1C_N(T)u\right)=T^{}U^{}C_N(T^{}),$$

as desired. ∎

###### 10.9 Lemma (\[OW1, Lem. 2.4\]).

If $`H`$ is compatible with $`A`$, then either

1. $`H=(HA)(HN)`$; or
2. there is a positive root $`\omega `$, a nontrivial group homomorphism $`\psi :\mathrm{ker}\omega N_\omega N_{2\omega }`$, and a closed, connected subgroup $`U`$ of $`N`$, such that

1. $`H=\{a\psi (a)a\mathrm{ker}\omega \}U`$;
    2. $`U`$ is normalized by both $`\mathrm{ker}\omega `$ and $`\psi (\mathrm{ker}\omega )`$; and
    3. $`U\psi (\mathrm{ker}\omega )=e`$.

###### Proof.

Because $`H`$ is compatible with $`A`$, we have $`HTUC_N(T)`$, where $`T=A(HN)`$ and $`U=HN`$. We may assume that $`HTU`$, for otherwise (1) holds. Therefore $`C_N(T)e`$. Because $`𝔫`$ is a sum of root spaces, this implies that there is a positive root $`\omega `$, such that $`T\mathrm{ker}\omega `$. Because $`\text{-rank}G=2`$, we have $`dim(\mathrm{ker}\omega )=1`$, so we must have $`T=\mathrm{ker}\omega `$ (otherwise we would have $`T=e`$, so $`H=U=TU`$; hence (1) holds). Therefore, $`C_N(T)=U_\omega U_{2\omega }`$.

Because $`UHTUC_N(T)`$, we have $`H=U\left[H\left(TC_N(T)\right)\right]`$, so there is a nontrivial one-parameter subgroup $`\{x^t\}`$ in $`H\left(TC_N(T)\right)`$ that is not contained in $`U`$. Because $`T`$ centralizes $`C_N(T)`$, we may write $`x^t=a^tu^t`$ where $`\{a^t\}`$ is a one-parameter subgroup of $`T`$ and $`\{u^t\}`$ is a one-parameter subgroup of $`C_N(T)`$. Furthermore, this decomposition is unique, because $`TC_N(T)=e`$. (In fact, $`x^t=a^tu^t`$ is the Real Jordan Decomposition of $`x^t`$.) Define $`\psi :\mathrm{ker}\omega U_\omega U_{2\omega }`$ by $`\psi (a^t)=u^t`$ for all $`t`$.

(2a) For all $`t`$, we have $`a^t\psi (a^t)=a^tu^t=x^tH`$, which establishes one inclusion of (2a). The other will follow if we show that $`dimHdimU=1`$, so suppose $`dimHdimU2`$. Then Lemma 10.11(1) implies that $`AH`$, so it follows from Lemma 3.21 (with $`T=A`$ and $`\omega =0`$) that $`H=A(HN)`$, contradicting our assumption that $`HTU`$.

(2b) Because $`x^tH`$, we know that each of $`a^t`$ and $`u^t`$ normalizes $`H`$ (see 10.3(a)). Being in $`AN`$, they also normalize $`N`$. Therefore, they normalize $`HN=U`$.

(2c) Suppose $`U\psi (\mathrm{ker}\omega )e`$. Because the intersection $`U\psi (\mathrm{ker}\omega )`$ is connected (see 2b and 3.15(2)), and $`dim(\mathrm{ker}\omega )=1`$, we must have $`\psi (\mathrm{ker}\omega )U`$. Therefore $`a^t=x^tu^tHU=H`$, so $`TH`$. This contradicts our assumption that $`HTU`$. ∎

###### 10.10 Corollary.

If $`H`$ is compatible with $`A`$, then $`A(HN)`$ normalizes $`HN`$.

###### 10.11 Lemma (\[OW1, Lem. 2.8\]).

If $`dim\left(H/(HN)\right)dimA`$, then

1. $`H`$ contains a conjugate of $`A`$; and
2. $`H`$ is a Cartan-decomposition subgroup.

###### Proof.

(1) Let $`\pi :ANA`$ be the projection with kernel $`N`$, and let $`\overline{H}`$ be the Zariski closure of $`H`$. From the structure theory of solvable algebraic groups \[Br2, Thm. 10.6(4), pp. 137–138\], we know that $`\overline{H}=TU`$ is the semidirect product of a torus $`T`$ and and unipotent subgroup $`UN`$. Replacing $`H`$ by a conjugate under $`N`$, we may assume that $`TA`$. Since

$$dimAdim\left(H/(HN)\right)=dim\left(\pi (H)\right)dimA,$$

we must have $`\pi (H)=A`$, so

$$A=\pi (H)\pi (\overline{H})=\pi (TU)=T\overline{H}$$

normalizes $`H`$ (see 10.12). Then, since $`\pi (H)=A`$, we conclude that $`AH`$ (see 3.21).

(2) From (1), we see that, by replacing $`H`$ with a conjugate subgroup, we may assume $`AH`$. Because $`A`$ is a Cartan-decomposition subgroup (see 2.6), this implies $`H`$ is a Cartan-decomposition subgroup. ∎

The following basic result was used twice in the above arguments.

###### 10.12 Lemma ((cf. \[Zm2, pf. of Thm. 3.2.5, p. 42\])).

If $`H`$ is a closed, connected subgroup of $`G`$, then the Zariski closure of $`H`$ normalizes $`H`$.

### 10B. Subgroups with no nearly linear curve

Our goal is to prove Theorem 10.19; we begin with some preliminary results.

First, an observation that simplifies the calculations in some cases, by allowing us to assume that $`x_u=0`$.

###### 10.13 Lemma.

Let $`u𝔲`$. If $`dim_𝔽(𝔽x_u+𝔽y_u)1`$ and $`y_u0`$, then there is some $`gN_\alpha `$, such that

1. $`x_{g^1ug}=0`$,
2. $`\varphi _{g^1ug}=\varphi _u`$, and
3. $`y_{g^1ug}=y_u`$.

###### Proof.

Because $`dim_𝔽(𝔽x_u+𝔽y_u)1`$ and $`y_u0`$, there is some $`\lambda 𝔽`$, such that $`x_u=\lambda y_u`$. Let

* $`v`$ be the element of $`𝔫_\alpha `$ with $`\varphi _\alpha =\lambda `$,
* $`g=\mathrm{exp}(v)N_\alpha `$, and
* $`w=g^1ug`$.

From (7.23), we see that

* $`\varphi _w=\varphi _u`$,
* $`x_w=x_u+\varphi _vy_u=0`$, and
* $`y_w=y_u`$,

as desired. ∎

###### 10.14 Proposition.

If there does not exist a continuous curve $`h^t\mathrm{}`$ in $`U`$, such that $`\rho (h^t)h^t`$, then

1. for every nonzero element $`z`$ of $`𝔡_𝔥`$, we have $`|\eta _z|^2+𝗑_z𝗒_z0`$; and
2. for every element $`u`$ of $`𝔲_{\varphi =0}`$, such that $`dim_𝔽(𝔽x_u+𝔽y_u)=1`$, we have

$$𝗑_u|y_u|^2+𝗒_u|x_u|^2+2\mathrm{Im}(x_uy_u^{}\eta _u^{})0.$$

###### Proof of the contrapositive.

(1) Suppose there is a nonzero element $`z`$ of $`𝔡_𝔥`$ with $`\mathrm{\Delta }(z)=0`$. Let $`h^t=\mathrm{exp}(tz)=\mathrm{Id}+tz`$ (see 7.18). We have

$$h_{j,k}=\{\begin{array}{cc}O(t)\hfill & \text{for all }j,k,\hfill \\ O(1)\hfill & \text{if }(j,k)\{1,2\}\times \{n+1,n+2\}.\hfill \end{array}$$

Then, because $`\mathrm{\Delta }(h^t)=0`$, it is easy to see that $`\rho (h)t`$. Also, we have $`h^t=\mathrm{Id}+tzt`$, so $`\rho (h^t)th^t`$, as desired.

(2) Suppose there is an element $`u`$ of $`𝔲_{\varphi =0}`$, such that $`dim_𝔽(𝔽x_u+𝔽y_u)=1`$, and

(10.15)
$$𝗑_u|y_u|^2+𝗒_u|x_u|^2+2\mathrm{Im}(x_uy_u^{}\eta _u^{})=0.$$

Let $`h=h^t=\mathrm{exp}(tu)`$.

Case 1 . Assume $`x_u=0`$.  Because $`dim_𝔽(𝔽x_u+𝔽y_u)=1`$, we must have $`y_u0`$. Then, from (10.15), we know that $`𝗑_u=0`$. So, from (7.18), we see that

* $`h_{2,n+1}|y_u|^2t^2t^2`$,
* $`h_{j,k}=O(t)`$ whenever $`(j,k)(2,n+1)`$, and
* $`h_{j,k}=O(1)`$ whenever $`j2`$ and $`kn+1`$.

This implies that $`\rho (h)t^2h`$.

Case 2 . Assume $`y_u=0`$.  This is similar to Case 10B. (In fact, this can be obtained as a corollary of Case 10B by replacing $`H`$ with its conjugate under the Weyl reflection corresponding to the root $`\alpha `$.)

Case 3 . Assume $`y_u0`$.  Because $`dim_𝔽(𝔽x_u+𝔽y_u)=1`$, Lemma 10.13 implies there is some $`gN_\alpha `$, such that, letting $`w=g^1ug`$, we have $`\varphi _w=\varphi _u=0`$, $`x_w=0`$, and $`y_w=y_u0`$. We show below that (10.15) is satisfied with $`w`$ in the place of $`u`$, so, from Case 10B, we conclude that $`\rho \left(\mathrm{exp}(tw)\right)\mathrm{exp}(tw)`$. Thus, the desired conclusion follows from Lemma 8.25 (with $`k=1`$).

To complete the proof, we now show that (10.15) is satisfied with $`w`$ in the place of $`u`$. (This can be verified by direct calculation, but we give a more conceptual proof.) Because $`g^1N_\alpha `$, multiplication by $`g^1`$ on the left performs a row operation on the first two rows of $`h`$; likewise, multiplication by $`g`$ on the right performs a column operation on the last two columns of $`h`$. These operations do not change the determinant $`\mathrm{\Delta }(h)`$: thus

$$\mathrm{\Delta }\left(\mathrm{exp}(tw)\right)=\mathrm{\Delta }\left(\mathrm{exp}(tu)\right).$$

From (7.18) and the definition of $`\mathrm{\Delta }`$, we see that

$$\mathrm{\Delta }\left(\mathrm{exp}(tu)\right)=\frac{1}{4}\left(|x_u|^2|y_u|^2|x_uy_u^{}|^2\right)t^4+\left(𝗑_u|y_u|^2+𝗒_u|x_u|^2+2\mathrm{Im}(x_uy_u^{}\eta _u^{})\right)t^3+O(t^2).$$

Because $`dim_𝔽(𝔽x_u+𝔽y_u)=1`$, we have $`|x_u|^2|y_u|^2|x_uy_u^{}|^2=0`$, so this simplifies to

$$\mathrm{\Delta }\left(\mathrm{exp}(tu)\right)=\left(𝗑_u|y_u|^2+𝗒_u|x_u|^2+2\mathrm{Im}(x_uy_u^{}\eta _u^{})\right)t^3+O(t^2).$$

Thus, (10.15) is equivalent to the condition that $`\mathrm{\Delta }\left(\mathrm{exp}(tu)\right)=O(t^2)`$. Then, since

$$\mathrm{\Delta }\left(\mathrm{exp}(tw)\right)=\mathrm{\Delta }\left(\mathrm{exp}(tu)\right)=O(t^2),$$

we conclude that (10.15) is also valid for $`w`$. ∎

###### 10.16 Lemma.

If there does not exist a continuous curve $`h^t\mathrm{}`$ in $`U`$, such that $`\rho (h^t)h^t`$, then

$$dim𝔡_𝔥+dim𝔲/𝔲_{\varphi =0}2q1.$$

Furthermore, if equality holds, and $`𝔽=`$, then $`𝔲=𝔲_{\varphi =0}`$ and $`dim𝔡_𝔥=3`$.

###### Proof.

Case 1 . Assume $`𝔽=`$.  Because $`|\eta _z|^2+𝗑_z𝗒_z`$ is a quadratic form of signature $`(1,3)`$ on $`𝔡`$, we know, from 10.14(1), that $`dim𝔡_𝔥3=2q1`$.

Thus, we may assume $`𝔲/𝔲_{\varphi =0}0`$, so there is some $`u𝔲`$, such that $`\varphi _u0`$.

Subcase 1.1 . Assume there exists $`z𝔡_𝔥`$, such that $`𝗒_z=0`$ and $`\eta _z\varphi _u`$.  From (7.22), we see that $`[u,z]𝔡_𝔥`$, with $`𝗑_{[u,z]}=\mathrm{Im}(\varphi _u\overline{\eta _z})0`$ and $`𝗒_{[u,z]}=\eta _{[u,z]}=0`$. This contradicts 10.14(1).

Subcase 1.2 . Assume there exists $`z𝔡_𝔥`$, such that $`𝗒_z0`$.  From (7.22), we see that $`[u,z]`$ is an element of $`𝔡_𝔥`$, such that $`𝗒_{[u,z]}=0`$, and $`\eta _{[u,z]}=\varphi _u𝗒_z`$ is a purely imaginary multiple of $`\varphi _u`$. So Subcase 10B applies (with $`[u,z]`$ in the place of $`z`$).

Subcase 1.3 . Assume $`𝗒_z=0`$ and $`\eta _z\varphi _u`$, for all $`z𝔡_𝔥`$.  From 10.14(1), we see that $`𝔡_𝔥𝔫_{2\alpha +2\beta }=\{0\}`$, so the assumption of this subcase implies $`dim𝔡_𝔥1`$. Thus,

$$dim𝔡_𝔥+dim𝔲/𝔲_{\varphi =0}1+2=3=2q1,$$

so the desired inequality holds.

If equality holds, then $`dim𝔡_𝔥=1`$ and $`dim𝔲/𝔲_{\varphi =0}=2`$. Thus, we may choose $`z𝔡_𝔥`$, such that $`z0`$, and $`u^{}𝔲`$, such that $`\varphi _u+\varphi _u^{}=`$. From the assumption of this subcase, we know that $`\eta _z\varphi _u`$; thus, $`\eta _z\varphi _u^{}`$. Therefore, Subcase 10B applies, with $`u^{}`$ in the place of $`u`$.

Case 2 . Assume $`𝔽=`$.  Because $`dim𝔡_𝔥dim𝔫_{\alpha +2\beta }=1`$ and $`dim𝔲/𝔲_{\varphi =0}dim𝔫_\alpha =1`$, the desired inequality holds unless $`𝔡_𝔥0`$ and $`𝔲/𝔲_{\varphi =0}0`$. Thus, we may assume there is some nonzero $`z𝔡_𝔥`$ and some $`u𝔲`$, such that $`\varphi _u0`$.

Subcase 2.1 . Assume $`y_u=0`$.  We may assume $`|x_u|^2+\varphi _u\eta _u0`$ (by replacing $`u`$ with $`u+z`$, if necessary). Let $`h^t=\mathrm{exp}(tu)`$. From (7.19), we see that $`h_{1,n+2}^tt^2`$, but

$$h_{j,k}^t=\{\begin{array}{cc}O(t)\hfill & \text{if }(j,k)(1,n+2)\text{,}\hfill \\ O(1)\hfill & \text{if }j1\text{ and }kn+2.\hfill \end{array}$$

Therefore $`\rho (h^t)=O(t^2)=O(h^t)`$, so Lemma 8.24(1) implies that $`\rho (h^t)h^t`$. This is a contradiction.

Subcase 2.2 . Assume $`y_u0`$.  Let $`v`$ be the element of $`𝔫_\beta `$ with $`y_v=(1/\varphi _u)x_u`$, and let $`w=\mathrm{exp}(v)u\mathrm{exp}(v)`$. Then $`x_w=0`$ (see 7.23 and 7.22). Thus, by replacing $`H`$ with the conjugate $`\mathrm{exp}(v)H\mathrm{exp}(v)`$ (see 8.25), we may assume $`x_u=0`$. For any large real number $`t`$, let $`h=h^t`$ be the element of $`\mathrm{exp}(tu+𝔫_{\alpha +2\beta })`$ that satisfies $`\eta _h=\varphi _hy_h^2/12`$. Then, from (7.17), we see that

$$h=\left(\begin{array}{ccccc}1& \varphi _h& \frac{1}{2}\varphi _hy_h& \frac{1}{4}\varphi _h|y_h|^2& \frac{1}{8}\varphi _h^2|y_h|^2\\ & 1& y_h& \frac{1}{2}|y_h|^2& \frac{1}{4}\varphi |y_h|^2\\ & & 1& y_h^{}& \frac{1}{2}\varphi _hy_h^{}\\ & & & 1& \varphi _h\\ & & & & 1\end{array}\right).$$

Clearly, we have $`h\varphi _h^2|y_h|^2`$.

A calculation shows that $`\mathrm{\Delta }(h)=0`$, and certain other $`2\times 2`$ minors also have cancellation. With this in mind, it is not difficult to verify that $`\rho (h)\varphi _h^2|y_h|^2h`$ (see \[OW1, Case 3 of pf. of 5.12($`32`$)\] for details). This is a contradiction. ∎

###### 10.17 Proposition.

If there does not exist a continuous curve $`h^t\mathrm{}`$ in $`U_{\varphi =0}`$, such that $`\rho (h^t)h^t`$, then

$$dim𝔲_{\varphi =0}/𝔡_𝔥\{\begin{array}{cc}q(n2)\hfill & \text{if }n\text{ is even}\hfill \\ q(n3)\hfill & \text{if }n\text{ is odd and }n3\hfill \\ q1\hfill & \text{if }n=3.\hfill \end{array}$$

Furthermore,

1. if equality holds, and $`n`$ is even, then $`dim_𝔽(𝔽x_u+𝔽y_u)=2`$, for every $`u𝔲_{\varphi =0}𝔡_𝔥`$;
2. if equality holds, and $`n=3`$, then $`dim𝔡_𝔥q`$.

###### Proof.

By passing to a subgroup, we may assume $`𝔲=𝔲_{\varphi =0}`$. Let $`V`$ be the projection of $`𝔲`$ to $`𝔫_\beta +𝔫_{\alpha +\beta }`$; then $`dimV=dim𝔲/dim𝔡_𝔥`$.

Case 1 . Assume $`dim_𝔽(𝔽x_u+𝔽y_u)=2`$ for every $`u𝔲𝔡_𝔥`$.

Subcase 1.1 . Assume $`n`$ is even.  From Theorem 10.14(1), we know that $`V`$ does not intersect $`𝔫_\beta `$ (or $`𝔫_{\alpha +\beta }`$, either, for that matter), so

$$dimV+dim𝔫_\beta dim(𝔫_\beta +𝔫_{\alpha +\beta })=dim𝔫_\beta +dim𝔫_{\alpha +\beta }.$$

Therefore

$$dim𝔲/dim𝔡_𝔥=dimVdim𝔫_{\alpha +\beta }=q(n2),$$

as desired. (If equality holds, then we have Conclusion (a).)

Subcase 1.2 . Assume $`n`$ is odd.

Step 1.2.1 . We have $`dim𝔲/𝔡_𝔥q(n3)`$.  Suppose not: then

$$dimVq(n3)+1.$$

(This will lead to a contradiction.) Let $`X=\{x_vvV\}`$, so $`X`$ is a $``$-subspace of $`𝔽^{n2}`$. For each $`xX`$, there is some $`v=v(x)V`$, such that $`x_v=x`$; define $`f(x)=y_{v(x)}`$. By the assumption of this case, we know

$$V𝔫_\beta =\{0\},$$

so $`v(x)`$ is uniquely determined by $`x`$; thus, $`f:X𝔽^{n2}`$ is a well-defined $``$-linear map. Also, again from the assumption of this case, we know that

(10.18) $`f(x)𝔽x`$ for every nonzero $`xX`$.
Because $`V𝔫_\beta =0`$, we have

$$dimX=dimVq(n3)+1=dim𝔽^{n2}(q1).$$

If $`𝔽=`$ (that is, if $`q=1`$), this implies $`X=^{n2}`$, so $`f`$ is defined on all of $`^{n2}`$. Because $`n`$ is odd, this implies that $`f`$ has a real eigenvalue, which contradicts (10.18).

We may now assume $`𝔽=`$. Let

* $`E=(X\times ^{n2})/`$, where $`(x,v)(x,v)`$,
* $`X`$ be the projective space of the real vector space $`X`$, and
* $`\zeta (x,v)=[x]X`$, for $`(x,v)E`$,

so $`(E,\zeta )`$ is a vector bundle over $`X`$.

Define $`g:X^{n2}`$ by $`g(x)=ix`$. Any $``$-linear transformation $`Q:X^{n2}`$ is a continuous function, such that $`Q(x)=Q(x)`$ for all $`xX`$; that is, a section of $`(E,\zeta )`$. Thus, $`\mathrm{Id}`$, $`f`$, and $`g`$ each define a section of $`(E,\zeta )`$. Furthermore, these three sections are pointwise linearly independent over $``$, because (10.18) implies that $`x`$, $`f(x)`$, and $`ix`$ are linearly independent over $``$, for every nonzero $`xX`$. On the other hand, the theory of characteristic classes \[MS, Prop. 4, p. 39\] implies that $`(E,\zeta )`$ does not have three pointwise $``$-linearly independent sections (see \[IW, Lem. 8.2\] for details). This is a contradiction.

Step 1.2.2 . Completion of the proof of Subcase 10B.  From Step 10B, we see that the desired inequality holds.

We may now assume $`n=3`$ and $`dim𝔲/𝔡_𝔥=q1`$. Since $`dim𝔲/𝔡_𝔥q(n3)=0`$, we must have $`q=1`$, so $`𝔽=`$. Therefore $`𝔫_{2\alpha }=𝔫_{2\beta }=0`$, so

$$dim𝔡_𝔥dim𝔫_{\alpha +2\beta }=q,$$

as desired.

Case 2 . Assume there is some $`v𝔲𝔡_𝔥`$, such that $`dim_𝔽(𝔽x_v+𝔽y_v)=1`$.

Subcase 2.1 . Assume $`x_v=0`$.  Since $`v𝔡_𝔥`$, we must have $`y_v0`$. Then $`𝗑_{v+z}0`$ for every $`z𝔡_𝔥`$ (otherwise 10.14(2) yields a contradiction); this implies

$$𝗑_v0,$$

and

$`𝗑_z=0`$ for every $`z𝔡_𝔥`$.
Because $`𝗑_v0`$, we know that $`𝔽`$; so

$$𝔽=.$$

Since $`𝗑_z=0`$ for every $`z𝔡_𝔥`$, but $`𝔡_𝔥𝔫_{2\beta }=0`$ (see 10.14(1)), we must have $`\eta _z0`$ for every $`z𝔡_𝔥`$. Therefore

$$dim𝔡_𝔥dim𝔫_{\alpha +2\beta }=q=2.$$

Let $`p:V𝔫_{\alpha +\beta }`$ be the natural projection. Note that

$$dim\mathrm{ker}p=1.$$

(If $`v^{}𝔲`$, with $`x_v^{}=0`$, then there is some $`t`$, such that $`𝗑_{v^{}+tv}=𝗑_v^{}+t𝗑_v=0`$. We also have $`x_{v^{}+tv}=0`$, so, from 10.14(2), we see that $`v^{}+tv𝔡_𝔥`$. Thus $`v^{}v+𝔡_𝔥`$. So $`\mathrm{ker}p=(v+𝔡_𝔥)/𝔡_𝔥`$ is 1-dimensional.)

Because $`𝗑_z=0`$ for every $`z𝔡_𝔥`$, and $`𝔲`$ is a Lie algebra, we see, from (7.22), that $`p(V)`$ must be a totally isotropic subspace for the symplectic form $`i\mathrm{Im}(x\stackrel{~}{x}^{})`$, so

$$dimp(V)\frac{1}{2}dim𝔫_{\alpha +\beta }=n2.$$

Therefore

$$dim𝔲/𝔡_𝔥=dimV=dimp(V)+dim\mathrm{ker}p(n2)+1=n1.$$

This completes the proof if $`n3`$:

* If $`n`$ is even, then, because $`n4`$, we have $`n1<2(n2)=q(n2)`$.
* If $`n>3`$ is odd, then $`n5`$, so $`n12(n3)=q(n3)`$.

Now let $`n=3`$, and suppose $`dimV=2`$. (This will lead to a contradiction.) Because equality is attained in the proof above, we must have $`dimp(V)=n2=1`$, so there exists $`w𝔲`$ with $`x_w0`$. For $`t`$, let $`w_t=w+tv`$. Then

$`𝗑_{w_t}|y_{w_t}|^2+𝗒_{w_t}|x_{w_t}|^2+2\mathrm{Im}(x_{w_t}y_{w_t}^{}\eta _{w_t})`$ $`=t^3𝗑_v|y_v|^2+O(t^2)`$
$`\{\begin{array}{cc}+𝗑_v\mathrm{}\hfill & \text{as }t\mathrm{}\hfill \\ 𝗑_v\mathrm{}\hfill & \text{as }t\mathrm{}.\hfill \end{array}`$
Thus, this expression changes sign, so it must vanish for some $`t`$. On the other hand, since $`n=3`$, we have $`dim_{}(x+y)1`$ for every $`x,y^{n2}=`$. Thus 10.14(2) yields a contradiction.

Subcase 2.2 . Assume $`y_v=0`$.  This is similar to Subcase 10B. (In fact, this can be obtained as a corollary of Subcase 10B by replacing $`H`$ with its conjugate under the Weyl reflection corresponding to the root $`\alpha `$.)

Subcase 2.3 . Assume $`y_v0`$.  Because $`dim_𝔽(𝔽x_v+𝔽y_v)=1`$, Lemma 10.13 implies there is some $`gN_\alpha `$, such that, letting $`w=g^1vg`$, we have $`\varphi _w=\varphi _u=0`$, $`x_w=0`$, and $`y_w=y_v0`$. There is no harm in replacing $`H`$ with $`g^1Hg`$ (see 8.25). Then Subcase 10B applies (with $`w`$ in the place of $`v`$). ∎

###### 10.19 Theorem.

Recall that Assumptions 10.1 are in effect.

If there does not exist a continuous curve $`h^t\mathrm{}`$ in $`H`$, such that $`\rho (h^t)h^t`$, then

(10.20)
$$dimH\{\begin{array}{cc}qn\hfill & \text{if }n\text{ is even}\hfill \\ q(n1)\hfill & \text{if }n\text{ is odd.}\hfill \end{array}$$

Furthermore, if equality holds, and $`n`$ is even, then

1. $`𝔥=(\mathrm{ker}\alpha )𝔲`$;
2. $`\varphi _u=0`$ for every $`u𝔲`$;
3. $`dim_𝔽(𝔽x_u+𝔽y_u)=2`$, for every $`u𝔲𝔡_𝔥`$;
4. $`|\eta _z|^2+𝗑_z𝗒_z0`$ for every nonzero $`z𝔡_𝔥`$;
5. $`dim𝔲/𝔡_𝔥=q(n2)`$; and
6. $`dim𝔡_𝔥=2q1`$.

###### Proof.

Let

$$m=\{\begin{array}{cc}q(n2)\hfill & \text{if }n\text{ is even}\hfill \\ q(n3)\hfill & \text{if }n5\text{ is odd}\hfill \\ q1\hfill & \text{if }n=3.\hfill \end{array}$$

From Lemmas 10.11(1) and 10.16, and Proposition 10.17, we have

(10.21)
$$\begin{array}{cc}\hfill dimH& dim𝔥/𝔲+(dim𝔲/𝔲_{\varphi =0}+dim𝔡_𝔥)+dim𝔲_{\varphi =0}/𝔡_𝔥\hfill \\ & 1+(2q1)+m\hfill \\ & =2q+m.\hfill \end{array}$$

This implies the desired inequality, unless $`n=3`$, $`𝔽=`$, and we have equality in both Lemma 10.16 and Proposition 10.17. This is impossible, because equality in Lemma 10.16 requires $`dim𝔡_𝔥=3`$, but Proposition 10.17(b) implies $`dim𝔡_𝔥2`$.

Assumption. In the remainder of the proof, we assume that equality holds in (10.20), and that $`n`$ is even.  Proposition 10.14(1) implies (4).

Case 1 . Assume $`𝔽=`$.  Because equality holds, Lemma 10.16 implies (2) and (6). Then Proposition 10.17(a) implies (3) (because $`𝔲=𝔲_{\varphi =0}`$). Since $`𝔲=𝔲_{\varphi =0}`$ (see 2) and equality holds in (10.20), we have

(10.22)
$$dim𝔲/𝔡_𝔥=dim𝔲_{\varphi =0}/𝔡_𝔥=m=q(n2)$$

and

(10.23)
$$dim𝔡_𝔥=dim𝔲/𝔲_{\varphi =0}+dim𝔡_𝔥=2q1,$$

so (5) and (6) hold.

Let $`T=A(HN)`$. Corollary 10.10 implies that $`T`$ normalizes $`𝔲`$, so, from (3) and Lemma 3.21, we see that $`T\mathrm{ker}\alpha `$. On the other hand, $`dimT=dim𝔥/𝔲`$, so, from equality in (10.21), we conclude that $`dimT=1`$. Therefore $`T=\mathrm{ker}\alpha `$.

Suppose $`\psi :\mathrm{ker}\alpha N_\alpha `$ is any continuous group homomorphism, such that $`\psi (\mathrm{ker}\alpha )`$ normalizes $`U`$. From (3) and (7.22), we see that $`N_{N_\alpha }(U)=e`$, so $`\psi `$ must be trivial. This implies that 10.9(2) cannot apply here, so 10.9(1) yields (1).

Case 2 . Assume $`𝔽=`$.  Proposition 10.17(a) implies that $`dim_𝔽(𝔽x_u+𝔽y_u)=2`$ for every $`u𝔲_{\varphi =0}𝔡_𝔥`$.

Suppose (2) is false. Then there is some $`u𝔲`$, such that $`\varphi _u0`$. Also, because $`dim𝔲_{\varphi =0}/𝔡_𝔥=m>0`$, we may fix some $`v𝔲_{\varphi =0}𝔡_𝔥`$. Then, letting $`w=[u,v]`$, we see, from (7.22), that $`y_w=0`$ and $`x_w0`$, so $`dim_𝔽(𝔽x_w+𝔽y_w)=1`$. This contradicts the conclusion of the preceding paragraph.

Conclusion (3) follows from (2) and 10.17(a).

Conclusion (1) can be established by arguing as in the last two paragraphs of Case 10B.

Equations (10.22) and (10.23) establish (5) and (6). ∎

### 10C. Subgroups with no nearly quadratic curve

Our goal is to prove Theorem 10.26; we start with two preliminary results.

###### 10.24 Lemma.

If there does not exist a continuous curve $`h^t\mathrm{}`$ in $`U`$, such that $`\rho (h^t)h^t^2`$, then

1. for every element $`u`$ of $`𝔲_{\varphi =0}`$, we have $`dim_𝔽(𝔽x_u+𝔽y_u)1`$;
2. for every element $`z`$ of $`𝔡_𝔥`$, we have $`|\eta _z|^2+𝗑_z𝗒_z=0`$; and
3. for every element $`u`$ of $`𝔲`$, such that $`\varphi _u0`$, $`y_u=0`$, and $`𝗒_u=0`$, we have $`|x_u|^2+2\mathrm{Re}(\varphi _u\overline{\eta _u})0`$.

###### Proof of the contrapositive..

(1) Suppose there is an element $`u`$ of $`𝔲_{\varphi =0}`$, such that $`dim_𝔽(𝔽x_u+𝔽y_u)=2`$. Let $`h^t=\mathrm{exp}(tu)`$. Then, from (7.18), we see that $`h^t=O(t^2)`$. Furthermore,

$$\mathrm{\Delta }(h^t)=det\left(\begin{array}{cc}\eta _ut\frac{1}{2}x_uy_u^{}t^2& 𝗑_ut\frac{1}{2}|x_u|^2t^2\\ 𝗒_ut\frac{1}{2}|y_u|^2t^2& \overline{\eta _u}t\frac{1}{2}y_ux_u^{}t^2\end{array}\right)=\frac{1}{4}||x_u||y_u|^2|x_uy_u^{}|^2|t^4+O(t^3).$$

Because $`dim_𝔽(𝔽x_u+𝔽y_u)=2`$, we have $`|x_u||y_u|>|x_uy_u^{}|`$, so $`|x_u|^2|y_u|^2|x_uy_u^{}|^20`$; therefore $`\mathrm{\Delta }(h^t)t^4`$, so

$$h^t^2=O(t^4)=O\left(\mathrm{\Delta }(h^t)\right)=O\left(\rho (h^t)\right),$$

so Lemma 8.24(2) implies that $`\rho (h^t)h^t^2`$, as desired.

(2) Suppose there is an element $`z`$ of $`𝔡_𝔥`$, such that $`|\eta _z|^2+𝗑_z𝗒_z0`$; in other words, we have $`\mathrm{\Delta }(z)0`$. Let $`h^t=\mathrm{exp}(tz)=\mathrm{Id}+tz`$ (see 7.18). Then $`h^t=O(t)`$ and

$$t^2\mathrm{\Delta }(z)t^2=\mathrm{\Delta }(h^t)=O\left(\rho (h^t)\right),$$

so

$$h^t^2=O(t^2)=O\left(\rho (h^t)\right),$$

so Lemma 8.24(2) implies that $`\rho (h^t)h^t^2`$, as desired.

(3) Suppose there is an element $`u`$ of $`𝔲`$, such that $`\varphi _u0`$, $`y_u=0`$, $`𝗒_u=0`$, and $`|x_u|^2+2\mathrm{Re}(\varphi _u\overline{\eta _u})=0`$. Let $`h^t=\mathrm{exp}(tu)`$. From (7.19), we see that $`h^t=\mathrm{Id}+tu`$ (note that, because $`|x_u|^2+2\mathrm{Re}(\varphi _u\overline{\eta _u})=0`$, we have $`\mathrm{Re}h_{1,n+2}^2=0`$). Then $`h^t=O(t)`$ and

$$\rho (h^t)\left|det\left(\begin{array}{cc}h_{1,2}^t& h_{1,n+2}^t\\ h_{n+1,2}^t& h_{n+1,n+2}^t\end{array}\right)\right|=\left|det\left(\begin{array}{cc}t\varphi _u& \\ 0& t\varphi _u^{}\end{array}\right)\right|t^2.$$

So $`h^t^2=O(t^2)=O\left(\rho (h^t)\right)`$. Thus, Lemma 8.24(2) implies that $`\rho (h^t)h^t^2`$, as desired. ∎

The following lemma obtains a dimension bound from Condition 10.24(1).

###### 10.25 Lemma.

If $`V`$ is a $``$-subspace of $`𝔽^{n2}𝔽^{n2}`$, such that $`dim_𝔽(𝔽x+𝔽y)1`$ for every $`(x,y)V`$, then either

1. $`dimVq(n2)`$; or
2. $`n=3`$ and $`dimV2q`$.

###### Proof.

Because $`dim_{}𝔽^{n2}=q(n2)`$, we may assume that there exist nonzero $`x_0,y_0𝔽^{n2}`$, such that $`(x_0,0)V`$ and $`(0,y_0)V`$ (otherwise, the projection to one of the factors of $`𝔽^{n2}𝔽^{n2}`$ is injective when restricted to $`V`$, so (1) holds). Then $`(x_0,y_0)V`$, so, by assumption, we have $`dim_𝔽(𝔽x_0+𝔽y_0)1`$. Because $`x_0`$ and $`y_0`$ are nonzero, this implies $`𝔽x_0=𝔽y_0`$.

Step 1 . For all $`(x,y)V`$, we have $`y𝔽x_0`$.  We may assume $`y0`$ (otherwise the desired conclusion is obvious). Then, since $`dim_𝔽(𝔽x+𝔽y)1`$, we conclude that $`x𝔽y`$. Similarly, because

$$(x+x_0,y)=(x,y)+(x_0,0)V+V=V,$$

we must have $`x+x_0𝔽y`$. Therefore

$$x_0=(x+x_0)x𝔽y𝔽y=𝔽y.$$

Since $`x_00`$, this implies $`𝔽x_0=𝔽y`$, so $`y𝔽x_0`$, as desired.

Step 2 . We have $`V𝔽y_0𝔽x_0`$.  Given $`(x,y)V`$, Step 10C asserts that $`y𝔽x_0`$. By symmetry (interchanging the two factors of $`𝔽^{n2}𝔽^{n2}`$), we must also have $`x𝔽y_0`$. So $`(x,y)𝔽y_0𝔽x_0`$, as desired.

Step 3 . Completion of the proof.  From Step 10C, we have

$$dimVdim_{}(𝔽y_0𝔽x_0)=2q.$$

If $`n4`$, then (1) holds; otherwise, (2) holds. ∎

###### 10.26 Theorem.

Recall that Assumptions 10.1 are in effect.

If there does not exist a continuous curve $`h^t\mathrm{}`$ in $`H`$, such that $`\rho (h^t)h^t^2`$, then $`dimHqn`$.

Furthermore, if equality holds, then $`H`$ is of the form $`H=TU`$, where

1. $`T=\mathrm{ker}\beta `$,
2. $`𝔲=\left((𝔫_\alpha +𝔫_{\alpha +\beta }+𝔫_{\alpha +2\beta })𝔲\right)+𝔫_{2\alpha +2\beta }`$, and
3. $`|x_u|^2+2\mathrm{Re}(\varphi _u\overline{\eta _u})0`$ for every $`u𝔲𝔫_{2\alpha +2\beta }`$.

###### Proof.

Note that

$$dim𝔥/𝔲1$$

(see 10.11(1)) and

$$dim𝔲/𝔲_{\varphi =0}dim𝔫_\alpha =q.$$

Step 1 . We have $`dim𝔲_{\varphi =0}/𝔡_𝔥q(n2)`$.  Suppose not. Let $`V`$ be the projection of $`𝔲_{\varphi =0}`$ to $`𝔫_\beta +𝔫_{\alpha +\beta }`$. We have

$$dimV=dim𝔲_{\varphi =0}/𝔡_𝔥>q(n2),$$

and, for every $`u𝔲_{\varphi =0}`$ with $`x_u0`$, we have $`dim_𝔽(𝔽x_u+𝔽y_u)1`$ (see 10.24(1)), so Lemma 10.25 implies that $`n=3`$. Therefore $`dim𝔫_\beta =dim𝔫_{\alpha +\beta }=q`$. Then, because $`dimV>q(n2)=q`$, we know that $`V𝔫_\beta 0`$ and $`V𝔫_{\alpha +\beta }0`$; thus, there exist $`u,v𝔲_{\varphi =0}`$, such that

* $`x_u=0`$, $`y_u0`$; and
* $`x_v0`$, $`y_v=0`$.

Therefore $`[u,v]`$ is a nonzero element of $`𝔫_{\alpha +2\beta }`$ (see 7.22), so $`\mathrm{\Delta }\left([u,v]\right)0`$. This contradicts Lemma 10.24(2).

Step 2 . We have $`dim𝔡_𝔥q1`$.  Suppose not: then, because $`dim𝔫_{2\alpha +2\beta }=q1`$, there is some $`u𝔡_𝔥𝔫_{2\alpha +2\beta }`$, and, because $`dim𝔫_{2\beta }=q1`$, there is some nonzero $`v𝔡_𝔥`$, such that $`𝗒_v=0`$. We must have $`\eta _v=0`$ (otherwise 10.24(2) yields a contradiction); thus $`v𝔫_{2\alpha +2\beta }`$. We must have $`𝗒_u0`$ (otherwise 10.24(2) yields a contradiction). Thus, we see that

$$|\eta |^2+𝗑_{u+tv}𝗒_{u+tv}=|\eta _u|^2+(𝗑_u+t𝗑_v)(t𝗒_u)$$

is nonconstant as a function of $`t`$, so 10.24(2) yields a contradiction.

Step 3 . The desired inequality.  We have

$`dim𝔥`$ $`dim𝔥/𝔲+dim𝔲/𝔲_{\varphi =0}+dim𝔲_{\varphi =0}/𝔡_𝔥+dim𝔡_𝔥`$
$`1+q+q(n2)+(q1)`$
$`=qn,`$
as desired.

Assumption. In the remainder of the proof, we assume that $`dimH=qn`$.  We must have equality throughout the preceding paragraphs.

Step 4 . We have $`V𝔫_{\alpha +\beta }`$.  Suppose not: then there is some $`v𝔲_{\varphi =0}`$, such that $`y_v0`$. Let $`u𝔲𝔲_{\varphi =0}`$ and $`w=[u,v]`$. Then, from (7.22), we see that $`y_w=0`$ and $`x_w0`$, and that $`[v,w]𝔫_{\alpha +2\beta }+𝔫_{2\alpha +2\beta }`$. From 10.24(1), we have $`x_v𝔽y_v`$ and $`x_{v+2}𝔽y_{v+w}=𝔽y_v`$, so

$$x_w=x_{v+w}x_v𝔽y_v𝔽y_v=𝔽y_v.$$

Therefore $`x_wy_w^{}0`$, so $`\eta _{[v,w]}0`$ (see 7.22), so 10.24(2) yields a contradiction.

Step 5 . We have $`𝔡_𝔥=𝔫_{2\alpha +2\beta }`$.  From Step 10C, together with the fact that

$$dimV=dim𝔲_{\varphi =0}/𝔡_𝔥=q(n2)=dim𝔫_{\alpha +\beta },$$

we conclude that $`V=𝔫_{\alpha +\beta }`$. Therefore,

$$𝔲_{\varphi =0}+𝔡=V+𝔡=𝔫_{\alpha +\beta }+𝔡,$$

so

$`𝔲`$ $`[𝔲_{\varphi =0},𝔲_{\varphi =0}]`$
$`=[𝔲_{\varphi =0}+𝔡,𝔲_{\varphi =0}+𝔡]`$
$`=[𝔫_{\alpha +\beta }+𝔡,𝔫_{\alpha +\beta }+𝔡]`$
$`=[𝔫_{\alpha +\beta },𝔫_{\alpha +\beta }]`$
$`=𝔫_{2\alpha +2\beta }.`$
Because $`dim𝔡_𝔥=q1=dim𝔫_{2\alpha +2\beta }`$, we must have $`𝔡_𝔥=𝔫_{2\alpha +2\beta }`$.

Step 6 . We have $`𝔡_𝔥=𝔫_{2\alpha +2\beta }`$. Let $`T=(HN)A`$ be the projection of $`H`$ to $`A`$. Then there exists $`\sigma \{\beta ,\alpha +\beta ,\alpha +2\beta \}`$, such that $`T=\mathrm{ker}(\alpha \sigma )`$, and, in the notation of Lemma 3.21, we have

$$𝔲=(𝔲𝔫^{=\alpha })+(𝔲𝔫^\alpha ).$$

Because $`T`$ normalizes $`𝔲`$ (see 10.10), we know, from Lemma 3.21, that $`𝔲=(𝔲𝔫^{=\alpha })+(𝔲𝔫^\alpha )`$. Since $`dim𝔲/𝔲_{\varphi =0}=q`$, we know that $`𝔲𝔫^{=\alpha }`$ projects nontrivially (in fact, surjectively) to $`𝔫_\alpha `$. On the other hand, we know that $`𝔲𝔫_\alpha =0`$ (otherwise 10.24(3) yields a contradiction). Therefore $`𝔫^{=\alpha }𝔫_\alpha `$, so there must be a positive root $`\sigma \alpha `$, such that $`\sigma |_T=\alpha |_T`$. Then $`T\mathrm{ker}(\alpha \sigma )`$; since $`dimT=dimH/U=1`$, we must have $`T=\mathrm{ker}(\alpha \sigma )`$.

Because $`𝔲𝔫^\alpha 𝔲_{\varphi =0}`$, we have

$$dim(𝔲𝔫^{=\alpha })dim\frac{𝔲𝔫^{=\alpha }}{𝔲_{\varphi =0}𝔫^{=\alpha }}=dim\frac{𝔲/(𝔲𝔫^\alpha )}{𝔲_{\varphi =0}/(𝔲𝔫^\alpha )}=dim\frac{𝔲}{𝔲_{\varphi =0}}=q.$$

Then, since $`𝔲𝔫_\alpha =0`$, we must have

$$dim𝔫^{=\alpha }dim(𝔲𝔫^{=\alpha })+dim𝔫_\alpha q+q=2q.$$

By inspection, we see that this implies $`\sigma \{2\alpha ,2\beta \}`$, so we conclude that $`\sigma \{\beta ,\alpha +\beta ,\alpha +2\beta \}`$, as desired.

Step 7 . We have $`\sigma \{\alpha +\beta ,\alpha +2\beta \}`$, $`T=\mathrm{ker}\beta `$, and $`𝔫^{=\alpha }=𝔫_\alpha +𝔫_{\alpha +\beta }+𝔫_{\alpha +2\beta }`$.  Since $`\mathrm{ker}\beta =\mathrm{ker}2\beta `$, it suffices to show $`\sigma \beta `$. Thus, let us suppose $`\sigma =\beta `$. (This will lead to a contradiction.) We have $`𝔫^{=\alpha }=𝔫_\alpha +𝔫_\beta `$ (and recall that $`𝔲𝔫_\alpha =\{0\}`$), so there is some $`u𝔲`$, such that $`\varphi _u0`$ and $`y_u0`$. Because $`V=𝔫_{\alpha +\beta }`$, we have

$$dim\{vVx_v𝔽y_u\}=q>dim𝔽_{\text{imag}},$$

so there is some $`v𝔲_{\varphi =0}`$, such that $`0x_v𝔽y_u`$ and $`𝗒_v=0`$. Then $`[u,v]𝔫_{\alpha +2\beta }+𝔫_{2\alpha +2\beta }`$, with $`\eta _{[u,v]}0`$ (see 7.22), so 10.24(2) yields a contradiction.

Step 8 . We have $`H=(HA)(HN)`$.  Suppose not: because $`T=\mathrm{ker}\beta `$, we conclude that there is some nonzero $`w𝔫_\beta +𝔫_{2\beta }`$, such that $`w`$ normalizes $`𝔲`$ (see 10.9).

If $`y_w0`$, then, because $`V=𝔫_{\alpha +\beta }`$, there is some $`v𝔲_{\varphi =0}`$, such that $`y_w𝔽x_v`$ and $`𝗒_v=0`$. Then $`[w,v]𝔫_{\alpha +2\beta }+𝔫_{2\alpha +2\beta }`$, with $`\eta _{[w,v]}0`$ (see 7.22), so 10.24(2) yields a contradiction.

If $`y_w=0`$, then, since $`w0`$, we must have $`𝗒_w0`$. There is some $`v𝔲`$ with $`\varphi _v0`$. Then $`[w,v]𝔫_{\alpha +2\beta }+𝔫_{2\alpha +2\beta }`$, with $`\eta _{[w,v]}0`$ (see 7.22), so 10.24(2) yields a contradiction.

Step 9 . Completion of the proof.  (1) From Step 10C, We know that $`H=TU`$, and, from Step 10C, that $`T=\mathrm{ker}\beta `$.

(2) Since $`𝔫^{=\alpha }=𝔫_\alpha +𝔫_{\alpha +\beta }+𝔫_{\alpha +2\beta }`$, it suffices to show $`𝔲𝔫^\alpha =𝔫_{2\alpha +2\beta }`$: given $`v𝔲𝔫^\alpha `$, we wish to show $`v𝔫_{2\alpha +2\beta }`$. Because $`V=𝔫_{\alpha +\beta }`$, we know that $`y_v=0`$. Thus, all that remains is to show that $`𝗒_v=0`$. If not, then choosing $`u𝔲`$ with $`\varphi _u0`$, we see that $`\eta _{[u,v]}0`$ (see 7.22). So 10.24(2) yields a contradiction.

(3) From Lemma 10.24(3), we know that $`|x_u|^2+2\mathrm{Re}(\varphi _u\overline{\eta _u})0`$ for every $`u𝔲𝔫_{2\alpha +2\beta }`$. ∎

## 11. Homogeneous spaces of $`\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(2,n)`$

This section proves two main results. Both assume that $`G`$ is either $`\mathrm{SO}(2,n)`$ or $`\mathrm{SU}(2,n)`$.

1. Theorem 11.1 shows that if $`n`$ is odd, and one or two specific homogeneous spaces of $`G`$ do not have tessellations, then no interesting homogeneous space of $`G`$ has a tessellation.
2. Theorem 11.2 shows that if $`n`$ is even, then certain deformations of the examples found by R. Kulkarni and T. Kobayashi (see 1.10) are essentially the only interesting homogeneous spaces of $`G`$ that have tessellations.

The classification results of §10 (specifically, Theorems 10.26 and 10.19) play a crucial role in the proofs.

We use the notation $`\mathrm{SU}(2,n;𝔽)`$ of Section 7, to provide a fairly unified treatment of $`\mathrm{SO}(2,n)`$ and $`\mathrm{SU}(2,n)`$.

###### 11.1 Theorem.

Assume $`G=\mathrm{SU}(2,2m+1;𝔽)`$ with $`m1`$, and let $`H`$ be any closed, connected subgroup of $`G`$, such that neither $`H`$ nor $`G/H`$ is compact.

If Conjecture 1.17 is true, then $`G/H`$ does not have a tessellation.

###### Proof.

Assume Conjecture 1.17 is true, and suppose $`\mathrm{\Gamma }`$ is a crystallographic group for $`G/H`$. (This will lead to a contradiction.) Let

$`H_1=\mathrm{SU}(1,2m+1;𝔽)`$ and $`H_2=\mathrm{Sp}(1,m;𝔽)`$
(see 7.12). From (7.15), we have $`d(H_1)=q(2m+1)`$ and $`d(H_2)=q(2m)`$, where $`q=dim_{}𝔽`$. We may assume that $`HAN`$ (see 3.5), and that $`H`$ is compatible with $`A`$ (see 10.4).

Because $`H`$ is not a Cartan-decomposition subgroup (see 2.9), the contrapositive of Proposition 8.19 implies, for some $`k\{1,2\}`$, that there does not exist a continuous curve $`h^t\mathrm{}`$ in $`H`$, such that $`\rho (h^t)h^t^k`$. Therefore, either Theorem 10.19 (if $`k=1`$) or Theorem 10.26 (if $`k=2`$) implies that $`d(H)q(2m+1)=d(H_1)`$.

We consider two cases.

Case 1 . Assume that $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H_1`$.  Theorem 4.1(2) (combined with the fact that $`d(H)d(H_1)`$) implies that $`G/H_1`$ has a tessellation. This contradicts either Theorem 1.18 (if $`𝔽=`$) or Conjecture 1.17c (if $`𝔽=`$).

Case 2 . Assume that $`\mathrm{\Gamma }`$ does not act properly discontinuously on $`G/H_1`$.  From Lemma 8.21, we know that $`\mu (H_1)`$ and $`\mu (H_2)`$ are the two walls of $`A^+`$, so Corollary 4.11 (combined with the assumption of this case) implies that $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H_2`$. Therefore, since Conjecture 1.17ab asserts that $`G/H_2`$ does not have a tessellation, the contrapositive of Theorem 4.1(2) (with $`H_2`$ in the role of $`H_1`$) implies that $`d(H)>d(H_2)=q(2m)`$. Hence, the contrapositive of Theorem 10.19 implies there is a continuous curve $`h^t\mathrm{}`$ in $`H`$, such that $`\rho (h^t)h^t`$. Thus, there is a compact subset $`C`$ of $`G`$, such that $`H_1CHC`$ (see 8.22). Since $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H`$, this implies that $`\mathrm{\Gamma }`$ acts properly discontinuously on $`G/H_1`$ (see 2.4). This contradicts the assumption of this case. ∎

###### 11.2 Theorem.

Assume $`G=\mathrm{SU}(2,2m;𝔽)`$ with $`m2`$, and let $`H`$ be a closed, connected, nontrivial, proper subgroup of $`AN`$.

The homogeneous space $`G/H`$ has a tessellation if and only if either

1. there is an $``$-linear map $`B:𝔽^{n2}𝔽^{n2}`$, such that

1. $`\mathrm{Im}\left((vB)(wB)^{}\right)=\mathrm{Im}(vw^{})`$ for every $`v,w𝔽^{n2}`$ (see 9.8), and
    2. $`xB𝔽x`$, for every nonzero $`x𝔽^{n2}`$ (see 9.9), and
    3. $`H`$ is conjugate to $`H_B`$ (see 9.3 and 9.6); or
2. $`𝔽=`$ and $`H`$ is conjugate to $`\mathrm{SU}(1,2m;)AN`$ (see 7.12); or
3. $`𝔽=`$ and there exists $`c(0,1]`$, such that $`H`$ is conjugate to $`H_{[c]}`$ (see 9.11).

###### Proof.

($``$) See (1) Theorem 9.7(2), (2) Theorem 9.2 (and 7.15), or (3) Theorem 9.14(2).

($``$) Let $`n=2m`$, so $`G=\mathrm{SU}(2,n;𝔽)`$. By combining Remark 1.14, Corollary 4.12, Lemma 8.21, and Remark 7.15, we see that

$$dimH=d(H)\mathrm{min}\{d\left(\mathrm{SU}(1,n;𝔽)\right),d\left(\mathrm{Sp}(1,m;𝔽)\right)\}=qn.$$

Also, we may assume $`H`$ is compatible with $`A`$ (see 10.4). Because $`H`$ is not a Cartan-decomposition subgroup (see 2.9), Proposition 8.19 implies that one of the following two cases applies.

Case 1 . Assume there does not exist a continuous curve $`h^t\mathrm{}`$ in $`H`$, such that $`\rho (h^t)h^t`$.  Since $`dimHqn`$, Theorem 10.19 implies that $`dimH=qn`$, and that $`H`$ is of the form $`H=TU`$ (with $`UN`$), where

1. $`T=\mathrm{ker}\alpha `$;
2. $`\varphi _u=0`$ for every $`u𝔲`$;
3. $`dim_𝔽(𝔽x_u+𝔽y_u)=2`$, for every $`u𝔲𝔡_𝔥`$;
4. $`|\eta _z|^2+𝗑_z𝗒_z0`$ for every nonzero $`z𝔡_𝔥`$;
5. $`dim𝔲/𝔡_𝔥=q(n2)`$; and
6. $`dim𝔡_𝔥=2q1`$.

Step 1.1 . We may assume that $`𝔡_𝔥=\{z𝔡𝗑_z=𝗒_z\}`$.  Because $`dim𝔡_𝔥=2q1`$ (see vi), it suffices to show that $`𝗑_z=𝗒_z`$ for all $`z𝔡_𝔥`$. This is trivially true if $`𝔽=`$, as $`𝗑_z,𝗒_z𝔽_{\text{imag}}=\{0\}`$ in this case. Thus, we assume $`𝔽=`$.

For any $`z𝔡_𝔥`$ with $`\eta _z=𝗒_z=0`$, we know, from (iv), that $`z=0`$; therefore, Lemma 8.26(1) implies there exist $``$-linear maps $`R:i`$ and $`S:ii`$, such that $`𝗑_z=R(\eta _z)+S(𝗒_z)`$ for all $`z𝔡_𝔥`$. More concretely, we may say that there exist $`\lambda `$ and $`c`$, such that $`𝗑_z=\mathrm{Im}(\lambda \eta _z)+c𝗒_z`$ for all $`z𝔡_𝔥`$.

Let $`v`$ be the element of $`𝔫_\alpha `$ with $`\varphi _v=\overline{\lambda }/2`$, and let $`H^{}=\mathrm{exp}(v)H\mathrm{exp}(v)`$ be the conjugate of $`H`$ by $`\mathrm{exp}(v)`$. Then $`H^{}`$ satisfies the conditions imposed on $`H`$ (note that $`H^{}`$, like $`H`$, is compatible with $`A`$ (see 10.8)), so there exist $`\lambda ^{}`$ and $`c^{}`$, such that $`𝗑_z^{}=\mathrm{Im}(\lambda ^{}\eta _z^{})+c^{}𝗒_z^{}`$ for all $`z^{}𝔡_𝔥^{}`$. Given $`z^{}𝔡_𝔥^{}`$ with $`𝗒_z^{}=0`$, let $`z=\mathrm{exp}(v)z\mathrm{exp}(v)`$. Because $`𝗒_z^{}=0`$, we have $`[[z^{},v],v]=0`$, so, from Remark 7.23 and (7.22), we see that

$$𝗒_z=𝗒_z^{}=0,$$

$$\eta _z=\eta _z^{}\varphi _v𝗒_z^{}=\eta _z^{},$$

and

$$𝗑_z=𝗑_z^{}+2\mathrm{Im}(\varphi _v\overline{\eta _z^{}})=𝗑_z^{}+2\mathrm{Im}\left((\overline{\lambda }/2)\overline{\eta _z}\right)=𝗑_z^{}+\mathrm{Im}(\lambda \eta _z)=𝗑_z^{}+𝗑_z.$$

Therefore

$$0=𝗑_z^{}=\mathrm{Im}(\lambda ^{}\eta _z^{})+c^{}𝗒_z^{}=\mathrm{Im}(\lambda ^{}\eta _z^{}).$$

Since $`\eta _z^{}`$ is arbitrary, this implies $`\lambda ^{}=0`$. Thus, by replacing $`H`$ with $`H^{}`$, we may assume that $`\lambda =0`$. This means that $`𝗒_z=c𝗑_z`$ for all $`z𝔡_𝔥`$.

From (vi) (and because $`𝔽=`$, so $`q=2`$), we know that $`dim𝔡_𝔥=3>1`$, so there is some nonzero $`w𝔡_𝔥`$, such that $`𝗒_w=0`$. (So $`𝗑_w=c𝗒_w=0`$.) Then $`|\eta _w|^2+𝗑_w𝗒_w=|\eta _w|^2>0`$, so we see, from (iv), that $`|\eta _z|^2+𝗑_z𝗒_z>0`$ for every nonzero $`z𝔡_𝔥`$. Now, since

$$dim𝔡_𝔥=3>2=dim𝔫_{\alpha +2\beta },$$

there is some nonzero $`z𝔡_𝔥`$, such that $`\eta _z=0`$. We have

$$0<|\eta _z|^2+𝗑_z𝗒_z=0+c𝗒_z^2.$$

Because $`𝗒_z`$ is pure imaginary, we know that $`𝗒_z^2<0`$, so this implies that $`c<0`$. Thus, replacing $`H`$ by a conjugate under a diagonal matrix, we may assume $`c=1`$, as desired.

Step 1.2 . Setting $`𝔲^{}=(𝔫_\alpha +𝔫_{\alpha +\beta })𝔲`$, we have $`𝔲=𝔲^{}+𝔡_𝔥`$.  Since $`T=\mathrm{ker}\alpha `$ (see i), we have

* $`\beta |_T=(\alpha +\beta )|_T`$,
* $`2\beta |_T=(\alpha +2\beta )|_T=(2\alpha +2\beta )|_T`$, and
* $`\beta |_T2\beta |_T`$.

Thus, in the notation of Lemma 3.21, we have $`𝔫^{=\beta }𝔲=𝔲^{}`$ and $`𝔫^\beta 𝔲=𝔡_𝔥`$, so $`𝔲=𝔲^{}𝔡_𝔥`$, as desired. (Note that this is a direct sum of vector spaces, not of Lie algebras: we have $`[𝔲^{},𝔲^{}]𝔡_𝔥`$.)

Step 1.3 . Completion of the proof of Case 11.  For any $`u𝔲^{}`$ with $`x_u=0`$, we have

$$dim_𝔽(𝔽x_u+𝔽y_u)=dim_𝔽𝔽y_u1<2,$$

so $`u𝔲^{}𝔡_𝔥=\{0\}`$ (see iii); therefore, Lemma 8.26(1) implies there is a $``$-linear map $`B:𝔽^{n2}𝔽^{n2}`$, such that $`y_u=x_uB`$ for all $`u𝔲^{}`$. Then, because

$$dim𝔲_{\alpha +\beta }=dim_{}𝔽^{n2}=q(n2)=dim𝔲^{}$$

(see v), we must have

$$𝔲^{}=\{u𝔫_\beta +𝔫_{\alpha +\beta }y_u=x_uB\}.$$

Combining this with (i) and the conclusions of Steps 11 and 11, we see that $`𝔥=𝔥_B`$. Therefore $`H=H_B`$, so Conclusion (1c) holds.

From (iii), we see that Conclusion (1b) holds.

Letting $`z=[u,v]`$, for any $`u,v𝔲^{}`$, we see, from (7.22), that

$$𝗑_z=2\mathrm{Im}(x_ux_v^{})$$

and

$$𝗒_z=2\mathrm{Im}(y_uy_v^{})=2\mathrm{Im}\left((x_uB)(x_vB)^{}\right).$$

From Step 11, we know that $`𝗒_z=𝗑_z`$, so this implies that Conclusion (1a) holds.

Case 2 . Assume there does not exist a continuous curve $`h^t\mathrm{}`$ in $`H`$, such that $`\rho (h^t)h^t^2`$.  Since $`dimHqn`$, Theorem 10.26 implies that $`dimH=qn`$, and that $`H`$ is of the form $`H=TU`$, where

1. $`T=\mathrm{ker}\beta `$,
2. $`𝔲=((𝔫_\alpha +𝔫_{\alpha +\beta }+𝔫_{\alpha +2\beta })𝔲)+𝔫_{2\alpha +2\beta }`$, and
3. $`|x_u|^2+2\mathrm{Re}(\varphi _u\overline{\eta _u})0`$ for every $`u𝔲𝔫_{2\alpha +2\beta }`$.

Let

$$𝔲^{}=(𝔫_\alpha +𝔫_{\alpha +\beta }+𝔫_{\alpha +2\beta })𝔲$$

(so $`𝔲=𝔲^{}𝔫_{2\alpha +2\beta }`$). Let $`Q`$ be the sesquilinear form (or bilinear form, if $`𝔽=`$) on $`𝔽𝔽^{n2}𝔽`$ defined by

$$Q((\varphi _1,x_1,\eta _1),(\varphi _2,x_2,\eta _2))=\varphi _1\overline{\eta _2}+x_1x_2^{}+\eta _1\overline{\varphi _2}.$$

Let

$$V_𝔥=\{(\varphi _u,x_u,\eta _u)𝔽𝔽^{n2}𝔽u𝔲^{}\}.$$

From (iii), we see that the restriction of $`\mathrm{Re}Q`$ to $`V_𝔥`$ is a (positive-definite) inner product.

Let $`V_𝔥^{}`$ be the $`(\mathrm{Re}Q)`$-orthogonal complement to $`V_𝔥`$. As a form over $`𝔽`$, $`Q`$ has signature $`(1,n1)`$. Thus, as a form over $``$, $`\mathrm{Re}Q`$ has signature $`(q,q(n1))`$. Since

$$dimV_𝔥=dim𝔥dim𝔱dim𝔫_{2\alpha +2\beta }=qn1(q1)=q(n1),$$

we conclude that $`V_𝔥^{}`$ is a $`q`$-dimensional $``$-subspace on which $`\mathrm{Re}Q`$ is negative-definite.

Choose some nonzero $`uV_𝔥^{}`$. Multiplying by a real scalar to normalize, we may assume $`Q(u,u)=2`$. Because $`\mathrm{SU}(1,n1)`$ is transitive on the vectors of norm $`1`$, there is some $`g\mathrm{SU}(\mathrm{Re}Q)`$, such that $`g(u)=(1,0,1)`$. Thus, letting

(11.3)
$$\widehat{g}=\left(\begin{array}{ccc}1& 0& 0\\ 0& g& 0\\ 0& 0& 1\end{array}\right)\mathrm{SU}(2,n;𝔽),$$

and $`𝔥^{\mathrm{}}=\widehat{g}^1𝔥g`$, we have $`(1,0,1)V_𝔥^{\mathrm{}}`$, so, by replacing $`𝔥`$ with the conjugate $`𝔥^{\mathrm{}}`$, we may assume $`u=(1,0,1)`$.

Then

(11.4)
$$\begin{array}{cc}\hfill V_𝔥& u^{}\hfill \\ & =(1,0,1)^{}\hfill \\ & =\{(\varphi ,x,\eta )\mathrm{Re}Q((\varphi ,x,\eta ),(1,0,1))=0\}\hfill \\ & =\{(\varphi ,x,\eta )\mathrm{Re}\left(\varphi (1)+x(0^{})+\eta (1)\right)=0\}\hfill \\ & =\{(\varphi ,x,\eta )\mathrm{Re}\eta =\mathrm{Re}\varphi \}.\hfill \end{array}$$

Subcase 2.1 . Assume $`𝔽=`$.  By comparing (11.4) and (7.13) (with $`𝔽=`$), we conclude that

$$𝔥\mathrm{SU}(1,n;)(𝔞+𝔫).$$

By comparing dimensions, we see that equality must hold; this establishes Conclusion (2).

Subcase 2.2 . Assume $`𝔽=`$.  Choose some nonzero $`vV_𝔥^{}`$, such that $`v`$ is $`(\mathrm{Re}Q)`$-orthogonal to $`u`$. Multiplying by a real scalar to normalize, we may assume $`Q(v,v)=2`$. By replacing $`v`$ with $`v`$ if necessary, we may assume $`\left(\mathrm{Im}Q(u,v)\right)/i0`$.

Because $`v`$ is $`(\mathrm{Re}Q)`$-orthogonal to $`u=(1,0,1)`$, we have $`\mathrm{Re}\eta _v=\mathrm{Re}\varphi _v`$ (see 11.4). Let $`s=(\mathrm{Im}\varphi _v)/i`$ and $`t=(\mathrm{Im}\eta _v)/i`$. Then

$`0`$ $`\left(\mathrm{Im}Q(u,v)\right)/i`$
$`=\left(\mathrm{Im}\left((1)\overline{\eta _v}+0(x_v^{})+(1)\overline{\varphi _v}\right)\right)/i`$
$`=\left(\mathrm{Im}\eta _v+\mathrm{Im}\varphi _v\right)/i`$
$`=t+s,`$
so

$$\left(\mathrm{Im}Q(u,v)\right)/i=|st|.$$

Also,

$`2`$ $`=Q(v,v)`$
$`=|x_v|^2+2\mathrm{Re}(\varphi _v\overline{\eta _v})`$
$`=|x_v|^2+2(\mathrm{Re}\varphi _v)^22(\mathrm{Im}\varphi _v)(\mathrm{Im}\eta _v)`$
$`2(\mathrm{Im}\varphi _v)(\mathrm{Im}\eta _v)`$
$`=2st,`$
so $`st1`$. Thus, $`s`$ and $`t`$ are of opposite signs so, because $`|s||t|1`$, we have

$$\left(\mathrm{Im}Q(u,v)\right)/i=|st|=|s|+|t|2.$$

Therefore, we may choose $`c(0,1]`$, such that

$$\mathrm{Im}Q(u,v)=i\left(c+\frac{1}{c}\right).$$

Let

$$w=(\frac{i}{c},0,ic).$$

Then

$$Q(w,w)=|x_w|^2+2\mathrm{Re}(\varphi _w\overline{\eta _w})=0^2+2\left(i/c\right)\left(ic\right)=2=Q(v,v),$$

and

$$Q(u,w)=\varphi _u\overline{\eta _w}+x_ux_w^{}+\eta _u\overline{\varphi _w}=(1)(ic)+0+(1)(i/c)=i\left(c+\frac{1}{c}\right)=\mathrm{Im}Q(u,v).$$

Hence, there is some $`h\mathrm{SU}(Q)`$, such that $`h(u)=u`$ and $`h(v)=w`$. Thus, replacing $`𝔥`$ with the conjugate $`\widehat{h}^1𝔥\widehat{h}`$ (cf. 11.3), we may assume $`v=w`$.

Therefore

$`V_𝔥`$ $`v^{}`$
$`=(cv)^{}`$
$`=(i,0,ic^2)^{}`$
$`=\{(\varphi ,x,\eta )\mathrm{Re}Q((\varphi ,x,\eta ),(i,0,ic^2))=0\}`$
$`=\{(\varphi ,x,\eta )\mathrm{Re}\left(\varphi (ic^2)+x(0^{})+\eta (i)\right)=0\}`$
$`=\{(\varphi ,x,\eta )\mathrm{Im}\eta =c^2\mathrm{Im}\varphi \}.`$
By combining this with (11.4) and comparing with (9.11) (with $`𝔽=`$), we conclude that $`𝔥𝔥_{[c^2]}`$. By comparing dimensions, we see that equality must hold; this establishes Conclusion (3) (because $`0<c^21`$). ∎

Theorem 11.2′′ can be restated in the following more elementary (but less precise) form.

###### 11.5 Corollary.

Let $`H`$ be a closed, connected subgroup of $`G=\mathrm{SU}(2,2m;𝔽)`$ with $`m2`$, such that neither $`H`$ nor $`G/H`$ is compact, and let $`q=dim_{}𝔽`$.

The homogeneous space $`G/H`$ has a tessellation if and only if

1. $`d(H)=2qm`$; and
2. either $`H\mathrm{SU}(1,2m;𝔽)`$ or $`H\mathrm{Sp}(1,m;𝔽)`$.

###### Proof.

($``$) This is Theorem 9.2.

($``$) Theorem 11.2($``$) provides us with three possibilities.

(1) In each case, we have $`d(H)=2qm`$ (see 9.5, 7.15, and 9.12).

(2) In each case, there is some $`k\{1,2\}`$, such that $`\rho (h)h^k`$ for $`hH`$ (see 9.7(1), 8.21(1), and 9.14(1)). Then Corollary 8.23 implies either that $`H\mathrm{SU}(1,2m;𝔽)`$ (if $`k=1`$) or that $`H\mathrm{Sp}(1,m;𝔽)`$ (if $`k=2`$). ∎

The following proposition shows that no further restriction can be placed on $`c`$ in the statement of Theorem 11.2(3).

###### 11.6 Proposition.

If $`𝔽=`$, then $`H_{[c]}`$ is not conjugate to $`H_{[c^{}]}`$, unless $`c=c^{}`$ (for $`c,c^{}(0,1]`$).

###### Proof.

Suppose $`g^1H_{[c]}g=H_{[c^{}]}`$, for some $`gG=\mathrm{SU}(2,2m)`$. Because all maximal split tori in $`H_{[c^{}]}`$ are conjugate, we may assume that $`g`$ normalizes $`\mathrm{ker}\beta `$. Since all roots of $`\mathrm{ker}\beta `$ on both $`𝔥_{[c]}`$ and $`𝔥_{[c^{}]}`$ are positive, $`g`$ cannot invert $`\mathrm{ker}\beta `$, so we conclude that $`g`$ centralizes $`\mathrm{ker}\beta `$; that is, $`gC_G(\mathrm{ker}\beta )`$.

In the notation of Case 11 of the proof of Theorem 11.2, define

$$S=\{\widehat{h}h\mathrm{SU}(Q)\}$$

(cf. 11.3). Then $`C_G(\mathrm{ker}\beta )=(\mathrm{ker}\beta )S`$, so we may assume $`gS`$ (because $`\mathrm{ker}\beta `$, being a subgroup of $`H_{[c]}`$, obviously normalizes $`H_{[c]}`$). Write $`g=\widehat{h}`$. Then, because $`g^1H_{[c]}g=H_{[c^{}]}`$, we must have $`h(V_{𝔥_{[c]}})=V_{𝔥_{[c^{}]}}`$; hence $`h(V_{𝔥_{[c]}}^{})=V_{𝔥_{[c^{}]}}^{}`$.

For any basis $`\{u,v\}`$ of $`V_{𝔥_{[c]}}^{}`$ with $`Q(u,u)=Q(v,v)=2`$ and $`\mathrm{Re}Q(u,v)=0`$, we have

$$\mathrm{Im}Q(u,v)=\pm i\left(c+(1/c)\right).$$

Similarly for any $`(\mathrm{Re}Q)`$-orthonormal basis $`\{u^{},v^{}\}`$ of $`V_{𝔥_{[c^{}]}}^{}`$. Because $`h\mathrm{SU}(Q)`$, this implies $`c+(1/c)=c^{}+(1/c^{})`$. Because $`c,c^{}(0,1]`$, we conclude that $`c=c^{}`$, as desired. ∎