text:
# Quantum impurity in a magnetic environment

## I Introduction

The Kondo problem has played a central role in the development of the theory of correlated electron systems. At its simplest it consists of a single quantum spin, $`\widehat{S}_\alpha `$ ($`\alpha =x,y,z`$), interacting with the fermionic excitations in a metallic environment. In a modern perspective, many aspects of the Kondo problem can be understood in the framework of boundary conformal field theory: the fermionic excitations in the environment are represented by a $`1+1`$ dimensional, free, conformal field theory with central charge $`c=1`$, and this interacts with the quantum spin degree of freedom located at spatial co-ordinate $`x=0`$.

More recently, attention has focused on a new type quantum impurity problem. Here, we again consider a quantum spin $`\widehat{S}_\alpha `$, but it now interacts with bosonic excitations in the environment. Such models become appropriate when the environment is in the vicinity of a magnetic ordering transition, and there are low energy spin excitations in the bulk; the latter may be viewed as excitonic particle-hole bound states of a metal/insulator/superconductor which peel off below the continuum of a pair of fermionic particles or holes.

We begin by describing the simplest ‘Bose Kondo’ problem, and postpone a discussion of specific physical motivations till later, when we consider more realistic models. The simplest modelsy  has the Hamiltonian

$$_1=\lambda \varphi _\alpha \widehat{S}_\alpha $$
(1)
where $`\lambda `$ is a coupling constant, and the $`\widehat{S}_\alpha `$ obey the usual relations of a Heisenberg spin with angular momentum $`S`$,

$$[\widehat{S}_\alpha ,\widehat{S}_\beta ]=iϵ_{\alpha \beta \gamma }\widehat{S}_\gamma ;\widehat{S}_\alpha \widehat{S}_\alpha =S(S+1).$$
(2)
The Bose field $`\varphi _\alpha `$ has Gaussian correlations in the absence of its coupling to $`\widehat{S}_\alpha `$, with the two-point correlation obeying

$$\varphi _\alpha (\tau )\varphi _\alpha (0)_{\lambda =0}\frac{1}{|\tau |^\mu },$$
(3)
for large $`|\tau |`$ with $`\mu >0`$, where $`\tau `$ is imaginary time.

It is important to distinguish the above Bose Kondo problem, from the ‘spin boson’ problem which had been the focus of much earlier attention leggett . The latter deals with a two-level system coupled to a bath of harmonic oscillators. Upon interpreting the two-level system as a spin, the splitting between the energy levels behaves like a magnetic field on the spin. In this situation, the spin-inversion symmetry, $`\widehat{S}_\alpha \widehat{S}_\alpha `$ for any 2 of the 3 $`\alpha `$ values, is explicitly broken by the Hamiltonian. In contrast, in the Bose Kondo problem of interest here, this spin inversion symmetry is preserved (when combined with the transformation $`\varphi _\alpha \varphi _\alpha `$).

Despite the simple form of the Hamiltonian $`_1`$ and of the correlator (3), the spin commutation relations (2) make this a problem of some complexity which cannot be solved exactly. This is also evident from its path integral formulation, in which we integrate over $`\varphi _\alpha (\tau )`$ and over a unit length field $`n_\alpha (\tau )`$, where $`\widehat{S}_\alpha =Sn_\alpha `$:

$`𝒵_1={\displaystyle 𝒟\varphi _\alpha (\tau )𝒟n_\alpha (\tau )\delta \left(n_\alpha ^21\right)\mathrm{exp}\left(𝒮_\text{b}[\varphi _\alpha ]𝒮_{\text{imp}}\right)}`$
$`𝒮_{\text{imp}}={\displaystyle 𝑑\tau \left[iSA_\alpha (n)\frac{dn_\alpha (\tau )}{d\tau }\lambda S\varphi _\alpha (\tau )n_\alpha (\tau )\right]}`$
$`𝒮_\text{b}[\varphi _\alpha ]={\displaystyle \frac{1}{2}}{\displaystyle 𝑑\tau 𝑑\tau ^{}\varphi _\alpha (\tau )Q^1(\tau \tau ^{})\varphi _\alpha (\tau ^{})}.`$ (4)
In this formulation, all the non-linearities are in the first Berry phase term, which involves the vector potential of a unit Dirac monopole at the origin of spin space obeying

$$ϵ_{\alpha \beta \gamma }\frac{A_\gamma (n)}{n_\beta }=n_\alpha .$$
(5)
Also, $`Q(\tau )`$ is the two-point $`\varphi _\alpha `$ correlator at $`\lambda =0`$, and (3) implies that its Fourier transform, $`Q(i\omega )`$, has the spectral density $`\text{Im}Q(\omega )\text{sgn}(\omega )|\omega |^{\mu 1}`$ at small frequencies.

The problem (4) appeared in Ref. sy,  in the context of a mean-field theory of a quantum Heisenberg spin glass. It was solved here by generalizing the SU(2) symmetry to SU($`N`$), and taking the large $`N`$ limit. In this limit, the spin correlations obey

$$\widehat{S}_\alpha (\tau )\widehat{S}_\alpha (0)\frac{1}{|\tau |^{2\mu }}$$
(6)
for large $`\tau `$ and $`0<\mu <2`$, while for $`\mu 2`$ and small $`\lambda `$ there is a broken spin rotation symmetryanirvan  and the $`\widehat{S}_\alpha `$ two-point correlator reaches a non-zero value at large $`|\tau |`$. The exponent in (6) was also obtained using a one-loop renormalization group analysis anirvan ; qmsi , and was subsequently shown sbv ; gps  to hold to all orders in an expansion in $`2\mu `$. The result (6) has also been found to hold in certain quantum impurity models in which the spin $`\widehat{S}_\alpha `$ is coupled simultaneously to bosonic and fermionic excitations in its environment anirvan ; qmsi ; si ; demler , including cases with spin anisotropy.

It is important to note that (6) relies crucially on the presence of the Berry phase in (4). In the absence of this term, we can integrate over the Gaussian $`\varphi _\alpha `$ modes, and then (4) becomes equivalent to a classical ferromagnetic spin chain at finite ‘temperature’, with exchange interactions which decay as $`1/|\tau |^\mu `$. The properties of this classical modelkosterlitz  are very different and have an interesting ‘dual’ structure. Now, the ferromagnetic phase with broken spin rotation symmetry appears only for $`\mu <2`$ and low ‘temperatures’ (large $`\lambda `$)—in contrast, with the Berry phase term, as we noted above, spin rotation invariance was broken for $`\mu 2`$ and small $`\lambda `$. Furthermore, in the classical model without the Berry phase, the paramagnetic phase with preserved spin rotation symmetry (present for all ‘temperatures’ for $`\mu 2`$ and in the high ‘temperature’ (small $`\lambda `$) phase for $`\mu <2`$) has its two-point $`\widehat{S}_\alpha `$ correlator decaying as $`1/|\tau |^\mu `$—in contrast, with the Berry phase we found a rotationally invariant phase for $`\mu <2`$ and with the correlator (6).

Intriguing and interesting as the properties of $`𝒵_1`$ are, their physical interpretation and application require care and must be discussed in the context of the underlying model from which $`𝒵_1`$ was derived. In particular, a free Bose field with a gapless spectrum is a delicate object which can become unstable under infinitesimal perturbations (this should be contrasted from a free Fermi field, which has a robust stability). One instance of this instability is the response to an applied magnetic field, $`H_\alpha `$; there must be a coupling which imposes a precession of $`\varphi _\alpha `$ about the direction of magnetic field, and in these conditions the action is unstable to arbitrarily large fluctuations in $`\varphi _\alpha `$ in the directions orthogonal to the field. For the initial spin glass context in which $`𝒵_1`$ was studied, the ‘quantum critical’ state described by (6) was found to be unstable to the onset of spin glass order at low temperatures gps .

In the remainder of this paper we will review another context in which $`𝒵_1`$ has appeared: the theory of quantum impurities in insulators and superconductors with low energy quantum spin fluctuations. Here, as we will see below, it is essential to include a quartic $`\varphi _\alpha ^4`$ term for proper computations of the $`\widehat{S}_\alpha `$ correlations.

## II Quantum antiferromagnets: $`\varphi ^4`$ field theory

A concrete application of the ‘Bose Kondo’ theory, which is now reasonably well understood, is the problem of quantum impurities in two-dimensional antiferromagnets. For definiteness, consider the simple coupled ladder antiferromagnet, illustrated in Fig 1.

As the ratio of the exchange constants is varied, two distinct types of ground states are obtained. For weakly-coupled ladders, the ground state is a spin singlet and there is a gap to all excitations; the ground state is adiabatically connected to the state in which each spin is paired in a singlet with its partner across the rung of the ladder. In contrast, when the inter- and intra-ladder exchange constants are roughly equal, the model has the structure of the square lattice antiferromagnet, and so has antiferromagnetic Néel order in its ground state; in this case spin rotation symmetry is broken, and the spin operators have an average expectation value which has opposite signs on the two sublattices. Given the distinct nature of these two ground states, there must be a quantum phase transition between them. There is now quite convincing evidence ladders  that there is one second-order quantum critical point, and in its vicinity the spin fluctuations are described by the $`\varphi ^4`$ field theory, written here in $`d`$ spatial dimensions:

$$\stackrel{~}{𝒮}_\text{b}[\varphi _\alpha ]=d^dx𝑑\tau \left[\frac{1}{2}\left\{(_\tau \varphi _\alpha )^2+c^2(_x\varphi _\alpha )^2+r\varphi _\alpha ^2\right\}+\frac{u}{24}(\varphi _\alpha ^2)^2\right].$$
(7)
The field $`\varphi _\alpha `$ represents the staggered Néel order parameter, and the tuning parameter $`r`$ moves the system from the spin gap state at large $`r`$, to the Néel state at smaller $`r`$. The spin-wave velocity is $`c`$, and $`u`$ is the quartic non-linear coupling.

Now insert an arbitrary quantum impurity in the spin ladder system: two examples are shown in Fig 2.

In the spin gap state, the presence of such a impurity may liberate one or more spins from their partners, and this leads to a residual Curie spin susceptibility at a low temperature $`T`$:

$$\chi _{\text{imp}}=\frac{S(S+1)}{3k_BT}\text{;  spin gap in bulk antiferromagnet}.$$
(8)
Here $`S`$ is an integer or half-odd-integer which characterizes the impurity. A remarkable property sbv  of the low energy dynamics of the quantum impurity is that no other parameters are needed to described the spin dynamics in its vicinity, provided the bulk antiferromagnet is not too far from its quantum critical point. This result emerges from an analysis of the theory coupling the impurity to the $`\varphi ^4`$ field theory:

$`𝒵_2={\displaystyle 𝒟\varphi _\alpha (x,\tau )𝒟n_\alpha (\tau )\delta \left(n_\alpha ^21\right)\mathrm{exp}\left(\stackrel{~}{𝒮}_\text{b}[\varphi _\alpha ]\stackrel{~}{𝒮}_{\text{imp}}\right)}`$
$`\stackrel{~}{𝒮}_{\text{imp}}={\displaystyle 𝑑\tau \left[iSA_\alpha (n)\frac{dn_\alpha (\tau )}{d\tau }\lambda S\varphi _\alpha (x=0,\tau )n_\alpha (\tau )\right]},`$ (9)
with $`\stackrel{~}{𝒮}_\text{b}[\varphi _\alpha ]`$ given in (7). Notice the similarity of $`𝒵_2`$ to $`𝒵_1`$: at $`r=0`$, $`u=0`$, we can integrate out all the $`\varphi _\alpha (x0,\tau )`$, and then $`𝒵_2`$ reduces to $`𝒵_1`$ with $`\mu =d1`$. However, it is crucial in the proper theory of $`𝒵_2`$ that the non-linearity $`u`$ be treated at an equal footing with $`\lambda `$; there is a non-trivial ‘interference’ between $`u`$ and $`\lambda `$, and the interaction $`u`$ significantly modifies the magnetic environment coupling to the impurity. It is not permissible to treat the environment as a Gaussian quantum noise, and focus only on its Kondo-like coupling to the impurity.

A systematic renormalization group based analysis of $`𝒵_2`$ was carried outsbv  in an expansion in $`(3d)`$. The couplings $`\lambda ^2`$ and $`u`$ both approach fixed point values of order $`(3d)`$, and this is the reason the coupling between the bulk and impurity spin fluctuations becomes universal, as claimed above. At the critical point, the spin correlations decay as

$$\widehat{S}_\alpha (x,\tau )\widehat{S}_\alpha (x,0)\frac{1}{|\tau |^\eta ^{}};x0,$$
(10)
close to the impurity. The exponent $`\eta ^{}2\mu =3d`$, as would be implied by (6), because of the non-zero fixed point value of $`u`$. Well away from the impurity, the results are as in the absence of the impurity with

$$\widehat{S}_\alpha (x,\tau )\widehat{S}_\alpha (x,0)\frac{1}{|\tau |^{d1+\eta }};x\mathrm{},$$
(11)
where $`\eta `$ is the well-known anomalous dimensions of the $`\varphi ^4`$ field theory in $`d+1`$ spacetime dimensions. The value of $`\eta ^{}`$, and numerous other physical properties of the impurity on both sides of the bulk quantum critical point, were computed in Ref. sbv  to second order in an expansion in $`(3d)`$. Numerical studies troyer ; sandvik  have investigated some of these properties. Related theoretical results were obtained recently castro  in magnetically ordered states in the presence of spin anisotropy.

## III Quantum antiferromagnets: Non-linear sigma model

An alternative approach to the impurity dynamics discussed in Section II is provided by a different representation of the bulk spin fluctuations. It is well known that in low dimensions the $`\varphi ^4`$ field theory can be represented by the non-linear sigma model: the fluctuations of the amplitude, $`\varphi _\alpha ^2`$, become irrelevant, and we need only focus on the angular fluctuations of the Néel order parameter. These are represented by a unit-length field $`N_\alpha (x,\tau )`$. Such a ‘non-linear sigma model’ representation provides an expansion in powers of $`(d1)`$ for the bulk critical properties.

In the quantum impurity problems of interest here, such a fixed-length representation offers some benefits. One is that it allows systematic computation of some properties in the ‘renormalized classical’ regime directly in spatial dimension $`d=2`$. However, more importantly, in the fixed-length formulation the universal nature of the couplings between the bulk and impurity spin fluctuations can be accounted for at the outset. Indeed, it was arguedsv  that in the scaling limit of the fixed-length theory, the quantum impurity behaves as if it is in the $`\lambda \mathrm{}`$ limit, and hence the impurity spin orientation align along the direction of the bulk spin order; in other words, we have $`n_\alpha (\tau )=N_\alpha (x=0,\tau )`$. With these arguments, we can rewrite the model described by the partition function $`𝒵_2`$ as

$`𝒵_3={\displaystyle 𝒟N_\alpha (x,\tau )\delta \left(N_\alpha ^21\right)\mathrm{exp}\left(𝒮_\text{b}[N_\alpha ]\overline{𝒮}_{\text{imp}}\right)}`$
$`\overline{𝒮}_{\text{imp}}={\displaystyle 𝑑\tau \left[iSA_\alpha (n)\frac{dn_\alpha (\tau )}{d\tau }\right]\text{with }n_\alpha (\tau )=N_\alpha (x=0,\tau )}`$
$`𝒮_\text{b}[N_\alpha ]={\displaystyle \frac{1}{2cg}}{\displaystyle d^dx𝑑\tau \left[\left(_\tau N_\alpha \right)^2+c^2\left(_xN_\alpha \right)^2\right]},`$ (12)
where now $`g`$ is the coupling constant that tunes the bulk antiferromagnet across the quantum critical point. Notice that there is no other coupling constant, and hence the universal nature of the coupling between the bulk and impurity is explicit. A systematic $`(d1)`$ expansion of $`𝒵_3`$ was performedsv , and all results were found to be consistent with those reviewed in Section II. In particular, a $`(d1)`$ expansion was presented for the exponent $`\eta ^{}`$ in (10), associated with computation of a ‘boundary’ renormalization constant for the field $`N_\alpha (x=0,\tau )`$. Related results were also obtained in Ref. sushkov2 .

## IV Electron spectral function in Kondo insulators

Kondo insulators are another class of physically interesting systems displaying a magnetic transition. In Kondo lattice models with a commensurate density of conduction electrons, increasing the Kondo exchange can drive the ground state from an ordered Néel state to a paramagnetic insulator in which the conduction electrons and local moments are strongly hybridized. The low energy magnetic fluctuations near such a critical point are also believed to be described by the $`\varphi ^4`$ field theory (7).

Now, let us consider the photoemission spectrum of a conduction electron in such an insulator in the vicinity of the magnetic transition assaad . Away from the critical point, there will be a sharp quasiparticle/hole pole with a finite residue, and the position of this pole will disperse in the Brillouin zone. Focus on the spectral function at the minimum of this dispersionsushkov ; stv , where the momentum dependence is quadratic. It was argued stv  that near this minimum, and in the vicinity of the magnetic ordering transition where the action (7) applies, we can safely neglect the quadratic dispersion of the hole. We are therefore left with the problem of a static hole interacting with the magnetic environment described by (7). This problem is clearly analogous to the X-ray edge problem, where a static hole interacts with a Fermi liquid.

This “Bose X-ray edge” problem is described by the partition function

$`𝒵_4={\displaystyle 𝒟\varphi _\alpha (x,\tau )𝒟\psi _a(\tau )𝒟\psi _a^{}(\tau )\mathrm{exp}\left(\stackrel{~}{𝒮}_\text{b}[\varphi _\alpha ]𝒮_\psi \right)}`$
$`𝒮_\psi ={\displaystyle 𝑑\tau \left[\psi _a^{}\left(\frac{}{\tau }+\epsilon _0\right)\psi _a\frac{\lambda }{2}\varphi _\alpha (x=0,\tau )\psi _a^{}\sigma _{ab}^\alpha \psi _b\right]},`$ (13)
with $`\stackrel{~}{𝒮}_\text{b}[\varphi _\alpha ]`$ given in (7). The $`\psi _a`$ are Grassman variables representing the hole with $`a,b=,`$, and $`\sigma ^\alpha `$ are the Pauli matrices. A complication has been ignored in our presentation here of $`𝒵_4`$ here: as the field $`\varphi _\alpha `$ carrier spin fluctuations at a finite momentum, it actually couples fermionic excitations at two different points in the Brillouin zone. We have not included this effect here because keeping track of the fermionic momentum label does not modify the critical propertiesstv .

With the hole present, the quantum theory $`𝒵_4`$ is, in fact, identical to $`𝒵_2`$ (with $`S=1/2`$): we have simply realized the quantum spin by a single hole. Consequently, the renormalization group equations for the $`\lambda `$ coupling in (13) are identical for those for $`\lambda `$ in (9). However, the present formulation allows determination of a new renormalization constant associated with the insertion of a hole. This constant measures the overlap of the system wavefunctions with and without the hole; in contrast, in (9) the spin is always present and so this physics is inaccessible.

Specifically, in the paramagnetic phase, away from the critical point, the single hole Green’s function $`G`$, has a quasiparticle pole given by

$$G(\omega )=\frac{Z}{\omega \epsilon _0}$$
(14)
where $`Z`$ is the quasiparticle residue, and $`\epsilon _0`$ has absorbed a renormalization from the coupling of the hole to $`\varphi _\alpha `$. As we approach the critical point, there is an orthogonality catastrophe and $`Z0`$. Instead, at the critical point we havestv ; sushkov ; iss

$$G(\omega )\frac{1}{(\omega \epsilon _0)^{1\eta _f}}.$$
(15)
The exponent $`\eta _f`$ is distinct from $`\eta ^{}`$, and its determination requires a separate renormalization group analysis. The value of $`\eta _f`$ has been obtained in a two-loop expansion iss  in powers of $`(3d)`$, and by a numerical simulation stv .

###### Acknowledgements.

I thank Chiranjeeb Buragohain, Antoine Georges, Olivier Parcollet, Matthias Troyer, Matthias Vojta, and Jinwu Ye for fruitful collaborations on the topics reviewed here. This research was supported by US NSF Grant DMR 0098226.


format: text

text:
# Representations of the fundamental group of a closed oriented surface in Sp(4,ℝ)

## 1. Introduction

Let $`X`$ be a closed oriented surface of genus $`g2`$ and let $`\pi =\pi _1(X)`$ be its fundamental group. Let $`\mathrm{Sp}(4,)`$ be the group of linear transformations of $`^4`$ preserving its standard real symplectic form. Consider the set $`𝒳:=\mathrm{Hom}(\pi ,\mathrm{Sp}(4,))`$ of group homomorphisms from $`\pi `$ to $`\mathrm{Sp}(4,)`$. We also refer to the elements of $`𝒳`$ as representations of $`\pi `$ in $`\mathrm{Sp}(4,)`$. Using a set of generators of $`\pi `$, $`𝒳`$ can be embedded in $`\mathrm{Sp}(4,)^{2g}`$, acquiring in this way a natural topological structure. The goal of this paper is to compute the number of connected components of $`𝒳`$.

Given a representation of $`\pi `$ in $`\mathrm{Sp}(4,)`$, there is an integer associated to it, which is geometrically obtained by considering the flat $`\mathrm{Sp}(4,)`$-bundle corresponding to the representation and taking a reduction of the structure group to $`\mathrm{U}(2)`$ — the maximal compact subgroup of $`\mathrm{Sp}(4,)`$. The degree $`d`$ of this $`\mathrm{U}(2)`$-bundle is an invariant of the representation, which we call the degree of the representation. Let $`𝒳(d)𝒳`$ be the set of representations of degree $`d`$. There is a Milnor-Wood type inequality () which says that $`𝒳(d)`$ is empty unless $`|d|2g2`$. Our main result in this paper is the following.

###### Theorem 1.1.

Let $`d`$ be an integer such that $`|d|2g2`$. Then

(1) $`𝒳(d)`$ is non-empty and connected if $`|d|<2g2`$;

(2) $`𝒳(d)`$ has $`3.2^{2g}+2g4`$ non-empty connected components if $`|d|=2g2`$.

The proof of (2) and the case $`d=0`$ for the space of reductive representations is due to Gothen . Recall that a representation $`\rho `$ from $`\pi `$ to a real algebraic group $`G`$ is called reductive if the real Zariski closure of $`\rho (\pi )`$ in $`G`$ is a reductive group. When the closure coincides with $`G`$, the representation is said to be irreducible. Let $`𝒳^+𝒳`$ be the subspace of reductive representations of $`\pi `$ in $`\mathrm{Sp}(4,)`$. In Section 2 we show that the inclusion $`𝒳^+𝒳`$ induces a bijection of connected components (Theorem 2.3). In fact, we show, this result is valid replacing $`\mathrm{Sp}(4,)`$ by any reductive real algebraic group $`G`$ and $`\pi `$ by any finitely generated group.

Let $`𝒳^+(d)`$ be the space of reductive representations from $`\pi `$ to $`\mathrm{Sp}(4,)`$ of degree $`d`$. This space is invariant under the action of $`\mathrm{Sp}(4,)`$ by conjugation. The moduli space of representations of degree $`d`$ is defined as $`(d):=𝒳^+(d)/\mathrm{Sp}(4,)`$. The reductivity condition is precisely what is needed in order for $`(d)`$ to be a Hausdorff space. Gothen’s results can now be stated as follows.

###### Theorem 1.2 (Gothen ).

The moduli space $`(0)`$ is non-empty and connected, and $`(\pm (2g2))`$ has $`3.2^{2g}+2g4`$ non-empty connected components.

In this paper we settle the situation for the remaining cases. More precisely, we prove the following.

###### Theorem 1.3.

Let $`d`$ be any integer satisfying $`0<|d|<2g2`$. The moduli space $`(d)`$ is connected and the subspace of irreducible representations is non-empty.

To prove this theorem, in Section 3, we follow the methods of Gothen () and Hitchin () by choosing a complex structure on $`X`$ and exploting the relation between $`(d)`$ and the moduli space of polystable $`\mathrm{Sp}(4,)`$-Higgs bundles $`(d)`$. These are Higgs bundles $`(E,\mathrm{\Phi })`$, where $`E=VV^{}`$, for $`V`$ a rank 2 holomorphic vector bundle of degree $`d`$, and Higgs field $`\mathrm{\Phi }:EE^{}K`$ of the form

(1.1)
$$\mathrm{\Phi }=\left(\begin{array}{cc}0& \beta \\ \gamma & 0\end{array}\right):VV^{}(VV^{})K$$

with $`\beta H^0(S^2VK)`$ and $`\gamma H^0(S^2V^{}K)`$. The basic fact is that $`(d)`$ is a complex analytic variety which is homeomorphic to $`(d)`$ (by results of Hitchin, Donaldson, Simpson and Corlette). This homeomorphism induces a homeomorphism between the subspace of irreducible representation in $`(d)`$ and the subspace of stable Higgs bundles in $`(d)`$.

By solving Hitchin equations for a Hermitian metric on $`V`$ and considering the square of the $`L^2`$-norm of $`\mathrm{\Phi }`$, one has a proper function on $`(d)`$. The number of connected components of the local minima of this proper function gives an upper bound on the number of connected components of $`(d)`$ and hence of $`(d)`$. These local minima have been characterised by Gothen  for any $`d`$. In particular when $`|d|<2g2`$, Gothen shows that the minima coincide with the subvariety $`𝒩(d)(d)`$ for which either $`\beta =0`$ or $`\gamma =0`$ (Proposition 3.6). Which one of the sections actually vanishes is determined by the sign of $`d`$. We prove that if $`0<|d|<2g2`$, $`𝒩(d)`$ is connected. To show this, in Section 4 we study a more general situation that is of independent interest. Namely, we introduce a (poly)stability criterium for pairs $`(V,\beta )`$, where $`V`$ is a rank 2 holomorphic vector bundle of degree $`d`$, and $`\beta H^0(S^2VK)`$ which depends on a real parameter $`\alpha `$, thus defining moduli spaces $`𝒩_\alpha (d)`$. It turns out that the subvariety of minima $`𝒩(d)`$ can be identified with $`𝒩_0(d)`$ (Proposition 4.1). Our main result concerning $`𝒩_\alpha (d)`$ is the following Theorem proved in Section 5.

###### Theorem 1.4.

Let $`2(g1)<d<0`$ be any integer and let $`\alpha 0`$ be a real number. The moduli space $`𝒩_\alpha (d)`$ is connected. Moreover, the subvariety consisting of stable pairs is non-empty.

From this we deduce the following.

###### Theorem 1.5.

Let $`d`$ be any integer such that $`0<|d|<2g2`$. The moduli space $`(d)`$ of polystable $`\mathrm{Sp}(4,)`$-Higgs bundles of degree $`d`$ is connected. Moreover, the subvariety consisting of stable Higgs bundles is non-empty.

This proves Theorem 1.3, which combined with Gothen’s Theorem 1.2 and Theorem 2.3 proves Theorem 1.1, since the connectedness of $`\mathrm{Sp}(4,)`$ implies that the number of connected components of $`𝒳^+(d)`$ coincides with that of $`(d)`$.

## 2. Representations of the fundamental group

### 2.1. Reductive representations of $`\pi _1(X)`$

The results of this section and the next one apply also when $`\pi `$ is replaced by any finitely generated group.

Let $`X`$ be a closed oriented surface of genus $`g2`$ and let $`\pi =\pi _1(X)`$ denote its fundamental group. Let $`G\mathrm{GL}(N,)`$ be a non-compact real reductive algebraic group (when $`G`$ is compact everything which follows in this section and the next one are also true for obvious reasons). Let $`𝒳:=\mathrm{Hom}(\pi ,G)`$ be the set of representations of $`\pi `$ in $`G`$, and let

$$𝒳^+:=\{\rho 𝒳\overline{\rho (\pi )}G\text{ is reductive }\}$$

the set of reductive representations ($`\overline{\rho (\pi )}`$ denotes the real Zariski closure of $`\rho (\pi )`$ in $`G`$). Take generators $`\gamma _1,\mathrm{},\gamma _k`$ of $`\pi `$ and consider the inclusion $`j:𝒳𝔼:=\mathrm{End}(^N)^k`$ which sends $`\rho `$ to $`(\rho (\gamma _1),\mathrm{},\rho (\gamma _k))G^k\mathrm{End}(^N)^k`$. Consider on $`𝒳`$ the topology induced by $`j`$ and the standard topology on the vector space $`𝔼`$. This is independent of the choice of generators of $`\pi `$. We take on $`𝒳^+`$ the topology induced by the inclusion $`𝒳^+𝒳`$. Furthermore, $`j(𝒳)𝔼`$ is a real algebraic (affine) subvariety of $`G^k\mathrm{End}(^N)^k`$, whose equations are defined by requiring the coordinates of a point $`(\rho _1,\mathrm{},\rho _k)G^k`$ to satisfy any relation satisfied by the generators $`\{\gamma _j\}`$. It follows by a theorem of Whitney  that $`j(𝒳)`$ (and hence $`𝒳`$) has a finite number of connected components.

Consider the adjoint action of $`G`$ on $`𝒳`$: if $`gG`$ and $`\rho 𝒳`$ then $`g\rho `$ is the representation defined by $`g\rho (\gamma ):=g\rho (\gamma )g^1`$ for any $`\gamma \pi `$. We consider similarly the diagonal adjoint action of $`G`$ on $`𝔼=(\mathrm{End}^N)^k`$, in such a way that the inclusion $`j`$ is $`G`$-equivariant. One has the following.

###### Theorem 2.1 (Richardson, Theorem 11.4 in ).

A representation $`\rho 𝒳`$ is reductive if and only if the orbit $`Gj(\rho )𝔼`$ is closed in the usual topology of $`𝔼`$.

###### Remark 2.2.

If a real algebraic group acts linearly on a vector space then an orbit which is closed in the usual topology may fail to be closed in the real Zariski topology (in general its Zariski closure will consist of a finite number of orbits which are closed in the usual topology). This is in contrast to the situation over the complex numbers, where an orbit of a linear action is closed in the usual topology if and only if it is closed in the Zariski topology.

It follows from Theorem 2.1 that $`𝒳^+/G`$ with the quotient topology is a Hausdorff space. The space $`:=𝒳^+/G`$ is called the moduli space of representations of $`\pi `$ in $`G`$.

Our main concern is the study of the connectedness of $`𝒳`$, $`𝒳^+`$ and $``$. Of course, when $`G`$ is connected the number of connected components of $``$ coincides with that of $`𝒳^+`$.

### 2.2. From reductive representations to arbitrary representations

###### Theorem 2.3.

The inclusion $`i:𝒳^+𝒳`$ induces a bijection of connected components.

###### Proof.

To prove this we need the following. (All topological notions refer to the usual topology in $`𝔼`$.)

###### Theorem 2.4 (Luna, Theorem 2.7 in ).

Let $`G`$ be a real algebraic group acting linearly on a real vector space $`𝔼`$. Let $`𝔼^+:=\{x𝔼Gx𝔼\text{ is closed }\}`$. For any $`x𝔼`$ there is a unique closed orbit $`p(x)`$ contained in the closure of $`Gx`$. The space of orbits $`𝔼^+/G`$ endowed with the quotient topology is Hausdorff, and the map $`p:𝔼𝔼^+/G`$ is continuous.

In  this theorem is stated in a slightly different form. Luna defines a $`G`$-invariant subset $`A𝔼`$ to be $`G`$-saturated if it contains any orbit whose closure intersects the closure of any orbit inside $`A`$. Then he proves that for any $`G`$-invariant closed subset $`F𝔼`$ the smallest $`G`$-saturated subset of $`𝔼`$ which contains $`F`$ is also closed (this is Property (C) of ). To deduce from this the continuity of $`p`$ note that if $`C𝔼^+/G`$ is closed then $`p^1(C)`$ is the smallest $`G`$-saturated set containing the closure of $`q^1(C)𝔼`$, where $`q:𝔼^+𝔼^+/G`$ is the quotient map.

Now we prove Theorem 2.3. Denote by $`G_0G`$ the connected component of the identity. We first check that the map $`i_{}:\pi _0(𝒳^+)\pi _0(𝒳)`$ is onto. For that, take any $`\rho 𝒳`$. By Theorem 2.4 the closure of $`G_0j(\rho )`$ contains a closed orbit $`G_0j(\rho _0)`$. On the other hand, since $`G_0`$ is connected, the closure of $`G_0j(\rho )`$ is locally arc-connected, and $`j`$ is injective, it follows that there is a continuous map $`c:[0,1]𝒳`$ such that $`c(0)=\rho _0`$ and $`c(1)=\rho `$. Finally, Theorem 2.1 implies that $`\rho _0𝒳^+`$.

To prove that $`i_{}`$ is injective consider the following diagram, which commutes by the definition of $`p`$ (see Theorem 2.4)

Since $`G_0`$ is connected it follows that $`q_{}:\pi _0(𝒳^+)\pi _0(𝒳^+/G_0)`$ is a bijection, and since $`q_{}=p_{}i_{}`$ we deduce that $`i_{}`$ is an injection. ∎

### 2.3. Representations of $`\pi _1(X)`$ in $`\mathrm{Sp}(4,)`$

We use the same notation as in the previous two sections. In particular Let $`𝔼=\mathrm{End}(^4)^k`$. Let now $`G=\mathrm{Sp}(4,)`$ and let $`𝒳:=\mathrm{Hom}(\pi ,\mathrm{Sp}(4,))`$ be the set of representations of $`\pi `$ in $`\mathrm{Sp}(4)`$. To understand the geometric meaning of reductive representations of $`\pi `$ in $`\mathrm{Sp}(4,)`$, consider the standard symplectic structure $`\omega `$ on $`^4`$. Recall that a subspace $`𝕍^4`$ is called coisotropic if

$$𝕍^{}=\{v^4\omega (v,w)=0w𝕍\}𝕍.$$

One has the following.

###### Proposition 2.5.

Let $`\rho :\pi \mathrm{Sp}(4,)`$ be a representation. The following are equivalent:

(i) $`\rho `$ is reductive;

(ii) $`\mathrm{Sp}(4,)j(\rho )𝔼`$ is closed;

(iii) for any coisotropic subspace $`𝕍^4`$ which is preserved by $`\rho `$ there is a splitting

(2.2)
$$^4=𝕍/𝕍^{}(^4/𝕍𝕍^{})$$

such that the image of $`\rho `$ is contained in $`\mathrm{Sp}(𝕍/𝕍^{})\times \mathrm{Sp}(^4/𝕍𝕍^{})`$ (one can check that both $`𝕍/𝕍^{}`$ and $`^4/𝕍𝕍^{}`$ carry natural symplectic structures).

###### Proof.

The equivalence between (i) and (ii) is given by Theorem 2.1. We just sketch the proof of the equivalence between (ii) and (iii) (the argument is similar to that of Lemma 28 in ). Take a representation $`\rho `$ and suppose that there is a sequence $`\{g_j\}\mathrm{Sp}(4,)`$ which is not contained in any compact subset of $`\mathrm{Sp}(4,)`$ and such that for any $`k`$

$$\{g_j\rho (\gamma _k)g_j^1\}\gamma _k^{}\mathrm{Sp}(4,)\text{as }j\mathrm{}.$$

Define $`s_j:=g_j^{}g_j`$, where we take the adjoint of $`g_j`$ with respect to the standard Euclidean metric in $`^4`$. Then $`s_j`$ diagonialises in some basis $`e_1,\mathrm{},e_4`$ with eigenvalues $`\lambda _1\mathrm{}\lambda _4`$. Define $`𝕍^j:=e_2,e_3,e_4`$ if $`\lambda _1>\lambda _2`$ or $`𝕍^j:=e_3,e_4`$ otherwise. Since $`s_j\mathrm{Sp}(4,)`$, $`𝕍^j`$ is coisotropic. Let $`𝕍`$ be a limit of a partial sequence of $`\{𝕍^j\}`$. Then, since $`\{g_j\rho (\gamma _k)g_j^1\}`$ is bounded, $`\rho `$ preserves $`𝕍`$. Now, if $`\gamma _k^{}`$ belongs to the adjoint orbit of $`\gamma _k`$ then one must have a splitting as in (2.2). Finally, if the limit representation defined by $`\{\gamma _k^{}\}`$ does not belong to the orbit of $`\rho `$, then one can take a sequence $`\{g_j\}`$ in such a way that no splitting of the form (2.2) is preserved by $`\rho (\gamma _k)`$ for all $`k`$. ∎

###### Remark 2.6.

It is clear that a reductive representation $`\rho 𝒳`$ is irreducible if there is no nontrivial splitting $`^4=𝕍_1𝕍_2`$ with both $`𝕍_1`$ and $`𝕍_2`$ symplectic subspaces of $`^4`$ such that the image of $`\rho `$ is contained in $`\mathrm{Sp}(𝕍_1)\times \mathrm{Sp}(𝕍_2)`$.

### 2.4. Characteristic numbers

Our first step in determining the number of connected components of $`𝒳:=\mathrm{Hom}(\pi ,\mathrm{Sp}(4,))`$ is to study the topological invariants associated to a representation. For any $`G`$ there is a locally constant obstruction map

$$o:\mathrm{Hom}(\pi ,G)H^2(X,\pi _1(G)).$$

For $`G=\mathrm{Sp}(4,)`$, we have $`\pi _1(G)`$; picking the standard isomorphism, the obstruction map associates to any $`\rho 𝒳`$ an integer $`o(\rho )`$ which we call the degree of $`\rho `$, and which can be obtained more geometrically by considering the flat $`\mathrm{Sp}(4,)`$ bundle associated to $`\rho `$. This bundle admits a reduction of the structure group to $`\mathrm{U}(2)`$ — the maximal compact subgroup of $`\mathrm{Sp}(4,)`$ —, whose first Chern class is precisely the degree of the representation. (Note that the reduction will not be in general compatible with the flat structure). One can also show that this degree measures the obstruction to the existence of a Lagrangian subbundle of the flat $`\mathrm{Sp}(4,)`$-bundle.

Let $`d`$ be an integer; recall that $`𝒳(d)𝒳`$ consists of those representations whose degree is $`d`$. The subset $`𝒳^+(d)𝒳(d)`$ is given by the reductive representations of degree $`d`$ and $`(d)`$ given by $`𝒳^+(d)/\mathrm{Sp}(4,)`$ is the moduli space of representations of degree $`d`$. Our main goal is the study of the non-emptiness and connectedness of $`(d)`$.

A restriction on the possible degrees of an element in $`𝒳`$ is given by the following Milnor-Wood type inequality( Turaev , Domic and Toledo ).

###### Proposition 2.7.

Let $`\rho `$ be a representation of $`\pi `$ in $`\mathrm{Sp}(4,)`$, and let $`d`$ be the degree of $`\rho `$. Then

$$|d|2g2.$$

Hence $`𝒳(d)`$ is empty for $`|d|>2g2`$.

## 3. $`\mathrm{Sp}(4,)`$-representations of $`\pi `$ and Higgs bundles

In this section we follow the methods of Hitchin  and Gothen . We refer to their papers for details.

### 3.1. $`\mathrm{Sp}(4,)`$-Higgs bundles

As in the case of complex representations of $`\pi `$, representations in $`\mathrm{Sp}(4,)`$ are related to Higgs bundles. To recall the basic ingredients of this theory, we fix from now on a complex structure on $`X`$. A Higgs bundle on $`X`$ is a pair $`(E,\mathrm{\Phi })`$, where $`E`$ is a holomorphic vector bundle over $`X`$ and $`\mathrm{\Phi }:EEK`$ is a holomorphic map, where $`K`$ is the canonical line bundle of $`X`$. The Higgs bundle $`(E,\mathrm{\Phi })`$ is said to be stable if for any proper subbundle $`FE`$ such that $`\mathrm{\Phi }(F)FK`$ we have $`\mu (F)<\mu (E)`$, where $`\mu (F)=\mathrm{deg}F/\mathrm{rk}F`$. The Higgs bundle $`(E,\mathrm{\Phi })`$ is said to be polystable if it is the direct sum of stable Higgs bundles of the same slope $`\mu (E)`$. This condition appears as the requirement to solve Hitchin’s equations. More precisely, we have the following.

###### Theorem 3.1.

 Let $`(E,\mathrm{\Phi })`$ be a Higgs bundle with $`\mathrm{deg}E=0`$. Then $`E`$ admits a Hermitian metric $`H`$ satisfying

(3.3)
$$F_H+[\mathrm{\Phi },\mathrm{\Phi }^_H]=0$$

if and only if $`(E,\mathrm{\Phi })`$ is polystable. (Here $`F_H`$ is the curvature of the unique connection compatible with $`H`$ and the holomorphic structure of $`E`$.) Furthermore, the set of metrics $`H`$ which solve (3.3) is convex, i.e., if $`H,H^{}`$ are both solutions then for any real number $`t[0,1]`$ the metric $`tH+(1t)H^{}`$ is also a solution.

The last statement of the theorem follows from the fact that if $`(E,\mathrm{\Phi })`$ is stable then there is a unique Hermitian metric $`H`$ solving (3.3).

The particular class of Higgs bundles that will be of relevance in relation to representations of $`\pi `$ in $`\mathrm{Sp}(4,)`$ is given by pairs $`(E,\mathrm{\Phi })`$, where $`E=VV^{}`$, with $`V`$ a rank 2 holomorphic vector bundle and

(3.4)
$$\mathrm{\Phi }=\left(\begin{array}{cc}0& \beta \\ \gamma & 0\end{array}\right):VV^{}(VV^{})K,$$

where $`\beta H^0(S^2VK)`$ and $`\gamma H^0(S^2V^{}K)`$.

A $`\mathrm{Sp}(4,)`$-Higgs bundle can be regarded as a pair $`(V,\phi )`$ where $`V`$ is a rank 2 holomorphic bundle and $`\phi =(\beta ,\gamma )H^0(S^2VK)H^0(S^2V^{}K)`$. Two $`\mathrm{Sp}(4,)`$-Higgs bundles $`(V,\phi )`$ and $`(V^{},\phi ^{})`$ are isomorphic if there is an isomorphism $`\psi :V^{}V`$ such that $`\phi ^{}=\psi ^{}\phi `$. Let $`d`$ be an integer. Let $`(d)`$ be the moduli space of polystable $`\mathrm{Sp}(4,)`$-Higgs bundles $`(V,\phi )`$ such that $`\mathrm{deg}V=d`$. By stability of $`(V,\phi )`$ we mean stability of the corresponding Higgs bundle $`(E,\mathrm{\Phi })`$, with $`E=VV^{}`$, and $`\mathrm{\Phi }`$ given by (3.4).

The following refinement of Theorem 3.3 is necessary to relate the moduli space of $`\mathrm{Sp}(4,)`$-Higgs bundles $`(d)`$ to $`(d)`$.

###### Theorem 3.2.

A $`\mathrm{Sp}(4,)`$-Higgs bundle $`(V,\phi )`$ is polystable if and only if there exists a Hermitian metric $`h`$ on $`V`$ such that

(3.5)
$$F_h+(\beta \beta ^_h\gamma ^_h\gamma )=0.$$

where $`F_h`$ denotes the curvature of the unique connection compatible with $`h`$ and the holomorphic structure of $`V`$.

###### Proof.

Suppose that we have a metric $`h`$ on $`V`$ for which (3.5) holds. Then, setting $`H=h+h^{}`$ we get a solution to (3.3), so that $`(E,\mathrm{\Phi })`$ is polystable and hence (by definition) $`(V,\phi )`$ is also polystable.

To prove the converse, assume that $`(V,\phi )`$ is polystable, i.e. $`(E,\mathrm{\Phi })`$ is polystable. By Theorem 3.3 there is a metric $`H`$ on $`E`$ which solves (3.3). We want to show that this metric can be taken of the form $`H=h+h^{}`$ with respect to the splitting $`E=VV^{}`$. We can view $`H`$ as a section of $`E\overline{E}^{}`$, and using the splitting

$$E\overline{E}^{}=V\overline{V}^{}V(\overline{V^{}})^{}V^{}\overline{V}^{}V^{}(\overline{V^{}})^{}$$

we may write $`H=H_{00}+H_{01}+H_{10}+H_{11}`$. We claim that if $`H^{}:=H_{00}H_{01}H_{10}+H_{11}`$ is also a solution to (3.3). To prove the claim, consider a local holomorphic framing $`e_1,\mathrm{},e_{2n}`$ of $`E`$ whose first (resp. last) $`n`$ sections give a framing of $`V`$ (resp. $`V^{}`$). Let $`M_H`$ be the $`2n\times 2n`$ matrix whose $`(i,j)`$ entry is $`e_i,e_j_H`$. Define $`\rho :=\mathrm{diag}(𝐢\mathrm{Id}_V,𝐢\mathrm{Id}_V^{})`$. A simple computation shows that $`M_H^{}=\rho M_H\rho ^1`$. On the other hand, if we identify $`\mathrm{\Phi }`$ with a matrix by means of our framing, then we have $`\mathrm{\Phi }^_H=(M_H^{})^1\mathrm{\Phi }^{}M_H^{},`$ from which we easily compute

(3.6)
$$[\mathrm{\Phi },\mathrm{\Phi }^_H^{}]=\rho [\rho ^1\mathrm{\Phi }\rho ,(\rho ^1\mathrm{\Phi }\rho )^_H]\rho ^1=\rho [\mathrm{\Phi },\mathrm{\Phi }^_H]\rho ^1$$

(the last equality follows from $`\rho ^1\mathrm{\Phi }\rho =\mathrm{\Phi }`$, which is a consequence of (3.4)). On the other hand, since we took a holomorphic framing the Chern connection takes the form, in our trivialisation, $`d_H=d+(M_H)M_H^1`$ (see for example p. 73 in ), from which we deduce that $`d_H^{}=\rho d_H\rho ^1`$ and hence $`F_H^{}=\rho F_H\rho ^1`$. This, together with (3.6), implies that $`H^{}`$ is another solution to (3.3). Consequently, $`H^{\prime \prime }:=H+H^{}`$ is also a solution to (3.3) which satisfies $`H^{\prime \prime }=h_V+h_V^{}`$ for some metric $`h_V`$ (resp. $`h_V^{}`$) on $`V`$ (resp. $`V^{}`$). To finish the argument, observe that we have an isomorphism $`f:EE^{}`$ defined as $`f(u,v)=(v,u)`$; now, if $`H`$ is a solution to (3.3), then both $`f^{}H`$ and $`H^{}`$ (the latter denotes the natural metric on $`E^{}`$ induced by $`H`$) give solutions to the Hermite–Einstein equations for the pair $`(E^{},\mathrm{\Phi }^{})`$. It follows that $`H^{\prime \prime }+(f^1)^{}(H^{\prime \prime })^{}`$ is a solution to (3.3) which is of the form $`h+h^{}`$ for some metric $`h`$ on $`V`$. This metric gives then a solution to (3.5). ∎

###### Remark 3.3.

Theorem 3.2 follows also from the general Hitchin–Kobayashi correspondence proved in .

### 3.2. Homeomorphism between $`(d)`$ and $`(d)`$

The moduli space $`(d)`$ of polystable $`\mathrm{Sp}(4,)`$-Higgs bundles can be constructed in essentially the same way as that of the moduli space of ordinary Higgs bundles (see §9 of ). Alternatively one can do the following: consider the map from isomorphism classes of polystable $`\mathrm{Sp}(4,)`$-Higgs bundles to isomorphism classes of polystable Higgs bundles which sends $`(V,\phi )`$ to $`(E,\mathrm{\Phi })`$; this map is finite to one, so there is a unique structure of reduced scheme on the set of isomorphism classes of polystable $`\mathrm{Sp}(4,)`$-Higgs bundles which is compatible both with the map and with the scheme structure on the set of isomorphism classes of polystable Higgs bundles which one gets by looking at the latter as a moduli space.

Following either of these procedures, one ends up in particular with a structure of topological space on $`(d)`$. We will sketch here a description of this topological structure on $`(d)`$ in gauge-theoretic terms. This discussion will also be relevant for us in order to define the proper function from which we plan to extract information about the number of connected components of $`(d)`$ (a similar study can be seen in §5 of ).

Let $`𝐕`$ be a $`C^{\mathrm{}}`$ complex vector bundle over $`X`$ of rank $`2`$ and degree $`d`$. Let us fix a Hermitian metric $`h`$ on $`𝐕`$, and denote by $`A`$ the affine space of Hermitian connections on $`𝐕`$. Let also $`\mathrm{\Omega }:=\mathrm{\Omega }^{1,0}(M;S^2𝐕S^2𝐕^{})`$ and let $`G`$ be the group of Hermitian automorphisms of $`𝐕`$ — the gauge group. Consider the completions of $`A`$ and $`\mathrm{\Omega }`$ (resp. $`G`$) with respect to Sobolev $`L_1^2`$ (resp. $`L_2^2`$) norm, and denote the resulting completions by the same symbols. Then the quotient $`B:=(A\times \mathrm{\Omega })/G`$ is a Hausdorff topological space (the crucial point, Hausdorffness, follows from the existence of slices of the action of $`G`$ on $`A\times \mathrm{\Omega }`$). Finally, we define $`𝒮(d)`$ to be the set of gauge equivalence classes $`[A,\phi ]B`$ such that

(3.7)
$$\{\begin{array}{c}\overline{}_A\phi =0,\hfill \\ F_A+(\beta \beta ^{}\gamma ^{}\gamma )=0,\hfill \end{array}$$

where $`\phi =(\beta ,\gamma )`$. The set $`𝒮(d)`$ inherits a topology from its inclusion in $`B`$. By Theorem 3.2, the points of $`𝒮(d)`$ are naturally in bijection with $`(d)`$, the moduli space of polystable $`\mathrm{Sp}(4,)`$-Higgs bundles of degree $`d`$ . This bijection maps a gauge equivalence class $`[A,\phi ]`$ to the pair $`(V,\phi )`$, where $`V`$ is $`𝐕`$ equipped with the holomorphic structure defined by the $`(0,1)`$ part of the connection $`A`$.

The following is an $`\mathrm{Sp}(4,)`$-version of the general correspondence between complex representations of $`\pi `$ and Higgs bundles ().

###### Theorem 3.4.

Let $`d`$ be an integer. There is a homeomorphism $`(d)(d)`$ which, when $`d0`$, restricts to a homeomorphism between the space of irreducible representations in $`(d)`$ and the space of stable Higgs bundles in $`(d)`$.

###### Proof.

In fact we prove that both $`(d)`$ and $`(d)`$ are homeomorphic to $`𝒮(d)`$. To see that $`𝒮(d)`$ is homeomorphic to $`(d)`$ we can consider the latter space from the complex analytic point of view below): consider pairs $`(\overline{}_V,\phi )`$, where $`\overline{}_V`$ is a $`\overline{}`$-operators on the $`C^{\mathrm{}}`$ vector bundle $`𝐕`$ underlying $`V`$ and $`\phi \mathrm{\Omega }`$. Let $`C`$ be the set of such pairs for which $`\phi `$ is holomorphic and the associated $`\mathrm{Sp}(4,)`$-Higgs bundle is polystable. We can then view $`(d)`$ as the quotient of $`C`$ by the complex gauge group. We clearly have an inclusion of the space of pairs $`(A,\phi )A\times \mathrm{\Omega }`$ which solve (3.7) into $`C`$, which descends to give a continuous map from $`𝒮(d)`$ to $`(d)`$. Theorem 3.2 now shows that this map is in fact a homeomorphism.

Suppose that $`(E=VV^{},\mathrm{\Phi })`$ represents a point in $`(d)`$, i.e. suppose that it is a polystable $`\mathrm{Sp}(4,)`$-Higgs bundle of degree $`d`$. From Theorem 3.2, there is a metric $`h`$ in $`V`$ satisfying (3.5). Rewriting the equations in terms of the Higgs connection $`D=d_A+\theta `$, where $`A`$ is the metric connection and $`\theta =\mathrm{\Phi }+\mathrm{\Phi }^{}`$, we see that $`D`$ is a flat $`\mathrm{Sp}(4,)`$-connection and thus defines a point in $`(d)`$. Notice that $`d_A`$ takes values in $`𝔲(2)`$, while $`\theta `$ takes values in the orthogonal (w.r.t. the Killing pairing) complement of $`𝔲(2)𝔰p(4,)`$. Conversely, by Corlette’s theorem , every representation in $`(d)`$ arises in this way. The fact that this correspondence gives a homeomorphism follows by the same argument as the one given in  for ordinary Higgs bundles. When $`d0`$, the solution to (3.7) is irreducible if and only if the corresponding $`\mathrm{Sp}(4,)`$-Higgs bundle is stable, hence the corresponding element in $`(d)`$ is irreducible. When $`d=0`$ the solution to (3.7) in a polystable $`\mathrm{Sp}(4,)`$-Higgs bundle may be actually $`\mathrm{Sp}(4,)`$-irreducible. The reason for this lies in the fact that we have used the standard stability of $`(E,\mathrm{\Phi })`$ as the stability criterium for the $`\mathrm{Sp}(4,)`$-Higgs bundle. There is, however, a notion of stability of the $`\mathrm{Sp}(4,)`$-Higgs bundle in its own rigth, i.e. without using that of the corresponding Higgs bundle $`(E,\mathrm{\Phi })`$ (). It turns out that these two notions are equivalent when $`d0`$, but when $`d=0`$, stability of the Higgs bundle $`(V,\phi )`$ is only equivalent to polystability. This is, however, no so important for us since in Theorem 1.5, where this is used, we assume that $`d0`$. ∎

### 3.3. A proper function on $`(d)`$

We follow the ideas of , which reduce the proof of the connectedness of $`(d)`$ to proving connectedness of a smaller subspace $`𝒩(d)(d)`$. Let us briefly recall how this goes.

We define, for any $`[A,\phi ]𝒮(d)(d)`$,

$$f([A,\phi ]):=\beta _{L^2}^2+\gamma _{L^2}^2,$$

where $`\phi =(\beta ,\gamma )`$. This expression is gauge invariant and hence descends to give a map

$$f:(d).$$

One can prove that $`f`$ is proper, essentially by using Uhlenbeck’s compactness theorem (see ). So any connected component of $`(d)`$ contains a local minimum of $`f`$ and hence, we have the following.

###### Proposition 3.5.

Let $`𝒩(d)(d)`$ be the local minima of $`f`$. Then $`(d)`$ is connected if $`𝒩(d)`$ is connected.

By duality we have $`(d)(d)`$, so it suffices to consider the case $`d<0`$. The key result characterizing $`𝒩(d)`$ is the following.

###### Proposition 3.6 (Gothen, ).

Suppose that $`d`$ satisfies $`2(g1)<d<0`$. Then $`𝒩(d)`$ consists of the classes $`[A,\phi ]`$ such that $`\gamma =0`$.

The following sections are devoted to proving that if $`2(g1)<d<0`$, the subspace $`𝒩(d)`$ is connected and contains stable $`\mathrm{Sp}(4,)`$-Higgs bundles, thus finishing the proof of Theorem 1.5, and hence the main theorems stated in the introduction. To study $`𝒩(d)`$, we take a more general point of view and consider a moduli problem that depends on a real parameter $`\alpha `$. We can then identify $`𝒩(d)`$ with the moduli space for $`\alpha =0`$.

## 4. Quadratic pairs

### 4.1.

A quadratic pair $`(V,\beta )`$ on $`X`$ consists by definition of a holomorphic vector bundle $`V`$ on $`X`$ of rank $`2`$ and a holomorphic section $`\beta H^0(KS^2V)`$, where $`K`$ is the canonical bundle of $`X`$. The degree of $`(V,\beta )`$ is by definition the degree of $`V`$. Let $`\alpha `$. We say that $`(V,\beta )`$ is $`\alpha `$-polystable if, denoting by $`d`$ the degree of $`(V,\beta )`$,

1. $`d/2\alpha `$, and if $`\beta =0`$ then $`\alpha =d/2`$,
2. for any subbundle $`LV`$

1. if $`\beta H^0(KS^2L)`$ then $`\mathrm{deg}Ld\alpha `$, and in case there is equality there is a splitting $`V=LL^{}`$;
    2. if $`\beta H^0(KLV)`$ then $`\mathrm{deg}Ld/2`$, and if there is equality then we have a splitting $`V=LL^{}`$ in such a way that $`\beta H^0(KLL^{})`$;
    3. $`\mathrm{deg}L\alpha `$ in any case, and if there is equality then there is a splitting $`V=LL^{}`$.

We say that $`(V,\beta )`$ is $`\alpha `$-stable if equality never occurs in the inequalities required for polystability.

For any rational value of $`\alpha `$ there exists an algebraic coarse moduli space $`𝒩_\alpha (d)`$ for the moduli problem of families of $`\alpha `$-polystable quadratic pairs over $`X`$ of degree $`d`$. As in the case of the moduli space of symplectic Higgs bundles, we will not describe the algebraic construction of $`𝒩_\alpha (d)`$, since for our purposes it suffices to have a topological description of it. To construct the topological space underlying $`𝒩_\alpha (d)`$ one uses the same strategy as in Section 3.2. Namely, to use a version of the Hitchin–Kobayashi correspondence for $`\alpha `$-polystable quadratic pairs (analogous to Theorem 3.2), a particular case of the more general correspondence proved in , and then use gauge theory.

It is clear that a quadratic pair is a special case of $`\mathrm{Sp}(4,)`$-Higgs bundle. The next proposition says that when $`\alpha =0`$ the two possible notions of (poly)stability coincide.

###### Proposition 4.1.

Let $`2(g2)<d<0`$ be an integer, and let $`(V,\beta )`$ be a quadratic pair of degree $`d`$. Then, $`(V,\beta )`$ is a polystable (resp. stable) $`\mathrm{Sp}(4,)`$-Higgs bundle if and only if it is a $`0`$-polystable (resp. $`0`$-stable) quadratic pair. In other words, $`𝒩(d)=𝒩_0(d)`$.

###### Proof.

That polystability as $`\mathrm{Sp}(4,)`$-Higgs bundle implies $`0`$-polystability as quadratic pair is obvious. To prove the converse, let $`p:VV^{}V`$ and $`q:VV^{}V^{}`$ denote the projections. Assume that $`(V,\beta )`$ is $`0`$-polystable as quadratic pair, and consider some subbundle $`WVV^{}`$ satisfying $`\mathrm{\Phi }(W)KW`$. We need to check that $`\mathrm{deg}W0`$, and that equality implies splitting. We will prove the first claim, since the second one follows from very similar argument. Define $`A=p(W)`$, $`B=q(W)`$, $`A^{}=WV`$ and $`B^{}=WV^{}`$. Using $`0`$-polystability it is straightforward to check that

(4.8)
$$\mathrm{deg}(A^{}+B)+\mathrm{deg}(A+B^{})0$$

(simply consider the subbundle $`A+B^{}V`$). One computes

$`\mathrm{deg}(A+B^{})`$ $`=\mathrm{deg}A+\mathrm{deg}B\mathrm{deg}(A^{}+B)`$
$`\mathrm{deg}(A^{}+B)`$ $`=\mathrm{deg}A+\mathrm{deg}B\mathrm{deg}(A+B^{}).`$
Adding up and taking (4.8) into account we get $`\mathrm{deg}A+\mathrm{deg}B0`$. A similar argument implies that $`\mathrm{deg}A^{}+\mathrm{deg}B^{}0`$. Finally, combining these two inequalities with the exact sequences

$$0B^{}WA0\text{ and }0A^{}WB0$$

we get the required inequality $`\mathrm{deg}W0`$. ∎

### 4.2. Familes of quadratic pairs and moduli space

If $`U`$ is a scheme of finite type (resp. a complex manifold) we define a family of quadratic pairs on $`X`$ parametrized by $`U`$ to be a pair $`(V,B)`$, where $`V`$ is a rank $`2`$ vector bundle on $`U\times X`$ and $`BH^0(\pi _X^{}KS^2V)`$ is an algebraic (resp. holomorphic) section (here $`\pi _X:U\times XX`$ denotes the projection).

The following lemma states that stability for algebraic quadratic pairs is an open condition in the Zariski topology, as one would naturaly expect.

###### Lemma 4.2.

Fix a real number $`\alpha `$. Let $`U`$ be a scheme of finite type and let $`(V,B)`$ be an algebraic family of quadratic pairs on $`X`$ parametrized by $`U`$. For any $`uU`$ let $`(V_u,B_u)`$ denote the restriction of $`(V,B)`$ to $`X\times \{u\}`$. The set

$$U_s(V,B)=\{uU(V_s,B_s)\text{ is }\alpha \text{-stable }\}$$

is a Zariski open subset of $`U`$.

###### Proof.

Fix $`d`$ and let $`J:=\mathrm{Jac}_dX`$. Let $`PJ\times X`$ be the Poincaré bundle. Define $`F:=(\pi _U)_{}(\pi _{U\times X}^{}V\pi _{J\times X}^{}P^{}),`$ where $`\pi _U:U\times J\times XU`$ is the projection and the other maps $`\pi _\mathrm{?}`$ are the obvious analogs. Let $`LJ\times X`$ be a line bundle of high enough degree such that the natural map $`\pi _U^{}\pi _{U}^{}{}_{}{}^{}(\pi _{J\times X}^{}P^{}L)\pi _{J\times X}^{}P^{}L`$ is surjective, $`𝐑^j\pi _{U}^{}{}_{}{}^{}(\pi _{J\times X}^{}P^{}L)=0`$ for any $`j>0`$, and the same thing applies for $`\pi _{U\times X}^{}VL`$, $`\pi _{U\times X}^{}V^{}L`$ and $`\pi _X^{}KL^2`$. Denote by $``$ the image sheaf of the canonical map $`F\pi _{U}^{}{}_{}{}^{}(\pi _{J\times X}^{}P^{}L)\pi _{U}^{}{}_{}{}^{}(\pi _{U\times X}^{}VL)`$, and let $`G:=^{}\pi _{U}^{}{}_{}{}^{}(\pi _{U\times X}^{}V^{}L)`$ be the orthogonal subsheaf. Denote by $`Q:V^{}K`$ be the quadratic map induced by $`B`$ (of course this is not a morphism of vector bundles!). The map $`Q`$ induces another map $`Q:\pi _{U}^{}{}_{}{}^{}(\pi _{U\times X}^{}V^{}L)\pi _{U}^{}{}_{}{}^{}(\pi _X^{}KL^2)`$. Finally, let $`H:=GQ^1(0)`$. This is a coherent sheaf on $`U`$. Let $`U_d`$ be the support of $`H`$ with the reduced scheme structure. This is by definition a closed subscheme of $`U`$ with the property that a closed point $`uU`$ belongs to $`U_d`$ if and only if there is a line subbundle $`LV_u`$ with degree $`\mathrm{deg}Ld`$ and such that $`B_uH^0(LV_uK)`$. Similarly, one proves that the subscheme $`U_d^{}U`$ whose closed points are the $`uU`$ such that there is a line subbundle $`LV_u`$ with degree $`\mathrm{deg}Ld`$ satisfying $`B_uH^0(L^2K)`$ is closed in $`U`$, and that the subscheme $`U_d^{\prime \prime }U`$ whose closed points are the $`uU`$ such that there is a line subbundle $`LV_u`$ with degree $`\mathrm{deg}Ld`$ is also closed in $`U`$. ∎

###### Corollary 4.3.

Let $`U`$ be an irreducible algebraic manifold and let $`(V,B)`$ be a (holomorphic) family of quadratic pairs on $`X`$ parametrized by $`U`$. The set

$$U_s(V,B)=\{uU(V_s,B_s)\text{ is }\alpha \text{-stable }\}$$

is connected.

###### Proof.

By Serre’s GAGA the holomorphic family $`(V,B)`$ is induced by an algebraic family $`(V^{\mathrm{alg}},B^{\mathrm{alg}})`$. We then have $`U_s(V,B)=U_s(V^{\mathrm{alg}},B^{\mathrm{alg}})`$, and the latter is connected in the standard topology, since it is a Zariski open subset of the irreducible manifold $`U`$. ∎

The following continuous version of the algebraic notion of family will also be relevant for us. We say that a quadratic pair $`(V,\beta )`$ can be (continuously) deformed to another pair $`(V^{},\beta ^{})`$ if and only if $`V`$ and $`V^{}`$ are isomorphic as smooth vector bundles and, denoting by $`𝐕`$ the $`C^{\mathrm{}}`$ complex vector bundle on $`X`$ underlying both $`V`$ and $`V^{}`$ and by $`\overline{}_V`$ (resp. $`\overline{}_V^{}`$) the $`\overline{}`$-operator giving rise to the holomorphic structure of $`V`$ (resp. $`V^{}`$), there are continuous maps $`D:[0,1]\mathrm{\Omega }^{0,1}(\mathrm{End}𝐕)`$ and $`B:[0,1]\mathrm{\Omega }^0(S^2𝐕K)`$ such that:

* $`D(0)=0`$ and $`D(1)=\overline{}_V^{}\overline{}_V`$;
* $`B(0)=\beta `$ and $`B(1)=\beta ^{}`$;
* for any $`t[0,1]`$ we have $`(\overline{}_V+D(t))B(t)=0`$.

Denote by $`(V_t,\beta _t)`$ the quadratic pair defined by $`(\overline{}_V+D(t),B(t))`$. We say that the continuous deformation $`(D,B)`$ goes through $`\alpha `$-(poly)stable pairs if for any $`t`$ the pair $`(V_t,\beta _t)`$ is $`\alpha `$-(poly)stable.

### 4.3. Geometry of quadratic pairs

#### 4.3.1. Quadratic forms in $`^2`$

Let $`𝕍=^2`$, and let $`x,y𝕍`$ be a basis. Recall that the discriminant of $`f=ax^2+bxy+cy^2S^2𝕍`$ is by definition $`\mathrm{\Delta }_f:=(b^24ac)(xy)^2(\mathrm{\Lambda }^2𝕍)^2.`$ It is clear that $`\mathrm{\Delta }:S^2𝕍(\mathrm{\Lambda }^2𝕍)^2`$ is a $`\mathrm{GL}(𝕍)`$ equivariant map. Define the following subsets of $`S^2𝕍`$: $`O_0=\{0\}`$, $`O_1=\{fS^2𝕍f0,\mathrm{\Delta }_f=0\}`$ and $`O_2=\{fS^2𝕍f0,\mathrm{\Delta }_f0\}`$.

###### Lemma 4.4.

(i) An element $`0fS^2𝕍`$ belongs to $`O_1`$ if and only if there exists $`0x𝕍`$ so that $`f=x^2`$, and in this case the span $`x𝕍`$ only depends on $`f`$. (ii) $`f`$ belongs to $`O_2`$ if and only if there exist linearly independent elements $`x,y𝕍`$ so that $`f=xy`$, and in this case the set of lines $`\{x,y\}`$ only depends on $`f`$. (iii) The orbits of the action of $`\mathrm{GL}(𝕍)`$ on $`S^2𝕍`$ are $`O_0,O_1,O_2`$.

###### Proof.

(i) and (ii) are easy, and (iii) follows from them. ∎

One readily sees (for example, by computing the dimension of the stabilizers of $`\mathrm{GL}(𝕍)`$ acting on $`S^2𝕍`$) that $`O_2S^2𝕍`$ is open. This implies the following.

###### Lemma 4.5.

Let $`fS^2𝕍`$, and let $`\rho _f:\mathrm{End}(𝕍)T_f(S^2𝕍)S^2𝕍`$ be the map given by the infinitesimal action of $`\mathrm{End}(𝕍)=\mathrm{Lie}\mathrm{GL}(𝕍)`$ on $`S^2𝕍`$. If $`fO_2`$, then $`\rho _f`$ is onto.

The following lemma is straightforward.

###### Lemma 4.6.

Let $`fS^2𝕍`$, and let $`\rho _f:\mathrm{End}(𝕍)T_f(S^2𝕍)S^2𝕍`$ be as above. If $`f=xyO_2`$ then, in the basis $`x,y`$ of $`𝕍`$,

$$\mathrm{Ker}\rho _f=\{\left(\begin{array}{cc}\lambda & 0\\ 0& \lambda \end{array}\right)\lambda \}.$$

###### Lemma 4.7.

If $`f,gS^2𝕍`$ are linearly independent, then for a generic value of $`\lambda `$ $`f+\lambda g`$ belongs to $`O_2`$.

###### Proof.

We have to check that $`O_1`$ does not contain any plane. This is equivalent to proving that the subset $`O_1(S^2𝕍)`$ induced by $`O_1`$ does not cointain any line. But $`O_1`$ is given by the equation $`4ac=b^2`$, hence is a nondegenerate conic and does not contain any line. ∎

#### 4.3.2. Discriminant of quadratic pairs

Extending fibrewise the definition of the discriminant we get a quadratic map $`\mathrm{\Delta }:KS^2VK^2(\mathrm{\Lambda }^2V)^2.`$ For any section $`\beta :𝒪KS^2V`$ we denote $`\mathrm{\Delta }\beta `$ by $`\mathrm{\Delta }_\beta `$. Note that we always have $`\beta ^1(0)\mathrm{\Delta }_\beta ^1(0)`$. It follows from Lemma 4.4 that, given a pair $`(V,\beta )`$, we can distinguish four possibilities.

1. $`\beta =0`$.
2. $`\beta 0`$ and $`\mathrm{\Delta }_\beta =0`$. Then there exists a line subbundle $`LV`$ so that $`\beta H^0(KL^2)`$.
3. $`\mathrm{\Delta }_\beta 0`$ and there exists a square root $`\mathrm{\Delta }^{1/2}(\beta )H^0(K\mathrm{\Lambda }^2V)`$; then there exist two different line subbundles $`L_1,L_2V`$ so that $`\beta H^0(KL_1L_2)`$.
4. There is no square root of $`\mathrm{\Delta }_\beta `$; then there exists no line subbundle $`LV`$ so that $`\beta H^0(KLV)`$.

To understand the difference between (3) and (4), observe that by Lemma 4.4 for any $`xX`$ such that $`\mathrm{\Delta }_\beta (x)0`$ one has two different lines $`L_{x,1},L_{x,2}V_x`$. As $`x`$ moves around $`X\mathrm{\Delta }_\beta ^1(0)`$ these two lines give rise to a pair of line subbundles of $`V`$, unless the monodromy around points in $`\mathrm{\Delta }_\beta ^1(0)`$ interchanges the lines. Now, the existence of a square root of $`\mathrm{\Delta }_\beta `$ is equivalent to the triviality of the action of the monodromy on the pair of lines.

###### Lemma 4.8.

Suppose that a pair $`(V,\beta )`$ defined on $`X`$ satisfies $`\mathrm{\Delta }=\mathrm{\Delta }_\beta 0`$ and $`\mathrm{\Delta }^{1/2}`$ exists. Let $`L_1,L_2V`$ be the two different line bundles so that $`\beta H^0(KL_1L_2)`$. Then: (i) the pair $`\{L_1,L_2\}`$ is uniquely determined by $`V`$ and $`\beta `$; (ii) there is an exact sequence of sheaves

$$0L_1L_2V𝒪_T0;$$

here $`T`$ is the divisor in $`X`$ defined as $`T=_{xX}(r(x)/2z(x))x`$, where $`r(x)`$ (resp. $`z(x)`$) is the vanishing order of $`\mathrm{\Delta }`$ (resp. $`\beta `$) at $`x`$.

###### Proof.

(i) is true because the induced quadratic map $`\gamma :V^{}K`$ vanishes exactly at $`L_1^{}L_2^{}`$; (ii) follows from the next lemma. ∎

###### Lemma 4.9.

Let $`(V,\beta )`$ be a pair defined on some disk $`𝔻X`$, and assume that $`\mathrm{\Delta }=\mathrm{\Delta }_\beta 0`$, $`\mathrm{\Delta }^1(0)=\{0\}`$, and there is a square root $`\mathrm{\Delta }^{1/2}:𝔻`$ of $`\mathrm{\Delta }`$. Let $`r`$ (resp. $`z`$) be the vanishing order of $`\mathrm{\Delta }`$ (resp. $`\beta `$) at $`0`$. Let $`\theta `$ be a coordinate in $`𝔻`$. Then: (i) we have $`r/2z`$; (ii) let $`L_1,L_2V`$ be the line subbundles so that $`\beta H^0(KL_1L_2)`$; we have an exact sequence of sheaves: $`0L_1L_2V𝒪/\theta ^{r/2z}𝒪0`$.

###### Proof.

Pick a trivialisation $`V𝔻\times x,y`$ so that $`L_1=𝔻\times x`$. Then we can write $`\beta =x(ax+by)`$, where $`a,b`$ are holomorphic functions on $`𝔻`$, and we have $`\mathrm{\Delta }=b^2`$. We then have: $`a=\theta ^z(a_0+\theta a_1)`$ and $`b=\theta ^{r/2}(b_0+\theta b_1)`$, where $`a_0,b_0^\times `$ and $`a_1,b_1`$ are holomorphic. To prove (i), observe that $`\theta ^z`$ divides $`b`$. (ii) follows from $`L_2=ax+by`$. ∎

### 4.4. Local study of moduli space

Take an isomorphism class $`[V,\beta ]𝒩_\alpha (d)`$. By a result of Biswas and Ramanan  the Zariski tangent space of $`𝒩_\alpha (d)`$ at $`[V,\beta ]`$ is given by the first hypercohomology group of the following $`2`$-term complex:

$$C_{V,\beta }:\mathrm{End}V\psi \rho _{V,\beta }(\psi )KS^2V,$$

where $`\rho _{V,\beta }`$ is the map induced fibrewise by the infinitesimal action of $`\mathrm{End}(^2)`$ on $`S^2^2`$ along $`\beta KS^2V`$. Applying hypercohomology to this exact sequence of complexes

$$0KS^2V[1]C_{V,\beta }\mathrm{End}V[0]0$$

(as usual, if $`E`$ is a sheaf, $`E[d]`$ denotes the complex whose only nonzero term is a copy of $`E`$ in the position $`d`$) we get the following long exact sequence:

(4.9) $`0`$ $`^0(C_{V,\beta })H^0(\mathrm{End}V)H^0(KS^2V)`$
$`^1(C_{V,\beta })H^1(\mathrm{End}V)H^1(KS^2V)^2(C_{V,\beta })0.`$
###### Lemma 4.10.

Suppose that $`^0(C_{V,\beta })=^2(C_{V,\beta })=0`$. Then $`dim^1(C_{V,\beta })=7(g1)+3d.`$

###### Proof.

Using (4.9) and Riemann–Roch we have

$`dim^1(C_{V,\beta })`$ $`=\chi (C_{V,\beta })=\chi (\mathrm{End}V)+\chi (KS^2V)`$
$`=4(1g)+(3(2g2)+3d+3(1g))=7(g1)+3d.`$
Biswas and Ramanan also prove that the points $`[V,\beta ]𝒩_\alpha (d)`$ at which $`^0=^2=0`$ are smooth. The preceeding lemma (together with the identification $`T_{V,\beta }𝒩_\alpha (d)^1`$ given by Biswas and Ramanan) implies that the dimension of $`𝒩_\alpha (d)`$ at these points is $`7(g1)+3d`$.

### 4.5. Vanishing of $`^0`$

###### Lemma 4.11.

Suppose that $`(V,\beta )`$ is $`\alpha `$-polystable for some $`\alpha `$ and $`\mathrm{\Delta }_\beta 0`$. Then we have $`^0(C_{V,\beta })=0`$ unless there is a splitting $`V=L_1L_2`$ in line bundles with $`\mathrm{deg}L_1=\mathrm{deg}L_2=d/2`$ and $`\beta H^0(KL_1L_2)`$.

###### Proof.

Thanks to (4.9) the vanishing of $`^0`$ is equivalent to the injectivity of the map $`H^0(\mathrm{End}V)H^0(KS^2V)`$. Suppose the map is not injective. Then there is some nonzero $`sH^0(\mathrm{End}V)`$ so that $`\rho _{V,\beta }(s)=0`$.

Since $`\mathrm{\Delta }_\beta 0`$, Lemma 4.6 implies that if $`xX`$ and $`s(x)0`$ then $`s(x)`$ has two different eigenvalues. On the other hand, since the characteristic polynomial of $`s`$ has holomorphic coefficients in $`X`$, hence constant coefficients, the eigenvalues of $`s(x)`$ are constant as $`x`$ varies along $`X`$. Hence we can split $`V=L_1L_2`$ in eigensubbundles of $`s`$. Using again Lemma 4.6, we know that $`\beta H^0(KL_1L_2)`$. Applying the semistability condition we get $`\mathrm{deg}L_1d/2`$, $`\mathrm{deg}L_2d/2`$, and hence $`\mathrm{deg}L_1=\mathrm{deg}L_2=d/2`$. ∎

### 4.6. Vanishing of $`^2`$

###### Lemma 4.12.

Assume that $`\mathrm{\Delta }_\beta 0`$. Then $`^2(C_{V,\beta })=0`$.

###### Proof.

It suffices to show that the map $`H^1(\mathrm{End}V)H^1(KS^2V)`$ in (4.9) is onto (this map is the one induced by $`\rho =\rho _{V,\beta }`$ in cohomology). By Serre duality, this is equivalent to the injectivity of the map $`H^0(S^2V^{})H^0(K\mathrm{End}V^{})`$ induced by $`\tau _K(\rho ^{}):S^2V^{}K\mathrm{End}V^{}`$. But it follows from Lemma 4.5 that the map of sheaves $`\rho :\mathrm{End}VKS^2V`$ is onto, so the map $`\rho ^{}:K^{}S^2V^{}\mathrm{End}V^{}`$ is injective. Then, by Lemma 4.13 below, $`\tau _K(\rho ^{}):S^2V^{}K\mathrm{End}V^{}`$ is also injective. ∎

###### Lemma 4.13.

Let $`L`$ be a line bundle on $`X`$ and let $`A,B`$ be coherent sheaves on $`X`$. Let $`\tau _L:\mathrm{Hom}(L^{}A,B)\mathrm{Hom}(A,LB)`$ be the standard isomorphism and let $`f\mathrm{Hom}(L^{}A,B)`$. Then $`\mathrm{Ker}\tau _L(f)=L\mathrm{Ker}f.`$

###### Proof.

The sheaf $`L`$ is flat since it is locally free, and hence the functor $`L:\mathrm{Coh}(X)\mathrm{Coh}(X)`$ is exact. But $`\tau _L`$ is the map induced by $`L`$ on morphisms in $`\mathrm{Coh}(X)`$. ∎

## 5. Proof of Theorem 1.4

We briefly sketch our strategy. We first classify the pairs $`(V,\beta )`$ in three types as follows:

* type A: pairs $`(V,\beta )`$ with $`\mathrm{\Delta }_\beta =0`$;
* type B: pairs $`(V,\beta )`$ with $`\mathrm{\Delta }_\beta 0`$ and $`^0(C_{V,\beta })0`$;
* type C: pairs $`(V,\beta )`$ with $`\mathrm{\Delta }_\beta 0`$ and $`^0(C_{V,\beta })=0`$.

In section 5.1 we prove that any $`\alpha `$-polystable pair of type A can be deformed to a $`\alpha `$-polystable pair of type B or C, and in section 5.2 we prove that any $`\alpha `$-polystable pair of type B can be deformed to a $`\alpha `$-polystable pair of type C (in both cases, and everywhere below, we mean continuous deformations). Finally, in section 5.3 we prove that the subset of $`𝒩_\alpha (d)`$ consisting of pairs of type $`C`$ is indeed connected. In Subsection 5.4 we prove that $`𝒩_\alpha (d)`$ contains at least one stable object, thus proving the last claim in Theorem 1.4.

### 5.1. Pairs of type A

In this section we study the $`\alpha `$-polystable pairs $`(V,\beta )`$ for which $`\mathrm{\Delta }_\beta =0`$. By Subsection 4.3.2, for any such pair there is a line subbundle $`LV`$ so that $`\beta H^0(KL^2)`$. Let $`l`$ be the degree of $`L`$. Since $`0\beta H^0(KL^2)`$, we have $`\mathrm{deg}KL^2=2g2+2l0`$. Also, if $`(V,\beta )`$ is $`\alpha `$-polystable for some $`\alpha 0`$ then we must have $`ld`$. So in this section we assume that $`d(g1)`$.

###### Lemma 5.1.

There exists a manifold $`S_{d,l}`$ and a family of pairs $`(V,B)`$ on $`X`$ parametrized by $`S_{d,l}`$ such that any pair $`(V,\beta )`$ with $`\mathrm{deg}V=d`$, $`\beta H^0(KL^2)`$ and $`\mathrm{deg}L=l`$, is isomorphic to $`(V|_{\{s\}\times X},B|_{\{s\}\times X})`$ for at least one $`sS_{d,l}`$.

###### Proof.

Let $`j=2g2+2l`$, and let $`\mu :S^jX\mathrm{Jac}_j(X)`$ be the map which sends an effective divisor $`D`$ to the bundle $`𝒪(D)`$. Let also $`q:\mathrm{Pic}^l(X)\mathrm{Jac}_j(X)`$ the map defined as $`q(L)=KL^2`$. Finally, let $`\mathrm{\Sigma }=S^jX\times _{\mathrm{Jac}_j(X)}\mathrm{Jac}_l(X).`$ Since $`\mu `$ is étale it follows that $`\mathrm{\Sigma }`$ is a manifold. Let $`\lambda :\mathrm{\Sigma }\mathrm{Jac}_l(X)`$ be the projection. We define $`S_{d,l}`$ to be the vector bundle over $`\mathrm{\Sigma }\times \mathrm{Jac}_{dl}(X)`$ whose fibre over $`(\sigma ,L^{})`$ is $`\mathrm{Ext}^1(L^{},\lambda (\sigma ))`$ (this vector space has constant dimension because for any $`(L,L^{})\mathrm{Jac}_l(X)\times \mathrm{Jac}_{dl}(X)`$ we have $`H^0(X;LL_{}^{}{}_{}{}^{})=0`$, since $`\mathrm{deg}LL_{}^{}{}_{}{}^{}=2ld<0`$). It is clear how to construct the family $`(V,B)`$. ∎

###### Lemma 5.2.

Suppose that $`(g1)<d0`$. There is a nonempty dense Zariski open subset $`S_{d,l}^{}S_{d,l}`$ such that for any $`sS_{d,l}^{}`$ the pair $`(V|_{\{s\}\times X},B|_{\{s\}\times X})`$ can be deformed to a pair with $`\mathrm{\Delta }0`$ through a path of $`\alpha `$-stable pairs.

###### Proof.

Let us prove that generically $`h^0(KL^2)<h^0(KS^2V)`$. This means that there is some section $`\beta ^{}`$ of $`KS^2`$ which is not entirely contained in $`KL^2`$. Then Lemma 4.7 implies that for generic $`\lambda `$ the section $`\beta (\lambda )=\beta +\lambda \beta ^{}`$ satisfies $`\mathrm{\Delta }_{\beta (\lambda )}0`$, thus proving the claim.

First of all, we have $`\chi (KL^2)<\chi (KS^2V).`$ Indeed, $`\chi (KS^2V)=3(g1+d)`$ and $`\chi (KL^2)=g1+2l`$. Now,

$$g1+2l<3(g1+d)0<(2(g1)+d)+(2d2l),$$

and the right hand side follows from $`2(g1)+d>0`$ (because $`d(g1)`$) and $`2d2l0`$ (by stability). On the other hand, since $`(g1)<d`$ we have

(5.10)
$$h^0(KS^2V)\chi (KS^2V)3.$$

Now it follows from standard Brill–Noether theory  that for a generic $`L`$ in $`\lambda (\mathrm{\Sigma })\mathrm{Jac}_l(X)`$ we have $`h^0(KL^2)=\mathrm{max}\{1,\chi (KL^2)\}.`$ If $`h^0(KL^2)=\chi (KL^2)`$ then we are done, since we have $`\chi (KL^2)<\chi (KS^2V)h^0(KS^2).`$ If instead $`h^0(KL^2)=1`$, then we simply use (5.10). ∎

###### Lemma 5.3.

Suppose that $`(g1)<d<0`$. Any $`\alpha `$-stable element in the family $`(V,B)`$ can be deformed to a pair with $`\mathrm{\Delta }0`$ through a path of $`\alpha `$-stable pairs.

###### Proof.

Let $`S_{d,l}^sS_{d,l}`$ be the set of points $`sS_{d,l}^s`$ for which $`(V_{\{s\}\times X},B_{\{s\}\times X})`$ is $`\alpha `$-stable. By Corollary 4.3, $`S_{d,l}^s`$ is connected. If $`S_{d,l}^s`$ is empty, then there is nothing to prove. Otherwise, by Lemma 5.2 the intersection $`S_{d,l}^sS_{d,l}^{}`$ is nonempty, and the result follows. ∎

###### Lemma 5.4.

A $`\alpha `$-polystable element in the family $`(V,B)`$ parametrized by $`S_{(g1),(g1)}`$ can be deformed to a pair with $`\mathrm{\Delta }0`$ through a path of $`\alpha `$-stable pairs.

###### Proof.

By definition, a $`\alpha `$-polystable element $`(V,\beta )`$ appearing in the family $`S_{(g1),(g1)}`$ must split $`V=LL^{}`$, where $`L^2=K`$ and $`\mathrm{deg}L^{}=0`$. Now, $`S^2=L^2LL^{}L_{}^{}{}_{}{}^{2}`$ and $`h^0(KL_{}^{}{}_{}{}^{2})\chi (KL_{}^{}{}_{}{}^{2})=g1`$, so $`h^0(KL^2)<h^0(KS^2V)`$. The result now follows from the same argument as in the preceeding lemma. ∎

### 5.2. Pairs of type B

By Lemma 4.11 the pairs $`(V,\beta )`$ which are $`\alpha `$-polystable for some value of $`\alpha `$ and so that $`\mathrm{\Delta }_\beta 0`$ and $`^0(C_{V,\beta })0`$ are precisely those of the form

(5.11)
$$V=L_1L_2,\mathrm{deg}L_1=\mathrm{deg}L_2=d/2,\beta H^0(KL_1L_2).$$

In this section we assume that $`\delta =d/2`$ is an integer.

###### Lemma 5.5.

Any pair $`(V,\beta )`$ of the form (5.11) is $`\alpha `$-polystable for any $`\alpha 0`$.

###### Proof.

If $`LL_1L_2`$ is a line subbundle, then at least for one value of $`i`$ the projection $`LL_i`$ is nonzero, from which it follows that $`\mathrm{deg}L\mathrm{deg}L_i=d/2<0`$. We cannot have $`\beta H^0(KL^2)`$, so $`\mathrm{deg}Ld/2<0`$ implies $`\alpha `$-polystability for any $`\alpha 0`$. ∎

###### Lemma 5.6.

There exists an irreducible manifold $`S_d`$ an a family of pairs $`(V,B)`$ on $`X`$ parametrized by $`S_d`$ such that any pair $`(V,\beta )`$ of the form (5.11) is isomorphic to $`(V|_{\{s\}\times X},B|_{\{s\}\times X})`$ for at least one $`sS_d`$.

###### Proof.

Define $`S_d=S^{2g2+d}X\times \mathrm{Jac}_\delta (X)`$. Let $`\pi :S_d\times X\mathrm{Jac}_\delta (X)\times \mathrm{Jac}_\delta (X)\times X`$ be the map defined by $`\pi (D,L,x)=(\mu (D)K^1L^1,L,x),`$ where $`\mu :S^{2g2+d}X\mathrm{Jac}_{2g2+d}(X)`$ sends an effective divisor to the line bundle it represents. We have two Poincaré bundles $`L_1,L_2\mathrm{Jac}_\delta (X)\times \mathrm{Jac}_\delta (X)\times X`$ corresponding to each Jacobian. Let us define $`V=\pi ^{}L_1\pi ^{}L_2`$.

On the other hand, we have a canonical bundle $`L`$ over $`S^{2g2+d}X\times X`$ (the pullback of the Poincaré bundle through the projection $`\mu \times \mathrm{Id}`$) and a canonical section $`\beta H^0(L)`$. Furthermore, we have an isomorphism $`\pi _X^{}K\pi ^{}L_1\pi ^{}L_2p^{}L,`$ where $`p:S^{2g2+d}X\times \mathrm{Jac}_\delta (X)\times XS^{2g2+d}X\times X`$ is the projection. Then we define $`B`$ to be the canonical section $`H^0(\pi _X^{}K\pi ^{}L_1\pi ^{}L_2)`$ induced by $`\beta `$. It is clear that $`S_d`$ is irreducible and that the family $`(V,B)`$ represents every isomorphism class of pairs of the form (5.11). ∎

###### Lemma 5.7.

Any $`\alpha `$-polystable pair of the form (5.11) can be deformed through a path of $`\alpha `$-stable pairs to a pair $`(V,\beta )`$ with $`\mathrm{\Delta }_\beta 0`$ and $`^0(C_{V,\beta })=0`$.

###### Proof.

By Lemmae 5.5 and 5.6 it suffices to prove that at least one pair of the form (5.11) can be deformed to a pair $`(V,\beta )`$ with $`\mathrm{\Delta }_\beta 0`$ and $`^0(C_{V,\beta })=0`$. We will find two nonisomorphic line bundles $`L_1,L_2`$ of degree $`\delta `$ and nonzero sections $`\beta H^0(KL_1L_2)`$ and $`\gamma H^0(KL_1^2)`$. Then the family $`\{(V_ϵ,\beta _ϵ)=(L_1L_2,\beta +ϵ\gamma )ϵ𝔻\}`$ is such desired deformation, since $`(V_ϵ,\beta _ϵ)`$ is of the form (5.11) if and only if $`ϵ=0`$. To find $`L_1,L_2`$, take two different points $`\lambda ,\lambda ^{}\mu (S^{2g2+d}X)`$ and solve the following system of linear equations in $`\mathrm{Jac}_{2g2+d}(X)`$: (i) $`K+L_1+L_2=\lambda `$, (ii) $`K+2L_1=\lambda ^{}`$. ∎

### 5.3. Pairs of type C

Let $`(V,\beta )`$ be a pair with $`\mathrm{\Delta }=\mathrm{\Delta }_\beta 0`$, and let $`x=(x_1,\mathrm{},x_k)`$ be the vanishing locus of $`\mathrm{\Delta }`$. Define the type of $`(V,\beta )`$ (with respect to the ordering of the elements of $`x`$) to be

$$T=T(V,\beta )=((r_1,z_1),\mathrm{},(r_k,z_k))$$

where $`r_j`$ (resp. $`z_j`$) is by definition the vanishing order of $`\mathrm{\Delta }`$ (resp. $`\beta `$) at $`x_j`$. Note that by (4.9) we always have $`2z_jr_j`$. We also have

(5.12)
$$\underset{j}{}r_j=\mathrm{deg}(K^2(\mathrm{\Lambda }^2S)^2)=2(2g2+d).$$

We define the generic type to be following list of $`2(2g2+d)`$ pairs

$$T^{\mathrm{gen}}=((1,0),\mathrm{},(1,0)).$$

###### Theorem 5.8.

For any type $`T=((r_1,z_1),\mathrm{},(r_k,z_k))`$ there is a manifold $`S_T`$ with a covering $`U_i=S_T`$ and a family of pairs $`(V_i,B_i)`$ on $`X`$ parametrized by $`U_i`$ for any $`i`$ so that

(i) if $`sU_iU_j`$ then the pairs $`(V_i|_{\{s\}\times X},B_i|_{\{s\}\times X})`$ and $`(V_j|_{\{s\}\times X},B_j|_{\{s\}\times X})`$ are isomorphic;

(ii) if $`T=T^{\mathrm{gen}}`$ then $`S_T`$ is irreducible;

(iii) any isomorphism class of pairs of type $`T`$ which is $`\alpha `$-stable for some value of $`\alpha `$ is represented by at least one $`(V,\beta )=(V_i|_{\{s\}\times X},B_i|_{\{s\}\times X})`$;

(iv) if at least one of the $`r_j`$ is odd, the dimension of $`S_T`$ is $`k+g1+|x_R|/2`$, where $`|x_R|`$ denotes the cardinal of $`\{jr_j\text{ is odd }\}`$; this number is $`7(g1)+3d`$, with equality if and only if $`T=T^{\mathrm{gen}}`$;

(v) if all $`r_j`$ are even then the dimension of $`S_T`$ is strictly less than $`7(g1)+3d`$.

###### Proof.

Let $`k`$ be a positive integer and define $`B_T=X^k\mathrm{\Delta }_{\mathrm{mult}}`$, where $`\mathrm{\Delta }_{\mathrm{mult}}`$ denotes the multidiagonal. Let us fix a type $`T=((r_1,z_1),\mathrm{},(r_k,z_k))`$.

Case 1. Suppose first that at least one $`r_j`$ is odd.

Fix some $`x=(x_1,\mathrm{},x_k)B_T`$, let $`x_R=\{x_jxr_j\text{ is odd }\}`$, and let $`p:X^{}X`$ be the $`2:1`$ covering ramified at $`x_R`$. Take a pair $`(V,\beta )`$ with $`\mathrm{\Delta }_\beta ^1(0)=x`$ and with type $`T`$. Since $`x_R\mathrm{}`$ there is no square root of $`\mathrm{\Delta }_\beta `$. Let $`(V^{},\beta ^{})=p^{}(V,\beta )`$, and let $`\mathrm{\Delta }^{}=\mathrm{\Delta }_\beta ^{}`$. By construction there is a square root of $`\mathrm{\Delta }^{}`$, so there are line bundles $`L_1,L_2V^{}`$ satisfying $`\beta ^{}H^0(p^{}KL_1L_2)`$.

Let $`\sigma :X^{}X^{}`$ the Galois transformation. There is a canonical lift $`\sigma :V^{}V^{}`$ which leaves $`\beta ^{}`$ invariant and swaps $`L_1`$ and $`L_2`$ (this follows from (i) in Lemma 4.8 and the fact that if $`\sigma ^{}L_1=L_1`$ then $`L_1`$ would descend to a line bundle $`L_0X`$ satisfying $`\beta H^0(KL_0V)`$, hence $`\mathrm{\Delta }_\beta `$ should have a square root). Denoting $`L=L_1`$ it then follows that $`\beta ^{}H^0(p^{}KL\sigma ^{}L)`$.

By Lemma 4.8 we have an exact sequence

(5.13)
$$0L\sigma ^{}LV^{}𝒪_{T_x}0,$$

where $`T_x`$ is the following divisor on $`X^{}`$:

$$T=\underset{p(y)=x_jxx_R}{}(r_j/2z_j)y+\underset{p(y)=x_jx_R}{}(r_j2z_j)y.$$

In order for the sequence to be $`\sigma `$-equivariant one has to take the natural action of $`\sigma `$ on $`𝒪(T_x)`$ obtained by identifying the sections of $`𝒪(T_x)`$ with meromorphic functions on $`X^{}`$. So we have

$$2\mathrm{deg}L=\mathrm{deg}V^{}\underset{j}{}r_j+2\underset{j}{}z_j=2d\underset{j}{}r_j+2\underset{j}{}z_j.$$

Consider the divisor

$$D_x=\underset{p(y)=x_jxx_R}{}z_jy+\underset{p(y)=x_jx_R}{}2z_jy$$

in $`X^{}`$. Since the numbers $`z_j`$ describe the vanishing order of $`\beta `$, it follows that $`p^{}KL\sigma ^{}L𝒪(D_x)`$.

So begining from the pair $`(V,\beta )`$ we have constructed a line bundle $`L`$ on $`X^{}`$ of degree $`d_jr_j/2+_jz_j`$ satisfying $`p^{}KL\sigma ^{}L𝒪(D_x)`$.

Conversely, we can recover the isomorphism class of $`(V,\beta )`$ as follows. Let us denote

$$\mathrm{Ext}^1(𝒪_{T_x},L\sigma ^{}L)_{\mathrm{free}}^\sigma \mathrm{Ext}^1(𝒪_{T_x},L\sigma ^{}L)^\sigma $$

the set of elements giving locally free extensions. The $`\sigma `$-equivariant extensions of the type (5.13) are classified by the elements of $`\mathrm{Ext}^1(𝒪_{T_x},L\sigma ^{}L)^\sigma `$, and the isomorphism classes of $`\sigma `$-equivariant vector bundles $`V^{}`$ which are obtained through extensions of the form (5.13) are classified by

$$\frac{\mathrm{Ext}^1(𝒪_{T_x},L\sigma ^{}L)_{\mathrm{free}}^\sigma }{\mathrm{Aut}(L\sigma ^{}L)^\sigma \times \mathrm{Aut}(𝒪_{T_x})}=\frac{\mathrm{Ext}^1(𝒪_{T_x},L)_{\mathrm{free}}}{\mathrm{Aut}(L)\times \mathrm{Aut}(𝒪_{T_x})},$$

and the latter quotient has a unique element, represented by the class of any extension

(5.14)
$$0LL𝒪(T_x)𝒪_{T_x}L𝒪_{T_x}0.$$

On the other hand, the section $`\beta ^{}`$ (and hence $`\beta `$) is uniquely determined (up to multiplication by scalars) by the fact that it induces an isomorphism $`p^{}KL\sigma ^{}L𝒪(D_x)`$.

Let $`d_T=d_jr_j/2+_jz_j`$. We are now going to study the set

$$𝒮_x:=\{L\mathrm{Jac}_{d_T}(X^{})p^{}KL\sigma L𝒪(D_x)\}.$$

The following lemma implies that $`𝒮_x`$ is a torus of dimension $`3(g1)+d`$.

###### Lemma 5.9.

Let $`\tau :\mathrm{Jac}_{d_T}(X^{})\mathrm{Jac}_{2d_T}(X^{})`$ be the map which sends $`L`$ to $`L\sigma L`$. The image of $`\tau `$ coincides with $`\mathrm{Jac}_{2d_T}(X^{})^\sigma `$, and the fibres of $`\tau `$ can be identified to the torus

$$𝒮_0=\{\mathrm{\Lambda }\mathrm{Jac}_0(X^{})\mathrm{\Lambda }\sigma \mathrm{\Lambda }𝒪\},$$

which has dimension $`g1+|x_R|/2`$.

###### Proof.

It is clear that the fibres of $`\tau `$ are the orbits of the action of $`𝒮_0`$ on $`\mathrm{Jac}_{d_T}(X^{})`$ given by restricting the canonical action of $`\mathrm{Jac}_0(X^{})`$.

For any $`L\mathrm{Jac}_{2d_T}(X^{})^\sigma `$ there exists a square root $`L^{1/2}\mathrm{Jac}_{d_T}(X^{})^\sigma `$, which consequently satisfies $`\tau (L^{1/2})=L`$. It follows that $`\mathrm{Jac}_{2d_T}(X^{})^\sigma \mathrm{Im}\tau `$. The other inclusion is obvious, so we get $`\mathrm{Jac}_{2d_T}(X^{})^\sigma =\mathrm{Im}\tau `$.

Let us now compute the dimension of $`𝒮_0`$. Recall that we have an identification $`\mathrm{Jac}_0(X^{})H^1(X^{};)/H^1(X^{};)`$, which is $`\sigma `$-equivariant, since it comes from identifying $`\mathrm{Jac}_0(X^{})`$ with the set of gauge equivalence classes of flat $`\mathrm{U}(1)`$-connections on the trivial bundle on $`X^{}`$. Let $`H^1(X^{};)^\pm `$ be the eigenspace of $`\sigma `$ corresponding to the eigenvalue $`\pm 1`$. We then have $`T_\underset{¯}{}𝒮_0H^1(X^{};)^{}`$ and $`H^1(X;)H^1(X^{};)^+`$. So (using complex dimension everywhere)

$`dim𝒮_0`$ $`=dimH^1(X^{};)^{}=dimH^1(X^{};)dimH^1(X;)`$
$`=g(X^{})g=g1+|x_R|/2.`$
Here we have used Hurwitz’ formula $`\chi (X^{})=2\chi (X)|x_R|`$ to deduce $`g(X^{})=1\chi (X^{})/2=1\chi (X)+|x_R|/2=1+2g+|x_R|/2`$

Define $`\pi :S_TB_T`$ to be the fibration whose fibre over $`xB_T`$ is $`𝒮_x`$. As $`x`$ moves along $`B_T`$ the divisors $`x_R`$ sweep a divisor in $`B_T\times X`$ which we denote by $`𝐱_R`$. Let $`p:X_T^{}B_T\times X`$ be the $`2:1`$ covering ramified along $`𝐱_R`$, and let $`\sigma :X_T^{}X_T^{}`$ be the Galois transformation. Let $`D_T`$ (resp. $`T_T`$) be the divisor in $`X_T^{}`$ swept by the divisors $`D_x`$ (resp. $`T_x`$) as $`x`$ moves along $`B_T`$.

There is a universal bundle $`LS_T\times _{B_T}X_T^{}`$ for which there is an isomorphism of bundles over $`S_T\times _{B_T}X_T^{}`$

(5.15)
$$𝒪(D_T)\stackrel{}{}KL\sigma ^{}L$$

(we omit the pullbacks). Now take a covering of $`𝒮_T`$ by open sets $`U_i`$ admiting trivialisations

$$L𝒪_{T_T}|_{U_i\times X}𝒪_{T_T}|_{U_i\times X}.$$

Using each of these trivialisation we can obtain families of pairs on $`U_i\times X`$ beginning from the extensions (5.14). By the preceeding arguments it follows that any isomorphism class of pairs $`(V,\beta )`$ is represented in the family $`(V,B)`$. It is also clear that the dimension of $`S_T`$ is $`k+g1+|x_R|/2`$. Since $`B_T`$ and the fibres of $`S_TB_T`$ are connected, we deduce that $`S_T`$ is connected.

Finally, observe that $`T^{\mathrm{gen}}`$ falls in the case we are now considering. In this situation, $`k=|x_R|=2(2g2+d)`$, so that

$$dim𝒮_{T^{\mathrm{gen}}}=7(g1)+3d.$$

When $`TT^{\mathrm{gen}}`$ then both $`k`$ and $`|x_R|`$ are $`2(2g2+d)`$, and one does not have equality in both cases (i.e., $`k=|x_R|=2(2g2+d)`$ only holds for $`T=T^{\mathrm{gen}}`$).

Case 2. Suppose that all the $`r_j`$ are even. Let $`(V,\beta )`$ be a pair of type $`T`$ and vanishing locus $`xB_T`$. There is a square root $`\mathrm{\Delta }^{1/2}`$ of $`\mathrm{\Delta }_\beta `$, so we have two line bundles $`L_1,L_2`$ on $`X`$, an exact sequence

$$0L_1L_2V𝒪_{T_x}0,$$

with $`T_x=_j(r_j/2z_j)x_j`$ (see Lemma 4.8), and an isomorphism

(5.16)
$$𝒪(D_x)KL_1L_2,$$

where $`D_x`$ is the divisor $`_jz_jx_j`$ in $`X`$. It follows that the isomorphism classes of pairs $`(V,\beta )`$ with type $`T`$ and vanishing locus $`x`$ are in 1—1 correspondence with line bundles $`L=L_1\mathrm{Jac}(X)`$ (indeed, once $`L_1`$ has been chosen, we set $`L_2=𝒪(D_x)K^1L_1^1`$) and a choice of a class in

(5.17)
$$\frac{\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)_{\mathrm{free}}}{\mathrm{Aut}(L_1L_2)^\sigma \times \mathrm{Aut}(𝒪_{T_x})}.$$

On the other hand, $`\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)_{\mathrm{free}}`$ is the following open subset of $`\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)`$

$$\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)_{\mathrm{free}}=\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)\underset{f:𝒪_{T_x}𝒪_T}{}f^{}\mathrm{Ext}^1(𝒪_T,L_1L_2),$$

where the union is over all torsion sheaves $`𝒪_T`$ admitting a surjection $`f:𝒪_{T_x}𝒪_T`$ and $`f^{}:\mathrm{Ext}^1(𝒪_T,L_1L_2)\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)`$ is the map induced by $`f`$.

To compute $`\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)`$ we use Serre duality to obtain $`\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)=\mathrm{Hom}(L_1L_2,𝒪_{T_x}K),`$ and deduce from it that

$$dim\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)=2r_j2z_j.$$

On the other hand, since we have $`dim\mathrm{Aut}(𝒪_T)=r_j2z_j`$, we obtain:

(5.18)
$$dim\frac{\mathrm{Ext}^1(𝒪_{T_x},L_1L_2)_{\mathrm{free}}}{\mathrm{Aut}(L_1L_2)^\sigma \times \mathrm{Aut}(𝒪_{T_x})}=\left(r_j2z_j\right)2r_j2=2(2g2+d)2.$$

Now, if the pair $`(V,\beta )`$ is $`\alpha `$-polystable for some value of $`\alpha `$ then we necessarily have $`\mathrm{deg}L_1d/2`$ and $`\mathrm{deg}L_2d/2`$. (5.16) implies that

$`\mathrm{deg}L_1`$ $`=\mathrm{deg}(D_x)\mathrm{deg}K\mathrm{deg}L_2`$
$`={\displaystyle z_j}+22g\mathrm{deg}L_2,`$
so semistability implies that

$$\mathrm{deg}L_1[m_T,M_T]:=[z_j+22gd/2,d/2].$$

Consequently we can define

$$S_T:=\underset{m_TdM_T}{}E^d(X),$$

where $`E^d\mathrm{Jac}_d(X)\times B_T`$ is the bundle whose fibre over $`(L,x)`$ is

$$\frac{\mathrm{Ext}^1(𝒪_{T_x},L𝒪(D_x)K^1L^1)}{\mathrm{Aut}(L𝒪(D_x)K^1L^1)\times \mathrm{Aut}(𝒪_{T_x})}.$$

By (5.18) this has dimension $`dimE^d5(g1)+2d+k1.`$ Finally, since all zeros of $`\mathrm{\Delta }_\beta `$ have even order (hence $`2`$) we have $`k2(g1)+d=\mathrm{deg}(K^2(\mathrm{\Lambda }^2V)^2)/2`$ so $`dimE^d7(g1)+3d1.`$ The definitions of the local families $`V_i`$ and $`B_i`$ can be given exactly as in Case 1. ∎

### 5.4. Existence of stable objects

Here we prove that for any $`2(g1)<d0`$ and $`\alpha 0`$ there is a $`\alpha `$-stable pair $`(V,\beta )`$ with $`\mathrm{deg}V=0`$. More concretely, we show that there exists a $`\alpha `$-stable pair of degree $`d`$ and generic type, i.e., such that $`\mathrm{\Delta }_\beta `$ has simple zeroes (and consequently $`\beta `$ never vanishes). By the results in Subsection 5.3, the isomorphism classes of pairs of degree $`d`$ of generic type are in 1—1 correspondence with choices of:

* a $`(2:1)`$ covering $`p:X^{}X`$, ramified along $`2(2g2+d)`$ different points of $`X`$;
* a line bundle $`L`$ on $`X^{}`$ of degree $`2(g1)`$ such that

(5.19)
$$p^{}KL\sigma ^{}L\underset{¯}{},$$

where $`\sigma :X^{}X^{}`$ is the unique nontrivial automorphism of $`p`$.

The pair $`(V,\beta )`$ corresponding to one such choice is related to this data as follows: there is an exact sequence

(5.20)
$$0L\sigma ^{}Lp^{}V𝒪_R0,$$

where $`𝒪_R`$ is the structure sheaf of the ramification locus, and $`p^{}\beta H^0(p^{}KL\sigma ^{}L)`$ is a nonvanishing section giving rise to the isomorphism (5.19).

Define, for every $`k`$, $`\mathrm{Jac}_k^+:=\{L\mathrm{Jac}_k(X^{})\sigma ^{}LL\}`$ and $`\mathrm{Jac}_k^{}:=\{L\mathrm{Jac}_k(X^{})\sigma ^{}LL^1\}`$. It is easy to check that for any pair $`p,q`$ the map given by tensor product $`\mathrm{Jac}_p^+\times \mathrm{Jac}_q^{}\mathrm{Jac}_{p+q}(X^{})`$ is a covering map.

Pick some integer $`\delta 2(2g2+d)`$ and define

$$W_\delta :=\{L\mathrm{Jac}_{2(g1)\delta }(X^{})h^0(L)>0\}.$$

This is a complex submanifold of $`\mathrm{Jac}_{2(g1)\delta }(X^{})`$, and we have

(5.21)
$$dimW_\delta 2(g1)\delta .$$

Take a square root $`K^{1/2}\mathrm{Jac}(X)`$ of $`K`$. Let also $`Q`$ be the quotient $`\mathrm{Jac}_{2(g1)\delta }/\mathrm{Jac}_\delta ^+`$, and consider the diagram

where $`\alpha `$ maps $`L`$ to its class $`[L]`$ in $`Q`$ and $`f`$ maps $`L_0\mathrm{Jac}_0^{}`$ to $`[p^{}K^{1/2}L_0]`$. The map $`f`$ is a covering, hence $`dimQ=dim\mathrm{Jac}_0^{}`$, and this dimension is equal to

$$dimT_\underset{¯}{}\mathrm{Jac}_0^{}=dimH^1(X^{};)^{}=3(g1)+d.$$

(see the proof of Lemma 5.9). It is then straightforward to check that for any $`\delta 2(2g2+d)`$ we have $`dimW_\delta <dimQ`$. So, if we set

$$W:=\underset{\delta 2(2g2+d)}{}W_\delta ,$$

which is a finite union because of (5.21), then $`c(W)Q`$. Hence there is some $`L_0\mathrm{Jac}_0^{}`$ such that $`f(L_0)c(W)`$. Let $`L:=p^{}K^{1/2}L_0`$, and let $`(V,\beta )`$ be the pair constructed using the ideas in Subsection 5.3.

We now prove that $`(V,\beta )`$ is $`\alpha `$-stable. First of all, by construction $`\mathrm{\Delta }_\beta `$ does not admit a square root. Hence, $`(V,\beta )`$ could only be unstable if there were a line subbundle $`\mathrm{\Lambda }V`$ such that $`\mathrm{deg}\mathrm{\Lambda }c0`$. In this case, by (5.20), we would have a diagram

in which $`T`$ is a torsion sheaf which injects into $`𝒪_R`$ by $`i`$. Consequently, we have

$$\delta :=\mathrm{deg}M\mathrm{deg}p^{}\mathrm{\Lambda }|𝒪_R|=2c2(2g2+d)2(2g2+d).$$

On the other hand, $`M\mathrm{Jac}_\delta ^+`$ and the map $`(j,j^{})`$ is $`\sigma `$ invariant, hence $`j^{}=\sigma ^{}j`$. And, since $`(j,j^{})`$ is an inclusion, it follows that $`j:ML`$ is a nonzero holomorphic map. Hence $`L:=M^{}LW_\delta `$. But this implies that $`f(L_0)=[L]c(W_\delta )`$, contradicting our assumption. So there does not exist any subbundle $`\mathrm{\Lambda }V`$ with $`\mathrm{deg}\mathrm{\Lambda }0`$, thus $`(V,\beta )`$ is $`\alpha `$-stable.

Acknowledgements. We wish to thank Nigel Hitchin for very useful discussions. Much of the research in this paper was done while the authors were visiting the Mathematical Institute of Oxford (which we warmly thank for its hospitality) during Michaelmas term of 2001. We thank the EPSRC and EDGE for supporting the visists of the first and second authors respectively. The authors are members of VBAC (Vector Bundles on Algebraic Curves), which is partially supported by EAGER (EC FP5 Contract no. HPRN-CT-2000-00099) and by EDGE (EC FP5 Contract no. HPRN-CT-2000-00101).