As in his previous paper [ibid. 51, 333-399 (1984; see the preceeding review)], the author studies properties of the Weil lifting between the dual reductive pair (Sp, O(Q)), which in this case is given in concrete terms by a kernel function \(\theta_{\phi}(G,g)\) on \(Sp_ n\times O(Q)\) associated to the Weil representation and to a Schwartz function \(\phi\) on the space of \(m\times n\) matrices, m the dimension of the nondegenerate quadratic form.
Let \(L^ 2_{cusp}(O(Q))\) [resp. \(L^ 2_{cusp}(Sp_ n)]\) denote the adelic space of cusp forms on the orthogonal group O(Q) [resp. the symplectic group \(Sp_ n]\). Moreover, let R(Q) [resp. I(Q)] be the orthogonal part in \(L^ 2_{cusp}(O(Q))\) [resp. \(L^ 2_{cusp}(Sp_ n)]\) to the subspace of cusp forms whose lift in \(L^ 2(Sp_ 1)\) [resp. \(L^ 2(O(Q))]\) vanishes. The paper presents, in the case \(n=1\) and Q anisotropic of dimension \(>6\), a complete description of the representations occuring in I(Q) and shows that the lift of R(Q) in \(L^ 2_{cusp}(Sp_ 1)\) has as image the space I(-Q).
The method is to get a relation between the inner product of two lifted forms \(\ell_{\phi,\psi}\) on O(Q) and the initial inner product of the \(\psi\) on \(Sp_ n\), namely: \(<\ell_{\phi,\psi_ 1}| \ell_{\phi ',\psi_ 2}>=<\psi_ 1*{\mathcal L}(\phi,\phi ')| \psi_ 2^{\Lambda}>\), where \({\mathcal L}(\phi,\phi ')\) is an \(L^ 1\) function of \(Sp_ n\) and \(\Lambda\) a certain involution on \(L^ 2_{cusp}(Sp_ n)\). Then the term \(<\psi *{\mathcal L}(\phi,\phi ')| \phi^{\Lambda}>\) is shown to be the product \(L_ 1\cdot L_ 2\), where \(L_ 1\) is the special value of a Langlands L-function associated to the representation \(\pi\) which carries \(\psi\), and \(L_ 2\) is a finite product of local integrals over the primes v where the local component \(\pi_ v\) is not a spherical representation. In the case \(n=1\), the author establishes a formula which shows that \(L_ 1\neq 0\); the nonvanishing of \(L_ 2\) depends on whether the components \(\pi_ v\) occur in the associated local Weil representations. Then using results of a paper by the author and G. Schiffmann [Mem. Am. Math. Soc. 231 (1980; Zbl 0442.22006)], the description of I(Q) is obtained.
On the other hand, the use of the inner product formula gives a comparison of traces of Hecke operators acting, respectively, on the space of modular cusp forms of weight 4t and on the image in \(L^ 2_{cusp}(O(Q))\) of the lifting. It also provides a way to determine the algebraic nature of the ratio \(<\ell_{\phi,\psi_ 1}| \ell_{\phi ',\psi_ 2}>/<\psi_ 1| \psi_ 2^{\Lambda}>\) when \(\psi_ 1\) is an eigenfunction of \({\mathcal L}(\phi,\phi ').\)
Finally, as a third application, the author gives a construction of a global cuspidal automorphic representation of O(Q) such that a local component at infinity has nonvanishing Lie algebra cohomology of a certain described level.