On a two-component Bose–Einstein condensate with steep potential wells

1034 Accesses
Explore all metrics

Abstract

In this paper, we study the following two-component systems of nonlinear Schrödinger equations

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta u-(\lambda a(x)+a_0(x))u+\mu _1u^3+\beta v^2u=0&{}\quad \text {in }\mathbb {R}^3,\\ \Delta v-(\lambda b(x)+b_0(x))v+\mu _2v^3+\beta u^2v=0&{}\quad \text {in }\mathbb {R}^3,\\ u,v\in H^1(\mathbb {R}^3), u,v>0&{}\quad \text {in }\mathbb {R}^3, \end{array}\right. \end{aligned}$$

where $\lambda ,\mu _1,\mu _2>0$ and $\beta <0$ are parameters; $a(x), b(x)\ge 0$ are steep potentials and $a_0(x),b_0(x)$ are sign-changing weight functions; a(x), b(x), $a_0(x)$ and $b_0(x)$ are not necessarily to be radial symmetric. By the variational method, we obtain a ground state solution and multi-bump solutions for such systems with $\lambda $ sufficiently large. The concentration behaviors of solutions as both $\lambda \rightarrow +\infty $ and $\beta \rightarrow -\infty $ are also considered. In particular, the phenomenon of phase separations is observed in the whole space $\mathbb {R}^3$. In the Hartree–Fock theory, this provides a theoretical enlightenment of phase separation in $\mathbb {R}^3$ for the 2-mixtures of Bose–Einstein condensates.

Existence and local uniqueness of normalized solutions for two-component Bose–Einstein condensates

Article 04 October 2021

A perturbation method for k-mixtures of Bose–Einstein condensates

Article 19 October 2014

Expansion dynamics of a two-component quasi-one-dimensional Bose–Einstein condensate: Phase diagram, self-similar solutions, and dispersive shock waves

Article 01 April 2017

Use our pre-submission checklist

Avoid common mistakes on your manuscript.

1 Introduction

In this paper, we consider the following two-component systems of nonlinear Schrödinger equations

where $\lambda ,\mu _1,\mu _2>0$ and $\beta <0$ are parameters. The potentials $a(x), b(x), a_0(x)$ and $b_0(x)$ satisfy the following conditions:

$(D_1)$ :

$a(x), b(x)\in C(\mathbb {R}^3)$ and $a(x), b(x)\ge 0$ on $\mathbb {R}^3$.

$(D_2)$ :

There exist $a_\infty >0$ and $b_\infty >0$ such that $\mathcal {D}_a:=\{x\in \mathbb {R}^3\mid a(x)<a_\infty \}$ and $\mathcal {D}_b:=\{x\in \mathbb {R}^3\mid b(x)<b_\infty \}$ are nonempty and have finite measures.

$(D_3)$ :

$\Omega _a=\text {int} a^{-1}(0)$ and $\Omega _b=\text {int} b^{-1}(0)$ are nonempty bounded sets and have smooth boundaries. Moreover, $\overline{\Omega }_a=a^{-1}(0)$, $\overline{\Omega }_b=b^{-1}(0)$ and $\overline{\Omega }_a\cap \overline{\Omega }_b=\emptyset $.

$(D_4)$ :

$a_0(x),b_0(x)\in C(\mathbb {R}^3)$ and there exist $R,d_a,d_b>0$ such that

$$\begin{aligned} a_0^-(x)\le d_a(1+a(x))\quad \text {and}\quad b_0^-(x)\le d_b(1+b(x))\quad \text {for }|x|\ge R, \end{aligned}$$

where $a_0^-(x)=\max \{-a_0(x),0\}$ and $b_0^-(x)=\max \{-b_0(x),0\}$.

$(D_5)$ :

$\inf \sigma _{a}(-\Delta +a_0(x))>0$ and $\inf \sigma _{b}(-\Delta +b_0(x))>0$, where $\sigma _{a}(-\Delta +a_0(x))$ is the spectrum of $-\Delta +a_0(x)$ on $H_0^1(\Omega _{a})$ and $\sigma _{b}(-\Delta +b_0(x))$ is the spectrum of $-\Delta +b_0(x)$ on $H_0^1(\Omega _{b})$.

Remark 1.1

If $a_0(x),b_0(x)\in C(\mathbb {R}^3)$ are bounded, then the condition $(D_4)$ is trivial. However, under the assumptions of $(D_4){-}(D_5)$, $a_0(x)$ and $b_0(x)$ may be sign-changing and unbounded.

Two-component systems of nonlinear Schrödinger equations like $(\mathcal {P}_{\lambda ,\beta })$ appear in the Hartree–Fock theory for a double condensate, that is, a binary mixture of Bose–Einstein condensates in two different hyperfine states $|1\rangle $ and $|2\rangle $ (cf. [25]), where the solutions u and v are the corresponding condensate amplitudes, $\mu _j$ are the intraspecies and interspecies scattering lengths. The interaction is attractive if $\beta >0$ and repulsive if $\beta <0$. When the interaction is repulsive, it is expected that the phenomenon of phase separations will happen, that is, the two components of the system tend to separate in different regions as the interaction tends to infinity. This kind of systems also arises in nonlinear optics (cf. [2]). Due to the important application in physics, the following system

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta u- \lambda _1 u+\mu _1u^3+\beta v^2u=0&{}\quad \text {in } \Omega ,\\ \Delta v-\lambda _2v+\mu _2v^3+\beta u^2v=0&{}\quad \text {in }\Omega ,\\ u,v=0 \quad \text {on }\partial \Omega ,\end{array}\right. \end{aligned}$$

(1.1)

where $\Omega \subset \mathbb {R}^2 $ or $\mathbb {R}^3$, has attracted many attentions of mathematicians in the past decade. We refer the readers to [7,8,9, 15, 17, 19, 20, 29,30,31, 33, 34, 36, 37, 39, 40, 44]. In these literatures, various existence theories of the solutions were established for the Bose–Einstein condensates in $\mathbb {R}^2$ and $\mathbb {R}^3$. Recently, some mathematicians devoted their interest to the two coupled Schrödinger equations with critical Sobolev exponent in the high dimensions, and a number of the existence results of the solutions for such systems were also established. See for example [13,14,15,16, 18].

On the other hand, if the parameter $\lambda $ is sufficiently large, then $\lambda a(x)$ and $\lambda b(x)$ are called the steep potential wells under the conditions $(D_1){-}(D_3)$. The depth of the wells is controlled by the parameter $\lambda $. Such potentials were first introduced by Bartsch and Wang in [3] for the scalar Schrödinger equations. An interesting phenomenon for this kind of Schrödinger equations is that one can expect to find the solutions which are concentrated at the bottom of the wells as the depth goes to infinity. Due to this interesting property, such topic for the scalar Schrödinger equations was studied extensively in the past decade. We refer the readers to [4, 5, 10, 24, 35, 42, 43, 45, 51] and the references therein. In particular, in [24], by assuming that the bottom of the steep potential wells consists of finitely many disjoint bounded domains, the authors obtained multi-bump solutions for scalar Schrödinger equations with steep potential wells, which are concentrated at any given disjoint bounded domains of the bottom as the depth goes to infinity.

We wonder what happens to the two-component Bose–Einstein condensate $(\mathcal {P}_{\lambda ,\beta })$ with steep potential wells? In the current paper, we shall explore this problem to find whether the solutions of such systems are concentrated at the bottom of the wells as $\lambda \rightarrow +\infty $ and when the phenomenon of phase separations of such systems can be observed in the whole space $\mathbb {R}^3$.

We remark that the phenomenon of phase separations for (1.1) was observed in [13, 16, 20, 21, 38, 49, 50] for the ground state solution when $\Omega $ is a bounded domain. In particular, this phenomenon was also observed on the whole spaces $\mathbb {R}^2$ and $\mathbb {R}^3$ by [48], where the system is radial symmetric! However, when the system is not necessarily radial symmetric, the phenomenon of phase separations for Bose–Einstein condensates on the whole space $\mathbb {R}^3$ has not been obtained yet. For other kinds of elliptic systems with strong competition, the phenomenon of phase separations has also been well studied; we refer the readers to [11, 12, 22] and references therein.

We recall some definitions in order to state the main results in the current paper. We say that $(u_0, v_0)\in H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ is a non-trivial solution of $(\mathcal {P}_{\lambda ,\beta })$ if $(u_0, v_0)$ is a solution of $(\mathcal {P}_{\lambda ,\beta })$ with $u_0\not =0$ and $v_0\not =0$. We say $(u_0,v_0)\in E$ is a ground state solution of $(\mathcal {P}_{\lambda ,\beta })$ if $(u_0,v_0)$ is a non-trivial solution of $(\mathcal {P}_{\lambda ,\beta })$ and

$$\begin{aligned} J_{\lambda ,\beta }(u_0,v_0)=\inf \{J_{\lambda ,\beta }(u,v)\mid (u,v)\text { is a non-trivial solution of }(\mathcal {P}_{\lambda ,\beta })\}, \end{aligned}$$

where $J_{\lambda ,\beta }(u,v)$ is the corresponding functional of $(\mathcal {P}_{\lambda ,\beta })$ and given by

$$\begin{aligned} J_{\lambda ,\beta }(u,v)= & {} \frac{1}{2}\int _{\mathbb {R}^3}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x+\frac{1}{2}\int _{\mathbb {R}^3}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2\mathrm{d}x\nonumber \\&-\,\frac{\mu _1}{4}\int _{\mathbb {R}^3}u^4\mathrm{d}x-\frac{\mu _2}{4}\int _{\mathbb {R}^3}v^4\mathrm{d}x-\frac{\beta }{2}\int _{\mathbb {R}^3}u^2v^2\mathrm{d}x. \end{aligned}$$

(1.2)

Remark 1.2

In Sect. 2, we will give a variational setting of $(\mathcal {P}_{\lambda ,\beta })$ and show that the solutions of $(\mathcal {P}_{\lambda ,\beta })$ are equivalent to the positive critical points of $J_{\lambda ,\beta }(u,v)$ in a suitable Hilbert space E.

Let $I_{\Omega _a}(u)$ and $I_{\Omega _b}(v)$ be two functionals, respectively, defined on $H_0^1(\Omega _a)$ and $H_0^1(\Omega _b)$, which are given by

$$\begin{aligned} I_{\Omega _a}(u)=\frac{1}{2}\int _{\Omega _a}|\nabla u|^2+a_0(x)u^2\mathrm{d}x-\frac{\mu _1}{4}\int _{\Omega _a}u^4\mathrm{d}x \end{aligned}$$

and by

$$\begin{aligned} I_{\Omega _b}(v)=\frac{1}{2}\int _{\Omega _b}|\nabla v|^2+b_0(x)v^2\mathrm{d}x-\frac{\mu _2}{4}\int _{\Omega _b}v^4\mathrm{d}x. \end{aligned}$$

Then, by the condition $(D_5)$, it is well known that $I_{\Omega _a}(u)$ and $I_{\Omega _b}(v)$ have least energy nonzero critical points. We denote the least energy of nonzero critical points for $I_{\Omega _a}(u)$ and $I_{\Omega _b}(v)$ by $m_a$ and $m_b$, respectively. Now, our first result can be stated as follows.

Theorem 1.1

Assume $(D_1){-}(D_5)$. Then there exists $\Lambda _{*}>0$ independent of $\beta $ such that $(\mathcal {P}_{\lambda ,\beta })$ has a ground state solution $(u_{\lambda ,\beta }, v_{\lambda ,\beta })$ for all $\lambda \ge \Lambda _{*}$ and $\beta <0$, which has the following properties:

(1):: $\int _{\mathbb {R}^3\backslash \Omega _{a}}|\nabla u_{\lambda ,\beta }|^2+u_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0$ and $\int _{\mathbb {R}^3\backslash \Omega _{b}}|\nabla v_{\lambda ,\beta }|^2+v_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0$ as $\lambda \rightarrow +\infty $.
(2):: $\int _{\Omega _{a}}|\nabla u_{\lambda ,\beta }|^2+a_0(x)u_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 4m_{a}$ and $\int _{\Omega _{b}}|\nabla v_{\lambda ,\beta }|^2+b_0(x)v_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 4m_{b}$ as $\lambda \rightarrow +\infty $.

Furthermore, for each $\{\lambda _n\}\subset [\Lambda _*, +\infty )$ satisfies $\lambda _n\rightarrow +\infty $ as $n\rightarrow \infty $ and $\beta <0$, there exists $(u_{0,\beta },v_{0,\beta })\in (H^1(\mathbb {R}^3)\backslash \{0\})\times (H^1(\mathbb {R}^3)\backslash \{0\})$ such that

(3):: $(u_{0,\beta },v_{0,\beta })\in H_0^1(\Omega _{a})\times H_0^1(\Omega _{b})$ with $u_{0,\beta }\equiv 0$ outside $\Omega _{a}$ and $v_{0,\beta }\equiv 0$ outside $\Omega _{b}$.
(4):: $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta },v_{0,\beta })$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $ up to a subsequence.
(5):: $u_{0,\beta }$ is a least energy nonzero critical point of $I_{\Omega _{a}}(u)$ and $v_{0,\beta }$ is a least energy nonzero critical point of $I_{\Omega _{b}}(v)$.

Remark 1.3

Roughly speaking, under the conditions $(D_1){-}(D_5)$, Theorem 1.1 obtain a solution of the system $(\mathcal {P}_{\lambda ,\beta })$ in the following form $(u_0+w_{\lambda ,\beta }^1, v_0+w_{\lambda ,\beta }^2)$, where $u_0$ and $v_0$ are, respectively, the least energy nonzero critical points of $I_{\Omega _a}(u)$ and $I_{\Omega _b}(v)$ and $w_{\lambda ,\beta }^1$ and $w_{\lambda ,\beta }^2$ are two perturbations with $w_{\lambda ,\beta }^1\rightarrow 0$ and $w_{\lambda ,\beta }^2\rightarrow 0$ strongly in $H^1(\mathbb {R}^3)$ as $\lambda \rightarrow +\infty $. It is worth to point out that the assumption $\overline{\Omega }_a\cap \overline{\Omega }_b=\emptyset $ is not essential in proving Theorem 1.1. For instance, in the case $\overline{\Omega }_a=\overline{\Omega }_b=\Omega $, our method to prove Theorem 1.1 (with some necessary modifications) still works to find out a solution of the system $(\mathcal {P}_{\lambda ,\beta })$ in the form $(u_\beta +w_{\lambda ,\beta }^3, v_\beta +w_{\lambda ,\beta }^4)$, where $(u_\beta , v_\beta )$ is the ground state solution of the following system

$$\begin{aligned} \left\{ \begin{array}{ll} \Delta u-a_0(x)u+\mu _1u^3+\beta v^2u=0&{}\quad \text {in }\Omega ,\\ \Delta v-b_0(x)v+\mu _2v^3+\beta u^2v=0&{}\quad \text {in }\Omega ,\\ u,v>0&{}\quad \text {in }\Omega ,\\ u=v=0&{}\quad \text {on }\partial \Omega ,\end{array}\right. \end{aligned}$$

and $w_{\lambda ,\beta }^3$ and $w_{\lambda ,\beta }^4$ are also two perturbations with $w_{\lambda ,\beta }^3\rightarrow 0$ and $w_{\lambda ,\beta }^4\rightarrow 0$ strongly in $H^1(\mathbb {R}^3)$ as $\lambda \rightarrow +\infty $.

Next, we assume that the bottom of the steep potential wells consists of finitely many disjoint bounded domains. It is natural to ask whether the two-component Bose–Einstein condensate $(\mathcal {P}_{\lambda ,\beta })$ with such steep potential wells has multi-bump solutions which are concentrated at any given disjoint bounded domains of the bottom as the depth goes to infinity. Our second result is devoted to this study. Similar to [24], we need the following conditions on the potentials a(x), b(x), $a_0(x)$ and $b_0(x)$.

$(D_3')$ :: $\Omega _a=\text {int} a^{-1}(0)$ and $\Omega _b=\text {int} b^{-1}(0)$ satisfy $\Omega _a=\underset{i_a=1}{\overset{n_a}{\cup }}\Omega _{a,i_a}$ and $\Omega _b=\underset{j_b=1}{\overset{n_b}{\cup }}\Omega _{b,j_b}$, where $\{\Omega _{a,i_a}\}$ and $\{\Omega _{b,j_b}\}$ are all nonempty bounded domains with smooth boundaries, and $\overline{\Omega }_{a,i_a}\cap \overline{\Omega }_{a,j_a}=\emptyset $ for $i_a\not =j_a$ and $\overline{\Omega }_{b,i_b}\cap \overline{\Omega }_{b,j_b}=\emptyset $ for $i_b\not =j_b$. Moreover, $\overline{\Omega }_a=a^{-1}(0)$ and $\overline{\Omega }_b=b^{-1}(0)$ with $\overline{\Omega }_a\cap \overline{\Omega }_b=\emptyset $.
$(D_5')$ :: $\inf \sigma _{a,i_a}(-\Delta +a_0(x))>0$ for all $i_a=1,\ldots ,n_a$ and $\inf \sigma _{b,j_b}(-\Delta +b_0(x))>0$ for all $j_b=1,\ldots ,n_b$, where $\sigma _{a,i_a}(-\Delta +a_0(x))$ is the spectrum of $-\Delta +a_0(x)$ on $H_0^1(\Omega _{a,i_a})$ and $\sigma _{b,j_b}(-\Delta +b_0(x))$ is the spectrum of $-\Delta +b_0(x)$ on $H_0^1(\Omega _{b,j_b})$.

Remark 1.4

Under the conditions $(D_3')$ and $(D_4)$, it is easy to see that the condition $(D_5')$ is equivalent to the condition $(D_5)$. For the sake of clarity, we use the condition $(D_5')$ in the study of multi-bump solutions.

We define $I_{\Omega _{a,i_a}}(u)$ on $H_0^1(\Omega _{a,i_a})$ for each $i_a=1,\ldots ,n_a$ by

$$\begin{aligned} I_{\Omega _{a,i_a}}(u)=\frac{1}{2}\int _{\Omega _{a,i_a}}|\nabla u|^2+a_0(x)u^2\mathrm{d}x-\frac{\mu _1}{4}\int _{\Omega _{a,i_a}}u^4\mathrm{d}x \end{aligned}$$

and $I_{\Omega _{b,j_b}}(v)$ on $H_0^1(\Omega _{b,j_b})$ for each $j_b=1,\ldots ,n_b$ by

$$\begin{aligned} I_{\Omega _{b,j_b}}(v)=\frac{1}{2}\int _{\Omega _{b,j_b}}|\nabla v|^2+b_0(x)v^2\mathrm{d}x-\frac{\mu _2}{4}\int _{\Omega _{b,j_b}}v^4\mathrm{d}x. \end{aligned}$$

Then by the conditions $(D_3')$ and $(D_5')$, it is well known that $I_{\Omega _{a,i_a}}(u)$ and $I_{\Omega _{b,j_b}}(v)$ have least energy nonzero critical points for every $i_a=1,\ldots ,n_a$ and every $j_b=1,\ldots ,n_b$, respectively. We denote the least energy of nonzero critical points for $I_{\Omega _{a,i_a}}(u)$ and $I_{\Omega _{b,j_b}}(v)$ by $m_{a,i_a}$ and $m_{b,j_b}$, respectively. Now, our second result can be stated as the following.

Theorem 1.2

Assume $\beta <0$ and the conditions $(D_1){-}(D_2)$, $(D_3')$, $(D_4)$ and $(D_5')$ hold. If the set $J_a\times J_b\subset \{1,\ldots ,n_a\}\times \{1,\ldots ,n_b\}$ satisfying $J_a\not =\emptyset $ and $J_b\not =\emptyset $, then there exists $\Lambda _{*}(\beta )>0$ such that $(\mathcal {P}_{\lambda ,\beta })$ has a non-trivial solution $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})$ for $\lambda \ge \Lambda _{*}(\beta )$ with the following properties:

(1):: $\int _{\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}}|\nabla u_{\lambda ,\beta }^{J_a}|^2+(u_{\lambda ,\beta }^{J_a})^2\mathrm{d}x\rightarrow 0$ and $\int _{\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}}|\nabla v_{\lambda ,\beta }^{J_b}|^2+(v_{\lambda ,\beta }^{J_b})^2\mathrm{d}x\rightarrow 0$ as $\lambda \rightarrow +\infty $, where $\Omega _{a,0}^{J_a}=\underset{i_a\in J_a}{{\cup }}\Omega _{a,i_a}$ and $\Omega _{b,0}^{J_b}=\underset{j_b\in J_b}{{\cup }}\Omega _{b,j_b}$.
(2):: $\int _{\Omega _{a,i_a}}|\nabla u_{\lambda ,\beta }^{J_a}|^2+a_0(x)(u_{\lambda ,\beta }^{J_a})^2\mathrm{d}x\rightarrow 4m_{a,i_a}$ and $\int _{\Omega _{b,j_b}}|\nabla v_{\lambda ,\beta }^{J_b}|^2+b_0(x)(v_{\lambda ,\beta }^{J_b})^2\mathrm{d}x\rightarrow 4m_{b,j_b}$ as $\lambda \rightarrow +\infty $ for all $i_a\in J_a$ and $j_b\in J_b$.

Furthermore, for each $\beta <0$ and $\{\lambda _n\}\subset [\Lambda _*(\beta ), +\infty )$ satisfying $\lambda _n\rightarrow +\infty $ as $n\rightarrow \infty $, there exists $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in (H^1(\mathbb {R}^3)\backslash \{0\})\times (H^1(\mathbb {R}^3)\backslash \{0\})$ such that

(3):: $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in H_0^1(\Omega _{a,0}^{J_a})\times H_0^1(\Omega _{b,0}^{J_b})$ with $u_{0,\beta }^{J_a}\equiv 0$ outside $\Omega _{a,0}^{J_a}$ and $v_{0,\beta }^{J_b}\equiv 0$ outside $\Omega _{b,0}^{J_b}$.
(4):: $(u_{\lambda _n,\beta }^{J_a},v_{\lambda _n,\beta }^{J_b})\rightarrow (u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $ up to a subsequence.
(5):: the restriction of $u_{0,\beta }^{J_a}$ on $\Omega _{a,i_a}$ lies in $H_0^1(\Omega _{a,i_a})$ and is a least energy nonzero critical point of $I_{\Omega _{a,i_a}}(u)$ for all $i_a\in J_a$, while the restriction of $v_{0,\beta }^{J_b}$ on $\Omega _{b,j_b}$ lies in $H_0^1(\Omega _{b,j_b})$ and is a least energy nonzero critical point of $I_{\Omega _{b,j_b}}(v)$ for all $j_b\in J_b$.

Corollary 1.1

Suppose $\beta <0$ and the conditions $(D_1){-}(D_2)$, $(D_3')$, $(D_4)$ and $(D_5')$ hold. Then, $(\mathcal {P}_{\lambda ,\beta })$ has at least $(2^{n_a}-1)(2^{n_b}-1)$ non-trivial solutions for $\lambda \ge \Lambda _{*}(\beta )$.

Remark 1.5

(i) To the best of our knowledge, it seems that Theorem 1.2 is the first result for the existence of multi-bump solutions to system $(\mathcal {P}_{\lambda ,\beta })$.

(ii):

Under the condition $(D_3')$, we can see that

$$\begin{aligned} I_{\Omega _a}(u)=\sum _{i_a=1}^{n_a}I_{\Omega _{a,i_a}}(u)\quad \text {and}\quad I_{\Omega _b}(v)=\sum _{j_b=1}^{n_b}I_{\Omega _{b,j_b}}(v). \end{aligned}$$

Let $m_{a,0}=\min \{m_{a,1},\ldots ,m_{a,n_a}\}$ and $m_{b,0}=\min \{m_{b,1},\ldots ,m_{b,n_b}\}$. Then, we must have $m_{a,0}=m_a$ and $m_{b,0}=m_b$. Without loss of generality, we assume $m_{a,0}=m_{a,1}$ and $m_{b,0}=m_{b,1}$. Now, by Theorem 1.2, we can find a solution of $(\mathcal {P}_{\lambda ,\beta })$ with the same concentration behavior as the ground state solution obtained in Theorem 1.1 as $\lambda \rightarrow +\infty $. However, we do not know these two solutions are the same or not.

Next we consider the phenomenon of phase separations for System $(\mathcal {P}_{\lambda ,\beta })$, i.e., the concentration behavior of the solutions as $\beta \rightarrow -\infty $. In the following theorem, we may observe such a phenomenon on the whole space $\mathbb {R}^3$.

Theorem 1.3

Assume $(D_1){-}(D_5)$. Then, there exists $\Lambda _{**}\ge \Lambda _*$ such that $\beta ^2\int _{\mathbb {R}^3}u_{\lambda ,\beta }^2v_{\lambda ,\beta }^2\rightarrow 0$ as $\beta \rightarrow -\infty $ for $\lambda \ge \Lambda _{**}$, where $(u_{\lambda ,\beta },v_{\lambda ,\beta })$ is the ground state solution of $(\mathcal {P}_{\lambda ,\beta })$ obtained by Theorem 1.1. Furthermore, for every $\{\beta _n\}\subset (-\infty , 0)$ with $\beta _n\rightarrow -\infty $ and $\lambda \ge \Lambda _{**}$, there exists $(u_{\lambda ,0}, v_{\lambda ,0})\in (H^1(\mathbb {R}^3)\backslash \{0\})\times (H^1(\mathbb {R}^3)\backslash \{0\})$ satisfying the following properties:

(1):

$(u_{\lambda ,\beta _n},v_{\lambda ,\beta _n})\rightarrow (u_{\lambda ,0}, v_{\lambda ,0})$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $ up to a subsequence.

(2):

$u_{\lambda ,0}\in C(\mathbb {R}^3)$ and $v_{\lambda ,0}\in C(\mathbb {R}^3)$.

(3):

$u_{\lambda ,0}\ge 0$ and $v_{\lambda ,0}\ge 0$ in $\mathbb {R}^3$ with $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}=\mathbb {R}^3\backslash \overline{\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}}$. Furthermore, $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}$ and $\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}$ are connected domains.

(4):

$u_{\lambda ,0}\in H_0^1(\{u_{\lambda ,0}>0\})$ and is a least energy solution of

$$\begin{aligned} -\Delta u+(\lambda a(x)+a_0(x))u=\mu _1 u^3,\quad u\in H_0^1(\{u_{\lambda ,0}>0\}), \end{aligned}$$

(1.3)

while $v_{\lambda ,0}\in H_0^1(\{v_{\lambda ,0}>0\})$ and is a least energy solution of

$$\begin{aligned} -\Delta v+(\lambda a(x)+a_0(x))v=\mu _1 v^3,\quad v\in H_0^1(\{v_{\lambda ,0}>0\}). \end{aligned}$$

(1.4)

Remark 1.6

(1) Theorem 1.3 is based on Theorem 1.1, Thus, due to Remark 1.3, the assumption $\overline{\Omega }_a\cap \overline{\Omega }_b=\emptyset $ is also not necessary in Theorem 1.3 and it still holds for $\overline{\Omega }_a=\overline{\Omega }_b=\Omega $.

(2)
It is natural to ask that whether the multi-bump solutions founded in Theorem 1.2 have the same phenomenon of phase separations as that of the ground state solution described in Theorem 1.3. However, by checking the proof of Theorem 1.2, we can see that $\Lambda _*(\beta )\rightarrow +\infty $ as $\beta \rightarrow -\infty $. Thus, our method to prove Theorem 1.2 is invalid to assert that the system $(\mathcal {P}_{\lambda ,\beta })$ has multi-bump solutions for $\lambda $ large but $\beta $ fixed and $\beta $ diverging. Due to this reason, we can not obtain the phenomenon of phase separations to the multi-bump solutions founded in Theorem 1.2.

Before closing this section, we would like to cite other references studying the equations with steep potential wells. For example, in [46], the Kirchhoff-type elliptic equation with a steep potential well was studied. The Schrödinger–Poisson systems with a steep potential well were considered in [32, 52]. Non-trivial solutions were obtained in [26,27,28] for quasilinear Schrödinger equations with steep potential wells, while the multi-bump solutions were also obtained in [28] for such equations.

In this paper, we will always denote the usual norms in $H^1(\mathbb {R}^3)$ and $L^p(\mathbb {R}^3)$ ($p\ge 1$) by $\Vert \cdot \Vert $ and $\Vert \cdot \Vert _p$, respectively; C and $C'$ will be indiscriminately used to denote various positive constants; $o_n(1)$ will always denote the quantities tending toward zero as $n\rightarrow \infty $.

2 The variational setting

In this section, we mainly give a variational setting for $(\mathcal {P_{\lambda ,\beta }})$. Simultaneously, an important estimate is also established in this section, which is used frequently in this paper.

Let

$$\begin{aligned} E_{a}=\{u\in D^{1,2}(\mathbb {R}^3)\mid \int _{\mathbb {R}^3}(a(x)+a_0^+(x))u^2\mathrm{d}x<+\infty \} \end{aligned}$$

and

$$\begin{aligned} E_{b}=\{u\in D^{1,2}(\mathbb {R}^3)\mid \int _{\mathbb {R}^3}(b(x)+b_0^+(x))u^2\mathrm{d}x<+\infty \}, \end{aligned}$$

where $a_0^+(x)=\max \{a_0(x), 0\}$ and $b_0^+(x)=\max \{b_0(x), 0\}$. Then, by the conditions $(D_1)$ and $(D_4)$, $E_a$ and $E_b$ are Hilbert spaces equipped with the inner products

$$\begin{aligned}&\langle u,v\rangle _{a}=\int _{\mathbb {R}^3}\nabla u\nabla v+(a(x)+a_0^+(x))uv\mathrm{d}x\quad \text {and}\\&\quad \langle u,v\rangle _{b}=\int _{\mathbb {R}^3}\nabla u\nabla v+(b(x)+b_0^+(x))uv\mathrm{d}x, \end{aligned}$$

respectively. The corresponding norms of $E_a$ and $E_b$ are, respectively, given by

$$\begin{aligned} \Vert u\Vert _{a}=\bigg (\int _{\mathbb {R}^3}|\nabla u|^2+(a(x)+a_0^+(x))u^2\mathrm{d}x\bigg )^{\frac{1}{2}} \end{aligned}$$

and by

$$\begin{aligned} \Vert v\Vert _{b}=\bigg (\int _{\mathbb {R}^3}|\nabla v|^2+(b(x)+b_0^+(x))v^2\mathrm{d}x\bigg )^{\frac{1}{2}}. \end{aligned}$$

Since the conditions $(D_1){-}(D_2)$ hold, by a similar argument as that in [46], we can see that

$$\begin{aligned} \Vert u\Vert \le \bigg (\max \{1+|\mathcal {D}_a|^{\frac{2}{3}}S^{-1},\frac{1}{a_\infty }\}\bigg )^{\frac{1}{2}}\Vert u\Vert _a\quad \text {for all }u\in E_{a} \end{aligned}$$

(2.1)

and

$$\begin{aligned} \Vert v\Vert \le \bigg (\max \{1+|\mathcal {D}_b|^{\frac{2}{3}}S^{-1},\frac{1}{b_\infty }\}\bigg )^{\frac{1}{2}}\Vert v\Vert _b\quad \text {for all }v\in E_{b}, \end{aligned}$$

(2.2)

where S is the best Sobolev embedding constant from $D^{1,2}(\mathbb {R}^3)$ to $L^6(\mathbb {R}^3)$ and given by

$$\begin{aligned} S=\inf \{\Vert \nabla u\Vert _2^2 \mid u\in D^{1,2}(\mathbb {R}^3), \Vert u\Vert _6^2=1\}. \end{aligned}$$

It follows that both $E_a$ and $E_b$ are embedded continuously into $H^1(\mathbb {R}^3)$. Moreover, by applying the Hölder and Sobolev inequalities, we also have

$$\begin{aligned} \Vert u\Vert _4\le \bigg (\max \{1+|\mathcal {D}_a|^{\frac{2}{3}}S^{-1},\frac{1}{a_\infty }\}\bigg )^{\frac{1}{2}}S^{-\frac{3}{8}}\Vert u\Vert _a\quad \text {for all }u\in E_{a} \end{aligned}$$

(2.3)

and

$$\begin{aligned} \Vert v\Vert _4\le \bigg (\max \{1+|\mathcal {D}_b|^{\frac{2}{3}}S^{-1},\frac{1}{b_\infty }\}\bigg )^{\frac{1}{2}}S^{-\frac{3}{8}}\Vert v\Vert _b\quad \text {for all }v\in E_{b}. \end{aligned}$$

(2.4)

On the other hand, by the conditions $(D_2)$ and $(D_3)$, there exist two bounded open sets $\Omega _a'$ and $\Omega _b'$ with smooth boundaries such that $\Omega _a\subset \Omega _a'\subset \mathcal {D}_a$, $\Omega _b\subset \Omega _b'\subset \mathcal {D}_b$, $\overline{\Omega _a'}\cap \overline{\Omega _b'}=\emptyset $, dist$(\Omega _a, \mathbb {R}^3\backslash \Omega _a')>0$ and that dist$(\Omega _b, \mathbb {R}^3\backslash \Omega _b')>0$. Furthermore, by the condition $(D_4)$, the Hölder and the Sobolev inequalities, there exists $\Lambda _0>2\max \{1,d_a+\frac{d_a+C_{a,0}}{a_\infty }, d_b+\frac{d_b+C_{b,0}}{b_\infty }\}$ such that

$$\begin{aligned} \int _{\mathbb {R}^3}a_0^-(x)u^2\mathrm{d}x\le \int _{B_R(0)}C_{a,0}u^2\mathrm{d}x+\int _{\mathbb {R}^3\backslash B_R(0)}d_a(1+a(x))u^2\mathrm{d}x \le \frac{\lambda }{2}\Vert u\Vert _a^2 \end{aligned}$$

(2.5)

and

$$\begin{aligned} \int _{\mathbb {R}^3}b_0^-(x)v^2\mathrm{d}x\le \int _{B_R(0)}C_{b,0}v^2\mathrm{d}x+\int _{\mathbb {R}^3\backslash B_R(0)}d_b(1+b(x))v^2\mathrm{d}x \le \frac{\lambda }{2}\Vert v\Vert _b^2 \end{aligned}$$

(2.6)

for $\lambda \ge \Lambda _0$, where $B_R(0)=\{x\in \mathbb {R}^3\mid |x|< R\}$, $C_{a,0}=\sup _{B_R(0)}a_0^-(x)$ and $C_{b,0}=\sup _{B_R(0)}b_0^-(x)$. Combining (2.3)–(2.6) and the Hölder inequality, we can see that $(\mathcal {P}_{\lambda ,\beta })$ has a variational structure in the Hilbert space $E=E_{a}\times E_{b}$ for $\lambda \ge \Lambda _0$, where E is endowed with the norm $\Vert (u,v)\Vert =(\Vert u\Vert _{a}^2+\Vert v\Vert _{b}^2)^\frac{1}{2}$. The corresponding functional of $(\mathcal {P}_{\lambda ,\beta })$ is given by (1.2). Furthermore, by applying (2.3)–(2.6) in a standard way, we can also see that $J_{\lambda ,\beta }(u,v)$ is $C^2$ in E and the solution of $(\mathcal {P}_{\lambda ,\beta })$ is equivalent to the positive critical point of $J_{\lambda ,\beta }(u,v)$ in E for $\lambda \ge \Lambda _0$. In the case of $(D_3')$, we can choose $\Omega _{a}'$ and $\Omega _{b}'$ as follows:

(I):: $\Omega _a'=\underset{i_a=1}{\overset{n_a}{\cup }}\Omega '_{a,i_a}\subset \mathcal {D}_a$, where $\Omega _{a,i_a}\subset \Omega '_{a,i_a}$ and dist$(\Omega _{a,i_a}, \mathbb {R}^3\backslash \Omega _{a,i_a}')>0$ for all $i_a=1,\ldots , n_a$ and $\overline{\Omega '_{a,i_a}}\cap \overline{\Omega '_{a,j_a}}=\emptyset $ for $i_a\not =j_a$.
(II):: $\Omega _b'=\underset{i_b=1}{\overset{n_b}{\cup }}\Omega '_{b,i_b}\subset \mathcal {D}_b$, where $\Omega _{b,i_b}\subset \Omega '_{b,i_b}$ and dist$(\Omega _{b,i_b}, \mathbb {R}^3\backslash \Omega _{b,i_b}')>0$ for all $j_b=1,\ldots , n_b$ and $\overline{\Omega '_{b,i_b}}\cap \overline{\Omega '_{b,j_b}}=\emptyset $ for $i_b\not =j_b$.
(III):: $\overline{\Omega _a'}\cap \overline{\Omega _b'}=\emptyset $.

Thus, (2.5)–(2.6) still hold for such $\Omega _{a}'$ and $\Omega _{b}'$ with $\lambda $ sufficiently large. Without loss of generality, we may assume that (2.5)–(2.6) still hold for such $\Omega _{a}'$ and $\Omega _{b}'$ with $\lambda \ge \Lambda _0$. It follows that the solution of $(\mathcal {P}_{\lambda ,\beta })$ is also equivalent to the positive critical point of the $C^2$ functional $J_{\lambda ,\beta }(u,v)$ in E for $\lambda \ge \Lambda _0$ under the conditions $(D_1){-}(D_2)$, $(D_3')$ and $(D_4)$.

The remaining of this section will be devoted to an important estimate, which is used frequently in this paper and essentially due to Ding and Tanaka [24].

Lemma 2.1

Assume $(D_1){-}(D_5)$. Then, there exist $\Lambda _1\ge \Lambda _0$ and $C_{a,b}>0$ such that

$$\begin{aligned} \inf _{u\in E_{a}\backslash \{0\}}\frac{\int _{\mathbb {R}^3}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x}{\int _{\mathbb {R}^3}u^2\mathrm{d}x}\ge C_{a,b} \end{aligned}$$

and

$$\begin{aligned} \inf _{v\in E_{b}\backslash \{0\}}\frac{\int _{\mathbb {R}^3}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2\mathrm{d}x}{\int _{\mathbb {R}^3}v^2\mathrm{d}x}\ge C_{a,b} \end{aligned}$$

for all $\lambda \ge \Lambda _1$.

Proof

Since the conditions $(D_1){-}(D_4)$ hold, by a similar argument as [24, Lemma 2.1], we have

$$\begin{aligned} \lim _{\lambda \rightarrow +\infty }\inf \sigma _{a,*}(-\Delta +\lambda a(x)+a_0(x))=\inf \sigma _{a}(-\Delta +a_0(x)), \end{aligned}$$

where $\sigma _{a,*}(-\Delta +\lambda a(x)+a_0(x))$ is the spectrum of $-\Delta +\lambda a(x)+a_0(x)$ on $H^1(\Omega _{a}')$. Denote $\inf \sigma _{a}(-\Delta +a_0(x))$ by $\nu _{a}$. Then, by the condition $(D_5)$, there exists $\Lambda _1'\ge \Lambda _0$ such that

$$\begin{aligned} \sigma _{a,*}(-\Delta +\lambda a(x)+a_0(x))\ge \frac{\nu _a}{2}\quad \text {for}\quad \lambda \ge \Lambda _1'. \end{aligned}$$

(2.7)

On the other hand, by the conditions $(D_2)$ and $(D_4)$, we have $a_0^-(x)\le C_{a,0}+d_a+d_aa_\infty $ for $x\in \mathcal {D}_a$. Let $\mathcal {D}_{a,\overline{R}}=\mathcal {D}_a\cap B^c_{\overline{R}}$, where $B^c_{\overline{R}}=\{x\in \mathbb {R}\mid |x|\ge \overline{R}\}$. Then, by the condition $(D_2)$ once more, $|\mathcal {D}_{a,\overline{R}}|\rightarrow 0$ as $\overline{R}\rightarrow +\infty $, which then implies that there exists $\overline{R}_0>0$ such that $|\mathcal {D}_{a,\overline{R}_0}|S^{-1}(C_{a,0}+d_a+d_aa_\infty +1)\le \frac{1}{2}$. Thanks to the conditions $(D_1){-}(D_4)$, there exists $\Lambda _1=\Lambda _1(\overline{R}_0)\ge \Lambda _1'$ such that

$$\begin{aligned} \lambda a(x)+a_0(x)+(C_{a,0}+d_a+d_aa_\infty +1)\chi _{\mathcal {D}_{a,\overline{R}_0}}\ge 1\quad \text {for all }x\in \mathbb {R}^3\backslash \Omega _a'\text { and }\lambda \ge \Lambda _1, \end{aligned}$$

where $\chi _{\mathcal {D}_{a,\overline{R}_0}}$ is the characteristic function of the set $\mathcal {D}_{a,\overline{R}_0}$. It follows from the Hölder and the Sobolev inequalities that

$$\begin{aligned} \int _{\mathbb {R}^3\backslash \Omega _a'}u^2\mathrm{d}x\le (1+2|\mathcal {D}_{a,\overline{R}_0}|S^{-1})\int _{\mathbb {R}^3\backslash \Omega _a'}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x \end{aligned}$$

(2.8)

for all $u\in E_a\backslash \{0\}$ and $\lambda \ge \Lambda _1$. Combining (2.7)–(2.8) and the choice of $\Omega _a'$, we have

$$\begin{aligned} \int _{\mathbb {R}^3}u^2\mathrm{d}x\le C_a\int _{\mathbb {R}^3}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x \quad \text {for all }u\in E_a\backslash \{0\}\text { and }\lambda \ge \Lambda _1, \end{aligned}$$

where $C_a=\max \{\frac{2}{\nu _a}, 1+2|\mathcal {D}_{a,\overline{R}_0}|S^{-1}\}$. By similar arguments as (2.7) and (2.8), we can also have

$$\begin{aligned} \int _{\mathbb {R}^3}v^2\mathrm{d}x\le C_b\int _{\mathbb {R}^3}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2\mathrm{d}x \end{aligned}$$

for all $v\in E_b\backslash \{0\}$ and $\lambda \ge \Lambda _1$, where $C_b=\max \{\frac{2}{\nu _b}, 1+2|\mathcal {D}_{b,\overline{R}_0}|S^{-1}\}$, $\nu _b=\inf \sigma _{b}(-\Delta +b_0(x))$ and $\mathcal {D}_{b,\overline{R}_0}=\mathcal {D}_b\cap B^c_{\overline{R}_0}$. We completes the proof by taking $C_{a,b}=(\min \{C_a,C_b\})^{-1}$. $\square $

Remark 2.1

Under the conditions $(D_1){-}(D_2)$, $(D_3')$, $(D_4)$ and $(D_5')$, we can see that

$$\begin{aligned} \nu _a=\min _{i_a=1,2,\ldots ,n_a}\bigg \{\inf \sigma _{a,i_a}(-\Delta +a_0(x))\bigg \}\quad \text {and}\quad \nu _b=\min _{j_b=1,2,\ldots ,n_b}\bigg \{\inf \sigma _{b,j_b}(-\Delta +b_0(x))\bigg \}. \end{aligned}$$

Now, by a similar argument as (2.7), we get that

$$\begin{aligned} \int _{\Omega _{a,i_a}'}u^2\mathrm{d}x\le \frac{2}{\nu _a}\int _{\Omega _{a,i_a}'}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2 \mathrm{d}x \end{aligned}$$

(2.9)

and

$$\begin{aligned} \int _{\Omega _{b,j_b}'}v^2\mathrm{d}x\le \frac{2}{\nu _b}\int _{\Omega _{b,j_b}'}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2 \mathrm{d}x \end{aligned}$$

(2.10)

for all $i_a=1,\ldots ,n_a$ and $j_b=1,\ldots ,n_b$ if $\lambda $ sufficiently large. Without loss of generality, we may assume (2.9) and (2.10) hold for $\lambda \ge \Lambda _1$. It follows that Lemma 2.1 still holds under the conditions $(D_1){-}(D_2)$, $(D_3')$, $(D_4)$ and $(D_5')$.

By Lemma 2.1, we observe that $\int _{\mathbb {R}^3}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x$ and $\int _{\mathbb {R}^3}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2\mathrm{d}x$ are norms of $E_a$ and $E_b$ for $\lambda \ge \Lambda _1$, respectively. Therefore, we set

$$\begin{aligned} \Vert u\Vert _{a,\lambda }^2=\int _{\mathbb {R}^3}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x;\quad \Vert v\Vert _{b,\lambda }^2=\int _{\mathbb {R}^3}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2\mathrm{d}x. \end{aligned}$$

3 A ground state solution

Our interest in this section is to find a ground state solution to $(\mathcal {P}_{\lambda ,\beta })$ under the conditions $(D_1){-}(D_5)$. For the sake of convenience, we always assume the conditions $(D_1){-}(D_5)$ hold in this section. Since $J_{\lambda ,\beta }(u,v)$, the corresponding energy functional of $(\mathcal {P}_{\lambda ,\beta })$, is $C^2$ in E, it is well known that all non-trivial solutions of $(\mathcal {P}_{\lambda ,\beta })$ lie in the Nehari manifold of $J_{\lambda ,\beta }(u,v)$, which is given by

$$\begin{aligned} \mathcal {N}_{\lambda ,\beta }= & {} \{(u,v)\in E\mid u\not =0,v\not =0, \langle D[J_{\lambda ,\beta }(u,v)],(u,0)\rangle _{E^*,E}\\= & {} \langle D[J_{\lambda ,\beta }(u,v)],(0,v)\rangle _{E^*,E}=0\}, \end{aligned}$$

where $D[J_{\lambda ,\beta }(u,v)]$ is the Frechét derivative of the functional $J_{\lambda ,\beta }$ in E at (u, v) and $E^*$ is the dual space of E. If we can find $(u_{\lambda ,\beta }, v_{\lambda ,\beta })\in E$ such that $J_{\lambda ,\beta }(u_{\lambda ,\beta }, v_{\lambda ,\beta })=m_{\lambda ,\beta }$ and $D[J_{\lambda ,\beta }(u_{\lambda ,\beta }, v_{\lambda ,\beta })]=0$ in $E^*$, then $(u_{\lambda ,\beta }, v_{\lambda ,\beta })$ must be a ground state solution of $(\mathcal {P}_{\lambda ,\beta })$, where $m_{\lambda ,\beta }=\inf _{\mathcal {N}_{\lambda ,\beta }}J_{\lambda ,\beta }(u,v)$. In what follows, we drive some properties of $\mathcal {N}_{\lambda ,\beta }$.

Let $(u,v)\in (E_a\backslash \{0\})\times (E_b\backslash \{0\})$ and define $T_{\lambda ,\beta ,u,v}:\mathbb {R}^+\times \mathbb {R}^+\rightarrow \mathbb {R}$ by $T_{\lambda ,\beta ,u,v}(t,s)=J_{\lambda ,\beta }(tu,sv)$. These functions are called the fibering maps of $J_{\lambda ,\beta }(u,v)$, which are closely linked to $\mathcal {N}_{\lambda ,\beta }$. Clearly, $\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial t}(t,s)=\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial s}(t,s)=0$ is equivalent to $(tu,sv)\in \mathcal {N}_{\lambda ,\beta }$. In particular, $\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial t}(1,1)=\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial s}(1,1)=0$ if and only if $(u,v)\in \mathcal {N}_{\lambda ,\beta }$. Let

$$\begin{aligned} \mathcal {A}_{\beta }=\{(u,v)\in E\mid \mu _1\mu _2\Vert u\Vert _4^4\Vert v\Vert _4^4-\beta ^2\Vert u^2v^2\Vert ^2_1>0\}. \end{aligned}$$

(3.1)

Then, $\mathcal {A}_{\beta }\not =\emptyset $ for every $\beta <0$. Now, our first observation on $\mathcal {N}_{\lambda ,\beta }$ can be stated as follows.

Lemma 3.1

Assume $\lambda \ge \Lambda _1$ and $\beta <0$. Then, we have the following.

(1):

If $(u,v)\in \mathcal {A}_{\beta }$, then there exists a unique $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))\in \mathbb {R}^+\times \mathbb {R}^+$ such that

$$\begin{aligned} (t_{\lambda ,\beta }(u,v)u,s_{\lambda ,\beta }(u,v)v)\in \mathcal {N}_{\lambda ,\beta }, \end{aligned}$$

where $t_{\lambda ,\beta }(u,v)$ and $s_{\lambda ,\beta }(u,v)$ are given by

$$\begin{aligned} t_{\lambda ,\beta }(u,v)=\bigg (\frac{\mu _2\Vert v\Vert _4^4\Vert u\Vert _{a,\lambda }^2-\beta \Vert u^2v^2\Vert _1\Vert v\Vert _{b,\lambda }^2}{\mu _1\mu _2\Vert u\Vert _4^4\Vert v\Vert _4^4-\beta ^2\Vert u^2v^2\Vert ^2_1}\bigg )^{\frac{1}{2}} \end{aligned}$$

(3.2)

and by

$$\begin{aligned} s_{\lambda ,\beta }(u,v)=\bigg (\frac{\mu _1\Vert u\Vert _4^4\Vert v\Vert _{b,\lambda }^2-\beta \Vert u^2v^2\Vert _1\Vert u\Vert _{a,\lambda }^2}{\mu _1\mu _2\Vert u\Vert _4^4\Vert v\Vert _4^4-\beta ^2\Vert u^2v^2\Vert ^2_1}\bigg )^{\frac{1}{2}}. \end{aligned}$$

(3.3)

Moreover, $T_{\lambda ,\beta ,u,v}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))=\max _{t\ge 0,s\ge 0}T_{\lambda ,\beta ,u,v}(t,s)$.

(2):

If $(u,v)\in E\backslash \mathcal {A}_{\beta }$, then $\mathcal {B}_{u,v}\cap \mathcal {N}_{\lambda ,\beta }=\emptyset $, where $\mathcal {B}_{u,v}=\{(tu,sv)\mid (t,s)\in \mathbb {R}^+\times \mathbb {R}^+\}$.

Proof

(1) The proof is similar to [20, Lemma 2.2], where $\mathcal {N}_{0,\beta }$ with $a_0(x)=a_0>0$ and $b_0(x)=b_0>0$ was studied. For the reader’s convenience, we give the details here. Let $T^1_{\lambda ,\beta ,u,v}$ and $T^2_{\lambda ,\beta ,u,v}$ be two functions on $\mathbb {R}^+\times \mathbb {R}^+$ defined by

$$\begin{aligned} T^1_{\lambda ,\beta ,u,v}(t,s)=\Vert u\Vert _{a,\lambda }^2-\mu _1\Vert u\Vert _4^4t^2 -\beta \Vert u^2v^2\Vert _1s^2 \end{aligned}$$

and by

$$\begin{aligned} T^2_{\lambda ,\beta ,u,v}(t,s)=\Vert v\Vert _{b,\lambda }^2-\mu _2\Vert v\Vert _4^4s^2 -\beta \Vert u^2v^2\Vert _1t^2. \end{aligned}$$

Then, it is easy to see that

$$\begin{aligned} \frac{\partial T_{\lambda ,\beta ,u,v}}{\partial t}(t,s)=tT^1_{\lambda ,\beta ,u,v}(t,s)\quad \text {and}\quad \frac{\partial T_{\lambda ,\beta ,u,v}}{\partial s}(t,s)=sT^2_{\lambda ,\beta ,u,v}(t,s). \end{aligned}$$

(3.4)

Suppose $(u,v)\in \mathcal {A}_{\beta }$, $\lambda \ge \Lambda _1$ and $\beta <0$. Then, by Lemma 2.1, the two-component systems of algebraic equations, given by

$$\begin{aligned} \left\{ \begin{array}{l} T^1_{\lambda ,\beta ,u,v}(t,s)=0,\\ T^2_{\lambda ,\beta ,u,v}(t,s)=0,\end{array}\right. \end{aligned}$$

(3.5)

has a unique nonzero solution $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))$ in $\mathbb {R}^+\times \mathbb {R}^+$, where $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))$ is characterized as (3.2) and (3.3). Hence, by (3.4), $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))$ is the unique one in $\mathbb {R}^+\times \mathbb {R}^+$ such that $\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial t}(t,s)=\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial s}(t,s)=0$, that is, $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))$ is the unique one in $\mathbb {R}^+\times \mathbb {R}^+$ such that $(t_{\lambda ,\beta }(u,v)u,s_{\lambda ,\beta }(u,v)v)\in \mathcal {N}_{\lambda ,\beta }$. It remains to show that $T_{\lambda ,\beta ,u,v}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))=\max _{t\ge 0,s\ge 0}T_{\lambda ,\beta ,u,v}(t,s)$. Indeed, by a direct calculation, we have

$$\begin{aligned} \frac{\partial ^2T_{\lambda ,\beta ,u,v}}{\partial t^2}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))=-2\mu _1\Vert u\Vert _4^4[t(u,v)]^2<0 \end{aligned}$$

and

$$\begin{aligned}&\begin{vmatrix} \frac{\partial ^2T_{\lambda ,\beta ,u,v}}{\partial t^2}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))&\frac{\partial ^2T_{\lambda ,\beta ,u,v}}{\partial t\partial s}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))\\ \frac{\partial ^2T_{\lambda ,\beta ,u,v}}{\partial s\partial t}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))&\frac{\partial ^2T_{\lambda ,\beta ,u,v}}{\partial s^2}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v)) \end{vmatrix}\\&\quad = \begin{vmatrix} -2[t_{\lambda ,\beta }(u,v)]^2\mu _1\Vert u\Vert _4^4&-2t_{\lambda ,\beta }(u,v)s_{\lambda ,\beta }(u,v)\beta \Vert u^2v^2\Vert _1\\ -2t_{\lambda ,\beta }(u,v)s_{\lambda ,\beta }(u,v)\beta \Vert u^2v^2\Vert _1&-2[s_{\lambda ,\beta }(u,v)]^2\mu _2\Vert v\Vert _4^4 \end{vmatrix}\\&\quad =4[t_{\lambda ,\beta }(u,v)]^2[s_{\lambda ,\beta }(u,v)]^2(\mu _1\mu _2\Vert u\Vert _4^4\Vert v\Vert _4^4-\beta ^2\Vert u^2v^2\Vert ^2_1)>0, \end{aligned}$$

which implies that $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))$ is a local maximum of $T_{\lambda ,\beta ,u,v}(t,s)$ in $\mathbb {R}^+\times \mathbb {R}^+$. It follows from the uniqueness of $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))$, $T_{\lambda ,\beta ,u,v}(t,s)>0$ for |(t, s)| sufficiently small and $T_{\lambda ,\beta ,u,v}(t,s)\rightarrow -\infty $ as $|(t,s)|\rightarrow +\infty $ that $(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))$ must be the global maximum of $T_{\lambda ,\beta ,u,v}(t,s)$ in $\mathbb {R}^+\times \mathbb {R}^+$. Thus, we have

$$\begin{aligned} T_{\lambda ,\beta ,u,v}(t_{\lambda ,\beta }(u,v),s_{\lambda ,\beta }(u,v))=\max _{t\ge 0,s\ge 0}T_{\lambda ,\beta ,u,v}(t,s). \end{aligned}$$

(2) Suppose $(u,v)\not \in \mathcal {A}_{\beta }$, $\lambda \ge \Lambda _1$ and $\beta <0$. If $\mathcal {B}_{u,v}\cap \mathcal {N}_{\lambda ,\beta }\not =\emptyset $, then there exists $(t,s)\in \mathbb {R}^+\times \mathbb {R}^+$ such that $\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial t}(t,s)=\frac{\partial T_{\lambda ,\beta ,u,v}}{\partial s}(t,s)=0$. It follows from (3.4) that (t, s) is a solution of (3.5) in $\mathbb {R}^+\times \mathbb {R}^+$. On the other hand, since $(u,v)\not \in \mathcal {A}_{\beta }$, $\lambda \ge \Lambda _1$ and $\beta <0$, by Lemma 2.1, (3.5) has no solution in $\mathbb {R}^+\times \mathbb {R}^+$, which is a contradiction. Hence, we must have $\mathcal {B}_{u,v}\cap \mathcal {N}_{\lambda ,\beta }=\emptyset $ if $(u,v)\not \in \mathcal {A}_{\beta }$, $\lambda \ge \Lambda _1$ and $\beta <0$. $\square $

By Lemma 3.1, we know that $\mathcal {N}_{\lambda ,\beta }\subset \mathcal {A}_{\beta }$ for $\lambda \ge \Lambda _1$ and $\beta <0$. Moreover, $m_{\lambda ,\beta }$ is well defined and nonnegative for $\lambda \ge \Lambda _1$ and $\beta <0$. Let

$$\begin{aligned} I_{a,\lambda }(u)=\frac{1}{2}\Vert u\Vert ^2_{a,\lambda }-\frac{\mu _1}{4}\Vert u\Vert _4^4\quad \text {and}\quad I_{b,\lambda }(v)=\frac{1}{2}\Vert v\Vert ^2_{b,\lambda }-\frac{\mu _2}{4}\Vert v\Vert _4^4. \end{aligned}$$

Then, by (2.3)–(2.6), $I_{a,\lambda }(u)$ is well defined on $E_{a}$ and $I_{b,\lambda }(v)$ is well defined on $E_{b}$. Moreover, by a standard argument, we can see that $I_{a,\lambda }(u)$ and $I_{b,\lambda }(v)$ are of $C^2$ in $E_a$ and $E_b$, respectively. Denote

$$\begin{aligned} \mathcal {N}_{a,\lambda }=\{u\in E_{a}\backslash \{0\}\mid I_{a,\lambda }'(u)u=0\}\quad \text {and}\quad \mathcal {N}_{b,\lambda }=\{u\in E_{b}\backslash \{0\}\mid I_{b,\lambda }'(u)u=0\}. \end{aligned}$$

Clearly, $\mathcal {N}_{a,\lambda }$ and $\mathcal {N}_{b,\lambda }$ are nonempty, which together with Lemma 2.1 implies $m_{a,\lambda }=\inf _{\mathcal {N}_{a,\lambda }}I_{a,\lambda }(u)$ and $m_{b,\lambda }=\inf _{\mathcal {N}_{b,\lambda }}I_{b,\lambda }(v)$ are well defined and nonnegative. Due to this fact, we have the following.

Lemma 3.2

Assume $\lambda \ge \Lambda _1$ and $\beta <0$. Then, $m_{\lambda ,\beta }\in [m_{a,\lambda }+m_{b,\lambda }, m_a+m_b]$, where $m_a$ and $m_b$ are the least energy of nonzero critical points for $I_{\Omega _a}(u)$ and $I_{\Omega _b}(v)$, respectively.

Proof

Suppose $(u,v)\in \mathcal {N}_{\lambda ,\beta }$. Then, by Lemma 2.1 and $(u,v)\in \mathcal {N}_{\lambda ,\beta }\subset \mathcal {A}_{\beta }$, we can see that $\Vert u\Vert _{a,\lambda }^2>0$, $\Vert v\Vert _{b,\lambda }^2>0$, $\Vert u\Vert _4>0$ and $\Vert v\Vert _4>0$. It follows that there exists a unique $(t^*(u), s^*(v))\in \mathbb {R}^+\times \mathbb {R}^+$ such that $(t^*(u)u, s^*(v)v)\in \mathcal {N}_{a,\lambda }\times \mathcal {N}_{b,\lambda }$. Note that $\beta <0$, so by Lemma 3.1, we have

$$\begin{aligned} J_{\lambda ,\beta }(u,v)\ge J_{\lambda ,\beta }(t^*(u)u,s^*(v)v)\ge I_{a,\lambda }(t^*(u)u)+I_{b,\lambda }(s^*(v)v)\ge m_{a,\lambda }+m_{b,\lambda }. \end{aligned}$$

Since $(u,v)\in \mathcal {N}_{\lambda ,\beta }$ is arbitrary, we must have $m_{\lambda ,\beta }\ge m_{a,\lambda }+m_{b,\lambda }$ for $\lambda \ge \Lambda _1$ and $\beta <0$. It remains to show that $m_{\lambda ,\beta }\le m_a+m_b$ for $\lambda \ge \Lambda _1$ and $\beta <0$. In fact, let $w_{a}\in H_0^1(\Omega _a)$ and $w_{b}\in H^1_0(\Omega _b)$ be the least energy nonzero critical points of $I_{\Omega _a}(u)$ and $I_{\Omega _b}(v)$, respectively. Then, by the conditions $(D_3)$ and $(D_5)$, it is well known that

$$\begin{aligned} I_{\Omega _a}(w_{a})=\max _{t\ge 0}I_{\Omega _a}(tw_{a})\quad \text {and}\quad I_{\Omega _b}(w_{b})=\max _{s\ge 0}I_{\Omega _b}(sw_{b}). \end{aligned}$$

On the other hand, by the condition $(D_3)$, we can extend $w_{a}$ and $w_{b}$ to $\mathbb {R}^3$ by letting $w_a=0$ outside $\Omega _a$ and $w_b=0$ outside $\Omega _b$ such that $w_a,w_b\in H^1(\mathbb {R}^3)$. Thanks to the condition $(D_3)$ again, we can see that $(w_{a}, w_{b})\in \mathcal {A}_{\beta }$. It follows from Lemma 3.1 that there exists $(t_{\lambda ,\beta }(w_{a}, w_{b}), s_{\lambda ,\beta }(w_{a}, w_{b}))\in \mathbb {R}^+\times \mathbb {R}^+$ such that

$$\begin{aligned} (t_{\lambda ,\beta }(w_{a}, w_{b})w_a, s_{\lambda ,\beta }(w_{a}, w_{b})w_b)\in \mathcal {N}_{\lambda ,\beta }\quad \text {for }\lambda \ge \Lambda _1, \end{aligned}$$

which together with the condition $(D_3)$ once more implies

$$\begin{aligned} m_a+m_b= & {} I_{\Omega _a}(w_{a})+I_{\Omega _b}(w_{b})\\\ge & {} I_{\Omega _a}(t_{\lambda ,\beta }(w_{a}, w_{b})w_{a})+I_{\Omega _b}(s_{\lambda ,\beta }(w_{a}, w_{b})w_{b})\\= & {} J_{\lambda ,\beta }(t_{\lambda ,\beta }(w_{a}, w_{b})w_{a},s_{\lambda ,\beta }(w_{a}, w_{b})w_{b})\\\ge & {} m_{\lambda ,\beta } \end{aligned}$$

for $\lambda \ge \Lambda _1$ and $\beta <0$. $\square $

Clearly, $m_{a,\lambda }$ and $m_{b,\lambda }$ are nondecreasing for $\lambda $. On the other hand, since Lemma 2.1 hold, by the conditions $(D_1){-}(D_5)$, it is easy to show that $m_{a,\lambda }$ and $m_{b,\lambda }$ are positive for $\lambda \ge \Lambda _1$ and $m_{a,\lambda }\rightarrow m_{a}$ and $m_{b, \lambda }\rightarrow m_{b}$ as $\lambda \rightarrow +\infty $. This fact will help us to observe the following property of $\mathcal {N}_{\lambda ,\beta }$, which is based on Lemma 3.2.

Lemma 3.3

Assume $\lambda \ge \Lambda _1$ and $\beta <0$. Then, there exists $d_{\lambda ,\beta }>0$ such that $\mathcal {N}_{\lambda ,\beta }\subset \mathcal {A}_{\beta }^{d_{\lambda ,\beta }}$, where $\mathcal {A}_{\beta }^{d_{\lambda ,\beta }}=\{(u,v)\in E \mid u\not =0,v\not =0,\mu _1\mu _2\Vert u\Vert _4^4\Vert v\Vert _4^4-\beta ^2\Vert u^2v^2\Vert ^2_1>d_{\lambda ,\beta }\}$.

Proof

A similar result was obtained in [16]. But as we will see, some new ideas are needed due to the fact that $a_0(x)$ and $b_0(x)$ are sign-changing. Suppose the contrary. Since $\mathcal {N}_{\lambda ,\beta }\subset \mathcal {A}_{\beta }$, there exists a sequence $\{(u_n,v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$ such that $\mu _1\mu _2\Vert u_n\Vert _4^4\Vert v_n\Vert _4^4=\beta ^2\Vert u_n^2v_n^2\Vert ^2_1+o_n(1)$, where $\mathcal {A}_{\beta }$ is given in (3.1). Clearly, one of the following two cases must happen:

(a):: $\Vert u_n\Vert _4^4\Vert v_n\Vert _4^4\ge C+o_n(1)$.
(b):: $\Vert u_n\Vert _4^4\Vert v_n\Vert _4^4=o_n(1)$ up to a subsequence.

Suppose case (a) happens. We claim that $\mu _1\Vert u_n\Vert _4^4+\beta \Vert u_n^2v_n^2\Vert _1=o_n(1)$ and $\mu _2\Vert v_n\Vert _4^4+\beta \Vert u_n^2v_n^2\Vert _1=o_n(1)$ up to a subsequence. If not, then up to a subsequence, we have

$$\begin{aligned} \mu _1\Vert u_n\Vert _4^4+\beta \Vert u_n^2v_n^2\Vert _1\ge C_1+o_n(1)\quad \text {and}\quad \mu _2\Vert v_n\Vert _4^4+\beta \Vert u_n^2v_n^2\Vert _1\ge C_2+o_n(1) \end{aligned}$$

for $\lambda \ge \Lambda _1$ and $\beta <0$, where $C_1, C_2$ are nonnegative constants with $C_1+C_2>0$. It follows from $\beta <0$ that

$$\begin{aligned} \mu _1\mu _2\Vert u_n\Vert _4^4\Vert v_n\Vert _4^4\ge & {} (C_1+o_n(1)+|\beta |\Vert u_n^2v_n^2\Vert _1)(C_2+o_n(1)+|\beta |\Vert u_n^2v_n^2\Vert _1)\\\ge & {} \beta ^2\Vert u_n^2v_n^2\Vert ^2_1+(C_1+C_2+o_n(1))|\beta |\Vert u_n^2v_n^2\Vert _1+C_1C_2+o_n(1)\\\ge & {} \beta ^2\Vert u_n^2v_n^2\Vert ^2_1+\frac{1}{2}(C_1+C_2)\sqrt{C}+o_n(1) \end{aligned}$$

for n large enough, which contradicts to $\mu _1\mu _2\Vert u_n\Vert _4^4\Vert v_n\Vert _4^4=\beta ^2\Vert u_n^2v_n^2\Vert ^2_1+o_n(1)$. This together with $\{(u_n,v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$ implies that $\Vert u_n\Vert _{a,\lambda }=o_n(1)$ and $\Vert v_n\Vert _{b,\lambda }=o_n(1)$ up to a subsequence. Note that $J_{\lambda ,\beta }(u_n,v_n)=\frac{1}{4}(\Vert u_n\Vert _{a,\lambda }^2+\Vert v_n\Vert _{b,\lambda }^2)$. So $m_{\lambda ,\beta }\le 0$ in case (a), which is impossible for $\lambda \ge \Lambda _1$ and $\beta <0$ due to Lemma 3.2. Now, we must have case (b). It follows that $\Vert u_n\Vert _4=o_n(1)$ or $\Vert v_n\Vert _4=o_n(1)$ up to a subsequence. Without loss of generality, we assume $\Vert u_n\Vert _4=o_n(1)$. Since $\mu _1\mu _2\Vert u_n\Vert _4^4\Vert v_n\Vert _4^4=\beta ^2\Vert u_n^2v_n^2\Vert ^2_1+o_n(1)$, we also have $\beta \Vert u_n^2v_n^2\Vert _1=o_n(1)$ in this case. These together with $\{(u_n,v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$ imply $\Vert u_n\Vert _{a,\lambda }=o_n(1)$. Therefore, $J_{\lambda ,\beta }(u_n,v_n)=I_{b,\lambda }(v_n)+o_n(1)$. On the other hand, since $\lambda \ge \Lambda _1$ and $\{(u_n,v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$, by Lemma 2.1 and $\mathcal {N}_{\lambda ,\beta }\subset \mathcal {A}_{\beta }$, for all $n\in \mathbb {N}$, there exists a unique $t^*(u_n)>0$ such that $t^*(u_n)u_n\in \mathcal {N}_{a,\lambda }$. It follows from Lemma 3.1 and $\beta <0$ that

$$\begin{aligned} J_{\lambda ,\beta }(u_n,v_n)\ge & {} J_{\lambda ,\beta }(t^*(u_n)u_n,v_n)\\\ge & {} I_{a,\lambda }(t^*(u_n)u_n)+I_{b,\lambda }(v_n)\\\ge & {} m_{a,\lambda }+I_{b,\lambda }(v_n)\\= & {} m_{a,\lambda }+J_{\lambda ,\beta }(u_n,v_n)+o_n(1) \end{aligned}$$

for $\lambda \ge \Lambda _1$ and $\beta <0$, which is also impossible for n large enough. Thus, there exists $d_{\lambda ,\beta }>0$ such that $\mathcal {N}_{\lambda ,\beta }\subset \mathcal {A}_{\beta }^{d_{\lambda ,\beta }}$ for $\lambda \ge \Lambda _1$ and $\beta <0$. $\square $

We also have the following.

Lemma 3.4

Suppose $\lambda \ge \Lambda _1$ and $\beta <0$. Then, $\mathcal {N}_{\lambda ,\beta }$ is a natural constraint.

Proof

Let $\varphi _{\lambda ,\beta }(u,v)=\langle D[J_{\lambda ,\beta }(u,v)],(u,v)\rangle _{E^*,E}$. Then, by (2.3)–(2.6), $\varphi _{\lambda ,\beta }(u,v)$ is $C^2$ in E for $\lambda \ge \Lambda _1$ and $\beta <0$. Since $\lambda \ge \Lambda _1$ and $(u,v)\in \mathcal {N}_{\lambda ,\beta }$, we have

$$\begin{aligned} \langle D[\varphi _{\lambda ,\beta }(u,v)],(u,v)\rangle _{E^*,E}=-2(\mu _1\Vert u\Vert _4^4+\mu _2\Vert v\Vert _4^4+2\beta \Vert u^2v^2\Vert _1) \le -8m_{\lambda ,\beta }. \end{aligned}$$

It follows from Lemma 3.2 that $\mathcal {N}_{\lambda ,\beta }$ is a natural constraint for $\lambda \ge \Lambda _1$ and $\beta <0$. $\square $

Now, we can obtain a ground state solution for $(\mathcal {P}_{\lambda ,\beta })$.

Proposition 3.1

There exists $\Lambda _2\ge \Lambda _1$ such that $(\mathcal {P}_{\lambda ,\beta })$ has a ground state solution $(u_{\lambda ,\beta },v_{\lambda ,\beta })$ for $\lambda \ge \Lambda _2$ and $\beta <0$. Furthermore, we have

$$\begin{aligned} \lim _{\lambda \rightarrow +\infty }\int _{\mathbb {R}^3\backslash \Omega _a'}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2 \mathrm{d}x=0 \end{aligned}$$

(3.6)

and

$$\begin{aligned} \lim _{\lambda \rightarrow +\infty }\int _{\mathbb {R}^3\backslash \Omega _b'}|\nabla v_{\lambda ,\beta }|^2+(\lambda b(x)+b_0(x))v_{\lambda ,\beta }^2 \mathrm{d}x=0. \end{aligned}$$

(3.7)

Proof

Let $\lambda \ge \Lambda _1$ and $\beta <0$. Then, for every $\{(u_n,v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$ satisfying $J_{\lambda ,\beta }(u_n,v_n)=m_{\lambda ,\beta }+o_n(1)$, we can see from Lemma 2.1 that

$$\begin{aligned} m_{\lambda ,\beta }+o_n(1)\ge & {} J_{\lambda ,\beta }(u_n,v_n)-\frac{1}{4}\langle D[J_{\lambda ,\beta }(u_n,v_n)], (u_n,v_n)\rangle _{E^*,E}\nonumber \\= & {} \frac{1}{4}\Vert u_n\Vert ^2_{a,\lambda }+\frac{1}{4}\Vert v_n\Vert ^2_{b,\lambda }\nonumber \\\ge & {} \frac{1}{4C_{a,b}}(\Vert u_n\Vert _2^2+\Vert v_n\Vert _2^2), \end{aligned}$$

(3.8)

which together with the condition $(D_4)$ and $\lambda \ge \Lambda _1$ implies

$$\begin{aligned} m_{\lambda ,\beta }+o_n(1)\ge & {} \frac{1}{4}\Vert u_n\Vert ^2_{a,\lambda } +\frac{1}{4}\Vert v_n\Vert ^2_{b,\lambda }\nonumber \\\ge & {} \frac{1}{8}\Vert (u_n,v_n)\Vert ^2-C(m_{\lambda ,\beta }+o_n(1)). \end{aligned}$$

(3.9)

It follows that $\Vert (u_n,v_n)\Vert \le 8(C+1)(m_{\lambda ,\beta }+o_n(1))$. Now, by Lemma 3.3, we can apply the implicit function theorem and the Ekeland variational principle in a standard way (cf. [13, 36]) to show that there exists $\{(u_n, v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$ such that $D[J_{\lambda ,\beta }(u_n,v_n)]=o_n(1)$ strongly in $E^*$ and $J_{\lambda ,\beta }(u_n,v_n)=m_{\lambda ,\beta }+o_n(1)$. Since $m_{\lambda ,\beta }\le m_{a}+m_{b}$, by similar arguments as (3.8) and (3.9), we have $\Vert (u_n,v_n)\Vert \le 8(C+1)(m_{a}+m_{b}+o_n(1))$ and $(u_n,v_n)\rightharpoonup (u_{\lambda ,\beta },v_{\lambda ,\beta })$ weakly in E as $n\rightarrow \infty $ for some $(u_{\lambda ,\beta },v_{\lambda ,\beta })\in E$. Clearly, $D[J_{\lambda ,\beta }(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$. Suppose $u_{\lambda ,\beta }=0$. Then, by the fact that $E_a$ is embedded continuously into $H^1(\mathbb {R}^3)$, we have

$$\begin{aligned} u_n=o_n(1)\quad \text {strongly in }L^p_{loc}(\mathbb {R}^3)\quad \text {for }2\le p<6. \end{aligned}$$

Combining with the condition $(D_2)$ and the Hölder and the Sobolev inequalities, we get

$$\begin{aligned} \int _{\mathbb {R}^3}|u_n|^4\mathrm{d}x= & {} \int _{\mathcal {D}_a}|u_n|^4\mathrm{d}x +\int _{\mathbb {R}^3\backslash \mathcal {D}_a}|u_n|^4\mathrm{d}x\nonumber \\= & {} \int _{\mathbb {R}^3\backslash \mathcal {D}_a}|u_n|^4\mathrm{d}x+o_n(1)\nonumber \\\le & {} \bigg (\frac{1}{a_\infty }\bigg )^{\frac{1}{2}} \int _{\mathbb {R}^3\backslash \mathcal {D}_a}[a(x)]^{\frac{1}{2}}|u_n|^4\mathrm{d}x+o_n(1)\nonumber \\\le & {} \bigg (\frac{1}{a_\infty S^3}\bigg )^{\frac{1}{2}} \bigg (\int _{\mathbb {R}^3\backslash \mathcal {D}_a}a(x)|u_n|^2\mathrm{d}x\bigg )^{\frac{1}{2}} \bigg (\int _{\mathbb {R}^3}|\nabla u_n|^2\mathrm{d}x\bigg )^{\frac{3}{2}}+o_n(1). \qquad \qquad \end{aligned}$$

(3.10)

Since $u_n=o_n(1)$ strongly in $L^p_{loc}(\mathbb {R}^3)$ for $2\le p<6$, by the conditions $(D_2)$ and $(D_4)$, $\int _{\mathcal {D}_a}(\lambda a(x)+a_0(x))u_n^2\mathrm{d}x=o_n(1)$. It follows from (3.8) and (3.10) that

$$\begin{aligned} \int _{\mathbb {R}^3}|u_n|^4\mathrm{d}x\le & {} \bigg (\frac{1}{a_\infty S^3}\bigg )^{\frac{1}{2}} \bigg (\int _{\mathbb {R}^3\backslash \mathcal {D}_a}a(x)|u_n|^2\mathrm{d}x\bigg )^{\frac{1}{2}} \bigg (\int _{\mathbb {R}^3}|\nabla u_n|^2\mathrm{d}x\bigg )^{\frac{3}{2}}+o_n(1)\nonumber \\\le & {} \bigg (\frac{2}{a_\infty S^3\lambda }\bigg )^{\frac{1}{2}} \Vert u_n\Vert _{a,\lambda }(\Vert u_n\Vert _{a,\lambda }^2+o_n(1))^{\frac{3}{2}}+o_n(1)\nonumber \\\le & {} \bigg (\frac{2}{a_\infty S^3\lambda }\bigg )^{\frac{1}{2}}\Vert u_n\Vert _{a,\lambda }^4+o_n(1) \end{aligned}$$

(3.11)

for $\lambda \ge \Lambda _1$. Note that $\{(u_n, v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$ and $\beta <0$, from (3.8) and (3.11), we have

$$\begin{aligned} \Vert u_n\Vert _{a,\lambda }^2\le \bigg (\frac{2}{a_\infty S^3\lambda }\bigg )^{\frac{1}{2}}\Vert u_n\Vert _{a,\lambda }^4+o_n(1)\le 4(m_a+m_b)\bigg (\frac{2}{a_\infty S^3\lambda }\bigg )^{\frac{1}{2}}\Vert u_n\Vert _{a,\lambda }^2+o_n(1),\nonumber \\ \end{aligned}$$

(3.12)

which then implies that there exists $\Lambda _2\ge \Lambda _1$ such that $\Vert u_n\Vert _{a,\lambda }=o_n(1)$ for $\lambda \ge \Lambda _2$ and $\beta <0$. It follows from Lemma 2.1, the Hölder and the Sobolev inequalities and the boundedness of $\{(u_n,v_n)\}$ in E that $\Vert u_n\Vert _4=o_n(1)$, hence, $\mu _1\mu _2\Vert u_n\Vert _4^4\Vert v_n\Vert _4^4=o_n(1)$ for $\lambda \ge \Lambda _2$ and $\beta <0$. However, it is impossible, since $\{(u_n, v_n)\}\subset \mathcal {N}_{\lambda ,\beta }$, $\Lambda _2\ge \Lambda _1$ and Lemma 3.3 holds for $\lambda \ge \Lambda _1$. Therefore, there exists $\Lambda _2\ge \Lambda _1$ such that $u_{\lambda ,\beta }\not =0$ for $\lambda \ge \Lambda _2$ and $\beta <0$. Similarly, we can also show that $v_{\lambda ,\beta }\not =0$ for $\lambda \ge \Lambda _2$ and $\beta <0$. Since $(u_n,v_n)\rightharpoonup (u_{\lambda ,\beta },v_{\lambda ,\beta })$ weakly in E as $n\rightarrow \infty $, by the fact that E is embedded continuously into $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$, we have $(u_n,v_n)\rightarrow (u_{\lambda ,\beta },v_{\lambda ,\beta })$ strongly in $L_{loc}^{p}(\mathbb {R}^3)\times L_{loc}^{p}(\mathbb {R}^3)$ as $n\rightarrow \infty $ for $2\le p<6$. It follows from the boundedness of $\{(u_n,v_n)\}$ in E and the conditions $(D_2)$ and $(D_4)$ that $\int _{\mathcal {D}_a}a_0^-(x)u_n^2\mathrm{d}x=\int _{\mathcal {D}_a}a_0^-(x)u_{\lambda ,\beta }^2\mathrm{d}x+o_n(1)$ and $\int _{\mathcal {D}_b}b_0^-(x)v_n^2\mathrm{d}x=\int _{\mathcal {D}_b}b_0^-(x)v_{\lambda ,\beta }^2\mathrm{d}x+o_n(1)$, which together with $D[J_{\lambda ,\beta }(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$, the Fatou lemma and the conditions $(D_2)$ and $(D_4)$, implies

$$\begin{aligned} m_{\lambda ,\beta }\le & {} J_{\lambda ,\beta }(u_{\lambda ,\beta },v_{\lambda ,\beta })-\frac{1}{4}\langle D[J_{\lambda ,\beta }(u_{\lambda ,\beta },v_{\lambda ,\beta })],(u_{\lambda ,\beta },v_{\lambda ,\beta })\rangle _{E^*,E}\nonumber \\= & {} \frac{1}{4}(\Vert u_{\lambda ,\beta }\Vert _{a,\lambda }^2+\Vert v_{\lambda ,\beta }\Vert _{b,\lambda }^2)\nonumber \\\le & {} \liminf _{n\rightarrow \infty }\frac{1}{4}(\Vert u_{n}\Vert _{a,\lambda }^2+\Vert v_{n}\Vert _{b,\lambda }^2)\nonumber \\= & {} \liminf _{n\rightarrow \infty }(J_{\lambda ,\beta }(u_{\lambda ,\beta },v_{\lambda ,\beta })-\frac{1}{4}\langle D[J_{\lambda ,\beta }(u_{n},v_{n})],(u_{n},v_{n})\rangle _{E^*,E})\nonumber \\= & {} m_{\lambda ,\beta }+o_n(1). \end{aligned}$$

(3.13)

Therefore, $J_{\lambda ,\beta }(u_{\lambda ,\beta },v_{\lambda ,\beta })=m_{\lambda ,\beta }$. Since $J_{\lambda ,\beta }(|u_{\lambda ,\beta }|,|v_{\lambda ,\beta }|)=m_{\lambda ,\beta }$ and $(|u_{\lambda ,\beta }|,|v_{\lambda ,\beta }|)\in \mathcal {N}_{\lambda ,\beta }$, $(|u_{\lambda ,\beta }|,|v_{\lambda ,\beta }|)$ is a local minimizer of $J_{\lambda ,\beta }(u,v)$ on $\mathcal {N}_{\lambda ,\beta }$. Note that by Lemma 3.4, $\mathcal {N}_{\lambda ,\beta }$ is a natural constraint, we can follow the argument as used in [6, Theorem 2.3] to show that $D[J_{\lambda ,\beta }(|u_{\lambda ,\beta }|,|v_{\lambda ,\beta }|)]=0$ in $E^*$. Thus, without loss of generality, we may assume $u_{\lambda ,\beta }$ and $v_{\lambda ,\beta }$ both are nonnegative. Now, since $(u_{\lambda ,\beta }, v_{\lambda ,\beta })\in E$, by (2.1) and (2.2), we have $u_{\lambda ,\beta }, v_{\lambda ,\beta }\in H^1(\mathbb {R}^3)$. It follows from the conditions $(D_1)$ and $(D_4)$ and the Calderon-Zygmund inequality that $u_{\lambda ,\beta }, v_{\lambda ,\beta }\in W_{loc}^{2,2}(\mathbb {R}^{3})$. By combining the Sobolev embedding theorem and the Harnack inequality, $u_{\lambda ,\beta }$ and $v_{\lambda ,\beta }$ are both positive. Hence, $(u_{\lambda ,\beta }, v_{\lambda ,\beta })$ is a ground state solution of $(\mathcal {P}_{\lambda ,\beta })$ for $\beta <0$ and $\lambda \ge \Lambda _2$. It remains to show that (3.6) and (3.7) are true. Indeed, let $\Omega _{a}''$ be a bounded domain with smooth boundary in $\mathbb {R}^3$ satisfying $\Omega _a\subset \Omega _{a}''\subset \Omega _a'$, $dist(\Omega _{a}'', \mathbb {R}^3\backslash \Omega _{a}')>0$ and $dist(\mathbb {R}^3\backslash \Omega _{a}'', \Omega _{a})>0$. Then, by a similar argument as (2.7), we can show that

$$\begin{aligned} \int _{\Omega _a''}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2\mathrm{d}x\ge \frac{\nu _a}{2}\int _{\Omega _a''}u_{\lambda ,\beta }^2\mathrm{d}x \end{aligned}$$

(3.14)

for $\lambda $ large enough. Without loss of generality, we assume (3.14) holds for $\lambda \ge \Lambda _2$. Since $(u_{\lambda ,\beta }, v_{\lambda ,\beta })$ is a ground state solution for $\lambda \ge \Lambda _2$, by combining Lemma 2.1, (3.14) and similar arguments of (3.8) and (3.9), we can see that $\Vert (u_{\lambda ,\beta }, v_{\lambda ,\beta })\Vert ^2\le 8(4(C_{a,0}+d_a)C_{a,b}+1)(m_a+m_b)$ and $8(4(C_{a,0}+d_a)C_{a,b}+1)(m_a+m_b)\ge \lambda \int _{\mathbb {R}^3\backslash \Omega _a''}a(x)u_{\lambda ,\beta }^2\mathrm{d}x$ for $\lambda \ge \Lambda _2$, which together with the conditions $(D_1){-}(D_3)$ imply $\int _{(\mathbb {R}^3\backslash \Omega _a'')\cap \mathcal {D}_a}u_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0$ as $\lambda \rightarrow +\infty $. It follows from the condition $(D_2)$ and $8(4(C_{a,0}+d_a)C_{a,b}+1)(m_a+m_b)\ge \lambda \int _{\mathbb {R}^3\backslash \Omega _a''}a(x)u_{\lambda ,\beta }^2\mathrm{d}x$ once more that

$$\begin{aligned} \lim _{\lambda \rightarrow +\infty }\int _{\mathbb {R}^3\backslash \Omega _a''} u_{\lambda ,\beta }^2\mathrm{d}x=0. \end{aligned}$$

(3.15)

Now, we choose $\Psi \in C^\infty (\mathbb {R}^3, [0, 1])$ satisfying

$$\begin{aligned} \Psi =\left\{ \begin{array}{ll} 1,&{}\quad x\in \mathbb {R}^3\backslash \Omega _a',\\ 0,&{}\quad x\in \Omega _a''.\end{array}\right. \end{aligned}$$

(3.16)

Then, $u_{\lambda ,\beta }\Psi \in E_a$. Note $D[J_{\lambda ,\beta }(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$ for $\lambda \ge \Lambda _2$, we have that

$$\begin{aligned}&\int _{\mathbb {R}^3}(|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2)\Psi \mathrm{d}x+\int _{\mathbb {R}^3}(\nabla u_{\lambda ,\beta }\nabla \Psi ) u_{\lambda ,\beta } \mathrm{d}x\\&\quad =\mu _1\int _{\mathbb {R}^3}u_{\lambda ,\beta }^4\Psi \mathrm{d}x+\beta \int _{\mathbb {R}^3}v_{\lambda ,\beta }^2u_{\lambda ,\beta }^2\Psi \mathrm{d}x. \end{aligned}$$

It follows from the Hölder and the Sobolev inequalities that

$$\begin{aligned}&\int _{\mathbb {R}^3\backslash \Omega _a'}(|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2)\mathrm{d}x\nonumber \\&\quad \le \mu _1\int _{\mathbb {R}^3\backslash \Omega _a''}u_{\lambda ,\beta }^4\mathrm{d}x+\beta \int _{\mathbb {R}^3\backslash \Omega _a'}v_{\lambda ,\beta }^2u_{\lambda ,\beta }^2\mathrm{d}x +\int _{\Omega _a'\backslash \Omega _a''}|\nabla u_{\lambda ,\beta }||\nabla \Psi ||u_{\lambda ,\beta }|\mathrm{d}x\nonumber \\&\quad \le \mu _1S^{-\frac{3}{2}}\Vert u_{\lambda ,\beta }\Vert _a^3(\int _{\mathbb {R}^3\backslash \Omega _a''}u_{\lambda ,\beta }^2\mathrm{d}x)^{\frac{1}{2}} +\max _{\mathbb {R}^3}|\nabla \Psi |\Vert u_{\lambda ,\beta }\Vert _a(\int _{\mathbb {R}^3\backslash \Omega _a''}u_{\lambda ,\beta }^2\mathrm{d}x)^{\frac{1}{2}}.\qquad \end{aligned}$$

(3.17)

Since $\Vert (u_{\lambda ,\beta }, v_{\lambda ,\beta })\Vert ^2\le 8(4(C_{a,0}+d_a)C_{a,b}+1)(m_a+m_b)$, we can conclude from (2.8), (3.15) and (3.17) that (3.6) holds. Similarly, we can also conclude that (3.7) is true. $\square $

We close this section by

Proof of Theorem 1.1

By Proposition 3.1, we know that there exists $\Lambda _2\ge \Lambda _1$ such that $(\mathcal {P}_{\lambda ,\beta })$ has a ground state solution $(u_{\lambda ,\beta },v_{\lambda ,\beta })$ for $\lambda \ge \Lambda _2$ and $\beta <0$. In what follows, we will show that $(u_{\lambda ,\beta },v_{\lambda ,\beta })$ has the concentration behaviors for $\lambda \rightarrow +\infty $ described as (1)–(5). We first verify (3)–(5). Let $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })$ be the ground state solution of $(\mathcal {P}_{\lambda _n,\beta })$ obtained by Proposition 3.1 with $\lambda _n\rightarrow +\infty $ as $n\rightarrow \infty $. Then, by Lemma 3.2 and Proposition 3.1, $\{(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\}$ is bounded in E with

$$\begin{aligned} \lim _{n\rightarrow +\infty }\int _{\mathbb {R}^3\backslash \Omega _a'}|\nabla u_{\lambda _n,\beta }|^2+({\lambda _n} a(x)+a_0(x))u_{\lambda _n,\beta }^2\mathrm{d}x=0 \end{aligned}$$

(3.18)

and

$$\begin{aligned} \lim _{n\rightarrow +\infty }\int _{\mathbb {R}^3\backslash \Omega _b'}|\nabla v_{\lambda _n,\beta }|^2+({\lambda _n} b(x)+b_0(x))v_{\lambda _n,\beta }^2\mathrm{d}x=0. \end{aligned}$$

(3.19)

Without loss of generality, we assume $(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\rightharpoonup (u_{0,\beta }, v_{0,\beta })$ weakly in E as $n\rightarrow \infty $ for some $(u_{0,\beta }, v_{0,\beta })\in E$. By (2.1) and (2.2), $(u_{0,\beta }, v_{0,\beta })\in H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$. For the sake of clarity, the verification of (3)–(5) is further performed through the following three steps.

Step 1 We prove that $(u_{0,\beta }, v_{0,\beta })\in H_0^1(\Omega _{a})\times H_0^1(\Omega _{b})$ with $u_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _{a}$ and $v_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _{b}$.

Indeed, since $(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })$ is a ground state solution of $(\mathcal {P}_{\lambda _n,\beta })$, by Lemma 3.2 and a similar argument as (3.8), we get that $4C_{a,b}(m_a+m_b)\ge (\Vert u_{\lambda _n,\beta }\Vert _2^2+\Vert v_{\lambda _n,\beta }\Vert _2^2)$. Now, by the condition $(D_4)$ and a similar argument as (3.8) again, we have

$$\begin{aligned} m_{a}+m_{b}\ge & {} J_{\lambda _n,\beta }(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })-\frac{1}{4}\langle D[J_{\lambda _n,\beta }(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })], (u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\rangle _{E^*,E}\\\ge & {} \frac{1}{4}\int _{\mathbb {R}^3}(\lambda _n a(x)+a_0(x))u^2_{\lambda _n,\beta }\mathrm{d}x+\frac{1}{4}\int _{\mathbb {R}^3}({\lambda _n} b(x)+b_0(x))v_{\lambda _n,\beta }^2\mathrm{d}x\\\ge & {} \frac{\lambda _n}{8}\int _{\mathbb {R}^3}a(x)u^2_{\lambda _n,\beta }+b(x)v_{\lambda _n,\beta }^2\mathrm{d}x-C(\Vert (u_{\lambda _n,\beta }\Vert _2^2+\Vert v_{\lambda _n,\beta })\Vert _2^2) \\\ge & {} \frac{\lambda _n}{8}\int _{\mathbb {R}^3}a(x)u^2_{\lambda _n,\beta }+b(x)v_{\lambda _n,\beta }^2\mathrm{d}x-C'. \end{aligned}$$

It follows that $\lim _{n\rightarrow +\infty }\int _{\mathbb {R}^3}a(x)u_{\lambda _n,\beta }^2\mathrm{d}x=\lim _{n\rightarrow +\infty }\int _{\mathbb {R}^3}b(x)v_{\lambda _n,\beta }^2\mathrm{d}x=0$. By the Fatou Lemma and the conditions $(D_1)$ and $(D_3)$, we can see that $u_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _a$ and $v_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _b$. Since $(u_{0,\beta }, v_{0,\beta })\in H^1(\mathbb {R}^3)$, by the condition $(D_3)$ again, we must have $u_{0,\beta }\in H^1_0(\Omega _a)$ and $v_{0,\beta }\in H^1_0(\Omega _b)$.

Step 2 We prove that $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta },v_{0,\beta })$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $ up to a subsequence.

Indeed, by the choice of $\Omega _a'$ and the Sobolev embedding theorem, we can see that $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $L^2(\Omega _a')$ as $n\rightarrow \infty $ up to a subsequence. Without loss of generality, we may assume $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $L^2(\Omega _a')$. It follows from $u_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _a$, (2.8) and (3.18) that $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $L^2(\mathbb {R}^3)$ as $n\rightarrow \infty $. By the Hölder and the Sobolev inequalities and the boundedness of $\{(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\}$ in E, we can see that $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $L^4(\mathbb {R}^3)$ as $n\rightarrow \infty $. On the other hand, by a similar argument as (3.13), we have

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla u_{0,\beta }|^2+a_0(x)u_{0,\beta }^2\mathrm{d}x= & {} \int _{\mathbb {R}^3}|\nabla u_{0,\beta }|^2\mathrm{d}x+(\lambda _na(x)+a_0(x))u_{0,\beta }^2\mathrm{d}x\\\le & {} \liminf _{n\rightarrow \infty }\int _{\mathbb {R}^3}|\nabla u_{\lambda _n,\beta }|^2+(\lambda _na(x)+a_0(x))u_{\lambda _n,\beta }^2\mathrm{d}x, \end{aligned}$$

which together with $D[J_{\lambda _n,\beta }(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })]=0$ in $E^*$ and $\beta <0$ implies

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla u_{0,\beta }|^2+a_0(x)u_{0,\beta }^2\mathrm{d}x\le \liminf _{n\rightarrow \infty }\mu _1\int _{\mathbb {R}^3}u_{\lambda _n,\beta }^4\mathrm{d}x=\mu _1\int _{\Omega _{a}}u_{0,\beta }^4\mathrm{d}x. \end{aligned}$$

Note that $(u_{0,\beta },v_{0,\beta })\in H_0^1(\Omega _{a})\times H_0^1(\Omega _{b})$ with $u_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _a$ and $v_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _b$ and $D[J_{\lambda _n,\beta }(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })]=0$ in $E^*$, by the condition $(D_3)$, we can show that $I_{\Omega _{a}}'(u_{0,\beta })=0$ in $H^{-1}(\Omega _{a})$ and $I_{\Omega _{b}}'(v_{0,\beta })=0$ in $H^{-1}(\Omega _{b})$. Recalling that $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $L^2(\mathbb {R}^3)$ as $n\rightarrow \infty $, $\{(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\}$ is bounded in E and the conditions $(D_2)$ and $(D_4)$ hold, we must have $\int _{\mathcal {D}_a}a_0^-(x)u_{\lambda _n,\beta }^2\mathrm{d}x=\int _{\mathcal {D}_a}a_0^-(x)u_{0,\beta }^2\mathrm{d}x+o_n(1)$. It follows from the conditions $(D_2)$ and $(D_4)$ again and the Fatou Lemma that

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla u_{0,\beta }|^2+a_0(x)u_{0,\beta }^2\mathrm{d}x= & {} \liminf _{n\rightarrow \infty }\int _{\mathbb {R}^3}|\nabla u_{\lambda _n,\beta }|^2+(\lambda _na(x)+a_0(x))u_{\lambda _n,\beta }^2\mathrm{d}x\nonumber \\\ge & {} \liminf _{n\rightarrow \infty }(\int _{\mathbb {R}^3}|\nabla u_{\lambda _n,\beta }|^2+\frac{\lambda _n}{2} a(x)u_{\lambda _n,\beta }^2\mathrm{d}x)\nonumber \\&+\int _{\mathbb {R}^3}a_0^+(x)u_{0,\beta }^2\mathrm{d}x+\int _{\mathcal {D}_a}a_0^-(x)u_{0,\beta }^2\mathrm{d}x. \end{aligned}$$

(3.20)

By the conditions $(D_1){-}(D_3)$, the Fatou lemma and the fact $u_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _a$, we can see from (3.20) that $\nabla u_{\lambda _n,\beta }\rightarrow \nabla u_{0,\beta }$ strongly in $L^2(\mathbb {R}^3)$ as $n\rightarrow \infty $ up to a subsequence, which then implies $\liminf _{n\rightarrow \infty }\int _{\mathbb {R}^3}\lambda _na(x)u_{\lambda _n,\beta }^2\mathrm{d}x=0$ and $\liminf _{n\rightarrow \infty }\int _{\mathbb {R}^3}a_0(x)u_{\lambda _n,\beta }^2\mathrm{d}x=\int _{\mathbb {R}^3}a_0(x)u_{0,\beta }^2\mathrm{d}x$. These together with the conditions $(D_1){-}(D_4)$, $u_{0,\beta }=0$ on $\mathbb {R}^3\backslash \Omega _a$ and $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $L^2(\mathbb {R}^3)$ as $n\rightarrow \infty $ imply $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $E_a$ as $n\rightarrow \infty $ up to a subsequence. Without loss of generality, we assume $u_{\lambda _n,\beta }\rightarrow u_{0,\beta }$ strongly in $E_a$ as $n\rightarrow \infty $. Similarly, we also have $v_{\lambda _n,\beta }\rightarrow v_{0,\beta }$ strongly in $E_b$ as $n\rightarrow \infty $, that is, $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta },v_{0,\beta })$ strongly in E as $n\rightarrow \infty $. Since E is embedded continuously into $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$, $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta },v_{0,\beta })$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $.

Step 3 We prove that $u_{0,\beta }$ and $v_{0,\beta }$ are least energy nonzero critical points of $I_{\Omega _{a}}(u)$ and $I_{\Omega _{b}}(v)$, respectively.

Indeed, since $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })$ is the ground state solution of $(\mathcal {P}_{\lambda _n,\beta })$, by Lemma 3.2, we can see that

$$\begin{aligned} \Vert u_{\lambda _n,\beta }\Vert _{a,\lambda _n}+\Vert v_{\lambda _n,\beta }\Vert _{b,\lambda _n} =4(m_a+m_b)+o_n(1). \end{aligned}$$

(3.21)

By a similar argument as used in Step 2, we can show that

$$\begin{aligned} \lambda _n\int _{\mathbb {R}^3}a(x)u_{\lambda _n,\beta }^2\mathrm{d}x=\lambda _n\int _{\mathbb {R}^3}b(x) v_{\lambda _n,\beta }^2\mathrm{d}x=o_n(1) \end{aligned}$$

and

$$\begin{aligned}&\int _{\mathbb {R}^3}a_0(x)u_{\lambda _n,\beta }^2\mathrm{d}x=\int _{\mathbb {R}^3}a_0(x)u_{0,\beta }^2\mathrm{d}x +o_n(1)\quad \text {and}\\&\quad \int _{\mathbb {R}^3}b_0(x)v_{\lambda _n,\beta }^2\mathrm{d}x=\int _{\mathbb {R}^3}b_0(x)v_{0,\beta }^2\mathrm{d}x +o_n(1). \end{aligned}$$

These together with Step 2 and (3.21) imply

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla u_{0,\beta }|^2+a_0(x)u_{0,\beta }^2\mathrm{d}x+ \int _{\mathbb {R}^3}|\nabla v_{0,\beta }|^2+b_0(x)v_{0,\beta }^2\mathrm{d}x=4(m_a+m_b). \end{aligned}$$

(3.22)

We claim that

$$\begin{aligned} \int _{\mathbb {R}^3}u_{\lambda _n,\beta }^4\mathrm{d}x\ge C+o_n(1)\quad \text {and}\quad \int _{\mathbb {R}^3}v_{\lambda _n,\beta }^4\mathrm{d}x\ge C+o_n(1). \end{aligned}$$

(3.23)

Indeed, suppose the contrary, we have either $\int _{\mathbb {R}^3}u_{\lambda _n,\beta }^4\mathrm{d}x=o_n(1)$ or $\int _{\mathbb {R}^3}v_{\lambda _n,\beta }^4\mathrm{d}x=o_n(1)$ up to a subsequence. Without loss of generality, we assume $\lim _{n\rightarrow \infty }\int _{\mathbb {R}^3}u_{\lambda _n,\beta }^4\mathrm{d}x=0$. By the boundedness of $\{(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\}$ in E and the Hölder and Sobolev inequalities, $\beta \int _{\mathbb {R}^3}u_{\lambda _n,\beta }^2v_{\lambda _n,\beta }^2=o_n(1)$, which implies $J_{\lambda _n,\beta }(u_{\lambda _n,\beta },v_{\lambda _n,\beta })=I_{b,\lambda _n}(v_{\lambda _n,\beta })+o_n(1)$. By Lemma 2.1 and $\mathcal {N}_{\lambda ,\beta }\subset \mathcal {A}_{\beta }$, for every n, there exists a unique $t^*_n(\beta )>0$ such that $t^*_n(\beta )u_{\lambda _n,\beta }\in \mathcal {N}_{a,\lambda _n}$. It follows from Lemma 3.1 and $\beta <0$ that

$$\begin{aligned} J_{\lambda _n,\beta }(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\ge & {} J_{\lambda _n,\beta }(t^*_n(\beta )u_{\lambda _n,\beta },v_{\lambda _n,\beta })\nonumber \\\ge & {} I_{a,\lambda _n}(t^*_n(\beta )u_{\lambda _n,\beta })+I_{b,\lambda _n}(v_{\lambda _n,\beta })\nonumber \\\ge & {} m_{a,\lambda _n}+I_{b,\lambda _n}(v_{\lambda _n,\beta })\nonumber \\= & {} m_{a,\lambda _n}+J_{\lambda _n,\beta }(u_{\lambda _n,\beta },v_{\lambda _n,\beta })+o_n(1)\nonumber \\= & {} m_{a}+J_{\lambda _n,\beta }(u_{\lambda _n,\beta },v_{\lambda _n,\beta })+o_n(1). \end{aligned}$$

(3.24)

Since $m_{a}>0$, (3.24) is impossible for n large enough. Now, (3.23) together with Steps 1–2 implies

$$\begin{aligned} \int _{\Omega _a}u_{0,\beta }^4\mathrm{d}x\ge C>0\quad \text {and}\int _{\Omega _b}v_{0,\beta }^4\mathrm{d}x\ge C>0. \end{aligned}$$

(3.25)

Note that $I_{\Omega _{a}}'(u_{0,\beta })=0$ in $H^{-1}(\Omega _{a})$ and $I_{\Omega _{b}}'(v_{0,\beta })=0$ in $H^{-1}(\Omega _{b})$, by (3.25), we have

$$\begin{aligned} \int _{\mathbb {R}^3}|\nabla u_{0,\beta }|^2+a_0(x)u_{0,\beta }^2\mathrm{d}x\ge 4m_a\quad \text {and}\quad \int _{\mathbb {R}^3}|\nabla v_{0,\beta }|^2+b_0(x)v_{0,\beta }^2\mathrm{d}x\ge 4m_b. \end{aligned}$$

It follows from (3.22) that $u_{0,\beta }$ is a least energy nonzero critical point of $I_{\Omega _{a}}(u)$ and $v_{0,\beta }$ is a least energy nonzero critical point of $I_{\Omega _{b}}(v)$.

We close the proof of Theorem 1.1 by verifying (1) and (2). Supposing the contrary, there exists $\{\lambda _n\}$ with $\lambda _n\rightarrow +\infty $ as $n\rightarrow \infty $ such that one of the following cases must happen:

(a):: $\int _{\mathbb {R}^3\backslash \Omega _{a}}|\nabla u_{\lambda _n,\beta }|^2+u_{\lambda _n,\beta }^2\mathrm{d}x\ge C+o_n(1)$;
(b):: $\int _{\mathbb {R}^3\backslash \Omega _{b}}|\nabla v_{\lambda _n,\beta }|^2+v_{\lambda _n,\beta }^2\mathrm{d}x\ge C+o_n(1)$;
(c):: $|\int _{\Omega _{a}}|\nabla u_{\lambda _n,\beta }|^2+a_0(x)u_{\lambda _n,\beta }^2\mathrm{d}x-4m_{a}|\ge C+o_n(1)$;
(d):: $|\int _{\Omega _{b}}|\nabla v_{\lambda ,\beta }|^2+b_0(x)v_{\lambda _n,\beta }^2\mathrm{d}x-4m_{b}|\ge C+o_n(1)$.

By Steps 1–3 and the condition $(D_3)$, it is easy to see that (c) and (d) can not hold, which then implies that we must have (a) or (b). Since (3.21) holds for $\{(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\}$, by Steps 2–3 and the condition $(D_3)$, we can see that

$$\begin{aligned}&\int _{\mathbb {R}^3\backslash \Omega _a}|\nabla u_{\lambda _n,\beta }|^2+(\lambda _na(x)+a_0(x))u_{\lambda _n,\beta }^2\mathrm{d}x\\&\quad + \int _{\mathbb {R}^3\backslash \Omega _b}|\nabla v_{\lambda _n,\beta }|^2+(\lambda _nb(x)+b_0(x))v_{\lambda _n,\beta }^2\mathrm{d}x\rightarrow 0 \end{aligned}$$

as $n\rightarrow \infty $. It follows from the conditions $(D_2)$ and $(D_4)$ and Steps 1–2 that

$$\begin{aligned} \int _{(\mathbb {R}^3\backslash \Omega _a)\cap \mathcal {D}_a}a_0^-(x)u_{\lambda _n,\beta }^2\mathrm{d}x =\int _{(\mathbb {R}^3\backslash \Omega _b)\cap \mathcal {D}_b}b_0^-(x)v_{\lambda _n,\beta }^2\mathrm{d}x=o_n(1), \end{aligned}$$

which then together with the conditions $(D_2)$ and $(D_4)$ and Steps 1–2 once more implies

$$\begin{aligned}&\int _{\mathbb {R}^3\backslash \Omega _a}|\nabla u_{\lambda _n,\beta }|^2+(\lambda _na(x)+a_0(x))u_{\lambda _n,\beta }^2\mathrm{d}x\\&\qquad +\int _{\mathbb {R}^3\backslash \Omega _b}|\nabla v_{\lambda _n,\beta }|^2+(\lambda _nb(x)+b_0(x))v_{\lambda _n,\beta }^2\mathrm{d}x\\&\quad \ge \int _{\mathbb {R}^3\backslash \Omega _a}|\nabla u_{\lambda _n,\beta }|^2\mathrm{d}x +\int _{\mathbb {R}^3\backslash \Omega _a}\frac{\lambda _n}{2}a(x)u_{\lambda _n,\beta }^2\mathrm{d}x +\int _{\mathbb {R}^3\backslash \Omega _b}|\nabla v_{\lambda _n,\beta }|^2\mathrm{d}x\\&\qquad +\int _{\mathbb {R}^3\backslash \Omega _b}\frac{\lambda _n}{2}b(x)v_{\lambda _n,\beta }^2\mathrm{d}x+o_n(1)\\&\quad =\int _{\mathbb {R}^3\backslash \Omega _a}|\nabla u_{\lambda _n,\beta }|^2+u_{\lambda _n,\beta }^2\mathrm{d}x +\int _{\mathbb {R}^3\backslash \Omega _b}|\nabla v_{\lambda _n,\beta }|^2+v_{\lambda _n,\beta }^2\mathrm{d}x+o_n(1), \end{aligned}$$

Thus, $\int _{\mathbb {R}^3\backslash \Omega _a}|\nabla u_{\lambda _n,\beta }|^2+u_{\lambda _n,\beta }^2\mathrm{d}x +\int _{\mathbb {R}^3\backslash \Omega _b}|\nabla v_{\lambda _n,\beta }|^2+v_{\lambda _n,\beta }^2\mathrm{d}x\rightarrow 0$ as $n\rightarrow \infty $ and it is a contradiction. We now complete the proof by taking $\Lambda _*=\Lambda _2$. $\square $

4 Multi-bump solutions

The main task in this section is to find multi-bump solutions to $(\mathcal {P}_{\lambda ,\beta })$ described as in Theorem 1.2. For the sake of convenience, in the present section, we always assume the conditions $(D_1){-}(D_2)$, $(D_3')$, $(D_4)$ and $(D_5')$ hold. Due to the conditions $(D_3')$ and $(D_5')$, in the present section, $\Omega _a'$ and $\Omega _b'$ will be chosen as $(I){-}(III)$ given in Sect. 2.

4.1 The penalized functional and the (PS) condition

Since we want to find multi-bump solutions of $(\mathcal {P}_{\lambda ,\beta })$ described as in Theorem 1.2, we will make some modifications on $J_{\lambda ,\beta }(u,v)$. Similar technique was developed by del Pino and Felmer [23] and was also used in several other literatures, see for example [10, 24, 28, 45] and the references therein.

Let $J_a\times J_b$ be a given subset of $\{1,\ldots ,n_a\}\times \{1,\ldots ,n_b\}$ with $J_a\not =\emptyset $ and $J_b\not =\emptyset $. Without loss of generality, we assume $J_a\times J_b=\{1,\ldots ,k_a\}\times \{1,\ldots ,k_b\}$ with $1\le k_a\le n_a$ and $1\le k_b\le n_b$. Denote $\Omega _a^{J_a}=\underset{i_a=1}{\overset{k_a}{\cup }}\Omega _{a,i_a}'$ and $\Omega _b^{J_b}=\underset{j_b=1}{\overset{k_b}{\cup }}\Omega _{b,j_b}'$. We also denote the characteristic functions of $\Omega _a^{J_a}$ and $\Omega _b^{J_b}$ by $\chi _{\Omega _a^{J_a}}$ and $\chi _{\Omega _b^{J_b}}$, respectively. Now, let

$$\begin{aligned} \delta _\beta ^2=\frac{C_{a,b}}{2}\min \bigg \{1,\frac{1}{\mu _1+2|\beta |},\frac{1}{\mu _2+2|\beta |}\bigg \}, \end{aligned}$$

(4.1)

where $C_{a,b}$ is given by Lemma 2.1, and define $f_a(x,t)=\chi _{\Omega _a^{J_a}}(t^+)^3+(1-\chi _{\Omega _a^{J_a}})f(t)$, $f_b(x,t)=\chi _{\Omega _b^{J_b}}(t^+)^3+(1-\chi _{\Omega _b^{J_b}})f(t)$ and $h(x,t,s)=(\chi _{\Omega _b^{J_b}}+\chi _{\Omega _a^{J_a}})t^+s^+ +(1-\chi _{\Omega _a^{J_a}})(1-\chi _{\Omega _b^{J_b}})h(t,s)$, where $t^+=\max \{0, t\}$, $s^+=\max \{0, s\}$,

$$\begin{aligned} f(t)=\left\{ \begin{array}{ll} 0,&{}\quad t\le 0,\\ t^3,&{}\quad 0\le t\le \delta _\beta ,\\ \delta _\beta ^2t,&{}\quad t\ge \delta _\beta , \end{array}\right. \quad \text {and}\quad h(t,s)=\left\{ \begin{array}{ll} 0,&{}\quad \min \{t,s\}\le 0,\\ ts,&{}\quad 0\le t,s\le \delta _\beta ,\\ \delta _\beta t,&{}\quad 0\le t\le \delta _\beta \le s,\\ \delta _\beta s,&{}\quad 0\le s\le \delta _\beta \le t,\\ \delta _\beta ^2,&{}\quad \delta _\beta \le t,s. \end{array}\right. \end{aligned}$$

Then, it is easy to see that $f_a(x,t)$ and $f_b(x,t)$ are the modifications of $t^3$ and h(x, t, s) is the modification of ts. Let us consider the following functional defined on E,

$$\begin{aligned} J_{\lambda ,\beta }^*(u,v)= & {} \frac{1}{2}\Vert u\Vert _{a,\lambda }^2+\frac{1}{2}\Vert v\Vert _{b,\lambda }^2 -\mu _1\int _{\mathbb {R}^3}F_a(x,u)\mathrm{d}x\\&-\mu _2\int _{\mathbb {R}^3}F_b(x,v)\mathrm{d}x-\beta \int _{\mathbb {R}^3}H(x,u,v)\mathrm{d}x, \end{aligned}$$

where $F_a(x,u)=\int _0^uf_a(x,t)\mathrm{d}t$, $F_b(x,v)=\int _0^vf_b(x,t)\mathrm{d}t$ and $H(x,u,v)=2\int _0^u\int _0^vh(x,t,s)dsdt$. Clearly, by the construction of $f_a(x,t)$, $f_b(x,t)$ and h(x, t, s), we can see that

$$\begin{aligned} 0\le & {} \int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}F_a(x,u)\mathrm{d}x\le \frac{\delta _\beta ^2}{2}\Vert u^+\Vert _2^2,\quad 0\le \int _{\mathbb {R}^3\backslash \Omega _b^{J_b}}F_b(x,v)\mathrm{d}x\le \frac{\delta _\beta ^2}{2}\Vert v^+\Vert _2^2,\\ 0\le & {} \int _{\mathbb {R}^3\backslash (\Omega _a^{J_a}\cup \Omega _b^{J_b})}H(x,u,v)\mathrm{d}x\le 2\delta _\beta ^2\Vert u^+\Vert _2\Vert v^+\Vert _2\le \delta _\beta ^2(\Vert u^+\Vert _2^2+\Vert v^+\Vert _2^2). \end{aligned}$$

On the other hand,by Lemma 2.1, we have

$$\begin{aligned} \Vert u^+\Vert _2^2\le C_{a,b}^{-1}\Vert u\Vert _{a,\lambda }^2\le \lambda C_{a,b}^{-1}\Vert u\Vert _a^2\quad \text {for }\lambda \ge \Lambda _1\text { and }u\in E_a \end{aligned}$$

and

$$\begin{aligned} \Vert v^+\Vert _2^2\le C_{a,b}^{-1}\Vert v\Vert _{b,\lambda }^2\le \lambda C_{a,b}^{-1}\Vert v\Vert _b^2\quad \text {for }\lambda \ge \Lambda _1\text { and }v\in E_b. \end{aligned}$$

It follows that $J_{\lambda ,\beta }^*(u,v)$ is well defined on E for $\lambda \ge \Lambda _1$ and $\beta <0$. Moreover, by a standard argument, we can see that for $\lambda \ge \Lambda _1$ and $\beta <0$, $J_{\lambda ,\beta }^*(u,v)$ is $C^1$ on E and the critical point of $J_{\lambda ,\beta }^*(u,v)$ is the solution of the following two-component systems:

In what follows, we will make some investigations on the functional $J_{\lambda ,\beta }^*(u,v)$.

Lemma 4.1

Assume $(u,v)\in E$. Then,

$$\begin{aligned}&\int _{\mathbb {R}^3}\frac{1}{4}f_a(x,u)u-F_a(x,u)\mathrm{d}x\ge -\frac{\delta _\beta ^2}{4}\Vert u^+\Vert _2^2,\\&\qquad \int _{\mathbb {R}^3}\frac{1}{4}f_b(x,v)v-F_b(x,v)\mathrm{d}x\ge -\frac{\delta _\beta ^2}{4}\Vert v^+\Vert _2^2,\\&\quad 0\ge \int _{\mathbb {R}^3}\frac{u}{2}\int _0^vh(x,u,s)\mathrm{d}s+\frac{v}{2}\int _0^uh(x,t,v)\mathrm{d}t-H(x,u,v)\mathrm{d}x\ge -\delta _\beta ^2\Vert u^+v^+\Vert _1. \end{aligned}$$

Proof

By the construction of $f_a(x,t)$, it is easy to see that $\frac{1}{4}f_a(x,t)t-F_a(x,t)=0$ for $x\in \Omega _a^{J_a}$. If $x\not \in \Omega _a^{J_a}$, then by the construction of $f_a(x,t)$, we have

$$\begin{aligned} \frac{1}{4}f_a(x,t)t-F_a(x,t)=\left\{ \begin{array}{ll} 0,&{}\quad t\le \delta _\beta ,\\ \frac{\delta _\beta ^4}{4}-\frac{\delta _\beta ^2}{4}(t^+)^2,&{}\quad \delta _\beta \le t. \end{array}\right. \end{aligned}$$

It follows that for every $u\in E_a$, we have

$$\begin{aligned} \int _{\mathbb {R}^3}\frac{1}{4}f_a(x,u)u-F_a(x,u)\mathrm{d}x= \int _{\{u(x)\ge \delta _\beta \}\cap (\mathbb {R}^3\backslash \Omega _a^{J_a})}\frac{\delta _\beta ^4}{4}-\frac{\delta _\beta ^2}{4}(u^+)^2\mathrm{d}x\ge -\frac{\delta _\beta ^2}{4}\Vert u^+\Vert _2^2. \end{aligned}$$

By a similar argument, for every $v\in E_b$, we have

$$\begin{aligned} \int _{\mathbb {R}^3}\frac{1}{4}f_b(x,v)v-F_b(x,v)\mathrm{d}x\ge -\frac{\delta _\beta ^2}{4}\Vert v^+\Vert _2^2. \end{aligned}$$

On the other hand, since $\Omega _a^{J_a}\cap \Omega _b^{J_b}=\emptyset $, by the construction of h(x, t, s), we can see that $\frac{t}{2}\int _0^sh(x,t,\tau )\mathrm{d}\tau +\frac{s}{2}\int _0^th(x,\tau ,s)\mathrm{d}\tau -H(x,t,s)=0$ for $x\in \Omega _a^{J_a}\cup \Omega _{b}^{J_b}$. If $x\not \in \Omega _a^{J_a}\cup \Omega _{b}^{J_b}$, then also by the construction of h(x, t, s), we have

$$\begin{aligned}&\frac{t}{2}\int _0^sh(x,t,\tau )\mathrm{d}\tau +\frac{s}{2}\int _0^th(x,\tau ,s)\mathrm{d}\tau -H(x,t,s)\\&\quad =\left\{ \begin{array}{ll} 0,&{}\quad t,s\le \delta _\beta ,\\ \frac{(t^+)^2\delta _\beta }{4}(\delta _\beta -s), &{}\quad t\le \delta _\beta \le s,\\ \frac{(s^+)^2\delta _\beta }{4}(\delta _\beta -t),&{}\quad s\le \delta _\beta \le t,\\ \frac{\delta _\beta ^2}{4}[(t-\delta _\beta )(\delta _\beta -2s)+(s-\delta _\beta )(\delta _\beta -2t)], &{}\quad \delta _\beta \le t, s. \end{array}\right. \end{aligned}$$

It follows that for every $(u,v)\in E$, we have

$$\begin{aligned} 0\ge \int _{\mathbb {R}^3}\frac{u}{2}\int _0^vh(x,u,s)\mathrm{d}s+\frac{v}{2}\int _0^uh(x,t,v)\mathrm{d}t-H(x,u,v)\mathrm{d}x\ge -\delta _\beta ^2\Vert u^+v^+\Vert _1, \end{aligned}$$

which completes the proof. $\square $

With Lemma 4.1 in hands, we can verify that $J_{\lambda ,\beta }^*(u,v)$ actually satisfies the $(\textit{PS})$ condition for $\lambda \ge \Lambda _1$ and $\beta <0$.

Lemma 4.2

Assume $\lambda \ge \Lambda _1$ and $\beta <0$. Then, $J_{\lambda ,\beta }^*(u,v)$ satisfies the $(\textit{PS})_c$ condition for all $c\in \mathbb {R}$, that is, every $\{(u_n,v_n)\}\subset E$ satisfying $J_{\lambda ,\beta }^*(u_n,v_n)=c+o_n(1)$ and $D[J_{\lambda ,\beta }^*(u_n,v_n)]=o_n(1)$ strongly in $E^*$ has a strongly convergent subsequence in E.

Proof

Suppose $\{(u_n,v_n)\}\subset E$ satisfying $J_{\lambda , \beta }^*(u_n,v_n)=c+o_n(1)$ and $D[J_{\lambda ,\beta }^*(u_n,v_n)]=o_n(1)$ strongly in $E^*$. Then, by $\beta <0$, Lemmas 2.1 and 4.1 and a similar argument of (3.8), we have

$$\begin{aligned} c+o_n(1)+o_n(1)\Vert (u_n,v_n)\Vert \ge \frac{1}{4}(1-\mu _1\delta _\beta ^2C_{a,b}^{-1})\Vert u_n\Vert ^2_{a,\lambda }+\frac{1}{4}(1-\mu _2\delta _\beta ^2C_{a,b}^{-1})\Vert v_n\Vert ^2_{b,\lambda }.\nonumber \\ \end{aligned}$$

(4.2)

It follows from Lemma 2.1 and (4.1) that $c+o_n(1)+o_n(1)\Vert (u_n,v_n)\Vert \ge \frac{C_{a,b}}{8}(\Vert u_n\Vert _2^2+\Vert v_n\Vert _2^2)$. This together with Lemma 2.1 and the condition $(D_4)$ implies

$$\begin{aligned} c+o_n(1)+o_n(1)\Vert (u_n,v_n)\Vert\ge & {} \frac{1}{4}(1-\mu _1\delta _\beta ^2C_{a,b}^{-1})\Vert u_n\Vert ^2_{a,\lambda } +\frac{1}{4}(1-\mu _2\delta _\beta ^2C_{a,b}^{-1})\Vert v_n\Vert ^2_{b,\lambda }\\\ge & {} \frac{1}{4}(1-\mu _1\delta _\beta ^2C_{a,b}^{-1})\Vert u_n\Vert _a^2 +\frac{1}{4}(1-\mu _2\delta _\beta ^2C_{a,b}^{-1})\Vert v_n\Vert _b^2\\&-\,C(\Vert u_n\Vert _2^2+\Vert v_n\Vert _2^2)\\\ge & {} \frac{1}{8}\Vert (u_n,v_n)\Vert ^2-C'(c+o_n(1)+o_n(1)\Vert (u_n,v_n)\Vert ), \end{aligned}$$

since $\lambda \ge \Lambda _1$. Thus, $\{(u_n,v_n)\}$ is bounded in E and $(u_n,v_n)\rightharpoonup (u_0,v_0)$ weakly in E as $n\rightarrow \infty $ for some $(u_0,v_0)\in E$ up to a subsequence. Without loss of generality, we assume $(u_n,v_n)\rightharpoonup (u_0,v_0)$ weakly in E as $n\rightarrow \infty $. Since $D[J_{\lambda ,\beta }^*(u_n,v_n)]=o_n(1)$ strongly in $E^*$, it is easy to see that $D[J_{\lambda ,\beta }^*(u_0,v_0)]=0$ in $E^*$, which implies

$$\begin{aligned} o_n(1)= & {} \langle D[J_{\lambda ,\beta }^*(u_n,v_n)]-D[J_{\lambda ,\beta }^*(u_0,v_0)], (u_n,v_n)-(u_0,v_0)\rangle _{E^*,E}\nonumber \\= & {} \Vert u_n-u_0\Vert _{a,\lambda }^2+\Vert v_n-v_0\Vert _{b,\lambda }^2-\mu _1\int _{\mathbb {R}^3}(f_a(x,u_n)-f_a(x,u_0))(u_n-u_0)\mathrm{d}x\nonumber \\&-\,\mu _2\int _{\mathbb {R}^3}(f_b(x,v_n)-f_b(x,v_0))(v_n-v_0)\mathrm{d}x\nonumber \\&-\,2\beta \int _{\mathbb {R}^3}(\int _0^{v_n}h(x,u_n,s)\mathrm{d}s-\int _0^{v_0}h(x,u_0,s)\mathrm{d}s)(u_n-u_0)\mathrm{d}x\nonumber \\&-\,2\beta \int _{\mathbb {R}^3}(\int _0^{u_n}h(x,t,v_n)\mathrm{d}t-\int _0^{u_0}h(x,t,v_0)\mathrm{d}t)(v_n-v_0)\mathrm{d}x. \end{aligned}$$

(4.3)

By the construction of $f_a(x,t)$, we can see that

$$\begin{aligned}&|\int _{\mathbb {R}^3}(f_a(x,u_n)-f_a(x,u_0))(u_n-u_0)\mathrm{d}x|\\&\quad \le \int _{\Omega _a^{J_a}}|(f_a(x,u_n)-f_a(x,u_0))(u_n-u_0)|\mathrm{d}x\\&\qquad +\int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}|(f_a(x,u_n)-f_a(x,u_0))(u_n-u_0)|\mathrm{d}x\\&\quad \le \int _{\Omega _a^{J_a}}(|u_n|^3+|u_0|^3)|u_n-u_0|\mathrm{d}x+2\delta _\beta ^2\int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}|u_0||u_n-u_0|\mathrm{d}x+ \delta _\beta ^2\Vert u_n-u_0\Vert _2^2. \end{aligned}$$

Since $(u_n,v_n)\rightharpoonup (u_0,v_0)$ weakly in E as $n\rightarrow \infty $, by (2.1) and the Sobolev embedding theorem, we have

$$\begin{aligned} u_n\rightarrow u_0 \text { strongly in }L^p_{loc}(\mathbb {R}^3)\text { as }n\rightarrow \infty \quad \text { for }p\in [1, 6). \end{aligned}$$

(4.4)

Thus, $\int _{\Omega _a^{J_a}}(|u_n|^3+|u_0|^3)|u_n-u_0|\mathrm{d}x=o_n(1)$ due to the choice of $\Omega _a^{J_a}$ and the Hölder inequality. On the other hand, we also see from (4.4) and the Hölder inequality that $2\delta _\beta ^2\int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}|u_0||u_n-u_0|\mathrm{d}x=o_n(1)$. Hence, we have

$$\begin{aligned} |\int _{\mathbb {R}^3}(f_a(x,u_n)-f_a(x,u_0))(u_n-u_0)\mathrm{d}x|\le \delta _\beta ^2\Vert u_n-u_0\Vert _2^2+o_n(1). \end{aligned}$$

(4.5)

Since (2.2) holds, we can also obtain the following estimates in a similar way:

$$\begin{aligned} |\int _{\mathbb {R}^3}(f_b(x,v_n)-f_b(x,v_0))(v_n-v_0)\mathrm{d}x|\le \delta _\beta ^2\Vert v_n-v_0\Vert _2^2+o_n(1). \end{aligned}$$

(4.6)

On the other hand, by the construction of h(x, t, s), we can see that

$$\begin{aligned}&|\int _{\mathbb {R}^3}\left( \int _0^{v_n}h(x,u_n,s)\mathrm{d}s-\int _0^{v_0}h(x,u_0,s)\mathrm{d}s\right) (u_n-u_0)\mathrm{d}x|\nonumber \\&\quad \le \delta _\beta ^2\int _{\mathbb {R}^3}|u_n-u_0||v_n-v_0|\mathrm{d}x+2\delta _\beta ^2\int _{\mathbb {R}^3}|v_0||u_n-u_0|\mathrm{d}x+o_n(1) \end{aligned}$$

(4.7)

and

$$\begin{aligned}&|\int _{\mathbb {R}^3}\left( \int _0^{u_n}h(x,t,v_n)\mathrm{d}t-\int _0^{u_0}h(x,t,v_0)\mathrm{d}t\right) (v_n-v_0)\mathrm{d}x|\nonumber \\&\quad \le \delta _\beta ^2\int _{\mathbb {R}^3}|u_n-u_0||v_n-v_0|\mathrm{d}x+2\delta _\beta ^2\int _{\mathbb {R}^3}|u_0||v_n-v_0|\mathrm{d}x+o_n(1). \end{aligned}$$

(4.8)

By using similar arguments of (4.5) and (4.6), we can see from (4.7) and (4.8) that

(4.9)

and

(4.10)

Combining (4.3), (4.5)–(4.6) and (4.9)–(4.10), we can conclude that

$$\begin{aligned} o_n(1)\ge \Vert u_n-u_0\Vert _{a,\lambda }^2+\Vert v_n-v_0\Vert _{b,\lambda }^2 -\delta _\beta ^2(\mu _1+2|\beta |)\Vert u_n-u_0\Vert _2^2-\delta _\beta ^2 (\mu _2+2|\beta |)\Vert v_n-v_0\Vert _2^2. \end{aligned}$$

(4.11)

It follows from Lemma 2.1 and (4.1) that $o_n(1)\ge \frac{C_{a,b}}{2}(\Vert u_n-u_0\Vert _2^2+\Vert v_n-v_0\Vert _2^2)$, which implies $u_n\rightarrow u_0$ and $v_n\rightarrow v_0$ strongly in $L^2(\mathbb {R}^3)$ as $n\rightarrow \infty $. This together the condition $(D_4)$, implies

$$\begin{aligned} o_n(1)\ge \Vert u_n-u_0\Vert _{a,\lambda }^2+\Vert v_n-v_0\Vert _{b,\lambda }^2\ge \Vert u_n-u_0\Vert _a^2+\Vert v_n-v_0\Vert _b^2+o_n(1) \end{aligned}$$

for $\lambda \ge \Lambda _1$ and $\beta <0$. Thus, $(u_n,v_n)\rightarrow (u_0,v_0)$ strongly in E as $n\rightarrow \infty $ for $\lambda \ge \Lambda _1$ and $\beta <0$, which completes the proof. $\square $

In the final of this section, we will show that $J_{\lambda ,\beta }^*(u,v)$ is actually a penalized functional of $J_{\lambda ,\beta }(u,v)$ in the sense that, some special critical points of $J_{\lambda ,\beta }^*(u,v)$ are also critical points of $J_{\lambda ,\beta }(u,v)$.

Lemma 4.3

Assume $\lambda \ge \Lambda _1$ and $\beta <0$. Let $M>0$ be a constant and $(u_{\lambda ,\beta },v_{\lambda ,\beta })\in E$ satisfy $J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })\le M$ and $D[J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$. Then,

(1)
There exists $M_1>0$ such that $\Vert (u_{\lambda ,\beta },v_{\lambda ,\beta })\Vert \le M_1$.
(2)
$\int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0$ and $\int _{\mathbb {R}^3\backslash \Omega _b^{J_b}}|\nabla v_{\lambda ,\beta }|^2+(\lambda b(x)+b_0(x))v_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0$ as $\lambda \rightarrow +\infty $.
(3)
There exists $\Lambda _1^*(\beta ,M)\ge \Lambda _1$ such that $|u_{\lambda ,\beta }|\le \delta _\beta $ on $\mathbb {R}^3\backslash \Omega _a^{J_a}$ and $|v_{\lambda ,\beta }|\le \delta _\beta $ on $\mathbb {R}^3\backslash \Omega _b^{J_b}$ for $\lambda \ge \Lambda _1^*(\beta ,M)$.

Proof

(1) Since $\lambda \ge \Lambda _1$ and $\beta <0$, by a similar argument as (4.2), we can conclude that

$$\begin{aligned} M\ge \frac{1}{4}(1-\mu _1\delta _\beta ^2C_{a,b}^{-1})\Vert u_{\lambda ,\beta }\Vert _{a,\lambda }^2 +\frac{1}{4}(1-\mu _2\delta _\beta ^2C_{a,b}^{-1})\Vert v_{\lambda ,\beta }\Vert _{b,\lambda }^2. \end{aligned}$$

(4.12)

It follows from Lemma 2.1 and (4.1) that $8MC_{a,b}^{-1}\ge \Vert u_{\lambda ,\beta }\Vert _2^2+\Vert v_{\lambda ,\beta }\Vert _2^2$. Now, applying the condition $(D_4)$, we can see that

$$\begin{aligned} 8M+8MC_{a,b}^{-1}(C_{a,0}+d_a+C_{b,0}+d_b)\ge \Vert (u_{\lambda ,\beta }, v_{\lambda ,\beta })\Vert ^2. \end{aligned}$$

(4.13)

We complete this proof by taking $M_1=8M+8MC_{a,b}^{-1}(C_{a,0}+d_a+C_{b,0}+d_b)$.

(2) Since $\mathbb {R}^3\backslash \Omega _a^{J_a}=(\mathbb {R}^3\backslash \Omega _a')\cup (\underset{i_a=k_a+1}{\overset{n_a}{\cup }}\Omega _{a,i_a}')$ and $\mathbb {R}^3\backslash \Omega _b^{J_b}=(\mathbb {R}^3\backslash \Omega _b')\cup (\underset{j_b=k_b+1}{\overset{n_b}{\cup }}\Omega _{b,j_b}')$, for the sake of clarity, we divide this proof into the following two steps.

Step 1 We prove that

$$\begin{aligned} \lim _{\lambda \rightarrow +\infty }\int _{\mathbb {R}^3\backslash \Omega _a'}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2 \mathrm{d}x=0 \end{aligned}$$

(4.14)

and

(4.15)

Indeed, let $\{\Omega _{a,i_a}''\}$ be a sequence of bounded domains with smooth boundaries in $\mathbb {R}^3$ and satisfy

(a):: $\Omega _{a,i_a}\subset \Omega _{a,i_a}''\subset \Omega _{a,i_a}'$ for all $i_a=1,\ldots ,n_a$.
(b):: $dist(\Omega _{a,i_a}'', \mathbb {R}^3\backslash \Omega _{a,i_a}')>0$ and $dist(\mathbb {R}^3\backslash \Omega _{a,i_a}'',\Omega _{a,i_a})>0$ for all $i_a=1,\ldots ,n_a$.

Denote $\Omega _a''=\underset{i_a=1}{\overset{n_a}{\cup }}\Omega _{a,i_a}''$. Then, by a similar argument as (2.7), we can show that

(4.16)

for $\lambda $ large enough. Without loss of generality, we assume (4.16) holds for $\lambda \ge \Lambda _1$. Since Lemma 2.1 and (4.12) hold, we can obtain $8((C_{a,0}+d_a)C_{a,b}^{-1}+1)M\ge \lambda \int _{\mathbb {R}^3\backslash \Omega _a''}a(x)u_{\lambda ,\beta }^2\mathrm{d}x$ for $\lambda \ge \Lambda _1$ by the condition $(D_4)$ and (4.16). Since the condition $(D_3')$ is contained in the condition $(D_3)$, by a similar argument as (3.15), we can see that

$$\begin{aligned} \int _{\mathbb {R}^3\backslash \Omega _a''}u_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0 \quad \text { as }\lambda \rightarrow +\infty . \end{aligned}$$

(4.17)

Let $\Psi \in C^\infty (\mathbb {R}^3)$ be given by (3.16). Then, $u_{\lambda ,\beta }\Psi \in E_a$. Note $D[J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$, we get that

$$\begin{aligned}&\int _{\mathbb {R}^3}(|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2)\Psi \mathrm{d}x+\int _{\mathbb {R}^3}(\nabla u_{\lambda ,\beta }\nabla \Psi ) u_{\lambda ,\beta } \mathrm{d}x\\&\quad =\mu _1\int _{\mathbb {R}^3}f_a(x,u_{\lambda ,\beta })u_{\lambda ,\beta }\Psi \mathrm{d}x+2\beta \int _{\mathbb {R}^3}(\int _0^{v_{\lambda ,\beta }}h(x,u_{\lambda ,\beta },s)\mathrm{d}s)u_{\lambda ,\beta }\Psi \mathrm{d}x. \end{aligned}$$

Since $\beta <0$, by (2.8) and the construction of $f_a(x,t)$ and h(x, t, s), we have

$$\begin{aligned} 0\le & {} \int _{\mathbb {R}^3\backslash \Omega _a'}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2 \mathrm{d}x\nonumber \\\le & {} \mu _1\int _{\Omega _a^{J_a}\cap (\mathbb {R}^3\backslash \Omega _a'')}u_{\lambda ,\beta }^4\mathrm{d}x +\,\mu _1\delta _\beta ^2\int _{(\mathbb {R}^3\backslash \Omega _a^{J_a})\cap (\mathbb {R}^3\backslash \Omega _a'')}u_{\lambda ,\beta }^2\mathrm{d}x\nonumber \\&+\,\int _{\Omega _a'\backslash \Omega _a''}|\nabla u_{\lambda ,\beta }||\nabla \Psi ||u_{\lambda ,\beta }|\mathrm{d}x\nonumber \\\le & {} \mu _1\delta _\beta ^2\int _{\mathbb {R}^3\backslash \Omega _a''}u_{\lambda ,\beta }^2\mathrm{d}x +\,(\mu _1S^{-\frac{3}{2}}\Vert u_{\lambda ,\beta }\Vert _a^3+\max _{\mathbb {R}^3}|\nabla \Psi |\Vert u_{\lambda ,\beta }\Vert _a)\bigg (\int _{\mathbb {R}^3\backslash \Omega _a''}u_{\lambda ,\beta }^2\mathrm{d}x\bigg )^{\frac{1}{2}}.\nonumber \\ \end{aligned}$$

(4.18)

Thanks to (4.13) and (4.17), we know from (4.18) that (4.14) holds. By a similar argument, we can also conclude that (4.15) is true.

Step 2 We prove that

$$\begin{aligned} \lim _{\lambda \rightarrow +\infty }\int _{\Omega _{a,i_a}'}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2 \mathrm{d}x=0\quad \text {for all }i_a\in \{1,\ldots ,n_a\}\backslash J_a \end{aligned}$$

(4.19)

and

$$\begin{aligned} \lim _{\lambda \rightarrow +\infty }\int _{\Omega _{b,j_b}'}|\nabla v_{\lambda ,\beta }|^2+(\lambda b(x)+b_0(x))v_{\lambda ,\beta }^2 \mathrm{d}x=0\quad \text {for all }j_b\in \{1,\ldots ,n_b\}\backslash J_b. \end{aligned}$$

(4.20)

In fact, let $\{\Omega _{a,i_a}'''\}$ be a sequence of bounded domains with smooth boundaries in $\mathbb {R}^3$ and satisfy

(i):: $\Omega _{a,i_a}'\subset \Omega _{a,i_a}'''$ and dist$(\Omega _{a,i_a}', \mathbb {R}^3\backslash \Omega _{a,i_a}''')>0$ for all $i_a\in \{1,\ldots ,n_a\}$.
(ii):: $\overline{\Omega _{a,i_a}'''}\cap \overline{\Omega _{a,j_a}'''}=\emptyset $ for all $i_a\not =j_a$.
(iii):: $(\overline{\underset{i_a=1}{\overset{n_a}{\cup }}\Omega _{a,i_a}'''})\cap \overline{\Omega _b'}=\emptyset $.

For every $i_a\in \{1,\ldots ,n_a\}\backslash J_a$, we choose $\Psi _{i_a}\in C^\infty (\mathbb {R}^3, [0, 1])$ satisfying

$$\begin{aligned} \Psi _{i_a}=\left\{ \begin{array}{ll} 1,&{}\quad x\in \Omega _{a,i_a}',\\ 0,&{}\quad x\in \mathbb {R}^3\backslash \Omega _{a,i_a}'''.\end{array}\right. \end{aligned}$$

Then, by a similar argument as (4.18), the choice of $\Omega _{a,i_a}'''$ and the construction of $f_{a}(x,t)$ and h(x, t, s), we can obtain that

$$\begin{aligned}&\int _{\Omega _{a,i_a}'}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2 \mathrm{d}x\\&\quad \le \mu _1\delta _\beta ^2\int _{\Omega _{a,i_a}'''}u_{\lambda ,\beta }^2\mathrm{d}x +\int _{\Omega _{a,i_a}'''\backslash \Omega _{a,i_a}'}|\nabla u_{\lambda ,\beta }||\nabla \Psi _{i_a}||u_{\lambda ,\beta }|\mathrm{d}x. \end{aligned}$$

Thanks to the choice of $\Omega _{a,i_a}'''$ and (4.13), for $i_a\in \{1,\ldots ,n_a\}\backslash J_a$, we have

$$\begin{aligned}&\int _{\Omega _{a,i_a}'}|\nabla u_{\lambda ,\beta }|^2+(\lambda a(x)+a_0(x))u_{\lambda ,\beta }^2 \mathrm{d}x\nonumber \\&\quad \le \mu _1\delta _\beta ^2\int _{\Omega _{a,i_a}'}u_{\lambda ,\beta }^2\mathrm{d}x +C\bigg (\int _{\mathbb {R}^3\backslash \Omega _a''}u_{\lambda ,\beta }^2\mathrm{d}x\bigg )^{\frac{1}{2}}. \end{aligned}$$

It follows from (2.9), (4.1) and (4.17) that (4.19) holds. A similar argument implies that (4.20) holds too. Now, the conclusion follows immediately from (4.14)–(4.15) and (4.19)–(4.20).

(3) By (2.9) and (4.19), we have

$$\begin{aligned} \int _{\Omega _{a,i_a}'}u_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0\quad \text {as }\lambda \rightarrow +\infty \quad \text {for } i_a\in \{1,\ldots ,n_a\}\backslash J_a, \end{aligned}$$

which together with (4.17) implies

$$\begin{aligned} \int _{\mathbb {R}^3\backslash \underset{i_a\in J_a}{\cup }\Omega _{a,i_a}''}u_{\lambda ,\beta }^2\mathrm{d}x\rightarrow 0 \quad \text { as }\lambda \rightarrow +\infty . \end{aligned}$$

(4.21)

Let $r=\frac{1}{3}dist(\Omega _a',\Omega _a'')$. Then, for every $x\in \mathbb {R}^3\backslash \Omega _a^{J_a}$, $B_{2r}(x)\subset \mathbb {R}^3\backslash \underset{i_a\in J_a}{\cup }\Omega _{a,i_a}''$. We define $\phi _L=\min \{|u_{\lambda ,\beta }|^{\alpha -1}, L\}u_{\lambda ,\beta }\overline{\rho }^2$, where $\overline{\rho }\in C_0^\infty (B_{2r}(x), [0, 1])$ with $\overline{\rho }=1$ on $B_{\frac{5r}{3}}(x)$ and $|\nabla \overline{\rho }|<\frac{C}{2r-\frac{5}{3}r}$, $\alpha >0$ and $L>0$. Then, $\phi _L\in E_a$. Since $D[J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$ and the conditions $(D_1)$ and $(D_4)$ hold, by multiplying $(\mathcal {P}_{\lambda ,\beta }^*)$ with $(\phi _L, 0)$ and letting $L\rightarrow +\infty $, we have

$$\begin{aligned} \frac{1}{2}\int _{\mathbb {R}^3}\overline{\rho }^2|u_{\lambda ,\beta }|^{\alpha -1}|\nabla u_{\lambda ,\beta }|^2\mathrm{d}x\le & {} C_\beta \bigg (\int _{\mathbb {R}^3}\overline{\rho }^2u_{\lambda ,\beta }^{\alpha +3}\mathrm{d}x+\int _{\mathbb {R}^3}\overline{\rho }^2u_{\lambda ,\beta }^{\alpha +1}\mathrm{d}x\bigg )\\&+\,4\int _{\mathbb {R}^3}|\nabla \overline{\rho }|^2u_{\lambda ,\beta }^{\alpha +1}\mathrm{d}x, \end{aligned}$$

where $C_\beta =\mu _1+\mu _1\delta _\beta ^2+C_{a,0}+d_a$. By the Sobolev embedding theorem, we can see that

$$\begin{aligned} \bigg (\int _{B_{\frac{5r}{3}}(x)}u_{\lambda ,\beta }^{3(\alpha +1)}\mathrm{d}x\bigg )^{\frac{1}{3}}\le C_\beta (\alpha +1)^2\bigg (\int _{B_{2r}(x)}u_{\lambda ,\beta }^{\alpha +3}\mathrm{d}x +(1+\frac{24}{r^2})\int _{B_{2r}(x)}u_{\lambda ,\beta }^{\alpha +1}\mathrm{d}x\bigg ).\nonumber \\ \end{aligned}$$

(4.22)

Let $\alpha _n=3\alpha _{n-1}$ with $\alpha _0=2$ and $r_n=(1+(\frac{2}{3})^{n-1})r$, $n\in \mathbb {N}$. Then, (4.22) can be rewritten as

$$\begin{aligned}&\bigg (\int _{B_{r_1}(x)}u_{\lambda ,\beta }^{3(\alpha _0+1)}\mathrm{d}x\bigg )^{\frac{1}{3}}\nonumber \\&\quad \le C_\beta (\alpha _0+1)^2\bigg (\int _{B_{r_0}(x)}u_{\lambda ,\beta }^{\alpha _0+3}\mathrm{d}x +(1+\frac{4}{|r_0-r_1|^2})\int _{B_{r_0}(x)}u_{\lambda ,\beta }^{\alpha _0+1}\mathrm{d}x\bigg ). \end{aligned}$$

(4.23)

We replace $\alpha _0$, $r_0$ and $r_1$ in (4.23) by $\alpha _n$, $r_n$ and $r_{n+1}$. Then, we can obtain

$$\begin{aligned}&\bigg (\int _{B_{r_{n+1}}(x)}u_{\lambda ,\beta }^{3(\alpha _n+1)}\mathrm{d}x\bigg )^{\frac{1}{3(\alpha _n+1)}}\nonumber \\&\quad \le [C_\beta (\alpha _n+1)^2]^{\frac{1}{(\alpha _n+1)}}\bigg (\int _{B_{r_n}(x)}u_{\lambda ,\beta }^{\alpha _n+3}\mathrm{d}x\bigg )^{\frac{1}{(\alpha _n+1)}}\nonumber \\&\qquad +\,[C_\beta (\alpha _n+1)^2]^{\frac{1}{(\alpha _n+1)}}(1+\frac{4}{|r_n-r_{n+1}|^2})^{\frac{1}{(\alpha _n+1)}} \bigg (\int _{B_{r_n}(x)}u_{\lambda ,\beta }^{\alpha _n+1}\mathrm{d}x\bigg )^{\frac{1}{(\alpha _n+1)}}.\qquad \quad \end{aligned}$$

(4.24)

Clearly, one of the following two cases must happen:

(1)
$\int _{B_{r_n}(x)}u_{\lambda ,\beta }^{\alpha _{n}+1}\mathrm{d}x\le \int _{B_{r_n}(x)}u_{\lambda ,\beta }^{\alpha _{n}+3}\mathrm{d}x$ up to a subsequence.
(2)
$\int _{B_{r_n}(x)}u_{\lambda ,\beta }^{\alpha _{n}+3}\mathrm{d}x\le \int _{B_{r_n}(x)}u_{\lambda ,\beta }^{\alpha _{n}+1}\mathrm{d}x$ up to a subsequence.

If case (1) happens, then by (4.24), we can see that

$$\begin{aligned}&\bigg (\int _{B_{r_{n+1}}(x)}u_{\lambda ,\beta }^{3(\alpha _{n}+1)}\bigg )^{\frac{1}{3(\alpha _{n}+1)}}\\&\quad \le \bigg (4C_\beta (\alpha _{n}+1)^2(1+\frac{1}{|r_n-r_{n+1}|^2})\bigg )^{\frac{1}{(\alpha _{n}+1)}} \bigg (\int _{B_{r_{n}}(x)}u_{\lambda ,\beta }^{\alpha _{n}+3}\bigg )^{\frac{1}{\alpha _{n}+1}}. \end{aligned}$$

By iterating (4.24) and using the choice of $r_n$ and $\alpha _n$, we have

$$\begin{aligned}&\lim _{n\rightarrow +\infty }\bigg (\int _{B_{r_{n+1}}(x)}u_{\lambda ,\beta }^{3(\alpha _{n}+1)}\bigg )^{\frac{1}{3(\alpha _{n}+1)}}\nonumber \\&\quad \le \bigg (\prod _{n=1}^{\infty }\bigg (4C_\beta (\alpha _{n}+1)^2(1+\frac{1}{|r_n-r_{n+1}|^2})\bigg )^{\frac{1}{(\alpha _{n}+1)}} \bigg (\int _{\mathbb {R}^3\backslash \underset{i_a\in J_a}{\cup }\Omega _{a,i_a}''}u_{\lambda ,\beta }^5\mathrm{d}x\bigg )^{\frac{1}{5}}\bigg ) ^{\underset{n=1}{\overset{+\infty }{\prod }}\frac{\alpha _n+3}{\alpha _n+1}}\nonumber \\&\quad \le C_\beta '\bigg (\int _{\mathbb {R}^3\backslash \underset{i_a\in J_a}{\cup }\Omega _{a,i_a}''}u_{\lambda ,\beta }^5\mathrm{d}x\bigg )^{\frac{C}{5}}, \end{aligned}$$

(4.25)

where $C_\beta '$ is a constant independent of $\lambda $ and x. If case (2) happens, then by iterating (4.24) and using the choice of $r_n$ and $\alpha _n$ once more, we have

$$\begin{aligned}&\lim _{n\rightarrow +\infty }\bigg (\int _{B_{r_{n+1}}(x)}u_{\lambda ,\beta }^{3(\alpha _{n}+1)}\bigg )^{\frac{1}{3(\alpha _{n}+1)}}\nonumber \\&\le \prod _{n=1}^{\infty }\bigg (4C_\beta (\alpha _{n}+1)^2(1+\frac{1}{|r_n-r_{n+1}|^2})|B_{r_n}(x)|^{\frac{2}{\alpha _{n}+3}}\bigg )^{\frac{1}{(\alpha _{n}+1)}} \bigg (\int _{\mathbb {R}^3\backslash \underset{i_a\in J_a}{\cup }\Omega _{a,i_a}''}u_{\lambda ,\beta }^5\mathrm{d}x\bigg )^{\frac{1}{5}}\nonumber \\&\le C_\beta '\bigg (\int _{\mathbb {R}^3\backslash \underset{i_a\in J_a}{\cup }\Omega _{a,i_a}''}u_{\lambda ,\beta }^5\mathrm{d}x\bigg )^{\frac{1}{5}}. \end{aligned}$$

(4.26)

By the Hölder and the Sobolev inequalities and (4.13) and (4.21), we can conclude that

$$\begin{aligned} \bigg (\int _{\mathbb {R}^3\backslash \underset{i_a\in J_a}{\cup }\Omega _{a,i_a}''}u_{\lambda ,\beta }^5\mathrm{d}x\bigg )^{\frac{1}{5}}\rightarrow 0\quad \text {as }\lambda \rightarrow +\infty . \end{aligned}$$

It follows from (4.25) and (4.26) that $\Vert u_{\lambda ,\beta }\Vert _{L^\infty (B_r(x))}\rightarrow 0$ as $\lambda \rightarrow +\infty $, which then implies $\Vert u_{\lambda ,\beta }\Vert _{L^\infty (\mathbb {R}^3\backslash \Omega _{a}^{J_a})}\rightarrow 0$ as $\lambda \rightarrow +\infty $. By similar arguments, we also have $\Vert v_{\lambda ,\beta }\Vert _{L^\infty (\mathbb {R}^3\backslash \Omega _{b}^{J_b})}\rightarrow 0$ as $\lambda \rightarrow +\infty $. Now, we can choose $\Lambda _1^*(\beta ,M)\ge \Lambda _1$ such that $|u_{\lambda ,\beta }|\le \delta _\beta $ a.e. on $\mathbb {R}^3\backslash \Omega _a^{J_a}$ and $|v_{\lambda ,\beta }|\le \delta _\beta $ a.e. on $\mathbb {R}^3\backslash \Omega _b^{J_b}$ for $\lambda \ge \Lambda _1^*(\beta )$. Note that by a similar argument as used in Theorem 1.1, we can see that $(u_{\lambda ,\beta }, v_{\lambda ,\beta })\in C(\mathbb {R}^3)\times C(\mathbb {R}^3)$. Hence, we must have $|u_{\lambda ,\beta }|\le \delta _\beta $ on $\mathbb {R}^3\backslash \Omega _a^{J_a}$ and $|v_{\lambda ,\beta }|\le \delta _\beta $ on $\mathbb {R}^3\backslash \Omega _b^{J_b}$ for $\lambda \ge \Lambda _1^*(\beta ,M)$. $\square $

4.2 Construction of critical points

In this section, we will construct critical values of $J_{\lambda ,\beta }^*(u,v)$ by a minimax argument. The idea of such a construction traces back to Séré [41] and also was applied in [10, 24, 28, 45].

We first recall some well-known results, which are useful in this construction. For all $i_a=1,\ldots ,n_a$ and $j_b=1,\ldots ,n_b$, we define $\mathcal {E}_{\Omega _{a,i_a}'}$ on $H^1(\Omega _{a,i_a}')$ and $\mathcal {E}_{\Omega _{b,j_b}'}$ on $H^1(\Omega _{b,j_b}')$ as follows:

$$\begin{aligned} \mathcal {E}_{\Omega _{a,i_a}'}(u)= & {} \frac{1}{2}\int _{\Omega _{a,i_a}'}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x-\frac{\mu _1}{4}\int _{\Omega _{a,i_a}'}u^4\mathrm{d}x,\\ \mathcal {E}_{\Omega _{b,j_b}'}(v)= & {} \frac{1}{2}\int _{\Omega _{b,j_b}'}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2\mathrm{d}x-\frac{\mu _2}{4}\int _{\Omega _{b,j_b}'}v^4\mathrm{d}x. \end{aligned}$$

By (2.9) and (2.10), $\mathcal {E}_{\Omega _{a,i_a}'}(u)$ and $\mathcal {E}_{\Omega _{b,j_b}'}(v)$ have the least energy nonzero critical point for all $i_a=1,\ldots ,n_a$ and $j_b=1,\ldots ,n_b$ if $\lambda \ge \Lambda _1$. We denote the ground state level of $\mathcal {E}_{\Omega _{a,i_a}'}(u)$ and $\mathcal {E}_{\Omega _{b,j_b}'}(v)$ by $m_{a,i_a,\lambda }$ and $m_{b,j_b,\lambda }$, respectively. Since $\{\Omega _{a,i_a}'\}$ and $\{\Omega _{b,j_b}'\}$ are two sequences of bounded domains, by the conditions $(D_1){-}(D_2)$, $(D_3')$, $(D_4)$ and $(D_5')$ and (2.9)–(2.10), it is easy to show that $m_{a,i_a,\lambda }$ and $m_{b,j_b,\lambda }$ are positive for $\lambda \ge \Lambda _1$. It follows that

$$\begin{aligned} m_{a,i_a,\lambda }=\inf \left\{ \mathcal {E}_{\Omega _{a,i_a}'}(u)\mid \int _{\Omega _{a,i_a}'}u^4\mathrm{d}x =\frac{4m_{a,i_a,\lambda }}{\mu _1}\right\} \quad \text {for all }i_a=1,\ldots ,n_a \end{aligned}$$

(4.27)

and

$$\begin{aligned} m_{b,j_b,\lambda }=\inf \left\{ \mathcal {E}_{\Omega _{b,j_b}'}(v)\mid \int _{\Omega _{b,j_b}'}v^4\mathrm{d}x =\frac{4m_{b,j_b,\lambda }}{\mu _2}\right\} \quad \text {for all }j_b=1,\ldots ,n_b. \end{aligned}$$

(4.28)

On the other hand, let $W_{a,i_a}\in H_0^1(\Omega _{a,i_a})$ and $W_{b,j_b}\in H_0^1(\Omega _{b,j_b})$ be the least energy nonzero critical points of $I_{\Omega _{a,i_a}}(u)$ and $I_{\Omega _{b,j_b}}(v)$, respectively. Then, by the conditions $(D_3')$ and $(D_5')$, it is well known that

$$\begin{aligned} I_{\Omega _{a,i_a}}(W_{a,i_a})=\max _{t\ge 0}I_{\Omega _{a,i_a}}(tW_{a,i_a})\quad \text {and}\quad I_{\Omega _{b,j_b}}(W_{b,j_b})=\max _{t\ge 0}I_{\Omega _{b,j_b}}(tW_{b,j_b}). \end{aligned}$$

(4.29)

Let $\gamma _{0,a}:[0, 1]^{k_a}\rightarrow E_a$ and $\gamma _{0,b}:[0, 1]^{k_b}\rightarrow E_b$ be

$$\begin{aligned} \gamma _{0,a}(t_1,\ldots ,t_{k_a})=\sum _{i_a=1}^{k_a}t_{i_a}RW_{a,i_a} \end{aligned}$$

(4.30)

and

$$\begin{aligned} \gamma _{0,b}(s_1,\ldots ,s_{k_b})=\sum _{j_b=1}^{k_b}s_{j_b}RW_{b,j_b}, \end{aligned}$$

(4.31)

where $R>2$ is a large constant satisfying

$$\begin{aligned}&I_{\Omega _{a,i_a}}(RW_{a,i_a})\le 0,\quad&R^4\int _{\Omega _{a,i_a}}W_{a,i_a}^4\mathrm{d}x\ge 2\frac{4m_{a,i_a}}{\mu _1},\end{aligned}$$

(4.32)

$$\begin{aligned}&I_{\Omega _{b,j_b}}(RW_{b,j_b})\le 0,\quad&R^4\int _{\Omega _{b,j_b}}W_{b,j_b}^4\mathrm{d}x\ge 2\frac{4m_{b,j_b}}{\mu _2}. \end{aligned}$$

(4.33)

for all $i_a=1,\ldots ,k_a$ and $j_b=1,\ldots ,k_b$. By the condition $(D_3')$, we can extend $W_{a,i_a}$ and $W_{b,j_b}$ to the whole space $\mathbb {R}^3$ by letting $W_{a,i_a}=0$ on $\mathbb {R}^3\backslash \Omega _{a,i_a}$ and $W_{b,j_b}=0$ on $\mathbb {R}^3\backslash \Omega _{b,j_b}$ such that $W_{a,i_a}\in H^1(\mathbb {R}^3)$ and $W_{b,j_b}\in H^1(\mathbb {R}^3)$ for all $i_a=1,\ldots ,n_a$ and $j_b=1,\ldots ,n_b$. Now, we can define a minimax value of $J_{\lambda ,\beta }^*(u,v)$ for $\lambda \ge \Lambda _1$ and $\beta <0$ as follows:

$$\begin{aligned} m_{J_a,J_b,\lambda ,\beta }=\inf _{(\gamma _a,\gamma _b)\in \Gamma }\sup _{[0, 1]^{k_a}\times [0, 1]^{k_b}}J_{\lambda ,\beta }^*(\gamma _a,\gamma _b), \end{aligned}$$

where

$$\begin{aligned} \Gamma= & {} \bigg \{(\gamma _a,\gamma _b)\mid (\gamma _a,\gamma _b)\in C([0, 1]^{k_a}\times [0, 1]^{k_b}, E_a\times E_b),\\&\quad (\gamma _a,\gamma _b)=(\gamma _{0,a},\gamma _{0,b})\text { on }\partial ([0, 1]^{k_a}\times [0, 1]^{k_b})\bigg \}. \end{aligned}$$

$m_{J_a,J_b,\lambda ,\beta }$ may be a critical value of $J_{\lambda ,\beta }^*(u,v)$. In order to show it, we need the following.

Lemma 4.4

Assume $(\gamma _a,\gamma _b)\in \Gamma $ and

$$\begin{aligned} (\xi _1,\ldots ,\xi _{k_a},\eta _1,\ldots ,\eta _{k_b})\in & {} \left[ 0, R^4\int _{\Omega _{a,1}}W_{a,1}^4\mathrm{d}x \right] \times \ldots \times \left[ 0, R^4\int _{\Omega _{a,k_a}}W_{a,k_a}^4\mathrm{d}x \right] \\&\quad \times \left[ 0, R^4\int _{\Omega _{b,1}}W_{b,1}^4\mathrm{d}x \right] \times \ldots \times \left[ 0, R^4\int _{\Omega _{b,k_b}}W_{b,k_b}^4\mathrm{d}x \right] . \end{aligned}$$

Then, there exist $(t_1',\ldots ,t_{k_a}')\in [0, 1]^{k_a}$ and $(s_1',\ldots ,s_{k_b}')\in [0, 1]^{k_b}$ such that

$$\begin{aligned} \int _{\Omega _{a,i_a}'}[\gamma _a(t_1',\ldots ,t_{k_a}')]^4(x)\mathrm{d}x=\xi _{i_a}\quad \text {for all }i_a=1,\ldots ,k_a \end{aligned}$$

and

$$\begin{aligned} \int _{\Omega _{b,j_b}'}[\gamma _b(s_1',\ldots ,s_{k_b}')]^4(x)\mathrm{d}x=\eta _{j_b}\quad \text {for all }j_b=1,\ldots ,k_b. \end{aligned}$$

Proof

For every $(\gamma _a,\gamma _b)\in \Gamma $, we define a map $\widetilde{\gamma }:[0, 1]^{k_a}\times [0, 1]^{k_b}\rightarrow \mathbb {R}^{k_a}\times \mathbb {R}^{k_b}$ as follows:

$$\begin{aligned}&\widetilde{\gamma }(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\\&\quad =\bigg (\int _{\Omega _{a,1}'}[\gamma _a(t_1,\ldots ,t_{k_a})]^4(x)\mathrm{d}x,\ldots , \int _{\Omega _{a,k_a}'}[\gamma _a(t_1,\ldots ,t_{k_a})]^4(x)\mathrm{d}x,\\&\qquad \int _{\Omega _{b,1}'}[\gamma _b(s_1,\ldots ,s_{k_b})]^4(x)\mathrm{d}x,\ldots , \int _{\Omega _{b,k_b}'}[\gamma _b(s_1,\ldots ,s_{k_b})]^4(x)\mathrm{d}x\bigg ). \end{aligned}$$

Note that for every $(\gamma _a,\gamma _b)\in \Gamma $, we have

$$\begin{aligned} (\gamma _a(t_1,\ldots ,t_{k_a}),\gamma _b(s_1,\ldots ,s_{k_b}))= (\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b})) \end{aligned}$$

if $(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\in \partial ([0, 1]^{k_a}\times [0, 1]^{k_b})$. Then, by the construction of $\{\Omega _{a,i_a}'\}$ and $\{\Omega _{b,j_b}'\}$, we can see that

$$\begin{aligned} \int _{\Omega _{a,i_a}'}[\gamma _a(t_1,\ldots ,t_{k_a})]^4(x)\mathrm{d}x=t_{i_a}^4R^4\int _{\Omega _{a,i_a}}W_{a,i_a}^4\mathrm{d}x \end{aligned}$$

and

$$\begin{aligned} \int _{\Omega _{b,j_b}'}[\gamma _b(s_1,\ldots ,s_{k_b})]^4(x)\mathrm{d}x=s_{j_b}^4R^4\int _{\Omega _{b,j_b}}W_{b,j_b}^4\mathrm{d}x \end{aligned}$$

for all $i_a=1,\ldots ,k_a$, $j_b=1,\ldots ,k_b$ and $(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\in \partial ([0, 1]^{k_a}\times [0, 1]^{k_b})$. It follows that

$$\begin{aligned} deg(\widetilde{\gamma },[0, 1]^{k_a}\times [0, 1]^{k_b}, (\xi _1,\ldots ,\xi _{k_a},\eta _1,\ldots ,\eta _{k_b}))=1, \end{aligned}$$

which completes the proof. $\square $

With Lemma 4.4 in hands, we can obtain the following energy estimate, which can be viewed as a linking structure of $J_{\lambda ,\beta }^*(u,v)$.

Lemma 4.5

Assume $\lambda \ge \Lambda _1$ and $\beta <0$. Then, we have the following results.

(1)
If $(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\in \partial ([0, 1]^{k_a}\times [0, 1]^{k_b})$, then
$$\begin{aligned}&J_{\lambda ,\beta }^*(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\\&\quad \le \sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b}-\min \{m_{a,1},\ldots ,m_{a,k_a},m_{b,1},\ldots ,m_{b,k_b}\}. \end{aligned}$$
(2)
$\underset{i_a=1}{\overset{k_a}{\sum }}m_{a,i_a,\lambda }+\underset{j_b=1}{\overset{k_b}{\sum }}m_{b,j_b,\lambda }\le m_{J_a,J_b,\lambda ,\beta }\le \underset{i_a=1}{\overset{k_a}{\sum }}m_{a,i_a}+\underset{j_b=1}{\overset{k_b}{\sum }}m_{b,j_b}$.

Proof

(1) Since $(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\in \partial ([0, 1]^{k_a}\times [0, 1]^{k_b})$, there exists $i_a'\in \{1,\ldots ,k_a\}$ or $j_b'\in \{1,\ldots ,k_b\}$ such that $t_{i_a'}\in \{0, 1\}$ or $s_{j_b'}\in \{0, 1\}$. Without loss of generality, we assume $t_1=1$. It follows from (4.29)–(4.33) and the condition $(D_3')$ that

$$\begin{aligned}&J_{\lambda ,\beta }^*(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\\&\quad = I_{a,1}(RW_{a,1})+\sum _{i_a=2}^{k_a}I_{a,i_a}(t_{i_a}RW_{a,i_a})+\sum _{j_b=1}^{k_b}I_{b,j_b}(s_{j_b}RW_{b,j_b})\\&\quad \le \sum _{i_a=2}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b}\\&\quad \le \sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b} -\min \{m_{a,1},\ldots ,m_{a,k_a},m_{b,1},\ldots ,m_{b,k_b}\}. \end{aligned}$$

(2) Since $(\gamma _{0,a},\gamma _{0,b})\in \Gamma $ and $R>2$, by the condition $(D_3')$, we must have

$$\begin{aligned} m_{J_a,J_b,\lambda ,\beta }\le & {} \sup _{[0, 1]^{k_a}\times [0, 1]^{k_b}}J_{\lambda ,\beta }^*\left( \sum _{i_a=1}^{k_a}t_{i_a}RW_{a,i_a},\sum _{j_b=1}^{k_b}s_{j_b}RW_{b,j_b}\right) \\\le & {} \sum _{i_a=1}^{k_a}I_{a,i_a}(W_{a,i_a})+\sum _{j_b=1}^{k_b}I_{b,j_b}(W_{b,j_b})\\= & {} \sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b}. \end{aligned}$$

On the other hand, since the condition $(D_3')$ holds, by the construction of $\Omega _{a,i_a}'$ and $\Omega _{b,j_b}'$, it is easy to show that $m_{a,i_a,\lambda }\le m_{a,i_a}$ and $m_{b,j_b,\lambda }\le m_{b,j_b}$ for all $i_a=1,\ldots ,n_a$, $j_b=1,\ldots ,n_b$ and $\lambda >0$. This together with (4.32)–(4.33) and Lemma 4.4 implies that for every $(\gamma _a,\gamma _b)\in \Gamma $, there exist $(t_1',\ldots ,t_{k_a}')\in [0, 1]^{k_a}$ and $(s_1',\ldots ,s_{k_b}')\in [0, 1]^{k_b}$ such that

$$\begin{aligned} \int _{\Omega _{a,i_a}'}[\gamma _a(t_1',\ldots ,t_{k_a}')]^4(x)\mathrm{d}x=\frac{4m_{a,i_a,\lambda }}{\mu _1}\quad \text {for all }i_a=1,\ldots ,k_a \end{aligned}$$

(4.34)

and

$$\begin{aligned} \int _{\Omega _{b,j_b}'}[\gamma _b(s_1',\ldots ,s_{k_b}')]^4(x)\mathrm{d}x=\frac{4m_{b,j_b,\lambda }}{\mu _2}\quad \text {for all }j_b=1,\ldots ,k_b. \end{aligned}$$

(4.35)

Denote $\gamma _a(t_1',\ldots ,t_{k_a}')$ and $\gamma _b(s_1',\ldots ,s_{k_b}')$ by $u_*$ and $v_*$. Then, by (2.8)–(2.10), we have

$$\begin{aligned} \int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}u_*^2\mathrm{d}x\le C_{a,b}^{-1}\int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}|\nabla u_*|^2+(\lambda a(x)+a_0(x))u_*^2\mathrm{d}x \end{aligned}$$

and

$$\begin{aligned} \int _{\mathbb {R}^3\backslash \Omega _b^{J_b}}v_*^2\mathrm{d}x\le C_{a, b}^{-1}\int _{\mathbb {R}^3\backslash \Omega _b^{J_b}}|\nabla v_*|^2+(\lambda b(x)+b_0(x))v_*^2\mathrm{d}x \end{aligned}$$

for $\lambda \ge \Lambda _1$. Note that $\{\Omega _{a,i_a}'\}$ and $\{\Omega _{b,j_b}'\}$ are two sequences of bounded domains with smooth boundaries, so the restriction of $u_*$ on $\Omega _{a,i_a}'$ lies in $H^1(\Omega _{a,i_a}')$ for every $i_a=1,\ldots ,n_a, $ while the restriction of $v_*$ on $\Omega _{b,j_b}'$ lies in $H^1(\Omega _{b,j_b}')$ for every $j_b=1,\ldots ,n_b$. Now, by $\beta <0$, (4.1) and the construction of $f_a(x,t)$, $f_b(x,t)$ and h(x, t, s), we have

$$\begin{aligned} J_{\lambda ,\beta }^*(u_*,v_*)\ge & {} \frac{1}{2}\int _{\mathbb {R}^3}|\nabla u_*|^2+(\lambda a(x)+a_0(x))u_*^2\mathrm{d}x-\mu _1\int _{\mathbb {R}^3}F_a(x,u_*)\mathrm{d}x\nonumber \\&+\frac{1}{2}\int _{\mathbb {R}^3}|\nabla v_*|^2+(\lambda b(x)+b_0(x))v_*^2\mathrm{d}x-\mu _2\int _{\mathbb {R}^3}F_b(x,v_*)\mathrm{d}x\nonumber \\\ge & {} \frac{1}{2}\left( \int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}|\nabla u_*|^2+(\lambda a(x)+a_0(x))u_*^2\mathrm{d}x\nonumber \right. \\&\left. -\delta _\beta ^2\int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}u_*^2\mathrm{d}x\right) +\sum _{i_a=1}^{k_a}\mathcal {E}_{\Omega _{a,i_a}',\lambda }(u_*)\nonumber \\&+\frac{1}{2}\left( \int _{\mathbb {R}^3\backslash \Omega _b^{J_b}}|\nabla v_*|^2+(\lambda b(x)+b_0(x))v_*^2\mathrm{d}x\nonumber \right. \\&\left. -\delta _\beta ^2\int _{\mathbb {R}^3\backslash \Omega _b^{J_b}}v_*^2\mathrm{d}x\right) +\sum _{j_b=1}^{k_b}\mathcal {E}_{\Omega _{b,j_b}',\lambda }(v_*)\nonumber \\\ge & {} \sum _{i_a=1}^{k_a}\mathcal {E}_{\Omega _{a,i_a}',\lambda }(u_*)+\sum _{j_b=1}^{k_b}\mathcal {E}_{\Omega _{b,j_b}',\lambda }(v_*). \end{aligned}$$

(4.36)

Thanks to (4.27)–(4.28) and (4.34)–(4.35), (4.36) implies $J_{\lambda ,\beta }^*(u_*,v_*)\ge \sum _{i_a=1}^{k_a} m_{a,i_a,\lambda }+\sum _{j_b=1}^{k_b}m_{b,j_b,\lambda }$ for $\lambda \ge \Lambda _1$ and $\beta <0$. Since $(\gamma _a,\gamma _b)\in \Gamma $ is arbitrary, we must have $\sum _{i_a=1}^{k_a} m_{a,i_a,\lambda }+\sum _{j_b=1}^{k_b} m_{b,j_b,\lambda }\le m_{J_a,J_b,\lambda ,\beta }$ for $\lambda \ge \Lambda _1$ and $\beta <0$, which completes the proof. $\square $

Let $m_{a,b}:=\underset{i_a=1}{\overset{n_a}{\sum }}m_{a,i_a}+\underset{j_b=1}{\overset{n_b}{\sum }}m_{b,j_b}$. Then, we can obtain the following proposition.

Proposition 4.1

Suppose $\beta <0$. Then, there exists $\Lambda _2^*(\beta )\ge \Lambda _1^*(\beta , m_{a,b})$ such that $m_{J_a,J_b,\lambda ,\beta }$ is a critical value of $J_{\lambda ,\beta }^*(u,v)$ for $\lambda \ge \Lambda _2^*(\beta )$, that is, for all $\lambda \ge \Lambda _2^*(\beta )$, there exists $(u_{\lambda ,\beta },v_{\lambda ,\beta })\in E$ satisfying $D[J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$ and $J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })=m_{J_a,J_b,\lambda ,\beta }$, where $\Lambda _1^*(\beta , m_{a,b})$ is given by Lemma 4.3. Furthermore, for every $\{\lambda _n\}\subset [\Lambda _2^*(\beta ), +\infty )$ satisfying $\lambda _n\rightarrow +\infty $ as $n\rightarrow \infty $, there exists $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in E$ such that

(1)
$(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in H_0^1(\Omega _{a,0}^{J_a})\times H_0^1(\Omega _{b,0}^{J_b})$ with $u_{0,\beta }^{J_a}=0$ on $\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}$ and $v_{0,\beta }^{J_b}=0$ on $\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}$.
(2)
$(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $ up to a subsequence.
(3)
The restriction of $u_{0,\beta }^{J_a}$ on $\Omega _{a,i_a}$ lies in $H_0^1(\Omega _{a,i_a})$ and is a critical point of $I_{\Omega _{a,i_a}}(u)$ for every $i_a\in J_a$, while the restriction of $v_{0,\beta }^{J_b}$ on $\Omega _{b,j_b}$ lies in $H_0^1(\Omega _{b,j_b})$ and is a critical point of $I_{\Omega _{b,j_b}}(v)$ for every $j_b\in J_b$.

Proof

Since the conditions $(D_1){-}(D_2)$, $(D_3')$, $(D_4)$ and $(D_5')$ hold, by a similar argument as [24, Lemma 3.1], we can see that $\lim _{\lambda \rightarrow +\infty }m_{a,i_a,\lambda }=m_{a,i_a}$ and $\lim _{\lambda \rightarrow +\infty }m_{b,j_b,\lambda }=m_{b,j_b}$ for all $i_a=1,\ldots ,n_a$ and $j_b=1,\ldots ,n_b$. Note that $\beta <0$, so by Lemma 4.5, there exists $\Lambda _2^*(\beta )\ge \Lambda _1^*(\beta , m_{a,b})$ such that $m_{J_a,J_b,\lambda ,\beta }>J_{\lambda ,\beta }^*(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))$ for $\lambda \ge \Lambda _2^*(\beta )$. Thanks to the construction of $m_{J_a,J_b,\lambda ,\beta }$ and Lemma 4.2, we can use the linking theorem (cf. [1]) to show that $m_{J_a,J_b,\lambda ,\beta }$ is a critical value of $J_{\lambda ,\beta }^*(u,v)$ for $\lambda \ge \Lambda _2^*(\beta )$, that is, there exists $(u_{\lambda ,\beta },v_{\lambda ,\beta })\in E$ satisfying $D[J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })]=0$ in $E^*$ and $J_{\lambda ,\beta }^*(u_{\lambda ,\beta },v_{\lambda ,\beta })=m_{J_a,J_b,\lambda ,\beta }$ for all $\lambda \ge \Lambda _2^*(\beta )$. In what follows, we will show that (1)–(3) hold. Suppose $\{\lambda _n\}\subset [\Lambda _2^*(\beta ), +\infty )$ satisfying $\lambda _n\rightarrow +\infty $ as $n\rightarrow \infty $. Then, by Lemmas 4.3 and 4.5, $\{(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\}$ is bounded in E with

$$\begin{aligned} \int _{\mathbb {R}^3\backslash \Omega _a^{J_a}}|\nabla u_{\lambda _n,\beta }|^2+(\lambda _n a(x)+a_0(x))u_{\lambda _n,\beta }^2\mathrm{d}x\rightarrow 0\quad \text {as }n\rightarrow \infty \end{aligned}$$

(4.37)

and

$$\begin{aligned} \int _{\mathbb {R}^3\backslash \Omega _b^{J_b}}|\nabla v_{\lambda _n,\beta }|^2+(\lambda _n b(x)+b_0(x))v_{\lambda _n,\beta }^2\mathrm{d}x\rightarrow 0\quad \text {as } n\rightarrow \infty . \end{aligned}$$

(4.38)

Without loss of generality, we assume $(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })\rightharpoonup (u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})$ weakly in E as $n\rightarrow \infty $ for some $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in E$. For the sake of clarity, we divide the following proof into two steps.

Step 1 We prove that $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in H_0^1(\Omega _{a,0}^{J_a})\times H_0^1(\Omega _{b,0}^{J_b})$ with $u_{0,\beta }^{J_a}=0$ on $\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}$ and $v_{0,\beta }^{J_b}=0$ on $\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}$.

Indeed, since $\beta <0$, by Lemmas 4.1 and 4.5 and a similar argument as used in Step 1 of the proof for Theorem 1.1, we can conclude that $u_{0,\beta }^{J_a}=0$ on $\mathbb {R}^3\backslash \Omega _{a}$ and $v_{0,\beta }^{J_b}=0$ on $\mathbb {R}^3\backslash \Omega _{b}$. On the other hand, combining (2.9) and (4.37), we can see that $\int _{\Omega _{a,i_a}'}u_{\lambda _n,\beta }^2\mathrm{d}x\rightarrow 0$ for $i_a\in \{1,\ldots ,n_a\}\backslash J_a$ as $n\rightarrow \infty $, which together with the Fatou lemma implies $u_{0,\beta }^{J_a}=0$ on $\Omega _{a,i_a}'$ for $i_a\in \{1,\ldots ,n_a\}\backslash J_a$. Since (4.38) holds, by a similar argument, we also have $v_{0,\beta }^{J_b}=0$ on $\Omega _{b,j_b}'$ for $j_b\in \{1,\ldots ,n_b\}\backslash J_b$. Note that $\{\Omega _{a,i_a}\}$ and $\{\Omega _{b,j_b}\}$ are two sequences of disjoint bounded domains with smooth boundaries. So $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in H_0^1(\Omega _{a,0}^{J_a})\times H_0^1(\Omega _{b,0}^{J_b})$ with $u_{0,\beta }^{J_a}=0$ on $\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}$ and $v_{0,\beta }^{J_b}=0$ on $\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}$.

Step 2 We prove that $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $ up to a subsequence, and the restriction of $u_{0,\beta }^{J_a}$ on $\Omega _{a,i_a}$ lies in $H_0^1(\Omega _{a,i_a})$ and is a critical point of $I_{\Omega _{a,i_a}}(u)$ for every $i_a\in J_a$, while the restriction of $v_{0,\beta }^{J_b}$ on $\Omega _{b,j_b}$ lies in $H_0^1(\Omega _{b,j_b})$ and is a critical point of $I_{\Omega _{b,j_b}}(v)$ for every $j_b\in J_b$.

Indeed, since $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in H_0^1(\Omega _{a,0}^{J_a})\times H_0^1(\Omega _{b,0}^{J_b})$ with $u_{0,\beta }^{J_a}=0$ on $\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}$ and $v_{0,\beta }^{J_b}=0$ on $\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}$, by $D[J_{\lambda _n,\beta }^*(u_{\lambda _n,\beta }, v_{\lambda _n,\beta })]=0$ in $E^*$ and the condition $(D_3')$, we can see that the restriction of $u_{0,\beta }^{J_a}$ on $\Omega _{a,i_a}$, denoted by $u_{i_a,\beta }^{J_a}$, lies in $H_0^1(\Omega _{a,i_a})$ and $I_{\Omega _{a,i_a}}'(u_{i_a,\beta }^{J_a})=0$ in $H^{-1}(\Omega _{a,i_a})$ for all $i_a\in J_a$, while the restriction of $v_{0,\beta }^{J_b}$ on $\Omega _{b,j_b}$, denoted by $v_{j_b,\beta }^{J_b}$, lies in $H_0^1(\Omega _{b,j_b})$ and $I_{\Omega _{b,j_b}}'(v_{j_b,\beta }^{J_b})=0$ in $H^{-1}(\Omega _{b,j_b})$ for all $j_b\in J_b$. Now, since $\beta <0$ and the condition $(D_3)$ contains the condition $(D_3')$, by the construction of $f_a(x,t)$, $f_b(x,t)$ and h(x, t, s) and a similar argument as used in Step 2 of the proof for Theorem 1.1, we can conclude that $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta }^{J_a},b_{0,\beta }^{J_b})$ in E as $n\rightarrow \infty $. Since E is embedded continuously into $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$, $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})$ in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $. $\square $

In the following part, we will use a deformation argument to obtain the solution described by Theorem 1.2. Let $\varepsilon _0=\frac{1}{4}\min \{2\sqrt{m_{a,1}},\ldots ,2\sqrt{m_{a,k_a}},2\sqrt{m_{b,1}},\ldots ,2\sqrt{m_{b,k_b}}\}$. For $0<\varepsilon <\varepsilon _0$, we define

$$\begin{aligned} \mathcal {D}_{a,\varepsilon }= & {} \bigg \{u\in E_a\mid \bigg (\int _{\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}}|\nabla u|^2+u^2\mathrm{d}x\bigg )^{\frac{1}{2}}<\varepsilon ,\\&\quad \bigg |\bigg (\int _{\Omega _{a,i_a}}|\nabla u|^2+a_0(x)u^2\mathrm{d}x\bigg )^{\frac{1}{2}}-2\sqrt{m_{a,i_a}}\bigg |<\varepsilon ,\forall i_a\in J_a\bigg \} \end{aligned}$$

and

$$\begin{aligned} \mathcal {D}_{b,\varepsilon }= & {} \bigg \{v\in E_b\mid \bigg (\int _{\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}}|\nabla v|^2+v^2\mathrm{d}x\bigg )^{\frac{1}{2}}<\varepsilon ,\\&\quad \bigg |\bigg (\int _{\Omega _{b,j_b}}|\nabla v|^2+b_0(x)v^2\mathrm{d}x\bigg )^{\frac{1}{2}}-2\sqrt{m_{b,j_b}}\bigg |<\varepsilon , \forall j_b\in J_b\bigg \}. \end{aligned}$$

Let $\mathcal {D}_{\varepsilon }=\mathcal {D}_{a,\varepsilon }\cap \mathcal {D}_{b,\varepsilon }$ and $J_{\lambda ,\beta }^{m_{J_a,J_b}}=\bigg \{(u,v)\in E\mid J^*_{\lambda ,\beta }(u,v)\le m_{J_a,J_b}\bigg \}$, where $m_{J_a,J_b}=\sum _{i_a=1}^{k_a} m_{a,i_a}+\sum _{j_b=1}^{k_b} m_{b,j_b}$. Then, we have the following.

Proposition 4.2

Assume $\beta <0$ and $0<\varepsilon <\varepsilon _0$. Then, there exists $\Lambda _3^*(\beta ,\varepsilon )\ge \Lambda _2^*(\beta )$ such that $J^*_{\lambda ,\beta }(u,v)$ has a critical point in $\mathcal {D}_{\varepsilon }\cap J_{\lambda ,\beta }^{m_{J_a,J_b}}$ for $\lambda \ge \Lambda ^*_3(\beta ,\varepsilon )$.

Proof

Suppose the contrary, since Lemma 4.2 holds, there exist $\{\lambda _n\}$ and $\{c_{n,\beta }\}$ with $\lambda _n\rightarrow +\infty $ as $n\rightarrow \infty $ and $c_{n,\beta }>0$ for all n such that

$$\begin{aligned} \Vert D[J^*_{\lambda _n,\beta }(u,v)]\Vert _{E^*}\ge c_{n,\beta }\quad \text {for all }(u,v)\in \mathcal {D}_{\varepsilon }\cap J_{\lambda _n,\beta }^{m_{J_a,J_b}}. \end{aligned}$$

(4.39)

For the sake of clarity, we divide the following proof into several steps.

Step 1 We prove that there exists $N\in \mathbb {N}$ and a constant $\sigma _0>0$ such that $\Vert J^*_{\lambda _n,\beta }(u,v)\Vert _{E^*}\ge \sigma _0$ for every $(u,v)\in (\mathcal {D}_{3\varepsilon }\backslash \mathcal {D}_{\varepsilon })\cap J_{\lambda _n,\beta }^{m_{J_a,J_b}}$ and $n\ge N$.

Suppose the contrary, there exists a subsequence of $\{\lambda _n\}$, still denoted by $\{\lambda _n\}$, such that $\Vert J^*_{\lambda _n,\beta }(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\Vert _{E^*}\rightarrow 0$ as $n\rightarrow \infty $ for some $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\in (\mathcal {D}_{3\varepsilon }\backslash \mathcal {D}_{\varepsilon })\cap J_{\lambda _n,\beta }^{m_{J_a,J_b}}$. By a similar argument as used in Proposition 4.1, we can see that $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\rightarrow (u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ and E as $n\rightarrow \infty $ for some $(u_{0,\beta }^{J_a},v_{0,\beta }^{J_b})\in H_0^1(\Omega _{a,0}^{J_a})\times H_0^1(\Omega _{b,0}^{J_b})$ with $u_{0,\beta }^{J_a}=0$ on $\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}$ and $v_{0,\beta }^{J_b}=0$ on $\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}$. The restriction of $u_{0,\beta }^{J_a}$ on $\Omega _{a,i_a}$, denoted by $u_{i_a,\beta }^{J_a}$, lies in $H_0^1(\Omega _{a,i_a})$ and $I_{\Omega _{a,i_a}}'(u_{i_a,\beta }^{J_a})=0$ in $H^{-1}(\Omega _{a,i_a})$ for all $i_a\in J_a$, while the restriction of $v_{0,\beta }^{J_b}$ on $\Omega _{b,j_b}$, denoted by $v_{j_b,\beta }^{J_b}$, lies in $H_0^1(\Omega _{b,j_b})$ and $I_{\Omega _{b,j_b}}'(v_{j_b,\beta }^{J_b})=0$ in $H^{-1}(\Omega _{b,j_b})$ for all $j_b\in J_b$. Clearly, one of the following two cases must occur:

$(1^*)$ :: $\int _{\Omega _{a,i_a}}(u_{i_a,\beta }^{J_a})^4\mathrm{d}x\ge C$ for all $i_a\in J_a$ and $\int _{\Omega _{b,j_b}}(v_{j_b,\beta }^{J_b})^4\mathrm{d}x\ge C$ for all $j_b\in J_b$.
$(2^*)$ :: There exists $i_a'\in J_a$ or $j_b'\in J_b$ such that $\int _{\Omega _{a,i_a'}}(u_{i_a',\beta }^{J_a})^4\mathrm{d}x=0$ or $\int _{\Omega _{b,j_b'}}(v_{j_b',\beta }^{J_b})^4\mathrm{d}x=0$.

If case $(1^*)$ happens, then we must have $I_{\Omega _{a,i_a}}(u_{i_a,\beta }^{J_a})\ge m_{a,i_a}$ and $I_{\Omega _{b,j_b}}(v_{j_b,\beta }^{J_b})\ge m_{b,j_b}$ for all $i_a\in J_a$ and $j_b\in J_b$. Since $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}$, by a similar argument as used in Step 3 of the proof for Theorem 1.1, we can show that $I_{\Omega _{a,i_a}}(u_{i_a,\beta }^{J_a})= m_{a,i_a}$ and $I_{\Omega _{b,j_b}}(v_{j_b,\beta }^{J_b})= m_{b,j_b}$ for all $i_a\in J_a$ and $j_b\in J_b$. It follows from the condition $(D_4)$ that

$$\begin{aligned} \int _{\Omega _{a,i_a}}|\nabla u_{\lambda _n,\beta }|^2+a_0(x)u_{\lambda _n,\beta }^2\mathrm{d}x\rightarrow 4m_{a,i_a}\quad \text {for all }i_a\in J_a \end{aligned}$$

and

$$\begin{aligned} \int _{\Omega _{b,j_b}}|\nabla v_{\lambda _n,\beta }|^2+b_0(x)v_{\lambda _n,\beta }^2\mathrm{d}x\rightarrow 4m_{b,j_b}\quad \text {for all }j_b\in J_b \end{aligned}$$

as $n\rightarrow \infty $, which then implies $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\in \mathcal {D}_{\varepsilon }$ for n large enough. It is impossible since $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\in (\mathcal {D}_{3\varepsilon }\backslash \mathcal {D}_{\varepsilon })\cap J_{\lambda _n,\beta }^{m_{J_a,J_b}}$ for all n. Thus, we must have the case $(2^*)$. Without loss of generality, we assume $\int _{\Omega _{a,1}}(u_{1,\beta }^{J_a})^4\mathrm{d}x=0$. It follows from the condition $(D_4)$ that

$$\begin{aligned} \bigg |\bigg (\int _{\Omega _{a,1}}|\nabla u_{\lambda _n,\beta }|^2+a_0(x)u_{\lambda _n,\beta }^2\mathrm{d}x\bigg )^{\frac{1}{2}}-2\sqrt{m_{a,1}}\bigg |\rightarrow 2\sqrt{m_{a,1}}=4\varepsilon _0\quad \text {as }n\rightarrow \infty , \end{aligned}$$

which implies $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\in E\backslash \mathcal {D}_{3\varepsilon }$ for n large enough. It also contradicts to the fact that $(u_{\lambda _n,\beta },v_{\lambda _n,\beta })\in (\mathcal {D}_{3\varepsilon }\backslash \mathcal {D}_{\varepsilon })\cap J_{\lambda _n,\beta }^{m_{J_a,J_b}}$ for all n.

Step 2 We construct a descending flow on $J_{\lambda _n,\beta }^{m_{J_a,J_b}}$ for every $n\ge N$.

Let $\eta :E\rightarrow [0, 1]$ be a local Lipschitz continuous function and satisfy

$$\begin{aligned} \eta (u,v)=\left\{ \begin{array}{ll} 1,&{}\quad (u,v)\in \mathcal {D}_{\frac{3}{2}\varepsilon },\\ 0,&{}\quad (u,v)\in E\backslash \mathcal {D}_{2\varepsilon }.\end{array}\right. \end{aligned}$$

Since $J^*_{\lambda _n,\beta }(u,v)$ is $C^1$ for every $n\ge N$, there exists a pseudo-gradient vector field of $J^*_{\lambda _n,\beta }(u,v)$, denoted by $\widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}=(\widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}_1, \widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}_2)$, which satisfies

$(a_*)$ :: $\langle D[J^*_{\lambda _n,\beta }(u,v)], \widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}\rangle _{E^*,E^*}\ge \frac{1}{2}\Vert D[J^*_{\lambda _n,\beta }(u,v)]\Vert _{E^*}^2$ for all $(u,v)\in E$;
$(b_*)$ :: $\Vert \widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}\Vert _{E^*}\le 2\Vert D[J^*_{\lambda _n,\beta }(u,v)]\Vert _{E^*}$ for all $(u,v)\in E$.

Let $\overrightarrow{\mathcal {V}}_{n}:J_{\lambda _n,\beta }^{m_{J_a,J_b}}\rightarrow E^*$ be a continuous map and given by

$$\begin{aligned} \overrightarrow{\mathcal {V}}_{n}(u,v)= & {} (\mathcal {V}_{1,n}(u,v),\mathcal {V}_{2,n}(u,v))\\= & {} -\frac{\eta (u,v)}{\Vert \widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}\Vert _{E^*}}(\widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}_1, \widetilde{D[J^*_{\lambda _n,\beta }(u,v)]}_2) \end{aligned}$$

for $(u,v)\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}\backslash \mathcal {K}=\{(u,v)\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}\mid D[J^*_{\lambda _n,\beta }(u,v)]\not =0\}$ and

$$\begin{aligned} \overrightarrow{\mathcal {V}}_{n}(u,v)=(\mathcal {V}_{1,n}(u,v),\mathcal {V}_{2,n}(u,v)) =(0, 0) \end{aligned}$$

for $(u,v)\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}\cap \mathcal {K}=\{(u,v)\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}\mid D[J^*_{\lambda _n,\beta }(u,v)]=0\}$. Clearly, $\Vert \overrightarrow{\mathcal {V}}_{n}(u,v)\Vert _{E^*}\le 1$ for all $(u,v)\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}$. Furthermore, by (4.39), Step 1 and the definition of $\eta $, we can see that $\overrightarrow{\mathcal {V}}_{n}(u,v)$ is locally Lipschitz. Now, let us consider the flow $\overrightarrow{\rho }_{n}(\tau )=(\rho _{1,n}(\tau ),\rho _{2,n}(\tau ))$ given by the following two-component system of ODE

$$\begin{aligned} \left\{ \begin{array}{l} \frac{d\rho _{1,n}(\tau )}{\mathrm{d}\tau }=\mathcal {V}_{1,n}(\rho _{1,n}(\tau ),\rho _{2,n}(\tau )),\\ \frac{d\rho _{2,n}(\tau )}{\mathrm{d}\tau }=\mathcal {V}_{2,n}(\rho _{1,n}(\tau ),\rho _{2,n}(\tau )),\\ (\rho _{1,n}(0),\rho _{2,n}(0))=(u,v)\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}.\end{array}\right. \end{aligned}$$

By $(a_*)$, $(b_*)$ and a direct calculation, we can see that

$$\begin{aligned}&\frac{dJ^*_{\lambda _n,\beta }(\rho _{1,n}(\tau ),\rho _{2,n}(\tau ))}{\mathrm{d}\tau }\\&\quad =\bigg \langle (\frac{\partial J^*_{\lambda _n,\beta }}{\partial u}(\rho _{1,n},\rho _{2,n}),\frac{\partial J^*_{\lambda _n,\beta }}{\partial v}(\rho _{1,n},\rho _{2,n})),(\mathcal {V}_{1,n}(\rho _{1,n},\rho _{2,n}), \mathcal {V}_{2,n}(\rho _{1,n},\rho _{2,n}))\bigg \rangle _{E^*,E^*}\\&\quad \le -\frac{1}{4}\eta (\rho _{1,n},\rho _{2,n})\Vert D[J^*_{\lambda _n,\beta }(\rho _{1,n},\rho _{2,n})]\Vert _{E^*}\\&\quad \le 0. \end{aligned}$$

It follows that $\overrightarrow{\rho }_{n}(\tau )=(\rho _{1,n}(\tau ),\rho _{2,n}(\tau ))$ is a descending flow on $J_{\lambda _n,\beta }^{m_{J_a,J_b}}$. Furthermore, for every $\tau >0$, we have $\overrightarrow{\rho }_{n}(\tau )=\overrightarrow{\mathcal {\rho }}_{n}(0)$ if $\overrightarrow{\mathcal {\rho }}_{n}(0)=(u,v)\in E\backslash \mathcal {D}_{2\varepsilon }$.

Step 3 For every $n\ge N$, we construct a map $\overrightarrow{\rho }^0_{n}(\tau )=(\rho ^0_{1,n}(\tau ), \rho ^0_{2,n}(\tau ))\in \Gamma $ for all $\tau >0$ such that

$$\begin{aligned} \sup _{[0, 1]^{k_a}\times [0, 1]^{k_b}}J^*_{\lambda _n,\beta }(\rho ^0_{1,n}(\tau _n), \rho ^0_{2,n}(\tau _n))<m_{J_a,J_b}-\sigma _0^*\quad \text {for some }\tau _n>0, \end{aligned}$$

(4.40)

where $\sigma _0^*>0$ is a constant.

Indeed, let $n\ge N$ and $\gamma _{0,a}$ and $\gamma _{0,b}$ be given by (4.30) and (4.31). We consider $\overrightarrow{\rho }^0_{n}(\tau )=(\rho ^0_{1,n}(\tau ), \rho ^0_{2,n}(\tau ))$, where $(\rho ^0_{1,n}(0), \rho ^0_{2,n}(0))=(\gamma _{0,a},\gamma _{0,b})$. Since $W_{a,i_a}$ and $W_{b,j_b}$ are the least energy nonzero critical points of $I_{\Omega _{a,i_a}}(u)$ and $I_{\Omega _{b,j_b}}(v)$ for all $i_a\in J_a$ and $j_b\in J_b$, by the choice of R and $\varepsilon $, we can see that

$$\begin{aligned} (\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\in E\backslash \mathcal {D}_{2\varepsilon } \end{aligned}$$

for every $(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\in \partial ([0, 1]^{k_a}\times [0,1]^{k_b})$. It follows from the construction of $(\rho ^0_{1,n}(\tau ), \rho ^0_{2,n}(\tau ))$ that

$$\begin{aligned} (\rho ^0_{1,n}(\tau )(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau )(s_1,\ldots ,s_{k_b})) =(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b})) \end{aligned}$$

for every $(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\in \partial ([0, 1]^{k_a}\times [0,1]^{k_b})$ and $\tau >0$. Thus, $\overrightarrow{\rho }^0_{n}(\tau )\in \Gamma $ for all $\tau >0$. It remains to show (4.40) holds. For every $(t_1,\ldots ,t_{k_a},s_1,\ldots ,s_{k_b})\in [0, 1]^{k_a}\times [0,1]^{k_b}$, one of the following two cases must occur:

$(a^*)$ :: $(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\in E\backslash \mathcal {D}_{\varepsilon }$.
$(b^*)$ :: $(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\in \mathcal {D}_{\varepsilon }$.

If case $(a^*)$ happens, then by Step 2, we must have

$$\begin{aligned}&J^*_{\lambda _n,\beta }(\rho ^0_{1,n}(\tau )(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau )(s_1,\ldots ,s_{k_b}))\\&\quad \le J^*_{\lambda _n,\beta }(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b})) \end{aligned}$$

for all $\tau >0$. Moreover, by (4.29) and the choice of $\{W_{a,i_a}\}$ and $\{W_{b,j_b}\}$, we can see that

$$\begin{aligned} J^*_{\lambda _n,\beta }(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b})) =\sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b} \end{aligned}$$

if and only if $t_{i_a}=s_{j_b}=\frac{1}{R}$ for all $i_a\in J_a$ and $j_b\in J_b$. Since $(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\in E\backslash \mathcal {D}_{\varepsilon }$ in this case, there exists $i_a'\in J_a$ or $j_b'\in J_b$ such that $t_{i_a'}\not =\frac{1}{R}$ or $s_{j_b'}\not =\frac{1}{R}$. It follows the construction of $\gamma _{0,a}$ and $\gamma _{0,b}$ and the condition $(D_3')$ that

$$\begin{aligned} m_{a,b}^*=\sup _{(u,v)\in \mathbb {P}}J^*_{\lambda _n,\beta }(u,v)=\sup _{(u,v)\in \mathbb {P}}(I_{\Omega _a}(u)+I_{\Omega _b}(v)) <\sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b}, \end{aligned}$$

(4.41)

where $\mathbb {P}=(\gamma _{0,a}([0, 1]^{k_a})\times \gamma _{0,b}([0,1]^{k_b}))\backslash \mathcal {D}_{\varepsilon }$. If case $(b^*)$ happens, then two subcases may occur:

$(b^*_1)$ :: $(\rho ^0_{1,n}(\tau )(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau )(s_1,\ldots ,s_{k_b}))\in \mathcal {D}_{\frac{3}{2}\varepsilon }$ for all $\tau >0$.
$(b^*_2)$ :: There exists $\tau _{n}^*>0$ such that $(\rho ^0_{1,n}(\tau _n^*)(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau _n^*)(s_1,\ldots ,s_{k_b}))\in E\backslash \mathcal {D}_{\frac{3}{2}\varepsilon }$.

In the subcase $(b^*_1)$, by Step 2 and the Taylor expansion, we can calculate that

$$\begin{aligned}&J^*_{\lambda _n,\beta }(\rho ^0_{1,n}(\tau )(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau )(s_1,\ldots ,s_{k_b}))\nonumber \\&\quad \le J^*_{\lambda _n,\beta }(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\nonumber \\&\qquad -\int _0^\tau \frac{1}{4}\eta (\rho _{1,n},\rho _{2,n})\Vert D[J^*_{\lambda _n,\beta }(\rho _{1,n},\rho _{2,n})]\Vert _{E^*}d\nu \nonumber \\&\quad \le \sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b}-\frac{1}{4}\tau \min \{c_{n,\beta },\sigma _0\}, \end{aligned}$$

(4.42)

where $c_{n,\beta }$ is given by (4.39) and $\sigma _0$ is given by Step 1. It follows that

$$\begin{aligned} J^*_{\lambda _n,\beta }(\rho ^0_{1,n}(\tau )(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau )(s_1,\ldots ,s_{k_b}))<\underset{i_a=1}{\overset{k_a}{\sum }}m_{a,i_a}+\underset{j_b=1}{\overset{k_b}{\sum }}m_{b,j_b}-\sigma _0 \end{aligned}$$

for $\tau \ge \tau _n^0=\frac{8\sigma _0}{\min \{c_{n,\beta },\sigma _0\}}$. In the subcase $(b^*_2)$, there must exist $0\le \tau _{n,a}^*<\tau _{n,b}^*\le \tau _{n}^*$ such that

$$\begin{aligned} (\rho ^0_{1,n}(\tau _{n,a}^*)(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau _{n,a}^*)(s_1,\ldots ,s_{k_b}))\in \partial \mathcal {D}_{\varepsilon } \end{aligned}$$

and

$$\begin{aligned} (\rho ^0_{1,n}(\tau _{n,b}^*)(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau _{n,b}^*)(s_1,\ldots ,s_{k_b}))\in \partial \mathcal {D}_{\frac{3}{2}\varepsilon }. \end{aligned}$$

For the sake of convenience, we, respectively, denote $(\rho ^0_{1,n}(\tau _{n,a}^*)(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau _{n,a}^*)(s_1,\ldots ,s_{k_b}))$ and $(\rho ^0_{1,n}(\tau _{n,b}^*)(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau _{n,b}^*)(s_1,\ldots ,s_{k_b}))$ by $(u_{n,1},v_{n,1})$ and $(u_{n,2}, v_{n,2})$. Since Lemma 2.1 holds for $\lambda \ge \Lambda _1$, by a similar argument as used for (4.14) in [24], we can see that one of the following four cases must happen:

$(a^{**})$ :: $\int _{\Omega _a}|\nabla (u_{n,1}-u_{n,2})|^2+a_0(x)(u_{n,1}-u_{n,2})^2\mathrm{d}x\ge \frac{\varepsilon ^2}{4}$.
$(b^{**})$ :: $\int _{\Omega _b}|\nabla (v_{n,1}-v_{n,2})|^2+b_0(x)(v_{n,1}-v_{n,2})^2\mathrm{d}x\ge \frac{\varepsilon ^2}{4}$.
$(c^{**})$ :: $\int _{\mathbb {R}^3\backslash \Omega _{a,0}^{J_a}}|\nabla (u_{n,1}-u_{n,2})|^2+(u_{n,1}-u_{n,2})^2\mathrm{d}x\ge \frac{\varepsilon ^2}{4}$.
$(d^{**})$ :: $\int _{\mathbb {R}^3\backslash \Omega _{b,0}^{J_b}}|\nabla (v_{n,1}-v_{n,2})|^2+(v_{n,1}-v_{n,2})^2\mathrm{d}x\ge \frac{\varepsilon ^2}{4}$.

By similar arguments as (2.1)–(2.2), we can see that in any case, there exists a constant $C(\varepsilon )>0$ such that $\Vert (u_{n,1},v_{n,1})-(u_{n,2}, v_{n,2})\Vert \ge C(\varepsilon )$. On the other hand, by Step 2, we can see that $(u_{n,1},v_{n,1})\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}$ and $(u_{n,2},v_{n,2})\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}$. It follows from $\Vert \overrightarrow{\mathcal {V}}_{n}(u,v)\Vert _{E^*}\le 1$ for all $(u,v)\in J_{\lambda _n,\beta }^{m_{J_a,J_b}}$ and the Taylor expansion that $\tau _{n,b}^*-\tau _{n,a}^*\ge C(\varepsilon )$. Now, by Step 1, we have

$$\begin{aligned}&J_{\lambda _n,\beta }(\rho ^0_{1,n}(\tau _n^*)(t_1,\ldots ,t_{k_a}),\rho ^0_{2,n}(\tau _n^*)(s_1,\ldots ,s_{k_b}))\nonumber \\&\quad \le J_{\lambda _n,\beta }(\gamma _{0,a}(t_1,\ldots ,t_{k_a}),\gamma _{0,b}(s_1,\ldots ,s_{k_b}))\nonumber \\&\qquad -\int _{\tau _{n,a}^*}^{\tau _{n,b}^*}\frac{1}{4}\eta (\rho _{1,n},\rho _{2,n})\Vert D[J_{\lambda _n,\beta }(\rho _{1,n},\rho _{2,n})]\Vert _{E^*}d\nu \nonumber \\&\quad \le \sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b}m_{b,j_b}-\frac{1}{4}C(\varepsilon )\sigma _0. \end{aligned}$$

(4.43)

Let $\tau _n=\max \{\tau _n^0,\tau _n^*\}$ and $\sigma _0^*=\min \{\frac{1}{4}C(\varepsilon )\sigma _0,\sigma _0,\sum _{i_a=1}^{k_a}m_{a,i_a}+\sum _{j_b=1}^{k_b} m_{b,j_b}-m_{a,b}^*\}$. Then, (4.40) follows from (4.41)–(4.43) and Step 2.

Since $\overrightarrow{\rho }^0_{n}(\tau )=(\rho ^0_{1,n}(\tau ), \rho ^0_{2,n}(\tau ))\in \Gamma $ for all $\tau >0$, by the definition of $m_{J_a,J_b,\lambda _n,\beta }$ and Step 3, we can see that $m_{J_a,J_b,\lambda _n,\beta }\le m_{J_a,J_b}-\sigma _0^*$, which is impossible since Lemma 4.5 holds and $m_{a,i_a,\lambda _n}\rightarrow m_{a,i_a}$ and $m_{b,j_b,\lambda _n}\rightarrow m_{b,j_b}$ as $n\rightarrow \infty $ for all $i_a\in J_a$ and $j_b\in J_b$. $\square $

We close this section by

Proof of Theorem 1.2

Suppose $\beta <0$. Then, by Proposition 4.2, for every $\varepsilon \in (0, \varepsilon _0)$, there exists $\Lambda ^*_3(\beta ,\varepsilon )>\Lambda _2^*(\beta )$ such that $J_{\lambda ,\beta }^*(u,v)$ has a critical point $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})\in \mathcal {D}_{\varepsilon }\cap J_{\lambda ,\beta }^{m_{J_a,J_b}}$ for all $\lambda \ge \Lambda _3^*(\beta ,\varepsilon )$. Thanks to Lemma 4.3 and the choice of $\Lambda ^*_3(\beta ,\varepsilon )$, $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})$ is also a critical point of $J_{\lambda ,\beta }(u,v)$. Since $u_{\lambda ,\beta }^{J_a}$ and $v_{\lambda ,\beta }^{J_b}$ are both nonnegative by the construction of $J_{\lambda ,\beta }^*(u,v)$, we can use a similar argument as used in the proof of Theorem 1.1 to show that $u_{\lambda ,\beta }^{J_a}$ and $v_{\lambda ,\beta }^{J_b}$ are both positive, which implies $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})$ is a solution of $(\mathcal {P}_{\lambda ,\beta })$. Clearly, $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})$ satisfies the concentration behaviors of (1) and (2), since $\Lambda ^*_3(\beta ,\varepsilon )\rightarrow +\infty $ as $\varepsilon \rightarrow 0$ and $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})\in \mathcal {D}_{\varepsilon }$. Furthermore, since $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})\in J_{\lambda ,\beta }^{m_{J_a,J_b}}$, by a similar argument as used in Proposition 4.1, we can see that the properties (3) and (4) are also hold. It remains to show that the concentration behavior (5) is also true. Indeed, by a similar argument as used in Proposition 4.1, the restriction of $u_{0,\beta }^{J_a}$ on $\Omega _{a,i_a}$, denoted by $u^{J_a}_{i_a,\beta }$, lies in $H_0^1(\Omega _{a,i_a})$ and is a critical point of $I_{\Omega _{a,i_a}}(u)$ for every $i_a\in J_a$, while the restriction of $v_{0,\beta }^{J_b}$ on $\Omega _{b,j_b}$, denoted by $v^{J_b}_{j_b,\beta }$, lies in $H_0^1(\Omega _{b,j_b})$ and is a critical point of $I_{\Omega _{b,j_b}}(v)$ for every $j_b\in J_b$. Since $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})\in \mathcal {D}_{\varepsilon }$ for all $\lambda \ge \Lambda _3^*(\beta ,\varepsilon )$, it is easy to see that $u^{J_a}_{i_a,\beta }$ and $v^{J_b}_{j_b,\beta }$ are nonzero for all $i_a\in J_a$ and $j_b\in J_b$. It follows from $(u_{\lambda ,\beta }^{J_a},v_{\lambda ,\beta }^{J_b})\in J_{\lambda ,\beta }^{m_{J_a,J_b}}$ that $u^{J_a}_{i_a,\beta }$ and $v^{J_b}_{j_b,\beta }$ are the least energy nonzero critical points of $I_{\Omega _{a,i_a}}(u)$ and $I_{\Omega _{b,j_b}}(v)$ for every $i_a\in J_a$ and $j_b\in J_b$, respectively. The proof of this theorem can be finished by taking $\Lambda _{*}(\beta )=\Lambda _3^*(\beta ,\frac{\varepsilon _0}{2})$. $\square $

5 The phenomenon of phase separations

In this section, we study the phenomenon of phase separations to $(\mathcal {P}_{\lambda ,\beta })$, that is, we study the concentration behavior of the solutions for $(\mathcal {P}_{\lambda ,\beta })$ as $\beta \rightarrow -\infty $.

Proof of Theorem 1.3

Suppose $\lambda \ge \Lambda _*$ and $\{\beta _n\}\subset (-\infty , 0)$ satisfying $\beta _n\rightarrow -\infty $ as $n\rightarrow \infty $. Let $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})$ be the ground state solution of $(\mathcal {P}_{\lambda ,\beta _n})$ obtained by Theorem 1.1. Then, by Lemma 3.2 and a similar argument of (3.9), we can see that $\{(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\}$ is bounded in E. Without loss of generality, we assume $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\rightharpoonup (u_{\lambda ,0}, v_{\lambda ,0})$ weakly in E as $n\rightarrow \infty $ for some $(u_{\lambda ,0}, v_{\lambda ,0})\in E$. Since E is embedded continuously into $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$, we have $(u_{\lambda ,0}, v_{\lambda ,0})\in H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$. Clearly, $u_{\lambda ,0}\ge 0$ and $v_{\lambda ,0}\ge 0$ in $\mathbb {R}^3$. In what follows, we verify that $(u_{\lambda ,0}, v_{\lambda ,0})$ satisfies (1)–(4). For the sake of clarity, we divide the following proof into several steps.

Step 1 We prove that there exists $\Lambda _{**}\ge \Lambda _*$ such that $u_{\lambda ,0}\not =0$ and $v_{\lambda ,0}\not =0$ in $\mathbb {R}^3$ for $\lambda \ge \Lambda _{**}$ in the sense of almost everywhere.

Indeed, suppose $u_{\lambda ,0}=0$ a.e. in $\mathbb {R}^3$. Since $u_{\lambda ,\beta _n}\rightharpoonup u_{\lambda ,0}$ weakly in $E_a$ as $n\rightarrow \infty $ and $\{(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\}$ is bounded in E, by a similar argument as (3.12), we can see that

$$\begin{aligned} \Vert u_{\lambda ,\beta _n}\Vert _{a,\lambda }^2\le C\lambda ^{-\frac{1}{2}}\Vert u_{\lambda ,\beta _n}\Vert _{a,\lambda }^2+o_n(1). \end{aligned}$$

It follows that there exists $\Lambda _{**}\ge \Lambda _*$ such that $\Vert u_{\lambda ,\beta _n}\Vert _{a,\lambda }^2=o_n(1)$ for $\lambda \ge \Lambda _{**}$, which together Lemma 2.1 and the boundedness of $\{(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\}$ in E implies $\Vert u_{\lambda ,\beta _n}\Vert _4=o_n(1)$. Thanks to the fact that $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})$ is the ground state solution of $(\mathcal {P}_{\lambda ,\beta _n})$ with $\beta _n<0$ for every n, we also have $\beta _n\int _{\mathbb {R}^3}u_{\lambda ,\beta _n}^2v_{\lambda ,\beta _n}^2\mathrm{d}x\rightarrow 0$ as $n\rightarrow \infty $. Hence, $J_{\lambda ,\beta _n}(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})=I_{b,\lambda }(v_{\lambda ,\beta _n})+o_n(1)$. On the other hand, by Lemma 2.1, for every $n\in \mathbb {N}$, there exists $t_n>0$ such that $t_nu_{\lambda ,\beta _n}\in \mathcal {N}_{a,\lambda }$ with $\lambda \ge \lambda _{**}$. It follows from Lemma 3.1 and $\beta _n<0$ that

$$\begin{aligned} J_{\lambda ,\beta _n}(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\ge & {} J_{\lambda ,\beta _n}(t_nu_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\nonumber \\\ge & {} I_{a,\lambda }(t_nu_{\lambda ,\beta _n})+I_{b,\lambda }(v_{\lambda ,\beta _n})\nonumber \\\ge & {} m_{a,\lambda }+J_{\lambda ,\beta _n}(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})+o_n(1). \end{aligned}$$

(5.1)

Since $m_{a,\lambda }>0$ for $\lambda \ge \lambda _{**}$, (5.1) is impossible for n large enough. By a similar argument, we can also show that $v_{\lambda ,0}\not =0$ in $\mathbb {R}^3$ for $\lambda \ge \Lambda _{**}$ in the sense of almost everywhere.

Step 2 We prove that $\{u_{\lambda ,\beta _n}\}, \{v_{\lambda ,\beta _n}\}\subset C(\mathbb {R}^3)$ and $\Vert u_{\lambda ,\beta _n}\Vert _{C(\mathbb {R}^3)}\le C_0$ and $\Vert v_{\lambda ,\beta _n}\Vert _{C(\mathbb {R}^3)}\le C_0$ for some $C_0>0$.

Indeed, by a similar argument as used in the proof of Theorem 1.1, we have $\{u_{\lambda ,\beta _n}\}, \{v_{\lambda ,\beta _n}\}\subset C(\mathbb {R}^3)$. On the other hand, since $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})$ is the ground state solution of $(\mathcal {P}_{\lambda ,\beta _n})$ obtained by Theorem 1.1 and $\{(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\}$ is bounded in E, we can use a similar argument as used in (3) of Lemma 4.3 to show that $\{u_{\lambda ,\beta _n}\}$ and $\{v_{\lambda ,\beta _n}\}$ are bounded in $L^\infty (\mathbb {R}^3)$, that is, $\Vert u_{\lambda ,\beta _n}\Vert _{L^\infty (\mathbb {R}^3)}\le C_0$ and $\Vert v_{\lambda ,\beta _n}\Vert _{L^\infty (\mathbb {R}^3)}\le C_0$ for some $C_0>0$, which implies $\Vert u_{\lambda ,\beta _n}\Vert _{C(\mathbb {R}^3)}\le C_0$ and $\Vert v_{\lambda ,\beta _n}\Vert _{C(\mathbb {R}^3)}\le C_0$.

Step 3 We prove that $u_{\lambda ,0}, v_{\lambda ,0}\in C(\mathbb {R}^3)$ and are all local Lipschitz in $\mathbb {R}^3$.

Indeed, since the conditions $(D_1){-}(D_5)$ hold, by [47, Theorem 1.7] and Step 2, we can see that $\{\nabla u_{\lambda ,\beta _n}\}$ and $\{\nabla v_{\lambda ,\beta _n}\}$ are bounded in $L^\infty (\mathbb {R}^3)$. On the other hand, for every n, by a similar argument as used in (3) of Lemma 4.3, we can show that $u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n}\in L^\gamma (\mathbb {R}^3)$ for all $\gamma \ge 2$. Thanks to the Calderon-Zygmund inequality and conditions $(D_1){-}(D_5)$, we have $u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n}\in W_{loc}^{2,\gamma }(\mathbb {R}^3)$ for all $\gamma \ge 2$. Together with the Sobolev embedding theorem, it implies $u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n}\in C^1(\mathbb {R}^3)$. It follows that $\{u_{\lambda ,\beta _n}\}$ and $\{v_{\lambda ,\beta _n}\}$ are bounded in $C^1(\mathbb {R}^3)$. Now, by applying the Ascoli-Arzelá theorem, we can conclude that $u_{\lambda ,\beta _n}\rightarrow u_{\lambda ,0}$ and $v_{\lambda ,\beta _n}\rightarrow v_{\lambda ,0}$ strongly in $C_{loc}(\mathbb {R}^3)$ as $n\rightarrow \infty $ with $u_{\lambda ,0}, v_{\lambda ,0}\in C(\mathbb {R}^3)$. This together with the boundness of $\{u_{\lambda ,\beta _n}\}$ and $\{v_{\lambda ,\beta _n}\}$ in $C^1(\mathbb {R}^3)$ again implies $u_{\lambda ,0}$ and $v_{\lambda ,0}$ are all local Lipschitz in $\mathbb {R}^3$.

Step 4 We prove that $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\rightarrow (u_{\lambda ,0}, v_{\lambda ,0})$ strongly in $H^1(\mathbb {R}^3)\times H^1(\mathbb {R}^3)$ as $n\rightarrow \infty $. Furthermore, $u_{\lambda ,0}\in H_0^1(\{u_{\lambda ,0}>0\})$ and is a least energy solution of (1.3), while $v_{\lambda ,0}\in H_0^1(\{v_{\lambda ,0}>0\})$ and is a least energy solution of (1.4).

Indeed, since $u_{\lambda ,0}\in C(\mathbb {R}^3)$ and is local Lipschitz in $\mathbb {R}^3$, we can conclude that $\partial \{u_{\lambda ,0}>0\}$, the boundary of the set $\{u_{\lambda ,0}>0\}$, is local Lipschitz. It follows from $u_{\lambda ,0}\in H^1(\mathbb {R}^3)$ and $u_{\lambda ,0}=0$ in $\mathbb {R}^3\backslash \{u_{\lambda ,0}>0\}$ that $u_{\lambda ,0}\in H_0^1(\{u_{\lambda ,0}>0\})$. Similarly, we have $v_{\lambda ,0}\in H_0^1(\{v_{\lambda ,0}>0\})$. Let $I_{a,\lambda }^*(u)$ and $I_{b,\lambda }^*(v)$, respectively, be the corresponding functional of (1.3) and (1.4). By a similar argument as used in Lemma 2.1, we can show that

$$\begin{aligned} C\int _{\{u_{\lambda ,0}>0\}}u^2\mathrm{d}x\le \int _{\{u_{\lambda ,0}>0\}}|\nabla u|^2+(\lambda a(x)+a_0(x))u^2\mathrm{d}x\quad \text {for all }u\in H_0^1(\{u_{\lambda ,0}>0\}) \end{aligned}$$

(5.2)

and

$$\begin{aligned} C\int _{\{v_{\lambda ,0}>0\}}v^2\mathrm{d}x\le \int _{\{v_{\lambda ,0}>0\}}|\nabla v|^2+(\lambda b(x)+b_0(x))v^2\mathrm{d}x\quad \text {for all }v\in H_0^1(\{v_{\lambda ,0}>0\}) \end{aligned}$$

(5.3)

if $\lambda \ge \Lambda _1$. It follows that the Nehari manifolds of $I_{a,\lambda }^*(u)$ and $I_{b,\lambda }^*(v)$ are both well defined if $\lambda \ge \Lambda _1$. Let

$$\begin{aligned} m_{a,\lambda }^*=\inf _{\mathcal {N}_{a,\lambda }^*}I_{a,\lambda }^*(u)\quad \text {and}\quad m_{b,\lambda }^*=\inf _{\mathcal {N}_{b,\lambda }^*}I_{b,\lambda }^*(v), \end{aligned}$$

where $\mathcal {N}_{a,\lambda }^*$ and $\mathcal {N}_{b,\lambda }^*$ are, respectively, the Nehari manifolds of $I_{a,\lambda }^*(u)$ and $I_{b,\lambda }^*(v)$. Then, $m_{a,\lambda }^*>0$ and $m_{b,\lambda }^*>0$ if $\lambda \ge \Lambda _1$. For every $\varepsilon >0$, there exist $u_\varepsilon \in \mathcal {N}_{a,\lambda }^*$ and $v_\varepsilon \in \mathcal {N}_{b,\lambda }^*$ such that

$$\begin{aligned} I_{a,\lambda }^*(u_\varepsilon )<m_{a,\lambda }^*+\varepsilon \quad \text {and}\quad I_{b,\lambda }^*(v_\varepsilon )<m_{b,\lambda }^*+\varepsilon . \end{aligned}$$

(5.4)

Since $\{(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\}$ is bounded in E, by the fact that $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})$ is the ground state solution $(\mathcal {P}_{\lambda ,\beta _n})$ and $\beta _n\rightarrow -\infty $, we have $\int _{\mathbb {R}^3}u_{\lambda ,\beta _n}^2v_{\lambda ,\beta _n}^2\mathrm{d}x\rightarrow 0$ as $n\rightarrow \infty $. It follows from the Fatou lemma that $\int _{\mathbb {R}^3}u_{\lambda ,0}^2v_{\lambda ,0}^2\mathrm{d}x=0$, which implies $\{u_{\lambda ,0}>0\}\cap \{v_{\lambda ,0}>0\}=\emptyset $. Hence, by $u_\varepsilon \in \mathcal {N}_{a,\lambda }^*$ and $v_\varepsilon \in \mathcal {N}_{b,\lambda }^*$, we can see that $(u_\varepsilon ,v_\varepsilon )\in \mathcal {N}_{\lambda ,\beta _n}$ for all n. Now, by (5.4), we have

$$\begin{aligned} 2\varepsilon +m_{a,\lambda }^*+m_{b,\lambda }^*\ge I_{a,\lambda }^*(u_\varepsilon )+I_{b,\lambda }^*(v_\varepsilon )=J_{\lambda ,\beta _n}(u_\varepsilon ,v_\varepsilon )\ge m_{\lambda ,\beta _n}\quad \text {for all }n. \end{aligned}$$

Since $\varepsilon >0$ and $n\in \mathbb {N}$ is arbitrary, we can conclude that

$$\begin{aligned} m_{a,\lambda }^*+m_{b,\lambda }^*\ge \limsup _{n\rightarrow \infty }m_{\lambda ,\beta _n}. \end{aligned}$$

(5.5)

On the other hand, note that $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})$ is the ground state solution $(\mathcal {P}_{\lambda ,\beta _n})$ and $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\rightharpoonup (u_{\lambda ,0}, v_{\lambda ,0})$ weakly in E as $n\rightarrow \infty $, by $\beta _n<0$, we can see that

$$\begin{aligned} \int _{\{u_{\lambda ,0}>0\}}|\nabla u_{\lambda ,0}|^2+(\lambda a(x)+a_0(x))u_{\lambda ,0}^2\mathrm{d}x\le \mu _1\int _{\{u_{\lambda ,0}>0\}}u_{\lambda ,0}^4\mathrm{d}x \end{aligned}$$

and

$$\begin{aligned} \int _{\{v_{\lambda ,0}>0\}}|\nabla v_{\lambda ,0}|^2+(\lambda b(x)+b_0(x))v_{\lambda ,0}^2\mathrm{d}x\le \mu _2\int _{\{v_{\lambda ,0}>0\}}v_{\lambda ,0}^4\mathrm{d}x. \end{aligned}$$

It follows from (5.2) and (5.3) that there exist $0<t_0\le 1$ and $0<s_0\le 1$ such that $t_0u_{\lambda ,0}\in \mathcal {N}_{a,\lambda }^*$ and $s_0v_{\lambda ,0}\in \mathcal {N}_{b,\lambda }^*$. Now, since $(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\rightharpoonup (u_{\lambda ,0}, v_{\lambda ,0})$ weakly in E as $n\rightarrow \infty $, by a similar argument as (3.13), we can see that

$$\begin{aligned} \liminf _{n\rightarrow +\infty }m_{\lambda ,\beta _n}= & {} \liminf _{n\rightarrow +\infty } J_{\lambda ,\beta _n}(u_{\lambda ,\beta _n}, v_{\lambda ,\beta _n})\nonumber \\= & {} \frac{1}{4}\liminf _{n\rightarrow +\infty }(\Vert u_{\lambda ,\beta _n}\Vert _{a,\lambda }^2 +\Vert v_{\lambda ,\beta _n}\Vert _{b,\lambda }^2)\nonumber \\\ge & {} \frac{1}{4}(\Vert u_{\lambda ,0}\Vert _{a,\lambda }^2 +\Vert v_{\lambda ,0}\Vert _{b,\lambda }^2)\nonumber \\\ge & {} \frac{1}{4}(\Vert t_0u_{\lambda ,0}\Vert _{a,\lambda }^2 +\Vert s_0v_{\lambda ,0}\Vert _{b,\lambda }^2)\nonumber \\= & {} I_{a,\lambda }^*(t_0u_{\lambda ,0})+I_{b,\lambda }^*(s_0v_{\lambda ,0})\nonumber \\\ge & {} m_{a,\lambda }^*+m_{b,\lambda }^*. \end{aligned}$$

(5.6)

Hence, by combining (5.5) and (5.6), we must have the following results:

$(a^{***})$ :: $\lim _{n\rightarrow \infty }\Vert u_{\lambda ,\beta _n}\Vert _{a,\lambda }^2=\Vert u_{\lambda ,0}\Vert _{a,\lambda }^2$ and $\lim _{n\rightarrow \infty }\Vert v_{\lambda ,\beta _n}\Vert _{b,\lambda }^2=\Vert v_{\lambda ,0}\Vert _{b,\lambda }^2$.
$(b^{***})$ :: $u_{\lambda ,0}\in \mathcal {N}_{a,\lambda }^*$ with $I_{a,\lambda }^*(u_{\lambda ,0})=m_{a,\lambda }^*$ and $v_{\lambda ,0}\in \mathcal {N}_{b,\lambda }^*$ with $I_{b,\lambda }^*(v_{\lambda ,0})=m_{b,\lambda }^*$.

By $(a^{***})$ and Lemma 2.1, we know that $\Vert u_{\lambda ,\beta _n}-u_{\lambda ,0}\Vert _{a,\lambda }=\Vert v_{\lambda ,\beta _n}-v_{\lambda ,0}\Vert _{b,\lambda }=o_n(1)$. Thanks to Lemma 2.1 once more and the condition $(D_4)$, we observe that $\Vert u_{\lambda ,\beta _n}-u_{\lambda ,0}\Vert _a=\Vert v_{\lambda ,\beta _n}-v_{\lambda ,0}\Vert _b=o_n(1)$ for $\lambda \ge \Lambda _1$. Since by (5.2) and (5.3), $\mathcal {N}_{a,\lambda }^*$ and $\mathcal {N}_{b,\lambda }^*$ are a natural constraint in $H_0^1(\{u_{\lambda ,0}>0\})$ and $H_0^1(\{v_{\lambda ,0}>0\})$, respectively, which together with $(b^{***})$ implies $u_{\lambda ,0}$ and $v_{\lambda ,0}$ are a least energy solution of (1.3) and (1.4), respectively.

Step 5 We prove that $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}$ and $\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}$ are connected domains and $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}=\mathbb {R}^3\backslash \overline{\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}}$.

Indeed, since $u_{\lambda ,0}$ and $v_{\lambda ,0}$ are, respectively, a least energy solution of (1.3) and (1.4), we must have $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}$ and $\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}$ are connected domains. It remains to show that $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}=\mathbb {R}^3\backslash \overline{\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}}$. To the contrary, we suppose that there exists an open set $\Omega $ satisfying $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}\subsetneqq \Omega \subsetneqq \mathbb {R}^3\backslash \overline{\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}}$. Furthermore, it has a locally lipschitz boundary. Since $\Omega \subsetneqq \mathbb {R}^3\backslash \overline{\{x\in \mathbb {R}^3\mid v_{\lambda ,0}(x)>0\}}$, by a similar argument as in Step 3, we can show that $u_{\lambda ,0}$ is a least energy solution of the following equation:

$$\begin{aligned} -\Delta u+(\lambda a(x)+a_0(x))u=\mu _1 u^3,\quad u\in H_0^1(\Omega ). \end{aligned}$$

Since $u_{\lambda ,0}\ge 0$ in $\Omega $, by a similar argument as in the proof of Theorem 1.1, we can conclude that $u_{\lambda ,0}>0$ on $\Omega $, which contradicts $\{x\in \mathbb {R}^3\mid u_{\lambda ,0}(x)>0\}\subsetneqq \Omega $.

We complete the proof by showing that $\beta ^2\int _{\mathbb {R}^3}u_{\lambda ,\beta }^2v_{\lambda ,\beta }^2\rightarrow 0$ as $\beta \rightarrow -\infty $, where $(u_{\lambda ,\beta },v_{\lambda ,\beta })$ is the ground state solution of $(\mathcal {P}_{\lambda ,\beta })$ obtained by Theorem 1.1. Indeed, if not, then there exists $\{\beta _n\}\subset (-\infty , 0)$ such that $\beta _n^2\int _{\mathbb {R}^3}u_{\lambda ,\beta _n}^2v_{\lambda ,\beta _n}^2\le -C$. By (5.5) and (5.6), we must have $\beta _n^2\int _{\mathbb {R}^3}u_{\lambda ,\beta _n}^2v_{\lambda ,\beta _n}^2\rightarrow 0$ as $\beta _n\rightarrow -\infty $ up to a subsequence, which is a contradiction. $\square $

References

Ambrosetti, A., Rabinowitz, P.: Dual variational methods in critical point theory and applications. J. Funct. Anal. 14, 349–381 (1973)
Article MathSciNet MATH Google Scholar
Akhmediev, N., Ankiewicz, A.: Partially coherent solitons on a finite background. Phys. Rev. Lett. 82, 2661–2664 (1999)
Article Google Scholar
Bartsch, T., Wang, Z.-Q.: Existence and multiplicity results for superlinear elliptic problems on $\mathbb{R}^n$. Commun. Partial Differ. Equ. 20, 1725–1741 (1995)
Article MathSciNet MATH Google Scholar
Bartsch, T., Wang, Z.-Q.: Multiple positive solutions for a nonlinear Schrödinger equations. Z. Angew. Math. Phys. 51, 366–384 (2000)
Article MathSciNet MATH Google Scholar
Bartsch, T., Pankov, A., Wang, Z.-Q.: Nonlinear Schrödinger equations with steep potential well. Commun. Contemp. Math. 3, 549–569 (2001)
Article MathSciNet MATH Google Scholar
Brown, K., Zhang, Y.: The Nehari manifold for a semilinear elliptic equation with a sign-changing weight function. J. Differ. Equ. 193, 481–499 (2003)
Article MathSciNet MATH Google Scholar
Bartsch, T., Wang, Z.-Q., Wei, W.: Bound states for a coupled Schrödinger system. J. Fixed Point Theory Appl. 2, 353–367 (2007)
Article MathSciNet MATH Google Scholar
Bartsch, T., Dancer, N., Wang, Z.-Q.: A Liouville theorem, a-priori bounds, and bifurcating branches of positive solutions for a nonlinear elliptic system. Calc. Var. Partial Differ. Equ. 37, 345–361 (2010)
Article MathSciNet MATH Google Scholar
Bartsch, T.: Bifurcation in a multicomponent system of nonlinear Schrödinger equations. J. Fixed Point Theory Appl. 13, 37–50 (2013)
Article MathSciNet MATH Google Scholar
Bartsch, T., Tang, Z.: Multibump solutions of nonlinear Schrödinger equations with steep potential well and indefinite potential. Discrete Contin. Dyn. Syst. 33, 7–26 (2013)
MathSciNet MATH Google Scholar
Caffarelli, L.A., Lin, F.-H.: Singularly perturbed elliptic systems and multi-valued harmonic functions with free boundaries. J. Am. Math. Soc. 21, 847–862 (2008)
Article MathSciNet MATH Google Scholar
Caffarelli, L.A., Roquejoffre, J.M.: Uniform Höder estimates in a class of elliptic systems and applications to singular limits in models for diffusion flames. Arch. Ration. Mech. Anal. 183, 457–487 (2007)
Article MathSciNet MATH Google Scholar
Chen, Z., Zou, W.: Positive least energy solutions and phase separation for coupled Schrödinger equations with critical exponent. Arch. Ration. Mech. Anal. 205, 515–551 (2012)
Article MathSciNet MATH Google Scholar
Chen, Z., Zou, W.: Ground states for a system of Schrödinger equations with critical exponent. J. Funct. Anal. 262, 3091–3107 (2012)
Article MathSciNet MATH Google Scholar
Chen, Z., Zou, W.: An optimal constant for the existence of least energy solutions of a coupled Schrödinger system. Calc. Var. Partial Differ. Equ. 48, 695–711 (2013)
Article MATH Google Scholar
Chen, Z., Zou, W.: Positive least energy solutions and phase separation for coupled Schrödinger equations with critical exponent: higher dimensional case. Calc. Var. Partial Differ. Equ. (accepted). Arxiv:1209.2522v2 [math.AP]
Chen, Z., Zou, W.: Standing waves for coupled Schrödinger equations with decaying potentials. J. Math. Phys. 54(111505), 1–21 (2013)
MATH Google Scholar
Chen, Z., Zou, W.: Existence and symmetry of positive ground states for a doubly critical Schrödinger system. Trans. Am. Math. Soc. (accepted). Arxiv:1211.1514v2 [math.AP]
Chen, Z., Lin, C.S., Zou, W.: Multiple sign-changing and semi-nodal solutions for coupled Schrödinger equations. J. Differ. Equ. 255, 4289–4311 (2013)
Article MATH Google Scholar
Chen, Z., Lin, C.S., Zou, W.: Infinitely many sign-changing and semi-nodal solutions for a nonlinear Schrödinger system. Arxiv:1212.3773v1 [math.AP]
Chen, Z., Lin, C.S., Zou, W.: Sign-changing solutions and phase separation for an elliptic system with critical exponent. Commun. Partial Differ. Equ. 39, 1827–1859 (2014)
Article MathSciNet MATH Google Scholar
Conti, M., Terracini, S., Verzini, G.: Asymptotic estimates for the spatial segregation of competitive systems. Adv. Math. 195, 524–560 (2005)
Article MathSciNet MATH Google Scholar
del Pino, M., Felmer, P.: Local mountain passes for semilinear elliptic problems in unbounded domains. Calc. Var. Partial Differ. Equ. 4, 121–137 (1996)
Article MathSciNet MATH Google Scholar
Ding, Y., Tanaka, K.: Multiplicity of positive solutions of a nonlinear Schrödinger equation. Manuscr. Math. 112, 109–135 (2003)
Article MATH Google Scholar
Esry, B., Greene, C., Burke, J., Bohn, J.: Hartree–Fock theory for double condensates. Phys. Rev. Lett. 78, 3594–3597 (1997)
Article Google Scholar
Furtado, M., Silva, E., Xavier, M.: Multiplicity and concentration of solutions for elliptic systems with vanishing potentials. J. Differ. Equ. 249, 2377–2396 (2010)
Article MathSciNet MATH Google Scholar
Guo, Y., Tang, Z.: Ground state solutions for quasilinear Schrödinger systems. J. Math. Anal. Appl. 389, 322–339 (2012)
Article MathSciNet MATH Google Scholar
Guo, Y., Tang, Z.: Multibump bound states for quasilinear Schrödinger systems with critical frequency. J. Fixed Point Theory Appl. 12, 135–174 (2012)
Article MathSciNet MATH Google Scholar
Hirano, N.: Multiple existence of nonradial positive solutions for a coupled nonlinear Schrödinger system. NoDEA Nonlinear Differ. Equ. Appl. 16, 159–188 (2009)
Article MATH Google Scholar
Ikoma, N.: Uniqueness of positive solutions for a nonlinear elliptic system. NoDEA Nonlinear Differ. Equ. Appl. 16, 555–567 (2009)
Article MathSciNet MATH Google Scholar
Ikoma, N., Tanaka, K.: A local mountain pass type result for a system of nonlinear Schrödinger equations. Calc. Var. Partial Differ. Equ. 40, 449–480 (2011)
Article MATH Google Scholar
Jiang, Y., Zhou, H.-S.: Schrödinger–Poisson system with steep potential well. J. Differ. Equ. 251, 582–608 (2011)
Article MATH Google Scholar
Lin, T.C., Wei, W.: Spikes in two coupled nonlinear Schrödinger equations. Ann. Inst. H. Poincaré Anal. Non Linéaire 22, 403–439 (2005)
Article MathSciNet MATH Google Scholar
Lin, T.C., Wei, W.: Spikes in two-component systems of nonlinear Schrödinger equations with trapping potentials. J. Differ. Equ. 229, 538–569 (2006)
Article MATH Google Scholar
Liu, X., Huang, Y., Liu, J.: Sign-changing solutions for an asymptotically linear Schrödinger equations with deepening potential well. Adv. Differ. Equ. 16, 1–30 (2011)
MATH Google Scholar
Lin, T.C., Wu, T.F.: Existence and multiplicity of positive solutions for two coupled nonlinear Schrödinger equations. Discret. Contin. Dyn. Syst. 33, 2911–2938 (2013)
Article MATH Google Scholar
Maia, L.A., Silva, E.: On a class of coupled elliptic systems in $\mathbb{R}^n$. NoDEA Nonlinear Differ. Equ. Appl. 14, 303–313 (2007)
Article MathSciNet MATH Google Scholar
Noris, B., Tavares, H., Terracini, S., Verzini, G.: Uniform Hölder bounds for nonlinear Schrödinger systems with strong competition. Commun. Pure Appl. Math. 63, 267–302 (2010)
Article MATH Google Scholar
Noris, B., Verzini, G.: A remark on natural constraints in variational methods and an application to superlinear Schrödinger systems. J. Differ. Equ. 254(3), 1529–1547 (2013)
Article MATH Google Scholar
Royo-Letelier, J.: Segregation and symmetry breaking of strongly coupled two-component Bose–Einstein condensates in a harmonic trap. Calc. Var. Partial Differ. Equ. 49, 103–124 (2014)
Article MathSciNet MATH Google Scholar
Séré, E.: Existence of infinitely many homoclinic orbits in Hamiltonian systems. Math. Z. 209, 27–42 (1992)
Article MathSciNet MATH Google Scholar
Stuart, C., Zhou, H.-S.: Positive eigenfunctions of a Schröinger operator. J. London Math. Soc. 72, 429–441 (2005)
Article MathSciNet MATH Google Scholar
Stuart, C., Zhou, H.-S.: Global branch of solutions for nonlinear Schrödinger equations with deepening potential well. Proc. London Math. Soc. 92, 655–681 (2006)
Article MathSciNet MATH Google Scholar
Sirakov, B.: Least energy solitary waves for a system of nonlinear Schrödinger equations in $\mathbb{R}^n$. Commun. Math. Phys. 271, 199–221 (2007)
Article MATH Google Scholar
Sato, Y., Tanaka, K.: Sign-changing multi-bump solutions nonlinear Schrödinger equations with steep potential wells. Trans. Am. Math. Soc. 361, 6205–6253 (2009)
Article MATH Google Scholar
Sun, J., Wu, T.F.: Ground state solutions for an indefinite Kirchhoff type problem with steep potential well. J. Differ. Equ. 256, 1771–1792 (2014)
Article MathSciNet MATH Google Scholar
Soave, N., Zilio, A.: Uniform bounds for strongly competing systems: the optimal Lipschitz case. arXiv:1407.6674v1 [math.AP]
Terracini, S., Verzini, G.: Multipulse phases in k-mixtures of Bose–Einstein condensates. Arch. Ration. Mech. Anal. 194, 717–741 (2009)
Article MathSciNet MATH Google Scholar
Wei, J., Weth, T.: Radial solutions and phase separation in a system of two coupled Schrödinger equations. Arch. Ration. Mech. Anal. 190, 83–106 (2008)
Article MathSciNet MATH Google Scholar
Wei, J., Weth, T.: Asymptotic behaviour of solutions of planar elliptic systems with strong competition. Nonlinearity 21, 305–317 (2008)
Article MathSciNet MATH Google Scholar
Wang, Z., Zhou, H.-S.: Positive solutions for nonlinear Schrödinger equations with deepening potential well. J. Eur. Math. Soc. 11, 545–573 (2009)
Article MathSciNet MATH Google Scholar
Zhao, L., Liu, H., Zhao, F.: Existence and concentration of solutions for the Schrödinger–Poisson equations with steep well potential. J. Differ. Equ. 255, 1–23 (2013)
Article MATH Google Scholar

Download references

Acknowledgements

This work was partly completed when the first author was visiting Tsinghua University, and he is grateful to the members in the Department of Mathematical Sciences at Tsinghua University for their invitation and hospitality. The first author was supported by NSFC (11626226). The second author was supported by the National Science Council and the National Center for Theoretical Sciences (South), Taiwan. The third author was supported by NSFC (11025106, 11371212, 11271386) and the Both-Side Tsinghua Fund. Y. Wu and W. Zou thank Prof. A. Szulkin for his suggestions and discussions when he was visiting Tsinghua University. They are also grateful to Dr. Zhijie Chen for careful reading the manuscript and some suggestions to improve it. The authors also would like to thank the anonymous referee for carefully reading the manuscript and valuable comments that greatly helped improve this paper, especially for pointing out Remarks 1.3 and 1.6.

Author information

Authors and Affiliations

College of Sciences, China University of Mining and Technology, Xuzhou, 221116, People’s Republic of China
Yuanze Wu
Department of Applied Mathematics, National University of Kaohsiung, Kaohsiung, 811, Taiwan
Tsung-fang Wu
Department of Mathematical Sciences, Tsinghua University, Beijing, 100084, People’s Republic of China
Wenming Zou

Authors

Yuanze Wu
View author publications
You can also search for this author in PubMed Google Scholar
Tsung-fang Wu
View author publications
You can also search for this author in PubMed Google Scholar
Wenming Zou
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuanze Wu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Wu, Y., Wu, Tf. & Zou, W. On a two-component Bose–Einstein condensate with steep potential wells. Annali di Matematica 196, 1695–1737 (2017). https://doi.org/10.1007/s10231-017-0635-6

Download citation

Received: 26 July 2016
Accepted: 24 January 2017
Published: 06 February 2017
Issue Date: October 2017
DOI: https://doi.org/10.1007/s10231-017-0635-6

On a two-component Bose–Einstein condensate with steep potential wells

Abstract

Similar content being viewed by others

Existence and local uniqueness of normalized solutions for two-component Bose–Einstein condensates

A perturbation method for k-mixtures of Bose–Einstein condensates

Expansion dynamics of a two-component quasi-one-dimensional Bose–Einstein condensate: Phase diagram, self-similar solutions, and dispersive shock waves

1 Introduction

Remark 1.1

Remark 1.2

Theorem 1.1

Remark 1.3

Remark 1.4

Theorem 1.2

Corollary 1.1

Remark 1.5

Theorem 1.3

Remark 1.6

2 The variational setting

Lemma 2.1

Proof

Remark 2.1

3 A ground state solution

Lemma 3.1

Proof

Lemma 3.2

Proof

Lemma 3.3

Proof

Lemma 3.4

Proof

Proposition 3.1

Proof

Proof of Theorem 1.1

4 Multi-bump solutions

4.1 The penalized functional and the (PS) condition

Lemma 4.1

Proof

Lemma 4.2

Proof

Lemma 4.3

Proof

4.2 Construction of critical points

Lemma 4.4

Proof

Lemma 4.5

Proof

Proposition 4.1

Proof

Proposition 4.2

Proof

Proof of Theorem 1.2

5 The phenomenon of phase separations

Proof of Theorem 1.3

References

Acknowledgements

Author information

Authors and Affiliations

Corresponding author

Rights and permissions

About this article

Cite this article

Share this article

Keywords

Mathematics Subject Classification

Search

Navigation