If you followed the installation instructions for Apigee hybrid for GKE and own the domain you used for your Cloud DNS zone, you already have all the necessary components to provision trusted SSL certificates for your API endpoints. Having certificates that are issued by a trusted CA is important, especially when exposing public APIs, as the certificates establish a chain of trust and prove domain ownership.
This post quickly outlines the necessary steps to configure an Cert-Manager issuer for Let’s Encrypt and how to use the automatically provisioned certificates on your Apigee hybrid ingress.
To recap the Apigee hybrid installation instructions, we assume you have provisioned and configured the following:
The approach described below is only one possible option to automatically provision trusted certificates for the Apigee hybrid ingress. Other options include:
In order to allow Cert-Manager to configure the necessary records in Cloud DNS to obtain a trusted certificate, we need to authorize it using a GCP service account. The following commands create a service account in GCP and add its key as a Kubernetes secret.
gcloud iam service-accounts create dns01-solver --display-name "dns01-solver" gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/dns.admin gcloud iam service-accounts keys create key.json \ --iam-account dns01-solver@$PROJECT_ID.iam.gserviceaccount.com kubectl create secret generic clouddns-dns01-solver-svc-acct \ --from-file=key.json -n istio-system
Because we already installed Cert-Manager and set up a Cloud DNS zone as part of the Apigee hybrid installation, we only need to create a Kubernetes resource to declare a Cert-Manager issuer based on Cloud DNS entries:
cat <<EOF | kubectl apply -f - apiVersion: cert-manager.io/v1alpha2 kind: Issuer metadata: name: cloud-dns-issuer namespace: istio-system spec: acme: email: cert-admin@your-domain.com server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: cloud-dns-issuer-account-key solvers: - dns01: clouddns: project: $PROJECT_ID serviceAccountSecretRef: name: clouddns-dns01-solver-svc-acct key: key.json EOF
And then create a Certificate resource to start the request for a trusted certificate for our API ingress gateway:
cat <<EOF | kubectl apply -f - apiVersion: cert-manager.io/v1alpha2 kind: Certificate metadata: name: cert-manager-secret-$PROJECT_ID-default namespace: istio-system spec: secretName: cert-manager-secret-$PROJECT_ID-default issuerRef: name: cloud-dns-issuer commonName: '*.apigee.example.com' # Change this dnsNames: - apigee.example.com # Change this - '*.apigee.example.com' # Change this EOF
After a few minutes the status of your certificate resource will change to “ready”:
kubectl get Certificate cert-manager-secret-$PROJECT_ID-default -n istio-system
And your TLS secret is ready to be used:
kubectl get secret cert-manager-secret-$PROJECT_ID-default -n istio-system
First off, we need to locate the ApigeeRouteConfig that corresponds to the ingress we would like to add our new certificate to:
kubectl get apigeerouteconfigs.apigee.cloud.google.com -n apigee
We then replace the secret in the ApigeeRouteConfig to point to our newly created and Cert-Manager controlled secret:
cat <<EOF | kubectl apply -f - apiVersion: apigee.cloud.google.com/v1alpha1 kind: ApigeeRouteConfig metadata: name: $ROUTE_CONFIG_NAME_FROM_BEFORE # Change this namespace: apigee spec: selector: app: istio-ingressgateway connectTimeout: 300 tls: mode: SIMPLE secretNameRef: "istio-system/cert-manager-secret-$PROJECT_ID-default" # Change this EOF
Note that the Apigee hybrid quickstart in Apigee DevRel is now automatically creating let's encrypt certificates. The steps above are still needed for manual installations.
Edit (8/9/2021): Note that the Apigee hybrid quickstart in Apigee DevRel is now automatically creating let's encrypt certificates. The steps above are still needed for manual installations. You can also use an L7 load balancer with your Apigee hybrid deployment and use trusted google managed certificates directly on the load balancer as described in this article.
Awesome! Love this, Daniel.
This was super helpful, thanks Daniel! I had a problem that my sub-domain wasn't properly configured to the Cloud DNS NS (as NS record in the Domain DNS config), but once that was corrected, everything worked perfectly.
The original post from Daniel states:
This post quickly outlines the necessary steps to configure a Cert-Manager issuer for Let’s Encrypt and how to use the automatically provisioned certificates on your Apigee hybrid ingress.
The original article is still relevant, but with hybrid 1.8, things are a little different. For one thing, the namespaces change. For another, the yaml changes slightly.
I'll describe what I did to get this to work with Hybrid 1.8. The main differences:
- I'm going to use the apigee namespace for the issuer, not the istio-system namespace
- my cloud DNS is in a different GCP project than the kubernetes cluster. This is just how I have my cloud DNS set up, it is unrelated to the switch to hybrid 1.8.
gcloud config set project my-gcp-project-containing-gke
export PROJECT_ID=my-gcp-project-containing-gke
export DNS_PROJECT_ID=gcp-project-that-hosts-cloud-DNS
gcloud iam service-accounts create dns01-solver --display-name "dns01-solver" --project $PROJECT_ID
gcloud projects add-iam-policy-binding $DNS_PROJECT_ID \
--member serviceAccount:dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/dns.admin
gcloud iam service-accounts keys create key.json \
--iam-account dns01-solver@$PROJECT_ID.iam.gserviceaccount.com
kubectl create secret generic clouddns-dns01-solver-svc-acct \
--from-file=key.json -n apigee
Then, the issuer. Moving to cert-manager.io/v1 , this must use cloudDNS as the name of the solver, not clouddns. I think the case is significant.
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: cloud-dns-issuer
namespace: apigee
spec:
acme:
email: dchiesa.cert-admin@google.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: cloud-dns-issuer-account-key
solvers:
- dns01:
cloudDNS:
project: $DNS_PROJECT_ID
serviceAccountSecretRef:
name: clouddns-dns01-solver-svc-acct
key: key.json
EOF
Then the certificate:
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: cert-manager-$PROJECT_ID-default
namespace: apigee
spec:
secretName: cert-manager-$PROJECT_ID-tls
issuerRef:
name: cloud-dns-issuer
commonName: '*.apigee-hybrid.my-domain.com'
dnsNames:
- eval.apigee-hybrid.my-domain.com
- '*.apigee-hybrid.my-domain.com'
EOF
That configuration assumes that I have my cloud DNS already resolving eval.apigee-hybrid.my-domain.com to the IP address of the TCP External Load Balancer for my hybrid cluster.
To check status of that update, I used
watch kubectl get Certificate -n apigee cert-manager-$PROJECT_ID-default -o yaml
And after about 60 seconds it showed as "Ready". Then, to get those warm-n-fuzzy feelings, check the secret:
kubectl get secret cert-manager-$PROJECT_ID-tls -n apigee -o yaml
And finally, check the ApigeeRouteConfig :
kubectl get apigeerouteconfigs.apigee.cloud.google.com -n apigee $ROUTE_CONFIG_NAME -o yaml
And update it with the appropriate new Secret:
cat <<EOF | kubectl apply -f -
apiVersion: apigee.cloud.google.com/v1alpha1
kind: ApigeeRouteConfig
metadata:
name: $ROUTE_CONFIG_NAME
namespace: apigee
spec:
connectTimeout: 300
selector:
app: apigee-ingressgateway
tls:
mode: SIMPLE
secretNameRef: cert-manager-$PROJECT_ID-tls
EOF
After that, I was able to invoke my hybrid ingress controller using 1-way TLS validation. In other words I was able to do:
curl https://api.hybrid-apigee.my-domain.com/basepath
...and the TLS was properly negotiated.