

Deploy HTTPS web application on Kubernetes with Citrix ingress controller and Hashicorp Vault using cert-manager

This topic provides a sample workflow that uses HashiCorp Vault as self-signed CA to automate TLS certificate provisioning, revocation, and renewal for ingress resources deployed with Citrix ingress controller using cert-manager.

Specifically, the workflow uses the Vault PKI Secrets Engine to create a CA. This tutorial assumes that you have a Vault server installed and reachable from the Kubernetes cluster. The PKI secrets engine of Vault is suitable for internal applications and for external facing applications that require public trust, you can refer automating TLS certificates using letsencrypt CA

The workflow uses a Vault secret engine and authentication methods, refer the following Vault documentation for full list of features:

• Vault Secrets Engines

• Vault Authentication Methods

This topic provides you information on how to deploy an HTTPS web application on a Kubernetes cluster, using:

• Citrix ingress controller
• JetStack's cert-manager to provision TLS certificates from Hashicorp Vault
• Hashicorp Vault

Prerequisites

Ensure that you have:

• The Vault server is installed, unsealed and is reachable from Kubernetes cluster.

• Enabled RBAC on your Kubernetes cluster.

• Deployed Citrix ADC MPX, VPX or CPX in Tier 1 or Tier 2 deployment model.

  In the Tier 1 deployment model, Citrix ADC MPX or VPX is used as an Application Delivery Controller (ADC) and Citrix ingress controller running in Kubernetes cluster configures the virtual services for the services running on Kubernetes cluster. Citrix ADC runs the virtual service on the publicly routable IP address and offloads SSL for client traffic with the help of Let's Encrypt generated certificate.

  Similarly in the Tier 2 deployment model, a TCP service is configured on the Citrix ADC (VPX/MPX) running outside the Kubernetes cluster to forward the traffic to Citrix ADC CPX instances running in the Kubernetes cluster. Citrix ADC CPX ends the SSL session and load-balances the traffic to actual service pods.

• Deployed Citrix ingress controller. Click here for various deployment scenarios.

• Administrator permissions for all the deployment steps. If you encounter failures due to permissions, make sure you have administrator permission.

Deploy cert-manager using the manifest file

Perform the following steps to deploy cert-manager using the supplied YAML manifest file. To keep things simple, skip cert-manager's Helm installation, and instead use the supplied YAML manifests.

1. Download the latest source of cert-manager from github.com/jetstack/cert-manager repository using the following command.

   wget https://github.com/jetstack/cert-manager/archive/v0.6.2.tar.gz
   tar -zxvf v0.6.2.tar.gz
   

2. Deploy the cert-manager using the following command.

   kubectl apply -f deploy/manifests/cert-manager.yaml
   

   You can also install the cert-manager with Helm, for more information see cert-manager documentation

3. Verify that the cert-manager is up and running using the following command.

   % kubectl -n cert-manager get all
   NAME                                       READY   STATUS    RESTARTS   AGE
   pod/cert-manager-77fd74fb64-d68v7          1/1     Running   0          4m41s
   pod/cert-manager-webhook-67bf86d45-k77jj   1/1     Running   0          4m41s
   
   NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
   service/cert-manager-webhook   ClusterIP   10.108.161.154   <none>        443/TCP   13d
   
   NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/cert-manager           1/1     1            1           13d
   deployment.apps/cert-manager-webhook   1/1     1            1           13d
   
   NAME                                             DESIRED   CURRENT   READY   AGE
   replicaset.apps/cert-manager-77fd74fb64          1         1         1       13d
   replicaset.apps/cert-manager-webhook-67bf86d45   1         1         1       13d
   
   NAME                                                COMPLETIONS   DURATION   AGE
   job.batch/cert-manager-webhook-ca-sync              1/1           22s        13d
   job.batch/cert-manager-webhook-ca-sync-1549756800   1/1           21s        10d
   job.batch/cert-manager-webhook-ca-sync-1550361600   1/1           19s        3d8h
   
   NAME                                         SCHEDULE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   cronjob.batch/cert-manager-webhook-ca-sync   @weekly    False     0        3d8h            13d
   

Deploy a sample web application

Perform the following steps to deploy a sample web application.

Note

Kuard, a kubernetes demo application is used for reference in this topic.

1. Create a deployment YAML file (kuard-deployment.yaml) for Kuard with the following configuration.

   apiVersion: extensions/v1beta1
   kind: Deployment
   metadata:
     name: kuard
   spec:
     replicas: 1
     template:
       metadata:
         labels:
           app: kuard
       spec:
         containers:
         - image: gcr.io/kuar-demo/kuard-amd64:1
           imagePullPolicy: Always
           name: kuard
           ports:
           - containerPort: 8080
   

2. Deploy Kuard deployment file (kuard-deployment.yaml) to your cluster, using the following commands.

   % kubectl create -f kuard-deployment.yaml
   deployment.extensions/kuard created
   % kubectl get pod -l app=kuard
   NAME                     READY   STATUS    RESTARTS   AGE
   kuard-6fc4d89bfb-djljt   1/1     Running   0          24s
   

3. Create a service for the deployment. Create a file called service.yaml with the following configuration.

   apiVersion: v1
   kind: Service
   metadata:
     name: kuard
   spec:
     ports:
     - port: 80
       targetPort: 8080
       protocol: TCP
     selector:
       app: kuard
   

4. Deploy and verify the service using the following command.

   % kubectl create -f service.yaml
   service/kuard created
   % kubectl get svc kuard
   NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
   kuard   ClusterIP   10.103.49.171   <none>        80/TCP    13s
   

5. Expose this service to outside world by creating an Ingress that is deployed on Citrix ADC CPX or VPX as Content switching virtual server.

   Note

   Ensure that you change kubernetes.io/ingress.class to your ingress class on which Citrix ingress controller is started.

   apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kuard annotations: kubernetes.io/ingress.class: "citrix" spec: rules: - host: kuard.example.com http: paths: - backend: serviceName: kuard servicePort: 80

   Important

   Change the value of spec.rules.host to the domain that you control. Ensure that a DNS entry exists to route the traffic to Citrix ADC CPX or VPX.

6. Deploy the Ingress using the following command.

   % kubectl apply -f ingress.yml
   ingress.extensions/kuard created
   root@ubuntu-vivek-225:~/cert-manager# kubectl get ingress
   NAME    HOSTS               ADDRESS   PORTS   AGE
   kuard   kuard.example.com             80      7s
   

7. Verify if the ingress is configured on Citrix ADC CPX or VPX using the following command.

   kubectl exec -it cpx-ingress-5b85d7c69d-ngd72 /bin/bash
   root@cpx-ingress-5b85d7c69d-ngd72:/# cli_script.sh 'sh cs vs'
   exec: sh cs vs
   1) k8s-10.244.1.50:80:http (10.244.1.50:80) - HTTP Type: CONTENT
     State: UP
     Last state change was at Thu Feb 21 09:02:14 2019
     Time since last state change: 0 days, 00:00:41.140
     Client Idle Timeout: 180 sec
     Down state flush: ENABLED
     Disable Primary Vserver On Down : DISABLED
     Comment: uid=75VBGFO7NZXV7SCI4LSDJML2Q5X6FSNK6NXQPWGMDOYGBW2IMOGQ====
     Appflow logging: ENABLED
     Port Rewrite : DISABLED
     State Update: DISABLED
     Default: Content Precedence: RULE
     Vserver IP and Port insertion: OFF
     L2Conn: OFF Case Sensitivity: ON
     Authentication: OFF
     401 Based Authentication: OFF
     Push: DISABLED Push VServer:
     Push Label Rule: none
     Listen Policy: NONE
     IcmpResponse: PASSIVE
     RHIstate:  PASSIVE
     Traffic Domain: 0
   Done
   root@cpx-ingress-5b85d7c69d-ngd72:/# exit
   exit
   

8. Verify if the page is correctly being served when requested using the curl command.

   % curl -sS -D - kuard.example.com -o /dev/null
   HTTP/1.1 200 OK
   Content-Length: 1458
   Content-Type: text/html
   Date: Thu, 21 Feb 2019 09:09:05 GMT
   

Configure Hashicorp Vault as Certificate Authority

Set up an intermediate CA certificate signing request using Hashicorp Vault. This Vault endpoint is used by cert-manager to sign the certificate for the ingress resources.

Prerequisites

Ensure that you have installed the jq utility.

Create a root CA

For the sample workflow you can generate your own Root Certificate Authority within the Vault. In a production environment, you should use an external Root CA to sign the intermediate CA that Vault uses to generate certificates. If you have a root CA generated elsewhere, skip this step.

Note

PKI_ROOT is a path where you mount the root CA, typically it is 'pki'. ${DOMAIN} in this procedure is example.com

% vault secrets enable -path="${PKI_ROOT}" pki

# Set the max TTL for the root CA to 10 years
% vault secrets tune -max-lease-ttl=87600h "${PKI_ROOT}"

% vault write -format=json "${PKI_ROOT}"/root/generate/internal \
 common_name="${DOMAIN} CA root" ttl=87600h | tee \
>(jq -r .data.certificate > ca.pem) \
>(jq -r .data.issuing_ca > issuing_ca.pem) \
>(jq -r .data.private_key > ca-key.pem)

#Configure the CA and CRL URLs:

% vault write "${PKI_ROOT}"/config/urls \
       issuing_certificates="${VAULT_ADDR}/v1/${PKI_ROOT}/ca" \
       crl_distribution_points="${VAULT_ADDR}/v1/${PKI_ROOT}/crl"

Generate an intermediate CA

After creating the root CA, perform the following steps to create an intermediate CSR using the root CA.

1. Enable pki from a different path PKI_INT from root CA, typically pki\_int. Use the following command:

   % vault secrets enable -path=${PKI_INT} pki
   Set the max TTL to 3 year
   
   % vault secrets tune -max-lease-ttl=26280h ${PKI_INT}
   

2. Generate CSR for ${DOMAIN} that needs to be signed by the root CA, the key is stored internally to vault. Use the following command:

   % vault write -format=json "${PKI_INT}"/intermediate/generate/internal \
   common_name="${DOMAIN} CA intermediate" ttl=43800h | tee \
   >(jq -r .data.csr > pki_int.csr) \
   >(jq -r .data.private_key > pki_int.pem)
   

3. Generate and sign the ${DOMAIN} certificate as an intermediate CA using root CA, store it as intermediate.cert.pem. Use the following command:

   % vault write -format=json "${PKI_ROOT}"/root/sign-intermediate csr=@pki_int.csr \
           format=pem_bundle \
           | jq -r '.data.certificate' > intermediate.cert.pem
   

   If you are using an external root CA, skip the above step and sign the CSR manually using root CA.

4. Once the CSR is signed and the root CA returns a certificate, it needs to added back into the Vault using the following command:

   % vault write "${PKI_INT}"/intermediate/set-signed certificate=@intermediate.cert.pem
   

5. Set the CA and CRL location using the following command.

   vault write "${PKI_INT}"/config/urls issuing_certificates="${VAULT_ADDR}/v1/${PKI_INT}/ca" crl_distribution_points="${VAULT_ADDR}/v1/${PKI_INT}/crl"
   

An intermediate CA is setup and can be used to sign certificates for ingress resources.

Configure a role

A role is a logical name which maps to policies, administrator can control the certificate generation through the roles.

Create a role for the intermediate CA that provides a set of policies for issuing or signing the certificates using this CA.

There are many configurations that can be configured when creating roles, for more information see, Vault role documentation.

For the workflow, create a role "kube-ingress" that allows you to sign certificates of ${DOMAIN} and its subdomains with a TTL of 90 days.

# with a Max TTL of 90 days
vault write ${PKI_INT}/roles/kube-ingress \
          allowed_domains=${DOMAIN} \
          allow_subdomains=true \
          max_ttl="2160h"

Create Approle based authentication

After configuring an intermediate CA to sign the certificates, you need to provide an authentication mechanism for cert-manager to use the vault for signing the certificates. Cert-manager supports Approle authentication method which provides a way for the applications to access the Vault defined roles.

An "AppRole" represents a set of Vault policies and login constraints that must be met to receive a token with those policies. For detailed understanding of this authentication method, see Approle documentation.

Create an Approle

Create an approle named "Kube-role". The secret id for cert-manager should not be expired to use this Approle for authentication, hence do not set a TTL, or set it to 0.

% vault auth enable approle of token ttl 5 minutes

% vault write auth/approle/role/kube-role \
    token_ttl=5m \
    token_max_ttl=10m

Associate a policy with the Approle

Perform the following steps to associate a policy with an Approle.

1. Create a file pki_int.hcl with the following configuration to allow the signing endpoints of the intermediate CA.

   path "${PKI_INT}/sign/*" {
         capabilities = ["create","update"]
       }
   

2. Add the file to a new policy called kube_allow_sign using the following command.

   vault policy write kube-allow-sign pki_int.hcl
   

3. Update this policy to the approle using the following command.

   vault write auth/approle/role/kube-role policies=kube-allow-sign
   

The kube-role approle allows you to sign the CSR with intermediate CA.

Generate the Role id and Secret id

Role id and Secret id is used by cert-manager to authenticate with Vault.

Generate role id and Secret id and encode the secret_id with Base64. Perform the following:

% vault read auth/approle/role/kube-role/role-id
role_id     db02de05-fa39-4855-059b-67221c5c2f63

% vault write -f auth/approle/role/kube-role/secret-id
secret_id               6a174c20-f6de-a53c-74d2-6018fcceff64
secret_id_accessor      c454f7e5-996e-7230-6074-6ef26b7bcf86

# encode secret_id with base64
% echo 6a174c20-f6de-a53c-74d2-6018fcceff64 | Base64
NmExNzRjMjAtZjZkZS1hNTNjLTc0ZDItNjAxOGZjY2VmZjY0Cg==

Configure issuing certificates in Kubernetes

After you have configured the Vault as intermediate CA, and the Approle authentication method for cert-manager to access the Vault. You need to configure the certificate for the ingress.

Create a secret with Approle secret-id

Perform the following to create a secret with Approle secret id.

1. Create a secret file called secretid.yaml with following configuration.

   apiVersion: v1
   kind: Secret
   type: Opaque
   metadata:
     name: cert-manager-vault-approle
     namespace: cert-manager
   data:
     secretId: "NmExNzRjMjAtZjZkZS1hNTNjLTc0ZDItNjAxOGZjY2VmZjY0Cg=="
   

   Note

   data.secretId is the base64 encoded Secret Id generated in Generate the Role id and Secret id. If you're using an Issuer resource in the next step, Secret must be in same namespace as the Issuer. For ClusterIssuer, secret must be in cert-manager namespace.

2. Deploy the secret file (secretid.yaml) using the following command.

   % kubectl create -f secretid.yaml
   

Deploy the Vault cluster issuer

Certmanager supports two different CRDs for configuration, an Issuer, which is scoped to a single namespace, and a ClusterIssuer, which is cluster-wide. For the workflow, you need to use ClusterIssuer.

Perform the following steps to deploy the Vault cluster issuer.

1. Create a file called issuer-vault.yaml with the following configuration.

   apiVersion: certmanager.k8s.io/v1alpha1
   kind: ClusterIssuer
   metadata:
     name: vault-issuer
   spec:
     vault:
       path: ${PKI_INT}/sign/kube-ingress
       server: https://vault_ip
       caBundle: <base64 encoded caBundle PEM file>
       auth:
         appRole:
           path: approle
           roleId: "db02de05-fa39-4855-059b-67221c5c2f63"
           secretRef:
             name: cert-manager-vault-approle
             key: secretId
   

   Replace PKI_INT with appropriate path of the intermediate CA. SecretRef is the kubernetes secret name created in the previous step. Replace roleId with the role_id retrieved from Vault. An optional base64 encoded caBundle in PEM format can be provided to validate the TLS connection to the Vault Server. When caBundle is set it replaces the CA bundle inside the container running the cert-manager. This parameter has no effect if the connection used is in plain HTTP.

2. Deploy the file (issuer-vault.yaml) using the following command.

   % kubectl create -f issuer-vault.yaml
   

3. Using the following command verify if the Vault cluster issuer is successfully authenticated with the Vault.

   % kubectl describe clusterIssuer vault-issuer  | tail -n 7
     Conditions:
       Last Transition Time:  2019-02-26T06:18:40Z
       Message:               Vault verified
       Reason:                VaultVerified
       Status:                True
       Type:                  Ready
   Events:                    <none>
   

Create a "certificate" CRD object for the certificate

Once the issuer is successfully registered , you need to get the certificate for the ingress domain kuard.example.com.

You need to create a "certificate" resource with the commonName and dnsNames. For more information, see cert-manager documentation. You can specify multiple dnsNames which will be used for SAN field in the certificate.

To create a "certificate" CRD object for the certificate, perform the following:

1. Create a file called certificate.yaml with the following configuration.

   apiVersion: certmanager.k8s.io/v1alpha1
   kind: Certificate
   metadata:
     name: kuard-example-tls
     namespace: default
   spec:
     secretName: kuard-example-tls
     issuerRef:
       kind: ClusterIssuer
       name: vault-issuer
     commonName: kuard.example.com
     duration: 720h
     #Renew before 7 days of expiry
     renewBefore: 168h
     commonName: kuard.example.com
     dnsNames:
     - www.kuard.example.com
   

   The certificate will have CN=kuard.example.com and SAN=Kuard.example.com,www.kuard.example.com. spec.secretName is the name of the secret where the certificate is stored after the certificate is issued successfully.

2. Deploy the file (certificate.yaml) on the Kubernetes cluster using the following command.

   kubectl create -f certificate.yaml
   certificate.certmanager.k8s.io/kuard-example-tls created
   

Verify if the certificate is issued

You can watch the progress of the certificate as it is issued using the following command:

% kubectl describe certificates kuard-example-tls  | grep -A5 Events
Events:
  Type    Reason      Age   From          Message
  ----    ------      ----  ----          -------
  Normal  CertIssued  48s   cert-manager  Certificate issued successfully

Important

You may encounter some errors due to Vault policies. If you encounter any such errors, return to vault and fix it.

After successful signing, a kubernetes.io/tls secret is created with the secretName specified in the Certificate resource.

% kubectl get secret kuard-example-tls
NAME                TYPE                DATA   AGE
kuard-exmaple-tls   kubernetes.io/tls   3      4m20s

Modify the ingress to use the generated secret

Perform the following steps to modify the ingress to use the generated secret.

1. Edit the original ingress and add a spec.tls section specifying the secret kuard-example-tls as follows.

   apiVersion: extensions/v1beta1
   kind: Ingress
   metadata:
     name: kuard
     annotations:
       kubernetes.io/ingress.class: "citrix"
   spec:
     tls:
     - hosts:
       - kuard.example.com
       secretName: kuard-example-tls
     rules:
     - host: kuard.example.com
       http:
         paths:
         - backend:
             serviceName: kuard
             servicePort: 80
   

2. Deploy the ingress using the following command.

   % kubectl apply -f ingress.yml
   ingress.extensions/kuard created
   
   % kubectl get ingress kuard
   NAME    HOSTS               ADDRESS   PORTS     AGE
   kuard   kuard.example.com             80, 443   12s
   

3. Log on to CPX and verify if the Certificate is bound to the SSL virtual server.

   kubectl exec -it cpx-ingress-668bf6695f-4fwh8 bash
   cli_script.sh 'shsslvs'
   exec: shsslvs
   1) Vserver Name: k8s-10.244.3.148:443:ssl
     DH: DISABLED
     DH Private-Key Exponent Size Limit: DISABLED Ephemeral RSA: ENABLED Refresh Count: 0
     Session Reuse: ENABLED Timeout: 120 seconds
     Cipher Redirect: DISABLED
     SSLv2 Redirect: DISABLED
     ClearText Port: 0
     Client Auth: DISABLED
     SSL Redirect: DISABLED
     Non FIPS Ciphers: DISABLED
     SNI: ENABLED
     OCSP Stapling: DISABLED
     HSTS: DISABLED
     HSTS IncludeSubDomains: NO
     HSTS Max-Age: 0
     SSLv2: DISABLED  SSLv3: ENABLED  TLSv1.0: ENABLED  TLSv1.1: ENABLED  TLSv1.2: ENABLED  TLSv1.3: DISABLED
     Push Encryption Trigger: Always
     Send Close-Notify: YES
     Strict Sig-Digest Check: DISABLED
     Zero RTT Early Data: DISABLED
     DHE Key Exchange With PSK: NO
     Tickets Per Authentication Context: 1
   Done
   
   root@cpx-ingress-668bf6695f-4fwh8:/# cli_script.sh 'shsslvs k8s-10.244.3.148:443:ssl'
   exec: shsslvs k8s-10.244.3.148:443:ssl
   
     Advanced SSL configuration for VServer k8s-10.244.3.148:443:ssl:
     DH: DISABLED
     DH Private-Key Exponent Size Limit: DISABLED Ephemeral RSA: ENABLED Refresh Count: 0
     Session Reuse: ENABLED Timeout: 120 seconds
     Cipher Redirect: DISABLED
     SSLv2 Redirect: DISABLED
     ClearText Port: 0
     Client Auth: DISABLED
     SSL Redirect: DISABLED
     Non FIPS Ciphers: DISABLED
     SNI: ENABLED
     OCSP Stapling: DISABLED
     HSTS: DISABLED
     HSTS IncludeSubDomains: NO
     HSTS Max-Age: 0
     SSLv2: DISABLED  SSLv3: ENABLED  TLSv1.0: ENABLED  TLSv1.1: ENABLED  TLSv1.2: ENABLED  TLSv1.3: DISABLED
     Push Encryption Trigger: Always
     Send Close-Notify: YES
     Strict Sig-Digest Check: DISABLED
     Zero RTT Early Data: DISABLED
     DHE Key Exchange With PSK: NO
     Tickets Per Authentication Context: 1
   , P_256, P_384, P_224, P_5216) CertKey Name: k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO Server Certificate for SNI
   
   7) Cipher Name: DEFAULT
     Description: Default cipher list with encryption strength >= 128bit
   Done
   
   root@cpx-ingress-668bf6695f-4fwh8:/# cli_script.sh 'sh certkey k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO'
   exec: sh certkey k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO
     Name: k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO Status: Valid,   Days to expiration:0
     Version: 3
     Serial Number: 524C1D9306F784A2F5277C05C2A120D5258D9A2F
     Signature Algorithm: sha256WithRSAEncryption
     Issuer:  CN=example.com CA intermediate
     Validity
       Not Before: Feb 26 06:48:39 2019 GMT
       Not After : Feb 27 06:49:09 2019 GMT
     Certificate Type: "Client Certificate" "Server Certificate"
     Subject:  CN=kuard.example.com
     Public Key Algorithm: rsaEncryption
     Public Key size: 2048
     Ocsp Response Status: NONE
     2) URI:http://127.0.0.1:8200/v1/pki_int/crl
     3) VServer name: k8s-10.244.3.148:443:ssl Server Certificate for SNI
   Done
   

The HTTPS webserver is UP with the vault signed certificate. Cert-manager automatically renews the certificate as specified in the 'RenewBefore" parameter in the Certificate, before expiry of the certificate.

Note

The vault signing of the certificate fails if the expiry of a certificate is beyond the expiry of the root CA or intermediate CA. You should ensure that the CA certificates are renewed in advance before the expiry.