Skip to content
Was this article helpful?

Deploy HTTPS web application on Kubernetes with Citrix ingress controller and Hashicorp Vault using cert-manager

This topic provides a sample workflow that uses HashiCorp Vault as self-signed CA to automate TLS certificate provisioning, revocation, and renewal for ingress resources deployed with Citrix ingress controller using cert-manager.

Specifically, the workflow uses the Vault PKI Secrets Engine to create a CA. This tutorial assumes that you have a Vault server installed and reachable from the Kubernetes cluster. The PKI secrets engine of Vault is suitable for internal applications and for external facing applications that require public trust, you can refer automating TLS certificates using letsencrypt CA

The workflow uses a Vault secret engine and authentication methods, refer the following Vault documentation for full list of features:

This topic provides you information on how to deploy an HTTPS web application on a Kubernetes cluster, using:

Prerequisites

Ensure that you have:

  • The Vault server is installed, unsealed and is reachable from Kubernetes cluster.

  • Enabled RBAC on your Kubernetes cluster.

  • Deployed Citrix ADC MPX, VPX or CPX in Tier 1 or Tier 2 deployment model.

    In the Tier 1 deployment model, Citrix ADC MPX or VPX is used as an Application Delivery Controller (ADC) and Citrix ingress controller running in Kubernetes cluster configures the virtual services for the services running on Kubernetes cluster. Citrix ADC runs the virtual service on the publicly routable IP address and offloads SSL for client traffic with the help of Let's Encrypt generated certificate.

    Similarly in the Tier 2 deployment model, a TCP service is configured on the Citrix ADC (VPX/MPX) running outside the Kubernetes cluster to forward the traffic to Citrix ADC CPX instances running in the Kubernetes cluster. Citrix ADC CPX ends the SSL session and load-balances the traffic to actual service pods.

  • Deployed Citrix ingress controller. Click here for various deployment scenarios.

  • Administrator permissions for all the deployment steps. If you encounter failures due to permissions, make sure you have administrator permission.

Deploy cert-manager using the manifest file

Perform the following steps to deploy cert-manager using the supplied YAML manifest file. To keep things simple, skip cert-manager's Helm installation, and instead use the supplied YAML manifests.

  1. Download the latest source of cert-manager from github.com/jetstack/cert-manager repository using the following command.

    wget https://github.com/jetstack/cert-manager/archive/v0.6.2.tar.gz
tar -zxvf v0.6.2.tar.gz

  2. Deploy the cert-manager using the following command.

    kubectl apply -f deploy/manifests/cert-manager.yaml

    You can also install the cert-manager with Helm, for more information see cert-manager documentation

  3. Verify that the cert-manager is up and running using the following command.

    % kubectl -n cert-manager get all
NAME                                       READY   STATUS    RESTARTS   AGE
pod/cert-manager-77fd74fb64-d68v7          1/1     Running   0          4m41s
pod/cert-manager-webhook-67bf86d45-k77jj   1/1     Running   0          4m41s

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/cert-manager-webhook   ClusterIP   10.108.161.154   <none>        443/TCP   13d

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager           1/1     1            1           13d
deployment.apps/cert-manager-webhook   1/1     1            1           13d

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-77fd74fb64          1         1         1       13d
replicaset.apps/cert-manager-webhook-67bf86d45   1         1         1       13d

NAME                                                COMPLETIONS   DURATION   AGE
job.batch/cert-manager-webhook-ca-sync              1/1           22s        13d
job.batch/cert-manager-webhook-ca-sync-1549756800   1/1           21s        10d
job.batch/cert-manager-webhook-ca-sync-1550361600   1/1           19s        3d8h

NAME                                         SCHEDULE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/cert-manager-webhook-ca-sync   @weekly    False     0        3d8h            13d

Deploy a sample web application

Perform the following steps to deploy a sample web application.

Note

Kuard, a kubernetes demo application is used for reference in this topic.

  1. Create a deployment YAML file (kuard-deployment.yaml) for Kuard with the following configuration.

    apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: kuard
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kuard
    spec:
      containers:
      - image: gcr.io/kuar-demo/kuard-amd64:1
        imagePullPolicy: Always
        name: kuard
        ports:
        - containerPort: 8080

  2. Deploy Kuard deployment file (kuard-deployment.yaml) to your cluster, using the following commands.

    % kubectl create -f kuard-deployment.yaml
deployment.extensions/kuard created
% kubectl get pod -l app=kuard
NAME                     READY   STATUS    RESTARTS   AGE
kuard-6fc4d89bfb-djljt   1/1     Running   0          24s

  3. Create a service for the deployment. Create a file called service.yaml with the following configuration.

    apiVersion: v1
kind: Service
metadata:
  name: kuard
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  selector:
    app: kuard

  4. Deploy and verify the service using the following command.

    % kubectl create -f service.yaml
service/kuard created
% kubectl get svc kuard
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kuard   ClusterIP   10.103.49.171   <none>        80/TCP    13s

  5. Expose this service to outside world by creating an Ingress that is deployed on Citrix ADC CPX or VPX as Content switching virtual server.

    Note

    Ensure that you change kubernetes.io/ingress.class to your ingress class on which Citrix ingress controller is started.

    apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kuard annotations: kubernetes.io/ingress.class: "citrix" spec: rules: - host: kuard.example.com http: paths: - backend: serviceName: kuard servicePort: 80

    Important

    Change the value of spec.rules.host to the domain that you control. Ensure that a DNS entry exists to route the traffic to Citrix ADC CPX or VPX.

  6. Deploy the Ingress using the following command.

    % kubectl apply -f ingress.yml
ingress.extensions/kuard created
root@ubuntu-vivek-225:~/cert-manager# kubectl get ingress
NAME    HOSTS               ADDRESS   PORTS   AGE
kuard   kuard.example.com             80      7s

  7. Verify if the ingress is configured on Citrix ADC CPX or VPX using the following command.

    kubectl exec -it cpx-ingress-5b85d7c69d-ngd72 /bin/bash
root@cpx-ingress-5b85d7c69d-ngd72:/# cli_script.sh 'sh cs vs'
exec: sh cs vs
1) k8s-10.244.1.50:80:http (10.244.1.50:80) - HTTP Type: CONTENT
  State: UP
  Last state change was at Thu Feb 21 09:02:14 2019
  Time since last state change: 0 days, 00:00:41.140
  Client Idle Timeout: 180 sec
  Down state flush: ENABLED
  Disable Primary Vserver On Down : DISABLED
  Comment: uid=75VBGFO7NZXV7SCI4LSDJML2Q5X6FSNK6NXQPWGMDOYGBW2IMOGQ====
  Appflow logging: ENABLED
  Port Rewrite : DISABLED
  State Update: DISABLED
  Default: Content Precedence: RULE
  Vserver IP and Port insertion: OFF
  L2Conn: OFF Case Sensitivity: ON
  Authentication: OFF
  401 Based Authentication: OFF
  Push: DISABLED Push VServer:
  Push Label Rule: none
  Listen Policy: NONE
  IcmpResponse: PASSIVE
  RHIstate:  PASSIVE
  Traffic Domain: 0
Done
root@cpx-ingress-5b85d7c69d-ngd72:/# exit
exit

  8. Verify if the page is correctly being served when requested using the curl command.

    % curl -sS -D - kuard.example.com -o /dev/null
HTTP/1.1 200 OK
Content-Length: 1458
Content-Type: text/html
Date: Thu, 21 Feb 2019 09:09:05 GMT

Configure Hashicorp Vault as Certificate Authority

Set up an intermediate CA certificate signing request using Hashicorp Vault. This Vault endpoint is used by cert-manager to sign the certificate for the ingress resources.

Prerequisites

Ensure that you have installed the jq utility.

Create a root CA

For the sample workflow you can generate your own Root Certificate Authority within the Vault. In a production environment, you should use an external Root CA to sign the intermediate CA that Vault uses to generate certificates. If you have a root CA generated elsewhere, skip this step.

Note

PKI_ROOT is a path where you mount the root CA, typically it is 'pki'. ${DOMAIN} in this procedure is example.com

% vault secrets enable -path="${PKI_ROOT}" pki

# Set the max TTL for the root CA to 10 years
% vault secrets tune -max-lease-ttl=87600h "${PKI_ROOT}"

% vault write -format=json "${PKI_ROOT}"/root/generate/internal \
 common_name="${DOMAIN} CA root" ttl=87600h | tee \
>(jq -r .data.certificate > ca.pem) \
>(jq -r .data.issuing_ca > issuing_ca.pem) \
>(jq -r .data.private_key > ca-key.pem)

#Configure the CA and CRL URLs:

% vault write "${PKI_ROOT}"/config/urls \
       issuing_certificates="${VAULT_ADDR}/v1/${PKI_ROOT}/ca" \
       crl_distribution_points="${VAULT_ADDR}/v1/${PKI_ROOT}/crl"

Generate an intermediate CA

After creating the root CA, perform the following steps to create an intermediate CSR using the root CA.

  1. Enable pki from a different path PKI_INT from root CA, typically pki\_int. Use the following command:

    % vault secrets enable -path=${PKI_INT} pki
Set the max TTL to 3 year

% vault secrets tune -max-lease-ttl=26280h ${PKI_INT}

  2. Generate CSR for ${DOMAIN} that needs to be signed by the root CA, the key is stored internally to vault. Use the following command:

    % vault write -format=json "${PKI_INT}"/intermediate/generate/internal \
common_name="${DOMAIN} CA intermediate" ttl=43800h | tee \
>(jq -r .data.csr > pki_int.csr) \
>(jq -r .data.private_key > pki_int.pem)

  3. Generate and sign the ${DOMAIN} certificate as an intermediate CA using root CA, store it as intermediate.cert.pem. Use the following command:

    % vault write -format=json "${PKI_ROOT}"/root/sign-intermediate csr=@pki_int.csr \
        format=pem_bundle \
        | jq -r '.data.certificate' > intermediate.cert.pem

    If you are using an external root CA, skip the above step and sign the CSR manually using root CA.

  4. Once the CSR is signed and the root CA returns a certificate, it needs to added back into the Vault using the following command:

    % vault write "${PKI_INT}"/intermediate/set-signed certificate=@intermediate.cert.pem

  5. Set the CA and CRL location using the following command.

    vault write "${PKI_INT}"/config/urls issuing_certificates="${VAULT_ADDR}/v1/${PKI_INT}/ca" crl_distribution_points="${VAULT_ADDR}/v1/${PKI_INT}/crl"

An intermediate CA is setup and can be used to sign certificates for ingress resources.

Configure a role

A role is a logical name which maps to policies, administrator can control the certificate generation through the roles.

Create a role for the intermediate CA that provides a set of policies for issuing or signing the certificates using this CA.

There are many configurations that can be configured when creating roles, for more information see, Vault role documentation.

For the workflow, create a role "kube-ingress" that allows you to sign certificates of ${DOMAIN} and its subdomains with a TTL of 90 days.

# with a Max TTL of 90 days
vault write ${PKI_INT}/roles/kube-ingress \
          allowed_domains=${DOMAIN} \
          allow_subdomains=true \
          max_ttl="2160h"

Create Approle based authentication

After configuring an intermediate CA to sign the certificates, you need to provide an authentication mechanism for cert-manager to use the vault for signing the certificates. Cert-manager supports Approle authentication method which provides a way for the applications to access the Vault defined roles.

An "AppRole" represents a set of Vault policies and login constraints that must be met to receive a token with those policies. For detailed understanding of this authentication method, see Approle documentation.

Create an Approle

Create an approle named "Kube-role". The secret id for cert-manager should not be expired to use this Approle for authentication, hence do not set a TTL, or set it to 0.

% vault auth enable approle of token ttl 5 minutes

% vault write auth/approle/role/kube-role \
    token_ttl=5m \
    token_max_ttl=10m

Associate a policy with the Approle

Perform the following steps to associate a policy with an Approle.

  1. Create a file pki_int.hcl with the following configuration to allow the signing endpoints of the intermediate CA.

    path "${PKI_INT}/sign/*" {
      capabilities = ["create","update"]
    }

  2. Add the file to a new policy called kube_allow_sign using the following command.

    vault policy write kube-allow-sign pki_int.hcl

  3. Update this policy to the approle using the following command.

    vault write auth/approle/role/kube-role policies=kube-allow-sign

The kube-role approle allows you to sign the CSR with intermediate CA.

Generate the Role id and Secret id

Role id and Secret id is used by cert-manager to authenticate with Vault.

Generate role id and Secret id and encode the secret_id with Base64. Perform the following:

% vault read auth/approle/role/kube-role/role-id
role_id     db02de05-fa39-4855-059b-67221c5c2f63

% vault write -f auth/approle/role/kube-role/secret-id
secret_id               6a174c20-f6de-a53c-74d2-6018fcceff64
secret_id_accessor      c454f7e5-996e-7230-6074-6ef26b7bcf86

# encode secret_id with base64
% echo 6a174c20-f6de-a53c-74d2-6018fcceff64 | Base64
NmExNzRjMjAtZjZkZS1hNTNjLTc0ZDItNjAxOGZjY2VmZjY0Cg==

Configure issuing certificates in Kubernetes

After you have configured the Vault as intermediate CA, and the Approle authentication method for cert-manager to access the Vault. You need to configure the certificate for the ingress.

Create a secret with Approle secret-id

Perform the following to create a secret with Approle secret id.

  1. Create a secret file called secretid.yaml with following configuration.

    apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: cert-manager-vault-approle
  namespace: cert-manager
data:
  secretId: "NmExNzRjMjAtZjZkZS1hNTNjLTc0ZDItNjAxOGZjY2VmZjY0Cg=="

    Note

    data.secretId is the base64 encoded Secret Id generated in Generate the Role id and Secret id. If you're using an Issuer resource in the next step, Secret must be in same namespace as the Issuer. For ClusterIssuer, secret must be in cert-manager namespace.

  2. Deploy the secret file (secretid.yaml) using the following command.

    % kubectl create -f secretid.yaml

Deploy the Vault cluster issuer

Certmanager supports two different CRDs for configuration, an Issuer, which is scoped to a single namespace, and a ClusterIssuer, which is cluster-wide. For the workflow, you need to use ClusterIssuer.

Perform the following steps to deploy the Vault cluster issuer.

  1. Create a file called issuer-vault.yaml with the following configuration.

    apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    path: ${PKI_INT}/sign/kube-ingress
    server: https://vault_ip
    caBundle: <base64 encoded caBundle PEM file>
    auth:
      appRole:
        path: approle
        roleId: "db02de05-fa39-4855-059b-67221c5c2f63"
        secretRef:
          name: cert-manager-vault-approle
          key: secretId

    Replace PKI_INT with appropriate path of the intermediate CA. SecretRef is the kubernetes secret name created in the previous step. Replace roleId with the role_id retrieved from Vault. An optional base64 encoded caBundle in PEM format can be provided to validate the TLS connection to the Vault Server. When caBundle is set it replaces the CA bundle inside the container running the cert-manager. This parameter has no effect if the connection used is in plain HTTP.

  2. Deploy the file (issuer-vault.yaml) using the following command.

    % kubectl create -f issuer-vault.yaml

  3. Using the following command verify if the Vault cluster issuer is successfully authenticated with the Vault.

    % kubectl describe clusterIssuer vault-issuer  | tail -n 7
  Conditions:
    Last Transition Time:  2019-02-26T06:18:40Z
    Message:               Vault verified
    Reason:                VaultVerified
    Status:                True
    Type:                  Ready
Events:                    <none>

Create a "certificate" CRD object for the certificate

Once the issuer is successfully registered , you need to get the certificate for the ingress domain kuard.example.com.

You need to create a "certificate" resource with the commonName and dnsNames. For more information, see cert-manager documentation. You can specify multiple dnsNames which will be used for SAN field in the certificate.

To create a "certificate" CRD object for the certificate, perform the following:

  1. Create a file called certificate.yaml with the following configuration.

    apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: kuard-example-tls
  namespace: default
spec:
  secretName: kuard-example-tls
  issuerRef:
    kind: ClusterIssuer
    name: vault-issuer
  commonName: kuard.example.com
  duration: 720h
  #Renew before 7 days of expiry
  renewBefore: 168h
  commonName: kuard.example.com
  dnsNames:
  - www.kuard.example.com

    The certificate will have CN=kuard.example.com and SAN=Kuard.example.com,www.kuard.example.com. spec.secretName is the name of the secret where the certificate is stored after the certificate is issued successfully.

  2. Deploy the file (certificate.yaml) on the Kubernetes cluster using the following command.

    kubectl create -f certificate.yaml
certificate.certmanager.k8s.io/kuard-example-tls created

Verify if the certificate is issued

You can watch the progress of the certificate as it is issued using the following command:

% kubectl describe certificates kuard-example-tls  | grep -A5 Events
Events:
  Type    Reason      Age   From          Message
  ----    ------      ----  ----          -------
  Normal  CertIssued  48s   cert-manager  Certificate issued successfully

Important

You may encounter some errors due to Vault policies. If you encounter any such errors, return to vault and fix it.

After successful signing, a kubernetes.io/tls secret is created with the secretName specified in the Certificate resource.

% kubectl get secret kuard-example-tls
NAME                TYPE                DATA   AGE
kuard-exmaple-tls   kubernetes.io/tls   3      4m20s

Modify the ingress to use the generated secret

Perform the following steps to modify the ingress to use the generated secret.

  1. Edit the original ingress and add a spec.tls section specifying the secret kuard-example-tls as follows.

    apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kuard
  annotations:
    kubernetes.io/ingress.class: "citrix"
spec:
  tls:
  - hosts:
    - kuard.example.com
    secretName: kuard-example-tls
  rules:
  - host: kuard.example.com
    http:
      paths:
      - backend:
          serviceName: kuard
          servicePort: 80

  2. Deploy the ingress using the following command.

    % kubectl apply -f ingress.yml
ingress.extensions/kuard created

% kubectl get ingress kuard
NAME    HOSTS               ADDRESS   PORTS     AGE
kuard   kuard.example.com             80, 443   12s

  3. Log on to CPX and verify if the Certificate is bound to the SSL virtual server.

    kubectl exec -it cpx-ingress-668bf6695f-4fwh8 bash
cli_script.sh 'shsslvs'
exec: shsslvs
1) Vserver Name: k8s-10.244.3.148:443:ssl
  DH: DISABLED
  DH Private-Key Exponent Size Limit: DISABLED Ephemeral RSA: ENABLED Refresh Count: 0
  Session Reuse: ENABLED Timeout: 120 seconds
  Cipher Redirect: DISABLED
  SSLv2 Redirect: DISABLED
  ClearText Port: 0
  Client Auth: DISABLED
  SSL Redirect: DISABLED
  Non FIPS Ciphers: DISABLED
  SNI: ENABLED
  OCSP Stapling: DISABLED
  HSTS: DISABLED
  HSTS IncludeSubDomains: NO
  HSTS Max-Age: 0
  SSLv2: DISABLED  SSLv3: ENABLED  TLSv1.0: ENABLED  TLSv1.1: ENABLED  TLSv1.2: ENABLED  TLSv1.3: DISABLED
  Push Encryption Trigger: Always
  Send Close-Notify: YES
  Strict Sig-Digest Check: DISABLED
  Zero RTT Early Data: DISABLED
  DHE Key Exchange With PSK: NO
  Tickets Per Authentication Context: 1
Done

root@cpx-ingress-668bf6695f-4fwh8:/# cli_script.sh 'shsslvs k8s-10.244.3.148:443:ssl'
exec: shsslvs k8s-10.244.3.148:443:ssl

  Advanced SSL configuration for VServer k8s-10.244.3.148:443:ssl:
  DH: DISABLED
  DH Private-Key Exponent Size Limit: DISABLED Ephemeral RSA: ENABLED Refresh Count: 0
  Session Reuse: ENABLED Timeout: 120 seconds
  Cipher Redirect: DISABLED
  SSLv2 Redirect: DISABLED
  ClearText Port: 0
  Client Auth: DISABLED
  SSL Redirect: DISABLED
  Non FIPS Ciphers: DISABLED
  SNI: ENABLED
  OCSP Stapling: DISABLED
  HSTS: DISABLED
  HSTS IncludeSubDomains: NO
  HSTS Max-Age: 0
  SSLv2: DISABLED  SSLv3: ENABLED  TLSv1.0: ENABLED  TLSv1.1: ENABLED  TLSv1.2: ENABLED  TLSv1.3: DISABLED
  Push Encryption Trigger: Always
  Send Close-Notify: YES
  Strict Sig-Digest Check: DISABLED
  Zero RTT Early Data: DISABLED
  DHE Key Exchange With PSK: NO
  Tickets Per Authentication Context: 1
, P_256, P_384, P_224, P_5216) CertKey Name: k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO Server Certificate for SNI

7) Cipher Name: DEFAULT
  Description: Default cipher list with encryption strength >= 128bit
Done

root@cpx-ingress-668bf6695f-4fwh8:/# cli_script.sh 'sh certkey k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO'
exec: sh certkey k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO
  Name: k8s-LMO3O3U6KC6WXKCBJAQY6K6X6JO Status: Valid,   Days to expiration:0
  Version: 3
  Serial Number: 524C1D9306F784A2F5277C05C2A120D5258D9A2F
  Signature Algorithm: sha256WithRSAEncryption
  Issuer:  CN=example.com CA intermediate
  Validity
    Not Before: Feb 26 06:48:39 2019 GMT
    Not After : Feb 27 06:49:09 2019 GMT
  Certificate Type: "Client Certificate" "Server Certificate"
  Subject:  CN=kuard.example.com
  Public Key Algorithm: rsaEncryption
  Public Key size: 2048
  Ocsp Response Status: NONE
  2) URI:http://127.0.0.1:8200/v1/pki_int/crl
  3) VServer name: k8s-10.244.3.148:443:ssl Server Certificate for SNI
Done

The HTTPS webserver is UP with the vault signed certificate. Cert-manager automatically renews the certificate as specified in the 'RenewBefore" parameter in the Certificate, before expiry of the certificate.

Note

The vault signing of the certificate fails if the expiry of a certificate is beyond the expiry of the root CA or intermediate CA. You should ensure that the CA certificates are renewed in advance before the expiry.

Was this article helpful?