TLS Certificate Chaining with Strimzi

Michael Olson

Dec 19, 2024

Implementing TLS Certificate Chain with FreeIPA, Cert-Manager and Strimzi

Introduction
Demo Environment
Requirements
FreeIPA
Kubernetes

Introduction

This blog post goes over setting up integrated certificate chains with FreeIPA, Cert-Manager, Strimzi and Kafka. mTLS is tested internally and externally via Kafdrop to ensure proper functionality. Strimzi does not currently support automatic TLS certificate chaining, although based on Strimzi's presentation at KubeCon North America 2024 integration with Cert-Manager is currently being worked on and will be implemented soon. The issue this blog post solves is implemently a fully integrated solution for Kafka TLS, allowing for the Kafka Clients CA to create working internal user TLS certificates, and FreeIPA to create external user TLS certificates that are trusted by the Kafka Cluster CA signed listeners. This is a hacky solution that involves creating the Clients CA Secret to contain the full certificate chain within the ca.crt key (this allows the Kafka Clients CA to properly create the internal users), then appending the CA / Subsidiary CA certificates as additional keys (i.e. ca-2.crt, ca-3.crt) which appends those CA's as trusted CA's for the Kafka Cluster CA. Currently the process is not automated. Below is a high-level diagram demonstrating the desired TLS trust within our environment:

Visual representation of the finished product

Demo Environment

My environment includes configuring 2 Rocky v9.5 Virtual Machines on my local machine with the following resources for a minimal setup. These hosts are built with access to the internet to reduce complexity and focus on the important steps relevant to the purpose of this post. This post assumes that you have a FreeIPA server running, as well as a Kubernetes cluster, as well as administrative access to both services.

visual of my graphical environment used for testing

Requirements

FreeIPA (v4.2+)
Kubernetes (v1.23+)
Strimzi (v0.40.0+)
Cert-Manager (v1.15+)
Reflector (v7.1.288+)
Kafdrop (v3.30.0+)

FreeIPA

The following tasks can be performed on any machine within the FreeIPA realm. To keep this post simple I am running commands on my FreeIPA server host, if I had set up a Configuration Manager (i.e. Ansible, Salt, Chef, etc.) I would have run these commands from that machine and stored the certificates on there. If you receive an error relating to FreeIPA not receiving the Kerberos credentials you can rerun the kinit admin command as demonstrated below.


[michael@ipa ~]# kinit admin
[michael@ipa ~]# mkdir ~/cert_profiles ~/certs

FreeIPA Certificate Profiles

This section will go over creating the Certificate Profiles in FreeIPA. Certificate Profiles are necessary because by default, FreeIPA does not have a profile capable of generating Subsidiary CA certificates. The second Certificate Profile is necessary because Strimzi Kafka expects the Subject Name in certificates to only have the CN=<service name> attribute, by default FreeIPA includes the O=<organization name> attribute which Strimzi Kafka is unable to parse. FreeIPA includes templates during installation that we can use and alter as opposed to creating the Certificate Profiles from scratch, as demonstrated below.

Kafka CA Certificate Profile

The below setps are as follows: copy the caCACert.cfg file from FreeIPA (this is the default template we will be altering to produce our desired Subsidiary CA Certificate Profile), remove the ability to use insecure key sizes for TLS certificates, restrict the level of CA certificate generation to one (The Subsidiary Certificate used by Cert-Manager will be capable of generating CA certificates for Kafka), append a class ID to categorize the type of certificates this profile is responsible for, append a profile ID (Certificate Profile name), append an auth instance ID to associate the certificate(s) with an authentication instance, and finally importing the config into FreeIPA. At this stage you will be prompted to enter the Certificate Profile ID, you will want to make sure the name you enter matches the profileId attribute you appended into the config.


[michael@ipa ~]# cp /var/lib/pki/pki-tomcat/ca/profiles/ca/caCACert.cfg ~/cert_profiles/KafkaCACerts.cfg
[michael@ipa ~]# sed -i 's/keyParameters=1024,/keyParameters=/' ~/cert_profiles/KafkaCACerts.cfg
[michael@ipa ~]# sed -i 's/PathLen=-1/PathLen=1/' ~/cert_profiles/KafkaCACerts.cfg
[michael@ipa ~]# echo "classId=caEnrollImpl" >> ~/cert_profiles/KafkaCACerts.cfg
[michael@ipa ~]# echo "profileId=KafkaCACerts" >> ~/cert_profiles/KafkaCACerts.cfg
[michael@ipa ~]# echo "auth.instance_id=raCertAuth" >> ~/cert_profiles/KafkaCACerts.cfg
[michael@ipa ~]# ipa certprofile-import --desc="Kafka CA Certs" --store=True --file="$HOME/cert_profiles/KafkaCACerts.cfg"

Kafka External User Certificate Profile

The below steps are as follows: copy the caOtherCert.cfg file from FreeIPA (this is the default template we will be altering to produce our desired Kafka External Users Certificate Profile), remove the ability to use insecure key sizes for TLS certificates, append a class ID to categorize the type of certificates this profile is responsible for, append a profile ID (Certificate Profile name), append an auth instance ID to associate the certificate(s) with an authenticated instance, and finally importing the config into FreeIPA. At this stage you will be prompted to enter the Certificate Profile ID, you will want to make sure the name you enter matches the profileId attribute you appended into the config.


[michael@ipa ~]# cp /var/lib/pki/pki-tomcat/ca/profiles/ca/caOtherCert.cfg ~/cert_profiles/KafkaExternalUserCerts.cfg
[michael@ipa ~]# sed -i 's/keyParameters=1024,/keyParameters=/' ~/cert_profiles/KafkaExternalUserCerts.cfg
[michael@ipa ~]# echo "classId=caEnrollImpl" >> ~/cert_profiles/KafkaExternalUserCerts.cfg
[michael@ipa ~]# echo "profileId=KafkaExternalUserCerts" >> ~/cert_profiles/KafkaExternalUserCerts.cfg
[michael@ipa ~]# echo "auth.instance_id=raCertAuth" >> ~/cert_profiles/KafkaExternalUserCerts.cfg
[michael@ipa ~]# ipa certprofile-import --desc="Kafka External User Certs" --store=True --file="~/cert_profiles/KafkaExternalUserCerts.cfg"

FreeIPA Services

FreeIPA requires that you request certificates on behalf of entities (Users, Services, Hosts). This section goes over force creating services in FreeIPA that we will later reference when generating our certificates. The services are force-created because typically a service will be attached to a host running in the FreeIPA domain, but in our case the CA will be running within our Kubernetes cluster and external users do not need to reference physical hosts necessarily, it allows for freedom when generating TLS certificates.

Kafka CA Service

The below steps are as follows: force add the KafkaCACerts service to FreeIPA, then associate the host you will be requesting the certificate from with that service to allow certificate requests to be approved to the requesting machine.


[michael@ipa ~]# ipa service-add --force --skip-host-check KAFKA/KafkaCACerts
[michael@ipa ~]# ipa service-add-host KAFKA/KafkaCACerts --hosts=ipa.example.com

Kafka External Users Service

The below steps are as follows: force add the KafkaExternalUserCerts service to FreeIPA, then associate the host you will be requesting the certificate from with that service to allow certificate requests to be approved to the requesting machine.


[michael@ipa ~]# ipa service-add --force --skip-host-check KAFKA/KafkaExternalUserCerts
[michael@ipa ~]# ipa service-add-host KAFKA/KafkaExternalUserCerts --hosts=ipa.example.com

FreeIPA CA ACLs

FreeIPA ACLs (Access Control Lists) allow for restricted access to generating certificates with the previously created Certificate Profiles. Once the ACLs are created we should be ready to request our actual certificates.

Kafka CA Certs CA ACL

The below steps are as follows: create the initial KafkaCACerts CA ACL, add the FreeIPA CA to the ACL, add the KafkaCACerts service to the ACL, and finally add the KafkaCACerts Profile to the CA ACL.


[michael@ipa ~]# ipa caacl-add KafkaCACerts
[michael@ipa ~]# ipa caacl-add-ca KafkaCACerts --cas=ipa
[michael@ipa ~]# ipa caacl-add-service KafkaCACerts --services="KAFKA/KafkaCACerts"
[michael@ipa ~]# ipa caacl-add-profile KafkaCACerts --certprofiles=KafkaCACerts

Kafka External User Certs CA ACL

The below steps are as follows: create the initial KafkaExternalUserCerts CA ACL, add the FreeIPA CA to the ACL, add the KafkaExternalUserCerts service to the ACL, and finally add the KafkaExternalUserCerts Profile to the CA ACL.


[michael@ipa ~]# ipa caacl-add KafkaExternalUserCerts
[michael@ipa ~]# ipa caacl-add-ca KafkaExternalUserCerts --cas=ipa
[michael@ipa ~]# ipa caacl-add-service KafkaExternalUserCerts --services="KAFKA/KafkaExternalUserCerts"
[michael@ipa ~]# ipa caacl-add-profile KafkaExternalUserCerts --certprofiles=KafkaExternalUserCerts

FreeIPA Certificates

We made it! We are ready to generate our certificates, one for issuing CA certificates for Kafka via Cert-Manager, and one for demonstrating an external user communicating with the Kafka external listeners via mTLS.

Kafka CA Certificates

For this demo I will use the FreeIPA ipa-getcert command. the -I flag allows you to specify an ID for your certificate, --keyfile flag specifies destination key file, --certfile specifies destination certificate file, --ca-file flag allows us to specify the destination FreeIPA root CA certificate file, --keysize flag specifies the key size to use for increased security, --profile flag specifies the FreeIPA Certificate Profile we created previously, --issuer flag specifies the CA issuer, in this case the FreeIPA root CA, --subject-name flag represents the Subject Name value to be set on the generated certificate, --dns flag adds a Subject Alternative Name (can be used multiple times), and the --principal flag references the Service we created in a previous step.


[michael@ipa ~]# mkdir ~/certs/KafkaCACerts
[michael@ipa ~]# ipa-getcert request -I KafkaCACerts --keyfile=$HOME/certs/KafkaCACerts/KafkaCACerts.key --certfile=$HOME/certs/KafkaCACerts/KafkaCACerts.crt --ca-file=$HOME/certs/KafkaCACerts/ca.crt --key-size=4096 --profile=KafkaCACerts --issuer=ipa --subject-name="CN = KafkaCACerts" --dns="KafkaCACerts" --principal=KAFKA/KafkaCACerts

Kafka External User Certificates

The following is very similar to the above step, with the addition of adding a principal alias to the original Kafka/KafkaExternalUserCerts service. This allows us to request certificates for individual users and alter Subject Names on the certificates with minimal effort. See the above task for details on the FreeIPA ipa-getcert command and flags.


[michael@ipa ~]# mkdir --parents ~/certs/KafkaExternalUsers/external-user
[michael@ipa ~]# ipa service-add-principal KAFKA/KafkaExternalUserCerts KAFKA/external-user
[michael@ipa ~]# ipa-getcert request -I external-user --keyfile=$HOME/certs/KafkaExternalUsers/external-user/external-user.key --certfile=$HOME/certs/KafkaExternalUsers/external-user/external-user.crt --ca-file=$HOME/certs/KafkaExternalUsers/external-user/ca.crt --key-size=4096 --profile=KafkaExternalUserCerts --issuer=ipa --subject-name="CN = external-user" --dns="external-user" --principal=KAFKA/KafkaExternalUserCerts

Kubernetes

This section will go over how to set up our Kubernetes applications needed for simplifying management of tls certificates within the cluster. We will start by deploying the services via publicly accessible CNCF helm charts, commands were taken from Artifact Hub

Cert-Manager


[michael@k8s ~]# helm repo add cert-manager https://charts.jetstack.io
[michael@k8s ~]# helm install cert-manager cert-manager/cert-manager --set crds.enabled=true -n cert-manager --create-namespace

Strimzi


[michael@k8s ~]# helm install strimzi-kafka-operator oci://quay.io/strimzi-helm/strimzi-kafka-operator --set watchNamespaces={kafka} -n strimzi --create-namespace

Reflector


[michael@k8s ~]# helm repo add emberstack https://emberstack.github.io/helm-charts
[michael@k8s ~]# helm install reflector emberstack/reflector -n reflector --create-namespace

Kafka

Copy Kafka CA Certs from FreeIPA


[michael@k8s ~]# mkdir --parents ~/kafka/certs
[michael@k8s ~]# scp -r michael@ipa.example.com:/home/michael/certs/KafkaCACerts ~/kafka/certs/

Create Kafka Issuer Secret


[michael@k8s ~]# kubectl create namespace kafka
[michael@k8s ~]# kubectl create secret generic kafka-ca-issuer --namespace kafka --from-file=ca.crt=$HOME/kafka/certs/KafkaCACerts/ca.crt --from-file=tls.crt=$HOME/kafka/certs/KafkaCACerts/KafkaCACerts.crt --from-file=tls.key=$HOME/kafka/certs/KafkaCACerts/KafkaCACerts.key

Create Kafka Issuer


[michael@k8s ~]# mkdir --parents ~/kafka/issuers
[michael@k8s ~]# cat > ~/kafka/issuers/kafka-ca-issuer.yaml << EOL
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: kafka-ca
  namespace: kafka
spec:
  ca:
    secretName: kafka-ca-issuer
EOL
[michael@k8s ~]# kubectl apply -f ~/kafka/issuers/kafka-ca-issuer.yaml

Create Kafka CA Template Certificates


[michael@k8s ~]# mkdir --parents ~/kafka/certs/templates
[michael@k8s ~]# cat > ~/kafka/certs/templates/kafka-ca-cert-templates.yaml << EOL
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kafka-clients-ca-template
  namespace: kafka
spec:
  commonName: clients-ca
  dnsNames:
    - clients-ca
  isCA: true
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: kafka-ca
  secretName: kafka-clients-ca-template
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kafka-cluster-ca-template
  namespace: kafka
spec:
  commonName: cluster-ca
  dnsNames:
    - cluster-ca
  isCA: true
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: kafka-ca
  secretName: kafka-cluster-ca-template
EOL
[michael@k8s ~]# kubectl apply -f ~/kafka/certs/templates/kafka-ca-cert-templates.yaml

Configure Kafka Clients CA Secrets


[michael@k8s ~]# kubectl get secret kafka-clients-ca-template -n kafka -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/kafka/certs/templates/kafka-clients-ca.crt
[michael@k8s ~]# kubectl get secret kafka-clients-ca-template -n kafka -o jsonpath='{.data.ca\.crt}' | base64 -d >> ~/kafka/certs/templates/kafka-clients-ca.crt
[michael@k8s ~]# cat > ~/kafka/certs/KafkaCACerts/kafka-clients-ca.yaml << EOL
---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-clients-ca-cert
  namespace: kafka
  labels:
    strimzi.io/kind: Kafka
    strimzi.io/cluster: kafka
  annotations:
    strimzi.io/ca-cert-generation: "0"
data:
  ca.crt: $(base64 -w0 ~/kafka/certs/templates/kafka-clients-ca.crt)
  ca-2.crt: $(kubectl get secret kafka-clients-ca-template -n kafka -o jsonpath='{.data.tls\.crt}' | base64 -d | while read L; do echo $L; [[ "$L" =~ "-----END" ]] && exit 1; done | base64 -w0)
  ca-3.crt: $(kubectl get secret kafka-clients-ca-template -n kafka -o jsonpath='{.data.ca\.crt}')
---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-clients-ca
  namespace: kafka
  labels:
    strimzi.io/kind: Kafka
    strimzi.io/cluster: kafka
  annotations:
    strimzi.io/ca-key-generation: "0"
data:
  ca.key: $(kubectl get secret kafka-clients-ca-template -n kafka -o jsonpath='{.data.tls\.key}')
EOL
[michael@k8s ~]# kubectl create -f ~/kafka/certs/KafkaCACerts/kafka-clients-ca.yaml

Configure Kafka Cluster CA Secrets


[michael@k8s ~]# kubectl get secret kafka-cluster-ca-template -n kafka -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/kafka/certs/templates/kafka-cluster-ca.crt
[michael@k8s ~]# kubectl get secret kafka-cluster-ca-template -n kafka -o jsonpath='{.data.ca\.crt}' | base64 -d >> ~/kafka/certs/templates/kafka-cluster-ca.crt
[michael@k8s ~]# cat > ~/kafka/certs/KafkaCACerts/kafka-cluster-ca.yaml << EOL
---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-cluster-ca-cert
  namespace: kafka
  labels:
    strimzi.io/kind: Kafka
    strimzi.io/cluster: kafka
  annotations:
    strimzi.io/ca-cert-generation: "0"
data:
  ca.crt: $(base64 -w0 ~/kafka/certs/templates/kafka-cluster-ca.crt)
---
apiVersion: v1
kind: Secret
metadata:
  name: kafka-cluster-ca
  namespace: kafka
  labels:
    strimzi.io/kind: Kafka
    strimzi.io/cluster: kafka
  annotations:
    strimzi.io/ca-key-generation: "0"
data:
  ca.key: $(kubectl get secret kafka-cluster-ca-template -n kafka -o jsonpath='{.data.tls\.key}')
EOL
[michael@k8s ~]# kubectl create -f ~/kafka/certs/KafkaCACerts/kafka-cluster-ca.yaml

Deploy Kafka Cluster


[michael@k8s ~]# cat > ~/kafka/kafka.yaml << EOL
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka
  namespace: kafka
spec:
  clientsCa:
    generateCertificateAuthority: false
    generateSecretOwnerReference: false
  clusterCa:
    generateCertificateAuthority: false
    generateSecretOwnerReference: false
  kafka:
    version: 3.9.0
    replicas: 3
    listeners:
      - name: internal
        port: 9092
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9093
        type: nodeport
        tls: true
        authentication:
          type: tls
        configuration:
          bootstrap:
            nodePort: 31000
          brokers:
            - broker: 0
              nodePort: 31001
            - broker: 1
              nodePort: 31002
            - broker: 2
              nodePort: 31003
    config:
      offsets.topic.replication.factor: 1
      transaction.state.log.replication.factor: 1
      transaction.state.log.min.isr: 1
      default.replication.factor: 1
      min.insync.replicas: 1
      inter.broker.protocol.version: "3.9"
    authorization:
      type: simple
    superUsers:
      - CN=kafdrop
    storage:
      type: ephemeral
  zookeeper:
    replicas: 3
    storage:
      type: ephemeral
  entityOperator:
    topicOperator: {}
    userOperator: {}
EOL
[michael@k8s ~]# kubectl create -f ~/kafka/kafka.yaml

Kafdrop

Kafdrop User


[michael@k8s ~]# mkdir --parents ~/kafka/users ~/kafka/topics
[michael@k8s ~]# cat > ~/kafka/users/kafdrop.yaml << EOL
---
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
  name: kafdrop
  namespace: kafka
  labels:
    strimzi.io/cluster: kafka
spec:
  authentication:
    type: tls
  template:
    secret:
      metadata:
        annotations:
          reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
          reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "kafdrop"
          reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
          reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "kafdrop"
EOL
[michael@k8s ~]# kubectl create -f ~/kafka/users/kafdrop.yaml

Deploy Kafdrop


[michael@k8s ~]# mkdir --parents ~/kafka/kafdrop
[michael@k8s ~]# cat > ~/kafka/kafdrop/kafdrop.yaml << EOL
---
apiVersion: v1
kind: Namespace
metadata:
  name: kafdrop
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kafdrop
  name: kafka-properties
data:
  kafka.properties: |
    bootstrap.servers: kafka-kafka-bootstrap.kafka.svc.cluster.local:9092
    security.protocol: SSL
    ssl.keystore.type: PKCS12
    ssl.keystore.location: /lib/ssl/private/kafdrop-certs/user.p12
    ssl.keystore.password: $(kubectl get secret kafdrop -n kafka -o jsonpath='{.data.user\.password}' | base64 -d)
    ssl.truststore.type: PEM
    ssl.truststore.location: /lib/ssl/private/kafdrop-certs/ca.crt
---
apiVersion: v1
kind: Service
metadata:
  namespace: kafdrop
  name: kafdrop
spec:
  type: NodePort
  selector:
    app: kafdrop
  ports:
    - port: 9000
      nodePort: 31004
      protocol: TCP
      targetPort: http
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kafdrop
  name: kafdrop
spec:
  selector:
    matchLabels:
      app: kafdrop
  replicas: 1
  template:
    metadata:
      labels:
        app: kafdrop
    spec:
      automountServiceAccountToken: false
      containers:
        - name: kafdrop
          image: obsidiandynamics/kafdrop:3.30.0
          imagePullPolicy: IfNotPresent
          env:
            - name: KAFKA_PROPERTIES_FILE
              value: /etc/kafdrop/kafka.properties
          ports:
            - name: http
              containerPort: 9000
              protocol: TCP
          volumeMounts:
            - mountPath: /etc/kafdrop
              name: kafka-properties
            - mountPath: /lib/ssl/private/kafdrop-certs
              name: kafdrop-certs
      volumes:
        - name: kafka-properties
          configMap:
            name: kafka-properties
        - name: kafdrop-certs
          secret:
            secretName: kafdrop
EOL

blog.michaelolson.io