Audit a K3s Cluster in Wazuh

Configure the Wazuh Server

Special thanks to the awesome documentation from the Wazuh team. Where I got the first part of this blog post from.

Create certificates for communication between the Wazuh server and Kubernetes

Create a directory for the webhook endpoint.

 mkdir -p /var/ossec/integrations/kubernetes-webhook/

Add the following to /var/ossec/integrations/kubernetes-webhook/csr.conf. Replace <wazuh_server_ip> and <wazuh_server_ip> with your server’s IP address. The file content from the Wazuh documentation didn’t work for me.

  
 cat > /var/ossec/integrations/kubernetes-webhook/csr.conf << EOF
 [ req ]
 prompt = no
 default_bits = 2048
 default_md = sha256
 distinguished_name = req_distinguished_name
 x509_extensions = v3_req
 [req_distinguished_name]
 C = US
 ST = California
 L = San Jose
 O = Wazuh
 OU = Research and development
 emailAddress = info@wazuh.com
 CN = <wazuh_server_ip>
 [ v3_req ]
 authorityKeyIdentifier=keyid,issuer
 basicConstraints = CA:FALSE
 keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
 subjectAltName = @alt_names
 [alt_names]
 IP.1 = <wazuh_server_ip>
 EOF

Create the root CA public and private keys.

  
 openssl req -x509 -new -nodes -newkey rsa:2048 \
   -keyout /var/ossec/integrations/kubernetes-webhook/rootCA.key \
   -out /var/ossec/integrations/kubernetes-webhook/rootCA.pem \
   -batch -subj "/C=US/ST=California/L=San Jose/O=Wazuh"

Create the certificate signing request and the server’s private key.

  
 openssl req -new -nodes -newkey rsa:2048 \
   -keyout /var/ossec/integrations/kubernetes-webhook/server.key \
   -out /var/ossec/integrations/kubernetes-webhook/server.csr \
   -config /var/ossec/integrations/kubernetes-webhook/csr.conf

Generate the server’s certificate.

  
 openssl x509 -req -in /var/ossec/integrations/kubernetes-webhook/server.csr \
   -CA /var/ossec/integrations/kubernetes-webhook/rootCA.pem \
   -CAkey /var/ossec/integrations/kubernetes-webhook/rootCA.key \
   -CAcreateserial \
   -out /var/ossec/integrations/kubernetes-webhook/server.crt \
   -extfile /var/ossec/integrations/kubernetes-webhook/csr.conf \
   -extensions v3_req

Create Listener using Python

The listener acts as a webhook and parses all incoming requests from the Kubernetes cluster.

Install Flask using pip.

 /var/ossec/framework/python/bin/pip3 install flask

Replace <wazuh_server_ip> and copy the content into the file /var/ossec/integrations/custom-webhook.py.

  
 #!/var/ossec/framework/python/bin/python3
 import json
 from socket import socket, AF_UNIX, SOCK_DGRAM
 from flask import Flask, request

 PORT     = 8080
 CERT     = '/var/ossec/integrations/kubernetes-webhook/server.crt'
 CERT_KEY = '/var/ossec/integrations/kubernetes-webhook/server.key'
 socket_addr = '/var/ossec/queue/sockets/queue'

 def send_event(msg):
     string = '1:k8s:{0}'.format(json.dumps(msg))
     sock = socket(AF_UNIX, SOCK_DGRAM)
     sock.connect(socket_addr)
     sock.send(string.encode())
     sock.close()
     return True

 app = Flask(__name__)
 context = (CERT, CERT_KEY)
 @app.route('/', methods=['POST'])
 def webhook():
     if request.method == 'POST':
         if send_event(request.json):
             print("Request sent to Wazuh")
         else:
             print("Failed to send request to Wazuh")
     return "Webhook received!"

 if __name__ == '__main__':
     app.run(host='<wazuh_server_ip>', port=PORT, ssl_context=context)

Create a systemd service at /lib/systemd/system/wazuh-webhook.service.

 [Unit]
 Description=Wazuh webhook
 Wants=network-online.target
 After=network.target network-online.target

 [Service]
 ExecStart=/var/ossec/framework/python/bin/python3 /var/ossec/integrations/custom-webhook.py
 Restart=on-failure

 [Install]
 WantedBy=multi-user.target

Start the service

 systemctl daemon-reload
 systemctl enable wazuh-webhook.service
 systemctl start wazuh-webhook.service
 systemctl status wazuh-webhook.service

Enable access to port 8080 if a firewall is running on the Wazuh server.

Configure the Kubernetes Cluster

SSH into the server your k3s master node is running on.

Create log directory

  
 mkdir -p -m 700 /var/lib/rancher/k3s/server/logs

Create an audit policy at /var/lib/rancher/k3s/server/audit.yaml.

  
 apiVersion: audit.k8s.io/v1
 kind: Policy
 rules:
     # Don’t log requests to the following API endpoints
     - level: None
       nonResourceURLs:
           - '/healthz*'
           - '/logs'
           - '/metrics'
           - '/swagger*'
           - '/version'

     # Limit requests containing tokens to Metadata level so the token is not included in the log
     - level: Metadata
       omitStages:
           - RequestReceived
       resources:
           - group: authentication.k8s.io
             resources:
                 - tokenreviews

     # Extended audit of auth delegation
     - level: RequestResponse
       omitStages:
           - RequestReceived
       resources:
           - group: authorization.k8s.io
             resources:
                 - subjectaccessreviews

     # Log changes to pods at RequestResponse level
     - level: RequestResponse
       omitStages:
           - RequestReceived
       resources:
           # core API group; add third-party API services and your API services if needed
           - group: ''
             resources: ['pods']
             verbs: ['create', 'patch', 'update', 'delete']

     # Log everything else at Metadata level
     - level: Metadata
       omitStages:
           - RequestReceived

Replace <wazuh_server_ip> and create the webhook configuration file /var/lib/rancher/k3s/server/audit-webhook.yaml.

  
 apiVersion: v1
 kind: Config
 preferences: {}
 clusters:
   - name: wazuh-webhook
     cluster:
       insecure-skip-tls-verify: true
       server: https://<wazuh_server_ip>:8080 

 # kubeconfig files require a context. Provide one for the API server.
 current-context: webhook
 contexts:
 - context:
     cluster: wazuh-webhook
     user: kube-apiserver # Replace with name of API server if it’s different
   name: webhook

Add the Kubeapi parameters to load the auditing and webhook configurations.

  
 ExecStart=/usr/local/bin/k3s \
 server \
 '--kube-apiserver-arg=audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log' \
 '--kube-apiserver-arg=audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml' \
 '--kube-apiserver-arg=audit-webhook-config-file=/var/lib/rancher/k3s/server/audit-webhook.yaml' \
 '--kube-apiserver-arg=audit-webhook-batch-max-size=1' \

Create detection rule on Wazuh Server

Add the following rules to the Wazuh server at /var/ossec/etc/rules/local_rules.xml.

  
 <group name="k8s_audit,">
   <rule id="110002" level="0">
     <location>k8s</location>
     <field name="apiVersion">audit</field>
     <description>Kubernetes audit log.</description>
   </rule>

   <rule id="110003" level="5">
     <if_sid>110002</if_sid>
     <regex type="pcre2">requestURI\":.+", \"verb\": \"create</regex>
     <description>Kubernetes request to create resource</description>
   </rule>

   <rule id="110004" level="5">
     <if_sid>110002</if_sid>
     <regex type="pcre2">requestURI\":.+", \"verb\": \"delete</regex>
     <description>Kubernetes request to delete resource</description>
   </rule>
 </group>

Restart the server.
1 systemctl restart wazuh-manager

Test Configuration

Run the following command on the Kubernetes master node to create a new deployment.
1 kubectl create deployment hello-wazuh --image=k8s.gcr.io/echoserver:1.4
Run the following command to delete the deployment.
1 kubectl delete deployment hello-wazuh

Audit a K3s Cluster in Wazuh

Configure the Wazuh Server

Create certificates for communication between the Wazuh server and Kubernetes

Create Listener using Python

Configure the Kubernetes Cluster

Create detection rule on Wazuh Server

Test Configuration

Further Reading

Monitor Proxmox Firewall using Graylog

Discord Rat 2.0: Malware Analysis

Cyber Apocalypse 2023: Relic Maps