Gathering audit logs

You can gather audit logs, which are a security-relevant chronological set of records documenting the sequence of activities that have affected the system by individual users, administrators, or other components of the system.

For example, audit logs can help you understand where an autoscaling request is coming from. This is key information when backends are getting overloaded by autoscaling requests made by user applications and you need to determine which is the troublesome application.

Configuring audit logging

You can configure auditing for the Custom Metrics Autoscaler Operator by editing the KedaController custom resource. The logs are sent to an audit log file on a volume that is secured by using a persistent volume claim in the KedaController CR.

Prerequisites

• The Custom Metrics Autoscaler Operator must be installed.

Procedure

1. Edit the KedaController custom resource to add the auditConfig stanza:

   kind: KedaController
   apiVersion: keda.sh/v1alpha1
   metadata:
     name: keda
     namespace: openshift-keda
   spec:
   # ...
     metricsServer:
   # ...
       auditConfig:
         logFormat: "json" 
         logOutputVolumeClaim: "pvc-audit-log" 
         policy:
           rules: 
           - level: Metadata
           omitStages: "RequestReceived" 
           omitManagedFields: false 
         lifetime: 
           maxAge: "2"
           maxBackup: "1"
           maxSize: "50"

   1. Specifies the output format of the audit log, either legacy or json.
   2. Specifies an existing persistent volume claim for storing the log data. All requests coming to the API server are logged to this persistent volume claim. If you leave this field empty, the log data is sent to stdout.
   3. Specifies which events should be recorded and what data they should include:

      • None: Do not log events.

      • Metadata: Log only the metadata for the request, such as user, timestamp, and so forth. Do not log the request text and the response text. This is the default.

      • Request: Log only the metadata and the request text but not the response text. This option does not apply for non-resource requests.

      • RequestResponse: Log event metadata, request text, and response text. This option does not apply for non-resource requests.

   4. Specifies stages for which no event is created.
   5. Specifies whether to omit the managed fields of the request and response bodies from being written to the API audit log, either true to omit the fields or false to include the fields.
   6. Specifies the size and lifespan of the audit logs.

      • maxAge: The maximum number of days to retain audit log files, based on the timestamp encoded in their filename.

      • maxBackup: The maximum number of audit log files to retain. Set to 0 to retain all audit log files.

      • maxSize: The maximum size in megabytes of an audit log file before it gets rotated.

Verification

1. View the audit log file directly:

   1. Obtain the name of the keda-metrics-apiserver-* pod:

      oc get pod -n openshift-keda

      Example output

      NAME                                                  READY   STATUS    RESTARTS   AGE
      custom-metrics-autoscaler-operator-5cb44cd75d-9v4lv   1/1     Running   0          8m20s
      keda-metrics-apiserver-65c7cc44fd-rrl4r               1/1     Running   0          2m55s
      keda-operator-776cbb6768-zpj5b                        1/1     Running   0          2m55s

   2. View the log data by using a command similar to the following:

      $ oc logs keda-metrics-apiserver-<hash>|grep -i metadata 

      1. Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

         For example:

         $ oc logs keda-metrics-apiserver-65c7cc44fd-rrl4r|grep -i metadata

         Example output

          ...
         {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"4c81d41b-3dab-4675-90ce-20b87ce24013","stage":"ResponseComplete","requestURI":"/healthz","verb":"get","user":{"username":"system:anonymous","groups":["system:unauthenticated"]},"sourceIPs":["10.131.0.1"],"userAgent":"kube-probe/1.28","responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2023-02-16T13:00:03.554567Z","stageTimestamp":"2023-02-16T13:00:03.555032Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
          ...

2. Alternatively, you can view a specific log:

   1. Use a command similar to the following to log into the keda-metrics-apiserver-* pod:

      $ oc rsh pod/keda-metrics-apiserver-<hash> -n openshift-keda

      For example:

      $ oc rsh pod/keda-metrics-apiserver-65c7cc44fd-rrl4r -n openshift-keda

   2. Change to the /var/audit-policy/ directory:

      sh-4.4$ cd /var/audit-policy/

   3. List the available logs:

      sh-4.4$ ls

      Example output

      log-2023.02.17-14:50  policy.yaml

   4. View the log, as needed:

      sh-4.4$ cat <log_name>/<pvc_name>|grep -i <log_level> 

      1. Optional: You can use the grep command to specify the log level to display: Metadata, Request, RequestResponse.

         For example:

         sh-4.4$ cat log-2023.02.17-14:50/pvc-audit-log|grep -i Request

         Example output

          ...
         {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Request","auditID":"63e7f68c-04ec-4f4d-8749-bf1656572a41","stage":"ResponseComplete","requestURI":"/openapi/v2","verb":"get","user":{"username":"system:aggregator","groups":["system:authenticated"]},"sourceIPs":["10.128.0.1"],"responseStatus":{"metadata":{},"code":304},"requestReceivedTimestamp":"2023-02-17T13:12:55.035478Z","stageTimestamp":"2023-02-17T13:12:55.038346Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:discovery\" of ClusterRole \"system:discovery\" to Group \"system:authenticated\""}}
          ...