Configuring metrics for core platform monitoring

Configure the collection of metrics to monitor how cluster components and your own workloads are performing.

You can send ingested metrics to remote systems for long-term storage and add cluster ID labels to the metrics to identify the data coming from different clusters.

Additional resources

• Understanding metrics

Configuring remote write storage

You can configure remote write storage to enable Prometheus to send ingested metrics to remote systems for long-term storage. Doing so has no impact on how or for how long Prometheus stores metrics.

Prerequisites

• You have access to the cluster as a user with the cluster-admin cluster role.

• You have created the cluster-monitoring-config ConfigMap object.

• You have installed the OpenShift CLI (oc).

• You have set up a remote write compatible endpoint (such as Thanos) and know the endpoint URL. See the Prometheus remote endpoints and storage documentation for information about endpoints that are compatible with the remote write feature.

  Important

  Red Hat only provides information for configuring remote write senders and does not offer guidance on configuring receiver endpoints. Customers are responsible for setting up their own endpoints that are remote-write compatible. Issues with endpoint receiver configurations are not included in Red Hat production support.

• You have set up authentication credentials in a Secret object for the remote write endpoint. You must create the secret in the openshift-monitoring namespace.

  Warning

  To reduce security risks, use HTTPS and authentication to send metrics to an endpoint.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. Add a remoteWrite: section under data/config.yaml/prometheusK8s, as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com" 
           <endpoint_authentication_credentials> 

   1. The URL of the remote write endpoint.
   2. The authentication method and credentials for the endpoint. Currently supported authentication methods are AWS Signature Version 4, authentication using HTTP in an Authorization request header, Basic authentication, OAuth 2.0, and TLS client. See Supported remote write authentication settings for sample configurations of supported authentication methods.

3. Add write relabel configuration values after the authentication credentials:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs:
           - <your_write_relabel_configs> 

   1. Add configuration for metrics that you want to send to the remote endpoint.
      Example of forwarding a single metric called my_metric

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          prometheusK8s:
            remoteWrite:
            - url: "https://remote-write-endpoint.example.com"
              writeRelabelConfigs:
              - sourceLabels: [__name__]
                regex: 'my_metric'
                action: keep

      Example of forwarding metrics called my_metric_1 and my_metric_2 in my_namespace namespace

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          prometheusK8s:
            remoteWrite:
            - url: "https://remote-write-endpoint.example.com"
              writeRelabelConfigs:
              - sourceLabels: [__name__,namespace]
                regex: '(my_metric_1|my_metric_2);my_namespace'
                action: keep

4. Save the file to apply the changes. The new configuration is applied automatically.

Additional resources

• writeRelabelConfigs

• relabel_config (Prometheus documentation)

Supported remote write authentication settings

You can use different methods to authenticate with a remote write endpoint. Currently supported authentication methods are AWS Signature Version 4, basic authentication, authorization, OAuth 2.0, and TLS client. The following table provides details about supported authentication methods for use with remote write.

Authentication method	Config map field	Description
AWS Signature Version 4	sigv4	This method uses AWS Signature Version 4 authentication to sign requests. You cannot use this method simultaneously with authorization, OAuth 2.0, or Basic authentication.
Basic authentication	basicAuth	Basic authentication sets the authorization header on every remote write request with the configured username and password.
authorization	authorization	Authorization sets the Authorization header on every remote write request using the configured token.
OAuth 2.0	oauth2	An OAuth 2.0 configuration uses the client credentials grant type. Prometheus fetches an access token from tokenUrl with the specified client ID and client secret to access the remote write endpoint. You cannot use this method simultaneously with authorization, AWS Signature Version 4, or Basic authentication.
TLS client	tlsConfig	A TLS client configuration specifies the CA certificate, the client certificate, and the client key file information used to authenticate with the remote write endpoint server using TLS. The sample configuration assumes that you have already created a CA certificate file, a client certificate file, and a client key file.

Example remote write authentication settings

The following samples show different authentication settings you can use to connect to a remote write endpoint. Each sample also shows how to configure a corresponding Secret object that contains authentication credentials and other relevant settings. Each sample configures authentication for use with default platform monitoring in the openshift-monitoring namespace.

Sample YAML for AWS Signature Version 4 authentication

The following shows the settings for a sigv4 secret named sigv4-credentials in the openshift-monitoring namespace.

apiVersion: v1
kind: Secret
metadata:
  name: sigv4-credentials
  namespace: openshift-monitoring
stringData:
  accessKey: <AWS_access_key> 
  secretKey: <AWS_secret_key> 
type: Opaque

1. The AWS API access key.
2. The AWS API secret key.

The following shows sample AWS Signature Version 4 remote write authentication settings that use a Secret object named sigv4-credentials in the openshift-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://authorization.example.com/api/write"
        sigv4:
          region: <AWS_region> 
          accessKey:
            name: sigv4-credentials 
            key: accessKey 
          secretKey:
            name: sigv4-credentials 
            key: secretKey 
          profile: <AWS_profile_name> 
          roleArn: <AWS_role_arn> 

1. The AWS region.
2. The name of the Secret object containing the AWS API access credentials.
3. The key that contains the AWS API access key in the specified Secret object.
4. The key that contains the AWS API secret key in the specified Secret object.
5. The name of the AWS profile that is being used to authenticate.
6. The unique identifier for the Amazon Resource Name (ARN) assigned to your role.

Sample YAML for Basic authentication

The following shows sample Basic authentication settings for a Secret object named rw-basic-auth in the openshift-monitoring namespace:

apiVersion: v1
kind: Secret
metadata:
  name: rw-basic-auth
  namespace: openshift-monitoring
stringData:
  user: <basic_username> 
  password: <basic_password> 
type: Opaque

1. The username.
2. The password.

The following sample shows a basicAuth remote write configuration that uses a Secret object named rw-basic-auth in the openshift-monitoring namespace. It assumes that you have already set up authentication credentials for the endpoint.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://basicauth.example.com/api/write"
        basicAuth:
          username:
            name: rw-basic-auth 
            key: user 
          password:
            name: rw-basic-auth 
            key: password 

1. The name of the Secret object that contains the authentication credentials.
2. The key that contains the username in the specified Secret object.
3. The key that contains the password in the specified Secret object.

Sample YAML for authentication with a bearer token using a Secret Object

The following shows bearer token settings for a Secret object named rw-bearer-auth in the openshift-monitoring namespace:

apiVersion: v1
kind: Secret
metadata:
  name: rw-bearer-auth
  namespace: openshift-monitoring
stringData:
  token: <authentication_token> 
type: Opaque

1. The authentication token.

The following shows sample bearer token config map settings that use a Secret object named rw-bearer-auth in the openshift-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://authorization.example.com/api/write"
        authorization:
          type: Bearer 
          credentials:
            name: rw-bearer-auth 
            key: token 

1. The authentication type of the request. The default value is Bearer.
2. The name of the Secret object that contains the authentication credentials.
3. The key that contains the authentication token in the specified Secret object.

Sample YAML for OAuth 2.0 authentication

The following shows sample OAuth 2.0 settings for a Secret object named oauth2-credentials in the openshift-monitoring namespace:

apiVersion: v1
kind: Secret
metadata:
  name: oauth2-credentials
  namespace: openshift-monitoring
stringData:
  id: <oauth2_id> 
  secret: <oauth2_secret> 
type: Opaque

1. The Oauth 2.0 ID.
2. The OAuth 2.0 secret.

The following shows an oauth2 remote write authentication sample configuration that uses a Secret object named oauth2-credentials in the openshift-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://test.example.com/api/write"
        oauth2:
          clientId:
            secret:
              name: oauth2-credentials 
              key: id 
          clientSecret:
            name: oauth2-credentials 
            key: secret 
          tokenUrl: https://example.com/oauth2/token 
          scopes: 
          - <scope_1>
          - <scope_2>
          endpointParams: 
            param1: <parameter_1>
            param2: <parameter_2>

1. The name of the corresponding Secret object. Note that ClientId can alternatively refer to a ConfigMap object, although clientSecret must refer to a Secret object.
2. The key that contains the OAuth 2.0 credentials in the specified Secret object.
3. The URL used to fetch a token with the specified clientId and clientSecret.
4. The OAuth 2.0 scopes for the authorization request. These scopes limit what data the tokens can access.
5. The OAuth 2.0 authorization request parameters required for the authorization server.

Sample YAML for TLS client authentication

The following shows sample TLS client settings for a tls Secret object named mtls-bundle in the openshift-monitoring namespace.

apiVersion: v1
kind: Secret
metadata:
  name: mtls-bundle
  namespace: openshift-monitoring
data:
  ca.crt: <ca_cert> 
  client.crt: <client_cert> 
  client.key: <client_key> 
type: tls

1. The CA certificate in the Prometheus container with which to validate the server certificate.
2. The client certificate for authentication with the server.
3. The client key.

The following sample shows a tlsConfig remote write authentication configuration that uses a TLS Secret object named mtls-bundle.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        tlsConfig:
          ca:
            secret:
              name: mtls-bundle 
              key: ca.crt 
          cert:
            secret:
              name: mtls-bundle 
              key: client.crt 
          keySecret:
            name: mtls-bundle 
            key: client.key 

1. The name of the corresponding Secret object that contains the TLS authentication credentials. Note that ca and cert can alternatively refer to a ConfigMap object, though keySecret must refer to a Secret object.
2. The key in the specified Secret object that contains the CA certificate for the endpoint.
3. The key in the specified Secret object that contains the client certificate for the endpoint.
4. The key in the specified Secret object that contains the client key secret.

Example remote write queue configuration

You can use the queueConfig object for remote write to tune the remote write queue parameters. The following example shows the queue parameters with their default values for default platform monitoring in the openshift-monitoring namespace.

Example configuration of remote write parameters with default values

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        <endpoint_authentication_credentials>
        queueConfig:
          capacity: 10000 
          minShards: 1 
          maxShards: 50 
          maxSamplesPerSend: 2000 
          batchSendDeadline: 5s 
          minBackoff: 30ms 
          maxBackoff: 5s 
          retryOnRateLimit: false 
          sampleAgeLimit: 0s 

1. The number of samples to buffer per shard before they are dropped from the queue.
2. The minimum number of shards.
3. The maximum number of shards.
4. The maximum number of samples per send.
5. The maximum time for a sample to wait in buffer.
6. The initial time to wait before retrying a failed request. The time gets doubled for every retry up to the maxbackoff time.
7. The maximum time to wait before retrying a failed request.
8. Set this parameter to true to retry a request after receiving a 429 status code from the remote write storage.
9. The samples that are older than the sampleAgeLimit limit are dropped from the queue. If the value is undefined or set to 0s, the parameter is ignored.

Additional resources

• Prometheus REST API reference for remote write

• Remote write compatible endpoints (Prometheus documentation)

• Remote write tuning (Prometheus documentation)

• Understanding secrets

Table of remote write metrics

The following table contains remote write and remote write-adjacent metrics with further description to help solve issues during remote write configuration.

Metric	Description
prometheus_remote_storage_highest_timestamp_in_seconds	Shows the newest timestamp that Prometheus stored in the write-ahead log (WAL) for any sample.
prometheus_remote_storage_queue_highest_sent_timestamp_seconds	Shows the newest timestamp that the remote write queue successfully sent.
prometheus_remote_storage_samples_retried_total	The number of samples that remote write failed to send and had to resend to remote storage. A steady high rate for this metric indicates problems with the network or remote storage endpoint.
prometheus_remote_storage_shards	Shows how many shards are currently running for each remote endpoint.
prometheus_remote_storage_shards_desired	Shows the calculated needed number of shards based on the current write throughput and the rate of incoming versus sent samples.
prometheus_remote_storage_shards_max	Shows the maximum number of shards based on the current configuration.
prometheus_remote_storage_shards_min	Shows the minimum number of shards based on the current configuration.
prometheus_tsdb_wal_segment_current	The WAL segment file that Prometheus is currently writing new data to.
prometheus_wal_watcher_current_segment	The WAL segment file that each remote write instance is currently reading from.

Creating cluster ID labels for metrics

You can create cluster ID labels for metrics by adding the write_relabel settings for remote write storage in the cluster-monitoring-config config map in the openshift-monitoring namespace.

Prerequisites

• You have access to the cluster as a user with the cluster-admin cluster role.

• You have created the cluster-monitoring-config ConfigMap object.

• You have installed the OpenShift CLI (oc).

• You have configured remote write storage.

Procedure

1. Edit the cluster-monitoring-config config map in the openshift-monitoring project:

   $ oc -n openshift-monitoring edit configmap cluster-monitoring-config

2. In the writeRelabelConfigs: section under data/config.yaml/prometheusK8s/remoteWrite, add cluster ID relabel configuration values:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       prometheusK8s:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs: 
             - <relabel_config> 

   1. Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.
   2. Substitute the label configuration for the metrics sent to the remote write endpoint.

      The following sample shows how to forward a metric with the cluster ID label cluster_id:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: cluster-monitoring-config
        namespace: openshift-monitoring
      data:
        config.yaml: |
          prometheusK8s:
            remoteWrite:
            - url: "https://remote-write-endpoint.example.com"
              writeRelabelConfigs:
              - sourceLabels:
                - __tmp_openshift_cluster_id__ 
                targetLabel: cluster_id 
                action: replace 

   3. The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.
   4. Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.
   5. The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

3. Save the file to apply the changes. The new configuration is applied automatically.

Additional resources

• Adding cluster ID labels to metrics

• Obtaining your cluster ID