Configuring metrics for user workload monitoring

Configure the collection of metrics to monitor how cluster components and your own workloads are performing.

You can send ingested metrics to remote systems for long-term storage and add cluster ID labels to the metrics to identify the data coming from different clusters.

Additional resources

• Understanding metrics

Configuring remote write storage

You can configure remote write storage to enable Prometheus to send ingested metrics to remote systems for long-term storage. Doing so has no impact on how or for how long Prometheus stores metrics.

Prerequisites

• You have access to the cluster as a user with the cluster-admin cluster role or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.

• A cluster administrator has enabled monitoring for user-defined projects.

• You have installed the OpenShift CLI (oc).

• You have set up a remote write compatible endpoint (such as Thanos) and know the endpoint URL. See the Prometheus remote endpoints and storage documentation for information about endpoints that are compatible with the remote write feature.

  Important

  Red Hat only provides information for configuring remote write senders and does not offer guidance on configuring receiver endpoints. Customers are responsible for setting up their own endpoints that are remote-write compatible. Issues with endpoint receiver configurations are not included in Red Hat production support.

• You have set up authentication credentials in a Secret object for the remote write endpoint. You must create the secret in the openshift-user-workload-monitoring namespace.

  Warning

  To reduce security risks, use HTTPS and authentication to send metrics to an endpoint.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. Add a remoteWrite: section under data/config.yaml/prometheus, as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com" 
           <endpoint_authentication_credentials> 

   1. The URL of the remote write endpoint.
   2. The authentication method and credentials for the endpoint. Currently supported authentication methods are AWS Signature Version 4, authentication using HTTP in an Authorization request header, Basic authentication, OAuth 2.0, and TLS client. See Supported remote write authentication settings for sample configurations of supported authentication methods.

3. Add write relabel configuration values after the authentication credentials:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs:
           - <your_write_relabel_configs> 

   1. Add configuration for metrics that you want to send to the remote endpoint.
      Example of forwarding a single metric called my_metric

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheus:
            remoteWrite:
            - url: "https://remote-write-endpoint.example.com"
              writeRelabelConfigs:
              - sourceLabels: [__name__]
                regex: 'my_metric'
                action: keep

      Example of forwarding metrics called my_metric_1 and my_metric_2 in my_namespace namespace

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheus:
            remoteWrite:
            - url: "https://remote-write-endpoint.example.com"
              writeRelabelConfigs:
              - sourceLabels: [__name__,namespace]
                regex: '(my_metric_1|my_metric_2);my_namespace'
                action: keep

4. Save the file to apply the changes. The new configuration is applied automatically.

Additional resources

• writeRelabelConfigs

• relabel_config (Prometheus documentation)

Supported remote write authentication settings

You can use different methods to authenticate with a remote write endpoint. Currently supported authentication methods are AWS Signature Version 4, basic authentication, authorization, OAuth 2.0, and TLS client. The following table provides details about supported authentication methods for use with remote write.

Authentication method	Config map field	Description
AWS Signature Version 4	sigv4	This method uses AWS Signature Version 4 authentication to sign requests. You cannot use this method simultaneously with authorization, OAuth 2.0, or Basic authentication.
Basic authentication	basicAuth	Basic authentication sets the authorization header on every remote write request with the configured username and password.
authorization	authorization	Authorization sets the Authorization header on every remote write request using the configured token.
OAuth 2.0	oauth2	An OAuth 2.0 configuration uses the client credentials grant type. Prometheus fetches an access token from tokenUrl with the specified client ID and client secret to access the remote write endpoint. You cannot use this method simultaneously with authorization, AWS Signature Version 4, or Basic authentication.
TLS client	tlsConfig	A TLS client configuration specifies the CA certificate, the client certificate, and the client key file information used to authenticate with the remote write endpoint server using TLS. The sample configuration assumes that you have already created a CA certificate file, a client certificate file, and a client key file.

Example remote write authentication settings

The following samples show different authentication settings you can use to connect to a remote write endpoint. Each sample also shows how to configure a corresponding Secret object that contains authentication credentials and other relevant settings. Each sample configures authentication for use with monitoring for user-defined projects in the openshift-user-workload-monitoring namespace.

Sample YAML for AWS Signature Version 4 authentication

The following shows the settings for a sigv4 secret named sigv4-credentials in the openshift-user-workload-monitoring namespace.

apiVersion: v1
kind: Secret
metadata:
  name: sigv4-credentials
  namespace: openshift-user-workload-monitoring
stringData:
  accessKey: <AWS_access_key> 
  secretKey: <AWS_secret_key> 
type: Opaque

1. The AWS API access key.
2. The AWS API secret key.

The following shows sample AWS Signature Version 4 remote write authentication settings that use a Secret object named sigv4-credentials in the openshift-user-workload-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://authorization.example.com/api/write"
        sigv4:
          region: <AWS_region> 
          accessKey:
            name: sigv4-credentials 
            key: accessKey 
          secretKey:
            name: sigv4-credentials 
            key: secretKey 
          profile: <AWS_profile_name> 
          roleArn: <AWS_role_arn> 

1. The AWS region.
2. The name of the Secret object containing the AWS API access credentials.
3. The key that contains the AWS API access key in the specified Secret object.
4. The key that contains the AWS API secret key in the specified Secret object.
5. The name of the AWS profile that is being used to authenticate.
6. The unique identifier for the Amazon Resource Name (ARN) assigned to your role.

Sample YAML for Basic authentication

The following shows sample Basic authentication settings for a Secret object named rw-basic-auth in the openshift-user-workload-monitoring namespace:

apiVersion: v1
kind: Secret
metadata:
  name: rw-basic-auth
  namespace: openshift-user-workload-monitoring
stringData:
  user: <basic_username> 
  password: <basic_password> 
type: Opaque

1. The username.
2. The password.

The following sample shows a basicAuth remote write configuration that uses a Secret object named rw-basic-auth in the openshift-user-workload-monitoring namespace. It assumes that you have already set up authentication credentials for the endpoint.

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://basicauth.example.com/api/write"
        basicAuth:
          username:
            name: rw-basic-auth 
            key: user 
          password:
            name: rw-basic-auth 
            key: password 

1. The name of the Secret object that contains the authentication credentials.
2. The key that contains the username in the specified Secret object.
3. The key that contains the password in the specified Secret object.

Sample YAML for authentication with a bearer token using a Secret Object

The following shows bearer token settings for a Secret object named rw-bearer-auth in the openshift-user-workload-monitoring namespace:

apiVersion: v1
kind: Secret
metadata:
  name: rw-bearer-auth
  namespace: openshift-user-workload-monitoring
stringData:
  token: <authentication_token> 
type: Opaque

1. The authentication token.

The following shows sample bearer token config map settings that use a Secret object named rw-bearer-auth in the openshift-user-workload-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://authorization.example.com/api/write"
        authorization:
          type: Bearer 
          credentials:
            name: rw-bearer-auth 
            key: token 

1. The authentication type of the request. The default value is Bearer.
2. The name of the Secret object that contains the authentication credentials.
3. The key that contains the authentication token in the specified Secret object.

Sample YAML for OAuth 2.0 authentication

The following shows sample OAuth 2.0 settings for a Secret object named oauth2-credentials in the openshift-user-workload-monitoring namespace:

apiVersion: v1
kind: Secret
metadata:
  name: oauth2-credentials
  namespace: openshift-user-workload-monitoring
stringData:
  id: <oauth2_id> 
  secret: <oauth2_secret> 
type: Opaque

1. The Oauth 2.0 ID.
2. The OAuth 2.0 secret.

The following shows an oauth2 remote write authentication sample configuration that uses a Secret object named oauth2-credentials in the openshift-user-workload-monitoring namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://test.example.com/api/write"
        oauth2:
          clientId:
            secret:
              name: oauth2-credentials 
              key: id 
          clientSecret:
            name: oauth2-credentials 
            key: secret 
          tokenUrl: https://example.com/oauth2/token 
          scopes: 
          - <scope_1>
          - <scope_2>
          endpointParams: 
            param1: <parameter_1>
            param2: <parameter_2>

1. The name of the corresponding Secret object. Note that ClientId can alternatively refer to a ConfigMap object, although clientSecret must refer to a Secret object.
2. The key that contains the OAuth 2.0 credentials in the specified Secret object.
3. The URL used to fetch a token with the specified clientId and clientSecret.
4. The OAuth 2.0 scopes for the authorization request. These scopes limit what data the tokens can access.
5. The OAuth 2.0 authorization request parameters required for the authorization server.

Sample YAML for TLS client authentication

The following shows sample TLS client settings for a tls Secret object named mtls-bundle in the openshift-user-workload-monitoring namespace.

apiVersion: v1
kind: Secret
metadata:
  name: mtls-bundle
  namespace: openshift-user-workload-monitoring
data:
  ca.crt: <ca_cert> 
  client.crt: <client_cert> 
  client.key: <client_key> 
type: tls

1. The CA certificate in the Prometheus container with which to validate the server certificate.
2. The client certificate for authentication with the server.
3. The client key.

The following sample shows a tlsConfig remote write authentication configuration that uses a TLS Secret object named mtls-bundle.

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        tlsConfig:
          ca:
            secret:
              name: mtls-bundle 
              key: ca.crt 
          cert:
            secret:
              name: mtls-bundle 
              key: client.crt 
          keySecret:
            name: mtls-bundle 
            key: client.key 

1. The name of the corresponding Secret object that contains the TLS authentication credentials. Note that ca and cert can alternatively refer to a ConfigMap object, though keySecret must refer to a Secret object.
2. The key in the specified Secret object that contains the CA certificate for the endpoint.
3. The key in the specified Secret object that contains the client certificate for the endpoint.
4. The key in the specified Secret object that contains the client key secret.

Example remote write queue configuration

You can use the queueConfig object for remote write to tune the remote write queue parameters. The following example shows the queue parameters with their default values for monitoring for user-defined projects in the openshift-user-workload-monitoring namespace.

Example configuration of remote write parameters with default values

apiVersion: v1
kind: ConfigMap
metadata:
  name: user-workload-monitoring-config
  namespace: openshift-user-workload-monitoring
data:
  config.yaml: |
    prometheus:
      remoteWrite:
      - url: "https://remote-write-endpoint.example.com"
        <endpoint_authentication_credentials>
        queueConfig:
          capacity: 10000 
          minShards: 1 
          maxShards: 50 
          maxSamplesPerSend: 2000 
          batchSendDeadline: 5s 
          minBackoff: 30ms 
          maxBackoff: 5s 
          retryOnRateLimit: false 
          sampleAgeLimit: 0s 

1. The number of samples to buffer per shard before they are dropped from the queue.
2. The minimum number of shards.
3. The maximum number of shards.
4. The maximum number of samples per send.
5. The maximum time for a sample to wait in buffer.
6. The initial time to wait before retrying a failed request. The time gets doubled for every retry up to the maxbackoff time.
7. The maximum time to wait before retrying a failed request.
8. Set this parameter to true to retry a request after receiving a 429 status code from the remote write storage.
9. The samples that are older than the sampleAgeLimit limit are dropped from the queue. If the value is undefined or set to 0s, the parameter is ignored.

Additional resources

• Prometheus REST API reference for remote write

• Remote write compatible endpoints (Prometheus documentation)

• Remote write tuning (Prometheus documentation)

• Understanding secrets

Table of remote write metrics

The following table contains remote write and remote write-adjacent metrics with further description to help solve issues during remote write configuration.

Metric	Description
prometheus_remote_storage_highest_timestamp_in_seconds	Shows the newest timestamp that Prometheus stored in the write-ahead log (WAL) for any sample.
prometheus_remote_storage_queue_highest_sent_timestamp_seconds	Shows the newest timestamp that the remote write queue successfully sent.
prometheus_remote_storage_samples_retried_total	The number of samples that remote write failed to send and had to resend to remote storage. A steady high rate for this metric indicates problems with the network or remote storage endpoint.
prometheus_remote_storage_shards	Shows how many shards are currently running for each remote endpoint.
prometheus_remote_storage_shards_desired	Shows the calculated needed number of shards based on the current write throughput and the rate of incoming versus sent samples.
prometheus_remote_storage_shards_max	Shows the maximum number of shards based on the current configuration.
prometheus_remote_storage_shards_min	Shows the minimum number of shards based on the current configuration.
prometheus_tsdb_wal_segment_current	The WAL segment file that Prometheus is currently writing new data to.
prometheus_wal_watcher_current_segment	The WAL segment file that each remote write instance is currently reading from.

Creating cluster ID labels for metrics

You can create cluster ID labels for metrics by adding the write_relabel settings for remote write storage in the user-workload-monitoring-config config map in the openshift-user-workload-monitoring namespace.

Note

When Prometheus scrapes user workload targets that expose a namespace label, the system stores this label as exported_namespace. This behavior ensures that the final namespace label value is equal to the namespace of the target pod. You cannot override this default configuration by setting the value of the honorLabels field to true for PodMonitor or ServiceMonitor objects.

Prerequisites

• You have access to the cluster as a user with the cluster-admin cluster role, or as a user with the user-workload-monitoring-config-edit role in the openshift-user-workload-monitoring project.

• A cluster administrator has enabled monitoring for user-defined projects.

• You have installed the OpenShift CLI (oc).

• You have configured remote write storage.

Procedure

1. Edit the user-workload-monitoring-config config map in the openshift-user-workload-monitoring project:

   $ oc -n openshift-user-workload-monitoring edit configmap user-workload-monitoring-config

2. In the writeRelabelConfigs: section under data/config.yaml/prometheus/remoteWrite, add cluster ID relabel configuration values:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         remoteWrite:
         - url: "https://remote-write-endpoint.example.com"
           <endpoint_authentication_credentials>
           writeRelabelConfigs: 
             - <relabel_config> 

   1. Add a list of write relabel configurations for metrics that you want to send to the remote endpoint.
   2. Substitute the label configuration for the metrics sent to the remote write endpoint.

      The following sample shows how to forward a metric with the cluster ID label cluster_id:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: user-workload-monitoring-config
        namespace: openshift-user-workload-monitoring
      data:
        config.yaml: |
          prometheus:
            remoteWrite:
            - url: "https://remote-write-endpoint.example.com"
              writeRelabelConfigs:
              - sourceLabels:
                - __tmp_openshift_cluster_id__ 
                targetLabel: cluster_id 
                action: replace 

   3. The system initially applies a temporary cluster ID source label named __tmp_openshift_cluster_id__. This temporary label gets replaced by the cluster ID label name that you specify.
   4. Specify the name of the cluster ID label for metrics sent to remote write storage. If you use a label name that already exists for a metric, that value is overwritten with the name of this cluster ID label. For the label name, do not use __tmp_openshift_cluster_id__. The final relabeling step removes labels that use this name.
   5. The replace write relabel action replaces the temporary label with the target label for outgoing metrics. This action is the default and is applied if no action is specified.

3. Save the file to apply the changes. The new configuration is applied automatically.

Additional resources

• Adding cluster ID labels to metrics

• Obtaining your cluster ID

Setting up metrics collection for user-defined projects

You can create a ServiceMonitor resource to scrape metrics from a service endpoint in a user-defined project. This assumes that your application uses a Prometheus client library to expose metrics to the /metrics canonical name.

This section describes how to deploy a sample service in a user-defined project and then create a ServiceMonitor resource that defines how that service should be monitored.

Deploying a sample service

To test monitoring of a service in a user-defined project, you can deploy a sample service.

Prerequisites

• You have access to the cluster as a user with the cluster-admin cluster role or as a user with administrative permissions for the namespace.

Procedure

1. Create a YAML file for the service configuration. In this example, it is called prometheus-example-app.yaml.

2. Add the following deployment and service configuration details to the file:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: ns1
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: prometheus-example-app
     template:
       metadata:
         labels:
           app: prometheus-example-app
       spec:
         containers:
         - image: ghcr.io/rhobs/prometheus-example-app:0.4.2
           imagePullPolicy: IfNotPresent
           name: prometheus-example-app
   ---
   apiVersion: v1
   kind: Service
   metadata:
     labels:
       app: prometheus-example-app
     name: prometheus-example-app
     namespace: ns1
   spec:
     ports:
     - port: 8080
       protocol: TCP
       targetPort: 8080
       name: web
     selector:
       app: prometheus-example-app
     type: ClusterIP

   This configuration deploys a service named prometheus-example-app in the user-defined ns1 project. This service exposes the custom version metric.

3. Apply the configuration to the cluster:

   $ oc apply -f prometheus-example-app.yaml

   It takes some time to deploy the service.

4. You can check that the pod is running:

   $ oc -n ns1 get pod

   Example output

   NAME                                      READY     STATUS    RESTARTS   AGE
   prometheus-example-app-7857545cb7-sbgwq   1/1       Running   0          81m

Specifying how a service is monitored

To use the metrics exposed by your service, you must configure OpenShift Container Platform monitoring to scrape metrics from the /metrics endpoint. You can do this using a ServiceMonitor custom resource definition (CRD) that specifies how a service should be monitored, or a PodMonitor CRD that specifies how a pod should be monitored. The former requires a Service object, while the latter does not, allowing Prometheus to directly scrape metrics from the metrics endpoint exposed by a pod.

This procedure shows you how to create a ServiceMonitor resource for a service in a user-defined project.

Prerequisites

• You have access to the cluster as a user with the cluster-admin cluster role or the monitoring-edit cluster role.

• You have enabled monitoring for user-defined projects.

• For this example, you have deployed the prometheus-example-app sample service in the ns1 project.

  Note

  The prometheus-example-app sample service does not support TLS authentication.

Procedure

1. Create a new YAML configuration file named example-app-service-monitor.yaml.

2. Add a ServiceMonitor resource to the YAML file. The following example creates a service monitor named prometheus-example-monitor to scrape metrics exposed by the prometheus-example-app service in the ns1 namespace:

   apiVersion: monitoring.coreos.com/v1
   kind: ServiceMonitor
   metadata:
     name: prometheus-example-monitor
     namespace: ns1 
   spec:
     endpoints:
     - interval: 30s
       port: web 
       scheme: http
     selector: 
       matchLabels:
         app: prometheus-example-app

   1. Specify a user-defined namespace where your service runs.
   2. Specify endpoint ports to be scraped by Prometheus.
   3. Configure a selector to match your service based on its metadata labels.

      Note

      A ServiceMonitor resource in a user-defined namespace can only discover services in the same namespace. That is, the namespaceSelector field of the ServiceMonitor resource is always ignored.

3. Apply the configuration to the cluster:

   $ oc apply -f example-app-service-monitor.yaml

   It takes some time to deploy the ServiceMonitor resource.

4. Verify that the ServiceMonitor resource is running:

   $ oc -n <namespace> get servicemonitor

   Example output

   NAME                         AGE
   prometheus-example-monitor   81m

Example service endpoint authentication settings

You can configure authentication for service endpoints for user-defined project monitoring by using ServiceMonitor and PodMonitor custom resource definitions (CRDs).

The following samples show different authentication settings for a ServiceMonitor resource. Each sample shows how to configure a corresponding Secret object that contains authentication credentials and other relevant settings.

Sample YAML authentication with a bearer token

The following sample shows bearer token settings for a Secret object named example-bearer-auth in the ns1 namespace:

Example bearer token secret

apiVersion: v1
kind: Secret
metadata:
  name: example-bearer-auth
  namespace: ns1
stringData:
  token: <authentication_token> 

1. Specify an authentication token.

The following sample shows bearer token authentication settings for a ServiceMonitor CRD. The example uses a Secret object named example-bearer-auth:

Example bearer token authentication settings

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - authorization:
      credentials:
        key: token 
        name: example-bearer-auth 
    port: web
  selector:
    matchLabels:
      app: prometheus-example-app

1. The key that contains the authentication token in the specified Secret object.
2. The name of the Secret object that contains the authentication credentials.

Important

Do not use bearerTokenFile to configure bearer token. If you use the bearerTokenFile configuration, the ServiceMonitor resource is rejected.

Sample YAML for Basic authentication

The following sample shows Basic authentication settings for a Secret object named example-basic-auth in the ns1 namespace:

Example Basic authentication secret

apiVersion: v1
kind: Secret
metadata:
  name: example-basic-auth
  namespace: ns1
stringData:
  user: <basic_username> 
  password: <basic_password>  

1. Specify a username for authentication.
2. Specify a password for authentication.

The following sample shows Basic authentication settings for a ServiceMonitor CRD. The example uses a Secret object named example-basic-auth:

Example Basic authentication settings

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - basicAuth:
      username:
        key: user 
        name: example-basic-auth 
      password:
        key: password 
        name: example-basic-auth 
    port: web
  selector:
    matchLabels:
      app: prometheus-example-app

1. The key that contains the username in the specified Secret object.
2. The name of the Secret object that contains the Basic authentication.
3. The key that contains the password in the specified Secret object.

Sample YAML authentication with OAuth 2.0

The following sample shows OAuth 2.0 settings for a Secret object named example-oauth2 in the ns1 namespace:

Example OAuth 2.0 secret

apiVersion: v1
kind: Secret
metadata:
  name: example-oauth2
  namespace: ns1
stringData:
  id: <oauth2_id> 
  secret: <oauth2_secret> 

1. Specify an Oauth 2.0 ID.
2. Specify an Oauth 2.0 secret.

The following sample shows OAuth 2.0 authentication settings for a ServiceMonitor CRD. The example uses a Secret object named example-oauth2:

Example OAuth 2.0 authentication settings

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-example-monitor
  namespace: ns1
spec:
  endpoints:
  - oauth2:
      clientId:
        secret:
          key: id 
          name: example-oauth2 
      clientSecret:
        key: secret 
        name: example-oauth2 
      tokenUrl: https://example.com/oauth2/token 
    port: web
  selector:
    matchLabels:
      app: prometheus-example-app

1. The key that contains the OAuth 2.0 ID in the specified Secret object.
2. The name of the Secret object that contains the OAuth 2.0 credentials.
3. The key that contains the OAuth 2.0 secret in the specified Secret object.
4. The URL used to fetch a token with the specified clientId and clientSecret.

Additional resources

• Enabling monitoring for user-defined projects

• PodMonitor API

• ServiceMonitor API

• Scrape Prometheus metrics using TLS in ServiceMonitor configuration (Red Hat Customer Portal)