Skip to content

Reference: PodMonitor CRD

A PodMonitor describes how to scrape Prometheus metrics directly from pods. It selects pods by label and lists the named container ports to scrape, along with the path, interval, authentication and relabeling rules for each port. Use a PodMonitor when your pods are not behind a Service, or when you want to scrape a port that the service does not expose. If your application already has a Service, a ServiceMonitor is usually simpler.

  • API version: monitoring.coreos.com/v1
  • Kind: PodMonitor
  • Scope: Namespaced
  • Short name: pmon

This page documents Prometheus Operator v0.95.1.

Example

The following example scrapes the /metrics endpoint on the http container port of all pods labeled app: your-application every 30 seconds. The port name matches the container port in the Deployment from the getting started guide.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: your-application
  namespace: your-application-namespace
spec:
  selector:
    matchLabels:
      app: your-application
  podTargetLabels:
  - app
  podMetricsEndpoints:
  - port: http
    path: /metrics
    interval: 30s
    scrapeTimeout: 10s

Specification (spec)

Field Type Description
selector LabelSelector Required. Selects the Pod objects to scrape. See spec.selector.
podMetricsEndpoints []object Optional. The pod ports to scrape and how to scrape them. See spec.podMetricsEndpoints[].
namespaceSelector object Optional. The namespaces in which to discover pods. Defaults to the namespace of the PodMonitor. See spec.namespaceSelector.
jobLabel string Optional. The name of a label on the Pod whose value becomes the job label of all metrics. If you leave it empty, job defaults to <namespace>/<name> of the PodMonitor.
podTargetLabels []string Optional. Labels to copy from the Pod onto the ingested metrics.
attachMetadata object Optional. Additional metadata to attach to discovered targets. See spec.attachMetadata.
sampleLimit integer Optional. Per-scrape limit on the number of samples accepted. Minimum 0.
targetLimit integer Optional. Limit on the number of scraped targets accepted. Minimum 0.
labelLimit integer Optional. Per-scrape limit on the number of labels accepted for a sample. Minimum 0.
labelNameLengthLimit integer Optional. Per-scrape limit on the length of label names accepted for a sample. Minimum 0.
labelValueLengthLimit integer Optional. Per-scrape limit on the length of label values accepted for a sample. Minimum 0.
bodySizeLimit string Optional. Job-level limit on the uncompressed size of a scrape response, for example 10MiB. 0 means no limit.
keepDroppedTargets integer Optional. Per-scrape limit on the number of targets dropped by relabeling that are kept in memory. 0 means no limit.
scrapeProtocols []string Optional. Protocols to negotiate during a scrape, in order of preference. Allowed values: PrometheusProto, OpenMetricsText0.0.1, OpenMetricsText1.0.0, PrometheusText0.0.4, PrometheusText1.0.0. If unset, the scraper uses its default.
fallbackScrapeProtocol string Optional. Protocol to use if a scrape returns a blank, unparseable or otherwise invalid Content-Type. Allowed values are the same as for scrapeProtocols.
scrapeNativeHistograms boolean Optional. Enables scraping of native histograms.
scrapeClassicHistograms boolean Optional. Scrapes the classic version of a histogram that is also exposed as a native histogram.
convertClassicHistogramsToNHCB boolean Optional. Converts all scraped classic histograms into native histograms with custom buckets.
nativeHistogramBucketLimit integer Optional. Maximum number of buckets in a native histogram. Buckets are merged to stay within the limit. Minimum 0.
nativeHistogramMinBucketFactor int-or-string Optional. Minimum growth factor from one native histogram bucket to the next. Buckets are merged to reach the factor.
extraScrapeMetrics boolean Optional. Stores the additional series scrape_timeout_seconds, scrape_sample_limit and scrape_body_size_bytes for each scrape, so you can see how close targets are to their limits.
selectorMechanism string Optional. How pods are selected. Allowed values: RelabelConfig (default, filter discovered targets with relabeling) and RoleSelector (filter at discovery, which can be more efficient in large clusters).
scrapeClass string Optional. The name of a scrape class to apply. Scrape classes are defined by the operator of the scraping instance.

spec.selector

A standard Kubernetes LabelSelector with matchLabels and matchExpressions. The selector matches labels on the Pod objects.

spec.namespaceSelector

Field Type Description
any boolean Optional. If true, discovers pods in all namespaces.
matchNames []string Optional. The list of namespace names in which to discover pods.

spec.attachMetadata

Field Type Description
node boolean Optional. If true, attaches node metadata to discovered targets as __meta_kubernetes_node_* labels. These labels are not added to metrics automatically. Copy them with relabeling. Requires list and watch permissions on Node objects for the scraper.

spec.podMetricsEndpoints[]

Each entry describes one port to scrape on the selected pods.

Field Type Description
port string Optional. The name of the container port to scrape. If no pod exposes a port with this name, no targets are discovered. Takes precedence over portNumber.
portNumber integer Optional. The number of the container port to scrape. The pod must declare the port in its spec, or the target is dropped. Minimum 1.
path string Optional. The HTTP path to scrape. Defaults to /metrics.
scheme string Optional. The HTTP scheme. Allowed values: http, https (upper case is also accepted).
params map[string][]string Optional. HTTP URL query parameters to add to the scrape request.
interval string Optional. Scrape interval as a duration, for example 30s. Defaults to the global scrape interval.
scrapeTimeout string Optional. Timeout after which the scrape fails, for example 10s. Defaults to the global scrape timeout, or to interval if that is shorter. Must not be greater than interval, or the resource is rejected.
honorLabels boolean Optional. If true, keeps the metric's own labels when they collide with target labels.
honorTimestamps boolean Optional. If true, keeps timestamps exposed by the target.
trackTimestampsStaleness boolean Optional. Tracks staleness of metrics that carry an explicit timestamp. Has no effect if honorTimestamps is false.
filterRunning boolean Optional. Drops pods that are not running (for example Failed or Succeeded) during discovery. Enabled when unset.
followRedirects boolean Optional. Follows HTTP 3xx redirects.
enableHttp2 boolean Optional. Set to false to disable HTTP/2.
authorization object Optional. Credentials for the Authorization header. See spec.podMetricsEndpoints[].authorization. Mutually exclusive with basicAuth and oauth2.
basicAuth object Optional. Basic authentication credentials. See spec.podMetricsEndpoints[].basicAuth. Mutually exclusive with authorization and oauth2.
oauth2 object Optional. OAuth2 client credentials. See spec.podMetricsEndpoints[].oauth2. Mutually exclusive with authorization and basicAuth.
tlsConfig object Optional. TLS settings for the scrape. See spec.podMetricsEndpoints[].tlsConfig.
relabelings []object Optional. Relabeling rules applied to target metadata before the scrape. See relabeling. The original job name is available in the __tmp_prometheus_job_name label.
metricRelabelings []object Optional. Relabeling rules applied to samples before ingestion. Use these to drop or rename series. See relabeling.
proxyUrl string Optional. HTTP proxy to use. Must match ^(http\|https\|socks5)://.+$.
noProxy string Optional. Comma-separated IPs, CIDRs and domain names that bypass the proxy.
proxyFromEnvironment boolean Optional. Uses the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables.
proxyConnectHeader map[string][]SecretKeySelector Optional. Headers to send to the proxy during CONNECT requests. Each value references keys in Secrets.

spec.podMetricsEndpoints[].authorization

Field Type Description
type string Optional. The authentication type, case-insensitive. Defaults to Bearer. Basic is not supported, use basicAuth instead.
credentials SecretKeySelector Optional. A key in a Secret in the same namespace that holds the credentials. See Secret and ConfigMap references.

spec.podMetricsEndpoints[].basicAuth

Field Type Description
username SecretKeySelector Optional. A key in a Secret that holds the username.
password SecretKeySelector Optional. A key in a Secret that holds the password.

spec.podMetricsEndpoints[].oauth2

Field Type Description
tokenUrl string Required. The URL to fetch the token from. Must start with http:// or https://.
clientId SecretOrConfigMap Required. A key in a Secret or ConfigMap that holds the client ID.
clientSecret SecretKeySelector Required. A key in a Secret that holds the client secret.
scopes []string Optional. OAuth2 scopes for the token request.
endpointParams map[string]string Optional. HTTP parameters to append to the token URL.
tlsConfig object Optional. TLS settings for the connection to the token server. Same fields as spec.podMetricsEndpoints[].tlsConfig.
proxyUrl string Optional. HTTP proxy for the token request.
noProxy string Optional. Comma-separated IPs, CIDRs and domain names that bypass the proxy.
proxyFromEnvironment boolean Optional. Uses the proxy environment variables.
proxyConnectHeader map[string][]SecretKeySelector Optional. Headers to send to the proxy during CONNECT requests.

spec.podMetricsEndpoints[].tlsConfig

Field Type Description
ca SecretOrConfigMap Optional. The certificate authority used to verify the target's certificate.
cert SecretOrConfigMap Optional. The client certificate to present for client authentication.
keySecret SecretKeySelector Optional. A key in a Secret that holds the client private key.
serverName string Optional. The hostname used to verify the target's certificate.
insecureSkipVerify boolean Optional. Disables verification of the target's certificate.
minVersion string Optional. Minimum TLS version. Allowed values: TLS10, TLS11, TLS12, TLS13.
maxVersion string Optional. Maximum TLS version. Allowed values: TLS10, TLS11, TLS12, TLS13.

spec.podMetricsEndpoints[].relabelings[] and spec.podMetricsEndpoints[].metricRelabelings[]

Both lists use the Prometheus relabel_config format. relabelings run on target metadata before the scrape, and metricRelabelings run on each sample before ingestion.

Field Type Description
sourceLabels []string Optional. Labels whose values are concatenated with separator and matched against regex.
separator string Optional. The string placed between concatenated source label values. Prometheus defaults to ;.
regex string Optional. The regular expression matched against the concatenated value. Prometheus defaults to (.*).
targetLabel string Optional. The label to write the result to. Required for the replace, hashmod, lowercase, uppercase, keepequal and dropequal actions. Capture groups are available.
replacement string Optional. The value written by a replace action if regex matches. Capture groups are available. Prometheus defaults to $1.
modulus integer Optional. The modulus for the hashmod action. Minimum 0.
action string Optional. The action to perform. Allowed values: replace, keep, drop, hashmod, labelmap, labeldrop, labelkeep, lowercase, uppercase, keepequal, dropequal. Capitalized forms such as Replace are also accepted. Defaults to replace.

Secret and ConfigMap References

Credential fields use two reference types. Both must point to objects in the same namespace as the PodMonitor.

  • SecretKeySelector has the fields name (the Secret name), key (required, the key in the Secret) and optional (whether the Secret or key may be missing). It is the same type as secretKeyRef in the Kubernetes Pod API.
  • SecretOrConfigMap has two optional fields, secret (a SecretKeySelector) and configMap (a ConfigMapKeySelector with the same name, key and optional fields). Set one of them.

Deprecated Fields

The following fields are deprecated upstream and not documented above:

  • spec.podMetricsEndpoints[].targetPort. Use port or portNumber instead.
  • spec.podMetricsEndpoints[].bearerTokenSecret. Use authorization instead.

Status (status)

The Prometheus Operator writes status only when it runs with the StatusForConfigurationResources feature gate enabled, which is off by default upstream. The field can be empty.

Field Type Description
bindings[].resource string The type of workload that selected this resource, for example prometheuses or prometheusagents.
bindings[].name string The name of the workload that selected this resource.
bindings[].namespace string The namespace of that workload.
bindings[].conditions[].type string The condition type. Only Accepted is supported.
bindings[].conditions[].status string The condition status, for example True or False.
bindings[].conditions[].reason string A short reason for the last transition.
bindings[].conditions[].message string A human-readable explanation, for example why the operator rejected the resource.
bindings[].conditions[].observedGeneration integer The metadata.generation the condition is based on. If it is lower than the current generation, the condition is out of date.

Platform Notes

  • You can create PodMonitor resources in your own namespaces.
  • As an alternative to a PodMonitor, you can annotate your pods with prometheus.io/scrape, prometheus.io/path and prometheus.io/port, as described in the getting started guide.

Further Reading