Kubernetes ingress discovery with Prometheus

October 13, 2017

To make it easier to monitor sites and services running on Kubernetes, we added ingress discovery to the existing Kubernetes service discovery mechanisms in Prometheus.

This makes it even easier to get availability and performance metrics about your Kubernetes ingresses with the Prometheus blackbox exporter as well as Latency.at.

The changes were released last week in Prometheus stable 1.8.0 as well as in the new Prometheus 2.0 release candidate.

See the official documentation example for how to use the ingress discovery with the blackbox-exporter. You can also use Latency.at to probe your ingresses automatically.

Example Configuration

This example will automatically probe all ingresses marked with the annotation prometheus.io/probe from Frankfurt:

- job_name: 'latency-at-fra1'
  bearer_token: your-auth-token
  metrics_path: /probe
  params:
    module: [http_2xx]
  kubernetes_sd_configs:
    - role: ingress
  relabel_configs:
    - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
      action: keep
      regex: true
    - source_labels:
      [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
      regex: (.+);(.+);(.+)
      replacement: ${1}://${2}${3}
      target_label: __param_target
    - source_labels: []
      target_label: __scheme__
      replacement: https
    - target_label: __address__
      replacement: fra1.do.mon.latency.at
    - source_labels: [__param_target]
      target_label: instance
    - action: labelmap
      regex: __meta_kubernetes_ingress_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_ingress_name]
      target_label: kubernetes_name

You can duplicate this job for each region you want to collect metrics from. Just replace fra1.do with the region you are interested in.

If you’re using your own blackbox exporter, just replace abc.do.mon.latency.at (and scheme if served via HTTP).