Kubernetes ingress discovery with Prometheus

August 29, 2017

We contributed ingress discovery to the existing Kubernetes service discovery mechanisms in Prometheus.

This makes it even easier to monitor your Kubernetes ingresses with the Prometheus blackbox exporter as well as Latency.at.

See the official documentation example for how to use the ingress discovery with the blackbox-exporter. You can also use Latency.at to probe your ingresses automatically.

This example will automatically probe all ingresses marked with the annotation prometheus.io/probe from Frankfurt:

- job_name: 'latency-at-fra1'
  bearer_token: your-auth-token
  metrics_path: /probe
  params:
    module: [http_2xx]
  kubernetes_sd_configs:
    - role: ingress
  relabel_configs:
    - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
      action: keep
      regex: true
    - source_labels:
      [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
      regex: (.+);(.+);(.+)
      replacement: ${1}://${2}${3}
      target_label: __param_target
    - source_labels: []
      target_label: __scheme__
      replacement: https
    - target_label: __address__
      replacement: fra1.do.mon.latency.at
    - source_labels: [__param_target]
      target_label: instance
    - action: labelmap
      regex: __meta_kubernetes_ingress_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_ingress_name]
      target_label: kubernetes_name

You can duplicate this job for each region you want to collect metrics from. Just replace fra1.do with the region you are interested in.