Prometheus Federation match params do not work

Question

I have been trying to achive federation in my Prometheus setup. While doing this, I want to exclude some metrics to be scraped by my scraper Prometheus.

Here is my federation config:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'xxxxxxxx'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job!="kubernetes-nodes"}'
    static_configs:
      - targets:
        - 'my-metrics-source'

As it can be seen from the config, I want to exclude any metric that has kubernetes-nodes job label, and retrieve the rest of the metrics. However, when I deploy my config, no metric is scraped.

Is it a bug in Prometheus or I simply misunderstood how the match params work?

score 7 · Accepted Answer · answered Feb 13 '18 at 09:21

If you really need to do this you need a primary vector selector which includes results.

Otherwise you'll get the error vector selector must contain at least one non-empty matcher.

So for example with these matchers you'll get what you are trying to achieve:

curl -G --data-urlencode 'match[]={job=~".+", job!="kubernetes-nodes"}' http://your-url.example.com/federate

score 3 · Answer 2 · answered Feb 13 '18 at 09:16

As a safety measure to avoid you accidentally writing an instant vector that returns all the time series in your Prometheus, selectors must contain at least one matcher that does not match the empty string. Your selector has no such matcher (job!="kubernetes-nodes" matches an empty job label), so this is giving you an error.

You could add a selector such as __name__=~".+" however at a higher level this is an abuse of federation as it is not meant for pulling entire Prometheus servers. See https://www.robustperception.io/federation-what-is-it-good-for/

Prometheus Federation match params do not work

2 Answers2