Namespace-scoped Kubernetes Service Discovery for Prometheus
Recently I was helping a colleague setting up Prometheus monitoring for an application.
This application must be installed into the Kubernetes cluster without
Therefore ClusterRoleBindings are not an option.
Instead we need to use a regular RoleBinding for attaching the the ServiceAccount to the Role.
If you are not familiar with the relation between roles, service accounts and role bindings, please refer to Kubernetes’ documentation on RBAC.
It boils down to adding the following YAML snippets to the Prometheus deployment (inspired by the Prometheus Helm Chart):
and adding the service account to the PodTemplate specification:
Please note that with a namespace-scoped service account Prometheus won’t be able to scrape the commonly used jobs
kubernetes-nodes-cadvisor because they are using the
So far so good. Unfortunately, it still didn’t work. We were using the following, simplified Prometheus configuration:
And Prometheus was giving us these errors:
$ kubectl logs deploy/prometheus-server -c prometheus-server | tail Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:mynamespace:prometheus-server" cannot list resource "pods" in API group "" at the cluster scope
The last two words of the log message contain the the key:
Prometheus is still trying to scrape at the cluster-level, even though the associated service account only allows it to scrape at the namespace level.
Unfortunately, Prometheus documentation on Kubernetes Service Discovery is not very clear about how to configure namespace-only scraping. This Github issue comment led me to the correct configuration option:
For each job definition under the
kubernetes_sd_configs key we need to specify for which namespace(s) it is activate.
If no namespaces are specified, Prometheus will try to scrape all namespaces — and if it is not allowed to do that, the entire process will fail.
The updated example configuration shown above looks like this:
After this configuration change Prometheus discovered all the namespace-local services.
If you want to achieve the same effect when using the Prometheus Helm Chart (v13+), you need to specify the following Helm values: