Using OpenShift OAuth for Grafana Authentication

A common frustration among system administrators is user management: when I spin up a new tool in my cluster, the last thing I want to do is manage a separate set of credentials, or worse, hand out a single admin password for something like Grafana. I just want my users to log in with their existing cluster accounts.

You can go the route of configuring Grafana’s generic OAuth2 integration, but I’ve found that to be a bit fiddly. You have to create an OAuth client manually, manage client secrets, and keep all the redirect URLs and discovery endpoints in sync.

A cleaner solution is using OpenShift’s built-in oauth-proxy. This proxy sits in front of the Grafana application, intercepts requests, redirects users to the OpenShift login page, and then, upon successful login, passes their identity to Grafana.

Diagram visualizing request flow

This post is a hands-on guide to setting this up. It’s the method I now use for a variety of internal apps that need protection and authentication. The general setup is not specific to Grafana (in fact I have written about authentication proxies before) and can be applied to any application that allows passing user identity via HTTP headers.

#  Grafana Operator

Let’s start by installing the Grafana Operator from OperatorHub. This is the modern, recommended method for running Grafana instances on OpenShift. It’s especially useful if you’re planning to have more than one Grafana instance on the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: grafana-operator
  namespace: openshift-operators
spec:
  channel: v5
  installPlanApproval: Automatic
  name: grafana-operator
  source: community-operators
  sourceNamespace: openshift-marketplace

Note that the openshift-operators namespace comes with an OperatorGroup out-of-the-box, hence we don’t need to create it ourselves.

#  Grafana instance

Next, we can proceed with creating a new Grafana instance. A minimal example would look like the following snippet. We’ll gradually iterate on it in this section, at the end of the section you’ll find a full example that you can simply copy&paste.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  config:
    log:
      mode: "console"
    security:
      admin_user: root
      admin_password: secret

#  Configure Grafana to trust the proxy

As described in the introduction, instead of relying on its own authentication mechanism, Grafana should rely on the information provided by the authentication proxy. This is achieved as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
spec:
  ...
  config:
    auth:
      # Grafana should not offer authentication itself
      disable_login_form: "True"
      disable_signout_menu: "True"
    auth.proxy:
      enabled: "True"
      # automatically create accounts for "new" users
      auto_sign_up: 'True'
      enable_login_token: 'False'
      # this HTTP header is sent by the oauth-proxy to the Grafana server
      header_name: "X-Forwarded-User"
      header_property: username
      # additional information to identify the user
      headers: "Name:X-Forwarded-User Email:X-Forwarded-Email"

#  Injecting sidecar

Next, we use the spec.deployment field of the custom resource to patch the Deployment the operator creates for us. In this scenario we will be running the oauth-proxy as a sidecar, i.e. we will add an extra container to the existing Grafana pod. This sidecar container will handle the necessary authentication steps: if a request is not authenticated, the user will be redirected to OpenShift’s built-in OAuth endpoint. After successful login, an HTTP cookie is set. Only if a request is authenticated (i.e. it has the relevant cookie), it is forwarded to the Grafana server. To let Grafana know about the identity of the user (username, display name etc.), the oauth-proxy adds several HTTP headers to the request: most importantly X-Forwarded-User and X-Forwarded-Email.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
spec:
  ...
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: oauth-proxy
              image: image-registry.openshift-image-registry.svc:5000/openshift/oauth-proxy:v4.4
              args:
                - '--provider=openshift'
                - '--http-address=:9090'
                - '--https-address='
                - '--openshift-service-account=grafana-instance-sa'
                - '--upstream=http://localhost:3000'
                - '--cookie-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token'
              ports:
                - containerPort: 9090
                  name: oauth-proxy

Note that in this snippet we don’t use an external image for the oauth-proxy (such as quay.io/openshift/origin-oauth-proxy:latest), instead we reference an ImageStream in OpenShift’s image registry. This image is also used by other operators in the cluster and is kept up-to-date by regular OpenShift cluster upgrades (despite the misleading v4.4 tag).

With --http-address=:9090 and --https-address=, we enable only HTTP traffic (more on that later).

By specifying the name of the OpenShift service account, the oauth-proxy knows which credentials it should use to communicate with the OpenShift OAuth server.

#  Routing traffic to the proxy

To get traffic to the port of the oauth-proxy (9090) instead of the regular Grafana port (3000), we instruct the operator to adjust the resources it deploys:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
spec:
  ...
  serviceAccount:
    metadata:
      annotations:
        serviceaccounts.openshift.io/oauth-redirectreference.grafana: '{"kind":"OAuthRedirectReference","apiVersion":"v1","reference":{"kind":"Route","name":"grafana-instance-route"}}'

  service:
    spec:
      ports:
        # additional port definition for oauth-proxy
        - name: oauth-proxy
          port: 9090
          protocol: TCP
          targetPort: oauth-proxy

  route:
    spec:
      # optionally could set the hostname here
      # host: my-grafana.example.com
      port:
        # send traffic to oauth-proxy instead of Grafana directly
        targetPort: oauth-proxy
      tls:
        termination: edge
      to:
        kind: Service
        name: grafana-instance-service
        weight: 100
      wildcardPolicy: None

The annotation on the ServiceAccount allows OpenShift to dynamically retrieve the hostname of the associated Route. For us this means that we don’t have to manually set RedirectURIs for the OAuth configuration, which is quite a handy feature.

Then we add an additional port for oauth-proxy to the Service definition and modify the Route so the ingress controller sends traffic to the oauth-proxy instead of directly to Grafana.

You can view and download the complete grafana-authenticated.yaml resource file and then deploy it with:

1
oc apply -f grafana-authenticated.yaml

And that’s it! At this point we have a Grafana instance running with an oauth-proxy sidecar that handles authenticating all incoming requests:

1
2
3
4
5
6
7
8
9
$ oc get grafana
NAME               VERSION   STAGE      STAGE STATUS   AGE
grafana-instance   10.2.6    complete   success        38m
$ oc get deploy
NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
grafana-instance-deployment   1/1     1            1           38m
$ oc get pod
NAME                                           READY   STATUS    RESTARTS   AGE
grafana-instance-deployment-7d5b554996-dmkds   2/2     Running   0          9m51s

#  Securing network traffic

While we have a functioning and authenticated Grafana instance now, there are still two improvements we can apply: network isolation and traffic encryption.

The entire point of the proxy is to be the only gateway. If another Pod can access, or another Route or Ingress service points to Grafana’s port directly, an attacker could bypass the OpenShift authentication entirely.

As a “defense-in-depth” step, we can create a NetworkPolicy (“firewall”) that ensures other pods in the OpenShift cluster can only reach the port of the oauth-proxy (9090), but not Grafana directly (3000). The only other pod in the OpenShift cluster that has to reach our service is the Ingress controller. In addition, the Grafana operator must be able to reach the Grafana instance directly (port 3000) to apply its configuration.

The following NetworkPolicy implements these restrictions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: grafana
spec:
  podSelector:
    matchLabels:
      # Use 'oc get pod -n <your-ns> --show-labels' to find the right label
      app.kubernetes.io/name: grafana-instance

  policyTypes:
    - Ingress
  ingress:
    # Allow traffic from the OpenShift Ingress/Router
    - from:
        - namespaceSelector:
            matchLabels:
              network.openshift.io/policy-group: ingress
      ports:
        # HTTP port
        - protocol: TCP
          port: 9090
        # HTTPS port (see below)
        - protocol: TCP
          port: 9091

    # Allow traffic from Grafana operator (for applying configuration via API)
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: openshift-operators
          podSelector:
            matchLabels:
              app.kubernetes.io/name: grafana-operator
      ports:
        - protocol: TCP
          port: 3000

The final step is to add encryption between the ingress controller and the oauth-proxy. This can be achieved with the following modification to the Grafana custom resource (modified sections are below comments). This setup heavily relies on OpenShift’s Service CA, a component that is an internal (private) certificate authority which provides on-demand certificates to all services in the cluster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
spec:
  ...
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: oauth-proxy
              image: image-registry.openshift-image-registry.svc:5000/openshift/oauth-proxy:v4.4
              args:
                - '--provider=openshift'
                # Expose HTTPS listener instead of HTTP listener
                - '--http-address='
                - '--https-address=:9091'
                - '--openshift-service-account=grafana-instance-sa'
                - '--upstream=http://localhost:3000' # Points to Grafana in the same pod
                - '--cookie-secret-file=/var/run/secrets/kubernetes.io/serviceaccount/token'
                # Specify location for certificates and private keys
                - '-tls-cert=/etc/tls/private/tls.crt'
                - '-tls-key=/etc/tls/private/tls.key'
                - '-openshift-ca=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'

              # New port is 9091
              ports:
                - containerPort: 9091
                  name: oauth-proxy

              # Mount secret in this container
              volumeMounts:
                - mountPath: /etc/tls/private
                  name: oauth-proxy-tls

          # Additional volume for the Pod
          volumes:
          - name: oauth-proxy-tls
            secret:
              secretName: oauth-proxy-tls

  service:
    metadata:
      annotations:
        # Tell OpenShift to generate a certificate for this Service and store it in the Secret
        service.beta.openshift.io/serving-cert-secret-name: oauth-proxy-tls
    spec:
      ports:
        - name: oauth-proxy
          port: 9090
          protocol: TCP
          targetPort: oauth-proxy

  # Point Route to oauth-proxy instead of directly to Grafana
  route:
    spec:
      port:
        targetPort: oauth-proxy
      tls:
        # Tell OpenShift ingress controller to encrypt connection to this backend
        termination: reencrypt
      to:
        kind: Service
        name: grafana-instance-service
        weight: 100
      wildcardPolicy: None

That’s quite a lot to take in all at once. Let’s break down what is happening here:

  • a Service with the serving-cert-secret-name annotation is created;
  • this triggers the OpenShift Service CA to generate a certificate and private key (that is trusted by the cluster-internal private certificate authority) and store it in the Secret resource named oauth-proxy-tls;
  • this Secret gets mounted in the container for the oauth-proxy;
  • the oauth-proxy uses the certificate and private key for its web server when serving client requests;
  • when a request reaches the OpenShift ingress controller, the ingress controller connects to the oauth-proxy;
  • the ingress controller accepts the certificate presented by the oauth-proxy because it is signed by the Service CA, which is trusted by the ingress controller.

Note that it is not necessary to encrypt the traffic between the oauth-proxy and the Grafana server since both components are part of the same Pod (network namespace) and communicate only via the localhost interface.

#  Conclusion

And that’s it! We now have a fully functional Grafana instance that’s seamlessly integrated with OpenShift’s built-in authentication.

The main benefits are the single source of truth for user management (the cluster’s identity provider) - no more separate user lists or shared passwords, secure communication by default (encryption and network isolation are enforced wherever necessary and users can’t bypass the authentication), and that this is a reusable pattern that can be applied to many applications (wikis, admin UIs, dashboards etc.)

This setup is incredibly portable and easy to manage with GitOps, since we don’t have to juggle any client secrets or out-of-band configurations.

Happy dashboarding!

#  References