Using Kubernetes context in policies (importing data.kubernetes resources to go through existing cluster resources)

[aleksandar-ruskov]

I have been testing out building OPA policies for Kubernetes resources. I have previously seen and implemented policies which import data.kubernetes.namespaces as well as data.kubernetes.ingresses to get the existing namespaces and ingresses when I am checking for some duplications and stuff. However, when I tried fetching other existing k8s resource types such as Pods, PodDisruptionBudgets, ConfigMaps that doesn't seem to work.

data.kubernetes.ingresses[SOME_NAMESPACE] - gives me all ingresses in the specified namespace
data.kubernetes.pods[SOME_NAMESPACE] - doesn't seem to fetch the pods in the namespace.

So is there some other way I need to do that, or is it an issue with OPA?

[anderseknert]

OPA will will work with whatever data it has been provided — it has no idea about Kubernetes, pods or namespaces :)
I'm guessing you are using kube-mgmt to provide OPA Kubernetes data? I would look specifically at how that is configured if so.

[aleksandar-ruskov]

Yes, so it is using Kubernetes-mgmt, I will take a look at that but I just followed the configurations published in the Open Policy Agent documentation (followed https://www.openpolicyagent.org/docs/v0.48.0/kubernetes-tutorial/ when I was setting it up on my cluster).

Seems like kube-mgmt should be correctly set in the admission-controller.yaml, where it has a viewer clusterRole which should be allowing it to view all cluster resources.

[anderseknert]

The tutorial specifically demonstrates policy for namespaces and ingresses. Check the configuration of kube-mgmt in that guide and you'll see:

        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:2.0.1
          args:
            - "--replicate-cluster=v1/namespaces"
            - "--replicate=networking.k8s.io/v1/ingresses"

Update args if you want to replicate other types of resources.

[aleksandar-ruskov]

Yes, thank you I just saw that as well, I had completely missed that part of the admission controller.

[aleksandar-ruskov]

Even though adding the required args in Kubernetes-mgmt for pods and deployments worked, I faced another problem with other resources such as Roles, Secrets, etc. I followed the kube-mgmt readme when I was adding the K8S resources to be replicated in the admission controller (https://github.com/open-policy-agent/kube-mgmt/tree/0753a3f46f95a22062caf31e2c69b89d298e736e#readme).

Since Roles and Secrets did not seem to be replicated successfully and I couldn't access them through data.kubernetes.secrets or data.kubernetes.roles, I made an experiment where I try to replicate all K8S api resources and added all to the admission controller:

- name: kube-mgmt
          image: openpolicyagent/kube-mgmt:2.0.1
          args:
            - "--replicate-cluster=v1/componentstatuses"
            - "--replicate-cluster=v1/namespaces"
            - "--replicate-cluster=v1/nodes"
            - "--replicate-cluster=v1/persistentvolumes"
            - "--replicate-cluster=apiregistration.k8s.io/v1/apiservices"
            - "--replicate-cluster=authentication.k8s.io/v1/tokenreviews"
            - "--replicate-cluster=authorization.k8s.io/v1/selfsubjectaccessreviews"
            - "--replicate-cluster=authorization.k8s.io/v1/selfsubjectrulesreviews"
            - "--replicate-cluster=authorization.k8s.io/v1/subjectaccessreviews"
            - "--replicate-cluster=certificates.k8s.io/v1/certificatesigningrequests"
            - "--replicate-cluster=flowcontrol.apiserver.k8s.io/v1beta2/flowschemas"
            - "--replicate-cluster=flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations"
            - "--replicate-cluster=networking.k8s.io/v1/ingressclasses"
            - "--replicate-cluster=node.k8s.io/v1/runtimeclasses"
            - "--replicate-cluster=rbac.authorization.k8s.io/v1/clusterrolebindings"
            - "--replicate-cluster=rbac.authorization.k8s.io/v1/clusterroles"
            - "--replicate-cluster=scheduling.k8s.io/v1/priorityclasses"
            - "--replicate-cluster=storage.k8s.io/v1/csidrivers"
            - "--replicate-cluster=storage.k8s.io/v1/csinodes"
            - "--replicate-cluster=storage.k8s.io/v1/storageclasses"
            - "--replicate-cluster=storage.k8s.io/v1/volumeattachments"
            - "--replicate=v1/bindings"
            - "--replicate=v1/configmaps"
            - "--replicate=v1/endpoints"
            - "--replicate=v1/events"
            - "--replicate=v1/limitranges"
            - "--replicate=v1/persistentvolumeclaims"
            - "--replicate=v1/pods"
            - "--replicate=v1/podtemplates"
            - "--replicate=v1/replicationcontrollers"
            - "--replicate=v1/resourcequotas"
            - "--replicate=v1/secrets"
            - "--replicate=v1/serviceaccounts"
            - "--replicate=v1/services"
            - "--replicate=apps/v1/controllerrevisions"
            - "--replicate=apps/v1/daemonsets"
            - "--replicate=apps/v1/deployments"
            - "--replicate=apps/v1/replicasets"
            - "--replicate=apps/v1/statefulsets"
            - "--replicate=authorization.k8s.io/v1/localsubjectaccessreviews"
            - "--replicate=autoscaling/v2/horizontalpodautoscalers"
            - "--replicate=batch/v1/cronjobs"
            - "--replicate=batch/v1/jobs"
            - "--replicate=coordination.k8s.io/v1/leases"
            - "--replicate=discovery.k8s.io/v1/endpointslices"
            - "--replicate=events.k8s.io/v1/events"
            - "--replicate=networking.k8s.io/v1/ingresses"
            - "--replicate=networking.k8s.io/v1/networkpolicies"
            - "--replicate=policy/v1/poddisruptionbudgets"
            - "--replicate=rbac.authorization.k8s.io/v1/rolebindings"
            - "--replicate=rbac.authorization.k8s.io/v1/roles"
            - "--replicate=storage.k8s.io/v1/csistoragecapacities"

Then I tried to access each of them in a policy, where for each resource I follow the pattern:

  import data.kubernetes.<resource>
  deny[message]{
      not <resource>
      message := "<resource> is not correctly imported!"
  }

As a result the following resources triggered the policy (these seem to fail to import):

Cluster-level resources (overall only namespaces seem to be imported correctly):

• apiservices
• certificatesigningrequests
• clusterrolebindings
• clusterroles
• componentstatuses
• csidrivers
• csinodes
• flowschemas
• ingressclasses
• localsubjectaccessreviews
• nodes
• persistentvolumes
• priorityclasses
• prioritylevelconfigurations
• runtimeclasses
• selfsubjectaccessreviews
• selfsubjectrulesreviews
• storageclasses
• subjectaccessreviews
• token reviews
• volumeattachments

Namespace-level resources:

• bindings
• csistoragecapacities
• leases
• localsubjectaccessreviews
• podtemplates
• rolebindings
• roles
• secrets

So it is visible that most namespaces resources get imported although there are some which fail, but also almost all cluster-level resources are not imported into OPA (by kube-mgmt). I can't seem to find any pattern between the resources which fail to import. So I was wondering what could be causing the issue?

[ritazh]

@aleksandar-ruskov I’m curious if you have evaluated OPA Gatekeeper and why it did not address your scenario.