I spent a weekend debugging a “permission denied” error in a custom controller that only appeared when the pod migrated to a different node. The fix wasn’t in the code, but in a ClusterRoleBinding that I’d lazily set to cluster-admin six months prior, which had since been partially overridden by a namespace-level policy I forgot I implemented. It was a reminder that “just give it admin” is a technical debt bomb that eventually explodes.

If you’re running a small homelab, cluster-admin is tempting. It’s the path of least resistance. But once you start deploying AI agents that can execute code or industrial IoT pipelines that touch physical hardware, a compromised pod with cluster-wide permissions is a catastrophe. You need a way to give your apps exactly what they need to function and nothing more.

What I tried first

My first instinct with RBAC was to use ClusterRoles for everything. I figured if I defined the permissions once at the cluster level, I wouldn’t have to keep rewriting the same Role YAML for every new namespace I created. I’d create a ClusterRole for “pod-reader” and then bind it to the ServiceAccount in each namespace.

This worked until I realized I was creating a massive auditing nightmare. I had no easy way to see which specific pods in which namespaces had these permissions without grepping through dozens of ClusterRoleBindings. I’d run kubectl get clusterrolebindings and see a wall of text that told me nothing about the actual blast radius of a single compromised namespace.

Then I tried the opposite: creating hyper-specific Roles for every single microservice. I ended up with a YAML sprawl that was impossible to maintain. I was manually updating 15 different Role objects just to add a patch permission to a deployment. I was essentially treating RBAC like a manual checklist rather than a system. I spent more time fighting YAML indentation and naming conventions than actually building the services.

The gap was in the middle. I needed a pattern that was scalable but strictly scoped.

The actual solution

The goal is to move from “it works” to “it’s secure.” This requires a three-tier approach: a dedicated ServiceAccount, a scoped Role (or ClusterRole), and a RoleBinding that bridges them. I’m basing this on a K8s 1.31 environment, but the logic holds for anything 1.24+.

1. The Minimalist Service Account

Stop using the default service account. If you don’t specify one, Kubernetes assigns the default SA in that namespace. If you’ve accidentally granted permissions to that default account, every single pod in that namespace now has those permissions.

# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-runtime-sa
  namespace: ai-workloads
automountServiceAccountToken: true # Only true if the pod actually needs to talk to the K8s API

I set automountServiceAccountToken: false by default for any pod that doesn’t need to query the API server. This prevents the token from being injected into the pod’s filesystem at /var/run/secrets/kubernetes.io/serviceaccount, removing one more attack vector. If a hacker gets a shell in your pod, the first thing they do is check for that token to see if they can pivot to the API.

2. Scoping the Role

Instead of cluster-admin, I define the exact verbs and resources. For an AI agent that needs to monitor its own pods but not touch secrets, the Role looks like this:

# role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-monitor-role
  namespace: ai-workloads
rules:
- apiGroups: [""] # The core API group
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list"]

If the agent needs to operate across multiple namespaces but still maintain limited permissions, I use a ClusterRole but bind it with a RoleBinding (not a ClusterRoleBinding). This is a key distinction: a RoleBinding to a ClusterRole grants the permissions of that role only within the namespace of the binding. This allows me to define a “standard-viewer” ClusterRole once, but apply it to different namespaces with different identities.

3. The Binding

This is where we connect the identity (SA) to the permissions (Role).

# rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-monitor-binding
  namespace: ai-workloads
subjects:
- kind: ServiceAccount
  name: agent-runtime-sa
  namespace: ai-workloads
roleRef:
  kind: Role
  name: agent-monitor-role
  apiGroup: rbac.authorization.k8s.io

Handling the “Namespace-Aware” Edge Case

Sometimes you have a service that needs to look at resources in other namespaces, but you still don’t want to give it full cluster access. This is where resourceNames comes into play. If you know exactly which resources a service needs to touch, you can hardcode them into the ClusterRole.

# clusterrole-scoped.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: specific-resource-reader
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]
  resourceNames: ["global-config", "shared-settings"]

This is significantly safer than granting get on all configmaps across the whole cluster. It means even if the service account is leaked, the attacker can only read those two specific config maps.

Troubleshooting RBAC Failures

The most frustrating part of RBAC is the error messages. They are intentionally vague for security reasons. You’ll usually see something like:

Error from server (Forbidden): pods "my-pod" is forbidden: User "system:serviceaccount:ai-workloads:agent-runtime-sa" cannot get resource "pods" in namespace "ai-workloads"

When I hit this, I stop guessing and use kubectl auth can-i. This is the only way to stay sane. Instead of deploying a pod and waiting for it to crash, I test the permission directly:

# Test if the SA can list pods in its own namespace
kubectl auth can-i list pods --as=system:serviceaccount:ai-workloads:agent-runtime-sa -n ai-workloads
# Output: yes

# Test if the SA can delete pods (which it shouldn't be able to)
kubectl auth can-i delete pods --as=system:serviceaccount:ai-workloads:agent-runtime-sa -n ai-workloads
# Output: no

If you’re seeing a Forbidden error but can-i says yes, you’re likely dealing with an Admission Controller (like Kyverno) or a SecurityContext issue. I’ve seen this happen when fsGroup settings in the pod spec conflict with the underlying storage permissions on Longhorn, which looks like an RBAC error in the logs but is actually a filesystem permission issue.

Scaling with Policy-as-Code

Manually writing these for every service is tedious. I’ve started using Kyverno to automate the enforcement of these patterns. If a Job is created without a specific ServiceAccount, or if it’s using the default account, Kyverno can either block it or automatically generate the required RBAC.

I implemented a policy that ensures all batch/jobs are linked to a scoped role, which is particularly useful for the ephemeral nature of AI training jobs or data processing tasks. This prevents the “lazy developer” syndrome where someone deploys a one-off job with cluster-admin just to make it work.

# kyverno-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-job-rbac
spec:
  background: true
  rules:
  - name: require-scoped-sa-on-jobs
    match:
      resources:
        - group: batch
          resources: ["jobs"]
    validate:
      message: "Jobs must use a dedicated ServiceAccount. The 'default' account is forbidden."
      pattern:
        spec:
          template:
            spec:
              serviceAccountName: "!default"

This forces the engineer (me) to actually think about the permissions before the pod ever hits the scheduler. You can read more about how I use these controllers in my post on Kyverno Admission Controllers.

Why this works

The logic here is about reducing the blast radius. In a standard K8s setup, the default service account is a liability. By creating a unique SA for every workload, you create a clear audit trail. If you see a weird API call in the audit logs, you can trace it back to a specific application rather than just “some pod in the default namespace.”

also, separating the Role from the ServiceAccount allows you to rotate identities without changing permissions. If you need to rotate a token or move a workload to a different namespace, the Role remains a constant definition of “what this type of application is allowed to do.”

This approach also aligns with the two-tier credential strategy I’ve used for AI agents. By giving the agent a limited K8s SA for orchestration and a separate, more restricted set of credentials for actual tool execution, you ensure that a prompt-injection attack doesn’t lead to a kubectl delete nodes --all command. I’ve detailed that specific architecture in Agent Credential Management: Two-Tier Service Accounts.

Lessons Learned

The biggest lesson I learned is that RBAC is a living document. You cannot “set and forget” it. Every time you update a library or add a feature to your app, the required permissions might change.

What surprised me most was how often “official” Helm charts for third-party tools request cluster-admin by default. It’s a lazy way for chart maintainers to ensure their software “just works” regardless of the user’s cluster configuration. I’ve stopped applying those charts blindly. I now inspect the rbac.create and rbac.clusterRole flags and, where possible, I write my own scoped roles to replace the bloated defaults.

If I were to do this differently from the start, I would have implemented a “deny-by-default” posture using a Kyverno policy to block any pod that doesn’t have a non-default serviceAccountName specified. It’s easier to force the habit of least-privilege from day one than it is to audit and strip permissions from a running cluster.

Also, watch out for the automountServiceAccountToken: true default. It’s a silent security hole. If your app doesn’t explicitly call the Kubernetes API, there is zero reason for that token to exist inside the container. Turn it off. If the app breaks, you’ve just discovered a hidden dependency on the API that you can now properly document and scope.