Agent Glass-Break Patterns: Controlled Escalation for Production

I watched an autonomous ops agent attempt to “fix” a failing deployment by recursively deleting pods in a loop because it misinterpreted a CrashLoopBackOff as a transient networking glitch. The agent had the permissions to do it, the logic to justify it, and absolutely no circuit breaker to stop it from taking down the entire namespace. It was a classic case of giving a tool a hammer and watching it treat the entire infrastructure like a nail.

If you’re running agents in production, you’ve probably realized that the standard “system prompt” safety is a joke. Telling an LLM “please be careful with the production database” is not a security boundary. You need a glass-break pattern: a way for agents to operate within a strict sandbox, but with a controlled, audited path to escalate privileges when a human approves it or a specific condition is met.

What I tried first

My first instinct was to lean on centralized identity. I tried routing every agent tool call through an Authentik-protected gateway. The idea was simple: the agent requests a tool, the gateway checks the session, and the action is authorized.

It was a nightmare. The latency added by the OIDC handshake for every single tool call made the agent feel sluggish, and the integration overhead for low-sensitivity observability tools was absurd. I spent more time debugging JWT expiration and redirect loops than actually building agent capabilities. I was treating a low-sensitivity internal tool like a public-facing enterprise application.

Then I tried the “Super-User” approach. I gave the agent a high-privilege service account but wrapped it in a complex set of Python decorators that checked for “safe” keywords in the arguments. This failed immediately. LLMs are too good at prompt injection and parameter manipulation. A simple --force flag or a clever string concatenation bypassed my “safety” filters in minutes. I remember one specific instance where the agent bypassed a delete keyword filter by using a bash alias it had created in a previous step.

The Actual Solution: Controlled Escalation

The fix was to move the security boundary from the application layer to the infrastructure and execution layer. I implemented a three-pronged approach: Network-level isolation for internal tools, safeBins for execution control, and a manual escalation trigger.

1. Infrastructure-Level Isolation

Instead of forcing every internal tool through a heavy auth layer, I shifted to a LAN-only access model using Kubernetes NetworkPolicy. This ensures that only the agent orchestrator can talk to the tool, and only from a specific subnet.

For a tool like Agent Quest, I stripped out the Authentik dependency and locked it down at the pod level. This assumes you’re running a CNI that actually supports NetworkPolicies, like Calico or Cilium. If you’re on a basic Flannel setup, this won’t do anything.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: traefik-allow-egress-to-agentquest
  namespace: agent-tools
spec:
  podSelector:
    matchLabels:
      app: traefik
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.140/32 # The specific IP of the Agent Quest service
    ports:
    - protocol: TCP
      port: 4444

This removes the auth overhead while ensuring that no one outside the cluster (or even in other namespaces) can trigger the tool. It aligns with the privacy-routed inference pattern of keeping sensitive traffic off the open wire. I’ve found that relying on the network fabric is significantly more reliable than relying on application-level middleware that can be bypassed if the agent finds a way to call the API directly.

2. Execution Control with safeBins

For tools that actually execute shell commands, like mcporter, I stopped relying on regex filters. I implemented a safeBins pattern. This is essentially an allowlist of binaries and the specific flags they are permitted to use.

The implementation works as a wrapper. The agent doesn’t call kubectl or mcporter directly; it calls a wrapper script that validates the arguments against a JSON schema before spawning the child process.

{
  "safeBins": {
    "mcporter": {
      "allowedValueFlags": ["--config", "--timeout"],
      "forbiddenFlags": ["--force", "--recursive", "--delete-all"],
      "max_args": 5
    },
    "kubectl": {
      "allowedValueFlags": ["--dry-run=client", "-n", "--context"],
      "restrictedCommands": ["delete", "patch", "edit"],
      "allowedCommands": ["get", "describe", "logs"]
    }
  }
}

If the agent tries to pass a flag not in the allowedValueFlags list, the execution engine kills the process before it ever hits the shell. This prevents the “hallucinated flag” problem where an LLM tries to use a flag that doesn’t exist but happens to look like a dangerous command.

For example, if the agent attempts to run: kubectl delete pod nginx-deployment-xyz -n production

The wrapper catches the delete command in the restrictedCommands list and returns a hard error: ERROR: Command 'delete' is restricted. Please request escalation via the Glass-Break protocol.

3. The Glass-Break Escalation

When the agent hits a safeBins restriction or a NetworkPolicy block, it triggers an escalation event. I integrated this with an n8n workflow that sends a Slack notification to me with the exact command the agent wants to run and the reasoning behind it.

The workflow logic is handled as follows:

1. The wrapper script catches the restrictedCommands violation.
2. It sends a POST request to an n8n webhook containing the agent_id, the requested_command, and the llm_reasoning.
3. n8n sends a Slack message with two buttons: “Approve” and “Deny”.
4. If “Approve” is clicked, n8n updates a Redis key: escalation:agent_id:allowed = true with a TTL of 300 seconds.
5. The agent retries the command. The wrapper checks Redis, sees the active escalation, and allows the command to execute.

I’ve found that a 5-minute window is the sweet spot. It’s long enough for the agent to finish the specific task it requested, but short enough that I don’t have to worry about a “zombie” escalation leaving a high-privilege window open indefinitely.

Troubleshooting the Gap

When implementing this, I hit a few walls that the documentation for these tools doesn’t mention. The most annoying was the “Shell Escape” problem.

I initially used a simple subprocess.run(cmd, shell=True) in Python. I quickly discovered that the agent could just use a semicolon to chain commands. It would send: kubectl get pods; rm -rf /

The safeBins check saw kubectl get pods at the start and let it through. I had to switch to shell=False and pass the arguments as a list. This forced the OS to treat the entire string as a single argument to the binary, which caused the rm -rf part to be passed as a literal argument to kubectl, resulting in a harmless Error: unknown flag from the binary itself.

Another issue was the “Context Switch” error. When the agent is in a restricted state, it often gets confused and keeps trying the same failing command over and over. I had to modify the orchestrator to inject a “System Warning” into the agent’s context the moment a glass-break is triggered:

SYSTEM: Your last command was blocked by safeBins. You are now in Escalation Mode. Wait for human approval before retrying.

Without this, the agent enters a loop that can spam your Slack notifications 50 times in ten seconds.

Why it works

This works because it acknowledges that the LLM is an unreliable narrator. You cannot trust the agent to follow safety guidelines, but you can trust the Linux kernel and the Kubernetes API.

By moving the constraints to the binary level (safeBins) and the network level (NetworkPolicy), we create a hard boundary. The agent can hallucinate all it wants, but it cannot execute a --force flag if the execution wrapper doesn’t allow it.

Combining this with the two-tier service account model ensures that even if the agent escalates, it’s using a token with a limited scope. I run the agent’s default identity with a read-only ClusterRole, and the “Escalated” identity is a separate token stored in a secret that is only injected into the environment by the wrapper when the Redis key is present.

Feature	System Prompt Safety	Application Logic	Glass-Break Pattern
Enforcement	Probabilistic	Code-based	Infrastructure-based
Bypass Risk	High (Prompt Injection)	Medium (Logic errors)	Low (Kernel/Net level)
Audit Trail	LLM Logs	App Logs	System/Network Logs
Human-in-the-loop	None	Optional/Manual	Mandatory for high-risk

Lessons Learned

The biggest takeaway here is that “safe” is a relative term. I spent a lot of time trying to make the agent “smart” enough to not break things. The real victory came when I accepted that the agent will try to break things and instead focused on making the environment “dumb” enough to resist those attempts.

If I were to do this again, I’d implement the escalation at the OPA (Open Policy Agent) level using Gatekeeper. Instead of a custom Python wrapper, I’d use a mutating admission webhook to strip dangerous flags from the agent’s requests before they even reach the K8s API. It’s more complex to set up, but it’s a more native way to handle policy in Kubernetes.

One final caveat: be careful with your Redis TTL. If you set it too long, you’ve essentially just given the agent a permanent skeleton key. If you set it too short, the agent might timeout mid-operation, leaving your infrastructure in a half-migrated state. 5 to 10 minutes is usually the right balance for most ops tasks.

For those building these systems, I highly recommend looking into building agent skills to ensure that the agent knows exactly which tools are “safe” and which require the glass-break, reducing the number of failed attempts and Slack notifications.