Pushgateway Heartbeat Gotcha: When ndots and NetworkPolicy Silently Eat Your Alerts
How ndots:5, a wildcard DNS record, and a default-deny NetworkPolicy combine to swallow CronJob heartbeats without a single error or alert.
#kubernetes
38 posts
FastMCP Agent Mail: RBAC Tokens vs Anonymous Access, and the 403 Errors in Between
Why a FastMCP agent mail server that works anonymously in dev returns 403 behind TLS ingress, and how to wire bearer tokens without leaking them.
Kyverno allowLatestTag Pitfalls: How Policy Can Stop Pods From Restarting
A disallow-latest-tag policy only blocks new admissions. That gap turns a routine pod eviction into a permanent outage. Here's the failure mode and the fix.
Longhorn Read-Only Mounts: Detection, Recovery, and Closing the Silent Failure Window
A Longhorn volume can report Healthy while the filesystem inside your pod has been read-only for hours. How to detect it, recover it, and alert on it.
ForwardAuth with Authentik: Wiring SSO Across Multiple Apps Without Breaking Ingress
Authentik ForwardAuth on Traefik fails in ways that look like success. Outpost bindings, NetworkPolicies, secret shape, and header mapping across apps.
Backing Up Qdrant Snapshots Correctly: From emptyDir to NFS Persistent Backups
Qdrant snapshots on emptyDir vanish on restart, and snapshots on the data PVC die with the disk. Here's the NFS-backed pattern that survives both.
Traefik Host Collision: Why Duplicate Host Rules Break IngressRoutes
In Traefik v3, a wildcard router can outrank an exact Host match, because default priority is rule string length. Here's how it fails and how to fix it.
Moving Scheduled LLM Curation from Cloud APIs to Local Models
A comparison of cloud APIs vs local models for scheduled agent memory curation, and the headless-cron gotchas nobody warns you about.
Prometheus Alerting Rules That Don't Cry Wolf
How to write Prometheus alerts that carry context, tolerate transient scrape blips, and page only when something is actually broken.
SealedSecrets: Storing Secrets in Git Without the Risk
How SealedSecrets lets you commit encrypted secrets to Git safely, why the real risk is losing the controller's private key, and how to back it up.
MetalLB on Bare Metal: LoadBalancer Without a Cloud Provider
Your bare-metal LoadBalancer Service is stuck Pending. Here's how MetalLB hands out real LAN IPs, why L2 mode works, and the traps that cost me hours.
CloudNativePG: Running PostgreSQL in Kubernetes Without the Pain
Running PostgreSQL on Kubernetes with CloudNativePG: the Kyverno policy traps, SSL gotchas, read/write splitting, and the 16.4 segfault to avoid.
Network Policies with Calico: Default Deny and Namespace Isolation
Rolling out default-deny NetworkPolicies and namespace isolation with Calico without breaking DNS, ingress, or admission webhooks.
Agent Glass-Break Patterns: Controlled Escalation for Production
How to implement controlled escalation for AI agents using safeBins and network-level constraints to prevent production catastrophes.
Velero + MinIO: Kubernetes Backup Strategy for Bare Metal
Moving beyond cloud S3 to a bare-metal backup strategy using Velero, MinIO, and Longhorn. Lessons on CSI snapshots and ETCD persistence.
Grafana Dashboards: Information Density vs Readability
Stop cramming every metric into one screen. A practical look at balancing information density and performance in Grafana dashboards.
Kubernetes RBAC: Building Least-Privilege Service Accounts
Moving beyond cluster-admin for everything. A practical approach to scoping ServiceAccount permissions for production workloads and AI agents.
Longhorn Volume Health: The Gap Between 'Healthy' and Actually Working
Stop trusting the Longhorn UI blindly. Learn to monitor replication, fix stale mounts, and manage snapshot bloat in production K8s storage.
Privacy-Routed LLM Inference: Keeping Sensitive Data Out of the Cloud
How to build a routing layer for AI agents that ensures sensitive data stays on local hardware while leveraging cloud LLMs for non-private tasks.
Kyverno Admission Controllers: Policy-as-Code That Actually Works
Moving beyond the happy path of Kubernetes policy enforcement. Real-world Kyverno pitfalls, mutation loops, and the gap between docs and production.
AdGuard Home: Network-Wide DNS Filtering with Failover
Setting up AdGuard Home for network-wide DNS filtering with a robust failover strategy to prevent total internet outages.
Stop Merging Broken YAML: Kubernetes Manifest Validation in CI
Don't let invalid manifests break your GitOps pipeline. Learn how to use kubeconform and Kyverno exclusions to catch errors before they hit production.
cert-manager + Cloudflare DNS-01: Automated TLS for Everything
Automating TLS with cert-manager and Cloudflare DNS-01 in Kubernetes
SealedSecrets Key Backup: Don't Lose Your Encryption Keys
How to back up and recover SealedSecrets encryption keys in Kubernetes
Ollama on Kubernetes: Recreate Strategy and Single-GPU Deadlock
Deploying Ollama on Kubernetes can lead to GPU deadlocks. Here's how to avoid them.
Wildcard DNS + ndots:5: The TLS Nightmare and How to Fix It
Kubernetes default DNS settings can cause TLS certificate mismatches when using wildcard DNS. Here is how to debug and fix it.
Self-Improving AI Infrastructure: How Your Homelab Wiki Updates Itself
How to automate your homelab wiki with self-improving AI infrastructure
The 6-Layer Memory Architecture I Run for Claude Code
Open-sourcing the memory system behind my Claude Code setup: CLAUDE.md, path-scoped rules, wiki, vector search, cognitive memory. With the mistakes.
Building Karpathy's LLM Wiki: A Production Homelab Implementation
Implementing Karpathy's LLM Wiki in a homelab with real-world lessons and gotchas
AMD iGPU Stealing Your RAM: UMA Frame Buffer on Headless Servers
AMD iGPU steals RAM on headless servers, here's how to fix it
Agent Credential Management: Two-Tier Service Accounts for Secure AI Agent Workflows
Managing agent credentials with two-tier service accounts: a secure approach for AI agent orchestration
Pod Disruption Budgets: Why kubectl drain Gets Stuck on Longhorn
Pod Disruption Budgets can block kubectl drain on Longhorn. Here's how to avoid it.
Helm fullnameOverride: Naming Sanity in ArgoCD
Avoid naming chaos in ArgoCD by using Helm fullnameOverride effectively
NVIDIA Container Toolkit: Why the Default Runtime Matters
Fixing default runtime misconfigurations in NVIDIA Container Toolkit for GPU workloads
AMD Ryzen C-State Freezes: How `processor.max_cstate=1` Saved My Proxmox Node
Ryzen freezes in Proxmox? Learn how to disable deep C-states and stop random system lockups.
GPU Passthrough on Proxmox: A Field Guide to the Gotchas That Bit Me
The documentation won't warn you about D3cold bricking, PCIe bus renumbering, or why the NVIDIA device plugin silently fails. This is that guide.
GitOps for Homelabs: How ArgoCD App-of-Apps Scales Your Cluster
How the ArgoCD app-of-apps pattern brings real GitOps discipline to homelab Kubernetes — repo structure, examples, and what I'd do differently.
Building a Production Homelab: Multi-Node Proxmox Cluster with Kubernetes
How I built a multi-node Proxmox cluster running Kubernetes with GPU passthrough, GitOps, and dozens of services — and what broke along the way.