Building Agent Skills: A Pattern for Discoverable Capabilities

I spent three weeks building a set of “tools” for a custom agent that could manage my infrastructure, only to realize the agent had no idea how to actually use them in combination. I’d give it a read_file tool and a grep_search tool, and it would repeatedly try to read a 50MB log file into its context window instead of grepping for the error first. The tools existed, but the “skill” of knowing when and how to sequence them was missing.

If you’re building AI agents, you’ve probably hit this. Most frameworks treat tools as a flat list of functions. You dump 20 Python functions into the system prompt and hope the LLM’s reasoning is strong enough to pick the right one. It usually isn’t.

The False Start: The “Tool Soup” Approach

My first instinct was to just write better descriptions. I spent hours tweaking the docstrings of my functions, adding phrases like “Use this tool ONLY when the file is larger than 10KB.” I was treating the LLM like a junior dev who just needed better instructions.

The problem is that tool-calling is fundamentally different from skill execution. A tool is an atomic action (e.g., GET /api/v1/status). A skill is a capability (e.g., “Diagnose why the Kubernetes ingress is returning 502”).

I tried to solve this by creating “orchestrator” tools—basically giant functions that wrapped other functions. This just moved the complexity into my Python code. I ended up with a monolithic diagnose_k8s_issue() function that was 300 lines long and impossible to test. I had created a rigid script, not a flexible agent. I’d effectively turned my AI agent back into a bash script with a fancy interface.

I also tried “Few-Shotting” the behavior. I added five examples to the system prompt showing the agent using grep before read_file. This worked for about three turns, then the agent would drift back into its habit of trying to swallow the entire log file. The context window is a fickle thing; the more tools you add, the more the “noise” of the tool definitions drowns out the “signal” of the few-shot examples.

The Solution: Discoverable Skill Definitions

The shift happened when I stopped defining tools and that started defining skills as discoverable metadata. Instead of just exposing a function, I created a registry where skills are defined by their intent, the tools they require, and a suggested execution pattern.

I implemented this using a structured manifest. Instead of the LLM guessing which tool to use, the agent first queries a “Skill Registry” to find a capability that matches the user’s intent. This decouples the “What to do” from the “How to do it.”

Here is the pattern I’m using now. Each skill is a standalone definition that explicitly maps the capability to the underlying tool. I use a YAML-based registry because it’s easy to version control and can be updated without redeploying the entire agent binary.

# skill-registry.yaml
skills:
  - id: "log-error-search"
    name: "Search Logs for Errors"
    description: "Finds specific error patterns in system logs without loading entire files."
    required_tools: ["grep", "ls"]
    execution_pattern: |
      1. Use 'ls' to identify the relevant log file in /var/log.
      2. Use 'grep' with the --context flag to find the error and surrounding lines.
      3. If no results, try searching for 'FATAL' or 'CRITICAL'.
    usage_example: "/skill:search --tool=grep --pattern='timeout' --files='/var/log/syslog'"

  - id: "k8s-pod-debug"
    name: "Debug Pod CrashLoopBackOff"
    description: "Diagnoses pods stuck in CrashLoopBackOff by checking events and logs."
    required_tools: ["kubectl_get_pods", "kubectl_describe", "kubectl_logs"]
    execution_pattern: |
      1. Run 'kubectl get pods' to find the crashing pod.
      2. Run 'kubectl describe pod' to check for OOMKilled or Liveness probe failures.
      3. Run 'kubectl logs --previous' to see why the last instance died.
    usage_example: "/skill:k8s-debug --pod=api-gateway-7f8d"

To make this work in practice, I changed the agent’s loop. Instead of User -> LLM -> Tool, the flow became User -> LLM -> Skill Lookup -> LLM -> Tool Sequence.

When the agent identifies it needs to search logs, it doesn’t just call grep. It retrieves the log-error-search skill definition. This gives the LLM a “recipe” for the task. It’s the difference between giving someone a pile of ingredients and giving them a recipe book.

If you’re building these as MCP servers, you can implement this by creating a specific “discovery” tool that returns these manifests. I’ve written about building MCP servers with FastMCP, and applying this skill pattern there makes the tools significantly more reliable across different IDEs like Antigravity or Kiro.

Handling the “Dirty Work” of Execution

One of the biggest gaps in agent documentation is how to handle the actual execution of these skills when they hit real-world infrastructure. For example, if a skill requires searching through Kubernetes volumes, you can’t just assume the agent has the right permissions or that the volume is healthy.

I hit a wall where my “Log Search” skill would fail because the underlying Longhorn volumes were hitting snapshot limits, causing the filesystem to go read-only. The agent would just report “Permission Denied,” which is useless.

The error usually looked like this in the agent’s output: grep: /var/log/syslog: Read-only file system

The LLM would then try to “fix” this by attempting to chmod the file or sudo the command, which obviously fails in a containerized environment. I had to build “pre-flight” checks into the skill execution layer. If a skill involves storage, it first checks the volume health. If I see a bunch of stale snapshots, I have the agent run a cleanup before attempting the search.

# Example of a cleanup command the agent can trigger via a 'maintenance' skill
# This targets snapshots older than a specific date to prevent volume degradation
kubectl delete snapshots.longhorn.io -l "snapshot-name=old-snapshot-2025"

This is where the gap between “it works in the playground” and “it works in production” becomes obvious. If you’re running these agents on bare metal, you need to account for the infrastructure failures I’ve detailed in my posts on Longhorn volume health.

Deep Dive: Composability and Dependency Mapping

The real power of the skill pattern isn’t just the recipe; it’s the ability to compose skills. In a production environment, a “High Level Skill” often depends on several “Atomic Skills.”

For example, a Deploy-and-Verify skill might look like this:

1. Provision-Infrastructure (Atomic)
2. Deploy-Application (Atomic)
3. Health-Check-Endpoint (Atomic)

If the Health-Check-Endpoint fails, the agent doesn’t just throw a generic error. Because it’s following a skill manifest, it knows exactly which previous step failed and can trigger a specific Rollback-Infrastructure skill.

I’ve implemented this using a dependency graph. When the agent loads a skill, it also loads the dependencies. This prevents the “hallucination loop” where an agent tries to verify a deployment before the deployment tool has even returned a success code.

To manage the security of these compositions, I use a two-tier credential system. The “Discovery” phase of the skill uses a read-only service account, while the “Execution” phase requests a short-lived token for the specific tools required. I’ve detailed this approach in Agent Credential Management: Two-Tier Service Accounts.

Troubleshooting the Skill Loop

Even with a registry, things break. The most common failure mode I’ve seen is “Skill Drift,” where the LLM ignores the execution_pattern and tries to be “clever.”

The Symptom: The agent retrieves the k8s-pod-debug skill, but instead of running kubectl describe, it tries to call a non-existent kubectl_get_detailed_status tool it imagined based on the skill’s description.

The Fix: I added a “Strict Mode” to the tool executor. If the LLM calls a tool that isn’t listed in the required_tools section of the active skill, the executor rejects the call with a specific error: ERROR: Tool 'kubectl_get_detailed_status' is not authorized for the current skill 'k8s-pod-debug'. Please use the tools listed in the skill manifest.

This forces the LLM to re-read the manifest. It’s a bit like a guardrail for the reasoning process.

Another edge case is the “Infinite Loop of Discovery.” This happens when the agent can’t find a skill that perfectly matches the user’s request, so it keeps querying the registry with slight variations.

Example of the loop:

• User: “Fix the network”
• Agent: Queries registry for “Fix Network” -> No result.
• Agent: Queries registry for “Network Repair” -> No result.
• Agent: Queries registry for “Infrastructure Network” -> No result.

I solved this by adding a “Fallback Skill.” If the registry returns no matches after two attempts, the agent is forced to use a General-Investigation skill, which limits it to basic ls, ps, and top commands to gather more data before trying another lookup.

Why This Pattern Works

The reason this beats a flat list of tools is cognitive load. LLMs have a limited context window, and more importantly, a limited “attention” span (the lost-in-the-middle phenomenon). When you provide 50 tools, the probability of the LLM picking a suboptimal tool increases.

By using a skill registry, you’re implementing a form of “just-in-time” prompting. The agent only sees the detailed instructions for the specific skill it needs for the current step.

Feature	Tool-Based Approach	Skill-Based Approach
Context Usage	High (all tool docs in prompt)	Low (only active skill in prompt)
Reliability	Variable (depends on LLM reasoning)	High (follows a defined recipe)
Discoverability	Poor (LLM must guess tool)	High (explicit registry lookup)
Maintenance	Hard (must update all prompts)	Easy (update YAML manifest)
Composition	Manual/Implicit	Explicit/Graph-based

Lessons Learned

If I were doing this over again, I would have moved to a registry much sooner. I spent way too much time trying to “prompt engineer” my way out of a structural problem.

One thing that surprised me was how much the LLM actually appreciates the usage_example field in the YAML. Providing a concrete string like /skill:search --tool=grep acts as a powerful anchor. It reduces the variance in how the LLM formats its tool calls, which in turn makes the parsing logic on the backend much simpler.

Another caveat: don’t make your skills too granular. If you have a skill for “Read one line of a file” and another for “Read ten lines of a file,” you’ve just recreated the “Tool Soup” problem. The goal is to group atomic tools into a meaningful capability.

The most important takeaway is that the gap between a demo agent and a production agent is almost always found in the “unhappy path.” The docs show you how to call a tool; they don’t show you what to do when the agent tries to read a 50MB log file or when the underlying Kubernetes volume is read-only. Building a skill pattern isn’t just about organization—it’s about creating a predictable environment where the agent is guided by recipes rather than guesses.