Skill Testing and Eval: Measure Before You Ship

By Yanni Papoutsis | 6 min read | 2026-07-13

Testing a Claude skill means checking two separate things every time: whether it triggers when it should (and stays quiet when it should not), and whether its output clears a bar you defined in advance, measured against a small, repeatable set of prompts rather than one anecdotal try that happened to go well.

This is the sixth post in the skills-authoring series. It assumes the skill itself is built (see writing a skill from scratch) and its MCP dependencies, if any, are wired up correctly (see MCP tool access). Testing is the gate between "it worked once" and "I trust this."

TL;DR: A skill test has two axes: trigger accuracy (does the description fire on the right tasks and stay silent on unrelated ones) and output quality (does the body's instructions actually produce a good result). Build a small eval set of 8 to 12 prompts covering typical phrasing, edge cases, and a few prompts that should NOT trigger the skill at all, then run each one in a genuinely cold session, not a session where you have already been discussing the skill. Check for variance by running the same prompt more than once; a skill whose output quality swings wildly between runs needs tighter instructions, not just a lucky rerun. Every time you edit a skill, re-run the full eval set rather than just the prompt you were fixing, since a change aimed at one failure can quietly break something that used to work. A skill is ready to ship when it passes its own eval set consistently, not when a single demo looked good.

What are you actually testing when you test a skill?

Two different failure modes hide inside "the skill doesn't work," and they need different fixes.

Trigger accuracy

Does the skill load for the tasks it is meant to handle, and does it stay out of the way for tasks it is not meant to handle? This is entirely a function of the description field, covered in depth in the skill triggers guide. Testing it means trying real phrasing, not the phrasing you used while writing the skill.

Output quality

Given that the skill did load, does the body's instructions produce something you would actually accept? This is a function of how clear and checkable the instructions are. Vague instructions ("write a good summary") produce inconsistent quality even when the skill triggers perfectly every time.

How do you build an eval set?

Step 1: Write 8 to 12 representative prompts

Cover the phrasing variety a real user would produce: short requests, long requests, requests with extra context, requests using different words for the same underlying task.

Step 2: Include negative examples

At least two or three prompts in the set should be tasks the skill must NOT trigger for, even though they are adjacent to its actual job. This is the check most people skip, and it is where trigger collisions hide.

Eval set for `changelog-writer`:
1. "Write a changelog for this diff" (should trigger)
2. "Summarise what changed in this release" (should trigger)
3. "What changed in this release, in plain English for a customer email"
   (should trigger, different downstream use)
4. "Write a blog post about our new feature" (should NOT trigger --
   check it does not fire alongside a marketing-copy skill)
5. "What does this function do" (should NOT trigger -- code
   explanation, not change summarisation)

Step 3: Define a pass/fail bar for each prompt

Before running anything, write down what a passing result looks like. For a changelog skill: correct grouping (Added, Changed, Fixed, Removed), no invented changes, imperative mood. Without a bar written in advance, it is too easy to grade a mediocre result generously because you already know what you wanted it to say.

Step 4: Run the set in cold sessions

Trigger accuracy tests only mean something in a session that has not already been primed by discussing the skill. Open a new session for each prompt, or at minimum for a representative sample of them.

How do you check for variance?

Run the same prompt three times. If the output quality is close every time, the skill's instructions are tight enough. If it swings from excellent to unusable, the instructions probably leave too much to interpretation, common causes include missing concrete rules (see the "never invent a change" style rule from the skill-writing guide) or an ambiguous step ordering.

Variance check for changelog-writer, prompt #1, 3 runs:
Run 1: correct grouping, no invented changes -- pass
Run 2: correct grouping, no invented changes -- pass
Run 3: merged "Fixed" into "Changed" -- fail

Fix: add an explicit rule distinguishing the two categories
with one example each, then re-run all 3.

Regression testing when you edit a skill

Every edit to a SKILL.md is a change to both the trigger surface and the instructions, even if you only meant to touch one. Re-run the whole eval set after any edit, not just the single prompt that motivated the change. A fix for prompt #7 can easily break the trigger behaviour tested by prompt #4 if the description changed even slightly.

A simple scoring rubric example

Check Pass criteria
Triggers on positive prompts Loads for all prompts marked "should trigger"
Stays silent on negative prompts Does not load for prompts marked "should NOT trigger"
No invented content Output contains nothing absent from the source input
Consistent structure Same output format across repeated runs
Handles missing input gracefully Produces a clear message, not a guess, when input is incomplete

When is a skill "good enough" to ship?

When it passes its own eval set consistently across repeated runs, not when one demo run looked good. A skill that works once under ideal conditions and has never been tested cold, tested with negative examples, or tested for variance is not evaluated, it is anecdotal. Treat the eval set as part of the skill, versioned alongside SKILL.md, so the next edit has something concrete to check against.

Frequently asked questions

How many prompts does a useful eval set need?

Eight to twelve is a practical starting size for most skills: enough to cover typical phrasing, a couple of edge cases, and at least two negative examples. Larger, higher-stakes skills warrant a bigger set; simple ones can work with fewer as long as negative examples are included.

Do I need special software to run a skill eval?

No. A markdown file listing prompts and pass criteria, run manually in cold sessions, is a complete eval set. The discipline matters more than the tooling: write the bar down before you test, and re-run the full set after every edit.

What is the single most common thing an eval set catches that a single test misses?

Trigger collisions and variance. A one-off test can look perfect and still hide a skill that only works on the exact phrasing you happened to use, or one that produces a good result three times out of five. Both only show up once you test more than once and test the boundaries on purpose.

About the author

Yanni Papoutsis builds AI products, automation pipelines, and technical documentation with Claude, and publishes free tooling and guides at yanni.uk.

Next step: Once a skill passes its eval set, package and share it using the publishing and sharing guide, or explore 1,000+ AI tools at yanni.uk/ai-tools.

Sources