Introduction
Getting started
The end-to-end system for writing, testing, and grading the skills your AI agents run on.
Write a Skill
The exact format and philosophy a skill needs to trigger correctly and hand off clean output.
Run the Eval Harness
Get a variance score across multiple model runs before you ship anything.
Benchmark Iterations
Run two skill versions head-to-head and get a clear delta score.
API Reference
Full reference for the Skill Eval config, gates, and output schema.
Skill Eval gives your agent team a structured, measurable foundation. Every skill that ships has a score, a benchmark, and a repeatable pass/fail gate.
What is Skill Eval?
Skill Eval is the infrastructure layer between your agent ideas and your production fleet. It answers three questions before any skill touches a live workflow: Does it trigger correctly? Is the output consistent? Is version two better than version one?
Why structure matters
Without structure, agent skills fail silently. A skill that worked in one session breaks in another because the trigger wording changed, the model shifted, or the context window got crowded. Skill Eval catches all three failure modes with a gate that runs before merge.
// skill-eval.config.js
export default {
skill: 'skill-eval',
version: '2.1.0',
gates: {
varianceThreshold: 0.12,
minRuns: 5,
benchmarkDelta: true,
},
}
The gate runs the skill five times minimum, measures variance, and compares the delta against the previous version. If variance exceeds 0.12 or the delta is negative, the skill does not ship.
Built for real agent fleets.
Skill Eval is not a toy eval framework. It was built inside a production multi-agent operation and hardened against the edge cases you only find when agents actually run in client workflows.
The Three Gates
Skill Eval enforces three sequential gates before any skill reaches a production agent.
Gate 1 — Structure
The skill either follows the canonical format or it does not. Name, description, when-to-use triggers, anti-patterns, variation matrix. Missing fields fail the gate automatically.
Gate 2 — Variance
The eval harness runs the skill against a fixed prompt corpus across multiple model runs. If output variance exceeds the configured threshold, the skill is flagged as brittle. Fix the wording, tighten the context, and re-run.
Gate 3 — Benchmark delta
When you rewrite a skill, the harness runs both versions and computes a delta score. You get a clear answer: improvement, regression, or no meaningful change. The score goes into the commit record so the team knows exactly what changed and why.
Proof of Work
Client names withheld under NDA.
Teams using a structured skill-eval process ship reliable agent behaviors in a fraction of the debugging cycles that unstructured skill writing requires. Ask us to show you the before/after on a live call.
The results we see consistently: skills pass the gate on the first eval attempt roughly 40 percent of the time. The remaining 60 percent need one targeted fix — and they almost always pass on the second run. Zero skills that passed all three gates have required a hotfix in production.
Ready to run your first eval?
Book a demo call and we will walk through your current skill library, run a live eval on one of your existing skills, and show you exactly where the gates would catch problems you have not found yet.