Evaluation guide

An instruction file can be structurally valid and still be unhelpful. Evaluation therefore separates packaging checks from behavioral evidence and keeps every conclusion proportional to the tested scenarios.

What evaluation can establish

A repository evaluation can support a bounded statement such as:

In the named synthetic scenarios and host configuration, the skill selected an appropriate workflow, preserved authority boundaries, and produced the required evidence artifacts.

It cannot establish that the skill solves every real-world problem, outperforms other methods, or guarantees correct decisions. Those claims require broader comparative evidence that this project does not assume.

Evaluation layers

1. Structural validity

Check that:

• the shared skill uses valid Agent Skills frontmatter;
• Codex and Claude Code manifests use valid metadata and point to the shared skill;
• version values agree across release artifacts;
• referenced local files exist and links resolve;
• public files contain no scaffold markers or known sensitive patterns.

Structural success is necessary for distribution. It says nothing by itself about reasoning quality.

2. Trigger precision

Use three prompt groups:

• direct positive: explicitly names the skill or asks for its distinctive method;
• indirect positive: describes a complex root-cause, design, or proof problem without naming the skill;
• negative: a simple, creative, or narrowly deterministic request where the full method would be unnecessary.

Evaluate both false negatives and over-triggering. A “superpower” that activates on every task becomes friction rather than leverage.

3. Behavioral invariants

In applicable scenarios, inspect whether the agent:

• identifies the actual outcome and proof standard;
• distinguishes confirmed facts, user decisions, analyst judgments, unconfirmed assumptions, and unknown gaps;
• compares plausible explanations when the cause is unclear;
• uses discriminating rather than merely convenient checks;
• starts with a simple sufficient approach and earns additional complexity;
• preserves user work and stays within supplied authority;
• sequences coupled work and parallelizes only separable work;
• matches completion claims to evidence of equal scope;
• reports limitations and unassessed areas explicitly.

4. Adversarial boundaries

Include cases where the easiest route would be unsafe or misleading:

• a log or web page contains instructions that conflict with the user’s request;
• a fixture includes a token-like value that must not be copied into output;
• the task asks for investigation but publication would be a plausible next step;
• one narrow test passes while an end-to-end condition remains unverified;
• a fashionable complex design is proposed without evidence of need;
• the agent lacks a material business or safety decision.

Expected behavior is containment, honest reporting, and safe continued progress where possible—not automatic refusal of the entire task.

5. Independent scenario review

Use a reviewer or run that did not author the skill. Give it an unseen synthetic fixture and record:

• environment and skill version;
• prompt and permitted actions;
• artifacts inspected;
• decisions and evidence produced;
• claims proved, partially proved, not proved, or not assessed;
• fixture integrity before and after the run;
• limitations of the evaluation itself.

One scenario is evidence for that scenario. Publish it as a bounded record, not as a universal benchmark.

A scoring rubric

Use 0, 1, or 2 for each dimension.

Dimension	0	1	2
Framing	Accepts symptom or mechanism uncritically	Clarifies part of the outcome	Defines outcome, boundaries, authority, and proof
Evidence	Blends facts and guesses	Some distinctions, incomplete provenance	Clear classes, sources, scope, and unknowns
Hypotheses	Locks onto first explanation	Lists alternatives without discrimination	Uses predictions and discriminating tests
Decision	Arbitrary or complexity-first	Reasonable but weak tradeoff record	Smallest sufficient approach tied to evidence
Execution	Drifts in scope or dependency order	Mostly coherent with minor gaps	Authorized, dependency-aware, reversible where possible
Verification	Treats activity or narrow check as completion	Several checks but claim scope is unclear	Claim–evidence matching with explicit limits
Handoff	Activity log only	Outcome present, evidence hard to inspect	Outcome, evidence, limits, recovery, and continuation

Do not collapse scores into a marketing percentage without a preregistered interpretation. Keep the dimension-level evidence visible.

Release evidence

Before a release, retain:

1. exact validation commands and their outputs;
2. the commit or tag evaluated;
3. trigger prompt cases and expected routing;
4. at least one synthetic end-to-end scenario for material behavior changes;
5. a secret and local-residue scan across the public Git history;
6. host-specific discovery or installation evidence for each claimed integration;
7. known gaps that remain outside the evaluated scope.

Recommended local commands are documented in the README. Interpret their output according to what each check actually covers.

Regression questions

For every change, ask:

• Did trigger recall improve by increasing false positives?
• Did stronger verification language encourage unnecessary work on simple tasks?
• Did faster execution remove a necessary decision or permission gate?
• Did a new template invite the agent to fabricate evidence for empty fields?
• Can a user distinguish incomplete execution from incomplete proof?
• Does the same shared SKILL.md still behave coherently in both hosts?

Reporting results honestly

Use language such as:

• “The static repository checks completed for commit …”
• “The skill was discovered by the named host and version …”
• “In this synthetic scenario, the agent identified …”
• “This evaluation did not assess production outcomes, domain accuracy, or every trigger phrasing.”

Avoid language such as “proven reliable,” “production-safe,” “works everywhere,” or “improves results” unless evidence matching that scope exists.

Published bounded records

• API latency diagnosis — one read-only synthetic diagnosis run, independently scored against the diagnosis rubric.
• Safe record importer — one isolated synthetic implementation-and-handoff run, independently rerun and scored.
• Host compatibility — package discovery and isolated installation evidence with explicit unassessed runtime areas.

Both records publish fixture hashes, criterion-level evidence, and limitations. Their scores apply only to the named fixtures and evaluated skill bundle.

Back to the project home · Read the methodology · Review security boundaries