Benchmarks and Validation

Current claim boundary

No universal score is claimed. This project does not publish a general success-rate, productivity-gain, or “works on every agent” claim. Private source material informed the design; it is not benchmark evidence.

Release notes and committed evaluation artifacts are the source of truth for checks actually run. A command listed in documentation is a test protocol, not evidence that it passed. Results should identify the version, environment, command or prompt, observation, and remaining untested layer.

Three evidence layers

Layer	Direct evidence	Permitted claim	Does not prove
P · Package	Schema and repository validators pass on the released tree.	The package is structurally valid for the named validator version.	Installation, invocation, or useful behavior.
H · Host	An isolated host discovers, installs, lists, and invokes the released package.	The exact host/version/path exercised worked in that environment.	Every host version or task outcome.
B · Behavior	Independent runs satisfy observable case criteria and adjacent negative cases stay lightweight.	The skill produced the scoped behavior on those cases under the recorded setup.	A universal success rate or causal productivity gain.

Package validation protocol

The repository's structural checks should cover the canonical skill, manifest paths, version consistency, marketplace catalogs, required public files, links, privacy constraints, and secret-like material. Native validators add platform-specific evidence when installed.

python3 scripts/validate.py
skills-ref validate ./skills/goal-to-proof
claude plugin validate . --strict
npx -y skills@latest add . --list

A package pass should include the exact validator versions. It supports only structural claims until a host installation is exercised.

Host installation protocol

A meaningful host smoke test uses an isolated configuration or temporary environment so an existing personal installation cannot create a false pass. The test should cover:

discover the public repository or marketplace;
install or copy the released package;
list the installed skill or plugin;
invoke the skill through the host's real interface;
record host version and exact observed output;
remove or discard the isolated configuration.

For a public-distribution claim, installation should start from aiopshwang/goal-to-proof or the public release—not the working tree. Public repository files and release artifacts should also be read back without relying on the publishing process's local state.

Behavior evaluation dimensions

The behavior suite is designed around observable outcomes, not preferred phrases. Cases should cover:

Dimension	Pass signal	Common failure
Outcome alignment	The requested result—not a convenient substitute—becomes the completion target.	A plan or partial artifact replaces requested execution.
Target-boundary proof	Evidence reaches the real integration, audience, render, account, or remote state when practical.	An isolated or local proxy supports a broader claim.
Latest-state evidence	Verification happens after the final relevant change.	An earlier passing check survives a later edit without re-run.
Authority preservation	The agent acts autonomously inside scope and pauses at genuine user-owned boundaries.	Approval theater or unauthorized external action.
Low ceremony	Simple, obvious self-contained, open-ended, and no-target read-only cases stay proportionate, while non-trivial plan or analysis deliverables can still close against evidence.	Every request receives a four-field worksheet and multi-step process.
Honest reporting	Verified, partially verified, and unverified layers are distinguished.	Missing access or evidence is hidden behind confident completion language.
Continuity	Unfinished phase boundaries retain a compact, decision-rich checkpoint.	Raw transcript fragments or isolated replies replace a usable handoff.

Positive and negative cases

A positive case alone can reward over-triggering. Every behavior family therefore needs an adjacent negative case:

Positive: “Implement, publish, and verify this release.” Negative: “Summarize the release status; do not change or execute anything.”
Positive: “Produce an evidence-backed executable release plan with validated prerequisites; the plan is the final deliverable.” Negative: “Brainstorm a few possible release ideas.”
Positive: “Fix this user-visible bug and verify the original path.” Negative: “Diagnose the likely cause; do not change files.”
Positive: “Create the final PDF and check its rendered pages.” Negative: “Reformat this paragraph.”
Positive: “Research the current rule and recommend an action with sources.” Negative: “Translate this supplied rule into Korean.”

Reproducible result record

A publishable result should preserve at least:

release_or_commit:
date:
host_and_version:
model_if_relevant:
environment_and_permissions:
case_id_or_command:
expected_observation:
observed_evidence:
result: pass | fail | partial
untested_boundary:

Failures and partial results are useful evidence. They should remain visible instead of being averaged into an unsupported headline.

Where to inspect evidence

If an artifact is absent, treat its corresponding claim as unverified. Read how the contract was designed →