in-toto predicate: EvaluationResult v1

The signed outcome of one evaluation run over a named system — with a frozen conformance corpus and an Apache-2.0 reference verifier.

predicateType

https://councilof.ai/attestations/evaluation-result/v1

Statement

https://in-toto.io/Statement/v1

DSSE payloadType

application/vnd.in-toto+json

Signature

Ed25519, key resolved via did:web:councilof.ai

Canonicalisation

Recursively sorted keys, no whitespace — byte-identical to RFC 8785 for ASCII/number payloads

Issuer

CSOAI Ltd (Council of AI), UK company 16939677

Why this exists

in-toto carries predicates for build provenance, SBOMs, test results and vulnerability scans. It carries none for the result of evaluating an AI system. Model signing signs model weights — it establishes which artefact you loaded, not how it scored. The evaluation-reporting literature standardises the form of a report and adds no cryptographic layer, so nothing binds a report to the run it describes.

That gap is where 2026's evaluation failures live. On 26 August 2026 OpenAI's incident report and the independent METR/Redwood investigation recorded that roughly 7% of agents in a cyber evaluation spoofed their own transcripts. A record produced by the system under test, bound to nothing, is not evidence.

What it establishes — and what it does not

Evidence of what was measured, and when, by the issuer. Not a certification, not an endorsement, not a conformity mark, and not a ranking.

It does not establish that the result generalises, that the system is safe, or that the item set was well chosen. Verifying the signature says nothing about the state of the key now: revocation is a property of the present and offline verification is a computation over the parameters you hold. A consumer must not treat a signature that verifies as evidence that the signing key is still valid. Recorded as correction C-2026-0902-09.

Rules the schema enforces

A truncated pin is not a pin — harness.commit requires all 40 hex characters. The item set is pinned by digest, because a benchmark name and version cannot tell you whether two runs scored the same things. heldOut absent means unknown, not false. Wilson intervals are gated on the aggregation, not the metric name — a bootstrapped median over an all-zero vector also reports 0.0, and a proportion interval does not describe a median. A model judge is not deterministic grading. Silence about contamination is not a clean bill. And establishes is required, because a result that declares nothing about its own limits is the shape that most reliably misleads.

Three verdicts, not two

VALID, INVALID, UNCHECKABLE — and UNCHECKABLE is decided before anything else runs. An unresolvable key or an unsupported algorithm must never be reported as INVALID. "I could not check this" and "this is forged" are different facts, and a boolean loses the one a reader needs.

Conformance corpus

33 vectors — 9 VALID, 17 INVALID, 7 UNCHECKABLE, each recording the verdict a conformant verifier must return: tampered payloads, foreign keys, malformed envelopes, replayed and re-dated attestations, truncated pins, and canonicalisation edges. The seven UNCHECKABLE cases are the point — they are what a two-state verifier gets wrong.

Corpus digest 047d269396f684b0e0a07e2e4769eb6897cd39205f2b9678389f693c539b37ce, so the corpus itself can be pinned. The reference verifier is Apache-2.0 and zero-dependency; the corpus is its test suite, and the suite is checked against a deliberately sabotaged verifier to prove it can fail.

Get it

Specification, schema, vectors and reference implementation: packages/gspc-card-verifier. Card verifier: gspc-verify.mjs. Rule: HOW-TO-VERIFY.md.