measured · n=32 · bank published
content provenance & marking · seat of the instrument: Brussels · EU AI Act (Reg. (EU) 2024/1689), Article 50 · C2PA 2.x
Whether the marking the law asks for on AI-generated or AI-altered content is still
there and valid after the content has moved through the world. The bank puts marked assets to a
detector and reads one of two labels back — SURVIVES or
DESTROYED — and grades the run deterministically by macro-F1. No model judges
another model. The bank has been measured and is published on Hugging Face; it is not yet quotable, and
this page shows you no number, because the board does not yet separate anything.
csoai/gspc-prv.
Thirty-two items clear usable_n = 30, so the reason no score appears here is
not a shortage of items. It is that the board does not yet discriminate: item discrimination sits
below the 0.15 floor, so the items fail to separate stronger runs from weaker
ones. A number computed on a board that separates nothing would read like a finding and be none, so no
score, no macro-F1, no accuracy and no interval is published until a clean discriminating board exists.| layer | instrument | status of the citation |
|---|---|---|
| Statutory | EU AI Act — Regulation (EU) 2024/1689, Article 50 (transparency obligations). Article 50 requires that content generated or manipulated by an AI system — synthetic audio, image, video and text, and deep fakes — be marked in a machine-readable format and disclosed, and that a person be told when they are interacting with an AI system. | Cited at the article level. The article number and the regulation number are fixed. The exact sub-paragraph an item turns on (for example the provider marking duty versus the deployer disclosure duty) is fixed per item in the bank, not asserted here. |
| Technical | C2PA — Coalition for Content Provenance and Authenticity, specification version 2.x. The machine-readable manifest format the bank uses to carry provenance assertions on a content asset and, crucially, to validate them: a manifest can be present yet fail to validate. | The specific C2PA 2.x conformance clause is to be sourced. C2PA 2.x is named as the technical carrier and the source of the validity check; no section number or publication date is stated here because none has been read against the specification text on this page. |
Each item is a content asset that once carried an Article 50 marking, put through a route — a re-encode, a crop, a re-upload, a format change — and then handed to a detector. The gold label records what the marking should be after that route, and the run is graded on whether the detector's read matches. Two labels, one duty each.
| label | what it means |
|---|---|
SURVIVES | The marking is still present and still validates after the route. Only a marking that a verifier would accept as intact counts here. |
DESTROYED | The marking is gone, OR it is present but no longer validates. The v3 validity principle is the whole point of the axis: presence is not survival. A manifest that a verifier rejects is DESTROYED, not SURVIVES — a broken seal is a broken seal, however much of it is still stuck to the envelope. |
DESTROYED so the label tracks what a downstream verifier would actually
conclude, not what is merely stuck to the file.Two gates stand between a bank and a published number. The bank clears the first and not yet the second, and both are stated so the reader can see exactly where it stands.
| gate | state |
|---|---|
Enough items — n ≥ usable_n (30) | Cleared. n = 32. The lane is not held back for want of items, and this is not the n<30 case some axes are in. |
A board that discriminates — item discrimination above the 0.15 floor | Not cleared. On the current board the items do not separate stronger runs from weaker ones; discrimination sits below the floor. A macro-F1 read off a board that separates nothing is a number, not a measurement, so none is quoted. |
End to end means ten links, the same ten every measure in the Arena is held to. Four are complete on ProvBench today; the rest are outstanding and shown as such.
The bank is published and it has been run against a model — those links are real. The links that remain open are what would carry ProvBench from a measured bank to a quotable one, and the largest of them is not a link at all but the discrimination the board must first show.
A genuine skeleton, not pseudocode. It reads a JSONL bank, applies
the validity rule, extracts one label per completion, and returns exactly one of three states. It emits a
run's macro-F1 only when the run is MEASURED at usable n — and a separate
lane-level gate holds the quote until the board discriminates, which is why the estate publishes no number
for ProvBench today.
#!/usr/bin/env python3 # provbench_score.py — deterministic scorer for the ProvBench content-provenance axis. # # Three per-completion / per-run outcomes, never folded together: # MEASURED — a label was read from the completion and compared to gold. # UNPARSEABLE — no label could be read. Excluded from the score and counted on # its own. A completion we cannot read is not a wrong answer. # UNMEASURED — the run as a whole. Returned when no scorable item exists, when # usable_n is not reached, or when the instrument's own error # rate exceeds threshold. In this state NO score is emitted. # # v3 validity principle: a marking that is present but does not validate is # DESTROYED, not SURVIVES. Presence is not survival. # # Input line (JSONL): # {"id": "prov-0001", "route": "reupload", "gold": "DESTROYED", # "completion": "...detector output...", "transport_error": false} # # Usage: python3 provbench_score.py bank.jsonl import json, re, sys from collections import defaultdict USABLE_N = 30 # below this a run is UNMEASURED — no interval, ever MAX_INSTRUMENT_ERROR = 0.10 # above this the run is a failed run, not a low score DISCRIMINATION_FLOOR = 0.15 # below this the BOARD does not separate — lane is held, not quoted # Two-label survival vocabulary, one duty each. The whole axis turns on validity. LABELS = ["SURVIVES", "DESTROYED"] TOKEN = re.compile(r"\b(SURVIVES|DESTROYED)\b", re.I) def verdict(present, valid): # The validity rule in one place: SURVIVES only if the marking is present # AND still validates. Present-but-invalid folds into DESTROYED. return "SURVIVES" if (present and valid) else "DESTROYED" def extract(text): # Read exactly one label, or None. None is UNPARSEABLE, never a wrong answer. if not text: return None m = TOKEN.search(text) if not m: return None label = m.group(1).upper() return label if label in LABELS else None def wilson(k, n, z=1.959963985): # Interval on the accompanying accuracy rate. macro-F1 has no closed form, # so the rate that CAN carry an interval is the one published with one. if not n: return None p = k / n; d = 1 + z*z/n c = (p + z*z/(2*n)) / d h = z * ((p*(1-p)/n + z*z/(4*n*n)) ** 0.5) / d return [round(max(0.0, c-h), 4), round(min(1.0, c+h), 4)] def macro_f1(pairs): # Macro, not accuracy: always answering the commoner label must lose here. tp = defaultdict(int); fp = defaultdict(int); fn = defaultdict(int) for gold, pred in pairs: # only MEASURED pairs reach this loop if pred == gold: tp[gold] += 1 else: fn[gold] += 1; fp[pred] += 1 total = 0.0 for lab in LABELS: p = tp[lab] / (tp[lab] + fp[lab]) if tp[lab] + fp[lab] else 0.0 r = tp[lab] / (tp[lab] + fn[lab]) if tp[lab] + fn[lab] else 0.0 total += 2*p*r / (p + r) if p + r else 0.0 return total / len(LABELS) def score_run(path): measured, unparseable, instrument_errors, n_lines = [], 0, 0, 0 for line in open(path, encoding="utf-8"): line = line.strip() if not line: continue n_lines += 1 row = json.loads(line) if row.get("transport_error"): # the instrument failed, not the model instrument_errors += 1; continue pred = extract(row.get("completion")) if pred is None: unparseable += 1; continue # excluded from the score, never scored wrong measured.append((row["gold"], pred)) scored_n = len(measured) err_rate = instrument_errors / n_lines if n_lines else 0.0 # Run-level UNMEASURED gates, checked BEFORE any number is emitted. if scored_n == 0: return {"status": "UNMEASURED", "reason": "no scorable items", "instrument_error_rate": round(err_rate, 4)} if err_rate > MAX_INSTRUMENT_ERROR: return {"status": "UNMEASURED", "reason": "instrument error above threshold", "instrument_error_rate": round(err_rate, 4)} if scored_n < USABLE_N: return {"status": "UNMEASURED", "reason": "n below usable_n; no interval, no quote", "n": scored_n, "unparseable": unparseable, "instrument_error_rate": round(err_rate, 4)} correct = sum(1 for g, p in measured if g == p) return {"status": "MEASURED", "macro_f1": round(macro_f1(measured), 4), "accuracy": round(correct / scored_n, 4), "accuracy_ci95": wilson(correct, scored_n), # bare figure never ships alone "n": scored_n, "unparseable": unparseable, # reported alongside, not folded in "instrument_error_rate": round(err_rate, 4)} def item_discrimination(matrix): # matrix[item] = list of 1/0 across the runs on the board. Point-biserial # between each item and the run totals: does the item separate stronger runs # from weaker ones? Return the mean over items. A run scores itself; a BOARD # is what can discriminate, so this needs several named runs, not one. runs = len(next(iter(matrix.values()))) if matrix else 0 if runs < 2: return 0.0 totals = [sum(matrix[i][r] for i in matrix) for r in range(runs)] mt = sum(totals) / runs disc = [] for i in matrix: col = matrix[i]; mi = sum(col) / runs num = sum((col[r]-mi)*(totals[r]-mt) for r in range(runs)) di = sum((col[r]-mi)**2 for r in range(runs)) dt = sum((totals[r]-mt)**2 for r in range(runs)) disc.append(num / ((di*dt) ** 0.5) if di and dt else 0.0) return sum(disc) / len(disc) if disc else 0.0 def lane_quotable(board_matrix): # The lane gate the page obeys today: even with a MEASURED run at usable n, # the board must discriminate above the floor before any number is quoted. d = item_discrimination(board_matrix) return {"discrimination": round(d, 4), "quotable": d >= DISCRIMINATION_FLOOR, "reason": None if d >= DISCRIMINATION_FLOOR else "board discrimination below floor; measured, not quoted"} if __name__ == "__main__": print(json.dumps(score_run(sys.argv[1]), indent=2))
MEASURED from
score_run — the bank is real and the scorer reads it. But
lane_quotable is what the page obeys, and on the current board its
discrimination is below 0.15, so it returns quotable = false.
That is not the scorer failing; it is the estate refusing to quote a number off a board that does not yet
separate anything. The three states and the floor are enforced in code so the discipline survives a late
night and a deadline.Every published run in this estate is a signed observation, and the signature is narrow on purpose. It records the route — which marked assets were put to which detector, which completions came back, which the scorer could read, whether each marking still validated, and the error rate of the instrument that carried them. It is a record of a measurement CSOAI performed. A signed route is not a verdict: not a statement that a tool marks content correctly, not a finding that a system complies with Article 50, not a ruling on a C2PA manifest a producer attached to a file. Provenance measured here means we observed this content and recorded what we found. It is never read backwards into a judgement about the person or product that made the content.