1.20.0 · Schema: result-v1.schema.json ·
Source of truth: codecalc/contract.py
1.20.0 is a MINOR bump over 1.19.0, on the doctor schema
(doctor-v1.schema.json), additively. codecalc doctor --deep is now bounded
by one wall-clock budget for the whole pass (180 s by default), where before
the per-probe deadlines and the warm retry could multiply to 710 s on a host
where every version probe hangs. The report says what the budget did in four
new top-level keys, always present: deep_budget_s and deep_elapsed_ms
(null without --deep), deep_budget_exhausted (boolean) and deep_skipped
(integer). A runtime the budget cut short stays installed with a
probe_error saying it was not probed; the budget running out is not evidence
about any runtime, so it never demotes one or flips healthy, the same rule
1.13.0 set for a single probe’s timeout. A 1.19.0 client sees keys it did
not know before and nothing else changes.
1.19.0 is a MINOR bump over 1.18.0. It adds one value to unenforced,
additively: timeout_enforced_after_provider_response, reported by the
Piston provider for a compiled language (one with a compile stage). Piston
takes a separate timeout for each phase, so compile plus run can total twice
the caller’s timeout; the provider cannot share one deadline across them. The
adapter therefore checks the total after the response: past timeout, the
verdict is TLE in phase run with timed_out: true, and the output Piston
returned is kept as it was. Consumers that matched unenforced values by exact
string keep working; a 1.18.0 client only sees a value it did not know before.
1.18.0 is a MINOR bump over 1.17.0. It closes the rest of GH #321/THE-1086: trace_execution’s events items and its
truncated_reason enum both gain a value, additively —
truncated_reasongains"event_detail_over_cap". Reported when one trace event’s own detail (almost alwayslocals, on a function call with more than 200 changed locals) exceeded a per-event size cap. Before this fix, such an event was silently DISCARDED without advancing the parser’s internal step counter, which then rejected every later, otherwise- legitimate event too — the rest of the trace vanished, andlines_never_ executedreported lines the run’s ownstdoutproved had executed.events[]items gain an optionaldetail_droppedboolean, present andtrueonly on the stub this fix now admits in place of the eventevent_detail_over_capdescribes:step/line/event/funcare the real ones,localsis{}. The stub still counts towardlines_executedandbranches— dropping the whole event, an earlier draft of this fix, cost a real coverage sample along with the oversized detail (cross-vendor review finding #2 on this PR).events_consistent’s own definition gains one more condition:discarded_eventsmust be0. A sandboxed program can pre-write a forged, step-matching event for a step the harness has not emitted yet; the harness’s own later, GENUINE event for that same step number is then rejected as stale, but the count/step arithmetic alone still balances (cross-vendor review finding #3 on this PR — seecodecalc/tracing.py’s “TRUST BOUNDARY” section for the mechanism and why an honest harness never produces a discarded event on its own). No existing client’s read ofevents_consistentnarrows in a way that breaks it: this can only turn atruea1.16.0client would have seen into afalse, for a trace this exact attack applies to — the field’s own documented meaning (“a best-effort tamper/corruption signal, never a cryptographic guarantee”) already told a reader not to treat it as absolute proof.
codecalc/tracing.py’s module docstring (“TRUST BOUNDARY” section) and
tests/test_trace_execution.py for the full mechanism and the reporter’s
own repro, now fixed.
1.17.0 is a MINOR bump over 1.16.0. It adds stderr to compact_result,
closing GH #323 (THE-1088): execute_code(compact=True) dropped stderr
unconditionally, so a failed run could say THAT it failed but never WHY —
the same defect class as code/error/remedy on rejected (#117, below),
one shape over. stderr is now present whenever the run did not succeed
(ok is false, exit_code is a nonzero integer, or verdict is other
than OK — the third clause is what still catches an OLE run, which can
otherwise report ok: true, exit_code: 0) and absent on a clean run exactly
as before, so a 1.16.0 client that already ignores unknown fields sees no
change in shape on the success path it already validated against. See
codecalc/server.py’s compact_result (the _COMPACT_ALWAYS comment block)
for the full reasoning behind the three-way gate.
1.16.0 is a MINOR bump over 1.15.0. It adds a TWELFTH shape,
branch_reachability, for the new tool of the same name: every if/elif/else
arm and while/for(range, static bounds) loop in one python3 function,
decided reachable/dead/unknown by z3 rather than shown from one
concrete run —
supported—falseonly when a construct on some path could not be translated despite passing the tool’s own upfront scan (an undefined name is the one case that scan cannot catch, since name binding is a flow property, not a node-type property); the branch it broke is reportedverdict: "unknown"rather than aborting the whole call. A plain solver timeout leaves thistrue.inputs— resolvedname -> 'int'/'bool'/'str'for every analyzed parameter, after applying the caller’s owninputsoverride/narrowing.branches— one entry per if/elif/else arm and while/for line, each{line, kind, condition, verdict, witness?, boundary_inputs}.conditionis the accumulated guard text reaching that arm, withnot (...)spelled out for an elif/else’s implicit negation.witness(present only whenverdictisreachable) and every non-nullmin_input/max_input/equality_edge_inputinsideboundary_inputsare full parameter dicts, shaped to pass straight intocompare_edge_cases’stest_inputs.dead_count/reachable_count/unknown_count— the same totals a reader could derive frombranchesthemselves, kept as a cheap top-level summary.truncated—truewhenmax_branchescut discovery short.suggested_test_inputs— every witness and boundary input, deduped, ordered by the line of the branch it came from.
language, more
than one top-level function, or no function and no inputs declaring the
free variables — is {ok: false, error, code: "validation"} with no
shape-specific key, naming the construct and its line where one applies.
That already matches rejected below (ok + error, no verdict) and
needed no branch of its own, the same pattern compare_edge_cases’s and
verify_optimization’s own refusal shapes follow. See codecalc/ branch_reachability.py’s module docstring for the mechanism end to end and
docs/design/2026-09-08-branch-reachability.md for why the supported
subset and the boundary/loop semantics were drawn where they were.
1.14.0 (superseded by the two entries above) was a MINOR bump over
1.13.0. It added a TENTH shape,
execution_trace, for the new trace_execution tool: the identical
execution_envelope execute_code returns for the same python3 program
(same 20 envelope keys, same stdout/stderr/exit_code/verdict/
timings), plus a per-line event trace and a static branch/line-coverage
report computed from an AST parse of the submitted source —
events— ordered{step, line, event, func, locals}for USER-CODE frames only (library/stdlib frames excluded by filename).eventisline/call/return/exception;localson every event is only the names that CHANGED since the previous event in that same frame (never a full dump each step), each repr capped at ~200 chars. Areturnevent additionally carriesreturn_value; anexceptionevent carriesexception_type/exception_message. CPython’s own trace protocol fires a spuriousreturnevent witharg=Nonefor a frame an exception is UNWINDING through (indistinguishable, read naively, from a function that genuinely returnedNone) — the harness tracks which frames are currently unwinding and suppresses that event, so areturnineventsalways means the function actually returned.event_count/steps_before_truncation— the former islen(events); the latter is always equal to it by construction (nothing is recorded once tracing stops) and exists because a caller reasoning about a truncated trace wants the count of what stopped tracing without a second lookup.truncated/truncated_reason(max_events, an internal trace-byte ceiling, ortrace_file_exceeded— present only whentruncatedis true) — the first two caps stop RECORDING, never the program: the underlyingexec()keeps running to completion at full speed regardless, sostdout/exit_code/verdictare always the real, complete ones even wheneventsis partial. A hard kill (TLE/OLE/MLE) before either cap trips reportstruncated: false— that story is already told byverdict/timed_out/output_truncated, and this field is deliberately scoped to this tool’s own recording caps rather than overloaded with an unrelated meaning.trace_file_exceededis different in kind from the other two — seediscarded_events/events_consistentimmediately below.discarded_events/events_consistent— the trace is produced BY the traced process, at the SAME privilege it runs with, so a sandboxed program can derive the trace file’s own path from__file__and write to it directly (a cross-vendor review reproduced a forged trailingreturnevent this way, usingos._exit(0)to skip the harness’s own cleanup). This tool cannot make the trace an attestation of behaviour — it remains exactly as trustworthy as that program’s own stdout — but it bounds and surfaces tampering rather than trusting the file blindly: the (unsandboxed) parser never reads more than_MAX_TRACE_BYTES + 4 KiBoff disk regardless of how much a program appends (trace_file_exceededfires when the file is bigger than that), every candidate line is schema-validated (unknown keys, wrong types, or astepthat skips or repeats the harness’s own monotonic sequence are DISCARDED and counted indiscarded_events, never raised), and the harness writes a final{"event": "end", "step": N, "emitted": N}line on every path it returns through normally —events_consistentisfalsewhenever that line is missing, its count disagrees with what this parser actually accepted, or anything follows it. That catches crude tampering, not a deliberate forger: the program can read its own trace file, append a step-continuous forged event plus a matchingendline, and exit before the harness writes the real one, and the parser accepts that as consistent.events_consistent: trueis therefore not proof of an untampered trace; the trace is exactly as trustworthy as the program’s own stdout. Seecodecalc/tracing.py’s module docstring, “TRUST BOUNDARY” section, for exactly what this does and does not guarantee.branches— source line number (string key) -> execution count, for everyif/elif/while/for/tryline, from a staticastparse of the submitted code (never from the trace alone, so a branch that never executed still gets a0entry rather than being absent).lines_executed/lines_never_executed— the cheap “which branch actually ran” view: every statically-detected executable line, split by whether alineevent ever fired on it.
execute_code is disclosed through the
EXISTING unenforced array rather than a new field, because it describes
a guarantee this tool could not fully apply rather than new information
about the traced program:
"threads: only the main thread is traced"—sys.settraceis a per-THREAD hook; a program that starts its own thread runs code this tracer never sees events for (that code still executes normally — onlyevents’ completeness is affected, neverstdout/exit_code). Added whenever the harness observes more than one thread alive at any point.
1.12.0 client reading execute_code’s own
execution_envelope shape sees no change, because trace_execution is a
new tool with a new shape, not a change to an existing one.
execution_trace would otherwise ALSO match execution_envelope (this
schema’s additionalProperties is intentionally open, so extra keys alone
never disqualify a match) — closed by adding not: {required: [events]}
to execution_envelope’s own definition, the same technique compact
already uses against backend and rejected against verdict, so
oneOf’s “exactly one shape matches” claim still holds for a trace result,
a plain envelope, and every other shape in this document.
1.15.0 is a MINOR bump over 1.14.0. It adds an ELEVENTH shape,
session_snapshot_result, for the new session_snapshot tool — archive a
session’s workspace files to a snapshot stored outside the jailed workspace,
and restore one into a new or the same session. Purely additive: nothing an
existing 1.14.0 client already reads changes shape or meaning: no existing
tool’s result gained, lost, or renamed a field, and the new shape’s own
discriminator (action) appears on none of the other ten. A failing
session_snapshot call was already representable — it matches rejected,
same as every other tool — so this bump is the SUCCESS shape becoming
describable, not a new failure mode. See CHANGELOG.md’s [Unreleased] entry
for the tool’s own design (caps, the identity checks at save/replace, the
closed-enum refusals a hostile archive gets on restore) and “Examples” below
for one real transcript per action.
1.13.0 is a MINOR bump over 1.12.0, on the doctor schema
(doctor-v1.schema.json) rather than the execution result — the two share
this one version and policy, see “Two version numbers, on purpose” below.
It adds probe_ms to every runtimes[] row a --deep version probe was
attempted for, and narrows what a version-probe timeout is allowed to
mean, closing four separate 2026-09-08 hosted-runner failures (main among
them): three were a cold windows-latest runner’s FIRST rustc --version
going through the rustup proxy — an arg-forwarding shim that has to locate
and re-exec the real toolchain component before it can answer anything — and
exceeding the probe’s 10-second deadline; a fourth, on a later commit, was
plain go version doing the same with no proxy involved and no per-command
override for go at all. _probe_version reported that exactly like a
spawn failure (hard_failure=True either way), and report() trusted the
confirmed _VERSION_FLAG entry (both rustc and go are audited, GNU-style
commands) the same way it trusts a genuine nonzero exit, so a tested-tier
runtime that was never actually broken read unhealthy, healthy flipped
false, and tests/test_doctor.py’s real-host --deep assertions (added
for the earlier go/lua/zig regression, #282 — they assert every resolved
tested-tier language stays installed/available) failed alongside it. A
timeout means
“the probe did not answer within the deadline”; it is not evidence the
runtime is broken, and this version stops treating the two as the same
claim: _probe_version now returns hard_failure=False for a timeout, and
report() never promotes a bare timeout to unhealthy regardless of
whether the command has a confirmed _VERSION_FLAG entry — only a
_HELLO runtime whose own hello-world run independently fails still gets
demoted, exactly as it did before. Two mitigations reduce how often the
timeout fires at all: one retry (a slow-start proxy is warm on its second
call) and a raised per-command deadline for the toolchains audited as
routing through this shape on a cold host (rustc/rustup, dotnet, java,
kotlinc, swift). probe_ms is the new field: the total wall time a
--deep version probe took across every attempt, present whenever one was
attempted at all — a 1.12.0 client that already reads runtimes[].status
sees no change in what a nonzero exit or a spawn failure demotes; only the
timeout case narrows, and only in the direction of trusting less, so no
existing unhealthy classification is undone by this.
1.12.0 is a MINOR bump over 1.11.0. It adds stdout_raw to every
translation_verification case’s source/target (_translation_side_ properties) and to every compare_edge_cases per-language run
(_edge_case_run_properties), closing part of codecalc #286: _normalize
(translation.py) used str.splitlines() to compare two programs’ stdout,
which treats VT (0x0B), FF (0x0C), FS/GS/RS (0x1C-0x1E), NEL (0x85), U+2028
LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR as line breaks and folds all
seven into \n — so a source printing a real \n and a port printing U+2028
compared equal, and the stdout shown in the result was the NORMALIZED
string, so a reader had no way to see the difference even after being told
where to look. _normalize itself is fixed (see translation. NORMALIZE_TOLERANCE: only \r\n/\r -> \n, trailing per-line spaces/tabs,
and trailing blank lines are tolerated now — nothing else Python calls a line
or whitespace boundary is touched), which is a behavior change with no shape
to model in a schema. stdout_raw is the shape change: exactly what a side
wrote, before normalization, sitting next to the normalized stdout so a
reader can see the two are the same string, or are not, without re-running
anything. Both fields are always present, not only when they differ — a
1.11.0 client that already reads stdout sees no change; stdout_raw is
purely additive, hence MINOR.
1.11.0 is a MINOR bump over 1.10.0. It adds four fields to
optimization_verification.inference, closing a false accept reproduced live
in CI: verify_optimization certified IDENTICAL before/after code as a
verified speedup (macOS sandbox job, native executor, main; measured ratio
1.21x, sizes_rejecting 2/3, a bare majority satisfied). All four are
additive — nothing a 1.10.0 client already reads changes shape or meaning:
method, on everyinference.per_size[]row.stats.mann_whitney_ualready computes"exact"or"normal_approximation"per test;_infer_ speedupcomputed it and threw it away, so a caller could not tell an exact p-value from an asymptotic one produced at n=5 — a sample sizestats.py’s own module docstring says is “not a number worth calling alpha=0.05 against”. Both of the CI incident’s “rejecting” sizes werenormal_ approximation, and this is why a caller can now see that for themselves.ratio, on everyinference.per_size[]row. That size’s OWN before/after ratio, the same computationspeedup.per_sizealready reports per position.sizes_rejectingnow requires it to clear the caller’smin_speedupas well asp_value < effective_alpha— a size can be statistically significant on a difference too small to be the speedup the caller asked for, andp_valuealone could not previously show that.effective_alphaandcorrection, oninferenceitself. The family-wise-error-controlled barsizes_rejectingis actually compared against:"unanimity"(every counted size must reject;sizes_total <= 3) or"bonferroni"(alpha / sizes_totalper size, a majority of those required;sizes_total > 3), or"n/a"whensizes_totalis 0.alphaitself is unchanged in meaning — it stays the nominal, uncorrected level a caller already reads. A bare majority at the nominalalpha, always, is what let the CI incident’s two false-positive sizes (out of three) carry the vote; unanimity atsizes_total <= 3alone would already have refused it (2 of 3 is not 3 of 3). Unanimity over a SINGLE counted size is no correction at all (one uncorrected test at the nominalalpha), soacceptedis nevertruewithsizes_totalbelow 2;decision_basissays so in words when that is the reason.reason, on asizes_below_floor[]entry, present only when that entry was excluded for one of two NEW reasons: fewer thanstats.min_testable_n(alpha)runs a side (structurally unable to ever reject atalpha— replaces a flat “fewer than 2 runs” guard), or anormal_approximation-method result on fewer than 8 runs a side whose two samples’ raw-run ranges overlap (the literal “the timer cannot always say which side a given pair of runs favoured” case — seeoptimization._ranges_overlap; gating on the tie alone, without the overlap qualifier, was measured live as nearly blinding the tool to genuine wins, so it is deliberately narrower than “any tie”). An entry excluded for an OLDER reason (the visibility floor, an unmeasurable side) carries noreason— a1.10.0client that already matchessizes_below_ floorentries by their original three keys (size/before_ms/after_ms) still matches exactly the entries it always did.
codecalc/optimization.py’s _infer_speedup/_accept_decision/
_fwer_correction/_ranges_overlap for the full mechanism, and
CHANGELOG.md’s [Unreleased] entry for the measured before/after
false-accept counts.
1.10.0 is a MINOR bump over 1.9.0, and adds a new result shape plus two new
optional fields on an existing one:
- A
comparison_rowsbranch, forcompare_execution. Its result — the same code run in N languages side by side — never matched any of this document’soneOfbranches; it sat outside the published contract entirely, the same gapedge_case_comparisonclosed forcompare_ edge_casesin1.5.0. Closing it is what makes the second change below possible to describe, not just to ship: a field added to an unmodeled shape is not a documented contract change. error/code/remedy/code_inferredoncompare_execution’s per-languageresults[]rows, and oncompare_edge_cases’s per-languageruns[lang]entries. Before this version, a row or run that failed for a REQUEST-level reason — no runtime installed for that language, or a timeout — carried onlystdout/stderr/exit_code(orverdict), same as an ordinary program failure. A caller could not tell “this language’s own snippet is broken” from “this host does not have this language installed” without re-running that one language alone throughexecute_ codeand readingcodethere. Both tools’ OUTERokwas, and remains,Trueonce the comparison itself ran — it means “this tool produced a real table”, never “every language succeeded” — soserver.py’s_codedwrapper (which classifies a result viaerrors.ensure_code) never reached these nested rows either;errors.stamp_rowis the same classification applied at the row level instead. An ordinary program failure (a realRTE/OLEexit_code, the code ran and failed on its own terms) still carries none of the four fields, matching the INTENDED reading of “The program ran and failed” vs “Rejected before execution” below:codeis meant to mark a failed REQUEST, never a failed program. That is NOT yet what the top-level execution envelope itself does for every case: a plain RTE (sys.exit(3)) or a plain wall-clock timeout throughexecute_codecomes backcode: "internal"today (executor.executenever setserrorfor either, soerrors.ensure_code’s message matcher falls through tointernal) — pre-existing on the envelope, not something this version touches, and tracked separately;errors.stamp_rowapplies the rule as it is meant to work at the row level, not as the envelope currently does. This closes the SAME gap1.9.0closed one backend at a time (below): a Rust-backend spawn failure now setserroron the envelope itself, butcompare_execution/compare_edge_casesnever forwarded that field — on EITHER backend — into their own rows at all.
1.9.0 is a MINOR bump over 1.8.0. A spawn failure — the requested
language’s runtime or compiler could not be launched at all, typically
because it is not installed — used to reach a caller two different ways
depending on backend: the pure-Python fallback set error (and
code_inferred classified it runtime_unavailable), while the native Rust
executor put the identical fact only in stderr and left error unset,
which classified it internal — the wrong remedy for “install the
language”. Both backends now set error for this failure (worded to start
“runtime unavailable for the … phase: …”), and both now report
exit_code: null rather than the Rust backend’s previous internal sentinel
-2 — the SAME convention the contract already documented for “nothing ran”
elsewhere (see “Truncation, and how much was cut” below). The error
property itself is not new — it already appeared in the schema’s execution
envelope shape (shared with the dead-session-worker shape) — so nothing
about the SCHEMA changed at this bump, only what a real spawn-failure result
now populates: a 1.8.0 client already tolerant of an absent error on the
envelope shape sees no shape it cannot parse, and one that reads error
when present now gets it on a failure it previously had to infer from
stderr prose — hence MINOR, not MAJOR.
1.8.0 is a MINOR bump over 1.7.0. It ADDS a probe_error field to the
doctor diagnostic document’s runtimes[] entries, present whenever a
--deep version probe ran and failed — a nonzero exit, a timeout, or a
spawn error — whether or not that failure was trusted enough to move
status. This closes a report reproduced on macOS with no JDK:
/usr/bin/java (Apple’s stub, always present on PATH there) exits non-zero
printing “The operation couldn’t be completed. Unable to locate a Java
Runtime.”, and the version probe used to read that text as a perfectly good
version string and leave status at installed — the failure was observed
and then discarded. version never carries a failed probe’s text;
probe_error does.
A probe failure moves status to unhealthy only when it is TRUSTED: a
spawn failure or timeout (the probe never got an answer at all, regardless
of which flag was used) always is; a mere nonzero exit is trusted only when
the flag used is one this code has confirmed correct for that command (an
explicit override, the same list java’s -version was already on). This
distinction exists because --version is a GNU convention, not a universal
one — go --version exits 2 (go version is the real form), lua --version exits 1 (-v is correct), zig --version exits 1 (zig version is correct) — and a nonzero exit on an UNCONFIRMED flag is reported
as merely unmeasured (probe_error still recorded, status untouched),
never as evidence the runtime itself is broken. A language with a
hello-world check (_HELLO) is stronger evidence than either reading and is
always consulted first.
healthy also gained a narrow new condition alongside the existing
“workspace writable and a backend resolved” rule: it now goes false when a
tested-tier runtime (the tier a CI job genuinely executes and checks on
every PR — see tier_summary) resolved and then proved broken by a TRUSTED
failure, whether a non-executable file or a failed --deep probe. An
unhealthy best_effort/plan_only runtime — java and kotlin included —
still leaves healthy untouched, same as before: nothing ever promised
those work. No result a 1.7.0 client already understood changes meaning —
probe_error is additive and absent on every row it does not apply to, and
healthy’s new condition only ever turns a currently-true value false
in a situation this document already called “the install cannot execute
what it advertises” — hence MINOR, not MAJOR.
1.7.0 is a MINOR bump over 1.6.0. It ADDS n_after to
optimization_verification.speedup.per_size[] and size_after to
optimization_verification.inference.per_size[], both present ONLY when the
candidate’s own n at that position differs from the baseline’s — the
disclosure for a position _align_sizes (codecalc/optimization.py)
deliberately leaves unaligned: when the candidate exhausted its ENTIRE
rescale budget (_MAX_RESCALE_ROUNDS, 10^4x the starting size) without ever
clearing optimization._VISIBILITY_FLOOR_MS, re-measuring anyone would be
either unsafe (the candidate’s exhausted n was never validated as safe for
the baseline) or pointless (the baseline already tried its own smaller
sizes and failed to help). This bump also makes the visibility-floor
exclusion rule itself ASYMMETRIC: a position now lands in
inference.sizes_below_floor only when the BASELINE side is unmeasurable
or never clears the floor, regardless of the candidate. The PRIOR, symmetric
rule (excluded when EITHER side was below the floor) was a bug, not an extra
safety margin: a baseline visibly costing real time, paired against a
candidate too fast to register even after exhausting its own rescale
budget, is the single most decisive result this tool can produce — the
symmetric rule discarded exactly that pairing, driving inference.sizes_total
to 0 (and accepted to False with “no size had enough comparable runs”)
for the canonical genuine-O(1)-win case this whole auto-scale system exists
to certify. The new rule instead feeds the baseline’s sample at its own n
directly against the candidate’s sample at ITS OWN (necessarily larger) n —
conservative by monotonicity: a candidate still measured faster while doing
at least as much work is faster, full stop — and size_after on that
per_size entry (in both speedup and inference) discloses the
mismatched pairing rather than silently presenting it as same-n. No result
that a 1.6.0 client already understood changes meaning: a position that
was comparable before stays comparable with the same size/n, and the
new fields are additive, absent whenever the two sides ran at the same n
(still the common case) — hence MINOR, not MAJOR.
1.6.0 is a MINOR bump over 1.5.0. It ADDS sizes_below_floor to
optimization_verification’s inference object: every size excluded from
per_size/sizes_total/sizes_rejecting (the majority-of-sizes vote) for
ANY reason — never cleared optimization._VISIBILITY_FLOOR_MS within
_timed’s per-size auto-scale budget, an unmeasurable baseline/candidate,
or too few comparable runs — rather than tested on a still-noisy sample or
silently dropped; before_ms/after_ms are null when no duration was
ever recorded. See codecalc/optimization.py’s _VISIBILITY_FLOOR_MS
docstring for the bug this closes (a real, macOS-CI-observed false rejection
of a genuine 1.9-2.4x speedup, because the OLD floor was applied across all
sizes combined instead of per size). sizes_below_floor is required
(always present on inference, an empty list when every size cleared every
bar, the common case), same as sizes_rejecting/sizes_total always are —
the same choice tier made at 1.3.0, and with the same consequence: a
strict validator applying the 1.6.0 schema to a result STORED before this
bump (shaped like 1.5.0, with no sizes_below_floor key at all) rejects
it as missing a required field, exactly as it would a pre-1.3.0 doctor
document missing tier. No existing key moves or changes meaning on either
result a 1.5.0 client already understood, so that client is unaffected —
additions only, hence MINOR.
1.5.0 is a MINOR bump over 1.4.0. It ADDS three new oneOf branches —
translation_verification, optimization_verification, edge_case_comparison
— covering verify_translation, verify_optimization and
compare_edge_cases, which were previously stamped contract_version like
every other tool but matched none of the five execution shapes and were
explicitly scoped OUT of the “exactly one branch matches” claim by this same
document (see the removed caveat under “The five shapes” below, now “The
eight shapes”). Concretely:
translation_verificationisverify_translation’s result:ok,passed,reason,matched,mismatched,inconclusive,total,cases[](translation.aggregate’s evidence), plusgrade/grade_basis/grade_rules_version(every call is graded — seecodecalc/grades.py).translation.aggregate()had never setokat all before this bump — a real omission caught in review, not a shape to document around:verify_optimization’s own{"ok": True, "accepted": False, ...}early return already established thatokmeans “the tool completed and produced a real answer”, never the verdict, so a mismatch isok: trueexactly like a pass.aggregatenow sets it unconditionally, and this branch requiresokthe same as every other shape in this document.optimization_verificationisverify_optimization’s result:ok,accepted,reason, an embeddedverification(the BAREtranslation_verificationevidence, without the grade/contract_versionwrapper —verify_optimizationcallsverify_translation()directly, not through the graded MCP tool), plusspeedup(ratio +per_size[].{n,before_ms,after_ms,ratio,n_after}—n_after, added in1.7.0, present only when the candidate ran at a different n thann),inference(the per-size one-sided Mann-Whitney U test —test/alternative/alpha/per_size[].\{size,n_before,n_after,u,p_value, rank_biserial,size_after\}(size_after,1.7.0, present under the same condition asspeedup’sn_after— noten_before/n_afterhere are SAMPLE COUNTS, not sizes;size_afteris the candidate’s n)/sizes_rejecting/sizes_total/sizes_below_floor[].{size,before_ms,after_ms}(added in1.6.0; ASYMMETRIC as of1.7.0— excluded only on the BASELINE’s failure, never the candidate’s)/decision_basis—_accept_decisionrequires a MAJORITY of sizes to reject beforeacceptedcan be true),min_speedup,language, andgrade/grade_basis/grade_rules_version. A measurement failure (baseline or candidate timing failed — e.g. a timeout) is{ok: false, error, code}with noacceptedkey and already matchedrejectedbefore this bump; it is deliberately NOT re-modeled here, so the two branches cannot both match one result.edge_case_comparisoniscompare_edge_cases’s success result:ok,inputs,languages,divergence_count,results[](a per-input matrix,{input, runs: {language: {ok, stdout, verdict, stderr}}}), anddivergences[]({input, languages, kind: content|order, runs}). Its own refusal ({ok: false, error: "provide at least one ... snippet"}) already matchedrejectedand needed no new branch.
accepted and divergence_count appear on no other shape, and
ok and contract_version are required on all eight. A 1.4.0 client is
unaffected: nothing about the five execution shapes, the error taxonomy, or
the verdict enum moved — this bump only widens what the schema accepts, by
naming three shapes it was always silently serving.
1.4.0 is a MINOR bump over 1.3.0, and carries THREE additive changes
landed against the same 1.3.0 base — one bump, not three:
- It ADDS
artifacts_createdto a session-scoped result —session_runandexecute_code(session_id=...)— naming the files that run just created or modified in the session workspace, as{path, size, mime, resource}per file (resourceis acodecalc://session/{sid}/files/{path}URI, also readable viasession_read_file).session_runadditionally addstruncated_inline, settruewhen more artifacts exist than the reply’s inline-content budget (8 blocks / 4 MiB) could attach — the full list still appears inartifacts_createdeither way. The MCP content blockssession_runattaches alongside its JSON result (ImageContentfor a small image,EmbeddedResourcefor a small text file,ResourceLinkotherwise) are a TRANSPORT-level addition, not a result-v1 field — this document and the schema describe the JSON result only. - It is the wire gaining
outputSchemaandstructuredContent, described in the note below — a client that only read thecontenttext block is unaffected. - It ADDS an optional
dependenciesfield to the execution envelope and to the session result shape (per-run dependencies forexecute_code/session_run) —[{spec, language, ok, installer, elapsed_ms, unenforced?}], present only when a run declared dependencies via a PEP 723 block or thedependenciestool argument.
unenforced. None moves or removes what an existing field means, and none
is present on a result that never triggers it, so a 1.3.0 client is
unaffected by any of the three — additions only, hence MINOR.
1.3.0 is a MINOR bump over 1.2.0: it ADDS a tier field to every entry in
the doctor diagnostic document’s runtimes array, plus a tier_summary
block alongside the existing runtime_summary. tier is a
RELIABILITY claim — how much codecalc’s own CI has verified a language’s
toolchain actually works — orthogonal to the existing status field, which is
a RESOLUTION claim about what this one machine found on PATH. The execution
result shape is unchanged, so a 1.2.0 result client is unaffected; only
the doctor document gained fields, hence MINOR.
1.2.0 is a MINOR bump over 1.1.0: it ADDS a strict_runtime block to the
doctor diagnostic document — the gVisor strict-execution boundary’s measured
prerequisites (Docker present, cgroup v2, the runsc runtime registered, the
executor image present, and under --deep a real startup canary verified out of
band). The execution result shape is unchanged, so a 1.1.0 result client
is unaffected; the doctor schema is versioned by this same contract number
(see below), and gaining a field is an addition, hence MINOR.
1.1.0 was a MINOR bump over 1.0.0: it ADDED a fifth result shape
(run_lifecycle, for the run_submit/run_inspect/run_cancel background-run
tools) and added fields to existing results (an execution receipt with
session_id, and grade metadata). Additions only — a 1.0.0 client keeps
working. See the versioning policy below for why an addition is MINOR.
Every codecalc tool result carries contract_version. This document says what
that number promises, what may change under each part of it, and how a client
written against an earlier server should migrate.
The schema is JSON Schema 2020-12.
That dialect is not arbitrary: MCP 2026-07-28 defaults tool inputSchema and
outputSchema to 2020-12 when no $schema is present, so this document can be
handed to a client as an outputSchema and validated with no translation step.
Updated, 2026-09-07 — outputSchema is now emitted for most tools, and it
is a looser document than this one. Most tools are now annotated
-> dict[str, Any] (previously a bare -> dict, which the SDK cannot
schematise at all — verified, SDK 2.0.0: a bare -> dict return yields
output_schema: None and no structuredContent, while -> dict[str, T]
yields both). That flips a typed tool onto the SDK’s structured-output path,
so its tools/list entry carries an outputSchema and every tools/call
result carries structuredContent alongside the unchanged text block.
Three tools are deliberately left untyped, because their return value is not
always a dict:
session_read_filecan return a rawImageContentblock instead of a dict (as_image=True, or any image file read without it).session_runcan return a list of content blocks ([TextContent, *artifact blocks]) when the run created artifacts.list_languages/list_execution_providersreturnlist[dict], which the SDK schematises fine on its own (measured: identical output schema either way) — not an exception in practice, just not thedict[str, Any]form the other 50 tools (includingtrace_execution, added1.14.0) took.
dict[str, Any] | ImageContent/
dict[str, Any] | list[...] union builds a schema, but the SDK wraps EVERY
Union return in {"result": ...} (measured), which is a wire-shape change
to a path that already works — and for session_run specifically, a caller
whose run produced artifacts would then fail the SDK’s own output
validation (“validation error for DictModel … Input should be a valid
dictionary”) instead of getting a result. So session_read_file and
session_run keep their pre-existing untyped return, outputSchema stays
None for those two, and structuredContent stays unpopulated for them —
nothing else about their behavior changed.
What actually gets published is not this file. The SDK derives
outputSchema mechanically from the Python return-type annotation, and
dict[str, Any] produces the generic, permissive shape Python’s typing gives
it license to produce — measured on execute_code:
required, no enums. This document’s schema
(the twelve shapes, the 21 envelope fields, the verdict/code enumerations) is
still the one worth validating against; it is just not the one a client reads
back from tools/list. Getting the SDK to emit this schema verbatim would
need a TypedDict (or BaseModel) per shape, one of which — run_lifecycle vs
envelope vs rejected — a tool can return conditionally at runtime, and that
is future work, not done here. Until then: outputSchema on the wire tells a
client “this is a JSON object”, which is strictly true and worth having, and
this document tells it what the object actually contains.
One caveat for a client UI: in some MCP clients structuredContent displaces
the content text block’s default display. Nothing is actually lost — the SDK
still emits the text block on every call, tests/_mcp_client.py’s data()
helper still reads it as a fallback, and a client that only speaks content
still works exactly as before.
The twelve shapes
Every result carriesok and contract_version, and a client discriminates
the twelve shapes in this order:
The first six are execution shapes — their discriminators only ever
fire for something that ran code (or explicitly refused to). The next three,
added in
1.5.0, are verification shapes: verify_translation,
verify_optimization and compare_edge_cases were always stamped
contract_version by the same stamp() at the MCP tool boundary — every
tool gets that — but before 1.5.0 their own shapes matched none of the
execution branches and were explicitly scoped OUT of this document’s
“exactly one branch matches” claim. That claim now covers all nine of them:
none of the three verification shapes collides with any execution shape or
with each other (accepted and divergence_count appear on no other
shape), and a verify_optimization/compare_edge_cases FAILURE that
carries no shape-specific key falls through to rejected rather than
double-matching. translation.aggregate() also had a real omission behind
this gap: it never set ok at all, which would have made
translation_verification the one shape in this document without it.
Fixed at the source instead of carved into the schema — ok means “the
tool completed and produced a real answer”, never the verdict, so a
mismatch is ok: true exactly like a pass (see verify_optimization’s own
{"ok": True, "accepted": False, ...} early return for the same
distinction already in the contract) — so all nine of those shapes require
it.
execution_trace, added in 1.14.0, is trace_execution’s own shape —
see the version-history entry above for its fields. It is the one shape here
that would otherwise double-match another (envelope): both carry
verdict/backend, and this schema’s additionalProperties is
intentionally left open (see build_schema’s own docstring), so an
execution_trace result’s extra events key alone would not stop it from
ALSO validating against envelope. Closed the same way compact excludes
backend and rejected excludes verdict: envelope’s own definition
gains not: {required: [events]}, so a plain execute_code result (no
events key) matches envelope alone, and a trace_execution result
(always carries events, even as an empty array) matches execution_trace
alone.
comparison_rows, added in 1.10.0, is compare_execution’s own result
— the identical gap edge_case_comparison closed for its sibling tool in
1.5.0, found the same way: compare_execution matched none of the other
eight branches either, and had no branch of its own to fall back on
(compare_execution has no refusal shape — an empty snippets dict still
returns this shape, with count: 0). Unlike the three verification shapes,
it collides with none of them structurally (count/succeeded/fastest
appear on no other shape) and needed no failure-shape reasoning of its own,
because it has no failure shape: ok is unconditionally true once the
comparison ran, the same as edge_case_comparison.
session_snapshot_result, added in 1.15.0, is session_snapshot’s own
result — the identical “unmodeled success shape” gap comparison_rows and
edge_case_comparison each closed above, found the same way: every tool’s
result is stamped contract_version at the same _coded wrapper, but
session_snapshot matched none of the ten existing branches. It is not an
execution shape at all — no verdict, no backend, no run_id/state — and
collides with none of them: action (required, one of save/restore/
list/delete) appears on no other shape here. A failing call (ok: false)
falls through to rejected like every other tool, so this shape only ever
describes success. See “Examples” below for one real transcript per action.
branch_reachability, added in 1.16.0, is branch_reachability’s own
result — nothing here ran, so it collides with none of the six execution
shapes (no verdict, no backend) and, unlike the three verification
shapes and comparison_rows, its own refusal (an unsupported construct, a
non-python language, more than one top-level function, or no function and
no inputs) IS {ok: false, error, code} with no shape-specific key —
already rejected, needing no branch of its own, the same as compare_ edge_cases’s and verify_optimization’s own refusals. Only its ok: true
success shape needed one: supported/dead_count/reachable_count/
unknown_count together appear on no other shape here.
The run_lifecycle shape (added in 1.1.0) is the reply the background-run
tools return before a run finishes — run_submit’s handle, a poll of a
still-running run_inspect, or run_cancel’s acknowledgement. A terminal
run_inspect does not use it: once the run settles, run_inspect returns the
full envelope (or, on a failed run, the rejected error shape) merged with
the run’s own extras (run_id, state, cleaned, …). Forbidding
verdict/error/backend on this shape is what keeps it from colliding with
those, so exactly one branch matches any stamped run response.
The envelope is the full shape: 21 fields including contract_version, identical
across both backends because scripts/check_parity.py runs both and diffs their
key sets — and, since this release, because tests/test_contract.py does the
same for the compile path, which the parity gate cannot reach (it probes
python3, an interpreted language, so nothing it compares ever compiles).
The session shape omits what does not apply. A warm worker has no per-run
workdir, no fresh platform to report and no compile step, so those fields are
absent rather than filled with plausible nulls. unenforced is required here
and is the field that matters most: a worker cannot apply several ceilings a
fresh sandbox can — no_net enforcement (seccomp where the Linux kernel
supports it, a symbol shim otherwise) is applied at exec and the worker is
long-lived — and it says so,
per call.
artifacts_created (added in 1.4.0) is present on any session-scoped
result — this shape from a stateful python3/node worker, or the envelope
shape from a workspace-only session language — because both write into a
workspace that outlives the call. It lists, as {path, size, mime, resource},
every file the run just created or modified; empty when nothing changed.
Sessionless execute_code (no session_id) never carries it: it runs in a
Rust-owned temp directory deleted the moment the call returns, so there is no
workspace left to report on. session_run — a dedicated tool for running a
workspace entry file, covered above under the run_lifecycle/envelope split —
carries the same field, plus truncated_inline when its own inline-content
budget (8 MCP content blocks / 4 MiB) could not attach every artifact; see
CHANGELOG.md ([Unreleased]/“Inline artifacts”) or the _MAX_INLINE_*
constants in codecalc/server.py for the per-type caps (image/text/link)
that decide what an artifact becomes on the wire — kept out of the tool’s
own docstring (the live tools/list description a client sees) to avoid
diluting it with detail a lexical tool-selector never needs.
The compact shape drops diagnostics to save tokens. It never drops
unenforced or output_error — or, since 1.4.0, artifacts_created and
truncated_inline — which is the difference between the current
implementation and the one that was a defect (#117). Since 1.17.0,
stderr (#323) is present whenever the run did not succeed (ok false, a
nonzero integer exit_code, or verdict other than OK) and absent on a
clean run — the executed-and-failed counterpart to code/error/remedy
on rejected above, which cover the rejected-before-execution case.
An earlier version of this document claimed there were two shapes and that
executor.execute was the single point every execution result passes through.
Both were wrong, and a cross-vendor review found it: compact results, native
streaming and session workers all returned shapes the published schema
rejected. The version is now stamped at the MCP tool boundary, which is the
only place that actually reaches all 53 tools.
Padding the short shapes with nulls so one schema fits everything was considered
and rejected. It would have made “we could not tell you the exit code” and “the
exit code was null because the process was killed” the same value.
Versioning policy
contract_version is semver. What each component is
allowed to change:
Why enum expansion is MAJOR and not MINOR. An earlier draft of this policy called adding an error code or verdict a MINOR change, on the strength of the “treat an unknowncodeasinternal” rule below. That was wrong, and a cross-vendor review caught it.codeandverdictare closed enums in the published schema, so a client validating strictly against1.0.0rejects a result carrying a ninth code before any application logic gets to apply the unknown-code rule. OpenadditionalPropertiesmakes new fields additive; it does nothing for enum members. Either the enums are closed and expansion is breaking, or the schema accepts arbitrary strings and stops catching a typo. This contract chooses closed enums and an honest MAJOR, which also matches the taxonomy’s own design: eight deliberately coarse buckets, not a catalogue that grows. The unknown-code rule still earns its place — it is what lets a1.xclient survive contact with a2.0.0server instead of crashing.
The two rules a conforming client MUST follow
-
Ignore fields it does not recognise. The schema leaves
additionalPropertiesopen for exactly this reason. A client that rejects unknown fields turns every additive change into a breaking one, and the MINOR row above stops being true for it. -
Treat an unrecognised
codeasinternal. This does not make adding a code a MINOR change — see the note above; a strict validator rejects the result first. What it does is let a1.xclient survive contact with a2.0.0server instead of crashing on a value it has never seen. The server already applies the rule to itself:errors.error_result()degrades an unknown code tointernaland says so in the message rather than passing it through.
1.0.0 and keep working
across every 1.x without changes.
_meta is not part of this contract
contract_version and this document describe the shape of a tool’s
result — the dict a tool returns, stamped and validated. Server-side
_meta on tools/list (the anthropic/* policy keys, and MCP Apps’
ui.resourceUri) and on a ui:// resource’s own resources/read response
lives entirely outside it: it is wire metadata about the tool or resource
declaration, not about any result this contract versions, and an MCP client
is already required to ignore an unrecognised _meta key regardless of what
it names. Adding, removing, or changing a _meta key is therefore never a
contract_version bump on its own — see
docs/design/2026-09-08-mcp-apps-verification-views.md for where the two
verification tools’ ui key comes from.
Deprecation, and the compatibility window
Nothing in a MAJOR is removed without notice. A field or enum member being retired is first marked deprecated in this document and in the schema description, and keeps working for at least twelve months from that mark. The compatibility window is twelve months, full stop — the same number for every kind of deprecated thing this contract can carry: a result field, acode enum member, and a verdict enum member. There is no shorter window for
a smaller-looking change (e.g. one field) and no separate release-count clock
running alongside it (e.g. “N minor releases”) — a single duration, so a
deprecation’s expiry date is answerable by reading one mark and adding twelve
months, not by also checking how many MINOR releases happened to ship in
that time. Concretely: mark a member deprecated in this document and the
schema description on day zero; it keeps returning exactly as before through
every MINOR release for at least the next twelve months; only a MAJOR
release on or after that date may remove it.
That window is deliberately the same as
MCP’s own deprecation policy,
which guarantees twelve months minimum. A server that promised its callers less
than the protocol underneath it promises would be the shorter of the two in
practice, so matching it is the only number that means anything.
How a client discovers the version
Readcontract_version off any result. It is stamped at the MCP tool
boundary, which is the only place that reaches all 52 tools — so it is present
on success, on failure, on timeout, on a rejected request, on a compact result
and on a session result alike. executor.execute() stamps it too, for callers
using codecalc as a library rather than over MCP; the stamp uses setdefault,
so the two cannot disagree.
codecalc doctor also prints it, for checking a deployment without making a
tool call.
Error codes
Eight, deliberately coarse: a caller should be able to do something different for each one.remedy travels with the code in the result, so the fix does not live only in
this table.
install_package and update_runtimes(apply=True) gate on a caller
confirmation before doing anything (see codecalc/confirmation.py); a
declined or cancelled confirmation is permission_denied — the same
“elevation gate” this table already describes — and a missing or malformed
confirmation response is validation. Neither is a new code: both reuse the
existing eight-entry taxonomy, in the same \{"ok": false, "code", "error", "remedy"\} shape as every other rejection in this document.
code_inferred
A failing result may carry code_inferred: true. That means the code was
derived by matching the error message, not chosen where the failure was
raised — a weaker claim, marked as such so a caller can tell the difference.
Its absence is the strong claim. scripts/check_claims.py counts inferred codes
and floors the total, so the transitional half is visible and shrinking rather
than quietly permanent.
Truncation, and how much was cut
output_truncated says output was cut. stdout_bytes and stderr_bytes say by
how much: they are the bytes the program produced, before the response cap.
max_output_kb is one combined cap on stdout and stderr together. When the
two together exceed it, each stream keeps up to half the cap, and whatever one
stream leaves unused goes to the other, so a stream that prints a lot cannot
push a short stderr traceback out of the result. A cut stream ends in a marker
that sits outside the cap, one per stream that was cut: \n…[truncated]
(horizontal ellipsis) from the Python backends, the fallback and Piston, and
\n...[truncated] (three dots) from the Rust executor. The two spellings are not
unified, so a client that matches on the marker must accept both.
Without the counts, “printed 9 KiB” and “printed 4 MB” were the same answer, and
there was no way to size a retry. With them, a caller can decide whether raising
max_output_kb is worth it or whether the program is the problem.
WhenFour things to know:output_truncatedis true, the count is a LOWER BOUND, and the two backends can disagree. They differ in what they stop: the native executor lets the program run on under a separate file-size ceiling, while the pure-Python fallback runs it to its exit or its timeout. Measured on the same 200 KiB writer atmax_output_kb=1: native and fallback both report204800on this build. Each is an honest report of what it observed before the program ended or was stopped by its own limit; neither is “what the program would have printed”, and no number available to either backend is. Whenoutput_truncatedis false, the count is exact and both backends agree. An earlier version of this section promised the exact original size unconditionally. That was true for a program that prints everything in one burst and false for one that writes slowly, which is the shape of claim a cross-vendor review exists to catch.
-
nullmeans not measured, never zero. A program that printed nothing reports0.nullappears where nothing ran (a runtime that failed to spawn, an executor that could not create its I/O files), where the output stream could not be read to completion, and where the native backend could not stat its output file. In every one of those the alternative would have been an integer that looked exact and was not. -
They count the program’s output, not the field next to them. On a timeout,
stderrcarries codecalc’s own<killed: exceeded wall-clock timeout>whilestderr_bytesis0, because the program wrote nothing. The message is ours; the count is the child’s. - A program stopped by the file-size ceiling wrote less than it wanted. The count is bytes successfully written. No number available anywhere could say how much more it intended, because that limit is enforced by the kernel against the file rather than by codecalc against the program.
-
The count is sampled when the direct child has exited, not when every
descendant has. A grandchild that inherited the output descriptor can still be
writing, so on the native backend a count taken at that moment is what the file
held then. The fallback reports
nullrather than a number when its reader threads have not finished.
len(stdout).
Backends are not identical, and the contract says where
backend is rust or python. The fallback enforces strictly less, and the
contract exposes exactly where rather than papering over it:
unenforcedlists bounds this host could not apply, by name. An empty array is a positive claim that everything requested was enforced. A non-empty one is the sandbox telling you what it did not do.MLEis native-only. The Rust path infers a memory kill from a signal plus an RSS reading near the cap. The fallback has neither, so an OOM there is reported asRTErather than guessed asMLE.scripts/check_contract.pyre-derives both backends’ verdict vocabularies from source and fails if this asymmetry stops being true.- A process killed abnormally never reports
exit_code: nullon EITHER backend, on any OS — but the VALUE’s convention is the platform’s own, not this contract’s invention: negative (the POSIX signal number) on Linux/macOS, positive (the raw NTSTATUS) on Windows, which has no signals. Both backends agree with EACH OTHER on the same OS; neither backend invents a cross-platform encoding the OS itself does not use. See “A program killed by a signal” below.
Examples
Real transcripts, captured by running the product.workdir, timings and
peak_memory_kb vary per run; everything else is stable.
Success
The program ran and failed
Noteok: false. It reports “ran and exited 0”, not “codecalc worked” — a
program that behaves exactly as intended and exits 3 lands here. Read verdict
and exit_code to tell a failed program from a failed request; a failed
request has a code and no verdict. errors.ensure_code enforces this: a
real RTE/OLE/MLE verdict with no error of its own gets no code at
all, never a message-matched internal.
phase: "compile" instead of
"run" — a compile-then-run language (c, rust, …) whose source does
not compile. code is absent here too, deliberately: retrying the same
request cannot succeed either way, and the fix — a syntax error in this
case — is in the program, not something codecalc got wrong. stderr carries
the compiler’s own diagnostic in full; nothing summarizes or drops it.
Neither backend emits a verdict distinct from an ordinary run’s RTE for
this — a caller tells “failed to compile” from “compiled and then failed”
by reading phase, never by a code or a different verdict.
A program killed by a signal
exit_code is a real, non-null value here — never null — because the
process DID stop, just not via exit(). The SIGN, and which convention
applies, is PER-PLATFORM, not one portable constant:
- POSIX (Linux, macOS): negative — the signal number that killed it.
This is the SAME convention Python’s own
subprocess.Popen.returncodeuses, and both backends now agree on it: the pure-Python fallback always returned-Nfor a signal death (it issubprocess’s ownreturncode, unmodified); the native backend now matches it (exit_code_jsoninexecutor/src/main.rs) rather than reportingnull— which used to be indistinguishable from a runtime that never spawned at all. The exact NUMBER is not portable even across two POSIX hosts: the identical null-pointer dereference traps asSIGSEGV(11, soexit_code: -11) on Linux glibc, but asSIGTRAP(5, soexit_code: -5) on macOS/Apple silicon — measured, not assumed, after a first version of this contract’s own test suite hardcoded-11and failed identically on both backends on macOS. A caller checking for “was this killed by a signal” should testexit_code < 0, never a specific value. - Windows: positive — the NTSTATUS itself. Windows has no signals at
all (
platform/mod.rs’s own doc comment onStepResult::signal: alwaysNonethere); an abnormal termination like an access violation is just a large positive exit code —0xC0000005/3221225477forSTATUS_ACCESS_VIOLATION.exit_code_jsonalready passes this through unchanged on Windows (the-Nbranch only fires whensignalisSome), so nothing needed to change there.
null is reserved for the other case entirely: nothing ran to a stop at
all (a timeout kill, or a runtime that never spawned — see below), on
either OS.
SIGSEGV, hence -11); the identical
null-deref on macOS carries "platform": "darwin" and "exit_code": -5
instead, and on Windows "platform": "win32" and a positive
"exit_code": 3221225477, with everything else about the shape unchanged.
stderr is empty here because a segfault does not itself write anything —
a program that DID print diagnostics before crashing would still have them
in stdout/stderr, same as any other run. verdict is RTE — the same
verdict an ordinary nonzero exit gets — because neither backend infers a
verdict distinct from a signal-caused crash except MLE (a signal near the
memory ceiling; see “Backends are not identical” above). code is absent
for the same reason it is absent from an ordinary RTE above: the process
ran and failed on its own terms, which errors.ensure_code does not
classify as a REQUEST-level failure.
Timeout
exit_code is null because the process was killed rather than exiting, and
stderr says which clock did it. No partial result is returned. Unlike an
ordinary program failure above, a timeout DOES carry code — "timeout",
code_inferred: true — because it is something the caller can act on (raise
timeout), the same actionable shape every other coded failure has;
executor.execute never sets error for this either, so errors. ensure_code classifies it from verdict/timed_out directly rather than
falling through to internal the way it once did.
Spawn failure (a missing runtime)
Still an execution envelope —verdict is present, because the code ran as
far as it could — but exit_code is null for the SAME reason as the
timeout above: the process never produced a real exit status. error names
the phase and the binary that could not be launched; a caller at the MCP
tool boundary (where errors.ensure_code runs) sees this classified code: "runtime_unavailable" from that text.
stderr and error carry the identical text here — the Rust binary’s own
spawn-failure message already says “runtime unavailable”; error exists
so a caller can read it as a stable machine-facing field rather than
parsing stderr prose, not because the two texts need to differ.
A compile-then-run language missing its COMPILER reports the identical
shape with "phase": "compile" instead. stdout_bytes/stderr_bytes are
null rather than 0 on EITHER phase — nothing spawned, so there is no
program output to have counted (see “Truncation, and how much was cut”
above).
Rejected before execution
Noverdict, because nothing ran.
A coded failure from a symbolic tool
Symbolic tools do not produce the execution envelope — nothing was executed — but they carry the same error half, so a caller branches oncode identically.
A compare_execution row that needed a runtime this host does not have
The tool’s OUTER ok is true — the comparison itself ran — but the lua
row failed for a REQUEST-level reason, so it carries code/error/remedy
the same as a top-level coded failure. The python3 row ran and succeeded
and carries none of the three; a row that ran and failed on its OWN terms
(a real RTE/OLE exit_code) would carry none of them either, only
exit_code and timed_out saying so — see errors.stamp_row in
codecalc/errors.py for the full rule. compare_edge_cases’s per-language
runs[lang] entries follow the identical rule.
session_snapshot, one transcript per action
Captured from a real python3 session that wrote one file (model.py),
saved it, restored it into a new session, then deleted the snapshot.
restore without replace=True returns the NEW session’s own session_start
shape, plus what this call itself restored (restored_files, bytes —
uncompressed bytes actually written, not the archive’s own on-disk size):
snapshot_id, a workspace whose identity could
not be verified, a hostile archive on restore — matches rejected instead:
Migration from a pre-1.0.0 server
Servers before this release emitted no contract_version. Absence of the
field means 0.x — a contract that made no promises. Three changes matter:
No field was removed and no field changed type, so a
0.x client keeps
working on 1.0.0 — it simply cannot see the new information. The migration
is opt-in, and the only thing that is genuinely retired is the practice of
matching on error prose, which was never stable enough to be a contract in the
first place.
The request half: a canonical spec and its content hash
Everything above describes what comes back. This section describes what a request is, so that two parties can agree on a name for one without shipping the whole object around.providers.ComputationSpec is the canonical, transport-neutral request. Its
identity is a byte encoding plus a digest over it:
The schema describes the canonical document, not the dataclass:
sha256 the bytes. The result is prefixed with its algorithm:
The canonicalization rules
They are stated here because a second implementation has to reproduce them, and every one of them is a place where two implementations could silently disagree.- JSON, UTF-8, sorted keys,
(",", ":")separators. This is RFC 8785 for the value space a spec can hold. JCS orders keys by UTF-16 code unit and this implementation by code point; the two can only differ above the BMP, and every key is a dataclass field name. - Every field is emitted, defaults included. There is no “absent”. A spec built with defaults and one with the same values passed explicitly produce identical bytes, and the bytes reconstruct the request without consulting a default table.
nullis a value.workdir: null(provider chooses) andworkdir: ""(an empty path) are different requests with different hashes.- Booleans are not integers.
no_net: trueandno_net: 1do not collide. - Array order is kept; object key order is not. A sequence’s order was asked for; a mapping’s is an artifact of construction. Tuples and lists encode identically — JSON has one array type.
bytesencode as{"__bytes_b64__": "<base64>"}. A bare base64 string would collide with a text field holding the same characters, so the tag is reserved and a mapping may not use it as a key.- Floats are refused. Shortest-round-trip IEEE-754 printing is where canonicalizers diverge between runtimes, and NaN/Infinity are not JSON.
- Anything with no rule is refused — sets above all, which have no stable iteration order.
language: "python", "py" and "python3" all execute on python3, but they are
three different requests and get three different spec_hashes — the hash names
the request as given, and does not claim two spellings are the same computation.
Normalizing an alias before hashing would collapse distinct requests onto one
identity; this contract keeps them distinct. (max_output_kb: 0 is likewise
literal-not-magic: it selects the backend’s 64 KiB default, not an uncapped
stream.)
Secrets are never part of an identity
Canonical bytes are hashed, logged beside receipts and compared across processes, so a secret in them is a secret in all three.ComputationSpec
carries no credential-bearing field today — language, code, stdin,
timeout, workdir, max_memory_mb, max_output_kb, max_cpu, no_net — and
the encoder keeps it that way.
A field name is tokenised — split on separators and camelCase boundaries —
and refused if any token is in spec_identity.SECRET_BEARING_WORDS. Tokenising
rather than exact-matching is load-bearing: db_password, auth_token and
client_secret are what a real codebase calls these things, and an exact-match
rule that only knows password guards a name nobody uses. Substring matching was
rejected for the opposite reason — it fires on author. The same words are
applied to mapping keys one level deeper, since a field policy cannot see
inside a dict.
Two escape hatches, both declared on the dataclass:
They are separate on purpose: reaching for the first to silence a false positive
would silently drop a real value out of the request’s identity.
key is in the word set deliberately and is the aggressive entry — it catches
private_key and signing_key, and also sort_key. A false positive costs one
declared line; a false negative puts a private key inside a hash that gets
logged.
Changing a credential’s value does not change the request’s identity; adding or
removing one does.
Versioning policy for COMPUTATION_SPEC_VERSION
Independent of CONTRACT_VERSION: they version different documents and are free
to move apart.
Read this before you store a hash. Rule 2 means MINOR is additive for the
schema but not hash-preserving: adding a field changes every canonical
document, and therefore every hash. That is the price of bytes that stand alone,
and it is stated rather than papered over. A consumer that persists spec hashes
must persist
schema_version beside them and treat hashes from different
versions as incomparable rather than unequal.
Changing the request contract
- Edit
providers.ComputationSpecand its entry inCOMPUTATION_SPEC_FIELD_DOCS— the schema generator refuses to publish an undocumented field. python scripts/check_contract.py --writeregeneratescomputation-spec-v1.schema.json.- Bump
COMPUTATION_SPEC_VERSIONper the table above and say so here. - Hand-edit
computation-spec-v1.vectors.json. Nothing regenerates it, on purpose: a vector its own generator rewrites cannot detect a change to the encoding, which is the one failure a schema check cannot see — the field set never moves, and every previously published hash silently means something else.
Changing this contract
- Edit
codecalc/contract.py. It is the single source. - Run
python scripts/check_contract.py --writeto regenerate the schema. - Bump
CONTRACT_VERSIONper the table above, and say so here. python scripts/check_contract.pymust pass. It verifies the committed schema matches the module, that thecodeenum is exactlyerrors.ALL_CODES, that theverdictenum is the union of what both backends emit (re-derived frommain.rsandexecutor.py, not transcribed), and that this document names the current version.