packages/server/eval/perf on a GitHub-hosted
ubuntu-latest runner, committed to the repository as a baseline, and gated on each subsequent
push. Nothing here was measured by hand.
What makes these reproducible rather than merely publishedThe harness refuses to record a run whose own calibration channels look contended. That refusal
is the feature: a benchmark that always produces a number produces a number even when the machine
was busy. The gate, the thresholds and the raw sample series are all in the repository, so a third
party can re-run the same workflow and compare like for like.
Method
-
Fresh subprocess per sample, median aggregation (
eval/perf/isolate.ts). Timing a loop inside a process that already ran it measures a warm JIT, not the operation. -
5 samples per key, recorded in
isolated-medianmode. -
Two calibration channels run alongside the measurement and decide whether it counts at all
(
eval/perf/contention.ts):The I/O bound is deliberately looser than CPU’s: a quiet reference host was itself measured at io CV 0.229, so a 0.2 bound would refuse clean recordings. -
Recorded at
9d1b2fbf, 2026-08-09, onlinux / x64 / 4 CPUs.
What the runner actually achieved
Both recordings used well under a sixth of the variance they were allowed.
Scenarios
Both scenarios build the same synthetic corpus from a fixed seed (0x5eed): 100 notes → 200
chunks, 20 duplicate-body groups, link fan-out 3. densify differs in exactly one axis —
derived-edge densification (tagEdges + knnEdges) is ON. Holding the corpus identical is what
makes the pair a clean A/B on densification cost.
small — the default path
A warm HTTP request is 22.8× faster than a cold one, which is the connection-reuse path doing
its job rather than a caching claim about retrieval.
densify — what derived edges cost
Same corpus, densification on. It produces 865 shared-tag edges and 533 kNN edges — 1,398 edges
across 200 chunks, ~7.0 per chunk — and the cost lands on ingest:
What these numbers are not
- Not a competitive comparison. No other Obsidian MCP server has been run through this harness.
- Not retrieval quality evidence. The corpus is synthetic and 100 notes. Retrieval quality is measured separately, on a public third-party corpus, in EVALUATION.md.
- Not a merge gate. The
perfCI job runsif: github.event_name == 'push'— post-merge onmainonly, never on a PR, and deliberately not among the required checks. It is a trend signal. - Not a large-vault claim. These are 200-chunk scenarios. Scale scenarios exist in the harness
(
vault1k,medium,large,vault100k) but are not part of the committed baseline.
Reproducing this
packages/server/eval/perf/README.md documents the recording
and re-baselining procedure.
A baseline is a recording, not a config file. Hand-editing a value fails the provenance coherence
check: the sidecar snapshots every exact-class key at record time, so an edited key means the timing
keys beside it were measured against a different build.