src/web/fetchTier.ts and its surroundings, all about how
the server behaves as a runtime rather than what it fetches: extraction
runs off the event loop, a fast markdown hop skips extraction entirely on
sites that offer it, the UA identifies this server honestly instead of
impersonating a browser, and /mcp over HTTP gets DNS-rebinding guards
plus a per-client rate limit.
Extraction off the event loop
Tier 1’sparseHTML + Defuddle call used to run inline, on the same
thread that answers every other concurrent MCP request (including a plain
GET /health). Both are synchronous, CPU-bound DOM work; on a large page
this is multiple seconds, and for that whole span nothing else this
process is doing could make progress.
src/web/extractWorker.ts now holds the actual algorithm
(runExtraction(html, url)), and src/web/extract.ts runs it on one
lazily-started node:worker_threads worker instead of the main thread.
One worker, not one per call: extraction isn’t frequent enough to justify
a fresh thread’s startup cost each time, and a single worker gives a
simple place to hang a 30s per-job timeout and crash recovery - a job that
blows past the timeout, or a worker that crashes ('error'/'exit'),
tears the worker down and rejects whatever was still pending against it;
the next extractHtml() call lazily starts a replacement. If a Worker
can’t even be constructed (worker_threads unavailable in this
environment, as opposed to a transient crash), extractHtml() falls back
to calling runExtraction() directly, in-thread - same algorithm, same
result shape, just back on the main thread.
extractWorker.ts doubles as both the worker entry point and the
in-thread fallback’s implementation: it checks parentPort at module load
to decide whether to wire up a message handler, so the exact same file
works loaded as a worker (new Worker(...)) or imported as a plain module
(extract.ts’s fallback path). The worker script’s own file is located
relative to import.meta.url, matching the loading module’s extension
(.ts under native execution, .js after npm run build) rather than a
hardcoded one - this is what lets new Worker(...) resolve correctly
whether the process is running from src/ or from dist/.
Measured on the same two fixtures the pre-task Rust spike used
(pg1342.htm, Pride and Prejudice, 0.8 MB; a Byzantine Empire Wikipedia
mirror, 1.6 MB): run in-thread, extracting either page fully blocks the
event loop for the whole extraction - a concurrent /health request
simply cannot be answered until the extraction finishes (in one
measurement, zero of ~30 attempted /health pings landed during a 4.2s
extraction; the only one that got through was the one issued right after
it completed). Run through the worker, the same extractions kept /health
answering in single-digit milliseconds through most of the run, with
occasional spikes into the tens of milliseconds on this shared,
noisy 4-core box - see the task 13 report for the full numbers and an
honest read on the handful of samples that briefly touched ~50-70ms under
load.
Markdown for Agents
Cloudflare’s Markdown for Agents (seeresearch/cloudflare-platform.md
section 4): a participating zone answers Accept: text/markdown with
content-type: text/markdown and an x-markdown-tokens header instead of
its ordinary HTML. Tier 1 now sends Accept: text/markdown, text/html;q=0.9; when the response actually comes back as
text/markdown, that body is used directly as text (via: 'markdown'),
skipping Defuddle (and the worker hop above) entirely - there’s no DOM to
build and nothing left to strip. A zone that doesn’t participate just
answers its normal HTML at the q=0.9 fallback, exactly as before.
Honest User-Agent
The default UA sent by tier 1 changed from an impersonatedChrome/131.0.0.0 desktop string to Alexandria/\<version> (+https://github.com/The-40-Thieves/alexandria-mcp) - identifying,
versioned, and pointing back at the project, rather than pretending to be
a browser. Overridable per deployment via ALEXANDRIA_FETCH_UA.
Final wave (F1): that is now the UA for EVERY outbound call, not just tier
1’s. src/utils/http.ts - the path every REST adapter takes - used to send
library-mcp-server/1.0 (open source research tool), and four adapters
(wikipedia, wikidata, kev, ecosystems) hardcoded alexandria-mcp/10, frozen
at whatever major each was written on. All of them build from
src/utils/userAgent.ts now; the four that upstream asks to carry a contact
address use contactUserAgent(), which folds CONTACT_EMAIL in when set.
npm run probe was run once against every source before this change and
once after (see the task 13 report for the full before/after status
table). No source’s probe status regressed from the UA change alone; a
site that specifically depended on the browser UA would show up as a new
ERROR/EMPTY/TIMEOUT in that diff, and none did in this run. If a
future probe run does turn up a regression traceable to the UA, the fix is
a per-adapter headers override restoring the old browser UA for that one
call site (src/utils/userAgent.ts’s fetchUserAgent() reads
ALEXANDRIA_FETCH_UA first, so a global override is also available without
patching source), not reverting the default for everyone.
Residual limits of the SSRF guard (final wave)
Delegated renderers cannot be pinned. Tiers 2, 3 and 4 (jina, crawl4ai, Browser Run) do not fetch the target themselves; they hand the URL to a third party that resolves the hostname on its own machine.assertFetchableUrl still runs before each of them, so a target whose name
or whose one DNS answer is private is refused - but that check validates
Alexandria’s DNS answer, and nothing about it constrains the connection
the delegate then makes. A hostname that resolves publicly for Alexandria
and privately for the delegate reaches whatever the delegate can reach.
This is unfixable from here for jina and Browser Run (hosted, no way to
pass a validated address with the Host/SNI preserved). It matters most for
crawl4ai, because crawl4ai runs on the operator’s own tailnet: a
successful validation here does not stop it fetching a Cave-internal
service. Tier 1 (defuddle, the tier that fetches directly) has no such gap
- it pins every hop to the address
resolveFetchTargetjust validated.
worker_threads, extraction has no timeout. extractHtml and
extractPdfOffThread fall back to running the same extraction in-thread
when a Worker cannot be constructed at all. The 30 second per-job
timeout goes with the worker: it works by terminating that worker, and
there is nothing to terminate on the main thread. parseHTML + Defuddle
and unpdf’s PDF.js are both synchronous, so an in-thread timer could not
interrupt them even if one were set. On this fallback path a pathological
page or PDF blocks the event loop for as long as it takes. Every supported
runtime (Node 24, the engines floor) has worker_threads, so this is the
fallback’s documented cost, not the normal path.
HTTP guards (src/httpGuards.ts, TRANSPORT=http only)
Applied to /mcp only - /health and /metrics carry no session state
worth protecting the same way:
- Host/Origin validation (
checkOrigin): DNS-rebinding protection via@modelcontextprotocol/node’shostHeaderValidation/originValidation, checked againstALEXANDRIA_ALLOWED_ORIGINS(a comma-separated list of hostnames, no scheme or port). Loopback (localhost,127.0.0.1,[::1]) is always allowed, regardless of that setting. A request that fails either check gets a403(the guard itself writes that response, per@modelcontextprotocol/node’s own contract). The Host check applies only whenALEXANDRIA_ALLOWED_ORIGINSis set: applied unconditionally it403s every request to any deployment that has not set the variable, whose own hostname is in no allowlist because there is no allowlist. Setting the variable is what turns DNS-rebinding protection on, and one warning is logged at startup while it is unset. The check is deliberately NOT keyed on which interface the connection arrived on: a Cloudflare Tunnel terminates atlocalhost:PORTand forwards the public hostname inHost, so keying on loopback403ed the tunnelled request while letting the identicalHostthrough on a LAN address. The Origin check is unconditional (the SDK’soriginValidationpasses a request carrying noOriginheader at all, which is every non-browser MCP client). - Per-client-IP rate limit (
checkRateLimit): a token bucket keyed onreq.socket.remoteAddress(or, withALEXANDRIA_TRUSTED_PROXY=1,CF-Connecting-IPthen the rightmostX-Forwarded-Forentry - the one the last hop appended; the leftmost is whatever the original caller sent), capacityALEXANDRIA_HTTP_RATE_LIMIT(default 60), refilling continuously back toward that cap over a one-minute window. Starts full, so a client’s first burst up to the cap is never rejected - only sustained traffic above the configured per-minute rate is. Exceeding it returns429with a JSON-RPC error body (\{"jsonrpc":"2.0","error":\{"code":-32000,"message":"rate limit exceeded"\},"id":null\}), matching the shape of every other/mcprejection instead of a bare HTTP status page.
createHttpApp()’s /mcp branch, before the request
ever reaches handleMcpRequest/the MCP transport.