Skip to main content
npm run eval:answer scores library_answer (src/tools/libraryAnswer.ts) against the hand-written golden set at eval/answer-golden.yaml (20 questions spanning 16 of the registry’s clusters, each with 3-6 expected_nuggets - short, independently checkable factual statements - and expected_sources, used only to group the per-cluster table, the same role expected plays in eval/routing-golden.yaml). It needs both a synth and an embeddings role configured (library_answer itself requires synth; embeddings materially improves the routing that feeds it); like eval:routing, it prints a message and exits 0 when either is missing, rather than failing on a key this deployment simply doesn’t have. Three metrics, per question and averaged overall and per cluster:
  • citation precision - of the answer’s sentences that carry a citation marker, what fraction are actually warranted by the text of their cited source(s) - not just on-topic, but the same strength, scope, and time (the warrant-strength rubric from research/retrieval-sota.md section 4)? Judged with one chatJSON call per cited sentence, via a role read from the CLAIM_SUPPORT_ROLE constant in scripts/ eval-answer.ts (today: synth; Task 9 adds a dedicated verify role, and that constant is the one line that needs to change). A sentence whose cited source couldn’t be re-read at all is excluded from the denominator (nothing to judge it against), not counted as unwarranted.
  • nugget recall (the brief’s “citation recall” and “answer nugget recall” are the same computation under two names - see the module comment in scripts/eval-answer.ts) - of the golden question’s expected_nuggets, how many are covered by at least one of the answer’s cited sentences? Uncited prose doesn’t count. Judged the same warrant-strength way, with the nugget as the claim and the joined cited sentences as the evidence.
  • resolvability - of the citations that carry a URL, or an id shaped like a DOI, what fraction return a 2xx on a guarded HEAD (falling back to GET) through src/web/fetchTier.ts’s assertFetchableUrl, with a 5s timeout? Capped at 20 checks per question. A question with no URL/DOI-bearing citations at all contributes nothing to this average (see “resolvability is undefined, not zero” below) rather than a misleading 0.
Exit code: unlike eval:routing, this never fails on the scores themselves - this run is the first baseline, so there’s no floor yet to regress against. --gate instead fails (exit 1) only on an infrastructure problem: the required roles missing, or a golden question’s library_answer call throwing outright - so a CI environment that expects this eval to actually run can catch “it silently no-op’d” or “it crashed”, without asserting a numeric floor nobody has earned yet. Plain npm run eval:answer always exits 0. To be explicit: --gate has no score floor yet - it does not fail on a low citation precision, nugget recall, or resolvability number, only on the infrastructure problems above; a numeric floor is a follow-up once enough baseline runs exist to set one responsibly (the same way eval:routing’s 0.60/0.80 floors were set from an established run, not invented up front).

2026-09-03, first baseline (Cave LiteLLM gateway)

Models: synth role openai/gpt-4o-mini and router role openai/gpt-4o-mini (both via the gateway, with ALEXANDRIA_SYNTH_JSON_MODE=1 / ALEXANDRIA_ROUTER_JSON_MODE=1 - the gateway’s hostname isn’t api.openai.com, so chatJSON won’t request response_format: json_object on its own). embeddings role BAAI/bge-m3. The task brief named only ALEXANDRIA_SYNTH_MODEL for this run; ALEXANDRIA_ROUTER_MODEL was set the same way for the same reason docs/routing-eval.md’s Task 6 run set it - library_answer calls runAsk() -> planRoute() internally, so this eval exercises the router role too, and its unset default (gpt-4o-mini, no provider prefix) isn’t a route this gateway recognizes. Credential handling: LITELLM_AGENT_KEY was read from /data/llm-stack/.env with grep '^LITELLM_AGENT_KEY=' | cut -d= -f2- into a shell variable, passed only as ALEXANDRIA_API_KEY to the eval process, and never echoed, logged, or committed - only the variable name appears here. OPENAI_API_KEY was explicitly unset (env -u) for the run so no local key could shadow the intended gateway credential. Method: ALEXANDRIA_STATE_DB=:memory: (a fresh store; no routing cache carries over from any prior run). Two full runs were made against the golden set; the second (below) is the one recorded, since the mean() fix described under “resolvability is undefined, not zero” landed between the two.
developer shows n=0: both of its golden questions (the JWT/RFC 7519 question and the PEP question) hit the timeout below rather than completing - academic’s 1 (of 1) reflects the AlphaFold question completing while the Transformer question in the same cluster timed out.

Two questions timed out

Two of the 20 golden questions failed outright with Request timed out. Node.js fetch timed out waiting for response headers; configure a matching undici fetch and fetchOptions.dispatcher with an Agent whose headersTimeout is at least the SDK timeout - an error surfaced by the openai SDK’s own request path, not this script. A first run (discarded, made before the mean() fix below) saw three timeouts, overlapping on two of these same questions plus a third; neither run reproduced a consistent failing question, and src/utils/dispatcher.ts has no known reproducible hang (see docs/routing-eval.md’s “Gateway host gotcha (did not reproduce, final wave)” section, which investigated the same failure shape against the same gateway and found nothing to fix in this codebase). Treated the same way here: not chased as a bug in this task, since the harness’s own job - catching the per-question exception, logging it, and continuing to score the other 18 - worked exactly as intended (scripts/eval-answer.ts’s main() loop). A CI run with --gate would fail on this (2 of 20 golden questions didn’t run), which is the intended signal for “the harness didn’t actually complete,” independent of the scores.

Resolvability is undefined, not zero

Every one of the 18 completed questions shows resolvability: NaN - skipped resolvability (no url/DOI): 18 out of 18. This is not a 0% pass rate; it means none of the citations in this run carried a URL or a DOI-shaped id to check at all. Confirmed directly:
No url field. buildCitations() in src/tools/libraryAnswer.ts maps url: s.item.url - LibraryResult.url, not LibraryResult.previewUrl. Across src/sources/*.ts, 40 adapters set url: on their normalized result and 85 set previewUrl: instead (grep -rl '^\s*url:' src/sources/ *.ts | wc -l vs the previewUrl: equivalent); every source this golden set’s questions actually cite in these two runs - coingecko, wikipedia, arxiv, ietf, gutenberg, guardian, and others reached via hasFullText: true sources - is in the previewUrl-only group. Separately, none of the id values seen (bitcoin, arXiv ids, Wikipedia page titles, RFC numbers, …) are DOI-shaped, and the sources whose id is a DOI (crossref, datacite, opencitations, …) are metadata-only (hasFullText: false), so readTopSources() never selects them to be read and cited by library_answer in the first place. This is a real, reportable structural gap in the current Citation shape, not a bug in this eval script: resolvability, as specified, has almost no citations to measure against until Citation.url is populated more broadly or Task 9’s Citation.resolves wiring (task-9-brief.md item 2, liveness.ts) lands. scripts/eval-answer.ts’s mean() reports NaN rather than 0 for this exact reason - an empty judged set must not read as “0% resolved” - so a future run that fixes this gap will show a real number here, not a false-zero silently improving.

Reading the other two numbers

Citation precision, 0.5556 overall (10 judged sentences across 9 questions with at least one judgeable citation; 9 of 18 completed questions had none - skipped precision (no judgeable citation): 9) - a small-n, one-model number from the very first run of this harness, not a number to alarm on. It’s the number Task 9’s warrant-strength verify role work (task-9-brief.md item 1, claimCheck.ts) will be measured against before/after. Nugget recall, 0.1944 overall - low, and worth reading carefully before treating it as an answer-quality problem: library_answer’s readTop defaults to 4 sources, maxSources to 6, so a synthesized answer realistically covers a handful of an expected nugget’s angles, not every one of the golden set’s 3-6 independently-checkable facts per question. Nugget recall only credits a nugget covered by a cited sentence - an answer that states a fact from general knowledge without a source number attached to it (this run had several) scores that nugget as not covered, which is the intended, stricter reading of “citation recall” the brief specifies (see the module comment in scripts/eval-answer.ts), not a looser “did the answer mention this” recall.

To re-run

Without any roles configured, npm run eval:answer prints a short message and exits 0 (or 1 with --gate) instead of making any network calls.

2026-09-03, Task 9: claim verification, citation liveness, and grades

Task 9 (src/utils/claimCheck.ts, src/utils/liveness.ts, src/utils/citationGrade.ts) added a dedicated verify role for the citation-precision judgment above (CLAIM_SUPPORT_ROLE in scripts/eval-answer.ts flipped from synth to verify - verify falls back to synth’s own config when ALEXANDRIA_VERIFY_* is unset, so this run used the same model as before), fixed buildCitations() in src/tools/libraryAnswer.ts to fall back through previewUrl then downloadUrl when a source has no url (the structural gap the first baseline above found), and wired Citation.resolves through the new src/utils/liveness.ts (checkLiveness, moved out of scripts/eval-answer.ts’s own checkResolvable, which now just calls it). Before: the 2026-09-03 first-baseline run above - citation_precision=0.5556 nugget_recall=0.1944 resolvability=NaN (18 of 20 questions completed; Citation.url populated by 40 of 125 adapters, Citation.resolves didn’t exist yet). After (same gateway, same models, ALEXANDRIA_STATE_DB=:memory:, OPENAI_API_KEY unset so no local key could shadow the gateway credential; LITELLM_AGENT_KEY read from /data/llm-stack/.env via grep '^LITELLM_AGENT_KEY=' | cut -d= -f2- into a shell variable, passed only as ALEXANDRIA_API_KEY, never echoed/logged/committed):
Only 14 of 20 questions completed - 6 hit the same gateway-side “Request timed out… waiting for response headers” hiccup documented in the first baseline above and in docs/routing-eval.md’s “Gateway host gotcha” section, and 6 of them were among the 8 that finished sooner in the first run; this is the same unresolved, previously-investigated gateway issue, not a Task 9 regression - re-running showed a different subset of questions timing out each time, and the harness’s own job (catch, log, keep scoring the rest, never silently drop a question) worked as intended. Because a different, smaller subset of questions completed than the first baseline’s 18, citation precision (0.5556 -> 0.5833) and nugget recall (0.1944 -> 0.1607) are not a clean before/after comparison on the same question set - both stayed in the same small-n, noisy range, not a demonstrated regression or improvement from the verify role itself. Resolvability is the number this task set out to fix, and it moved from having nothing to measure to a real, mostly-passing number: resolvability=1.0000 over 5 checkable citations (9 of 14 completed questions still had no citation carrying a url or DOI-shaped id - markets/news_global/others cite sources whose id/text genuinely has neither). Those 5 came from literature, real_estate, science, standards, and video - each resolved live, confirming both halves of the fix work end to end: buildCitations()’s previewUrl/downloadUrl fallback actually produces a checkable URL for these previously-URL-less citations, and checkLiveness()’s guarded, pinned HEAD/GET (through src/utils/liveness.ts, cached 24h in the state store under live|<url>) correctly reaches and confirms them.

To re-run (with claim verification and grading)

Same env as “To re-run” above; claim verification is on by default (set ALEXANDRIA_CLAIM_CHECK=off to disable it) and uses the verify role, which needs no separate configuration - it falls back to whatever synth resolves to. To point verify at a different model, set ALEXANDRIA_VERIFY_MODEL (and _BASE_URL/_API_KEY if it’s a different gateway) alongside the vars above.

2026-09-03, Task 10: cross-encoder rerank backends and margin-gated multi-query

src/utils/rerank.ts (new) replaces fuse.ts’s llmRerank with a single rerank(query, candidates, { backend, top }) entry point behind ALEXANDRIA_RERANK: off | llm | cohere | workers-ai. library_answer hands it the RRF-fused pool capped to ALEXANDRIA_RERANK_POOL (default 60, was a fixed 40); the llm backend further caps its own listwise input to the top 20 of that pool, shuffled before the prompt (position-bias mitigation per research/retrieval-sota.md section 1). cohere POSTs <rerank base>/rerank in the Cohere/Jina/Voyage/LiteLLM-shared request shape; workers-ai POSTs Cloudflare’s {query, contexts} shape straight to ALEXANDRIA_RERANK_BASE_URL (already the full per-account model run URL for that backend). Both are covered by stubbed-endpoint tests in src/utils/rerank.test.ts, not by a live run here - see below.

Cohere-shape backend: not measured live

The brief asked for one live run with ALEXANDRIA_RERANK=cohere against the Cave LiteLLM gateway’s /rerank, fronting nvidia_nim/nvidia/llama-3_2-nv-rerankqa-1b-v2. Before wiring anything, that exact model was probed directly:
Both paths reach LiteLLM’s rerank handler correctly (it parses the request and reports “Received Model Group=…” before forwarding) - confirming the request shape itself is accepted at either /rerank or /v1/rerank - but the specific NVIDIA NIM-hosted reranker model is permanently gone upstream, not a network or routing problem this repo can work around. The gateway’s other two rerank-named models fail closed too:
No rerank endpoint on the configured gateway is reachable, so the cohere backend’s effect on citation precision/nugget recall/ resolvability is not measured in this environment. Running eval:answer with ALEXANDRIA_RERANK=cohere anyway would be scientifically meaningless here: every call would hit cohereRerank()’s catch block (confirmed by src/utils/rerank.test.ts’s stubbed-server tests) and fall back to the plain fused order, producing numbers indistinguishable from the off baseline below modulo ordinary run-to-run LLM/gateway noise - not a test of reranking quality at all. Workers AI cannot be probed here either (no Cloudflare account/token in this environment); its {query, contexts} request/response shape is implemented against the documented API and exercised only by rerank.test.ts’s stub.

Margin-gated multi-query: measured

ALEXANDRIA_MULTI_QUERY=1 only touches which sources library_ask’s stage 2 sees (docs/routing-eval.md’s Task 10 section has the routing-level numbers); its effect on library_answer’s own citation quality flows through indirectly, via which sources get searched, read, and cited. Two runs, same gateway/models/env as “To re-run” above (ALEXANDRIA_STATE_DB=:memory:, ALEXANDRIA_RERANK unset/off in both):
As with Task 9’s before/after, this is not a clean same-question-set comparison: 2 questions timed out in the off run and a different 1 timed out in the on run (the same gateway-side “Request timed out… waiting for response headers” hiccup docs/routing-eval.md’s “Gateway host gotcha” section already documents - not chased as a bug here), so 18 and 19 questions respectively were actually scored, not the same 18. The precision jump (0.5000 -> 0.7143) is consistent with multi-query surfacing better sources for the previously-hardest routing cases, but at n=18-19 with a different question set each time it is not strong evidence on its own - nugget recall (0.1944 -> 0.1974) and resolvability (0.8125 -> 0.8095) barely moved, which is the more representative signal at this sample size. Combined with the LLM-cost finding in docs/routing-eval.md (2.45x router-role calls on already-ambiguous queries), this is consistent with “does no harm, plausibly helps, costs more” rather than a confirmed answer-quality win - hence still shipping default-off.

To re-run

Same env as “To re-run (with claim verification and grading)” above, plus ALEXANDRIA_RERANK=cohere|workers-ai|llm (pointed at a reachable /rerank-shaped endpoint for cohere, or a Workers AI model run URL for workers-ai) and/or ALEXANDRIA_MULTI_QUERY=1.

2026-09-03, Task 11: objective-driven research stopping (before/after)

library_research did not have a library_answer-shaped question in the golden set to run through eval:answer (the harness scores library_answer citations, not a multi-round research loop), so this task’s brief calls for one live library_research run before and after the change instead, recording rounds used, elapsed seconds, and objectives covered. Before (this repo’s e2874b6, pre-Task 11: no outline/coverage step, stops only on depth, breadth, max_minutes, or a round with no new sources) and after (this branch): the same fixed topic, \{ depth: 3, breadth: 4, maxMinutes: 5 \}, against the Cave gateway (http://100.78.123.100:4001/v1, ALEXANDRIA_API_KEY read from /data/llm-stack/.env’s LITELLM_AGENT_KEY via grep | cut, never echoed/logged/committed), ALEXANDRIA_EMBEDDINGS_MODEL=BAAI/bge-m3, and ALEXANDRIA_ROUTER_MODEL=ALEXANDRIA_SYNTH_MODEL=ALEXANDRIA_RESEARCH_MODEL=openai/gpt-4o-mini (_JSON_MODE=1 on all three) - the same gateway/model settings docs/answer-eval.md’s first baseline above used, extended with a research-role model since this run exercises that role directly (the answer eval harness never does). Topic: “What retrieval techniques improve citation grounding in LLM-based research agents?”
  • Both runs used all 3 rounds, for different reasons. The after run’s round 3 found 0 new sources (newSources=0), so the pre-existing no-new-sources stop condition is what actually ended it here - not the new objective-coverage check, which had not yet reached 7/7 (one objective, “user feedback and interaction patterns… concerning citation grounding accuracy”, was never covered by a learning on this topic, plausibly because none of the sources found actually discuss user feedback). This is an honest single-topic result, not a demonstration of early stopping: on a topic whose objectives get covered before sources run dry, the new stop condition would end the loop before round 3, but this topic wasn’t that case. src/tools/libraryResearch.test.ts’s “stops early once every outlined objective is covered” test (mocked LLM/network) is where the actual early-stop mechanism is verified; this live run instead confirms the objectives/coverage plumbing produces sane values (a 3-to-7-item outline, index-aligned coverage booleans) end to end against a real gateway.
  • Elapsed time and citation count are not a clean before/after comparison - same caveat as every other before/after in this file: n=1 per side, live LLM/gateway variance (round 1 alone found 5 new sources before vs. 8 after for the same queries-so-far budget), and the after run makes two additional chatJSON calls per run (one generateObjectives up front, one updateCoverage per round) that the before code never made, yet still finished faster - not concluded to mean anything beyond “the added calls aren’t the bottleneck at this scale”.
  • after’s warnings was empty; before’s report had one sentence kept with a “matched 0 times” warning (checkCitations’s conservative removal, unrelated to this task).

To re-run

Same gateway/model env as above. Not wired into npm run eval:answer (that harness scores library_answer, not library_research) - run libraryResearch(topic, { depth, breadth, maxMinutes }) directly from a one-off script, same as this measurement did.