scripts/data/tool_select_prompts.jsonl, 241 hand-labeled prompts, none
containing their own target tool’s name):
Neither replaces the other. BM25 is a cheap, reproducible proxy that runs on
every PR with no external dependency; a model call is the real question but
costs money, is not bit-for-bit reproducible, and needs a live endpoint — so
it stays opt-in rather than gating every PR.
Method
For every prompt applicable to aCODECALC_TOOLS group (full/dev/core
— a prompt is “applicable” if at least one of its labeled expected tools
is present in that group’s surface), tool_select_llm_eval.py makes two
calls to the model, both temperature=0:
- TOOLS mode — the real tool catalog as OpenAI
tools=[...](name + description + the exact MCPinputSchema),tool_choiceforced to"required". The tool the model calls is top-1. - LIST mode — no
toolsparam; the same catalog inlined as text, the model asked for a ranked JSON array of its top three tool names. This is top-3, and it also supplies the top-1 fallback for any model (or prompt) where TOOLS mode fails outright — a live gateway 400 for a model with no function-calling support, or a response with notool_callsat all.
full’s 52-tool catalog and core’s 25-tool one cost roughly the same
number of calls but very different token counts per call (see the token
usage numbers below).
top-3 is STRICT: a hit iff expected intersects LIST mode’s own
three-name answer, nothing else. aggregate() also reports
top1_or_list_top3 — the union of TOOLS mode’s single pick with LIST mode’s
three — under its own name, because the two modes are genuinely different
questions (“what does LIST mode’s own ranking contain” vs. “does either mode
get it right”) and folding one into the other silently overstates the first.
A cross-vendor review of the first live run caught exactly this: v1’s top3
unioned in effective_top1 (usually TOOLS mode’s answer, from a DIFFERENT
call) before checking the hit, which counted a correct TOOLS-mode call as
“top-3” even on prompts where LIST mode’s own three picks never named the
tool at all (147 of 898 records in the first live run). See “Measured”
below for how large that gap actually was, per model.
The cache key folds in a hash of the exact tools=[...] payload
(tools_payload_hash) alongside (model, group, prompt) — editing a tool’s
description or inputSchema invalidates every cached call for that group
rather than silently replaying a response scored against wording that no
longer exists. A run that gets zero cache hits for a (model, group) with
an existing baseline entry prints a ::warning:: naming it, since that
shape means either the catalog changed since the baseline was measured or
the cache directory was cleared — either way, worth a human noticing before
trusting the regression compare.
Measured — 2026-10-29 (Haiku 5.5 generation, 241 prompts)
Models:anthropic/claude-haiku-5-5 (the script’s default since Haiku 4.5’s
retirement) and openai/gpt-4o-mini — one Anthropic model, one
non-Anthropic model, so the numbers below are not a single vendor’s
self-report. Both were live on the gateway’s GET /v1/models at the time of
this run. The corpus is the 241-prompt set (prompt_set_sha256
bddf4f17…), 45 prompts larger than the 2026-09-08 run’s, because the tools
added since then (percent_change, session_delete_file, the
solve_linear/diff/integrate/series surfacing) each brought labeled
prompts with them. Hit counts are therefore NOT comparable to the history
table below; the checked-in baseline was regenerated from this run, and the
table was generated from the run’s --json report, not hand-edited.
Which generation these numbers are from: Haiku 5.5 and gpt-4o-mini, one
live run on 2026-10-29, with the tool descriptions as a Python 3.12
interpreter serves them. Python 3.13+ strips the common indentation from
docstrings at compile time, so the exact tools=[...] payload (and with it
every cache key) differs between a 3.12 and a 3.14 server; the hit counts
are not expected to move on whitespace, but that was not measured. The next
model bump is a rerun of the command under “Reproducing this” plus a
regenerated baseline, not a hunt.
Total tokens across the whole run: 36,814,069 (prompt + completion, both
modes, both models, all three groups; the gateway’s own
usage field,
summed — no live pricing feed exists here, so this is tokens, not dollars).
Haiku 5.5 spends roughly 1.6x the tokens of gpt-4o-mini on the same prompts
(its tool-call responses are longer). TOOLS mode was 100% reliable for
both models (top1_source is {"tool": n} for every model/group — zero
list_fallback, zero none), so every top-1 number above came from a real
tool call, never the list-mode fallback.
Both models beat BM25’s top-1 by a wide margin on every surface, and top-1’s
score gets better as the surface shrinks, the same “less headroom to lose on
full, more on core” shape tool_select_eval.py’s own self-check
documents for BM25. Haiku 5.5 now leads gpt-4o-mini on top-1 on every
surface (96.7% vs 92.5% on full).
The strict top-3 gap that defined the Haiku 4.5 run is gone. Haiku 5.5’s
LIST-mode ranking contained the expected tool in its top three on every one
of the 549 prompts across the three groups (100.0% strict top-3), where Haiku
4.5 scored 68.4-78.0% because LIST mode sometimes declined outright (see “A
finding from the first live run” below) and sometimes ranked the right tool
outside its own top three. gpt-4o-mini is unchanged in shape: strict top-3
within a point of its top-1-or-list-top-3 column.
History — 2026-09-08 (Haiku 4.5 generation, 196 prompts)
Kept for comparison; superseded by the table above. Corpus: the 196-prompt set (prompt_set_sha256 b7a425b3…). anthropic/claude-haiku-4-5-20251001
was the script’s default until 2026-10-28.
Total tokens for that run: 16,563,555. Strict top-3 was where the two models
genuinely diverged: gpt-4o-mini’s LIST-mode ranking nearly always contained
its own top-1 pick, while Haiku 4.5’s strict top-3 (68.4-78.0%) sat 15-30pp
below its own top-1-or-list-top-3 column, exactly the set of prompts where
LIST mode’s three picks did not include what TOOLS mode correctly called
directly.
The disagreements column counts prompts where BM25’s own top-1 pick and
the model’s effective top-1 pick differ — most of these are not the model
being wrong, they are BM25 being wrong where a model’s semantic read of
the prompt succeeds; the full per-prompt list is in the run’s
--json
report (disagreements_with_bm25 per model/group), not reproduced here.
Reproducing this
scripts/data/.llm_eval_cache/
(gitignored), keyed by (model, group, prompt, tools_hash) — an interrupted
or partial run resumes without re-paying for calls it already made, and an
edited description or inputSchema (a different tools_hash) is a cache
miss rather than a stale hit. --no-cache forces a refresh regardless.
A finding from the first live run: LIST mode sometimes declines
TOOLS mode (which drives top-1) was 100% reliable across both models in the first live run — every prompt produced a realtool_calls response, zero
tool_errors. LIST mode is not: without a forced tool_choice, a model
shown a prompt like “summarize these latency numbers” with no actual numbers
attached sometimes replies conversationally asking for the missing data
instead of complying with the “just return a ranked JSON array” system
instruction — a genuine, reproducible model behavior, not a parsing bug (see
_parse_ranked_names’s own handling: it returns [] rather than raising).
This does not lower top-1 (TOOLS mode supplies it directly, unaffected). It
does cost the STRICT top-3 metric directly: a decline means list_top3 is
empty, so strict top-3 is a guaranteed miss on that prompt regardless of
whether TOOLS mode got it right — one contributor (alongside genuinely
ranking a correct tool outside its own top three) to the top-1 vs. strict
top-3 gap the 2026-09-08 history table shows for
anthropic/claude-haiku-4-5-20251001. Haiku 5.5 did not decline once in
the 2026-10-29 run (strict top-3 100.0% on every group).
top1_or_list_top3 recovers a decline the same way it recovers any other
list_top3 miss, since it unions in effective_top1 regardless of why LIST
mode came back empty — which is exactly why that column is reported
separately rather than folded into top-3, per the “Method” section above.
Not evidence the model could not rank three tools if asked differently (e.g.
with a forced JSON response format, which this script does not use so its
behavior stays comparable across gateways that may not support it).
What this does NOT prove
Same caveattool_select_eval.py’s own docstring states for BM25, mirrored
here for the model half: a single run against one prompt set and two models
is evidence, not a guarantee, that a different model, prompt phrasing, or
tool-description edit behaves the same way. The --baseline compare exists
to catch a regression against this corpus and these models over time —
it is not a claim that the numbers generalize beyond them.