​

Evidence, economics and limitations

Documentation for v0.0.1. Additional guides and API reference coverage are being expanded.

Whitepaper revision 3 · October 27, 2026 · SDK scope: published 0.0.1, with the original model-trial dependency identity preserved below The evidence answers two different questions: how long local lexical filtering took, and whether a configured model selected the expected outcomes on a fixed corpus. Neither answers how fast or inexpensive a complete production agent will be.

​

Measure three properties separately

Abstention also needs a task-aware rubric. needs_input is correct when required information is absent, but a semantic miss when an actionable case already supplies it. Reporting all abstentions as successes would conceal routing failures; reporting all as failures would penalize appropriate restraint.

​

Local lexical filtering: a qualified warm-query result

The October 24, 2026 run measured the published 0.0.1 core archive, SHA-256 621e27364027e28de25ddddf06f2193287f1e77690a0b43dcca6475511e68147. It used 1,000 synthetic local tools, 100 distinct warmup queries and 1,000 separate measured queries. Queries ran serially with the query cache disabled and returned five ranked tools each. The operation includes source capture, snapshot and eligibility checks, lexical ranking, sorting and result construction. It excludes registration, the first query, model inference, network discovery and tool execution. No model provider or tool handler ran. The benchmark guide records the qualification and full scope. Figure 5. Distribution recomputed from the retained 1,000 warm-query durations, not a new benchmark. Bins include their lower bound and exclude their upper bound; the final bin includes 50 ms and above. Source: the qualified October 24 filtering run summarized in Benchmarks. This is not end-to-end agent latency. Percentiles use nearest rank: sort the samples and select position ceil(p × N), counted from one. The p95 meets the run’s 50 ms threshold. The observed maximum is not a promised upper bound, and the cold costs prevent interpreting the warm result as a startup guarantee. The environment was an Ubuntu 24.04 ARM64 hosted VM, observed as Ubuntu 24.04.5, with Node 24.21.0 and Docker 28.0.4 using overlay2. It had two logical CPUs and approximately 7.73 GiB RAM; the CPU model was unknown. Qualification used an explicit run-scoped hosted-VM exception: dedicated physical hardware was not verified. The run establishes neither a production service-level agreement nor performance across arbitrary machines, catalogues or concurrency levels.

​

Routing trials: finite semantic evidence

Three original live-model trials repeated the same 120-case synthetic English holdout against a 12-tool catalogue. Each trial contained 72 actionable cases, 24 requiring input and 24 with no matching action. The recorded response model was gpt-5.6-luna-2026-07-09. That identifier describes the tested configuration; it is not a recommendation or evidence about a 1B–8B model. The frozen settings used default lexical selection, a shortlist of five, threshold zero, a 512-output-token cap and a 30-second case deadline. The maximum was two proposals; no repair, retry or fallback occurred in these trials. Temperature was unspecified, not zero, and an output-token cap does not demonstrate a particular reasoning budget. Figure 6. Original routing-trial outcomes, not a model comparison. Bars start at zero and show matched cases out of 120. The third trial’s one miss is shown separately and remains in the denominator. Source: original three-trial qualification summarized under separate model qualification. Repetitions are not 360 distinct tasks. The missed actionable case returned a schema-valid needs_input and did not dispatch. This is a semantic failure, not an authorization bypass. All 360 requests settled with no unknown requests; the miss was not dropped or retried to improve the score. The repeated corpus limits the statistical independence of a pooled success percentage, so the per-trial results remain the primary presentation. These measurements retain their original dependency-graph identity. The published replacement patches include bundled Undici 7.30.1; the selected profile uses Zod 4.6.6. Qualification carries the original model evidence forward through reviewed source-impact analysis and a 40-case offline differential comparison on each of Node 22 and Node 24. It is not a fresh live-model run on the replacement graph, nor proof of identical wire bytes, latency or universal behavior. The corpus does not establish multilingual performance, unseen enterprise tools, adversarial robustness, long-horizon planning or production task completion. Zero forbidden dispatches in these trials is a finite observation, not a security certification. The synthetic, AI-authored holdout was maintainer-visible and disjoint from development examples by case IDs and normalized prompts. That does not establish secrecy or independence from model training; shared-family and pretraining contamination remain unknown. Its designed task proportions are not estimates of production traffic. The aggregates recount retained scorer flags; raw historical provider responses are unavailable for independent retrospective semantic regrading.

​

The small-model hypothesis remains open

A constrained action space may make a specialized model worth evaluating. That is a testable design hypothesis, not a conclusion from these trials. Smaller parameter counts do not remove ambiguity, poor tool descriptions, missing context or the need for host authorization.

​

Model-tier opportunity matrix

The source paper’s companion matrix proposes different roles for 1B–8B models. The matrix below preserves that design direction while separating potential roles from capabilities demonstrated by the released SDK. Parameter count alone does not qualify a model for any row. The companion material groups the opportunity into five dimensions:

• Selection quality: a relevant shortlist may reduce distractions, but validation does not make a wrong action correct. Semantic ranking is an option, not an automatic prerequisite, and the SDK does not guarantee constrained decoding in every model runtime.
• Latency: fewer schemas may reduce prompt processing. Measure inference and the complete action path separately from the lexical-filtering result above.
• Cost: assigning routine decisions to a specialized model may reduce some model charges. Neither the proportion of routine calls nor the net savings is established here.
• Local deployment: a locally hosted model may limit some remote data transfers. The published SDK targets Node.js, not browser or mobile execution; tools, embeddings, telemetry and host services need their own egress review.
• Specialization: tool-selection and argument-generation examples may be useful fine-tuning targets. Training quality, fresh holdout performance and generalization remain to be tested; fine-tuning is not a built-in SDK service.

​

Illustrative cost, latency and throughput visual

Illustrative inputs — not benchmark results. The supplied web visual compares a frontier-model scenario, a cloud 8B router and a local 3B router. Figure 7 retains its numeric inputs so the proposed trade-off is visible. They are not measurements from the retained trials, current provider quotes or a validated comparison between models. No common workload, hardware, inference settings or timing boundary was supplied for these values. Figure 7. Static adaptation of the supplied web visual. Horizontal position represents assumed latency, vertical position represents assumed input-token price, and bubble area represents assumed token throughput. Both axes start at zero. These inputs illustrate the source proposal; they do not establish a small-model advantage. The zero local model-token charge excludes compute and operating costs. The original visual also attached schema-compliance percentages and “deterministic” badges to these scenarios. Those unsupported badges are not reproduced as guarantees. A schema-valid proposal can still choose the wrong tool or wrong arguments; permission and semantic success must be evaluated separately. The measured routing outcomes above illustrate that distinction directly.

​

How to test the proposed advantage

To establish a small-model advantage, compare exact model versions or weight hashes on the same frozen workload. Record the runtime, hardware, quantization, inference settings, prompt/schema format, shortlist size and repair budget. Report semantic success by task type alongside abstentions, invalid proposals, forbidden dispatches, token use and latency. Hold out fresh tasks after tuning, and disclose failures rather than repeating only the best run. Automatic escalation to a frontier model would introduce a further host-owned decision: when to escalate, what data may leave the environment, and what budget applies. The current SDK does not provide an autonomous fallback ladder. Any evaluation must count escalation latency and cost rather than attributing the final result solely to the smaller model.

​

Economics must include the complete workflow

Local inference can avoid a per-request provider invoice, but it is not cost-free. Hardware, idle capacity, memory, energy, deployment engineering and monitoring still contribute. Hosted pricing also depends on prompt size, output tokens, caching and retries. No controlled cost comparison in the retained evidence supports a universal savings multiplier. A deployment assessment should measure:

• Total latency: host reasoning, discovery, filtering, routing inference, repair, approval waits, tool execution and result handling. Overlapping stages require measured critical-path timing, not an unconditional sum.
• Total operating cost: model charges or local compute allocation, tool-service charges, escalation, retries and operational overhead.
• Cost per successful task: total cost divided by independently verified successful tasks, with failed and abandoned attempts included in the cost.
• Risk and recovery: unauthorized-action attempts, unknown effects, human interventions and reconciliation work, reported alongside throughput.

​

Workflow measurement matrix

The companion workflow matrix separates five categories worth instrumenting. Its call-volume, latency-share and spending percentages lack a trace dataset, sampling method and consistent accounting boundary, so they are not presented as observed production distributions. Use the categories below to measure your own workload rather than assuming a universal split. Use explicit span ownership to avoid counting nested work twice. Concurrent spans can overlap, so their durations are not additive shares of wall-clock runtime. Report category denominators and the complete workflow total; a routing optimization cannot be credited with removing host reasoning or external-service work that still occurs. Use comparable baselines: exact host calls for predetermined actions, a host model selecting directly from tools, and any proposed specialized router. Keep task mix, permissions, result quality and hardware or pricing assumptions explicit. A faster filter does not prove a faster agent if inference or the external service dominates its response time.

​

Adoption and qualification checklist

Begin with synthetic, read-only tasks and a deny-by-default execution policy. Verify exact-call behavior before adding a routing model. Then evaluate a frozen corpus containing overlapping tools, missing information, no-match cases, denied actions and stale bindings. For writes, test timeout recovery and service-level idempotency separately. Qualify the actual deployment artifact, dependency graph and infrastructure. Retain failed runs, machine details, sample denominators and scope exclusions. Recheck after changing models, prompts, schemas, policies or adapters; the original result does not automatically transfer to changed inputs. The architectural conclusion is narrower than a promise of cheaper autonomous agents: separating relevance and proposal from authorization gives the host a defined place to govern execution. Whether a specialized model improves quality, speed or economics remains a workload-specific empirical question.

​

References and evidence access

1. Qualified filtering benchmark and original model-trial summary: public methods, artifact identity, venue exception and dependency limitations. The histogram and percentile table derive from the same retained run, not an additional experiment.
2. Filtering contracts and model bindings: source selection, ranking alternatives and routing-model access.
3. Execution contract, result states and security boundaries: authorization, abstention, snapshots and recovery responsibilities.
4. Installation and compatibility and published core package, version 0.0.1: package and environment scope. Initial-release npm attestation is deferred as documented, not verified.
5. MCP tools specification, revision 2026-07-28: protocol context and client/server security considerations, not performance evidence.

This public paper presents sanitized aggregates and methods, not the internal raw trace archive. Readers cannot independently reproduce the measured results from these pages alone. The source report and comparison widget are not benchmark datasets; their binaries and executable code are not redistributed here. The adapted matrices and Figure 7 retain the proposal separately from the measured evidence. No competitor latency, cost, adoption or universal-correctness claim is asserted.