Skip to main content

​
Benchmarks

Documentation for v0.0.1. Additional guides and API reference coverage are being expanded.
Measurements below apply only to the identified artifact, workload and environment, not a universal performance guarantee. A benchmark measures how long a specific piece of work takes under recorded conditions. mcprouter’s local filtering benchmark asks: how quickly can the router find relevant tools in a catalogue of 1,000 tools? It measures the default, model-free tool filtering and ranking path. It does not measure a whole AI conversation or how quickly an external tool responds. The published 0.0.1 core archive has a qualified warm-query p95 of 24.893034 ms, meeting the p95 ≤ 50 ms target for the exact run described under current release evidence. That run used an explicitly accepted hosted-VM venue exception, not verified dedicated physical hardware.

​
What does p95 ≤ 50 ms mean?

The target is p95 ≤ 50 ms: at least 950 of the 1,000 measured queries must finish in 50 milliseconds or less.
  • p95 means the 95th percentile, not the average.
  • ≤ means “less than or equal to.” A p95 of exactly 50 ms meets the target.
  • 50 ms is 0.05 seconds, or one twentieth of a second.
To calculate p95, sort the 1,000 measured durations from fastest to slowest and take the 950th duration. This is the nearest-rank method: ceil(0.95 × N), counted from one. For example, if the 950th duration is 42 ms, the percentile meets the target. If it is 51 ms, it misses the target. These are illustrative examples, not measured results. Up to 50 queries can be slower than 50 ms while this percentile target still passes. It is not a maximum-latency guarantee, a promise for every machine, or a production service-level agreement. A numerical pass alone also does not establish release qualification.

​
What the test does

Each timed query includes source capture, bounds and snapshot checks, eligibility checks across all 1,000 tools, lexical ranking, sorting, and building the returned results. Timing uses performance.now() around the awaited filtering operation. The 100 warmup queries are excluded from the measured p95. Cold construction, registration, and the first query are also excluded from that percentile and retained as separate costs. Post-call assertions and report serialization are outside the timed filtering operation.

​
What it does not measure

  • AI response time: there are no inference requests, model repairs, or generated conversations in this workload.
  • Network or MCP discovery latency: the catalogue is already registered locally; the test does not contact a remote MCP server.
  • Tool execution time: filtering returns descriptors and never invokes the registered handlers.
  • Semantic ranking quality: lexical filtering speed does not establish whether a model or embedding ranker chooses the right tool.
  • Concurrent production traffic: a single-query workload does not establish throughput or latency under load.
Real model evaluation is a separate check of routing outcomes and request accounting. Passing this latency benchmark cannot substitute for it. Likewise, a fast ranking result is not permission to execute a tool.

​
Three different limits

Finishing within five minutes does not mean the 50 ms target passed. The workload can finish within its timeout and still have a p95 above 50 ms. A timed-out, interrupted, or incomplete run cannot supply a passing result. The job ceiling is not time spent inside every query and does not guarantee exclusive physical hardware.

​
How to read a benchmark report

A useful report ties the measurements to the exact candidate archive, dependency and harness inputs, Node version, operating system, architecture, and CPU/RAM allocation. Hardware and image-store compatibility checks support reproducibility; they are not latency measurements themselves. Look for:
  1. All raw warmup and measured durations, with the required sample counts and unique queries.
  2. The nearest-rank p95, mean, population variance, and standard deviation—not only the fastest run.
  3. Cold costs reported separately, plus the query-cache, concurrency, and zero-dispatch checks.
  4. The command, exit status, network isolation, cleanup result, and any failed or incomplete attempts.
  5. Review of the execution venue: dedicated hardware evidence or an explicitly scoped exception. A hosted VM must not be described as verified dedicated physical hardware.
Equivalent required checks can run in local Docker or CI. Local testing is useful for diagnosis; CI helps reproduce and retain results. Neither location automatically proves performance. Evidence applies only to the version, workload, and environment actually tested. Do not silently retry until a favourable result appears or reuse an old exception for a new run.

​
Current release evidence

The October 24, 2026 timed run was reviewed and qualified for the exact 0.0.1 core archive subsequently published on npm. The measured warm-query p95 is 24.893034 ms, below the 50 ms target. The cold costs make clear why the result is a warm-query qualification, not a cold-start guarantee or production SLA. The venue exception applies only to this run; it neither certifies dedicated hardware nor grants another hosted run. Other artifacts, dependency graphs, Docker stores, machines and concurrent workloads need their own evidence. Earlier untimed adapter self-tests passed in a local containerd-backed environment and on Docker 28.0.4 with a classic VFS store in a disposable Debian 13 ARM64 VM. Those remain image-binding/self-test compatibility evidence, not measured p95 results. Earlier development measurements retain their own artifact/environment identities and do not replace the qualified result above.

​
Separate model qualification

The original three complete model trials scored 120/120, 120/120 and 119/120, with 360 settled requests, no unknown requests and no forbidden dispatches. The third trial’s miss returned needs_input without dispatch; it was counted, not dropped or retried. These are finite routing-evaluation results, not filtering latency or universal provider accuracy. Those live measurements retain their original dependency-graph identity. The published replacement patches include bundled Undici 7.30.1 and use Zod 4.6.6 in the selected profile. Their bounded qualification carries the original model evidence forward through reviewed source-impact analysis and a 40-case offline differential comparison on each of Node 22 and Node 24, not a fresh live-model run on the replacement graph. That comparison does not establish universal behavior, wire-byte identity or latency equivalence. No new provider request is authorized by these results. For API usage, see Filter tools. For the difference between filtering and execution, see Choose an integration.