Performance & Conformance¶
Note
This page is auto-generated from the live test + benchmark harnesses on each docs
build (bench/gen_results_page.py, ADR-0032) — the published response surface, not a
hand-edited snapshot. All rates and latencies are absolute measured values,
representative of the CI runner. Each chapter opens with the method for its own surface;
the cross-cutting rules are §1–§3, and §4 says what
every harness in bench/ drives and reports.
🤖 CI-generated on 2026-08-23 01:13 UTC · commit 3d958fa · run 32609336162 · runner Linux.
How to read this page¶
Every number here belongs to exactly ONE measurement surface, and a value is only comparable to values from the same surface — surfaces use different harnesses, processes and units. A surface is a property of the series, not of the chapter it is drawn in: a chapter groups by subject, a subject can be answered by more than one instrument, and this table is what settles whether two numbers may be compared at all.
surface |
what it measures |
harness |
where it appears |
discipline |
|---|---|---|---|---|
Cross-core conformance |
byte-exactness across cores, not speed |
|
§5 |
any DISAGREE fails CI |
In-process timing |
per-operation latency and throughput in one process |
|
§6, §7, §8, on demand |
gated per PR and per |
Framed-hop timing |
what one wire frame costs, before and after the payload |
|
§7, on demand |
batch-amortized against the host clock, self-calibrating |
Allocation counting |
exact allocations and bytes around ONE operation — counted, never timed |
|
§8, on demand |
forward hop hard-gated at zero; per-vertex block count ratcheted exactly |
Network timing |
one-way latency over a real socket, two processes |
|
§9, on demand |
same topology for both engines; p50, the p99 tail and the p999 deep tail — published, never gated |
Engine comparison |
libtracer and Zenoh side by side, absolute |
|
§9 |
same runner, best of 3 rounds — no ratios |
Cross-core codec |
decode→encode roundtrip per implementation |
|
§10 |
the same v1 vectors for every core |
Concurrency & scaling |
how one shared object behaves as threads are added |
|
on demand |
many-core host, run on demand — not part of the published sweep |
Instrument self-check |
that the harness’s own estimator reproduces a distribution it was given |
|
on demand |
run before trusting a published percentile; touches neither clock nor library |
§2 is what stops a regression, §3 is what one number is worth against the runner lottery, and §4 says what each harness actually drives.
Every chart on this page is the same chart¶
One idiom, learned once. A family is a set of series answering one question, drawn as
lines on shared axes; the x-axis is recorded main commits, oldest to newest, except in
§9 where it is the swept parameter. Each chart names its own swept variable and
pinned scenario beneath its title — which is also why two charts both titled “throughput”
are not comparable to each other: different denominator, by construction.
🏷 dashed marker = a release tag (≈ when the tag’s commit is not itself a recorded point). 🔧 dotted = the bench changed at that commit, so points either side were taken with different rulers and are not comparable. Core changes are deliberately not marked: a core change moving the line is the signal the chart exists to show.
A series label keeps one color across every history chart. The §9 comparison charts use a fixed engine palette instead — their line dimension is which engine, not which parameter.
Families with a numeric parameter carry four switchable views — trend, sweep, heatmap and an isometric 3D surface — drawn only over the commits where every series of the family has a value, so a series that started late cannot fake a trend.
Every commit-axis view is drawn over a selectable commit range (two sliders per card, the full history by default). Narrowing it re-bases the axis and the release/instrument markers with it, so a short window is a real chart rather than a zoom.
The paired libtracer-vs-Zenoh cards (fan-out and payload) add a ratio toggle: zenoh ÷ libtracer per recorded commit, dimensionless, with a parity line at 1×. On a latency metric a quotient above 1 means libtracer is faster; on throughput the same statement is a quotient below 1, and the y-axis says which. This is the comparison to read across a long history: both arms are measured in the same pass on the same runner, so that machine’s speed on the day divides out of the quotient to first order, while the absolute lines above it carry the full shared-runner spread. Each point pairs the two engines at one commit; a commit where only one arm recorded contributes no point. The cancellation is partial on the hosted store and not total, because that store records the best of three runners per series, so a point’s two arms are not guaranteed to be the same runner’s — measured at fan 128/1024/8192 over that store’s last 60 recorded commits, the quotient’s spread is about a tenth lower than the libtracer line’s own. On the bench-local store, where every point is one pinned CPU (measured over its full store of 12 runs), it is about a third lower.
A
sourceselector heads each chart block: GitHub-hosted (the default — best of three runners per point, a portability envelope) or bench-local (one pinned self-hosted CPU, the absolute-trend instrument). One store at a time, page-wide, never overlaid — the two answer different questions (§13). A family the selected store has not recorded says so instead of drawing an empty axis.Hover any point for its exact value, the commit and its subject line — plus the host descriptor when the store records one, which the bench-local store does on every point.
Two latency series per in-process mode¶
Every <mode> row is timed one operation at a time; every <mode>-batch row times a
calibrated batch of the same operation and divides. The clock is a large fraction of a
sub-100 ns write, so the per-op percentiles snap to coarse steps and read high at small
fan-out, converging on the batch row as the operation outgrows the clock — the Clock
quantization chart in §6 plots exactly that gap. Use -batch to resolve a
small delta; use the per-op row for tail shape, since it is the one with a real p99. A
percentile of batch means measures interference between batches rather than the tail of an
operation, so the -batch rows publish no p99 at all rather than a fabricated one.
1 · What the metrics mean¶
Across the surfaces, a measurement is one of the following dimensions. Each has a distinct instrument, and a distinct rule for what a “worse” number means.
dimension |
unit |
instrument |
direction |
status |
|---|---|---|---|---|
latency |
ns (p50 / p99 / mean) |
wall-clock per op, |
lower better |
gated ✅ per-PR + push |
throughput |
deliveries/s, publishes/s |
ops / elapsed, |
higher better |
gated ✅ per-PR + push |
alloc bytes |
bytes & count per op |
counting allocator, |
lower better |
forward hop gated ✅ = 0; other probes tracked |
memory footprint |
live bytes / vertex, max RSS |
|
lower better |
gated ✅ per-vertex (+2% same-runner); RSS tracked |
wire bytes |
encoded frame bytes |
TLV frame size over the v1 vectors, codec surface |
lower better |
being promoted to a first-class series |
CPU |
work per op |
per-op cost on a pinned core |
lower better |
latency is today’s proxy; dedicated counter planned |
Three notes on reading this table honestly:
Latency vs CPU. Today’s per-op cost is measured as wall-clock latency on a quiesced, core-pinned runner, which on an idle machine is a close proxy for CPU work. A dedicated cycles/CPU-time counter is a finer instrument for the same dimension; where a chart is labelled “CPU” it means per-op compute isolated from I/O and wait.
Wire bytes are a codec property, not a dispatch property. They come from the encoded size of a message over the wire (the TLV frame), measured on the same v1 vectors the codec surface uses — so they are comparable across cores and independent of runner speed.
Alloc bytes vs footprint. Alloc bytes is the transient heap a single operation churns (gated to zero on the forward hop). Footprint is the resident memory the graph holds at rest (per-vertex live bytes, whole-run RSS). A design can be zero-churn yet heavy at rest, or lean at rest yet allocation-happy per op — so the two are tracked separately and never summed.
2 · What actually stops a regression¶
Absolute nanoseconds vary ~2× with the runner drawn, so the gates are all same-runner relative comparisons, where machine speed cancels. Three jobs, three thresholds, one hard invariant:
mechanism |
when |
comparison |
threshold |
effect |
|---|---|---|---|---|
per-PR hard gate ( |
every PR |
PR build vs |
p50 +15% · mean +12% · deliveries/s −12% · per-vertex bytes +2% — and disjoint ranges and a majority of pairs |
fails the PR |
push ratchet |
every |
HEAD vs its parent, three independently-drawn runners |
same as above |
advisory — each replica reports (see the tier note below) |
forward-hop zero-alloc gate |
every CI run |
absolute |
|
fails the build |
soft trend alert |
per |
vs previous point, cross-runner |
series drifts past 125% |
a comment, not a verdict |
Verdict tiers. The comparison is the same wherever it runs; what a caller declares is
whether a breached ratchet may stop the job. perf_gate.py --tier blocking fails on a
breach, --tier advisory prints the identical numbers and the identical verdict line and
exits 0 — with a ::warning:: annotation, so an unenforced breach is never quiet. The tier
is not a second, looser threshold set: an advisory run reports exactly what a blocking
run would have said. The per-PR gate is blocking; the three push replicas are advisory,
because a single noisy runner should not carry a verdict alone and the ruled escalation
(two of the three replicas agreeing) is not built yet. Omit the flag and you get
advisory — the safe default for a maintainer’s own machine, where the bar was never
calibrated; every CI invocation declares its tier explicitly and a unit test fails the
build if one does not.
Details that make these trustworthy:
The per-PR gate watches fifteen canonical points — a representative slice of the fan-out / payload / topic sweeps plus a fold-width point, one per gated family (
inprocandinproc-borrowshare one), so a pullback on any of those legs is caught and not just the 1:1 write. They are not the whole dispatch surface, and this page should not be read as claiming they are:inproc-deliver, the-batchtwins and theeptype-*sweep’s two lean arms are measured and charted but ungated. A pullback confined to those ships without the gate objecting. They areperf_gate.py’sPOINTS, named here in full because this page is hand-written and a bare count says nothing about what is covered; each is keyedmode/payload/fan-out/endpoints:inproc/64/1/1— the canonical 1:1 write;inproc-borrow/64/1/1— its zero-copy twin, the same write handed a borrowed view instead of an owned copy;inproc/64/1024/1— the 1024-subscriber fan-out loop;inproc-path/64/1/8192— the resolver canary, one registry lookup per write across 8192 registered vertices;mixed/0/6/128— the composed topology, 128 topics whose fan-out varies 1–16 (mean 6) over payloads of 1 B–8 KiB, which is why its payload column reads 0;fold-b4/512/1/1— the L0 fold walk, 512 bytes held constant across four rope links and timed over a batch; andlkv-store-heap/64/1/1+lkv-store-pool/64/1/1— the L1 rope-to-contiguous copy (rope_t::materialize: one segment allocated from the backend plus the payloadmemcpy), against the default heap and against a pooled backend; andinproc-target-handler/64/8/1+inproc-target-stored/64/8/1— the path-target dispatch legs at fan-out 8, edges carrying a target key rather than a callback, which is the leg a wireSUBSCRIBERactually takes.Those last two are here because of what happened without them (#1250): reshaping
rope_t::flatten’s wrapper cost 25–48% on every path throughmaterialize— branch and field writes,op_resolvereads, FWD COMPACT emission, the RX span sink — and no gated point at the time was downstream of that call, so the loss shipped with every gate green. They are read out ofRESULTrows the default sweep already emits, so they add no wall-clock. Note the name:lkv-store-*measures the copy-store allocation, not the last-known-value slot.The
inproc-target-*pair is gated at fan-out 8 and nowhere else, for two measured reasons (#1077). Fan 8 sits exactly onvertex_t::kInlineFanout, the no-heap small-fan-out boundary, so it is the width that prices the narrow-fan snapshot path — which is where that issue’s ~+5.5% step appeared, invisible to every gate at the time. And a layout control (the same source rebuilt at three-falign-functionssettings) measured this row as the most layout-stable point of the whole sweep, 0.62% across placements against ~15% forfold-b4, so the false-red risk from code placement is near zero here. Both legs are gated becausestoredcarries the same shape ashandler; it reads reliably now thathost_guard.pyrejects the contaminated windows that once put its own A/A null at −14.7%. Like thelkv-store-*pair they add no wall-clock: the default sweep already emits both rows at every fan width, so this is two more keys read out of output already collected. Both gate on all three legs at the nominal +15% / +12% / −12%: a fan-8 row’s p50 and mean time the whole 8-subscriber publish (~640 ns and ~800 ns p50 respectively, against per-delivery costs of ~62–80 ns), so they sit far above the sub-100 ns band where the tick guard would demand an extra +25 ns absolute and blunt them the way it bluntsfold-b4.eptype-stream/64/1/1is the fifteenth, and the reason it is gated while its two siblings are not is the whole point of adding it.eptype-leanandeptype-lean-cachedare theinprocandinproc-borrowcode paths re-emitted under an endpoint-type name — already gated, twice over, so gating them again would buy correlated evidence rather than coverage.eptype-streamis not a re-emission: it is the only point on the list that declares a bounded history depth, and therefore the only one downstream of theSTREAMrole’s retention work. Nothing else on the list touches that path, so a pullback confined to retention was invisible to all fourteen predecessors, which is the same guard-gap shape aslkv-store-*and the compact/demux arms below. It costs no wall-clock: the default sweep already emits the row. And all three legs bite at the nominal thresholds — it measures ~190 ns p50 and mean over five best-of-rounds on a busy 31-core host, far above the sub-100 ns band where the tick guard would demand an extra +25 ns absolute.What that row prices changed underneath it, and the change is worth naming because the series name did not. Until
bdd1066bthe ring was the producer’s: a write to a stream vertex appended to its own history under the stripe mutex, before fan-out. Under RFC-0025 §4.6.1 Amendment 2 a producer never queues — the queue belongs to the party that wants depth, so the ring lives on the receiving vertex and the admission runs after the last-known-value publish rather than before fan-out. What the gated row times today is therefore the receiver leg: retire the entry the depth intent pushes out, admit the new one by reserving its retained width against that vertex’s own injected block source, and append. The row keeps its name on purpose. The rule above — renaming beats reinterpreting — governs a change in what the instrument measures; the bench source is untouched and the operation it drives is still “one 64 B write into a depth-16 stream vertex”. The mechanism beneath it moved, and a core change moving the line is exactly the signal the series exists to show.The append-and-admission leg is read as a gap, not as a series of its own.
eptype-streamandeptype-leanare the same write at the same payload, fan-out and topic count, emitted from the same pass of the same binary and banked as separate series on the Endpoint-type family card — so the distance between the two lines is the receiver-ring cost, over the whole recorded history, with the runner shared by both arms. Two things that leg does not price, stated because a reader would otherwise assume them covered: a reservation is handed straight on when a retiring entry has the shape the new one needs, so a steady uniform stream costs its source zerotry_alloccalls per write and this row measures the admission bookkeeping, not an allocator round-trip; and the resident footprint of the receiver’s ring state has no memory probe. A bench for the shape-changing admission path — where the carried reservation cannot be reused andtry_allocfires on every write — does not exist, and neither does one for the ring’s resident bytes. Both are gaps in this page, not numbers it is withholding.Four of the fifteen come from OTHER bench binaries, and they are here because of what happened without them (#1173):
compact-forwardmoved +41% across the v0.8.0 → v0.9.0 window while every gated point stayed flat, so the gate had nothing to object to. They arecompact-forward/64/1/1andcompact-terminus/64/1/1— the compact-delivery tier’s forward hop and its terminus, frombench_compact_delivery; andfwd-demux-fixed/79/1/1andfwd-demux-scan/79/64/64— the fixed-slot and scanning arms of the FWD demux, frombench_forward_demux. EachPOINTSentry names the binary that produces it; every one of them emits the same 12-columnRESULTformat, so this costs two extra processes per arm per pair and no new parsing.The points are
modevalues of the benches described above, and the numbers are theirsize/fan-out/endpointcolumns. The binaries are run interleaved —A B / B A / A B / B A, four pairs, alternating which one starts — so a slow window in the machine is shared by both arms rather than donated to whichever one holds it. Because the baseline is the same PR’smainrebuilt on the same runner in the same pass, the comparison is machine-neutral.A point fails only when all three of these hold, and the gate prints every one of them: the medians breach the threshold (the effect is big enough), the two arms’ [min..max] ranges are disjoint (a sign flip inside the ranges reads as indistinguishable and can never fail), and a strict majority of the interleaved pairs breach on their own (the effect reproduces). All three are needed because a best-of-3 estimator rejects a bad sample but not a bad window: a runner that goes slow for the whole of one arm’s block produces a clean, reproducible, entirely false breach, and the majority-of-interleaved-pairs rule is what a window cannot fake.
Each arm’s own spread across the pairs is printed, and the baseline arm’s worst spread is reported as the run’s drift figure — the baseline binary cannot be moved by the change under test, so its spread is the invariant control leg. It does not gate; it tells a reader whether the run was worth believing.
The same gate additionally checks five memory probes (
perf_gate.py’sMEM_POINTS, named here in full because this page is hand-written and a bare count rots):vertex— a default leaf at rest;vertex_value— the increment one LKV write adds;vertex_app5— a leaf carrying a copied five-field app-field table;vertex_app5_static— the borrowed twin of that table (ADR-0058); andreg_escape— the global-heap blocks a runtime registration takes that the graph’s injectedmemory_resourcenever sees (ADR-0039 / RFC-0014), whose target is zero. Each ratchets on two quantities: live bytes per vertex, tolerant at +2% because it is host-allocator-dependent, and the number of heap blocks, which is host-independent and therefore ratchets exactly — one extra block is a regression, full stop. All come from the counting allocator (bench_forward_heap), so they are exact rather than sampled and need no repetition at all, with the baseline binary probed same-runner via--baseline-bench-fwd. Supplying that binary for one arm and not the other fails the gate as a wiring error; supplying it for neither prints an explicitSKIP— a probe that cannot run never passes silently.A per-vertex cost a ratified clause already prices is not a pullback, and the memory ratchet has one narrow way to say so: a charged step (
perf_gate.py’sMEM_CHARGED), declared per probe, in bytes, naming the clause that charges it. A charge absorbs at most its own bytes — a step of exactly that size passes, one byte more fails and the failure names the unpriced remainder — and it is printed on every run, spent or not, so an allowance can never be a gate that quietly moved. It also expires by construction: the paired baseline is built frommain, so once the step lands there the delta is zero, the charge prints asUNSPENT, and the entry is deleted. It is not a tolerance: it does not scale, does not accumulate, and is not a budget to spend later.The gate’s decision rules have their own unit tests (
bench/test_perf_gate.py), run in the same CI job before anything is timed, pinning both directions: the recorded false-failure sample sets must pass, and the recorded real regressions must still fail.The push ratchet re-runs that gate on three separate runner draws and requires the regression to reproduce — one noisy machine cannot fail
main, and a regression that slips through the PR gate still gets caught the moment it lands.The forward-hop zero-alloc gate is the one absolute gate: it is a structural invariant, not a speed target. Steady-state forwarding must allocate nothing, so the threshold is literally zero.
The soft alert compares across runners, so it is only a prompt to look at the trend, never a merge-blocker.
The per-PR gate never compares against a recorded number at all: both binaries are
in hand on the runner, so every sample on both sides is drawn in the same interleaved
rotation, minutes apart at most. (bench/perf_baseline.json still exists for the
local / root-commit path, where there is no baseline binary to interleave with. It is
host-specific and never committed as a fixed number, precisely so the gate can
never encode one machine’s speed as another machine’s target.)
3 · Noise, variance, and what one number is worth¶
Runner lottery. Shared CI runners vary ~2× in absolute speed. The tell: a move that hits every series at once — including unrelated ones like the pure-codec
fold-b*rows — is the runner; a move confined to one family is the code. Read trends across several commits, not the third digit of one point.Per-point noise floor. Each recorded point is the median of the repeated RESULT rows one run emits, so per-iteration jitter does not move a series. Points are then recorded as the best across three runner draws, approximating the code’s capability rather than the machine lottery. Sub-microsecond points sit on a ~10 ns timer grain — do not over-read a 5 ns wiggle.
Tail percentiles are published, not gated — and here is the measurement that decided it. The deeper into a distribution a statistic reaches, the fewer samples stand behind it and the more it moves for reasons that are not the code. Running the same binary against itself and taking the worst ratio between repeats gives the floor any threshold on that leg would have to clear before it stopped firing on its own noise. On the in-process gated points (18 runs, replayed through the gate’s then-current best-of-3 estimator, worst of 15 disjoint pairs — the numbers below are a property of the machine, not of the estimator, so the interleaved gate inherits them unchanged):
leg
worst same-binary ratio
threshold it would need
gated today
p50
1.16×
+16 %
yes, at +15 %
mean
1.22×
+22 %
yes, at +12 %
deliveries/s
1.36×
−36 %
yes, at −12 %
p99
1.67×
+67 %
no — published only
A p99 gate would have to sit near +67 %, and a regression that large has already tripped the +15 % p50 gate several times over: the leg would add no detection while adding a new false-fail source on every PR.
On the two-process network bench, the same experiment run at two probe counts isolates how much of the tail’s instability is simply too few samples (5 repeats each, same idle host, only
nchanged):probes per point
p50
p99
p999
worst sample
4 000
1.09×
5.01×
62×
21×
10 000 (published)
1.09×
1.40×
17×
13×
The median does not care — it is 1.09× either way — while the p99 tightens 3.6× and the p999 3.7× purely from sample count. That is why the published run pays for the larger count. Even so the deep tail stays 17× unstable: a loopback p999 is dominated by scheduler wake-up, not by either engine’s code path. Read the p999 chart as the jitter floor this topology inherits, and read engine-against-engine on the p50 and p99 charts, which at the published probe count are stable enough to carry a comparison. (Measured on an idle 24-core host; a shared CI runner is not quieter than that, so these are lower bounds.)
A throughput pullback with flat latency is a machine, not a regression. Every gated point is measured by two instruments over the same operation — a per-op clock (p50, mean) and a bulk timer (deliveries/s) — and a change in what the operation costs moves both. When only the bulk timer moves and both latency legs come back flat-or-better, the two instruments contradict each other, and the gate reports the contradiction rather than failing on it. This is not hypothetical: one PR that touched only L4 was failed at −33 % throughput on an L0 codec point it has no call path to, with p50 identical and the mean better. The guard is deliberately narrow — any upward move in either latency leg, of any size, leaves the failure standing.
Sign conventions in the history store. The latency suite is smaller-is-better nanoseconds; throughput also appears there inverted as
ns/deliveryso a slowdown always charts as a rise; memory metrics live in that same smaller-is-better suite. The throughput suite is bigger-is-better naturaldeliveries/s. The same measurement can therefore appear twice, in two units — by design, so each suite reads monotonically.A discontinuity in the instrument is annotated; a discontinuity in the code is not. When a commit changes a bench source —
bench_libtracer.cpp,bench_forward_demux.cpp, the shared harness, the emitter — every series that bench feeds is marked at that commit, because points either side of it were taken with different rulers and are not comparable. Changes undercore/**are deliberately never marked: a core change moving the line is the signal the series exists to show, and marking both would reduce the annotation to “something happened,” which is worth nothing. Comment- and whitespace-only bench edits are filtered out for the same reason — an alert that fires on nothing trains readers to ignore it.Renaming beats reinterpreting. If a fix changes what a row measures, the mode string changes too: the old series ends visibly and a new one starts at one point. What is never done is keeping the name while changing the meaning, which produces a continuous-looking line that silently stops being about the same thing. If a fix only makes the same quantity more accurately measured, the name is kept and the instrument marker carries the discontinuity.
Across rounds, take each arm’s BEST — never its median — and read the range as a contamination diagnostic. Contamination is one-sided: a busy neighbour can only make a round slower, never faster. A low order statistic across an arm’s rounds therefore rejects a dirty round, while the median merely counts them, and the median flips as soon as half the rounds are dirty — which is why the gate estimates best-of-N and why an ad-hoc driver must too. Measured on the pinned host with one binary against itself (#1358): in a window where a neighbouring job got busy, median-of-rounds put
fwd-rope-hopat −33 % … +54 % while best-of-rounds on the very same samples stayed inside 1.44 %; quiet, both estimators agree inside ±0.5 %. The[min..max]range is what tells the two situations apart — a spread near 2× whose ends sit in two clusters rather than spreading smoothly is a contaminated window, and the run must be repeated rather than reported, whatever the medians say.The same estimator rule governs the series, and a gap in it is a measurement defect. Points of the fixed-host store are rounds of a slower experiment, so they inherit both halves of the rule above. First, a baseline drawn from them is the window’s best, never its median:
store_guard.pycompares each banked point with the best of the last 20 trusted points, which is the comparison benchmark-action’s point-to-pointalert-thresholdcannot express — five merges of +4 % never trip a 15 % step, and the series ends 22 % slower with no alert ever raised. A best is a floor, though, so a breach must also land outside that window’s whole[min..max]: on the real store the percentage bar alone reports 59 breaches where the range guard leaves 2. Second, a flagged point is not a sample — it answers what the machine was doing, not what the code cost — so it neither sets the baseline nor counts as coverage. And coverage is itself guaranteed: a run that never happens (filtered out, failed, or cancelled while queued) leaves a hole nothing notices, which is how one +41 % step became a 16-merge range instead of a commit (#1173). A daily catch-up measures HEAD whenever the last trusted point is more than 8 bench-relevant merges behindmain, so the guarantee is a bound on the gap — narrow enough to bisect — rather than a point per commit.A banked series is a TREND instrument; the paired same-runner A/B is the gate. The two stores answer “where has this point been going” across machines and months. They do not answer “did this commit cost anything”, and reading them as if they did produces verdicts the code never earned. A textbook case, both halves measured on the same day: the rolling drift check on the banked series warned that
inproc 64B/fan1/1epp50 had gone to 400 ns against a 100 ns baseline (+300%), while the interleaved same-runner A/B on the identical commit read 1.02×. Nothing regressed — the banked baseline and the banked point were taken on different machines under different load, and the quotient of two absolutes across that gap is instrument drift wearing a percentage sign. So: a drift warning is a prompt to measure, never a verdict; a verdict comes only from two arms interleaved on one runner in one session, which is what the per-PR gate is and what every ratchet in the table above compares. This holds for every banked series on this page, the memory ones included.An absolute without its host is not a measurement, and this page never publishes one. Load alone is worth more than most of the effects anyone argues about: the same gated point read 3.6 M deliveries/s while CI was building on the box and 6.0 M/s quiet — 1.6×, no code between the two. Every absolute here therefore names the machine and the conditions it was taken under, and a figure that arrives without them is unusable rather than merely imprecise — #1495 is open on exactly that defect in a normative document: RFC-0025 §4.6.2 states throughput caps as bare cross-machine numbers, and unmodified
mainmisses two of them by 1.40× and 1.71× purely because nobody recorded which host they were cut on. Where this page mentions those caps it links that issue; it does not report them as met.Record the load context on both sides of every measurement, and wait for quiescence first.
python3 bench/host_guard.py waitbefore the run and/proc/loadavgeither side of it: an absolute figure without its load context cannot be re-judged later, and this is the fact that the same pair of binaries reads inside ±0.5 % at load average ~2 and up to +81 % at load average 16–36. The contaminant that matters is memory-subsystem contention, not runnable-task count — five pure CPU spinners (load average 6.8, just under the guard’s bar on the 31-CPU host) cost 0.83 %, while memory-bandwidth neighbours at the same nominal pin cost tens of percent.A per-op figure is only as honest as the batch it was timed in. Timing a batch and dividing charges the window’s two clock reads (~22 ns on the pinned host) once per sample, so the batch size is part of the measurement: forcing it on a ~238 ns operation reads 260 / 250 / 242 / ~238 ns at batch 1 / 2 / 4 / ≥8. A calibrator that picks the batch by comparing two timed quantities therefore lets the machine choose the answer, discretely —
bench_common.hpp’scalibrate_batch_for_windowdoubles until the window reaches 20 µs instead, so the batch follows the operation’s own cost and repeats across executions.
The A/B protocol — what a two-arm comparison may and may not vary. An A/B of a code change runs two binaries and attributes the difference to the change. That attribution is only sound if nothing else differed. Two things that look harmless routinely do.
Build directory: harmless, and now measured. A cross-worktree A/B — build
origin/main in one worktree, the change in another, interleave — was suspected
(#807, from PR #806) of measuring code layout rather than code: identical
origin/main source built at two paths appeared to differ by +1.7 % (65589 B
frames) to +6.7 % (53 B frames) on bench_terminus_tier’s terminus-arena leg
over 12 interleaved rounds, with disjoint ranges. That cause is refuted. Building
the same commit at two paths of different lengths produces, on this toolchain,
byte-identical output: all 27 libtracer object files, libtracer.a, and the
bench_terminus_tier executable cmp equal (one md5 for both arms). There is no
layout to be sensitive to, so no layout lever is warranted, and none ships —
-falign-functions=64 over the whole bench + library build reads −1.50 % to
+1.56 % against stock across 10 interleaved rounds, sign varying by frame size,
i.e. nothing. Re-run at two paths under this protocol the same leg reads −0.64 % to
+0.87 % (unpinned) and −0.64 % to +0.73 % (pinned), 12 interleaved rounds.
CPU placement: the real hazard, and it is large. The reference host is
heterogeneous — 4 Zen 5 cores at 5.16 GHz (cpu0–7) and 8 Zen 5c cores at
3.29 GHz (cpu8–23). The same binary on the terminus-arena leg reads +47.0 % to
+53.7 % slower pinned to a compact core than to a classic one, 5 rounds each, ranges
disjoint and tight within each arm (e.g. 53 B: 308–313 ns vs 474–481 ns). That is the
signature a confounded A/B wears: tight, reproducible, and entirely about the machine.
Even the choice of pinning moves the figure — at 53 B, 15 runs each: unpinned median
228 ns, taskset -c 2 234 ns, taskset -c 2,3 238 ns (+4.4 % across pinning
policies, from SMT sharing). A few percent between two arms needs no exotic
explanation on such a host; it needs only that the two arms were placed differently.
So the rules for any latency A/B, on this leg or any other:
Pin both arms identically, to the same single logical CPU on the same core class (
taskset -c 2). Pinning one arm and not the other, or to different core classes, is not a comparison.Interleave the arms round-robin within one session and report medians and ranges of at least ~10 rounds per arm. A single round per arm cannot separate the change from the machine.
Discard the first execution. A cold process’s first measured point on this leg reads ~313 ns against a 228 ns steady state (+37 %) — it is idle-state wake-up, not code.
Prefer same-directory A/B (
git stash, rebuild, re-run in one session) when it is available; it varies strictly less than a two-worktree run.When the expected effect is smaller than the leg’s own noise floor, do not reach for a stopwatch at all — use object-file
cmpagainst the baseline tree (the method PR #799/#806 established). A change that leaves 25 of 27 objects byte-identical has proved more about the unchanged paths than any bench can.
The stack-offset hypothesis was also tested and is negative: sweeping the environment block from 0 to 4000 bytes (which shifts the initial stack pointer, and with it the bench’s 64 KiB stack slab) moves the leg 2.8–4.4 %, non-monotonically — indistinguishable from run-to-run noise.
4 · The measurement surfaces, and the instruments behind them¶
Every number belongs to exactly one of these. They are deliberately kept separate
so a value is never silently compared against an incomparable one. Which surface a
harness serves, and which chapter it lands in, is the table at the top of this page —
joined from the instrument registry in bench/gen_results_page.py, so it cannot
disagree with the benches on disk.
Every harness in bench/,
including the ones whose results are not charted here — each one drives something specific
and reports it in its own units:
harness |
what it does |
reports |
where it lands |
|---|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. Every |
ns p50 / p99 / mean · deliveries/s |
§6, §7, §8, watched |
|
Drives one FWD forward hop through the inbound link’s own receiver against a registry of N links, with the target child registered first ( |
ns p50, batch-amortized |
§7, watched |
|
Drives the Nth |
ns p50 · allocations per frame |
§7, watched |
|
Replaces the global allocator with a counting wrapper and arms it around exactly one operation, so every number is an exact count rather than a sample. Eight probes: the forward hop, a terminus resolve, five resident-vertex shapes, and a wire-driven registration’s escape from the injected memory seam. Gate: forward hop = 0 allocations; per-vertex bytes +2%, block counts exact. |
allocations · live usable-size bytes |
§8, watched |
|
Decodes and re-encodes every shared v1 conformance vector through the C++ core — the |
ns per roundtrip · roundtrips/s |
§10 |
|
The Zenoh side of the in-process comparison: intra-session peer pub/sub over zenoh-c, sweeping the same fan-out / payload / endpoint matrix and emitting the same RESULT rows. |
ns p50 / p99 · deliveries/s |
§9 |
|
libtracer’s network latency: a publisher and a subscriber process on one host, one transport each (UDP / TCP / WebSocket), payload carrying a send timestamp so one-way latency is valid across the two processes. Every paced probe is kept, so the same run yields the deep tail and the sample count that backs it. |
ns p50 / p99 / p999 / max, one-way · samples per point |
§9 |
|
The Zenoh side of the same two-process network measurement, over a configured UDP endpoint with multicast scouting disabled so the pair talks only over that socket. It shares the subscriber-side accumulator with the libtracer arm, so both engines’ tails are estimated by the same code from the same number of samples. |
ns p50 / p99 / p999 / max, one-way · samples per point |
§9 |
|
libtracer’s composition-throughput arm: a publisher process ships a K-link rope as one |
values/s and messages/s observed · values per message · ns one-way per K-value group |
neither — on demand only |
|
The Zenoh arm of the same measurement, over a configured loopback UDP endpoint with multicast scouting disabled. It has no composite send, so the same K values are K puts; it shares the subscriber-side counter with the libtracer arm, so a value means the same thing on both sides. Its publisher is audited for wire use before any number is taken, so a run in which it never transmitted cannot report one. |
values/s and messages/s observed · values per message · ns one-way per K-value group |
neither — on demand only |
|
Forwards a frame that arrives as a multi-link rope, where TLV headers straddle link boundaries and every peek is a walk with stitching, swept over link count at a fixed frame and registry size. |
ns p50 · allocations per hop |
neither — on demand only |
|
Resolves the same frame through both reader tiers — the eager arena reader and the lazy rope reader — plus a flatten-then-arena arm, swept over frame size and link count. |
ns p50 · allocations per resolve |
neither — on demand only |
|
Drives the node that starts a remote operation: no inbound frame to read an address out of, so it encodes the |
ns p50 · wire bytes per frame |
neither — on demand only |
|
Five nodes and four hops: a full-path address against a minted label, and the cold first operation against the warm steady state, recording the frame bytes each hop actually carries so the address collapse is asserted rather than assumed. |
ns p50 per hop · wire bytes per hop |
neither — on demand only |
|
Carries one |
ns p50 per hop · originator frame bytes |
neither — on demand only |
|
Assembles the real transports’ |
allocations · bytes, per span count |
neither — on demand only |
|
Censuses the egress span counts the rope forward arm actually hands a transport, then times the gather at the spill width against an inline control pair at the same +1 delta. |
spans per frame · ns p50 |
neither — on demand only |
|
Drives real loopback TCP/UDP/WS sockets with thread-local allocation counters, so one frame’s egress cost on the sending thread and its ingress cost on the transport’s receive thread are attributed separately. |
allocations · bytes, per frame per direction |
neither — on demand only |
|
Stands up a real server transport and drives K raw client peers at it, reading the live heap balance when the server is up and quiesced, at K established connections, and after teardown. Gate: per-connection bytes drift past a measured ±8 B / ±0.5% band — WARN-first on the pinned host (bench/ram_census_pins.json carries the pins and the activation criterion for the flip to fail). |
live bytes per link · live bytes per connection |
§8, watched |
|
Prices a 100-vertex node — mixed int, array and stream values, 4–64 B each — stage by stage on the live heap balance: empty graph, vertices registered, values written, TCP listener up, a second-process peer connected, and the steady state after a mixed op storm, with a null arm and residual check bracketing every run. |
live bytes · blocks, per stage · bytes per vertex |
§8 |
|
Counts the blocks each peer-driven control-plane operation draws from the injected resource against those that escape to the global heap, and A/Bs the two growable-array guard shapes a nothrow migration chooses between. |
blocks per operation · ns per growth |
neither — on demand only |
|
Raises the number of simultaneous TCP peers against the server’s single poll thread and counts frames delivered, to find where that thread saturates before the host does. |
frames/s, per peer count |
neither — on demand only |
|
Times the WRITE store leg over a (payload x segment) grid with the copy branch and the pinned-subview branch rotating as interleaved arms inside one process, and reports each cell’s pin/copy split two independent ways so no timing is read off an arm that never reached the branch. Pairs with |
ns per store · pins and copies per cell |
neither — on demand only |
|
Sweeps the REGISTERED-VERTEX POPULATION over 10^3/10^4/10^5/10^6 and decomposes what grows with it. Each candidate leg is measured in isolation against the same population, in one process: a write through a pre-bound handle touching all N vertices, |
ns per operation, by leg · bytes per vertex |
neither — on demand only |
|
Prices the per-link departure index ( |
ns per index operation · bytes at rest per link |
neither — on demand only |
|
Drives the PRODUCER fan-out leg: one write to a vertex with N remote subscribers, which runs |
ns per write, median with range · deliveries and resolutions per write |
neither — on demand only |
|
Prices |
ns per delivery, p50 · canonical resolutions per delivery |
neither — on demand only |
|
Drives one whole forward hop — peek, mount descent, head rebuild, egress — through a registry of N mounts of width W segments, addressed to the last-registered mount, and counts the frames that actually reached the intended egress link so no cell’s timing is read off a hop that fell to the terminus instead. Written to compile unchanged against the pre-lift tree, so the two binaries interleave as A/B arms on one machine. |
ns per hop, p50 · hits vs iters per cell |
neither — on demand only |
|
Drives FWD{WRITE} across two processes over real UDP into a receiver whose RX backend is a bounded pool, counting deliveries in the receiver’s own graph and sampling the pool’s free-slot floor and refusal count while the load runs — so the throughput figure is a receiver’s count and the RAM figure is an observed exhaustion rather than arithmetic. |
deliveries/s · free slots · rx drops |
neither — on demand only |
|
Counts, per vertex shape, whether an extension block is allocated at all — the pay-for-what-you-use split RFC-0022 §3.B widened by deleting the registration parameter that could force one. Classifies through |
vertices per bucket |
neither — on demand only |
|
Separates the single poll thread’s three costs by moving one axis at a time: an |
frames/s + ns p50 / p99, per peer count |
neither — on demand only |
|
The same fan-in topology with timestamped payloads, on a fresh server per sweep point, so aggregate throughput and the full one-way latency distribution come from one run. |
frames/s · ns p50 / p99 / p999 / max |
neither — on demand only |
|
T threads clone and release one shared segment view — the per-subscriber delivery primitive, with T standing in for fan-out width — as T scales 1 → 128. |
clone+release/s, aggregate and per thread |
neither — on demand only |
|
One writer storms writes at a hot vertex while W threads each loop on |
writes/s · wakeups/s |
neither — on demand only |
|
T threads hammer |
ops/s, aggregate and per thread |
neither — on demand only |
|
T receive threads forward rope frames with the RX block source shared across all children, one pool shared, or one pool per child. One seam; the whole-node instrument is |
frames/s · ns p50, per thread count |
neither — on demand only |
|
Composes a whole node under four allocation-store configurations — the shipped all-heap baseline plus ADR-0079’s WIDE, MID and NARROW, at equal total slab — and drives one workload that draws all four of its allocation channels: a rope forward hop (router |
ns p50 · ops/s per thread · store bytes |
watched |
|
The memory half of the store sweep, in its own binary so a global |
heap bytes and allocations, per arm and window |
watched |
|
The same write at a vertex that RETAINS a last-known-value and at one that retains nothing, swept over subscriber width 0/1/4 and value link count 1/4 — the axis |
ns p50 · writes/s, per role, fan-out and link count |
watched |
|
The heap half of that question, in its own binary for the reason bench_store_escape gives: how many process-heap blocks one write costs at each role and link count. The count was the shed-on-OOM exposure — a HANDLER notify clone past the rope’s inline capacity drew a block on every write, and it was that block’s failure that shed an entire fan-out while the write still returned success. #1505 deleted the clone, so the HANDLER now reads exactly one block per write below the retaining role at every link count and the shed is impossible rather than counted. Gate: never gated — a decision input, reported as a count. |
allocations and frees per write, per role and link count |
watched |
|
Concurrent publishers and readers on one last-known-value slot across five reclamation arms, plus |
ops/s · ns p50, per thread count |
neither — on demand only |
|
Measures the machine rather than libtracer: nine arms isolating a thread-private counter, a shared read, contended read-modify-writes, false sharing and lock costs, so a claim about a shared line can be checked against the box in front of you. |
ns per op · ops/s, per thread count |
neither — on demand only |
|
Feeds the shared latency accumulator distributions whose quantiles are known by construction and prints measured against analytic. It touches neither the clock nor libtracer, so a failure is a defect in the estimator and nowhere else. |
quantile ns, measured vs analytic |
neither — on demand only |
29 of 43 harnesses reach neither a chart on this page nor a gate, by surface: Network timing — bench_compose_net, bench_zenoh_compose, bench_tcp_fanin, bench_pin_net, bench_tcp_peer_scaling, bench_tcp_baseline; Framed-hop timing — bench_forward_rope, bench_terminus_tier, bench_originate, bench_hop_chain, bench_path_label, bench_reply_leg, bench_mount_resolve; Allocation counting — bench_transport_iov, bench_iov_spill_cost, bench_wire_heap, bench_failable_census, bench_pin_ratio, bench_subscribe_index, bench_qos_census; In-process timing — bench_scale_sweep, bench_target_binding; Concurrency & scaling — bench_fanout_clone_storm, bench_await_wakeup_storm, bench_route_handle_contention, bench_rx_source_topology, bench_lkv_slot, bench_contention; Instrument self-check — bench_tail_validate. They are run on demand and read in a terminal. That is a deliberate state for the concurrency arms — they need a many-core host and a shared CI runner cannot produce a number worth recording — and a standing gap for the rest. A measurement that is neither published nor gated decays into a number nobody can check, so this count is derived here rather than kept in a list that could quietly go stale.
5 · Cross-core conformance (every native core must agree byte-for-byte)¶
How this surface is measured¶
The three native cores — the C++ golden reference, and the from-scratch
TypeScript and Rust reimplementations — must agree byte-for-byte. A shared set
of versioned conformance vectors (tests/conformance/vectors/v1) is decoded and
re-encoded by every enabled core; the driver diffs the results, and a single
DISAGREE fails CI
(ADR-0028).
This surface measures truth, not time — it is what lets every timed surface below
trust that a fast C++ number describes the same protocol the other cores speak.
Live driver summary:
cpp: 103/103 vectors ok
ts: 103/103 vectors ok
rust: 103/103 vectors ok
CONFORMANCE: PASS
6 · Dispatch — in-process latency & throughput¶
How this surface is measured¶
bench_libtracer exercises the hot path — resolve a vertex, write a value, notify
and deliver to subscribers — entirely in one process, and reports per-operation
latency (p50 / p99 / mean nanoseconds) and throughput (deliveries or
publishes per second). It sweeps three axes independently:
fan-out — subscribers per write (dispatch amortization);
payload — value size in bytes (copy cost);
topic count — number of registered vertices (registry / resolver pressure).
Several named modes isolate distinct costs on the same axes:
inproc— the full write (store + notify + deliver);inproc-borrow— the zero-alloc loaned-view path;inproc-deliver— deliver-only (propagate), value stored once;inproc-path— write-by-path, resolving the registry on every write. This is a deliberate resolver canary, not a hot pattern: real code resolves a path once and writes through the held handle. Judge dispatch cost againstinproc/inproc-borrow, never againstinproc-path.inproc-target-stored/inproc-target-handler— the path-target dispatch leg: edges carrying a target key instead of a callback, which is what a wireSUBSCRIBERproduces (the callback form is host-SDK sugar). Each delivery resolves the target in the registry, passes the fan-in ACL gate, clones the rope nothrow, and applies the target’s write effects — measured at roughly 10× a callback edge at fan-out. Thestored/handlerpair separates the target’s own store from the dispatch itself.
Every mode above except the inproc-target-* pair subscribes with an in-process
callback, so a fan-out curve reads the callback leg unless its mode says otherwise.
This is the surface that carries the microsecond thesis — the zero-copy substrate
(ADR-0016)
delivering values as loaned view_ts — and the one the per-PR gate watches most
closely.
harness |
what it does |
reports |
|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. Every |
ns p50 / p99 / mean · deliveries/s |
20 family charts · 392 series · x-axis = recorded
main commits (oldest → newest) · 🏷 dashed verticals mark release
tags (≈ = tag commit itself is not a recorded point; marker sits at the nearest
following recorded commit) · 🔧 dotted verticals mark commits where the BENCH
changed — points either side of one are not comparable. Each card carries every METRIC that point
recorded — p50 / p99 / ns per delivery / throughput — pick one under the title. Families with a numeric parameter
axis also offer trend / sweep / heatmap / 3D views — same data,
three axes (commit × parameter × value), over a selectable commit range. The paired
libtracer-vs-Zenoh cards add a ratio toggle: both engines run in the same pass on the
same runner, so their per-commit quotient cancels runner speed and is the comparison to read
across a long history. Hover any chart for exact per-commit values.
7 · Wire & routing — what a framed hop costs¶
How this surface is measured¶
The in-process surfaces above measure the graph. A separate set of benches measures the
network plane — what a frame costs between arriving and being applied — because the two
move independently and an improvement to one can hide a regression in the other. They cover
the three shapes a frame takes: a transit hop that never resolves, a cold terminus resolve
an established flow pays once, and the warm COMPACT frame that dominates a running system.
Every bench on this surface reports latency and exact allocation counts, and every one
calibrates its own batch size against the host clock rather than hardcoding a number: a
routing operation costs the same order as clock_gettime, so timing one operation at a time
measures the clock instead of the code.
bench_forward_demux and bench_compact_delivery are recorded into the build-to-build
history alongside the in-process series, so a routing or delivery regression shows up as a
trend rather than being noticed later. They emit the same RESULT rows as bench_libtracer,
so they need no separate aggregation. Their transcripts are tolerated-empty — a bench that
fails to run must not cost a commit its whole history point — but an empty one emits a build
warning naming the file, because a silently-empty transcript is otherwise a green job that
recorded nothing.
harness |
what it does |
reports |
|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. Every |
ns p50 / p99 / mean · deliveries/s |
|
Drives one FWD forward hop through the inbound link’s own receiver against a registry of N links, with the target child registered first ( |
ns p50, batch-amortized |
|
Drives the Nth |
ns p50 · allocations per frame |
6 family charts · 112 series · x-axis = recorded
main commits (oldest → newest) · 🏷 dashed verticals mark release
tags (≈ = tag commit itself is not a recorded point; marker sits at the nearest
following recorded commit) · 🔧 dotted verticals mark commits where the BENCH
changed — points either side of one are not comparable. Each card carries every METRIC that point
recorded — p50 / p99 / ns per delivery / throughput — pick one under the title. Families with a numeric parameter
axis also offer trend / sweep / heatmap / 3D views — same data,
three axes (commit × parameter × value), over a selectable commit range. The paired
libtracer-vs-Zenoh cards add a ratio toggle: both engines run in the same pass on the
same runner, so their per-commit quotient cancels runner speed and is the comparison to read
across a long history. Hover any chart for exact per-commit values.
Reading the two demux arms¶
fixed registers the target child first, so its lookup hits on the first compare and the
chart isolates the size-independent part of a hop. scan registers it last, so the lookup
walks the whole table. The rise of scan over fixed at the same registry size is the
scan’s marginal cost. Read them as a pair; neither means anything alone.
Per-module keying does not narrow that scan — it changes the key, not the container, and a node’s links overwhelmingly sit in one module. It earns its place by keeping two modules’ same-named connections distinct, which is correctness, not lookup time (ADR-0061).
What the scan costs a real frame¶
path |
registry work per frame |
|---|---|
plain |
mount descent + inbound lookup, per frame — the frame carries the full path, so there is nothing to cache against |
|
one dereference of the cached registry slot, first frame only — resolved once, then memoized |
remote delivery to a subscriber |
one lookup by link name, per frame — the subscriber record holds a name |
A binding holds the resolved target rather than a name, and both cached forms self-invalidate
without a callback: the terminus compares a retirement generation, and the forwarding hop
reads the registry slot, whose link teardown nulls in place — so a departed link reads
nullptr, the same clean miss an unresolved lookup gives (ADR-0062, ADR-0063).
The dominant term in a hop is TLV header parsing — the FWD header, the op, the dst PATH and
its segments, the selector peek and the src PATH on rebuild, each read exactly once — and it
sits at a local optimum for that structure. At -O3 the parse inlines with values flowing in
registers into the narrow struct’s stores, so there is no intermediate object to remove:
forcing the inline, or narrowing the header struct, each regress the hop by about 10 %.
A COMPACT re-emit builds no frame: the head goes to a 12-byte stack buffer and the payload
is handed to the transport by reference as a scatter-gather list, which is why the forward
charts are flat in payload size. Zero allocations in the router is not zero on the wire —
a transport that does not override the gather form concatenates once in
transport_t::send(iov).
8 · Memory & allocation¶
How this surface is measured¶
A different instrument entirely. bench_forward_heap replaces the global allocator
with a counting wrapper and arms it around exactly one operation, so these are
exact allocation counts and byte totals — not statistics, not sampling. Bytes are
read from malloc_usable_size, so a resident figure is what the allocator really
holds rather than what the caller asked for; whole-run max RSS comes from
/usr/bin/time -v and is the coarse process-level number beside them.
Two invariants sit on this surface, and the scope of the armed window is part of the
first one. The steady-state forward hop’s own work must touch no heap — the two-plane
forwarding model
(ADR-0038)
requires it — so bench_forward_heap arms the counting allocator around one hop and CI
runs it at ZEROHEAP_MAX=0: on a contiguous (single-link) frame the hop dispatches by
offset, builds its replacement heads in fixed stack buffers and its gather table in a stack
iov array, and a single stray malloc inside that window fails the job.
Two terms sit outside that window, so this is not the claim that a forward hop is
heap-free end to end. The gate drives capture_transport_t, a stub link that only sums the
spans it is handed, so the shipping transports’ ::iovec table is never assembled —
transport_udp and transport_tcp each hold 16 spans inline and spill to the heap above
that, measured at 17 caller spans / ~288 B by bench_transport_iov. And the
multi-link rope arm gathers into a block array drawn from the injected receive source,
because the sub-span count is the sender’s choice and is known only at run time — nothrow,
so an exhausted heap drops the frame rather than aborting, but not allocation-free. Read
the two benches together; neither is sufficient alone. A terminus, by contrast, may
allocate
(ADR-0041)
and is measured rather than gated. The per-vertex resident figure is the one the
constrained profile lives or dies by (a ~16 KB RAM budget on the ESP32 target),
which is why it is a series of its own rather than folded into RSS.
harness |
what it does |
reports |
|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. Every |
ns p50 / p99 / mean · deliveries/s |
|
Replaces the global allocator with a counting wrapper and arms it around exactly one operation, so every number is an exact count rather than a sample. Eight probes: the forward hop, a terminus resolve, five resident-vertex shapes, and a wire-driven registration’s escape from the injected memory seam. Gate: forward hop = 0 allocations; per-vertex bytes +2%, block counts exact. |
allocations · live usable-size bytes |
|
Stands up a real server transport and drives K raw client peers at it, reading the live heap balance when the server is up and quiesced, at K established connections, and after teardown. Gate: per-connection bytes drift past a measured ±8 B / ±0.5% band — WARN-first on the pinned host (bench/ram_census_pins.json carries the pins and the activation criterion for the flip to fail). |
live bytes per link · live bytes per connection |
|
Prices a 100-vertex node — mixed int, array and stream values, 4–64 B each — stage by stage on the live heap balance: empty graph, vertices registered, values written, TCP listener up, a second-process peer connected, and the steady state after a mixed op storm, with a null arm and residual check bracketing every run. |
live bytes · blocks, per stage · bytes per vertex |
Eight armed windows feed the store, each counted around exactly one operation:
probe |
what is armed around |
gate |
|---|---|---|
|
a |
hard — instrument self-check, aborts the run (#1420) |
|
the identical vector filled over the PROCESS HEAP — the escape must be SEEN, so an operator-new override gone blind cannot hand every zero below a free pass |
hard — instrument self-check, aborts the run (#1420) |
|
one FWD forward hop, from arrival to egress |
zero allocations, every CI run (ADR-0038 §16KB-RAM) |
|
one terminus resolve, which may allocate (ADR-0041) |
report-only |
|
a bare default leaf vertex at rest |
bytes +2% · blocks exact |
|
the same leaf plus the increment one small LKV write adds |
bytes +2% · blocks exact |
|
a leaf carrying an owning five-field app descriptor table (ADR-0058) |
bytes +2% · blocks exact |
|
the same table installed borrowed, its slots viewing caller flash |
bytes +2% · blocks exact |
|
one publish at a large subscriber count |
report-only |
|
a wire-driven |
report-only |
Every probe reports two independent quantities and both ratchet: bytes=, the live
usable-size balance, and allocs=, the number of heap blocks. They ratchet on different
terms because they are different kinds of number — bytes carry a 2 % tolerance since an
allocator size-class flip is not a regression, block counts carry none.
6 family charts · 73 series · x-axis = recorded
main commits (oldest → newest) · 🏷 dashed verticals mark release
tags (≈ = tag commit itself is not a recorded point; marker sits at the nearest
following recorded commit) · 🔧 dotted verticals mark commits where the BENCH
changed — points either side of one are not comparable. Each card carries every METRIC that point
recorded — p50 / p99 / ns per delivery / throughput — pick one under the title. Families with a numeric parameter
axis also offer trend / sweep / heatmap / 3D views — same data,
three axes (commit × parameter × value), over a selectable commit range. The paired
libtracer-vs-Zenoh cards add a ratio toggle: both engines run in the same pass on the
same runner, so their per-commit quotient cancels runner speed and is the comparison to read
across a long history. Hover any chart for exact per-commit values.
Note
Allocation churn is not resident footprint, and this page charts both — separately. Resident bytes are what a live object holds; the heap-footprint series are the transient churn of one operation; the timed rows are what it costs to get a block and give it back. Neither number bounds the other: a release can take per-frame allocations to zero and move resident bytes almost not at all. Churn also costs more on the target than these host figures show — glibc’s tcache serves a hot same-size malloc/free in tens of nanoseconds, an MCU allocator in hundreds — so a host reading of churn is a lower bound.
9 · libtracer vs Zenoh — measured, absolute¶
Both engines on one runner, best of 3 rounds, so the numbers are directly comparable on identical hardware: three in-process axes — subscriber fan-out, payload size and topic count — and a network latency comparison over the real loopback kernel path, one socket and one paced value per transport in two processes. The charts plot absolute throughput, latency and bandwidth as series on shared axes; there are no speed-up ratios.
libtracer is compiled from source at -O3; Zenoh is the upstream prebuilt zenoh-c 1.10.0
release binary that bench/fetch_zenoh.sh downloads, so its optimization profile is
upstream’s rather than a flag this repo sets. Both are optimized builds. Which rows do equal
work, why there is no network throughput comparison, and which transports are absent and
why, are all below — as are the three designated model boundaries, which say what the
two engines are respectively buying with the costs these charts show, and which include the
one axis where Zenoh is the better engine.
A side-by-side against Eclipse Zenoh (zenoh-c, peer mode). Both are
measured on the same runner in the same rounds — libtracer compiled from source at
-O3, Zenoh as the upstream prebuilt zenoh-c 1.10.0 release binary that
bench/fetch_zenoh.sh downloads (we do not build it, so its optimization profile is
upstream’s, not ours). The whole grid is swept 3 times and each point keeps its best
observation across those rounds (bench/best_of_rounds.py): contamination on a shared
runner is one-sided — a busy neighbour can only ever make a round slower — so the best
round is the estimator closest to what the code does, while a median would be an estimate
of how busy the machine was. Both engines are executed in the same loop the same number of
times; giving one arm N tries and the other one shot would be a thumb on the scale rather
than a measurement. The tail rows (p999 / max) are deliberately not reduced this
way — a minimum p999 would advertise the calmest moment the host ever had as though it
were a worst case — so those are left as measured. And the charts plot absolute
throughput / latency / bandwidth — both engines as series on shared axes. There are no
speed-up ratios: every point is a measured number you can read off directly. Fairness is
discussed in its own section below, because a naive put-vs-write comparison would be
misleading.
Designated model boundaries¶
Three properties of libtracer’s model are designed, not accidental, and each of them carries a cost this page measures. They are listed here so that cost can be read for what it is: the price of a capability that was chosen, not a defect awaiting a fix. Every comparison on this page — the Zenoh chapter above most of all — has to be read against them, because a comparison between two engines that solved different problems is only informative once the problems are named. Nothing below proposes a redesign; where a boundary has a mitigable constant, the mitigation is a measurement, and it is linked.
A boundary earns that name only from a measurement that isolates it, and the section is held to that standard in both directions. A number that moves for some other reason is not evidence for a wall, however convenient the shape of the curve — the first draft of boundary 1 argued from exactly such a curve and the isolating control arm refuted it. Where that has happened it is said outright below rather than quietly rewritten, because a boundaries section that only ever accumulates walls is an argument, not an instrument.
1 · Every distinct address is resident state. A vertex is not a name that a message
happens to carry — it is an object that exists between writes. That is what pays for the
last-known-value (a reader gets the current value without waiting for the next publish),
for await (a readiness sequence has to live somewhere), for composed reads across a
subtree, and for the per-vertex ACL. An engine that only routes keeps no such object and
so has nothing to scale in the number of addresses.
What residency costs is bytes. That is the axis the trade is paid on, and it is measured rather than derived: the decomposition sweep (#1485 / #1496) reads 132 B of heap and 140 B of RSS per vertex at 10⁶ vertices, converging from above as the upper tree’s fixed cost amortizes, against a computed ~120 B floor — +10 % over the arithmetic. A million addresses is 132 MB of heap and 140 MB of RSS. The O(#addresses) residency is the model and no footprint work turns it into O(1); the constant in front of it is fair game and is actively ratcheted on this page, bytes at +2 % and block counts exactly.
This is also where the boundary actually bites the target the project cares most about. On a wide host 132 MB for a million addresses is unremarkable; on the narrow end — the constrained MCU profile living inside a ~16 KB RAM budget — the per-vertex byte figure is the number that decides how many addresses a node may have at all. Memory, not latency, is the axis on which “every distinct address is resident state” is a real constraint.
Residency is NOT a per-operation LATENCY cost, and the control arm that decides this is unambiguous. Hold one million vertices resident and touch exactly one: a write costs 90 ns — the same 90 ns it costs with a thousand resident. Resolving a single hot address moves 70 → 90 ns across three decades. Descent is population-independent outright (ADR-0057, now verified rather than assumed): a fixed-shape probe address resolves in 60 ns at 10³ and 60 ns at 10⁶ warm, and 90 ns flat with its lines evicted between samples. Everything that grows, grows with the touched set, not the resident set.
Read that refutation at exactly its own width. It kills the claim that residency is a latency wall — which is what the earlier draft of this section asserted, and it was wrong on its own evidence. It does not say the residency model is free: the bytes above are the cost, and they are real. A boundary can be genuine on one axis and absent on another, and saying which is which is the whole job of this section.
The topic-count curve in the Zenoh chapter is not this boundary’s evidence — and the
completed comparison retires it. The fairness audit
(#1480) went looking in both
directions and reported exactly one axis where Zenoh was the better engine outright:
topic-count scaling, Zenoh’s p50 moving 220 → 230 ns (+5 %) across 1 → 8192 topics
against libtracer’s 140 → 190 ns (+36 %), narrowing our margin from 1.57× to 1.21×.
Both rows were measured correctly and neither is withdrawn. They were not the same
operation: the libtracer row (inproc-path) re-resolved the destination address inside
every timed iteration, while bench_zenoh published through a declared Publisher and
resolved nothing per put. A resolution term sat inside one arm and nowhere in the other, so
the narrowing could not be attributed to either engine’s topic scaling.
The topics-bound / topics-addr pair
(#1485, bench/run_topics.sh)
measures both spellings on both engines over one ladder, and it is now a result
rather than an instrument. 64 B payload, fan-1, deliveries counted at the subscriber,
best-of-5-rounds with both arm orders and both engine orders, two independent ladders
agreeing to ≤ 2 % on every arm:
p50 / mean ns per operation |
1 topic |
100 topics |
10 000 topics |
|---|---|---|---|
libtracer |
110 / 116 |
110 / 116 |
110 / 118 |
Zenoh |
210 / 214 |
210 / 215 |
220 / 232 |
libtracer |
140 / 140 |
170 / 174 |
190 / 196 |
Zenoh |
320 / 332 |
1 450 / 1 532 |
221 000 / 226 351 |
Read bound against bound — the like-for-like pair, and each engine’s own recommended spelling — libtracer moves 116 → 118 ns (+2 %) and Zenoh 214 → 232 ns (+8 %) across four decades of topic count, so the margin widens slightly, 1.84× → 1.97×. The audit’s “+36 % against +5 %” was our resolve-per-operation arm charted against their bound one. Matched, neither engine has a topic-count problem in the bound form.
Read resolve against resolve, the growth is on the other side and it is large.
libtracer’s per-operation find() descent moves 140 → 196 ns (+41 %) over the same
span — the audit’s +36 %, reproduced at the shape that actually produces it. Zenoh’s
undeclared Session::put moves 332 ns → 226 µs, and the completed run settles the
“is that a different code path or a scaling curve” question #1485 left open, in the only
sense that mattered: it is a curve, and a linear one. From 100 to 10 000 declared keys
the log-log slope is 1.08 — O(N) within the instrument — with a marginal cost of about
12 ns per additional declared key over the first decade and 23 ns over the last two.
It is of course reached by a different code path from Publisher::put; that is the arm’s
whole purpose. What it is not is an outlier, a congestion effect or a measurement artefact:
inside that path the cost is exactly the linear match a declared publisher exists to hoist.
So the honest statement of this axis, replacing the one above: both engines are near-flat in the bound spelling and both degrade in the resolve-per-operation spelling — libtracer by 41 % across four decades, Zenoh by a factor of about 680. The mitigation is the same on both sides and both APIs already offer it: bind, or declare, once. There is no axis here on which Zenoh scales better in topic count. There was an axis on which our own bench spelled its destination worse than theirs did.
Host, recorded with the numbers: 31 CPUs, no cc1plus alive at any 20-second sample of
either ladder, 1-minute load 1.4–3.3 throughout except one ~60–90 s excursion per ladder
(peaks 10.4 and 9.4, unattributed, on a shared desktop). Best-of-rounds is the specific
mitigation for one-sided contamination, and the two ladders’ ≤ 2 % agreement on every arm is
the evidence that it worked here.
Two caveats on that decomposition, because it is fresh. Its arms are not gated — they
are new and their run-to-run stability is unproven, and POINTS is a promise about
stability.
And the topics-* arms are not on the generated results page. One ladder costs about
nine minutes, almost all of it the Zenoh topics-addr rung at 10 000 keys, so wiring it
into the documentation job would add roughly twenty minutes to every docs build to
re-derive a result whose interesting content is structural. It is run by hand; the table
above is its output. Those numbers supersede the preliminary single-round, single-engine
Zenoh figures recorded on #1485, which were never quotable and are deliberately not
reproduced here. The decomposition beside them was taken on a quiet box (1-minute load
0.64–1.15, no cc1plus alive, 31 CPUs, best of 3 rounds with the arm order flipped on
alternate rounds, two full ladders agreeing to ~2 % at 10⁶).
One follow-up remains open and it is a ruling, not a redesign:
#1486, on memoizing
vertex_slot(), whose reverse scan is exactly O(total vertices) — measured at 450 ns at
10³ rising to 410 µs at 10⁶, taken under a shared mutex on every binding mint. That is
the number the ruling was blocked on; the ruling itself is #1486’s to make.
2 · Paths are routes, not location-independent names. A path is resolved from the vantage point of the graph doing the resolving; the same value can be reachable by different paths from different places, and a path handed to a peer does not carry a promise that it means the same thing there. This is the boundary that buys composition with no infrastructure: two graphs are joined by mounting one into the other, and the join needs no broker, no registry service, no name authority and no agreement between the parties beyond the mount itself. A location-independent naming scheme would make a path portable across vantage points, and would need exactly the infrastructure that absence is the point of not having. Comparisons that assume a global namespace are comparing against a system that has one.
3 · The producer’s write budget is fixed, fan-out is value-agnostic, and a producer
never queues. A write costs what it costs regardless of who is listening: fan-out
carries the value without inspecting it, and the depth a consumer wants is the consumer’s
own ring on the consumer’s own vertex, charged in bytes to a source that vertex injected
(RFC-0025 §4.6.1 Amendment 2). What this buys is the hot write path — the reason a write
is a sub-100 ns operation at all, and the reason its cost does not move when a subscriber
decides it wants history. What it forecloses is symmetric and worth stating: there is no
producer-side buffering to smooth a slow consumer with, no content-dependent routing
decision inside the fan-out loop, and no way for a subscriber to make a producer pay for
its own depth. The budget is instrumented rather than asserted — the write path’s compiled
size is ratcheted symbol by symbol (bench/symbol_ratchet.json), and a change that moves
it has to price the move on the bench before the pin is allowed to move with it.
Fairness in the comparison¶
An honest side-by-side has to account for the two engines doing different amounts of work per operation.
Deliveries are counted at the subscriber on both arms. Neither engine’s delivery figure is inferred from
publishes x fan-out. Each side’s timed window ends only once every delivery it owes has landed, each reports what its subscribers actually received, and each prints a warning naming any point that came up short. The two engines reach that guarantee differently, and the difference is not a handicap: libtracer dispatches inline, sowrite()returns only after the last subscriber callback and its publish loop is its delivery loop, whereas Zenoh delivers off the publishing thread and the harness spins on the receive counter inside the window until the backlog drains. This matters because the arithmetic form would not merely be imprecise, it would be unfalsifiable: libtracer’s wide-fan-out snapshot truncates to its inline prefix when its overflow reserve fails, and its HANDLER and STREAM legs shed an entire fan-out on a clone or ring-append failure — in every one of those caseswrite()still returns success, so apublishes x fan-outfigure would report deliveries that never happened and nothing in the harness would contradict it.The libtracer-only rows that audit deliberately left on the arithmetic — the
inproc-target-*pair andinproc-remote, which have no Zenoh row beside them — were finished on the same terms (#1481).inproc-target-handlerandinproc-remotecount at their consumer like every other row.inproc-target-storedhas no consumer to count at — its delivery terminates in the target’s last-known-value, and hanging a counting subscriber off each target to see it would add an edge per target to the topology being timed — so that one subtractsgraph_t::delivery_drops()from the ceiling instead. That is a measurement rather than a restatement of the arithmetic because RFC-0025 §4.4 forbids an unaccounted shed: every leg that drops one of those deliveries counts it.Write does strictly more than put. libtracer’s
writerow also persists the value (it becomes the vertex’s last-known-value) and bumps theawait/ readiness sequence on every op. Zenoh’sputis transient delivery only. So the libtracer write row is charted against a Zenoh row that does less semantic work — theinproc-deliver(propagate) series is the apples-to-apples counterpart: value stored once, each op only delivers, matching put semantics. Both libtracer series are shown so the reader sees the full-work and the like-for-like number side by side.The topic-count rows must agree on how the destination is spelled — and the pre-existing
inproc-pathpair did not. libtracer’sinproc-pathrow writes by address, so a registry resolution sits inside every timed iteration; the Zenoh row of the same name publishes through a declaredPublisher, which is the bound form and resolves nothing per put. A resolution term therefore sat inside one arm and nowhere in the other, and any narrowing of the margin along that ladder could not be attributed to either engine’s topic scaling. Thetopics-bound/topics-addrpair (bench/run_topics.sh, #1485) fixes it by measuring both spellings on both engines over the same ladder — pre-bound handle / declared publisher, and destination resolved inside the operation — so the resolution term is visible as its own difference instead of hidden inside one arm. What is compared is per-operation resolution: thepath_ton one side and theKeyExpron the other are both pre-built, because charging one engine for a string parse the other hoisted is the same mistake in the other direction. The completed two-sided run and what it does to the audit’s conclusion are in Designated model boundaries below; the short form is that bound against bound both engines are near-flat, and the axis the audit reported as Zenoh’s was the unmatched pair.ACL is disabled in the comparison rows. No subject resolver is installed, so the access gate is a single null check. The cost of enforcement is measured separately (the
acl-inheritrows), never hidden inside the comparison.No network throughput comparison is published. A valid one needs a real subscriber in a second process on both sides, with deliveries counted at the receiver rather than sends counted at the publisher — an engine whose publisher has no peer emits nothing to the wire at all, and a per-send rate multiplied by a composition width is arithmetic, not a measurement. That bench now exists (
bench/run_compose.sh): two processes per engine over real loopback UDP, the composition width K swept on both sides, and every rate taken from the subscriber’s own count over the subscriber’s own clock. It carries two guards, and reports nothing unless both hold — the receiver emits no row at all when it observed no values, any malformed record, or fewer throughput datagrams than the sample floor the driver hands it, and a wire-use audit runs the publisher understraceand fails the point below a send-syscall floor (COMPOSE_SEND_FLOOR, default 50), which is exactly the check whose absence let the withdrawn version report a rate for an engine that had only ever emitted scouting beacons. What is still absent is the chart: publishing this comparison is a claim, so it is a separate, reviewed step, and until it is taken the chapter states the gap rather than showing a number.Network latency is the surviving network comparison, and it is fair: a single-value, two-process, same-clock measurement over the real loopback kernel path, identical topology for both engines. p50, the p99 tail and the p999 deep tail are charted per transport — for a latency-first substrate the tail is the load-bearing number, and it is where transports separate, since an unreliable datagram path can win the median and still spike at p99.
The p999 has to earn its own publication. It is the cost of one message in a thousand — on a control path, the size of the deadline it misses — and it is the one figure on this page whose publication is gated rather than only its value. A p999 read off
nsamples is the order statistic atfloor(0.999·n), so onlyn − 1 − floor(0.999·n)samples lie above it: three at n = 4 000, and none at all at n ≤ 1 000, where “p999” is the maximum wearing a percentile’s name. The accumulator recordsnbeside every percentile and flags whether it clears its own adequacy floor; below the floor the number is withheld, and the shortfall is named in the transport-coverage note instead of being drawn as a line that moves for reasons the code never touched. The sample count, the number of samples above the p999, and the worst single message of the pass are printed under the charts, computed from the same rows that drew them.
A transport that is not charted always says so in the transport-coverage note under the charts, including when neither engine produced rows for it. WebSocket and QUIC are the standing cases: the WebSocket transport shows order-of-magnitude single-run p50 jitter under this bench that would make a published chart misleading, and QUIC needs the optional TLS module. An absent transport must never be readable as a tie.
Measured on the same runner in the same rounds — the whole grid swept
3 times with each point keeping its best observation, both engines in the same loop — so
this is like-for-like on identical hardware. libtracer is compiled from source at
-O3 here; Zenoh is the upstream zenoh-c 1.10.0 prebuilt release binary
(bench/fetch_zenoh.sh
downloads it — we do not build it, so its optimization profile is upstream's, not a flag we
set). Both are optimized builds. Absolute values on absolute
axes — no ratios, and every "reading" under a chart is computed from that chart's own
endpoints at render time. Read trends and orders of magnitude, not the third digit;
shared-runner variance is real. x-axis = the swept parameter (these are the only charts on
the page whose x-axis is not a commit).
- WS: not charted — libtracer's WebSocket transport shows large single-run p50 latency spikes (held until understood), and Zenoh has no WebSocket transport to compare against.
- QUIC: not charted — needs the optional
LIBTRACER_WITH_QUICmodule (msquic + TLS), gated like the dedicated quic CI job.
Each percentile here comes from 10,000 paced one-way probes at that payload, so at least 9 samples sit above the p999 and the worst single message of the whole pass was 2.9 ms. Across the charted points the p999 runs 1×–29× the p50 of the same point. Every charted point clears the accumulator's adequacy floor.
These are published, not gated. A two-process loopback tail is dominated by scheduler wake-up, not by either engine's code path, and it is unstable run-to-run by far more than the gate thresholds — measured, and quoted, in the noise chapter above. Read the deep tail as the jitter floor a real-time consumer on this topology inherits, and read engine-vs-engine on the p50 and p99 charts, which are stable enough to carry a comparison.
**🤖 CI-generated** on 2026-08-23 01:13 UTC · commit [`3d958fa`](https://github.com/avatarsd-llc/libtracer/commit/3d958fa1e9212c6b56d0a5311555b4b28810242c) · [run 32609336162](https://github.com/avatarsd-llc/libtracer/actions/runs/32609336162) · runner `Linux`.
10 · Cross-core codec performance (decode→encode roundtrip, same v1 vectors)¶
How this surface is measured¶
Each native core runs the same per-vector decode→encode roundtrip over the shared v1 vectors, reported as the median across all vectors (one decode + one encode = one roundtrip). Because the input is identical for all cores, this is a genuine like-for-like codec comparison across implementations; a core whose toolchain is absent in a given build degrades to a note rather than failing the docs build.
harness |
what it does |
reports |
|---|---|---|
Decodes and re-encodes every shared v1 conformance vector through the C++ core — the |
ns per roundtrip · roundtrips/s |
core |
throughput (median) |
p50 latency (median) |
mean (median) |
|---|---|---|---|
cpp-core |
1.7 M roundtrips/s |
606 ns |
615 ns |
ts-core |
0.6 M roundtrips/s |
1713 ns |
1823 ns |
rust-core |
2.1 M roundtrips/s |
496 ns |
500 ns |
11 · Test rollup (live ctest, unified with the perf surface)¶
suites |
passing |
wall time |
verdict |
|---|---|---|---|
222 |
222/222 |
78.56s |
✅ all green |
By area: Codec (L2/L3) 13/13 · Examples 70/70 · Graph (L4) 21/21 · Net (FWD plane) 21/21 · Substrate (L0/L1) 16/16 · Transport 9/9 · other 72/72. Full per-suite detail: Test report.
12 · Reproducing this page locally¶
The gates are same-runner by construction, so a local reading only means something against another local reading taken the same way:
# Build the bench Release (-O3) — same flags CI uses.
cmake -S bench -B bench/build -DCMAKE_BUILD_TYPE=Release
cmake --build bench/build -j
# Pin to a core, take the best of several runs, compare only same-machine numbers.
taskset -c 2 ./bench/build/bench_libtracer # the sweep matrix
taskset -c 2 ./bench/build/bench_forward_heap # the allocation probes (zero-alloc gate)
# The network plane (§3b). Each takes a per-point wall-clock budget via
# LIBTRACER_BENCH_SECONDS; longer means tighter percentiles, never a different measurement.
taskset -c 2 ./bench/build/bench_forward_demux # forward hop vs registry size
taskset -c 2 ./bench/build/bench_terminus_tier # terminus: eager arena vs lazy rope reader
taskset -c 2 ./bench/build/bench_compact_delivery # steady-state warm compacted delivery
# The comparison surface needs Zenoh vendored first:
bench/fetch_zenoh.sh && cmake --build bench/build -j
Compare a change against its own baseline on the same machine in the same
session (git stash, rebuild, re-run) — never against a number from a different
host or a different day, and never with one arm pinned differently from the other.
The full two-arm protocol, and the measurements behind each of its rules, is above
under The A/B protocol, in Reading the numbers. The single-CPU taskset -c 2 in the commands above is part of the measurement, not
decoration: this host is heterogeneous and an unpinned arm can land on a core class
50 % slower than its partner’s.
13 · Raw data & provenance¶
The charts above are one view of the two persisted stores this section describes — the
selector at the head of each chart block says which one you are reading. Every push to
main runs the full bench
on three independently-drawn runners, archives all raw transcripts as a per-commit CI
artifact (bench-results-<sha>, on the perf workflow run), and records every
(mode, size, fanout, endpoints) point — latency, throughput and memory footprint as
separate series — to a build-to-build history on the machine-maintained gh-pages branch
(benchmark-action/github-action-benchmark).
Per metric the recorded value is the best across the three runners, which approximates
the code’s capability rather than the machine lottery.
The store carries roughly three times as many series as this page charts — every recorded point, including the ones no family groups. Open the raw per-series trend browser ↗ — one chart per series, zoomable, with per-point commit links. It is the archive; the families above are the reading.
A second, parallel store records the same transcript set from a fixed self-hosted
machine (perf-local workflow), pinned to one logical CPU — the bench-local trend
browser ↗. Every family chart above
draws one store at a time, chosen by the source selector at the head of each chart
block (GitHub-hosted by default); switching redraws every chart on the page from the
other store and the choice is remembered. Both raw browsers are stock
benchmark-action pages with no selector of their own, so the deploy overlays a
three-way nav banner on the mirrored copies — each one names the other store and links
back here — rather than leaving a reader there believing the site has a single store. A family the selected store has not recorded
says so rather than drawing an empty axis. The two stores answer
different questions and are never mixed: GitHub-hosted runners vary ~2× in absolute speed
run to run, so the hosted store reads as a portability envelope (best-across-three-runners
per point), while the bench-local store is the absolute-trend instrument — same silicon
every point, host characteristics recorded on every point (each point’s tooltip carries the
host descriptor; host.txt next to the store holds the full lscpu capture). A two-point
regression verdict is only ever read from the bench-local store or from a same-host
interleaved A/B, never from the hosted one.
Provenance & auditability¶
Because the Performance page is regenerated on every docs build, each render carries
a CI stamp — date, commit, run, and runner OS — so any published figure is auditable
back to the exact deploy that produced it. Every main push additionally archives
all raw benchmark transcripts as a per-commit CI artifact and records every
(mode, size, fan-out, endpoints) point — latency, throughput, and memory — into a
persisted build-to-build history on the machine-maintained gh-pages branch. The
numbers on the Performance page are one run; that history is the durable signal, and
it is what the trend charts and the soft alert read from.