Performance & Conformance¶
Note
This page is auto-generated from the live test + benchmark harnesses on each docs
build (bench/gen_results_page.py, ADR-0032) — the published response surface, not a
hand-edited snapshot. All rates and latencies are absolute measured values,
representative of the CI runner. Each chapter opens with the method for its own surface;
the cross-cutting rules are §1–§3, and §4 says what
every harness in bench/ drives and reports.
🤖 CI-generated on 2026-10-06 07:08 UTC · commit b4ab133 · run 37426333810 · runner Linux.
How to read this page¶
Every number here belongs to exactly ONE measurement surface, and a value is only comparable to values from the same surface — surfaces use different harnesses, processes and units. A surface is a property of the series, not of the chapter it is drawn in: a chapter groups by subject, a subject can be answered by more than one instrument, and this table is what settles whether two numbers may be compared at all.
surface |
what it measures |
harness |
where it appears |
discipline |
|---|---|---|---|---|
Cross-core conformance |
byte-exactness across cores, not speed |
|
§5 |
any DISAGREE fails CI |
In-process timing |
per-operation latency and throughput in one process |
|
§6, §7, §8, on demand |
gated per PR and per |
Framed-hop timing |
what one wire frame costs, before and after the payload |
|
§7, on demand |
batch-amortized against the host clock, self-calibrating |
Allocation counting |
exact allocations and bytes around ONE operation — counted, never timed |
|
§8, on demand |
forward hop hard-gated at zero; per-vertex block count ratcheted exactly |
Network timing |
one-way latency over a real socket, two processes |
|
§9, on demand |
same topology for both engines; p50, the p99 tail and the p999 deep tail — published, never gated |
Engine comparison |
libtracer and Zenoh side by side, absolute |
|
§9 |
same runner, best of 3 rounds — no ratios |
Cross-core codec |
decode→encode roundtrip per implementation |
|
§10 |
the same v1 vectors for every core |
Concurrency & scaling |
how one shared object behaves as threads are added |
|
on demand |
many-core host, run on demand — not part of the published sweep |
Instrument self-check |
that the harness’s own estimator reproduces a distribution it was given |
|
on demand |
run before trusting a published percentile; touches neither clock nor library |
§2 is what stops a regression, §3 is what one number is worth against the runner lottery, and §4 says what each harness actually drives.
Every chart on this page is the same chart¶
One idiom, learned once. A family is a set of series answering one question, drawn as
lines on shared axes; the x-axis is recorded main commits, oldest to newest, except in
§9 where it is the swept parameter. Each chart names its own swept variable and
pinned scenario beneath its title — which is also why two charts both titled “throughput”
are not comparable to each other: different denominator, by construction.
🏷 dashed marker = a release tag (≈ when the tag’s commit is not itself a recorded point). 🔧 dotted = the bench changed at that commit, so points either side were taken with different rulers and are not comparable. Core changes are deliberately not marked: a core change moving the line is the signal the chart exists to show.
A series label keeps one color across every history chart. The §9 comparison charts use a fixed engine palette instead — their line dimension is which engine, not which parameter.
Families with a numeric parameter carry four switchable views — trend, sweep, heatmap and an isometric 3D surface — drawn only over the commits where every series of the family has a value, so a series that started late cannot fake a trend.
Every commit-axis view is drawn over a selectable commit range (two sliders per card, the full history by default). Narrowing it re-bases the axis and the release/instrument markers with it, so a short window is a real chart rather than a zoom.
The paired libtracer-vs-Zenoh cards (fan-out and payload) add a ratio toggle on the bench-local store: zenoh ÷ libtracer per recorded commit, dimensionless, with a parity line at 1×. On a latency metric a quotient above 1 means libtracer is faster; on throughput the same statement is a quotient below 1, and the y-axis says which. This is the comparison to read across a long history: on bench-local both arms of a point are measured in the same pass on one pinned CPU, so that machine’s speed on the day divides out of the quotient to first order, while the absolute lines carry the full spread. Each point pairs the two engines at one commit; a commit where only one arm recorded contributes no point. Measured over bench-local’s full store of 12 runs, the quotient’s spread is about a third lower than the libtracer line’s own. The ratio is computed from same-pass arms only: the hosted store records the best of three runners per series, so a point’s two arms need not be the same runner’s, and its cards carry no ratio toggle.
A
sourceselector heads each chart block: bench-local (the default — one pinned self-hosted CPU, the trend instrument) or GitHub-hosted (best of three runners per series, the portability envelope). One store at a time, page-wide, never overlaid — the two answer different questions (§13). A family the selected store has not recorded says so instead of drawing an empty axis.Hover any point for its exact value, the commit and its subject line — plus the host descriptor when the store records one, which the bench-local store does on every point. That descriptor carries the run’s clock floor: the clock’s resolution and the measured cost of one timed sample, the finest a per-op row could resolve on that host.
Two latency series per in-process mode¶
Every <mode> row is timed one operation at a time; every <mode>-batch row times a
calibrated batch of the same operation and divides. The clock is a large fraction of a
sub-100 ns write, so the per-op percentiles snap to coarse steps and read high at small
fan-out, converging on the batch row as the operation outgrows the clock — the Clock
quantization chart in §6 plots exactly that gap. Use -batch to resolve a
small delta; use the per-op row for tail shape, since it is the one with a real p99. A
percentile of batch means measures interference between batches rather than the tail of an
operation, so the -batch rows publish no p99 at all rather than a fabricated one. Every
batch row’s window is at least 20 µs and its figure is kept to the picosecond, so a 3 ns row
is not stepped in whole nanoseconds.
1 · What the metrics mean¶
Across the surfaces, a measurement is one of the following dimensions. Each has a distinct instrument, and a distinct rule for what a “worse” number means.
dimension |
unit |
instrument |
direction |
status |
|---|---|---|---|---|
latency |
ns (p50 / p99 / mean) |
wall-clock per op, |
lower better |
gated ✅ per-PR + push |
throughput |
deliveries/s, publishes/s |
ops / elapsed, |
higher better |
gated ✅ per-PR + push |
alloc bytes |
bytes & count per op |
counting allocator, |
lower better |
forward hop gated ✅ = 0; other probes tracked |
memory footprint |
live bytes / vertex, edge, link; RSS delta per family |
|
lower better |
gated ✅ per-vertex, edge and link (+2% same-runner); RSS tracked |
wire bytes |
encoded frame bytes |
TLV frame size over the v1 vectors, codec surface |
lower better |
being promoted to a first-class series |
CPU |
work per op |
per-op cost on a pinned core |
lower better |
latency is today’s proxy; dedicated counter planned |
Three notes on reading this table honestly:
Latency vs CPU. Today’s per-op cost is measured as wall-clock latency on a quiesced, core-pinned runner, which on an idle machine is a close proxy for CPU work. A dedicated cycles/CPU-time counter is a finer instrument for the same dimension; where a chart is labelled “CPU” it means per-op compute isolated from I/O and wait.
Wire bytes are a codec property, not a dispatch property. They come from the encoded size of a message over the wire (the TLV frame), measured on the same v1 vectors the codec surface uses — so they are comparable across cores and independent of runner speed.
Alloc bytes vs footprint. Alloc bytes is the transient heap a single operation churns (gated to zero on the forward hop). Footprint is the resident memory the graph holds at rest (per-vertex live bytes, whole-run RSS). A design can be zero-churn yet heavy at rest, or lean at rest yet allocation-happy per op — so the two are tracked separately and never summed.
2 · What actually stops a regression¶
Absolute nanoseconds vary ~2× with the runner drawn, so the gates are all same-runner relative comparisons, where machine speed cancels. Three jobs, three thresholds, one hard invariant:
mechanism |
when |
comparison |
threshold |
effect |
|---|---|---|---|---|
per-PR hard gate ( |
every PR |
PR build vs |
per row and leg, 3× the row’s A/A-null spread, floor 3% (flat p50 +15% · mean +12% · deliveries/s −12% where the null has no row) · per-vertex bytes +2% — and a bootstrap 95% interval on the median pair ratio that excludes 1 (a row at the flat threshold keeps the old disjoint-ranges + majority-of-pairs rule) |
fails the PR |
push ratchet |
every |
HEAD vs its parent, three independently-drawn runners |
same as above |
advisory — each replica reports (see the tier note below) |
forward-hop zero-alloc gate |
every CI run |
absolute |
|
fails the build |
soft trend alert |
per |
vs previous point, cross-runner |
series drifts past 125% |
a comment, not a verdict |
Verdict tiers. The comparison is the same wherever it runs; what a caller declares is
whether a breached ratchet may stop the job. perf_gate.py --tier blocking fails on a
breach, --tier advisory prints the identical numbers and the identical verdict line and
exits 0 — with a ::warning:: annotation, so an unenforced breach is never quiet. The tier
is not a second, looser threshold set: an advisory run reports exactly what a blocking
run would have said. The per-PR gate is blocking; the three push replicas are advisory,
because a single noisy runner should not carry a verdict alone and the ruled escalation
(two of the three replicas agreeing) is not built yet. Omit the flag and you get
advisory — the safe default for a maintainer’s own machine, where the bar was never
calibrated; every CI invocation declares its tier explicitly and a unit test fails the
build if one does not.
Details that make these trustworthy:
Only a clean run may PASS or FAIL. Every timed bench execution — inside
perf_gate.pyand in the transcript steps of both perf workflows — runs through one classifier,bench_conditions.py, which samples the conditions around it and records them: the measured CPU’s foreign time (busy time in/proc/stat— including irq and hypervisor steal — minus the bench process’s own CPU time), the bench’s involuntary context switches (thenonvoluntary_ctxt_switchescounter), and CPU pressure over the execution itself (the growth of the PSIsome total=stall counter between start and end, as a share of the wall time — notavg10, a 10 s moving average a short execution would inherit from the run before it). An execution shorter than 0.5 s is too short to measure a share on and is judged on foreign time alone, and a multi-threaded execution, which raises pressure itself, has its pressure recorded but not scored. On the pinned host the pressure that decides is the bench job’s own cgroup’scpu.pressure: its CPUs are cgroup-isolated, so host-wide/proc/pressure/cpusays nothing about them and is recorded for information only. On unpinned hosted runners, host-wide pressure decides. An execution is contended when foreign time exceeds 2 % of its window or pressure exceeds 5; a contended execution is re-run, up to three attempts, and the first clean one is kept. The per-PR gate judges each family’s own window (#1807): a pair whose execution stayed contended is dropped for that family, both arms, and the family is judged on its other pairs. Only when a family that emits a gated row loses more than two of its pairs (and, outside the per-PR gate, when any execution is still contended after its last attempt) does the gate printPERF: INCONCLUSIVEinstead of PASS or FAIL — the comparison’s numbers still print, marked?as unverified — and the blocking tier exits 3 so the job is re-run rather than merged on an unverified green. On the pinned host the same verdict is stamped next to the host descriptor on every recorded point (and so in every chart tooltip), and a contended sample carries theCONTAMINATEDflag: it stays in the store as evidence, but it never counts as coverage and never enters the rolling-drift baseline. GitHub-hosted runners cannot be isolated: they run unpinned, their conditions are read over the whole affinity set, and the interleaved A/B remains their primary defence. The allocation-count instruments (the zero-alloc gate, the memory probes, the RAM censuses) are exempt — load cannot move a count.The per-PR gate watches thirty-five canonical points — a representative slice of the fan-out / payload / topic sweeps plus a fold-width point, one per gated family (
inprocandinproc-borrowshare one), so a pullback on any of those legs is caught and not just the 1:1 write. They are not the whole dispatch surface, and this page should not be read as claiming they are:inproc-deliver, the-batchtwins and theeptype-*sweep’s two lean arms are measured and charted but ungated. A pullback confined to those ships without the gate objecting. They areperf_gate.py’sPOINTS, named here in full because this page is hand-written and a bare count says nothing about what is covered; each is keyedmode/payload/fan-out/endpoints:inproc/64/1/1— the canonical 1:1 write;inproc-borrow/64/1/1— its zero-copy twin, the same write handed a borrowed view instead of an owned copy;inproc/64/1024/1— the 1024-subscriber fan-out loop;inproc-path/64/1/8192— the resolver canary, one registry lookup per write across 8192 registered vertices;mixed/0/6/128— the composed topology, 128 topics whose fan-out varies 1–16 (mean 6) over payloads of 1 B–8 KiB, which is why its payload column reads 0;fold-b4/512/1/1— the L0 fold walk, 512 bytes held constant across four rope links and timed over a batch; andlkv-store-heap/64/1/1+lkv-store-pool/64/1/1— the L1 rope-to-contiguous copy (rope_t::materialize: one segment allocated from the backend plus the payloadmemcpy), against the default heap and against a pooled backend;lkv-store-heap/1024/1/1+lkv-alloc-heap/1024/1/1— the same copy and the bare alloc/free on the default heap at 1 KiB, where the heap backend’s segment layout changes; andinproc-target-handler/64/8/1+inproc-target-stored/64/8/1— the path-target dispatch legs at fan-out 8, edges carrying a target key rather than a callback, which is the leg a wireSUBSCRIBERactually takes; and three multi-threaded rows at four threads,inproc-mt4/64/1/4(parallel dispatch, one graph per thread),acl-inherit-d4-mt4/64/1/4(ACL-gated reads under a shared ancestor) andpoolalloc-mt4/64/1/1(the thread-safe pool’s alloc/free under contention); and seven 16 KiB payload-ladder rows, one per data-path family (#1806):inproc/16384/1/1,inproc-borrow/16384/1/1,lkv-store-heap/16384/1/1,lkv-store-pool/16384/1/1,eptype-stream/16384/1/1,compact-forward/16384/1/1andfwd-demux-value/16384/1/1(the forward hop keyed by its VALUE payload rather than its frame size); and eight store-latency rows frombench_store_sweep(#1869), the p50 of each of its four workload legs — the rope forward hop, a graph write, a composed subtree read, and all three — on the NARROW and the WIDE store profile:store-lat-narrow-net-fwd/32/1/1,store-lat-narrow-graph-write/32/1/1,store-lat-narrow-graph-read/32/1/1,store-lat-narrow-full/32/1/1,store-lat-wide-net-fwd/32/1/1,store-lat-wide-graph-write/32/1/1,store-lat-wide-graph-read/32/1/1andstore-lat-wide-full/32/1/1. The workload writes a 32 B value, the only size the sweep has, so no store row above 1 KiB exists to gate. These rows are batch medians in picoseconds, so the clock-tick guard does not apply to them; their mean is not gated. An A/A null of 40 runs on one pinned CPU put every row’s robust spread under 1.3 % of its median and failed none of 160 four-pair A/A verdicts, so they gate on the blocking tier like every other point, at the cost of about one second per gate.The multi-threaded rows are timed in their own invocation (
--family-set multi), after every single-threaded one, and that invocation is judged on foreign CPU time only (#1803). Their own threads queue behind each other on the pinned CPUs, which raises the bench’s own CPU pressure with nothing foreign present; scoring that pressure turned about one gate in three INCONCLUSIVE. A real intruder still shows as foreign CPU time and still makes the verdict INCONCLUSIVE. Every other invocation is judged on foreign time and pressure, as before.Those last two are here because of what happened without them (#1250): reshaping
rope_t::flatten’s wrapper cost 25–48% on every path throughmaterialize— branch and field writes,op_resolvereads, FWD COMPACT emission, the RX span sink — and no gated point at the time was downstream of that call, so the loss shipped with every gate green. They are read out ofRESULTrows the default sweep already emits, so they add no wall-clock. The two 1 KiB heap rows joined them after the same kind of miss (#1768): putting a segment’s header and payload in one block made the 64 B heap rows ~30% faster and the 1 KiB ones ~2x slower, because a 1072 B request misses glibc’s 1032 B per-thread cache, and with only the 64 B row gated the release went out green. Note the name:lkv-store-*measures the copy-store allocation, not the last-known-value slot. Eachlkv-*row times its whole loop as one block, so it carries one metric, its throughput; its latency columns read 0 (#1804) instead of repeating that same figure as a p50 and a mean.Beside those rows sits the ADR-0060 pool check, and it is not a speed ratio. The pool’s acceptance has one goal: it never takes an allocation from the system heap (#1695, maintainer clarification 2026-10-03). The LKV-ROUTE gate in
bench_forward_heapchecks exactly that. It runs the pooled alloc/free loop under the global operator-new counter and requires the pool arm to reach the heap zero times at 64 B and 1 KiB, with the heap arm as its control (it must reach the heap on every cycle, or the probe is blind). CI also runs it withLKV_ROUTE_BREAK=1, which routes the pool arm to the heap on purpose, and requires it to fail. That count is the only blocking pool check; a slower pool path is caught as thelkv-store-pool/64/1/1row above.The timed pool/heap ratio (
lkv-alloc-pooloverlkv-alloc-heap) is still printed, and never fails. It used to be gated, at a 2.0x floor with a 1.25x fallback threshold, and later paired against main in the same session (#1745). It measured the runner: across 254 best-of-3 readings on the CI runners with healthy code, the 64 B ratio ran 1.4x–6.5x (median 3.0x, bimodal), and main and an identical candidate read 2.1x and 1.6x in the same session. The per-PR report keeps the pairing so main’s same-session ratio sits beside the candidate’s.bench/test_perf_gate.pypins that no ratio fails.The
inproc-target-*pair is gated at fan-out 8 and nowhere else, for two measured reasons (#1077). Fan 8 sits exactly onvertex_t::kInlineFanout, the no-heap small-fan-out boundary, so it is the width that prices the narrow-fan snapshot path — which is where that issue’s ~+5.5% step appeared, invisible to every gate at the time. And a layout control (the same source rebuilt at three-falign-functionssettings) measured this row as the most layout-stable point of the whole sweep, 0.62% across placements against ~15% forfold-b4, so the false-red risk from code placement is near zero here. Both legs are gated becausestoredcarries the same shape ashandler; it reads reliably now thathost_guard.pyrejects the contaminated windows that once put its own A/A null at −14.7%. Like thelkv-store-*pair they add no wall-clock: the default sweep already emits both rows at every fan width, so this is two more keys read out of output already collected. Both gate on all three legs at the nominal +15% / +12% / −12%: a fan-8 row’s p50 and mean time the whole 8-subscriber publish (~640 ns and ~800 ns p50 respectively, against per-delivery costs of ~62–80 ns), so they sit far above the sub-100 ns band where the tick guard would demand an extra +25 ns absolute and blunt them the way it bluntsfold-b4.eptype-stream/64/1/1is on the list too, and the reason it is gated while its two siblings are not is the whole point of adding it.eptype-leanandeptype-lean-cachedare theinprocandinproc-borrowcode paths re-emitted under an endpoint-type name — already gated, twice over, so gating them again would buy correlated evidence rather than coverage.eptype-streamis not a re-emission: it is the only point on the list that declares a bounded history depth, and therefore the only one downstream of theSTREAMrole’s retention work. Nothing else on the list touches that path, so a pullback confined to retention was invisible to all fourteen predecessors, which is the same guard-gap shape aslkv-store-*and the compact/demux arms below. It costs no wall-clock: the default sweep already emits the row. And all three legs bite at the nominal thresholds — it measures ~190 ns p50 and mean over five best-of-rounds on a busy 31-core host, far above the sub-100 ns band where the tick guard would demand an extra +25 ns absolute.What that row prices changed underneath it, and the change is worth naming because the series name did not. Until
bdd1066bthe ring was the producer’s: a write to a stream vertex appended to its own history under the stripe mutex, before fan-out. Under RFC-0025 §4.6.1 Amendment 2 a producer never queues — the queue belongs to the party that wants depth, so the ring lives on the receiving vertex and the admission runs after the last-known-value publish rather than before fan-out. What the gated row times today is therefore the receiver leg: retire the entry the depth intent pushes out, admit the new one by reserving its retained width against that vertex’s own injected block source, and append. The row keeps its name on purpose. The rule above — renaming beats reinterpreting — governs a change in what the instrument measures; the bench source is untouched and the operation it drives is still “one 64 B write into a depth-16 stream vertex”. The mechanism beneath it moved, and a core change moving the line is exactly the signal the series exists to show.The append-and-admission leg is read as a gap, not as a series of its own.
eptype-streamandeptype-leanare the same write at the same payload, fan-out and topic count, emitted from the same pass of the same binary and banked as separate series on the Endpoint-type family card — so the distance between the two lines is the receiver-ring cost, over the whole recorded history, with the runner shared by both arms. Two things that leg does not price, stated because a reader would otherwise assume them covered: a reservation is handed straight on when a retiring entry has the shape the new one needs, so a steady uniform stream costs its source zerotry_alloccalls per write and this row measures the admission bookkeeping, not an allocator round-trip; and the resident footprint of the receiver’s ring state has no memory probe. A bench for the shape-changing admission path — where the carried reservation cannot be reused andtry_allocfires on every write — does not exist, and neither does one for the ring’s resident bytes. Both are gaps in this page, not numbers it is withholding.Fourteen of the thirty-five come from OTHER bench binaries, and they are here because of what happened without them (#1173):
compact-forwardmoved +41% across the v0.8.0 → v0.9.0 window while every gated point stayed flat, so the gate had nothing to object to. They arecompact-forward/64/1/1andcompact-terminus/64/1/1— the compact-delivery tier’s forward hop and its terminus, frombench_compact_delivery; andfwd-demux-fixed/61/1/1andfwd-demux-scan/61/64/64— the fixed-slot and scanning arms of the FWD demux, frombench_forward_demux; plus the two 16 KiB ladder rows from those binaries, named above; and the eight store-latency rows frombench_store_sweep, also named above. EachPOINTSentry names the binary that produces it. The compact and demux binaries emit the same 12-columnRESULTformat;bench_store_sweep’sRESULT_STORE_LATrow is folded into that shape by the gate. This costs three extra processes per arm per pair.The points are
modevalues of the benches described above, and the numbers are theirsize/fan-out/endpointcolumns. The binaries are run family by family (bench_libtracer --family NAME, and each sibling binary as one step), and each family’s pairs run back to back, ABBA —A B / B A / A B / …, eight pairs, alternating which one starts — so the two arms of a pair are seconds apart and a slow window in the machine is shared by both rather than donated to whichever one holds it. A single-threaded family is pinned to one logical CPU (the first ofBENCH_CPU, orBENCH_CPU_SINGLE); a multi-threaded family runs on the wholeBENCH_CPUset, sizes its thread count from that affinity mask, and runs after every single-threaded one. Because the baseline is the same PR’smainrebuilt on the same runner in the same pass, the comparison is machine-neutral.A point’s leg fails only when both of these hold, and the gate prints each, with the numbers: the median of the per-pair ratios candidate/baseline is past the row’s threshold (the effect is big enough), and a bootstrap 95% confidence interval on that median (2 000 resamples of the pairs, seeded) excludes 1 (the effect is real). The threshold is per row and per leg, from a banked A/A null (
bench/aa_null.json, written bybench/aa_null.py): 3× the robust spread of the gate’s own statistic between different builds of the same source (one source built at three function alignments, so code layout moves the way an unrelated change moves it), floor 3%, and capped at the flat thresholds above: a measured null may tighten a row, never loosen it. The allocator-cliff rows (cliff-alloc-heap,cliff-alloc-pool) take 5× instead of 3×: banked on the bench CPUs, three of them false-failed a held-out A/A replay at 3× and none at 5×, and raising their floor to 5% instead cleared fewer. A stable row gets as little as 3%, so a real 10% regression fails it. A row the null does not carry is gated on the flat thresholds, and the report saysflat. This replaced the disjoint-range and majority-of-pairs rules (#1807): three PRs that could not touch the rows they failed, and one with identical sources, failed at 15–22% with clean conditions, because within one session two builds of one source differ by a layout offset, and disjoint ranges read that offset as real. The null is banked on the gate’s own host and pin (bench.sliceCPUs 3–6, single-threaded families on CPU 3, the bench runner stopped while it measures): a null first measured on a shared runner CPU was tighter than the bench CPUs’ own spread and false-failed nine rows of a real gate run at the 3% floor. It is re-banked there when the CPU layout changes and when a gated row is added.Layout-sensitive, held at the flat threshold. On these rows 3× the banked spread is at or past the flat threshold, because builds of one source move them by several percent (the
-falign-functions=64build ranlkv-store-heap~40% slower), so they keep the verdict they had before the null existed — the flat threshold and the old rule (medians breach, the arms’ [min..max] ranges are disjoint, a strict majority of pairs breach) — and the report sayscap. A row the null does not carry is decided the same way. In the bench-CPU null the capped rows outside the cliff family are:acl-inherit-d4-mt4,eptype-stream/16384,fold-b4/512/1/1andpoolalloc-mt4(every leg);fwd-demux-value/16384,fwd-demux-scan,inproc/16384,inproc-borrow/64andinproc-path/64/1/8192(two legs); throughput on thelkv-store-*rows,lkv-alloc-heap/1024,inproc-borrow/16384andinproc-mt4; the mean oneptype-stream/64,inproc/64/1024/1andinproc-target-handler; and p50 on five of the eightstore-lat-*rows. 56 of the 98 cliff rows hold at least one leg at the cap. A same-source A/A between two layouts can still fail these rows, as it could before #1807; the null cannot remove that without loosening them, which was ruled out.The payload ladder (#1806). Every data-path family is swept over 64 B, 984 B, 985 B, 1 KiB, 4 KiB, 16 KiB and 64 KiB:
inproc,inproc-borrow, the fourlkv-*rows,eptype-stream, bothcompact-*arms andfwd-demux-value. 984 and 985 B sit either side of the heap’s one-block boundary (984 B plus the 48 B segment header is glibc’s 1032 B per-thread-cache ceiling). Rows a family did not have before run after its existing rows, so no existing row moves. Above 8 KiB the operation budget shrinks in proportion to the payload, and thecompact-*andfwd-demux-valueladder rows run at a quarter of their binary’s time budget, so the ladder adds seconds to a sweep rather than minutes. Only the 16 KiB rows are gated.The allocator-cliff family (#1806).
cliff-alloc-heapandcliff-alloc-pooltime one segment alloc/free, each family in its own fresh process, at every size of the cliff ladder: 960–1096 B in steps of 8 plus 985 B, and 2^k, 2^k − 48 and 2^k + 48 from 64 B to 64 KiB. They are batch rows (picosecond p50, every window at least 20 µs). The gate reads every size twice: against main at the same size (the paired rule above, at the p50 threshold), and against the next smaller size, where a step of more than 1.75× in the median and in a majority of pairs is a cliff. A cliff fails only when main does not have it; one main has too is printed as a warning. The exact half is the segment-draw ratchet:bench_forward_heapcounts what one heap segment asks the allocator for at every ladder size (draws, total bytes, largest block), which ismem_heap_request_size_testpromoted to a gate. No draw may pass the 1032 B fast-path ceiling unless the payload alone does, and no count may grow against main. Reverting the #1768 layout split fails it at 985 B (one 1033 B draw). The timed rows do not, on this host: with glibc 2.39 a 1033 B request costs about as much as the split’s two cached draws in a fresh process, so the 2x step the v0.17.0 sweep showed does not appear in an alloc-only row timed this way. The count is what holds the boundary.Each arm’s own spread across the pairs is printed, and the baseline arm’s worst spread is reported as the run’s drift figure — the baseline binary cannot be moved by the change under test, so its spread is the invariant control leg. It does not gate; it tells a reader whether the run was worth believing.
The same gate additionally checks five memory probes (
perf_gate.py’sMEM_POINTS, named here in full because this page is hand-written and a bare count rots):vertex— a leaf at rest, its tables drawn per object (see Per object, not per slab);vertex_value— the increment one LKV write adds;vertex_app5— a leaf carrying a copied five-field app-field table, drawn per object;vertex_app5_static— the borrowed twin of that table (ADR-0058), likewise; andreg_escape— the global-heap blocks a runtime registration takes that the graph’s injectedmemory_resourcenever sees (ADR-0039 / RFC-0014), whose target is zero. Each ratchets on two quantities: live bytes per vertex, tolerant at +2% because it is host-allocator-dependent, and the number of heap blocks, which is host-independent and therefore ratchets exactly — one extra block is a regression, full stop. All come from the counting allocator (bench_forward_heap), so they are exact rather than sampled and need no repetition at all, with the baseline binary probed same-runner via--baseline-bench-fwd. Supplying that binary for one arm and not the other fails the gate as a wiring error; supplying it for neither prints an explicitSKIP— a probe that cannot run never passes silently.The #1808 exact rows are gated beside them, on counts alone: the four RAM probes
edge_callback,edge_wire,linkandvertex_value_1k(perf_gate.py’sRAM_POINTS), the blocks per write, and the STREAM stripe-lock sections. Against main, no block or section count may grow at all and live bytes may not grow past the same +2%; a row main emits that the candidate does not fails. #1713’s claims are checked with no baseline: a steady STREAM write, a spilling one and four concurrent writers each take exactly one stripe-lock section per write, the steady write takes nothing from the global heap, and a write whose spill is refused delivers nothing until the next write.A per-vertex cost a ratified clause already prices is not a pullback, and the memory ratchet has one narrow way to say so: a charged step (
perf_gate.py’sMEM_CHARGED), declared per probe, in bytes, naming the clause that charges it. A charge absorbs at most its own bytes — a step of exactly that size passes, one byte more fails and the failure names the unpriced remainder — and it is printed on every run, spent or not, so an allowance can never be a gate that quietly moved. It also expires by construction: the paired baseline is built frommain, so once the step lands there the delta is zero, the charge prints asUNSPENT, and the entry is deleted. It is not a tolerance: it does not scale, does not accumulate, and is not a budget to spend later.The gate’s decision rules have their own unit tests (
bench/test_perf_gate.py), run in the same CI job before anything is timed, pinning both directions: the recorded false-failure sample sets must pass, and the recorded real regressions must still fail.The push ratchet re-runs that gate on three separate runner draws and requires the regression to reproduce — one noisy machine cannot fail
main, and a regression that slips through the PR gate still gets caught the moment it lands.The forward-hop zero-alloc gate is the one absolute gate: it is a structural invariant, not a speed target. Steady-state forwarding must allocate nothing, so the threshold is literally zero.
The soft alert compares across runners, so it is only a prompt to look at the trend, never a merge-blocker.
The per-PR gate never compares against a recorded number at all: both binaries are
in hand on the runner, so every sample on both sides is drawn in the same interleaved
rotation, minutes apart at most. (bench/perf_baseline.json still exists for the
local / root-commit path, where there is no baseline binary to interleave with. It is
host-specific and never committed as a fixed number, precisely so the gate can
never encode one machine’s speed as another machine’s target.)
Recent steps on bench-local¶
A step is a row that moves beyond its own threshold and holds there for 3 consecutive recorded points. The threshold is per row: 4× the row’s robust point-to-point spread, more than one tick of its clock, and never below 6%, the host guard’s A/A band. bench-local’s noise runs from about 1% to about 9% by row family, so one global percentage would either miss steps on quiet rows or fire on noisy ones. A single contaminated sample moves one point and is never a step. Each step is marked on its chart with ▲ (up) or ▼ (down) at the first commit that measured the new level; the merge that caused it is in the window between that commit and the recorded point before it. Rule and constants: bench/step_detect.py, which also prints the full list (python3 bench/step_detect.py --recent 20).
82 steps in the last 20 recorded commits on bench-local; the 40 largest, largest first:
Row |
Before → after |
First commit |
Merges in the window |
|---|---|---|---|
|
34 ns → 19.6 ns (-42%, better) |
#1835, #1830, #1800, #1756 +2 · |
|
|
29.4 M/s → 50.9 M/s (+73%, better) |
#1835, #1830, #1800, #1756 +2 · |
|
|
68.1 ns → 42.2 ns (-38%, better) |
||
|
68.1 ns → 42.2 ns (-38%, better) |
||
|
68.1 ns → 42.2 ns (-38%, better) |
||
|
14.2 M/s → 22.9 M/s (+61%, better) |
||
|
70.3 ns → 43.6 ns (-38%, better) |
||
|
69.5 ns → 43.4 ns (-38%, better) |
||
|
14.4 M/s → 23.0 M/s (+60%, better) |
||
|
69.2 ns → 43.6 ns (-37%, better) |
||
|
14.5 M/s → 22.9 M/s (+59%, better) |
||
|
43 ns → 68.2 ns (+59%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
43 ns → 68.1 ns (+58%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
43 ns → 68.1 ns (+58%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
10.6 µs → 15.9 µs (+49%, worse) |
||
|
21.2 M/s → 14.2 M/s (-33%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
47.3 ns → 70.3 ns (+49%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
49.5 ns → 69.2 ns (+40%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
20.2 M/s → 14.5 M/s (-28%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
359.3 µs → 292.0 µs (-19%, better) |
#1835, #1830, #1800, #1756 +2 · |
|
|
5.5 µs → 4.5 µs (-18%, better) |
#1835, #1830, #1800, #1756 +2 · |
|
|
128 ns → 156 ns (+22%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
162 ns → 134 ns (-17%, better) |
||
|
6.2 M/s → 7.4 M/s (+20%, better) |
||
|
156 ns → 130 ns (-17%, better) |
||
|
156 ns → 130 ns (-17%, better) |
||
|
130 ns → 156 ns (+20%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
6.1 M/s → 7.3 M/s (+20%, better) |
||
|
164 ns → 137 ns (-16%, better) |
||
|
158 ns → 133 ns (-16%, better) |
||
|
314.4 M/s → 372.7 M/s (+19%, better) |
||
|
3.2 ns → 2.7 ns (-16%, better) |
||
|
450 ns → 380 ns (-16%, better) |
#1842, #1758, #1827, #1759 +8 · |
|
|
23.1 M/s → 27.4 M/s (+18%, better) |
||
|
43.2 ns → 36.5 ns (-16%, better) |
||
|
113.8 M/s → 134.0 M/s (+18%, better) |
||
|
8.8 ns → 7.5 ns (-15%, better) |
||
|
17972 KB → 21168 KB (+18%, worse) |
||
|
135 ns → 159 ns (+18%, worse) |
#1842, #1758, #1827, #1759 +8 · |
|
|
195 ns → 166 ns (-15%, better) |
3 · Noise, variance, and what one number is worth¶
Runner lottery. Shared CI runners vary ~2× in absolute speed. The tell: a move that hits every series at once — including unrelated ones like the L0 rope-walk
fold-b*rows — is the runner; a move confined to one family is the code. Read trends across several commits, not the third digit of one point.Per-point noise floor. Each recorded point is the median of the repeated RESULT rows one run emits, so per-iteration jitter does not move a series. Points are then recorded from the best of three runner draws — that runner’s whole tuple (p50, p99 and throughput from one machine, never a per-metric minimum across runners, and never a minimum of p99s), approximating the code’s capability rather than the machine lottery. Sub-microsecond points sit on a ~10 ns timer grain — do not over-read a 5 ns wiggle.
Tail percentiles are published, not gated — and here is the measurement that decided it. The deeper into a distribution a statistic reaches, the fewer samples stand behind it and the more it moves for reasons that are not the code. Running the same binary against itself and taking the worst ratio between repeats gives the floor any threshold on that leg would have to clear before it stopped firing on its own noise. On the in-process gated points (18 runs, replayed through the gate’s then-current best-of-3 estimator, worst of 15 disjoint pairs — the numbers below are a property of the machine, not of the estimator, so the interleaved gate inherits them unchanged):
leg
worst same-binary ratio
threshold it would need
gated today
p50
1.16×
+16 %
yes, at +15 %
mean
1.22×
+22 %
yes, at +12 %
deliveries/s
1.36×
−36 %
yes, at −12 %
p99
1.67×
+67 %
no — published only
A p99 gate would have to sit near +67 %, and a regression that large has already tripped the +15 % p50 gate several times over: the leg would add no detection while adding a new false-fail source on every PR.
On the two-process network bench, the same experiment run at two probe counts isolates how much of the tail’s instability is simply too few samples (5 repeats each, same idle host, only
nchanged):probes per point
p50
p99
p999
worst sample
4 000
1.09×
5.01×
62×
21×
10 000 (published)
1.09×
1.40×
17×
13×
The median does not care — it is 1.09× either way — while the p99 tightens 3.6× and the p999 3.7× purely from sample count. That is why the published run pays for the larger count. Even so the deep tail stays 17× unstable: a loopback p999 is dominated by scheduler wake-up, not by either engine’s code path. Read the p999 chart as the jitter floor this topology inherits, and read engine-against-engine on the p50 and p99 charts, which at the published probe count are stable enough to carry a comparison. (Measured on an idle 24-core host; a shared CI runner is not quieter than that, so these are lower bounds.)
A throughput pullback with flat latency is a machine, not a regression. Every gated point is measured by two instruments over the same operation — a per-op clock (p50, mean) and a bulk timer (deliveries/s) — and a change in what the operation costs moves both. When only the bulk timer moves and both latency legs come back flat-or-better, the two instruments contradict each other, and the gate reports the contradiction rather than failing on it. This is not hypothetical: one PR that touched only L4 was failed at −33 % throughput on an L0 rope-walk point it has no call path to, with p50 identical and the mean better. The guard is deliberately narrow — any upward move in either latency leg, of any size, leaves the failure standing.
Sign conventions in the history store. The latency suite is smaller-is-better nanoseconds; throughput also appears there inverted as
ns/deliveryso a slowdown always charts as a rise; memory metrics live in that same smaller-is-better suite. The throughput suite is bigger-is-better naturaldeliveries/s. The same measurement can therefore appear twice, in two units — by design, so each suite reads monotonically.A discontinuity in the instrument is annotated; a discontinuity in the code is not. When a commit changes a bench source —
bench_libtracer.cpp,bench_forward_demux.cpp, the shared harness, the emitter — every series that bench feeds is marked at that commit, because points either side of it were taken with different rulers and are not comparable. Changes undercore/**are deliberately never marked: a core change moving the line is the signal the series exists to show, and marking both would reduce the annotation to “something happened,” which is worth nothing. Comment- and whitespace-only bench edits are filtered out for the same reason — an alert that fires on nothing trains readers to ignore it.Renaming beats reinterpreting. If a fix changes what a row measures, the mode string changes too: the old series ends visibly and a new one starts at one point. What is never done is keeping the name while changing the meaning, which produces a continuous-looking line that silently stops being about the same thing. If a fix only makes the same quantity more accurately measured, the name is kept and the instrument marker carries the discontinuity.
Across rounds, take each arm’s BEST — never its median — and read the range as a contamination diagnostic. Contamination is one-sided: a busy neighbour can only make a round slower, never faster. A low order statistic across an arm’s rounds therefore rejects a dirty round, while the median merely counts them, and the median flips as soon as half the rounds are dirty — which is why an ad-hoc driver estimates best-of-N. The per-PR gate does not: it runs the two binaries in interleaved pairs and compares medians, and fails only when the medians breach the threshold, the two arms’
[min..max]ranges are disjoint, and a majority of pairs breach on their own (perf_gate.pypaired_verdict); best-of-N survives only in its legacy mode (a recordedperf_baseline.json, no baseline build). Measured on the pinned host with one binary against itself (#1358): in a window where a neighbouring job got busy, median-of-rounds putfwd-rope-hopat −33 % … +54 % while best-of-rounds on the very same samples stayed inside 1.44 %; quiet, both estimators agree inside ±0.5 %. The[min..max]range is what tells the two situations apart — a spread near 2× whose ends sit in two clusters rather than spreading smoothly is a contaminated window, and the run must be repeated rather than reported, whatever the medians say.The same estimator rule governs the series, and a gap in it is a measurement defect. Points of the fixed-host store are rounds of a slower experiment, so they inherit both halves of the rule above. First, a baseline drawn from them is the window’s best, never its median:
store_guard.pycompares each banked point with the best of the last 20 trusted points, which is the comparison benchmark-action’s point-to-pointalert-thresholdcannot express — five merges of +4 % never trip a 15 % step, and the series ends 22 % slower with no alert ever raised. A best is a floor, though, so a breach must also land outside that window’s whole[min..max]: on the real store the percentage bar alone reports 59 breaches where the range guard leaves 2. Second, a flagged point is not a sample — it answers what the machine was doing, not what the code cost — so it neither sets the baseline nor counts as coverage. And coverage is itself guaranteed: a run that never happens (filtered out, failed, or cancelled while queued) leaves a hole nothing notices, which is how one +41 % step became a 16-merge range instead of a commit (#1173). A daily catch-up measures HEAD whenever the last trusted point is more than 8 bench-relevant merges behindmain, so the guarantee is a bound on the gap — narrow enough to bisect — rather than a point per commit.A banked series is a TREND instrument; the paired same-runner A/B is the gate. The two stores answer “where has this point been going” across machines and months. They do not answer “did this commit cost anything”, and reading them as if they did produces verdicts the code never earned. A textbook case, both halves measured on the same day: the rolling drift check on the banked series warned that
inproc 64B/fan1/1epp50 had gone to 400 ns against a 100 ns baseline (+300%), while the interleaved same-runner A/B on the identical commit read 1.02×. Nothing regressed — the banked baseline and the banked point were taken on different machines under different load, and the quotient of two absolutes across that gap is instrument drift wearing a percentage sign. So: a drift warning is a prompt to measure, never a verdict; a verdict comes only from two arms interleaved on one runner in one session, which is what the per-PR gate is and what every ratchet in the table above compares. This holds for every banked series on this page, the memory ones included.An absolute without its host is not a measurement, and this page never publishes one. Load alone is worth more than most of the effects anyone argues about: the same gated point read 3.6 M deliveries/s while CI was building on the box and 6.0 M/s quiet — 1.6×, no code between the two. Every absolute here therefore names the machine and the conditions it was taken under, and a figure that arrives without them is unusable rather than merely imprecise. The worked example is #1495, now corrected: RFC-0025 §4.6.2’s evidence appendix stated its figures as bare cross-machine absolutes, and unmodified
mainmissed two of them by 1.40× and 1.71× purely because nobody recorded which host they were cut on. The 2026-08-24 erratum to RFC-0025 restates that section as portable premiums and ratios and demotes the absolutes to a host-stamped basis — the same move this bullet asks for, applied to the document that most needed it. Note what the defect was not: §4.6.2 is informative, so those numbers were never normative caps, and the “53–55 ns” / “82–84 ns” figures quoted against them were derived downstream, never stated by the RFC. Where this page mentions them it links the erratum; it does not report them as met.Record the load context on both sides of every measurement, and wait for quiescence first.
python3 bench/host_guard.py waitbefore the run and/proc/loadavgeither side of it: an absolute figure without its load context cannot be re-judged later, and this is the fact that the same pair of binaries reads inside ±0.5 % at load average ~2 and up to +81 % at load average 16–36. The contaminant that matters is memory-subsystem contention, not runnable-task count — five pure CPU spinners (load average 6.8, just under the guard’s bar on the 31-CPU host) cost 0.83 %, while memory-bandwidth neighbours at the same nominal pin cost tens of percent.A per-op figure is only as honest as the batch it was timed in. Timing a batch and dividing charges the window’s two clock reads (~22 ns on the pinned host) once per sample, so the batch size is part of the measurement: forcing it on a ~238 ns operation reads 260 / 250 / 242 / ~238 ns at batch 1 / 2 / 4 / ≥8. A calibrator that picks the batch by comparing two timed quantities therefore lets the machine choose the answer, discretely —
bench_common.hpp’scalibrate_batch_for_windowdoubles until the window reaches 20 µs instead, so the batch follows the operation’s own cost and repeats across executions. Every batch row now times through one loop (time_batches, #1804): every window is asserted to be at least 20 µs, each per-op figure is kept in picoseconds rather than integer-divided to whole nanoseconds, and no batch row publishes a p99. Each run records the host’s clock floor — the clock’s resolution and the measured cost of one timed sample — and the bench-local store carries it in every point’s tooltip.
The A/B protocol — what a two-arm comparison may and may not vary. An A/B of a code change runs two binaries and attributes the difference to the change. That attribution is only sound if nothing else differed. Two things that look harmless routinely do.
Build directory: harmless, and now measured. A cross-worktree A/B — build
origin/main in one worktree, the change in another, interleave — was suspected
(#807, from PR #806) of measuring code layout rather than code: identical
origin/main source built at two paths appeared to differ by +1.7 % (65589 B
frames) to +6.7 % (53 B frames) on bench_terminus_tier’s terminus-arena leg
over 12 interleaved rounds, with disjoint ranges. That cause is refuted. Building
the same commit at two paths of different lengths produces, on this toolchain,
byte-identical output: all 27 libtracer object files, libtracer.a, and the
bench_terminus_tier executable cmp equal (one md5 for both arms). There is no
layout to be sensitive to, so no layout lever is warranted, and none ships —
-falign-functions=64 over the whole bench + library build reads −1.50 % to
+1.56 % against stock across 10 interleaved rounds, sign varying by frame size,
i.e. nothing. Re-run at two paths under this protocol the same leg reads −0.64 % to
+0.87 % (unpinned) and −0.64 % to +0.73 % (pinned), 12 interleaved rounds.
CPU placement: the real hazard, and it is large. The reference host is
heterogeneous — 4 Zen 5 cores at 5.16 GHz (cpu0–7) and 8 Zen 5c cores at
3.29 GHz (cpu8–23). The same binary on the terminus-arena leg reads +47.0 % to
+53.7 % slower pinned to a compact core than to a classic one, 5 rounds each, ranges
disjoint and tight within each arm (e.g. 53 B: 308–313 ns vs 474–481 ns). That is the
signature a confounded A/B wears: tight, reproducible, and entirely about the machine.
Even the choice of pinning moves the figure — at 53 B, 15 runs each: unpinned median
228 ns, taskset -c 2 234 ns, taskset -c 2,3 238 ns (+4.4 % across pinning
policies, from SMT sharing). A few percent between two arms needs no exotic
explanation on such a host; it needs only that the two arms were placed differently.
So the rules for any latency A/B, on this leg or any other:
Pin both arms identically, to the same single logical CPU on the same core class (
taskset -c 2). Pinning one arm and not the other, or to different core classes, is not a comparison.Interleave the arms round-robin within one session and report medians and ranges of at least ~10 rounds per arm. A single round per arm cannot separate the change from the machine.
Discard the first execution. A cold process’s first measured point on this leg reads ~313 ns against a 228 ns steady state (+37 %) — it is idle-state wake-up, not code.
Prefer same-directory A/B (
git stash, rebuild, re-run in one session) when it is available; it varies strictly less than a two-worktree run.When the expected effect is smaller than the leg’s own noise floor, do not reach for a stopwatch at all — use object-file
cmpagainst the baseline tree (the method PR #799/#806 established). A change that leaves 25 of 27 objects byte-identical has proved more about the unchanged paths than any bench can.
The stack-offset hypothesis was also tested and is negative: sweeping the environment block from 0 to 4000 bytes (which shifts the initial stack pointer, and with it the bench’s 64 KiB stack slab) moves the leg 2.8–4.4 %, non-monotonically — indistinguishable from run-to-run noise.
4 · The measurement surfaces, and the instruments behind them¶
Every number belongs to exactly one of these. They are deliberately kept separate
so a value is never silently compared against an incomparable one. Which surface a
harness serves, and which chapter it lands in, is the table at the top of this page —
joined from the instrument registry in bench/gen_results_page.py, so it cannot
disagree with the benches on disk.
Every harness in bench/,
including the ones whose results are not charted here — each one drives something specific
and reports it in its own units:
harness |
what it does |
reports |
where it lands |
|---|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. An |
ns p50 / p99 / mean · deliveries/s |
§6, §7, §8, watched |
|
Drives one FWD forward hop through the inbound link’s own receiver against a registry of N links, with the target child registered first ( |
ns p50, batch-amortized |
§7, watched |
|
Drives the Nth |
ns p50 · allocations per frame |
§7, watched |
|
Replaces the global allocator with a counting wrapper and arms it around exactly one operation, so every number is an exact count rather than a sample. Eight probes: the forward hop, a terminus resolve, five resident-vertex shapes, and a wire-driven registration’s escape from the injected memory seam. Gate: forward hop = 0 allocations; per-vertex bytes +2%, block counts exact. |
allocations · live usable-size bytes |
§8, watched |
|
Decodes and re-encodes every shared v1 conformance vector through the C++ core — the |
ns per roundtrip · roundtrips/s |
§10 |
|
The Zenoh side of the in-process comparison: intra-session peer pub/sub over zenoh-c, sweeping the same fan-out / payload / endpoint matrix and emitting the same RESULT rows. |
ns p50 / p99 · deliveries/s |
§9 |
|
libtracer’s network latency: a publisher and a subscriber process on one host, one transport each (UDP / TCP / WebSocket), payload carrying a send timestamp so one-way latency is valid across the two processes. Every paced probe is kept, so the same run yields the deep tail and the sample count that backs it. |
ns p50 / p99 / p999 / max, one-way · samples per point |
§9 |
|
The Zenoh side of the same two-process network measurement, over a configured UDP endpoint with multicast scouting disabled so the pair talks only over that socket. It shares the subscriber-side accumulator with the libtracer arm, so both engines’ tails are estimated by the same code from the same number of samples. |
ns p50 / p99 / p999 / max, one-way · samples per point |
§9 |
|
libtracer’s composition-throughput arm: a publisher process ships a K-link rope as one |
values/s and messages/s observed · values per message · ns one-way per K-value group |
neither — on demand only |
|
The Zenoh arm of the same measurement, over a configured loopback UDP endpoint with multicast scouting disabled. It has no composite send, so the same K values are K puts; it shares the subscriber-side counter with the libtracer arm, so a value means the same thing on both sides. Its publisher is audited for wire use before any number is taken, so a run in which it never transmitted cannot report one. |
values/s and messages/s observed · values per message · ns one-way per K-value group |
neither — on demand only |
|
Forwards a frame that arrives as a multi-link rope, where TLV headers straddle link boundaries and every peek is a walk with stitching, swept over link count at a fixed frame and registry size. |
ns p50 · allocations per hop |
neither — on demand only |
|
Resolves the same frame through both reader tiers — the eager arena reader and the lazy rope reader — plus a flatten-then-arena arm, swept over frame size and link count. |
ns p50 · allocations per resolve |
neither — on demand only |
|
Drives the node that starts a remote operation: no inbound frame to read an address out of, so it encodes the |
ns p50 · wire bytes per frame |
neither — on demand only |
|
Five nodes and four hops: a full-path address against a minted label, and the cold first operation against the warm steady state, recording the frame bytes each hop actually carries so the address collapse is asserted rather than assumed. |
ns p50 per hop · wire bytes per hop |
neither — on demand only |
|
Carries one |
ns p50 per hop · originator frame bytes |
neither — on demand only |
|
Assembles the real transports’ |
allocations · bytes, per span count |
neither — on demand only |
|
Censuses the egress span counts the rope forward arm actually hands a transport, then times the gather at the spill width against an inline control pair at the same +1 delta. |
spans per frame · ns p50 |
neither — on demand only |
|
Drives real loopback TCP/UDP/WS sockets with thread-local allocation counters, so one frame’s egress cost on the sending thread and its ingress cost on the transport’s receive thread are attributed separately. |
allocations · bytes, per frame per direction |
neither — on demand only |
|
Stands up a real server transport and drives K raw client peers at it, reading the live heap balance when the server is up and quiesced, at K established connections, and after teardown. Gate: per-connection bytes drift past a measured ±8 B / ±0.5% band — WARN-first on the pinned host (bench/ram_census_pins.json carries the pins and the activation criterion for the flip to fail). |
live bytes per link · live bytes per connection |
§8, watched |
|
Prices a 100-vertex node — mixed int, array and stream values, 4–64 B each — stage by stage on the live heap balance: empty graph, vertices registered, values written, TCP listener up, a second-process peer connected, and the steady state after a mixed op storm, with a null arm and residual check bracketing every run. |
live bytes · blocks, per stage · bytes per vertex |
§8 |
|
Counts every allocation and byte one publish costs, stage by stage along the value path — a local write with callback subscribers, the same write to a vertex that retains nothing, the same write fanned to K stored target vertices and to K HANDLER targets, a producer handing over its own bytes, a FWD{WRITE} through the terminus below and at-or-above the copy-or-share threshold, egress through the base scatter-gather, and the RFC-0028 one-block prototype — over a warm window of 1,000 publishes. |
allocs · heap bytes · payload copies, per publish per stage |
§8 |
|
Counts the blocks each peer-driven control-plane operation draws from the injected resource against those that escape to the global heap, and A/Bs the two growable-array guard shapes a nothrow migration chooses between. |
blocks per operation · ns per growth |
neither — on demand only |
|
Raises the number of simultaneous TCP peers against the server’s single poll thread and counts frames delivered, to find where that thread saturates before the host does. |
frames/s, per peer count |
neither — on demand only |
|
Times the WRITE store leg over a (payload x segment) grid with the copy branch and the shared-subview branch rotating as interleaved copy-or-share THRESHOLD arms inside one process (RFC-0028 §5.3), and reports each cell’s share/copy split two independent ways so no timing is read off an arm that never reached the branch. Pairs with |
ns per store · pins and copies per cell |
neither — on demand only |
|
Sweeps the REGISTERED-VERTEX POPULATION over 10^3/10^4/10^5/10^6 and decomposes what grows with it. Each candidate leg is measured in isolation against the same population, in one process: a write through a pre-bound handle touching all N vertices, |
ns per operation, by leg · bytes per vertex |
neither — on demand only |
|
Prices the per-link departure index ( |
ns per index operation · bytes at rest per link |
neither — on demand only |
|
Drives the PRODUCER fan-out leg: one write to a vertex with N remote subscribers, which runs |
ns per write, median with range · deliveries and resolutions per write |
neither — on demand only |
|
Prices |
ns per delivery, p50 · canonical resolutions per delivery |
neither — on demand only |
|
Drives one whole forward hop — peek, mount descent, head rebuild, egress — through a registry of N mounts of width W segments, addressed to the last-registered mount, and counts the frames that actually reached the intended egress link so no cell’s timing is read off a hop that fell to the terminus instead. Written to compile unchanged against the pre-lift tree, so the two binaries interleave as A/B arms on one machine. |
ns per hop, p50 · hits vs iters per cell |
neither — on demand only |
|
Drives FWD{WRITE} across two processes over real UDP into a receiver whose RX backend is a bounded pool, counting deliveries in the receiver’s own graph and sampling the pool’s free-slot floor and refusal count while the load runs — so the throughput figure is a receiver’s count and the RAM figure is an observed exhaustion rather than arithmetic. |
deliveries/s · free slots · rx drops |
neither — on demand only |
|
Counts, per vertex shape, whether an extension block is allocated at all — the pay-for-what-you-use split RFC-0022 §3.B widened by deleting the registration parameter that could force one. Classifies through |
vertices per bucket |
neither — on demand only |
|
Separates the single poll thread’s three costs by moving one axis at a time: an |
frames/s + ns p50 / p99, per peer count |
neither — on demand only |
|
The same fan-in topology with timestamped payloads, on a fresh server per sweep point, so aggregate throughput and the full one-way latency distribution come from one run. |
frames/s · ns p50 / p99 / p999 / max |
neither — on demand only |
|
T threads clone and release one shared segment view — the per-subscriber delivery primitive, with T standing in for fan-out width — as T scales 1 → 128. |
clone+release/s, aggregate and per thread |
neither — on demand only |
|
One writer storms writes at a hot vertex while W threads each loop on |
writes/s · wakeups/s |
neither — on demand only |
|
T threads hammer |
ops/s, aggregate and per thread |
neither — on demand only |
|
T receive threads forward rope frames with the RX block source shared across all children, one pool shared, or one pool per child. One seam; the whole-node instrument is |
frames/s · ns p50, per thread count |
neither — on demand only |
|
Composes a whole node under four allocation-store configurations — the shipped all-heap baseline plus ADR-0079’s WIDE, MID and NARROW, at equal total slab — and drives one workload that draws all four of its allocation channels: a rope forward hop (router |
ns p50 · ops/s per thread · store bytes |
watched |
|
The memory half of the store sweep, in its own binary so a global |
heap bytes and allocations, per arm and window |
watched |
|
The same write at a vertex that RETAINS a last-known-value and at one that retains nothing, swept over subscriber width 0/1/4 and value link count 1/4 — the axis |
ns p50 · writes/s, per role, fan-out and link count |
watched |
|
The heap half of that question, in its own binary for the reason bench_store_escape gives: how many process-heap blocks one write costs at each role and link count. The count was the shed-on-OOM exposure — a HANDLER notify clone past the rope’s inline capacity drew a block on every write, and it was that block’s failure that shed an entire fan-out while the write still returned success. #1505 deleted the clone, so the HANDLER now reads exactly one block per write below the retaining role at every link count and the shed is impossible rather than counted. Gate: never gated — a decision input, reported as a count. |
allocations and frees per write, per role and link count |
watched |
|
Separates the two halves of a write at 0 and 1 subscribers: |
ns p50 / p99 / mean · operations/s, per verb and subscriber count |
watched |
|
N writers on ONE vertex, plain last-writer-wins against a STREAM retaining a bounded history, swept 1 → 8 writers with the two arms run back to back at each width so the stream/lock-free ratio is same-run. The committed instrument for §4.6.2’s four-writer leg; it prices today’s receiving-vertex retention, not the producer-side ring that figure priced and PR #1490 deleted. Gate: never gated — a many-thread aggregate rate is a property of the host. |
aggregate and per-thread ops/s · ps per system-wide op · the same-run ratio |
watched |
|
Concurrent publishers and readers on one last-known-value slot across five reclamation arms, plus |
ops/s · ns p50, per thread count |
neither — on demand only |
|
Isolates hazard-slot NODE ACQUISITION, which every other LKV bench amortizes away: it holds thousands of never-written slots live so that each publish takes the allocating arm rather than the free list, and reports that arm, the retire-scan-free path and a free-list-hit control separately. Batch-derived per-operation figures, so the distribution columns carry no distribution. |
ns per acquisition · ns per release |
neither — on demand only |
|
Measures the machine rather than libtracer: nine arms isolating a thread-private counter, a shared read, contended read-modify-writes, false sharing and lock costs, so a claim about a shared line can be checked against the box in front of you. |
ns per op · ops/s, per thread count |
neither — on demand only |
|
Feeds the shared latency accumulator distributions whose quantiles are known by construction and prints measured against analytic. It touches neither the clock nor libtracer, so a failure is a defect in the estimator and nowhere else. |
quantile ns, measured vs analytic |
neither — on demand only |
30 of 47 harnesses reach neither a chart on this page nor a gate, by surface: Network timing — bench_compose_net, bench_zenoh_compose, bench_tcp_fanin, bench_pin_net, bench_tcp_peer_scaling, bench_tcp_baseline; Framed-hop timing — bench_forward_rope, bench_terminus_tier, bench_originate, bench_hop_chain, bench_path_label, bench_reply_leg, bench_mount_resolve; Allocation counting — bench_transport_iov, bench_iov_spill_cost, bench_wire_heap, bench_failable_census, bench_pin_ratio, bench_subscribe_index, bench_qos_census; In-process timing — bench_scale_sweep, bench_target_binding; Concurrency & scaling — bench_fanout_clone_storm, bench_await_wakeup_storm, bench_route_handle_contention, bench_rx_source_topology, bench_lkv_slot, bench_hazard_node, bench_contention; Instrument self-check — bench_tail_validate. They are run on demand and read in a terminal. That is a deliberate state for the concurrency arms — they need a many-core host and a shared CI runner cannot produce a number worth recording — and a standing gap for the rest. A measurement that is neither published nor gated decays into a number nobody can check, so this count is derived here rather than kept in a list that could quietly go stale.
5 · Cross-core conformance (every native core must agree byte-for-byte)¶
How this surface is measured¶
The three native cores — the C++ golden reference, and the from-scratch
TypeScript and Rust reimplementations — must agree byte-for-byte. A shared set
of versioned conformance vectors (tests/conformance/vectors/v1) is decoded and
re-encoded by every enabled core; the driver diffs the results, and a single
DISAGREE fails CI
(ADR-0028).
This surface measures truth, not time — it is what lets every timed surface below
trust that a fast C++ number describes the same protocol the other cores speak.
Live driver summary:
cpp: 106/106 vectors ok
ts: 106/106 vectors ok
rust: 106/106 vectors ok
CONFORMANCE: PASS
6 · Dispatch — in-process latency & throughput¶
How this surface is measured¶
bench_libtracer exercises the hot path — resolve a vertex, write a value, notify
and deliver to subscribers — entirely in one process, and reports per-operation
latency (p50 / p99 / mean nanoseconds) and throughput (deliveries or
publishes per second). It sweeps three axes independently:
fan-out — subscribers per write (dispatch amortization);
payload — value size in bytes (copy cost);
topic count — number of registered vertices (registry / resolver pressure).
Several named modes isolate distinct costs on the same axes:
inproc— the full write (store + notify + deliver), including the producer’s own work: each op allocates a segment and copies the payload into it before the write, so above 1 KiB part of the size slope is the producer’s copy, not the library’s;inproc-borrow— the loaned-view path: no payload copy, but not allocation-free — each write allocates a segment header for the borrow and the block the vertex stores;inproc-deliver— deliver-only (propagate), value stored once;inproc-path— write-by-path, resolving the registry on every write. This is a deliberate resolver canary, not a hot pattern: real code resolves a path once and writes through the held handle. Judge dispatch cost againstinproc/inproc-borrow, never againstinproc-path.inproc-target-stored/inproc-target-handler— the path-target dispatch leg: edges carrying a target key instead of a callback, which is what a wireSUBSCRIBERproduces (the callback form is host-SDK sugar). Each delivery resolves the target in the registry, passes the fan-in ACL gate, clones the rope nothrow, and applies the target’s write effects — measured at roughly 10× a callback edge at fan-out. Thestored/handlerpair separates the target’s own store from the dispatch itself.stream-w1/stream-w2/stream-w4— 1, 2 and 4 writer threads on one STREAM vertex, the single-lock admission of #1713 under contention; delivery is counted at the subscriber.stream-spilltimes a write whose take is wider than the in-frame slots (it spills), andstream-defera burst ofassigns delivered by one coveringpropagate; both per cycle, batch-timed (#1808).route-handle-egress-mt1/-mt2/-mt4— producer threads on one advertised route-handle flow, the egress reuse read a compacted delivery takes. Advisory: charted, never gated.seam-class-c1/-c8/-c32,seam-direct,seam-fallback— the allocation seam’s two decisions: finding a block’s size class among C classes, and a fullbump_source_tfalling back to its upstream pool (seam-directis that pool alone).inproc-pool-batch— the window-calibrated twin of the heap-viewinproc-poolrows.
Every mode above except the inproc-target-* pair subscribes with an in-process
callback, so a fan-out curve reads the callback leg unless its mode says otherwise.
This is the surface that carries the microsecond thesis — the zero-copy substrate
(ADR-0016)
delivering values as loaned view_ts — and the one the per-PR gate watches most
closely.
Process shape. Each family of the sweep (the fan-out ladder, the payload ladder,
lkv, …) runs in its own fresh process, under one fixed set of allocator settings
(GLIBC_TUNABLES: mmap and trim thresholds, arena count). Rows used to share one process
heap, aged by every row ahead of them, so a row’s value partly depended on its position in
the sweep. A heap state that matters is now a named row instead: lkv-*-heap runs on a
fresh heap and lkv-*-heap-aged on a deliberately fragmented one
(#1803).
Counted, kept and listed once. Every delivery rate in this sweep is counted at the
receiver, never computed as publishes × fan-out: eptype-stream, mixed and
inproc-mt* were the last rows that computed it, and a STREAM ring that sheds an entry
still lets write() succeed, so only a count can show the shed. Timed results that
nothing reads (the lkv-* copy, the path-parse result) pass through an empty asm
clobber so the compiler cannot delete the work. The dce-canary family checks that a
timed chain is not deleted: a dependent chain eight times longer must time at least four
times longer, or the run fails. It does not prove the clobber itself is load-bearing. Each key is emitted once: inproc/64/1/1 comes from the
payload sweep only (#1805).
harness |
what it does |
reports |
|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. An |
ns p50 / p99 / mean · deliveries/s |
21 family charts · 436 series · x-axis = recorded
main commits (oldest → newest) · 🏷 dashed verticals mark release
tags (≈ = tag commit itself is not a recorded point; marker sits at the nearest
following recorded commit) · 🔧 dotted verticals mark commits where the BENCH
changed — points either side of one are not comparable. Each card carries every METRIC that point
recorded — p50 / p99 / ns per delivery / throughput — pick one under the title. Families with a numeric parameter
axis also offer trend / sweep / heatmap / 3D views — same data,
three axes (commit × parameter × value), over a selectable commit range. The paired
libtracer-vs-Zenoh cards add a ratio toggle on bench-local: both engines run in the
same pass on one pinned CPU, so their per-commit quotient cancels runner speed and is the
comparison to read across a long history. The hosted store keeps the best runner per series,
so its two arms need not share a pass and its cards carry no ratio. On bench-local, a
▲ / ▼ on a trend line marks a sustained step (the row moved past its own
noise threshold and held for 3 points), at the first commit that measured the new level.
Also on bench-local, a trend line carries a noise band: over each series' last N recorded points
(default 10, picked per card) the p10–p90 is shaded and min/max drawn faint, and the legend and
tooltip give the window's coefficient of variation, so a dip can be read against the row's own
noise. A series with fewer than N points gets no band.
Hover any chart for exact per-commit values.
7 · Wire & routing — what a framed hop costs¶
How this surface is measured¶
The in-process surfaces above measure the graph. A separate set of benches measures the
network plane — what a frame costs between arriving and being applied — because the two
move independently and an improvement to one can hide a regression in the other. They cover
the three shapes a frame takes: a transit hop that never resolves, a cold terminus resolve
an established flow pays once, and the warm COMPACT frame that dominates a running system.
Every bench on this surface reports latency and exact allocation counts, and every one
calibrates its own batch size against the host clock rather than hardcoding a number: a
routing operation costs the same order as clock_gettime, so timing one operation at a time
measures the clock instead of the code.
bench_forward_demux and bench_compact_delivery are recorded into the build-to-build
history alongside the in-process series, so a routing or delivery regression shows up as a
trend rather than being noticed later. They emit the same RESULT rows as bench_libtracer,
so they need no separate aggregation. bench_compact_delivery drives every frame through
the child’s receiver ctx, the way a link’s receive thread delivers it, and asserts so
before timing: the public by-name on_frame door it used to call is one production frames
never take (#1808). Their transcripts are tolerated-empty — a bench that
fails to run must not cost a commit its whole history point — but an empty one emits a build
warning naming the file, because a silently-empty transcript is otherwise a green job that
recorded nothing.
harness |
what it does |
reports |
|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. An |
ns p50 / p99 / mean · deliveries/s |
|
Drives one FWD forward hop through the inbound link’s own receiver against a registry of N links, with the target child registered first ( |
ns p50, batch-amortized |
|
Drives the Nth |
ns p50 · allocations per frame |
6 family charts · 168 series · x-axis = recorded
main commits (oldest → newest) · 🏷 dashed verticals mark release
tags (≈ = tag commit itself is not a recorded point; marker sits at the nearest
following recorded commit) · 🔧 dotted verticals mark commits where the BENCH
changed — points either side of one are not comparable. Each card carries every METRIC that point
recorded — p50 / p99 / ns per delivery / throughput — pick one under the title. Families with a numeric parameter
axis also offer trend / sweep / heatmap / 3D views — same data,
three axes (commit × parameter × value), over a selectable commit range. The paired
libtracer-vs-Zenoh cards add a ratio toggle on bench-local: both engines run in the
same pass on one pinned CPU, so their per-commit quotient cancels runner speed and is the
comparison to read across a long history. The hosted store keeps the best runner per series,
so its two arms need not share a pass and its cards carry no ratio. On bench-local, a
▲ / ▼ on a trend line marks a sustained step (the row moved past its own
noise threshold and held for 3 points), at the first commit that measured the new level.
Also on bench-local, a trend line carries a noise band: over each series' last N recorded points
(default 10, picked per card) the p10–p90 is shaded and min/max drawn faint, and the legend and
tooltip give the window's coefficient of variation, so a dip can be read against the row's own
noise. A series with fewer than N points gets no band.
Hover any chart for exact per-commit values.
Reading the two demux arms¶
fixed registers the target child first, so its lookup hits on the first compare and the
chart isolates the size-independent part of a hop. scan registers it last, so the lookup
walks the whole table. The rise of scan over fixed at the same registry size is the
scan’s marginal cost. Read them as a pair; neither means anything alone.
Per-module keying does not narrow that scan — it changes the key, not the container, and a node’s links overwhelmingly sit in one module. It earns its place by keeping two modules’ same-named connections distinct, which is correctness, not lookup time (ADR-0061).
What the scan costs a real frame¶
path |
registry work per frame |
|---|---|
plain |
mount descent + inbound lookup, per frame — the frame carries the full path, so there is nothing to cache against |
|
one dereference of the cached registry slot, first frame only — resolved once, then memoized |
remote delivery to a subscriber |
one lookup by link name, per frame — the subscriber record holds a name |
A binding holds the resolved target rather than a name, and both cached forms self-invalidate
without a callback: the terminus compares a retirement generation, and the forwarding hop
reads the registry slot, whose link teardown nulls in place — so a departed link reads
nullptr, the same clean miss an unresolved lookup gives (ADR-0062, ADR-0063).
The dominant term in a hop is TLV header parsing — the FWD header, the op, the dst PATH and
its segments, the selector peek and the src PATH on rebuild, each read exactly once — and it
sits at a local optimum for that structure. At -O3 the parse inlines with values flowing in
registers into the narrow struct’s stores, so there is no intermediate object to remove:
forcing the inline, or narrowing the header struct, each regress the hop by about 10 %.
A COMPACT re-emit builds no frame: the head goes to a 12-byte stack buffer and the payload
is handed to the transport by reference as a scatter-gather list, which is why the forward
charts are flat in payload size. Zero allocations in the router is not zero on the wire —
a transport that does not override the gather form concatenates once in
transport_t::send(iov).
8 · Memory & allocation¶
How this surface is measured¶
A different instrument entirely. bench_forward_heap replaces the global allocator
with a counting wrapper and arms it around exactly one operation, so these are
exact allocation counts and byte totals — not statistics, not sampling. Bytes are
read from malloc_usable_size, so a resident figure is what the allocator really
holds rather than what the caller asked for. The coarse process-level number beside them
is each family’s RSS delta: every bench_libtracer family runs in its own process and
prints RSS family=<name> start_kb= peak_kb= delta_kb=, the high-water mark minus the
resident set the family started from. It replaced a whole-run “max RSS” from
/usr/bin/time -v, which was the harness’s peak, not any family’s footprint
(#1808).
Beside the per-vertex probes, bench_forward_heap prints RAM per callback edge
(edge_callback), per wire subscriber edge (edge_wire), per link (link, one
router child) and per 1 KiB value (vertex_value_1k), each over 256 units and multiplied
by 1000 so a fraction of a block survives the division; the heap blocks per write at every
payload-ladder size, from the injected source and from the global heap; and the STREAM
write’s stripe-lock sections per write (counted through --wrap=pthread_mutex_lock, the
instrument #1713’s own test uses).
Per object, not per slab
(#1778). Since a default graph draws
its tables from the host slab pool
(#1777, ADR-0083), a counting window
over a default graph sees whole slabs (64 KiB and up, one per size class a growing table
first reaches), not the object it is pricing. So the gated vertex, vertex_app5,
vertex_app5_static, edge_callback, edge_wire and vertex_value_1k probes run on a graph
injected with
tr::mem::heap_source() (and vertex_value_1k mints its segments through a backend over
that same source): every block is then its own counted operator new, which is what a build
without the slab pool (an MCU) pays per object. The same windows on a default graph are still
printed, ungated, as RESULT slabfoot <what> live_x1000= blocks_x1000= n= (the two app-table
rows have no such twin; slabfoot vertex already prices a leaf’s slabs), so the slab cost
stays visible. The
two methods differ only where the old one missed a block: on main before #1778, a wire
edge’s per-link candidate list came from the graph’s pmr pool, which the old window did not
see (7.035 blocks per wire edge counted per object, 6.035 the old way); and the old
vertex_value_1k row moved by a whole kept free slab between runs of the same code
(+384 B over 256 units).
Two invariants sit on this surface, and the scope of the armed window is part of the
first one. The steady-state forward hop’s own work must touch no heap — the two-plane
forwarding model
(ADR-0038)
requires it — so bench_forward_heap arms the counting allocator around one hop and CI
runs it at ZEROHEAP_MAX=0: on a contiguous (single-link) frame the hop dispatches by
offset, builds its replacement heads in fixed stack buffers and its gather table in a stack
iov array, and a single stray malloc inside that window fails the job.
Two terms sit outside that window, so this is not the claim that a forward hop is
heap-free end to end. The gate drives capture_transport_t, a stub link that only sums the
spans it is handed, so the shipping transports’ ::iovec table is never assembled —
transport_udp and transport_tcp each hold 16 spans inline and spill to the heap above
that, measured at 17 caller spans / ~288 B by bench_transport_iov. And the
multi-link rope arm gathers into a block array drawn from the injected receive source,
because the sub-span count is the sender’s choice and is known only at run time — nothrow,
so an exhausted heap drops the frame rather than aborting, but not allocation-free. Read
the two benches together; neither is sufficient alone. A terminus, by contrast, may
allocate
(ADR-0041)
and is measured rather than gated. The per-vertex resident figure is the one the
constrained profile lives or dies by (a ~16 KB RAM budget on the ESP32 target),
which is why it is a series of its own rather than folded into RSS.
harness |
what it does |
reports |
|---|---|---|
Drives the in-process hot path — resolve a vertex, write a value, notify, deliver — swept over fan-out, payload size, topic count, thread count, endpoint type, value backend and dispatch mode. An |
ns p50 / p99 / mean · deliveries/s |
|
Replaces the global allocator with a counting wrapper and arms it around exactly one operation, so every number is an exact count rather than a sample. Eight probes: the forward hop, a terminus resolve, five resident-vertex shapes, and a wire-driven registration’s escape from the injected memory seam. Gate: forward hop = 0 allocations; per-vertex bytes +2%, block counts exact. |
allocations · live usable-size bytes |
|
Stands up a real server transport and drives K raw client peers at it, reading the live heap balance when the server is up and quiesced, at K established connections, and after teardown. Gate: per-connection bytes drift past a measured ±8 B / ±0.5% band — WARN-first on the pinned host (bench/ram_census_pins.json carries the pins and the activation criterion for the flip to fail). |
live bytes per link · live bytes per connection |
|
Prices a 100-vertex node — mixed int, array and stream values, 4–64 B each — stage by stage on the live heap balance: empty graph, vertices registered, values written, TCP listener up, a second-process peer connected, and the steady state after a mixed op storm, with a null arm and residual check bracketing every run. |
live bytes · blocks, per stage · bytes per vertex |
|
Counts every allocation and byte one publish costs, stage by stage along the value path — a local write with callback subscribers, the same write to a vertex that retains nothing, the same write fanned to K stored target vertices and to K HANDLER targets, a producer handing over its own bytes, a FWD{WRITE} through the terminus below and at-or-above the copy-or-share threshold, egress through the base scatter-gather, and the RFC-0028 one-block prototype — over a warm window of 1,000 publishes. |
allocs · heap bytes · payload copies, per publish per stage |
Eight armed windows feed the store, each counted around exactly one operation:
probe |
what is armed around |
gate |
|---|---|---|
|
a |
hard — instrument self-check, aborts the run (#1420) |
|
the identical vector filled over the PROCESS HEAP — the escape must be SEEN, so an operator-new override gone blind cannot hand every zero below a free pass |
hard — instrument self-check, aborts the run (#1420) |
|
one FWD forward hop, from arrival to egress |
zero allocations, every CI run (ADR-0038 §16KB-RAM) |
|
one terminus resolve, which may allocate (ADR-0041) |
report-only |
|
a bare default leaf vertex at rest |
bytes +2% · blocks exact |
|
the same leaf plus the increment one small LKV write adds |
bytes +2% · blocks exact |
|
a leaf carrying an owning five-field app descriptor table (ADR-0058) |
bytes +2% · blocks exact |
|
the same table installed borrowed, its slots viewing caller flash |
bytes +2% · blocks exact |
|
one publish at a large subscriber count |
report-only |
|
a wire-driven |
report-only |
Every probe reports two independent quantities and both ratchet: bytes=, the live
usable-size balance, and allocs=, the number of heap blocks. They ratchet on different
terms because they are different kinds of number — bytes carry a 2 % tolerance since an
allocator size-class flip is not a regression, block counts carry none.
8 family charts · 82 series · x-axis = recorded
main commits (oldest → newest) · 🏷 dashed verticals mark release
tags (≈ = tag commit itself is not a recorded point; marker sits at the nearest
following recorded commit) · 🔧 dotted verticals mark commits where the BENCH
changed — points either side of one are not comparable. Each card carries every METRIC that point
recorded — p50 / p99 / ns per delivery / throughput — pick one under the title. Families with a numeric parameter
axis also offer trend / sweep / heatmap / 3D views — same data,
three axes (commit × parameter × value), over a selectable commit range. The paired
libtracer-vs-Zenoh cards add a ratio toggle on bench-local: both engines run in the
same pass on one pinned CPU, so their per-commit quotient cancels runner speed and is the
comparison to read across a long history. The hosted store keeps the best runner per series,
so its two arms need not share a pass and its cards carry no ratio. On bench-local, a
▲ / ▼ on a trend line marks a sustained step (the row moved past its own
noise threshold and held for 3 points), at the first commit that measured the new level.
Also on bench-local, a trend line carries a noise band: over each series' last N recorded points
(default 10, picked per card) the p10–p90 is shaded and min/max drawn faint, and the legend and
tooltip give the window's coefficient of variation, so a dip can be read against the row's own
noise. A series with fewer than N points gets no band.
Hover any chart for exact per-commit values.
Note
Allocation churn is not resident footprint, and this page charts both — separately. Resident bytes are what a live object holds; the heap-footprint series are the transient churn of one operation; the timed rows are what it costs to get a block and give it back. Neither number bounds the other: a release can take per-frame allocations to zero and move resident bytes almost not at all. Churn also costs more on the target than these host figures show — glibc’s tcache serves a hot same-size malloc/free in tens of nanoseconds, an MCU allocator in hundreds — so a host reading of churn is a lower bound.
9 · libtracer vs Zenoh — measured, absolute¶
Both engines on one runner, best of 3 rounds, so the numbers are directly comparable on identical hardware: two in-process comparison axes — subscriber fan-out and payload size — and a network latency comparison over the real loopback kernel path, one socket and one paced value per transport in two processes. The charts plot absolute throughput, latency and bandwidth as series on shared axes; there are no speed-up ratios.
The topic-count pair is charted for libtracer alone, and that is a fairness decision, not a gap in the run. The two arms are different operations — libtracer’s row re-resolves the destination address inside every timed iteration, Zenoh’s publishes through a declared handle and resolves nothing per put — so putting them on a shared axis would attribute a resolution term to topic scaling. The matched decomposition that replaced it (both spellings on both engines) is tabulated in the methodology below rather than charted here: one ladder costs about nine minutes and its content is structural.
libtracer is compiled from source at -O3; Zenoh is the upstream prebuilt zenoh-c 1.10.0
release binary that bench/fetch_zenoh.sh downloads, so its optimization profile is
upstream’s rather than a flag this repo sets. Both are optimized builds. Which rows do equal
work, why there is no network throughput comparison, and which transports are absent and
why, are all below — as are the three designated model boundaries, which say what the
two engines are respectively buying with the costs these charts show. Topic count was once
listed there as the one axis where Zenoh was the better engine; the matched
topics-bound / topics-addr decomposition retired that reading, and the boundary section
below states what replaced it.
A side-by-side against Eclipse Zenoh (zenoh-c, peer mode). Both are
measured on the same runner in the same rounds — libtracer compiled from source at
-O3, Zenoh as the upstream prebuilt zenoh-c 1.10.0 release binary that
bench/fetch_zenoh.sh downloads (we do not build it, so its optimization profile is
upstream’s, not ours). The whole grid is swept 3 times and each point keeps its best
observation across those rounds (bench/best_of_rounds.py): contamination on a shared
runner is one-sided — a busy neighbour can only ever make a round slower — so the best
round is the estimator closest to what the code does, while a median would be an estimate
of how busy the machine was. Both engines are executed in the same loop the same number of
times; giving one arm N tries and the other one shot would be a thumb on the scale rather
than a measurement. The tail rows (p999 / max) are deliberately not reduced this
way — a minimum p999 would advertise the calmest moment the host ever had as though it
were a worst case — so those are left as measured. And the charts plot absolute
throughput / latency / bandwidth — both engines as series on shared axes. There are no
speed-up ratios: every point is a measured number you can read off directly. Fairness is
discussed in its own section below, because a naive put-vs-write comparison would be
misleading.
Designated model boundaries¶
Three properties of libtracer’s model are designed, not accidental, and each of them carries a cost this page measures. They are listed here so that cost can be read for what it is: the price of a capability that was chosen, not a defect awaiting a fix. Every comparison on this page — the Zenoh chapter above most of all — has to be read against them, because a comparison between two engines that solved different problems is only informative once the problems are named. Nothing below proposes a redesign; where a boundary has a mitigable constant, the mitigation is a measurement, and it is linked.
A boundary earns that name only from a measurement that isolates it, and the section is held to that standard in both directions. A number that moves for some other reason is not evidence for a wall, however convenient the shape of the curve — the first draft of boundary 1 argued from exactly such a curve and the isolating control arm refuted it. Where that has happened it is said outright below rather than quietly rewritten, because a boundaries section that only ever accumulates walls is an argument, not an instrument.
1 · Every distinct address is resident state. A vertex is not a name that a message
happens to carry — it is an object that exists between writes. That is what pays for the
last-known-value (a reader gets the current value without waiting for the next publish),
for await (a readiness sequence has to live somewhere), for composed reads across a
subtree, and for the per-vertex ACL. An engine that only routes keeps no such object and
so has nothing to scale in the number of addresses.
What residency costs is bytes. That is the axis the trade is paid on, and it is measured rather than derived: the decomposition sweep (#1485 / #1496) reads 132 B of heap and 140 B of RSS per vertex at 10⁶ vertices, converging from above as the upper tree’s fixed cost amortizes, against a computed ~120 B floor — +10 % over the arithmetic. A million addresses is 132 MB of heap and 140 MB of RSS. The O(#addresses) residency is the model and no footprint work turns it into O(1); the constant in front of it is fair game and is actively ratcheted on this page, bytes at +2 % and block counts exactly.
This is also where the boundary actually bites the target the project cares most about. On a wide host 132 MB for a million addresses is unremarkable; on the narrow end — the constrained MCU profile living inside a ~16 KB RAM budget — the per-vertex byte figure is the number that decides how many addresses a node may have at all. Memory, not latency, is the axis on which “every distinct address is resident state” is a real constraint.
Residency is NOT a per-operation LATENCY cost, and the control arm that decides this is unambiguous. Hold one million vertices resident and touch exactly one: a write costs 90 ns — the same 90 ns it costs with a thousand resident. Resolving a single hot address moves 70 → 90 ns across three decades. Descent is population-independent outright (ADR-0057, now verified rather than assumed): a fixed-shape probe address resolves in 60 ns at 10³ and 60 ns at 10⁶ warm, and 90 ns flat with its lines evicted between samples. Everything that grows, grows with the touched set, not the resident set.
Read that refutation at exactly its own width. It kills the claim that residency is a latency wall — which is what the earlier draft of this section asserted, and it was wrong on its own evidence. It does not say the residency model is free: the bytes above are the cost, and they are real. A boundary can be genuine on one axis and absent on another, and saying which is which is the whole job of this section.
The topic-count curve in the Zenoh chapter is not this boundary’s evidence — and the
completed comparison retires it. The fairness audit
(#1480) went looking in both
directions and reported exactly one axis where Zenoh was the better engine outright:
topic-count scaling, Zenoh’s p50 moving 220 → 230 ns (+5 %) across 1 → 8192 topics
against libtracer’s 140 → 190 ns (+36 %), narrowing our margin from 1.57× to 1.21×.
Both rows were measured correctly and neither is withdrawn. They were not the same
operation: the libtracer row (inproc-path) re-resolved the destination address inside
every timed iteration, while bench_zenoh published through a declared Publisher and
resolved nothing per put. A resolution term sat inside one arm and nowhere in the other, so
the narrowing could not be attributed to either engine’s topic scaling.
The topics-bound / topics-addr pair
(#1485, bench/run_topics.sh)
measures both spellings on both engines over one ladder, and it is now a result
rather than an instrument. 64 B payload, fan-1, deliveries counted at the subscriber,
best-of-5-rounds with both arm orders and both engine orders, two independent ladders
agreeing to ≤ 2 % on every arm:
p50 / mean ns per operation |
1 topic |
100 topics |
10 000 topics |
|---|---|---|---|
libtracer |
110 / 116 |
110 / 116 |
110 / 118 |
Zenoh |
210 / 214 |
210 / 215 |
220 / 232 |
libtracer |
140 / 140 |
170 / 174 |
190 / 196 |
Zenoh |
320 / 332 |
1 450 / 1 532 |
221 000 / 226 351 |
Read bound against bound — the like-for-like pair, and each engine’s own recommended spelling — libtracer moves 116 → 118 ns (+2 %) and Zenoh 214 → 232 ns (+8 %) across four decades of topic count, so the margin widens slightly, 1.84× → 1.97×. The audit’s “+36 % against +5 %” was our resolve-per-operation arm charted against their bound one. Matched, neither engine has a topic-count problem in the bound form.
Read resolve against resolve, the growth is on the other side and it is large.
libtracer’s per-operation find() descent moves 140 → 196 ns (+41 %) over the same
span — the audit’s +36 %, reproduced at the shape that actually produces it. Zenoh’s
undeclared Session::put moves 332 ns → 226 µs, and the completed run settles the
“is that a different code path or a scaling curve” question #1485 left open, in the only
sense that mattered: it is a curve, and a linear one. From 100 to 10 000 declared keys
the log-log slope is 1.08 — O(N) within the instrument — with a marginal cost of about
12 ns per additional declared key over the first decade and 23 ns over the last two.
It is of course reached by a different code path from Publisher::put; that is the arm’s
whole purpose. What it is not is an outlier, a congestion effect or a measurement artefact:
inside that path the cost is exactly the linear match a declared publisher exists to hoist.
So the honest statement of this axis, replacing the one above: both engines are near-flat in the bound spelling and both degrade in the resolve-per-operation spelling — libtracer by 41 % across four decades, Zenoh by a factor of about 680. The mitigation is the same on both sides and both APIs already offer it: bind, or declare, once. There is no axis here on which Zenoh scales better in topic count. There was an axis on which our own bench spelled its destination worse than theirs did.
Host, recorded with the numbers: 31 CPUs, no cc1plus alive at any 20-second sample of
either ladder, 1-minute load 1.4–3.3 throughout except one ~60–90 s excursion per ladder
(peaks 10.4 and 9.4, unattributed, on a shared desktop). Best-of-rounds is the specific
mitigation for one-sided contamination, and the two ladders’ ≤ 2 % agreement on every arm is
the evidence that it worked here.
Two caveats on that decomposition, because it is fresh. Its arms are not gated — they
are new and their run-to-run stability is unproven, and POINTS is a promise about
stability.
And the topics-* arms are not on the generated results page. One ladder costs about
nine minutes, almost all of it the Zenoh topics-addr rung at 10 000 keys, so wiring it
into the documentation job would add roughly twenty minutes to every docs build to
re-derive a result whose interesting content is structural. The table above is the output
of a hand run. Since #1809 both
default sweeps also emit the pair (family topics), so the trend charts carry it on every
recorded commit, one round and one arm order per commit; a verdict on the axis still comes
from run_topics.sh. Those numbers supersede the preliminary single-round, single-engine
Zenoh figures recorded on #1485, which were never quotable and are deliberately not
reproduced here. The decomposition beside them was taken on a quiet box (1-minute load
0.64–1.15, no cc1plus alive, 31 CPUs, best of 3 rounds with the arm order flipped on
alternate rounds, two full ladders agreeing to ~2 % at 10⁶).
One follow-up remains open and it is a ruling, not a redesign:
#1486, on memoizing
vertex_slot(), whose reverse scan is exactly O(total vertices) — measured at 450 ns at
10³ rising to 410 µs at 10⁶, taken under a shared mutex on every binding mint. That is
the number the ruling was blocked on; the ruling itself is #1486’s to make.
2 · Paths are routes, not location-independent names. A path is resolved from the vantage point of the graph doing the resolving; the same value can be reachable by different paths from different places, and a path handed to a peer does not carry a promise that it means the same thing there. This is the boundary that buys composition with no infrastructure: two graphs are joined by mounting one into the other, and the join needs no broker, no registry service, no name authority and no agreement between the parties beyond the mount itself. A location-independent naming scheme would make a path portable across vantage points, and would need exactly the infrastructure that absence is the point of not having. Comparisons that assume a global namespace are comparing against a system that has one.
3 · The producer’s write budget is fixed, fan-out is value-agnostic, and a producer
never queues. A write costs what it costs regardless of who is listening: fan-out
carries the value without inspecting it, and the depth a consumer wants is the consumer’s
own ring on the consumer’s own vertex, charged in bytes to a source that vertex injected
(RFC-0025 §4.6.1 Amendment 2). What this buys is the hot write path — the reason a write
is a sub-100 ns operation at all, and the reason its cost does not move when a subscriber
decides it wants history. What it forecloses is symmetric and worth stating: there is no
producer-side buffering to smooth a slow consumer with, no content-dependent routing
decision inside the fan-out loop, and no way for a subscriber to make a producer pay for
its own depth. The budget is instrumented rather than asserted — the write path’s compiled
size is ratcheted symbol by symbol (bench/symbol_ratchet.json), and a change that moves
it has to price the move on the bench before the pin is allowed to move with it.
Fairness in the comparison¶
An honest side-by-side has to account for the two engines doing different amounts of work per operation.
Deliveries are counted at the subscriber on both arms. Neither engine’s delivery figure is inferred from
publishes x fan-out. Each side’s timed window ends only once every delivery it owes has landed, each reports what its subscribers actually received, and each prints a warning naming any point that came up short. The two engines reach that guarantee differently, and the difference is not a handicap: libtracer dispatches inline, sowrite()returns only after the last subscriber callback and its publish loop is its delivery loop, while the Zenoh harness spins on the receive counter inside the window until the backlog drains, so a delivery made on a runtime thread is still inside the window. Each Zenoh row is followed by aNOTE zenoh-runtimeline with its runtime threads’ CPU time over the window: on the in-process rows it is 0 ns, so intra-session delivery happens on the putting thread there, as libtracer’s does. This matters because the arithmetic form would not merely be imprecise, it would be unfalsifiable: libtracer’s wide-fan-out snapshot truncates to its inline prefix when its overflow reserve fails, and its HANDLER and STREAM legs shed an entire fan-out on a clone or ring-append failure — in every one of those caseswrite()still returns success, so apublishes x fan-outfigure would report deliveries that never happened and nothing in the harness would contradict it.The libtracer-only rows that audit deliberately left on the arithmetic — the
inproc-target-*pair andinproc-remote, which have no Zenoh row beside them — were finished on the same terms (#1481).inproc-target-handlerandinproc-remotecount at their consumer like every other row.inproc-target-storedhas no consumer to count at — its delivery terminates in the target’s last-known-value, and hanging a counting subscriber off each target to see it would add an edge per target to the topology being timed — so that one subtractsgraph_t::delivery_drops()from the ceiling instead. That is a measurement rather than a restatement of the arithmetic because RFC-0025 §4.4 forbids an unaccounted shed: every leg that drops one of those deliveries counts it.Write does strictly more than put. libtracer’s
writerow also persists the value (it becomes the vertex’s last-known-value) and bumps theawait/ readiness sequence on every op. Zenoh’sputis transient delivery only. So the libtracer write row is charted against a Zenoh row that does less semantic work — theinproc-deliver(propagate) series is the apples-to-apples counterpart: value stored once, each op only delivers, matching put semantics. Both libtracer series are shown so the reader sees the full-work and the like-for-like number side by side.The topic-count rows must agree on how the destination is spelled — and the pre-existing
inproc-pathpair did not. libtracer’sinproc-pathrow writes by address, so a registry resolution sits inside every timed iteration; the Zenoh row of the same name publishes through a declaredPublisher, which is the bound form and resolves nothing per put. A resolution term therefore sat inside one arm and nowhere in the other, and any narrowing of the margin along that ladder could not be attributed to either engine’s topic scaling. Thetopics-bound/topics-addrpair (bench/run_topics.sh, #1485) fixes it by measuring both spellings on both engines over the same ladder — pre-bound handle / declared publisher, and destination resolved inside the operation — so the resolution term is visible as its own difference instead of hidden inside one arm. What is compared is per-operation resolution: thepath_ton one side and theKeyExpron the other are both pre-built, because charging one engine for a string parse the other hoisted is the same mistake in the other direction. The completed two-sided run and what it does to the audit’s conclusion are in Designated model boundaries below; the short form is that bound against bound both engines are near-flat, and the axis the audit reported as Zenoh’s was the unmatched pair.Each compared row is matched on four conditions (#1809), and every libtracer-vs-Zenoh chart restates them:
One fresh process per engine and family. Both harnesses run their default sweep as families, each in a new child process with pinned allocator tunables, and the Zenoh family opens its own session. No row inherits a heap or a session that an earlier family aged.
The same pin, runtime threads included. Both binaries run under one CPU set, which Zenoh’s runtime threads share with the putting thread, and their CPU time is printed per row (above).
Equal payload bytes. A libtracer write moves the value plus its 4-byte VALUE TLV header (6 bytes from 64 KiB), so a Zenoh put carries the same count. Rows stay keyed by the value’s size.
Resolution against resolution. Zenoh’s
inproc-pathrow now puts by key, resolved on every put, as libtracer’s writes by path; the bound pair istopics-bound, which both default sweeps now emit, so the topic-count charts below pair each spelling with its own. The Zenohinproc-pathseries changes meaning at that commit and steps with it.
ACL is disabled in the comparison rows. No subject resolver is installed, so the access gate is a single null check. The cost of enforcement is measured separately (the
acl-inheritrows), never hidden inside the comparison.No network throughput comparison is published. A valid one needs a real subscriber in a second process on both sides, with deliveries counted at the receiver rather than sends counted at the publisher — an engine whose publisher has no peer emits nothing to the wire at all, and a per-send rate multiplied by a composition width is arithmetic, not a measurement. That bench now exists (
bench/run_compose.sh): two processes per engine over real loopback UDP, the composition width K swept on both sides, and every rate taken from the subscriber’s own count over the subscriber’s own clock. It carries two guards, and reports nothing unless both hold — the receiver emits no row at all when it observed no values, any malformed record, or fewer throughput datagrams than the sample floor the driver hands it, and a wire-use audit runs the publisher understraceand fails the point below a send-syscall floor (COMPOSE_SEND_FLOOR, default 50), which is exactly the check whose absence let the withdrawn version report a rate for an engine that had only ever emitted scouting beacons. What is still absent is the chart: publishing this comparison is a claim, so it is a separate, reviewed step, and until it is taken the chapter states the gap rather than showing a number.Network latency is the surviving network comparison, and it is fair: a single-value, two-process, same-clock measurement over the real loopback kernel path, identical topology for both engines. p50, the p99 tail and the p999 deep tail are charted per transport — for a latency-first substrate the tail is the load-bearing number, and it is where transports separate, since an unreliable datagram path can win the median and still spike at p99.
The p999 has to earn its own publication. It is the cost of one message in a thousand — on a control path, the size of the deadline it misses — and it is the one figure on this page whose publication is gated rather than only its value. A p999 read off
nsamples is the order statistic atfloor(0.999·n), so onlyn − 1 − floor(0.999·n)samples lie above it: three at n = 4 000, and none at all at n ≤ 1 000, where “p999” is the maximum wearing a percentile’s name. The accumulator recordsnbeside every percentile and flags whether it clears its own adequacy floor; below the floor the number is withheld, and the shortfall is named in the transport-coverage note instead of being drawn as a line that moves for reasons the code never touched. The sample count, the number of samples above the p999, and the worst single message of the pass are printed under the charts, computed from the same rows that drew them.
A transport that is not charted always says so in the transport-coverage note under the charts, including when neither engine produced rows for it. WebSocket and QUIC are the standing cases: the WebSocket transport shows order-of-magnitude single-run p50 jitter under this bench that would make a published chart misleading, and QUIC needs the optional TLS module. An absent transport must never be readable as a tie.
Measured on the same runner in the same rounds — the whole grid swept
3 times with each point keeping its best observation, both engines in the same loop — so
this is like-for-like on identical hardware. libtracer is compiled from source at
-O3 here; Zenoh is the upstream zenoh-c 1.10.0 prebuilt release binary
(bench/fetch_zenoh.sh
downloads it — we do not build it, so its optimization profile is upstream's, not a flag we
set). Both are optimized builds. Absolute values on absolute
axes — no ratios, and every "reading" under a chart is computed from that chart's own
endpoints at render time. Read trends and orders of magnitude, not the third digit;
shared-runner variance is real. x-axis = the swept parameter (these are the only charts on
the page whose x-axis is not a commit).
The in-process sweeps carry a history picker over the bench-local store: each banked commit is one pass on one pinned CPU with both engines in it, so a picked point is as like-for-like as this build's own pass. It opens on the latest banked commit; 🏷 marks a release tag; compare overlays a second pick, dashed, on the same axes. this build is the pass described above. The network charts are this build's pass only: the store banks no transport rows.
- WS: not charted — libtracer's WebSocket transport shows large single-run p50 latency spikes (held until understood), and Zenoh has no WebSocket transport to compare against.
- QUIC: not charted — needs the optional
LIBTRACER_WITH_QUICmodule (msquic + TLS), gated like the dedicated quic CI job.
Each percentile here comes from 10,000 paced one-way probes at that payload, so at least 9 samples sit above the p999 and the worst single message of the whole pass was 2.5 ms. Across the charted points the p999 runs 1×–11× the p50 of the same point. Every charted point clears the accumulator's adequacy floor.
These are published, not gated. A two-process loopback tail is dominated by scheduler wake-up, not by either engine's code path, and it is unstable run-to-run by far more than the gate thresholds — measured, and quoted, in the noise chapter above. Read the deep tail as the jitter floor a real-time consumer on this topology inherits, and read engine-vs-engine on the p50 and p99 charts, which are stable enough to carry a comparison.
**🤖 CI-generated** on 2026-10-06 07:08 UTC · commit [`b4ab133`](https://github.com/avatarsd-llc/libtracer/commit/b4ab133cde4d3cf5379b87982d4590b26f325332) · [run 37426333810](https://github.com/avatarsd-llc/libtracer/actions/runs/37426333810) · runner `Linux`.
10 · Cross-core codec performance (decode→encode roundtrip, same v1 vectors)¶
How this surface is measured¶
Each native core runs the same per-vector decode→encode roundtrip over the shared v1 vectors, reported as the median across all vectors (one decode + one encode = one roundtrip). Because the input is identical for all cores, this is a genuine like-for-like codec comparison across implementations; a core whose toolchain is absent in a given build degrades to a note rather than failing the docs build.
harness |
what it does |
reports |
|---|---|---|
Decodes and re-encodes every shared v1 conformance vector through the C++ core — the |
ns per roundtrip · roundtrips/s |
core |
throughput (median) |
p50 latency (median) |
mean (median) |
|---|---|---|---|
cpp-core |
1.7 M roundtrips/s |
621 ns |
623 ns |
ts-core |
0.5 M roundtrips/s |
1904 ns |
2050 ns |
rust-core |
2.0 M roundtrips/s |
521 ns |
523 ns |
11 · Test rollup (live ctest, unified with the perf surface)¶
suites |
passing |
wall time |
verdict |
|---|---|---|---|
287 |
287/287 |
145.39s |
✅ all green |
By area: Codec (L2/L3) 15/15 · Examples 70/70 · Graph (L4) 24/24 · Net (FWD plane) 26/26 · Substrate (L0/L1) 17/17 · Transport 9/9 · other 126/126. Full per-suite detail: Test report.
12 · Reproducing this page locally¶
The gates are same-runner by construction, so a local reading only means something against another local reading taken the same way:
# Build the bench Release (-O3) — same flags CI uses.
cmake -S bench -B bench/build -DCMAKE_BUILD_TYPE=Release
cmake --build bench/build -j
# Pin to a core, take the best of several runs, compare only same-machine numbers.
taskset -c 2 ./bench/build/bench_libtracer # the sweep matrix
taskset -c 2 ./bench/build/bench_forward_heap # the allocation probes (zero-alloc gate)
# The network plane (§3b). Each takes a per-point wall-clock budget via
# LIBTRACER_BENCH_SECONDS; longer means tighter percentiles, never a different measurement.
taskset -c 2 ./bench/build/bench_forward_demux # forward hop vs registry size
taskset -c 2 ./bench/build/bench_terminus_tier # terminus: eager arena vs lazy rope reader
taskset -c 2 ./bench/build/bench_compact_delivery # steady-state warm compacted delivery
# The comparison surface needs Zenoh vendored first:
bench/fetch_zenoh.sh && cmake --build bench/build -j
Compare a change against its own baseline on the same machine in the same
session (git stash, rebuild, re-run) — never against a number from a different
host or a different day, and never with one arm pinned differently from the other.
The full two-arm protocol, and the measurements behind each of its rules, is above
under The A/B protocol, in Reading the numbers. The single-CPU taskset -c 2 in the commands above is part of the measurement, not
decoration: this host is heterogeneous and an unpinned arm can land on a core class
50 % slower than its partner’s.
13 · Raw data & provenance¶
The charts above are one view of the two persisted stores this section describes — the
selector at the head of each chart block says which one you are reading. Every push to
main runs the full bench
on three independently-drawn runners, archives all raw transcripts as a per-commit CI
artifact (bench-results-<sha>, on the perf workflow run), and records every
(mode, size, fanout, endpoints) point — latency, throughput and memory footprint as
separate series — to a build-to-build history on the machine-maintained gh-pages branch
(benchmark-action/github-action-benchmark).
Per point the recorded values are the best runner’s whole tuple (its p50, p99 and
throughput together, never a per-metric minimum across runners), which approximates the
code’s capability rather than the machine lottery.
The store carries roughly three times as many series as this page charts — every recorded point, including the ones no family groups. Open the raw per-series trend browser ↗ — one chart per series, zoomable, with per-point commit links. It is the archive; the families above are the reading.
A second, parallel store records the same transcript set from a fixed self-hosted
machine (perf-local workflow), pinned to one logical CPU — the bench-local trend
browser ↗. Every family chart above
draws one store at a time, chosen by the source selector at the head of each chart
block (bench-local by default); switching redraws every chart on the page from the
other store and the choice is remembered. Both raw browsers are stock
benchmark-action pages with no selector of their own, so the deploy overlays a
three-way nav banner on the mirrored copies — each one names the other store and links
back here — rather than leaving a reader there believing the site has a single store. A family the selected store has not recorded
says so rather than drawing an empty axis. The two stores answer
different questions and are never mixed. bench-local is the trend instrument, and the
page opens on it: same silicon every point, and both arms of each paired point come from
the same pass. GitHub-hosted points swing by up to about 50% between consecutive commits —
bimodally, which looks like two runner machine types — so a 2× step in one row disappears
there: the 1 KiB LKV heap regression of 2026-10-01 reads 27 → 54 ns on bench-local and is
invisible on the hosted store. The hosted store therefore reads as a portability
envelope (best-across-three-runners per series): what the code does on whatever machine
a stranger rents. bench-local records host characteristics on every point (each point’s tooltip carries the
host descriptor and, beside it, the measurement conditions the point was taken under — the
bench CPU’s foreign time, CPU pressure and context switches, with a CONTAMINATED flag on a
sample that stayed contended; host.txt next to the store holds the full lscpu capture). A two-point
regression verdict is only ever read from the bench-local store or from a same-host
interleaved A/B, never from the hosted one.
Provenance & auditability¶
Because the Performance page is regenerated on every docs build, each render carries
a CI stamp — date, commit, run, and runner OS — so any published figure is auditable
back to the exact deploy that produced it. Every main push additionally archives
all raw benchmark transcripts as a per-commit CI artifact and records every
(mode, size, fan-out, endpoints) point — latency, throughput, and memory — into a
persisted build-to-build history on the machine-maintained gh-pages branch. The
numbers on the Performance page are one run; that history is the durable signal, and
it is what the trend charts and the soft alert read from.
The history is two stores. bench-local (dev/bench-local/data.js, one pinned
self-hosted CPU) is the trend instrument, and the trend charts open on it: every point
is the same silicon, and both arms of each libtracer-vs-Zenoh point come from the same
pass, so the zenoh ÷ libtracer ratio is computed there and only there. The GitHub-hosted
store (dev/bench/data.js) keeps the best of three hosted runners per series; its
consecutive points swing by up to about 50%, bimodally, so a 2× step in one row is lost in
it. It stays one selector away as the portability envelope, with no ratio view, because
its two arms need not come from the same runner.