Failable allocation and backpressure¶
Scope: this page describes the C++23 reference implementation’s allocation seams and its failure semantics. It is not the standard. The protocol-level obligation — that a receiver’s bounds are injected resources and that exhaustion is answered by value — is described implementation-independently in
../reference/09-memory-substrate.md; the normative surface is../spec/v1.md. For the deployment question these seams serve — which stage is allowed to queue or shed, and how to size the rest so it does not — see../reference/22-backpressure-and-sizing.md.
The rule¶
Every allocation a peer can provoke must be able to report exhaustion as a value. None may
abort. A frame arrives from the network, a peer chooses its nesting depth, its node count and
its link count, and several of those allocations sit behind no ACL. If any of them reports failure
by throwing, then on a -fno-exceptions target the throw lowers to the toolchain’s abort() stub
and a peer can reboot the node by sending a large frame.
std::pmr::memory_resource::allocate reports failure by throwing. That is not a defect of the
implementation using it — it is the contract of the type, and no wrapper can add a return channel
to a function whose only failure signal is an exception. So a target that compiles without
exceptions needs a second, structurally different seam for the failable allocations, one whose
failure is a nullptr return. That seam is tr::mem::block_source_t
(ADR-0065 — failable allocation gets its own seam).
The rejected alternative was routing failable allocation through the existing mr_ seam with a
non-throwing memory_resource subclass. It fails for a reason that has nothing to do with taste:
allocate has no way to say “no”, so a non-throwing resource must either abort or return a
pointer, and the caller has no branch to write.
Where the rule is not met today¶
The rule above is an obligation, not a description of the tree — read it as “must”, never as
“does”. One site still reports exhaustion by throwing, so “nothing on the delivery path can
abort” is not yet unconditionally true. (There were three. The CAN egress window table went away
in #1110, which deleted the table rather than bounding it — the windows are derivable from the
payload length and the mode, so split allocates nothing at all. The peer-driven label-table
binds of #603 defect 1 went away when route_handle_t moved onto the block_source_t substrate:
every byte of label state is drawn from an injected source and every door answers exhaustion by
value, degrading the flow to the full-route FWD{WRITE} form.) The one that remains is NOT
peer-provoked — read the “who provokes it” column rather than assuming it is, because the provoker
is what decides whether a bound is an admission-control problem or a sizing one. It is
named here so a bounded deployment prices it rather than rediscovers it; this list is the
sites found by the sweeps behind #603, #850 and the CAN egress review, and is not claimed to be
exhaustive:
Site |
Code |
Who provokes it |
|---|---|---|
|
|
on the MCU profile only, anything concurrent — a FreeRTOS context switch between the probe’s free and the |
The nothrow seams and the status legs described below are real and are what makes each covered site answer by value. They do not make the row above go away, and #848 (the WS/TCP/UDP/CAN egress gather) does not close it either.
The one injected source, and the four channels it feeds¶
graph_t’s constructor used to take four seams — a std::pmr::memory_resource*, a
mem_backend_t* and two block_source_t*. Since #873
phase 1 it takes one tr::mem::block_source_t*, defaulted so an unconfigured host gets the
platform heap and byte-identical behaviour. Since RFC-0028 slice 10 that is the ONE reference
constructor (core/include/libtracer/graph.hpp:graph_t::graph_t(mem::block_source_t& src; the parameter’s whole contract is
documented on it), which also takes the graph-wide graph_hooks_t.
The four channels are still four — the contracts genuinely differ, and that is why the graph builds adapters rather than pretending one vocabulary serves all of them — but a deployer now sizes and censuses ONE store.
Channel |
Reaches the source through |
What it allocates |
Exhaustion |
|---|---|---|---|
|
directly — the table sub-pool on a default graph, the injected root otherwise |
the graph’s own tables (#1778): the vertex tree, the vertex index, the link index, edge tables, the seam park, the creation catalog, the identity record and the declaration lists. Its pmr predecessor |
|
|
|
the graph’s payload byte |
|
|
directly — it IS the injected source |
every allocation a peer can provoke |
|
|
directly |
the graph-level value source (#1777: it replaced the ring alias |
|
The failure convention does not leak. The substrate speaks raw nullptr; nothing wraps a
refusal in a result_t, and no channel falls back to the global heap. Only the payload adapter
translates, at its own boundary and for its own contract’s reason.
The process default folds back onto the pre-#873 objects. A graph handed nothing points
value_backend_ at mem::heap_backend() — the
exact objects the retired defaults named, including the ADR-0047 §2 devirtualized HEAP reclaim
arm — so a default-built graph gains no indirection anywhere on the write path. The adapter is
constructed either way (it is three words) and used only when a host injected
something. ctl_ and the adapter are declared last in the object on purpose: no hot path
reads them, so placing them there leaves every other member at the byte offset it had before the
seam existed, which keeps the forward-hop bench measuring the same layout (graph.hpp:graph_t::ctl_).
What still bypasses the injection, at #873’s close. Two channels, both settled rather than
pending. The LKV hazard-slot nodes (lkv_slot.hpp) stay new (std::nothrow) on the global heap:
#873 phase 2 built the migration and reverted it on its own gate (+22.7 % on hazard-node
acquisition, +3.5 % on the free-list-hit steady arm, ranges disjoint against a −0.12 % A/A null),
which the ruling named an acceptable terminal outcome. And the plain std::vector<std::byte>
sites — the KEY containers and the read-back encoders’ staging buffers — are an allocator swap
only in appearance, because their container type is fixed by the signatures they cross; the
std::vector growth behind tr::detail::try_reserve / try_push_back migrates only where the
element type allows it, and the sites that cannot are annotated in place.
Phase 3 widened value_backend_’s row above. Every segment the graph’s READ-BACK encoders
mint — :point, :settings, :settings.app, :acl, :children, the identity record, the
stats block, an app field’s stored bytes and the subscriber / mount-route records subscribe
composes — was reaching the global heap through view::over_bytes’s one-argument overload, so a
peer’s READ escaped the injection. They now draw from the graph’s own backend (published as
graph_t::value_backend()), and a refusal is the BACKPRESSURE every one of those sites already
returned. The full ledger is
reference 09 §”The channel ledger, at #873’s close”.
fwd_router_t carries the same failable seam separately as its router_planes_t::rx plane
(core/include/libtracer/fwd_router.hpp:router_planes_t::rx), because the terminus arena decode belongs to the
router’s receive thread rather than to the graph. It carries a fourth injection beside it, and
for a different contract: flat (fwd_router.hpp:router_planes_t::flat, documented on the member), the mem_backend_t every rope flatten on
the forward and terminus paths draws its owned segment from — the byte-buffer seam, with cache
hooks and a refcount, which the block source is not. The split is graph_t’s ctl_ /
value_backend_ split one layer out. Like value_backend_, an injected flat MUST be
thread-safe (ADR-0060 §2): all of those sites but one run on a transport child’s receive thread
and the remaining one on the writer thread, and the segment it hands out self-routes its reclaim
on whichever thread drops the last reference. A bare pool_t is not thread-safe and must not be injected here;
synchronized_pool_t<Sync> (core/include/libtracer/mem_pool.hpp:synchronized_pool_t) is the in-tree
composition, and its critical section is a compile-time policy: synchronized_pool_t<> over the host guard for the multi-core
spinlock, tr::esp::critical_pool_t for the single-core interrupt-disable variant.
It carries a fifth injection with the same thread-safety force and, again, a different
contract: egress (fwd_router.hpp, documented on the constructor), the mem_backend_t the
terminus reply head and its RFC-0024 mint draw from. It is kept separate from flat because a
reply head is egress construction sized against route bytes, not the payload bytes a flatten
seam is sized against — folding it into flat would silently re-scope a slab already sized for
flattens (ADR-0074;
see What flat covers below).
A bounded node must inject all of them. Injecting the graph’s root alone leaves every
peer-driven allocation on the global heap, where the failure mode on a -fno-exceptions target is
the abort this seam exists to remove. A host reaching for “one slab, whole stack” points all three
plus the router’s rx, its flat and its egress, and the transport-receive backend at the
same underlying slab; the composition guidance is in
../reference/09-memory-substrate.md.
“One slab, whole stack” is a direction, not a state the tree is in, but the thread-safe
byte backend it needs is no longer what stops it. Both spellings of the synchronised pool exist:
synchronized_pool_t<> for a multi-core host’s mutex guard and tr::esp::critical_pool_t for a single-core
target’s interrupt-disable critical section (zero-copy and flatten §5),
so a constrained node points value_backend, flat and egress at a bounded pool rather than
heap_backend(). With egress injected there is no terminus byte source left on the global heap;
the reply head that was the last such source is detailed in
What flat covers below.
The resolver’s flattens and ownership copies are inside the bound on both tiers. The router
hands flat to its op_resolver_t, which threads it through whichever node reader runs — the rope
tier’s view_node::ensure_cache and its multi-link, non-host view_node::own_wire fallback
(core/src/op_resolve_view.cpp). The ownership COPY of a written value below its vertex’s
copy-or-share threshold is, since RFC-0028
slice 5, not a flat draw at all: it lands inline in the value’s own block, drawn from the
graph’s block_source_t (graph_t::control_source(), the source every published value is drawn
from), on both tiers — so the span tier asks flat for nothing. core/tests/terminus_flatten_backend_test.cpp
pins every half — the same request costs strictly fewer global allocations with the seam on a
slab, the stored value’s bytes are asserted to lie inside the injected slab, and a refusal is
answered rather than read short.
The bound covers the arena, and every rope flatten on the forward and terminus paths beside
it. Read the arena claim below as a claim about the arena only. An unguarded flatten is not a
visibly-failed one: view::over_bytes maps an empty span to an ENGAGED-empty optional by design
and graph_t::write stores it and reports success, so an ingress COMPACT flatten that OOMs
would REPLACE the subscriber’s last-known value with nothing. Both halves are closed at every one
of the router’s materialize() call sites — they draw from flat, and each answers a refusal by
value (the rows in Status legs below). Read that literally too: the named sites
in that table, not “the router’s allocations”.
What flat covers, and the two things beside it that it does not¶
#766 left three rope-tier heap sites outside the injection; #793 closed one of them and #801 closed
the span tier’s counterpart, which #766 had not counted at all; #795 closed the reply head — the
last one — with its own egress seam. None of what flat does not cover is an oversight, and the
reason each is out is different — so they are named here rather than left to be rediscovered:
Site |
Code |
Verdict |
|---|---|---|
|
|
Closed by #793, then moved by RFC-0028 slice 5 into the value’s own inline block on the graph’s source (the |
|
|
Closed by #801, then moved by RFC-0028 slice 5 like the rope tier’s: the copy is the value’s own inline block on the graph’s source, so this tier no longer allocates from |
The terminus arena |
|
Already bounded, by a different seam. It draws from the router’s injected |
The reply head segment (and its RFC-0024 mint) |
|
Closed by #795, by its OWN injection. Not folded into |
The composed-root folded READ’s per-node POINT headers |
|
Closed by #831, on the EXISTING value seam. No new injection: these are payload framing (each header’s length field wraps that node’s stored TLV and the name record below it), so they belong to |
The |
|
Closed by #831, same seam, same commit. The same defect class on the other folded read, and the one the wire actually reaches first: a |
The reply head is reachable from the bounded-node resolve path — every terminus reply allocates
it, on both tiers (assemble_reply lives in a shared TU, so the arena/MCU terminus runs
it too), and on a mint a second fixed 12-byte PATH_REF segment beside it. Its failure half was
always closed: a null segment returns an empty rope, which or_backpressure
(core/src/op_resolve_walk.hpp:or_backpressure) turns into an addressed kind=ERROR BACKPRESSURE rather than
a silent drop. The bound is closed by a dedicated egress injection on fwd_router_t and
op_resolver_t, threaded to both assemble_reply allocation sites — and, since #887, to
fwd_router.cpp’s bus-NAME-hop rejection reply, which now builds its head through the same
assembler rather than through a throwing std::vector of its own.
It is deliberately not flat, for a reason that is about the seam’s contract, not about
effort. flat is documented — in a public @param on fwd_router_t and op_resolver_t — as the
backend every rope flatten draws from, and a deployment sizes its slab against that sentence.
A reply head is not a flatten; it is egress construction, and its size tracks the route bytes
rather than the payload. Pointing it at flat would silently re-scope an injection somebody has
already sized, so a node that budgeted a small pool for flattens could begin refusing replies it
used to send. So the terminus egress gets its own mem_backend_t, defaulting to the global heap
so an un-injected node is byte-unchanged; a bounded node points it at its slab and this last
terminus allocation joins the bound. The seam decision is
ADR-0074.
The honest statement is now: flat covers every rope flatten and every rope-tier ownership copy
on the forward and terminus paths; the reply head and its mint draw from the injected egress
backend, answer exhaustion by value, and — with egress pointed at the node’s slab — are bounded
too, leaving no terminus byte source on the global heap.
The block source¶
block_source_t is a bytes-in / void*-out interface whose allocating method is
[[nodiscard]] void* try_alloc(std::size_t bytes, std::size_t align) noexcept
(core/include/libtracer/mem_source.hpp:bump_source_t::try_alloc). The noexcept is the whole point: the override
cannot throw, so the caller has exactly one branch to write.
Four implementations ship:
Source |
Construction |
Behaviour |
|---|---|---|
|
free function, process-wide |
wraps the platform allocator; the root of the host slab pool (#1777), and the default itself where a build binds |
|
free function, process-wide |
serves nothing; makes a |
|
|
carves from |
|
caller-supplied slab plus a caller-supplied span of size classes |
segregated exact-size free lists; recycles, so it suits a long-lived seam |
bump_source_t is the nothrow twin of std::pmr::monotonic_buffer_resource, and the upstream
parameter is what makes it a capability-preserving substitution. A monotonic resource also spills
past its buffer, but it spills to a throwing resource — the abort again. A bump_source_t spills
to whatever block_source_t the caller named, so the same large input still succeeds where memory
exists and fails as a value where it does not (mem_source.hpp:bump_source_t).
The decode arena¶
The branch-write decode allocates its arena on the calling thread’s stack and names the graph’s injected failable seam as the overflow upstream:
std::array<std::byte, 4096> stack;
mem::bump_source_t src(stack, *ctl_);
(core/src/graph.cpp:graph_t::write_branch.) Three properties follow, and each closes a different failure mode:
A bounded node that injected
ctlgets its own store here too. The overflow leg draws from that injection rather than from the global heap, so the node’s memory bound covers this arena (graph.cpp:graph_t::write_branch). Read that literally: it is a statement about the decode arena, not a general one about every allocation near it. Each seam is covered because it was injected and the site was pointed at it, one site at a time — the router’s flattens went uncovered for a release precisely because they looked like they were included in a sentence like this one (#730).The default is unbounded.
ctl_defaults todefault_root()(the host slab pool over the platform heap, #1777), so a host that injects nothing sees the same capability it had with an unbounded resource.Exhaustion is a value. A tree larger than the slab still decodes where the upstream can serve it, and where the upstream cannot,
decode_intoreturns an error the caller converts to a status.
The arena is structure storage — a node array and the walk’s open-node stacks — so its size is
independent of the payload’s byte count. No node-counting pre-pass exists, and none is needed: the
seam alone carries the failure. The three draws that make the RX decode path peer-provokable — the
node array’s growth, the sink’s open-node stack, and the walk stack’s spill past its inline slots —
are enumerated in the changelog (core/CHANGELOG.md:2512-2516), which is the citation for that leg being
closed.
TLV nesting has no depth constant. Depth is bounded by the receiver’s decode resources, and
exhaustion rejects with TLV_NESTING_TOO_DEEP — the status
RFC-0006
defines for “exceeds this receiver’s decode resources”.
Status legs¶
Allocation |
Site |
Failure answer |
|---|---|---|
Branch-write flatten into the value backend |
|
the refusal’s cause (#917): |
Field-write flatten into the value backend |
|
same two-verdict split as the branch-write row above |
Branch-write root key render ( |
|
|
Branch-write parse-key copy ( |
|
|
Branch-write decode arena |
|
decode error → |
Per-delivery COMPACT flatten (egress) |
|
the delivery is dropped |
Per-delivery frame build |
deleted (#885) — the COMPACT leg gathers off a stack head ( |
n/a: there is nothing left to refuse |
Ingress |
flatten |
the empty flatten fails |
Ingress |
flatten |
the delivery is dropped; the subscriber keeps its last-known value |
Bus-name rejection reply flatten (cold) |
flatten |
the frame is dropped by value — no reply |
Terminus per-node span materialize (rope tier) |
flatten |
a refusal on the reply’s own route bytes ⇒ |
Terminus ownership flatten (rope tier, ADR-0053 ⑤, MULTI-link) |
flatten |
the write is not stored — the vertex keeps its previous value — and the reply is |
Terminus ownership copy (rope tier, ADR-0041 §2, SINGLE-link) |
copy |
the write is not stored — the vertex keeps its previous value — and the reply is |
Terminus ownership copy (both tiers, ADR-0041 §2; one inline |
drawn from the graph’s source by |
the write is not stored — the vertex keeps its previous value — and the reply is |
Terminus reply head + mint (egress seam, both tiers) |
alloc |
a refused RESULT head yields an empty rope; |
The four terminus rows are the resolver’s, reached through the same flat the router injects
(#766, #793, #801), and all four are exercised by core/tests/terminus_flatten_backend_test.cpp —
including two sweeps that move the refusal point across every draw one request makes and require
each outcome to be a drop or an addressed BACKPRESSURE, never a kind=RESULT built on a short
span and never a vertex left holding a third value.
The fourth row is #801’s — the same ADR-0041 §2 copy on the span tier, whose refusal is answered
through the empty view alone and never through spans_intact(); see the seam table above.
The third is #793’s. It is worth stating separately from the flatten above it because
the two are branches of one function that used to allocate from two different allocators:
own_wire flattens a multi-link subrope through the injection and copied a single-link one
through view::over_bytes’s global heap. Which branch a peer takes is decided by where its
fragmentation happens to fall, and the common case — a whole payload TLV landing inside one RX
segment — took the uncovered one. Measured the way #766 was: with flat armed on a slab, a
two-link FWD{WRITE} whose payload TLV is contiguous consulted the seam zero times before
#793 and once after, and the arm’s global-heap new count falls 16 → 15 accordingly. over_bytes
gained a mem_backend_t& overload for it (core/include/libtracer/mem_heap.hpp); the pre-existing
one-argument form is untouched, which is how every other call site stays byte-identical.
All three router-ingress rows draw from the router’s injected flat backend, and all three are
exercised by core/tests/fwd_flatten_backend_test.cpp, which injects a backend that refuses on
command. Each of the three has its own case — a row nothing can fail is a row nothing pins, so
reverting any one site’s seam fails that site’s case and no other.
Two of those three rows are answered by a decode, not by a guard. The refusal early-outs
beside the ADVERTISE (core/src/fwd_router.cpp:if (route.empty() && head->child1_total != 0) return, still an empty() test — that arm reads a
span through the make-contiguous seam) and bus-name (core/src/fwd_router.cpp:fwd_router_t::on_frame_rope_impl, since
#917 a !flat test on the named refusal rather than an empty() guess) flattens are redundant
with the wire::tlv_node_t::over that follows each — deleting either changes nothing observable, verified by
ablation — and the code
says so at both sites. They are kept so the reason the operation failed is the OOM rather than the
codec’s leniency, and nothing here cites them as proven guards. What the test pins at those two
sites is the seam: with the site back on the default heap the flatten succeeds under the
injection and the case fails. Only the ingress COMPACT row has a guard that is independently
observable — remove it and the LKV-preservation assertion fails, because that is the site where an
empty flatten was stored and reported as success.
That file is the reason these rows are worth writing down at all. Guards alone, without an injectable seam behind them, are not a design: nothing can make those flattens fail, and a guard nobody can fail is a guard nobody can prove.
The key render and its parse copy are nothrow so that OOM soft-fails the branch write as
BACKPRESSURE, the injected-resource status — never an abort on the writer thread
(graph.cpp:graph_t::write_branch).
The remote-delivery leg answers differently on purpose. A stored write that reached its LKV has
succeeded; the fan-out to one subscriber is a separate obligation, and a subscriber missing
one value under heap exhaustion is valid delivery behaviour where failing the write is not. Every
per-delivery allocation on that writer-thread leg is nothrow, and a failed flatten or frame build
drops that one delivery (core/src/fwd_router.cpp:fwd_router_t::deliver_remote). Dropping invisibly is the part that
needs an answer, which is why graph_t::delivery_drops() exists
(core/include/libtracer/graph.hpp:graph_t::delivery_drops): four relaxed monotonic counters — no_target, denied,
out_of_memory, fan_out_truncated (graph.hpp:graph_t::delivery_drops_t) — incremented only on a drop, so the
delivering path is byte-identical while nothing drops. The net plane adds to the same four
through one public door, count_external_drop (#1068), so a COMPACT delivery shed for want of
memory is as visible as a local one; denied is not among that door’s causes because a refusal
is counted at the WRITE gate itself, on every plane. Nothing in the library reads them; a
deployment chooses whether to alarm. What they count is shed deliveries: the sharpest OOM shed
is an assign whose pending mark cannot be allocated, which abandons the vertex’s whole
subscriber set and still returns success (core/src/graph.cpp:graph_t::mark_pending), so it moves the counter by
the fan-out width rather than by one (#896). The sharpest used to be a HANDLER write whose notify
clone failed; #1505 deleted the clone — the handler’s value is delivered without one — so that
shed cannot occur at all, and the leg that counted it is gone rather than narrowed.
A dropped fresh ADVERTISE on the COMPACT leg self-heals: the peer answers the unknown label with
HANDLE_NACK and the next delivery re-advertises (fwd_router.cpp:fwd_router_t::deliver_remote). Since #885 the router
itself no longer has a way to drop one for want of memory — the frame is gathered off a stack head
— so the surviving drop is the transport’s, not the label plane’s.
Legs that throw, and their nothrow twins¶
rope_t::to_iovec builds the scatter-gather span table by value, and its reserve throws on OOM —
an abort() under -fno-exceptions (core/include/libtracer/rope.hpp:rope_t::to_iovec). The terminus reply
egress builds that table on every send, so on a fragmented heap it was a reachable abort. The
nothrow twin is rope_t::try_to_iovec(std::vector<std::span<const std::byte>>& out) noexcept
(rope.hpp:rope_t::try_to_iovec): it clears out, sizes it to link_count() through tr::detail::try_reserve,
and returns false without touching out further when the table cannot be grown — the caller drops
the reply (rope.hpp:rope_t::try_to_iovec).
Be exact about what that helper buys, because the twin is named for its signature, not for an
absolute guarantee. tr::detail::try_reserve (core/include/libtracer/mem_heap.hpp:try_reserve) routes
the ordinary throwing std::vector::reserve through try_grow (core/include/libtracer/mem_heap.hpp:try_grow), which
converts its failure into a false the caller answers with BACKPRESSURE — the whole of the
improvement over to_iovec. The allocation whose failure is reported is the one the vector
actually performs, so there is no probe-then-commit window to lose on a hosted build
(#923, which folded in
#850).
Under -fno-exceptions there is nothing to catch — libstdc++ turns the bad_alloc into a bare
abort() inside reserve — so there the helper still probes first and the window survives. That
is the try_reserve row of §”Where the rule is not met today” above, and every try_* helper on
this page inherits it on that profile. The fix for a site that must survive exhaustion there is
the ADR-0065 failable seam, not a better try_reserve.
Both forms remain. to_iovec is correct wherever the caller is on a host build with exceptions or
holds a table it sized beforehand; any path a peer can drive uses try_to_iovec.
The forward hop’s own entry table has a length of ~6 + link_count — again the sender’s choice, and
not even at the terminus. It is gathered into a mem::block_array_t over the injected rx, and
exhaustion drops the hop rather than emitting the entries that fit: a partial iov is a truncated
FWD on the wire, which is strictly worse than silence, and FWD is not delivery-guaranteed so the
sender retries.
The host TX gather¶
A host integration that marshals sends onto a separate task must copy the caller’s spans before returning, because the spans are gone by the time that task runs. The shape that matters is where the copy’s allocator lives.
A gather written as a braced initializer over a std::vector — building the work item and the
payload copy in one expression — allocates the vector with the throwing allocator even when the
work-item shell is guarded with new (std::nothrow). The guard covers the shell; the payload copy
inside the initializer is unguarded, so a reply-sized copy meeting heap exhaustion aborts the node.
The nothrow-end-to-end shape splits them: size the total, allocate the payload buffer with
new (std::nothrow), memcpy each span in, then allocate the work-item shell separately — also
new (std::nothrow) — moving the buffer in. If the shell allocation fails the initializer never
runs, so the buffer is not moved from and frees itself on return; there is no leak on either arm,
and an allocation failure becomes the same drop the link’s send contract defines
(httpd_ws_link_t::queue_send, integrations/esp-idf/libtracer/httpd_ws_link.cpp).
The copy is structural; the allocation is not. httpd_ws_link_t fronts the gather with a
once-per-link pool of TX work slots claimed lock-free (a CAS scan — senders run on any task, the
httpd task releases the slot as the send drains): a steady-state frame gathers straight into the
slot’s inline payload and allocates nothing. Past the inline capacity there may be a SECOND bounded
size class, if an integrator declared one (#1566, tx_large_bytes x tx_large_slots): a frame in
its band borrows a buffer from that class and still allocates nothing, and an exhausted class is a
named drop rather than a fallback, because a fallback would hand back the unbounded transient
footprint the declaration was made to bound. The nothrow-end-to-end heap shape above therefore
remains for exactly one arm — the exceptional tail past every declared class, which keeps its
pooled shell and takes a nothrow heap payload — with the drop-on-OOM contract on it. A momentarily exhausted pool has no arm: the
pool is the link’s outstanding-send bound, and a send that finds it full is dropped and counted
rather than posted from a heap work item, because the fallback bounded the in-flight depth by the
heap instead of by the queue behind it and left the event unobservable (#949; the exhaustion
policy is ADR-0039 §4 / ADR-0042 §2, drop and count). The RX mirror is a once-per-link scratch
buffer: a frame that fits reads into it and is delivered borrowed (the httpd task is the only RX
thread and delivery is synchronous, so one scratch needs no lock); only an oversized frame takes
the exact-size nothrow allocation.
That RX default serves the decode-and-forget consumer, which is what the router is. A consumer
that must OWN the payload past the callback copies it out of the borrowed scratch itself, so its
inbound path is the recv plus a second full copy per frame — and for that one, the link takes an
optional rx_backend (#1565): a bounded, INTEGRATOR-injected mem_backend_t the frame is recv’d
straight into, delivered up as an owning rope the sink may keep. The library still owns no buffer
and picks no size; a source that refuses is a counted drop (rx_dropped_pool) and never a heap
fallback, which is the same fail-closed answer the TX side gives. Borrowed remains the default,
and a link that names no backend is byte-for-byte the link that shipped before.
Pitfalls¶
A bump block is never reclaimed. bump_source_t has a cursor and no free list, so a source that
outlives one burst of work fills monotonically and then refuses everything — the node does not
abort, the seam behaves exactly as specified, it simply stops working. Construct one per operation
(as the branch-write decode does) or reset() it between operations (mem_source.hpp:bump_source_t::reset).
A long-lived bounded seam — a router’s rx, a graph’s ctl — wants pool_source_t, which
recycles (mem_source.hpp:bump_source_t).
bump_source_t is also single-threaded by contract: a bump cursor is not synchronized, and its
intended use is a function-scoped buffer on the calling thread’s stack (mem_source.hpp:bump_source_t).
Exhaustion of the block seam does not have one answer. The reject belongs to the operation, not to the seam, and the three in-tree consumers answer differently:
Consumer |
Answer on exhaustion |
Why |
|---|---|---|
Terminus decode |
|
the status RFC-0006 defines for “exceeds this receiver’s decode resources” |
Branch-write decode |
|
it cannot distinguish “the value did not parse” from “the arena ran out”, and does not try |
Rope forward hop |
nothing at all |
a forward hop has no reply channel; its only sound answer is silence |
BACKPRESSURE is what a store answers when its value backend is exhausted. It is not a
property of the block seam. A caller that assumes one status across all three will misclassify two
of them (../reference/09-memory-substrate.md:305).
A pool shared across receive threads is slower than the heap it replaced. See the topology
result below; fwd_router_t::add_child takes an optional per-child source
(fwd_router.hpp:fwd_router_t::add_child, resolved at fwd_router.hpp:fwd_router_t::rx_for) precisely so each transport’s receive
thread owns one. A source shared at wiring frequency — a graph’s ctl — is fine behind a lock.
A size_class_t span is a bound the caller sets, not the library. pool_source_t classes do
not share: a freed 64 B block cannot serve a 128 B request. classes_used() and overflowed()
report what to size the class span against.
Sizing a bounded seam¶
The bump seam’s frame count is a capacity result, not a rate¶
An 8 KiB bump_source_t wired as a router’s rx, decoding a 53-byte FWD, decoded 6 frames
and rejected the next 194
(ADR-0067 §1;
corroborated at core/include/libtracer/mem_source.hpp:decoded 6 frames and rejected the next 194 and
../reference/09-memory-substrate.md:280).
The frame size is load-bearing and a frames-served count without it is not a measurement: what the figure reports is 8192 bytes divided by the arena footprint of one decode of that frame, so a different frame shape gives a different count. Being capacity arithmetic rather than a rate, it does not vary with host or build flags — which is what makes it a design fact about the seam and not a benchmark.
The same result at the router’s own RX seam¶
Those rows come from a different seam. Instrument: the committed bench_rx_source_topology.
Host: 12-core / 24-thread. Median of three 300 ms runs, aggregate forwards/s. T receive threads each
drive a multi-link rope through their own inbound child to their own egress sink, so the source is
the only object two threads share.
T |
shared |
one shared |
per-child |
|---|---|---|---|
1 |
4.10 M |
4.09 M |
4.32 M |
4 |
10.99 M |
2.78 M |
10.80 M |
8 |
15.30 M |
1.90 M |
16.00 M |
24 |
21.77 M |
1.46 M |
21.81 M |
The shared pool falls to a sixty-seventh of its own single-thread rate — per-thread 244 ns → 16,428 ns — a deeper collapse than the fifteenth above, while the per-child pool tracks the scaling heap across the whole sweep. The one point where the medians diverge, T=16, heap 22.6 M against per-child 19.0 M, is inside the per-child run-to-run range and is explicitly not a finding.
Two control arms belong with the table and must not be dropped from it. At T=1 the pool is not faster than the heap at this seam either — 231 ns against 244 ns, ranges overlapping. A single run suggested a 27 % pool win; three runs deleted it. The rule that follows is ownership, not synchronization:
A
pool_source_tis owned by one thread wherever it sits on a per-frame path. Sharing one behind a lock is admissible only at wiring frequency.
Sizing, for a deployment bounding this seam: a 4-link rope’s forward hop settles at 128 B of slab per child in one size class, independent of T (ADR-0067 §3).