RFC 0004 — Remote operation addressing: path-as-route + the FWD/FIELD frames¶
Note
Status: accepted. This page is an accepted change proposal, kept as the record of why the specification reads as it does. Superseded in part by RFC-0029, RFC-0030. RFCs are proposals and history, not the standard. The normative specification is Protocol v1 and the annexes its §3 incorporates; where an RFC and the specification differ, the specification wins. All RFCs, with their status, are listed in the ADR and RFC index.
Field |
Value |
|---|---|
RFC |
0004 |
Title |
Remote operation addressing: path-as-route + the |
Status |
accepted (2026-06-28). Amended by RFC-0029 (accepted 2026-09-30, §12.4): §B’s “a reply does not accumulate |
Author(s) |
AvatarSD (maintainer) |
Created |
2026-06-28 |
Accepted |
2026-06-28 — maintainer/BDFL; no registered second-implementer to object, so the 14-day window is nominal (GOVERNANCE.md §Roles). Implementation tracked by ADR-0035. |
Tracking issue |
|
Target spec version |
v1 (draft refinement — no released v1 yet, so no v2 needed) |
Erratum (2026-07-30), #583: §A’s facet list includes
:statsand:status. Neither was ever implemented, in this RFC’s lifetime or before it; a field read or write to either answersERROR{tr::schema::not_found}. Neither name is in the field namespace, which is{subscribers, acl, children, settings, schema, identity}. §A’s bullet is left as written rather than edited, because deleting the two spellings from an accepted record would decide #584 — which asks whether they should exist — by silently editing the decision it is meant to inform.
Amendment proposed by RFC-0017 (2026-07-28) — still
draft, not in force: §C’sFIELDgrammar admits K = 0 — a selector carrying onlyindex/index_modeand no leadingNAMEaddresses the vertex’s own value, so[n]reaches the value plane and not only a:field. Levels areNAME-delimited, so aFIELDwhose first child is not aNAMEis unambiguously a value-plane selector. §D gains the element operations and the rule that a delivery mirrors the shape of the write that caused it.PATHis untouched — the “children MUST beNAME” invariant restated in §C still stands, and an element index never appears indstorsrc. The amendment takes effect only when RFC-0017 is accepted: as this RFC stands, §C’s grammar is the one reference/03 §index forms describes, where an index is a property of a field step. (ADR-0066 settles the concurrency half of the proposal in advance; it does not accept the RFC.)
Partially superseded by RFC-0008 (2026-07-06, amended 2026-07-06b): the value-based
delivery_modeQoS hint (EVERY/THROTTLED/ON_CHANGE) and themin_interval_nsthrottle referenced in §E are removed fromSUBSCRIBER.qos_settings— the runtime no longer filters delivery by comparing values.delivery_modesurvives redefined as a value-agnostic per-vertex policy (UNCONDITIONAL/IF_NEWER/EXPLICIT), not a per-subscriber value filter.delivery_compact(label compaction) is orthogonal and unaffected.
Summary¶
docs/spec/v1.md §3 (the wire encoding of a remote read/write/await against a path:field) is stubbed (“to be written”). Locally the three primitives are direct router calls and never hit the wire; the wire only appears for a remote operation across a transport, and today the only remote mechanism is a bridge mounting inbound data under a fixed prefix (reference/04 §bridge republish). There is no frame for “operate on an arbitrary remote path:field,” so a web UI cannot read/write/await/subscribe a vertex behind another node — which blocks the TypeScript client SDK higher operations (#56, ADR-0034), the declarative reconciler (#58), and transport-as-vertex orchestration (#83).
This RFC fills §3 with the smallest design consistent with the accepted model:
Path-as-route. A remote endpoint is addressed by its full path from the caller’s own root, walking through transport vertices. A transport/connection vertex (ADR-0027) mounts its peer’s graph under itself: its
:-facets are the link’s own control, its/-subtree is the peer’s tree. Routing is hop-by-hop source-routing — each transport vertex strips its own leading segment and forwards the unresolved remainder to its peer.FWD(0x0F) — a self-describing frameFWD{ op∈{READ,WRITE,AWAIT,REPLY}, PATH dst, FIELD? selector, PATH src, payload? }.dst(forward route) shrinks per hop;src(return route) grows per hop by a zero-copy prepend of the inbound-linkNAME, so forwarders stay stateless and the reply self-routes home viasrc. Loop-free by construction → needs none ofROUTER’s dedup/MAX_HOPS.FIELD(0x10) — encodes the:fieldtail (:subscribers[],:settings.x) thatPATH(NAME-segments only) cannot.Replies are stateless and source-routed back — a
FWD{ op=REPLY, kind∈{RESULT,ERROR} }whosedstis the accumulatedsrc. No end-to-end correlation-id (matching a reply to a specific concurrent request at an endpoint is the transport’s concern); works over unidirectional links; survives a hop reboot.Two planes.
READ/WRITE/AWAITare the one-shot plane (works on data and:fieldcontrol). Streaming never per-sample-remote-writes: a consumer wires oneSUBSCRIBER, then the producer produces locally + flushes, fanning out. A delivery is aFWD WRITEto the subscriber’s data vertex (delivery-is-a-write, ADR-0026), so a subscription delivery and a one-shot command are the same frame.
subscribe remains a WRITE to :subscribers[] (ADR-0006/ADR-0026); ROUTER (0x0D) is unchanged and earns its keep only on the cyclic/multi-path delivery side. The consumer-stored SUBSCRIBER.target is the src route its subscribe accumulated — producer-holds and this RFC are one mechanism.
Motivation¶
The whole point of the reference suite is cross-implementation interop. A second implementer (the TS client, #123) can build a VALUE/SUBSCRIBER/PATH and decode a delivered VALUE, but it cannot express “read /sensor/temp on the device behind this WebSocket” — there is no wire frame for it. The audit in #123 surfaced this precisely: spec §1–§3 are stubbed, no conformance vector carries a remote-operation envelope, and the C++ reference only mounts inbound data. Until §3 exists:
the web UI cannot read/write/await/subscribe a remote vertex (the browser↔robot thesis of ADR-0031 has no wire);
producer-holds fan-out to a remote
target(ADR-0026) has no carrier beyond a fixed mount — a producer cannot deliver to an arbitrary consumer-named target across a transport;the reconciler (#58) and remote
:children[]creation (#82/#83) have no addressing primitive.
This is the keystone wire gap for everything “remote.”
Proposed change¶
A. Path-as-route (normative model)¶
A remote operation is read/write/await (and subscribe/QoS as field-writes) issued against a path that traverses transport vertices. Per ADR-0027, a transport/connection is a / vertex; this RFC fixes its dual nature:
a transport vertex’s
:-facets (:settings,:stats,:acl,:children,:status) are the link’s own control surface, resolved locally;a transport vertex’s
/-subtree is its peer’s graph, mounted — any/-segment below it is not local; it is the address of a vertex on the peer, reached by forwarding the unresolved suffix.
Example (from a web UI rooted at its own node):
/net/<ws://board-ip>/can[0]/ow/<temp_sensor>
└─ local ws connection vertex ──┘ │ │
forwards "/can[0]/ow/<temp_sensor>" ───┘ │
over the ws link to the board │
board resolves "can[0]" (its CAN vertex), │
forwards "/ow/<temp_sensor>" over CAN bus 0 ┘
the 1-Wire bus resolves "<temp_sensor>"
Segments already carry the identifiers — can[0] is the bus number; <temp_sensor> is the device id deduced from CAN advertisement — so the path needs no separate name or destination field: the path-suffix is the address.
Consistency requirement. The suffix a caller routes through a transport vertex MUST equal the prefix that vertex mounts inbound data under (reference/04 §bridge republish). Send-side routing and receive-side mounting are duals and MUST agree.
Location-dependence (consequence, not a bug). A path encodes the route; it is relative to the caller’s root, like a URL or a filesystem mount path. The same physical vertex has different paths from different vantage points. Provenance a consumer needs still travels in the data per RFC-0003, not inferred from the route.
B. FWD — 0x0F (forward / remote-operation frame)¶
FWD is the frame a transport vertex emits to its peer to carry an operation one hop onward. Structured (opt.PL=1); source-routed forward, return-route-accumulated backward; children in order:
FWD (0x0F, PL=1) {
VALUE op ; required, FIRST child — u8: READ=0, WRITE=1, AWAIT=2, REPLY=3
PATH dst ; required — UNRESOLVED forward route (segments only); SHRINKS per hop
FIELD selector ; optional — the :field tail (§C), resolved at the final hop
PATH src ; required — accumulated RETURN route; GROWS per hop (§D)
VALUE kind ; REPLY only — u8: RESULT=0, ERROR=1
<payload TLV> ; WRITE: value/SUBSCRIBER/SETTINGS/… to write. READ/AWAIT: absent.
; REPLY: the result (VALUE for a READ result; STATUS for WRITE-ack/ERROR).
VALUE await_timeout ; optional, AWAIT only — u64 ns; the requester's own deadline, a hint
; the terminus MAY ignore (Amendment 3)
}
Forward (source-routed). The receiving transport vertex resolves the first segment of
dst. If it names one of its own transport children, it strips that segment fromdstand re-emitsFWDover that link. Whendstempties, it resolves to a local vertex here, and the op (+selector) is applied.Return-route accumulation (zero-copy prepend). On every forwarding hop, the vertex prepends to
srcthe oneNAMEthat — in this node’s own address space — names the link theFWDarrived on (the way back). Sincesrcis a structuredPATH(concatenatedNAMEchildren), the prepend is a rope head-insert: existing bytes never move — only the outerPATH/FWDlengthis rewritten (and the trailer is recomputed at egress per hop regardless, so no extra CRC cost). The originator seedssrcwith its own reply endpoint; whendstempties,srcis the complete reverse route in per-node-local segments. The consumer therefore also receives the full source route — exactly the provenance RFC-0003 wanted, now inherent.REPLY is itself a
FWDrouted back:dst = src(the accumulated return route),kind ∈ {RESULT, ERROR}, payload = the result. A reply expects no reply, so it does not accumulatesrc; thesrcchild of a REPLY is still required and is set to the responder’s own endpoint (the vertex that produced the result) — uniform with the other ops, available as provenance, unchanged hop-to-hop.Terminus-reply asymmetry (load-bearing). Each
srcsegment is meaningful only at the node that prepended it (its local name for an inbound link), so the terminus does not resolvedst[0]of the reply — it emits theFWD{REPLY}(whosedstis the request’s accumulatedsrc) unmodified over the link the request arrived on. The first reverse hop performs the firstdst-strip (by the same forward step), and so on back to the originator. A REPLY routes by the ordinary forward step but never accumulatessrc. This asymmetry is what makes the per-node-local return route compose correctly.opis the first child so a forwarder can dispatch without parsing the whole frame.FWDis loop-free by construction: the forwarddstis explicit and shrinks monotonically per hop, so a delivery travels exactly as far as its route names and no further. There is no visited-set and no revisitERROR— adstthat spells out a physical cycle simply routes around it as many times as the route names, then stops. It carries noorigin/hop_count/dedup — those stay inROUTERon the multi-path delivery side (§E). See the erratum below — this clause previously requiredERROR=INVALID_PATHfor adstrevisiting a node.
C. FIELD — 0x10 (control-plane selector)¶
The :field tail of an address. PATH (0x06) encodes only /-segment NAMEs; FIELD encodes the chain after the : separator (reference/03). Structured (opt.PL=1); one level per field-chain element, in order, depth ≤ 8:
FIELD (0x10, PL=1) {
level_1 ... level_K ; K ≤ 8
}
where each level =
NAME field_name ; e.g. "subscribers", "settings", "app"
VALUE index ; optional — u32 index for [N];
; absent + index_mode=ELEMENT ⇒ append/list "[]";
; index_mode=WILDCARD ⇒ "[*]" (subscriber-path targets only)
VALUE index_mode ; optional — u8: SCALAR=0 (no index), ELEMENT=1 ("[N]"/"[]"),
; WILDCARD=2 ("[*]"); default SCALAR
:subscribers[3] → FIELD{ NAME "subscribers", VALUE u32=3, VALUE u8 index_mode=ELEMENT }.
:subscribers[] (append) → FIELD{ NAME "subscribers", VALUE u8 index_mode=ELEMENT } (no index VALUE).
:settings.app → FIELD{ NAME "settings", NAME "app" } (the example changed with RFC-0022, which emptied the flat settings.<knob> namespace; the FIELD grammar did not).
[*] (index_mode=WILDCARD) MUST be rejected with ERROR=INVALID_PATH outside a subscriber-path context, mirroring reference/03.
PATH (0x06) is untouched — its “children MUST be NAME” invariant and every existing parser stand.
D. Operation semantics + replies¶
Amended by RFC-0021 (accepted 2026-08-01): a wire
SUBSCRIBER’s optionalPATHchild — undefined by this section — is the delivery route in the producer’s own frame, andSUBSCRIBEis sufficient authority to install it. RFC-0021’s implementation is deferred past v0.7.0, so the reference implementation still discards that child; the normative text is RFC-0021’s.
|
Payload |
One-shot |
|---|---|---|
|
none |
|
|
the TLV to write |
|
|
none (+ optional |
|
subscribe is a WRITE of a SUBSCRIBER to a :subscribers[] field — no new op (ADR-0006). QoS is a WRITE of SETTINGS to :settings….
A READ of an array :field (e.g. :subscribers[]) returns its members wrapped in a POINT (0x07) — the established structured introspection-result container (the :schema POINT already carries SUBSCRIBER children) — whose children are the populated slot TLVs in slot order, each a zero-copy view (§E / ADR-0035 reply rule). A READ of a single slot (:subscribers[3]) or scalar field returns that TLV directly.
Replies are stateless, source-routed back. A one-shot reply is a FWD{ op=REPLY } whose dst is the src that was accumulated on the way in (§B). Forwarders hold no per-hop request state — the return route lives in the frame, so a hop may even reboot mid-operation and the reply still routes. The reply works over unidirectional transports (it does not need the inbound link to be bidirectional). The reply-kind is just {RESULT, ERROR}; there is no DELIVERY/KEEPALIVE/DIGEST reply-kind — those are not replies (see §E). A kind=ERROR reply’s payload is a STATUS TLV carrying one ERROR (0x08) child (RFC-0002 error model); the concrete ERROR code set is pinned by the ERROR registry (RFC-0001 §C/E, #8), so until #8 lands the fwd-reply-error vector uses a provisional STATUS{ ERROR u8 } shape and the codes finalize with #8.
There is no end-to-end correlation-id. The src route delivers a reply to the right endpoint; matching it to a specific outstanding request at that endpoint (e.g. a consumer with many concurrent READs) is the transport’s concern — a transport-level stream tag for WebSocket, pipelining/ID-matching for CAN — never a field in FWD. Route = inter-hop; tag = intra-endpoint; a route is not an id.
Two planes — direct (one-shot) vs standing (streaming). FWD READ/WRITE/AWAIT are the one-shot/synchronous plane: read a snapshot, write once, block for one. Streaming data does not per-sample remote-write. A consumer creates a local receiving endpoint and WRITEs one SUBSCRIBER into the producer’s :subscribers[] (a control write, §C); thereafter the producer produces locally and fans out — fills its own endpoint’s segment memory (rope-chaining slices for large/scatter data) and flushes, which delivers to each subscriber. The flush, not a client, drives every delivery.
A delivery is a FWD WRITE (load-bearing). When a producer fans out to a remote subscriber, that delivery is a FWD{ op=WRITE, payload=VALUE } routed to the subscriber’s data vertex via the stored return-route. So a subscription delivery and a one-shot command are the identical wire frame — the only difference is whether a standing SUBSCRIBER produced it or a client issued it once; the target cannot (and per ADR-0026 claim 2 must not) tell them apart. Likewise keepalive / digest / liveness / async-write-ack are ordinary VALUE/STATUS writes on the standing channel, not reply-kinds.
E. Delivery / fanout is unchanged¶
Amended by RFC-0021 and RFC-0022 (both accepted 2026-08-01): RFC-0021 rules that a
SUBSCRIBERcarrying aPATHtarget binds the edge to the mount link the route names rather than to the session the subscribe-write arrived on, so a subscription outlives the orchestrator that installed it. RFC-0022 (as replaced by its Amendment 1) moves delivery policy to the subscription — two packed bytes in theSUBSCRIBER’sSETTINGSchild — and deletessettings_t, so the per-vertex:settings.<knob>write surface this section and §D refer to no longer exists.
Producer-holds fan-out (ADR-0026) is unified with this RFC, not changed by it: the src route a consumer’s subscribe-FWD accumulated on the way in (§B) is exactly what the producer stores as the SUBSCRIBER’s target. Each later delivery is a FWD{ op=WRITE, payload=VALUE } source-routed back along that stored route — i.e. a delivery is the same primitive as a one-shot write (§D). For a delivery that crosses a cyclic / multi-path region of the mesh (where the same data could arrive two ways), the delivery FWD is wrapped in ROUTER (0x0D) for (origin, ts) dedup + MAX_HOPS (ADR-0014), carrying the RFC-0003 concrete-path child. Strict source-routed deliveries (a single accumulated route) need no ROUTER; ROUTER earns its keep only where the topology folds.
E.1 Delivery compaction — the route-handle (generalized header-elision)¶
Taken literally, “a delivery is a FWD WRITE” (§D) makes every streamed sample carry its full return route. For a 1 kHz, 4-byte sensor over a 3-hop path that is ~60 B of route on a 4 B payload (~16×) — fine for one-shots, prohibitive for streams (and impossible on CAN, which has no room for a route at all). The fix is not a new mechanism: it is header-elided framing (ADR-0022) generalized to every transport — a compact per-link label that aliases an established delivery route.
Per-link label-switching. A label is meaningful only on the link it was bound for; each forwarding hop swaps it (label-in → its own label-out / route), exactly as a CAN-ID is re-resolved against each bus’s
identity↔pathmap. There is no end-to-end handle — “matching is the transport’s concern” (§D) applied to delivery.Advertise-driven binding (generalize ADR-0030). The upstream advertises the
label ↔ routebinding in-band when a flow starts; each hop learnslabel → (downstream link, out-label)and swaps. There is no setup handshake. Re-advertise on (re)connect is the self-heal (it is also what producer-holds already triggers on reconnect, ADR-0026); a delivery bearing an unknown/stale label is dropped with an error that prompts re-advertise. On a header-elided transport this advertise is the existing id-assignment (transport_can, #55) — no new code.Framing-mode decides whether labels are used (no global policy):
Header-elided transports (CAN): always labeled — the ID is the path; the
identity↔pathmap is mandatory; no threshold.Full-TLV transports (ws/UDP): default full-route (stateless forwarders, no label table). Labels are opt-in compaction, requested declaratively by a
SUBSCRIBERQoS hint (adelivery_compactflag inqos_settings, alongsidedelivery_mode/min_interval_ns). A transport MAY also promote a hot full-route flow to a label adaptively (SHOULD), but the hint is the contract.
State boundary (the cost, made precise). One-shot ops and cold/low-rate subscriptions stay stateless (route in the frame). A hop holds a
label↔routebinding only for the flows explicitly flagged compact that cross it — bounded by (number of compact subscriptions through this hop), and on CAN it is theidentity↔pathmap already paid for. So a constrained ws node forwarding 50 cold reads holds zero label state.
With a 2–4 B label, the 1 kHz example drops from ~16× to ~1.5× overhead. Net rule: one-shot ops pay the full route; high-rate established streams amortize it to a label, by the transport’s framing mode.
Implementation pins (ADR-0035 slice 4, ws side — descriptive, see reference/05 §route-handle). The C++ reference fixes the ws (full-TLV) encodings as transport-plane control frames that ride a link alongside FWD (not part of the FWD frame, no conformance vectors, so the cross-core machine is unperturbed):
Label — a per-link u16 (
VALUE, little-endian; 65 535 usable labels per link —0is reserved to mean “none”), allocated monotonically per link and swapped each hop.ADVERTISE(0x11) —{ VALUE label, PATH route }: bindslabel ↔ route; each forwarding hop stripsroute’s leading segment, allocates its own out-label, re-advertises downstream.COMPACT(0x12) —{ VALUE label, <payload TLV> }: the lean delivery; the terminus expands the label to the bound route and applies the write (delivery-is-a-write, §D).HANDLE_NACK(0x13) —{ VALUE label }: returned for an unknown/stale label (drop, never crash); prompts a re-advertise. Re-advertise on (re)connect is the self-heal (the same producer-holds reconnect trigger).Clearing a link is CROSS-LINK. A node clearing a link’s label state on (re)connect MUST also treat as stale every ingress binding it holds, on any link, whose downstream half crossed that link. A forwarding binding is keyed by the link the label arrives on but names the link the swapped label leaves by, so clearing only the reconnected link’s own tables leaves a mid-chain node aimed at an out-label that no longer exists — and its upstream, which never saw the reconnect, never re-advertises. Dropping the crossing bindings makes that upstream’s next
COMPACTmiss and draw the ordinary stale-labelHANDLE_NACK, which is what prompts its re-advertise. Recovery therefore uses only the three frames above, in situations already specified for them; the wire surface is unchanged. A terminus binding has no downstream half and is never dropped by this rule.And the clear is CONCURRENT with in-flight advertises. Dropping the crossing bindings reaches only the bindings that already exist. An
ADVERTISEruns on its inbound link’s receive thread while the downstream link’s reconnect runs on another, so a hop can mint its out-label and retain its egress route against the pre-clear downstream tables and bind the swap after the drop has scanned its inbound link — recreating the same aimed-at-nothing binding through a window rather than a steady state. A node clearing a link MUST therefore also refuse any forwarding binding whose downstream half was resolved against that link’s pre-clear state, and the refusal and the drop MUST NOT interleave. A refused binding degrades exactly like an unbindable label: the peer’s nextCOMPACTmisses and draws the stale-labelHANDLE_NACK. The wire surface is unchanged.Opt-in —
SUBSCRIBER.qos_settings.delivery_compact(NAME "delivery_compact" VALUE u8, optional/NAME-tagged ⇒ back-compatible). On CAN this advertise is the existingidentity↔pathid-assignment (#55), so the CAN half is unchanged.
F. ACL across hops¶
A FWD is gated twice, by the existing ACL machinery (ADR-0018/ADR-0020):
each intermediate transport vertex’s
:aclauthorizes “may thisorigin_peer_idforward through me” (the forward right);the target vertex’s
:aclauthorizes the actualREAD/WRITE/AWAITat the final hop.
No new ACL machinery; FWD is subject to the same subject-token model as a local field-write.
G. Type-code budget¶
FWD=0x0F and FIELD=0x10 consume two of the sixteen 0x0F–0x1F v1 fast-track slots (reference/05). Both are structured (opt.PL=1), each declaring its own purpose — no generic container is introduced.
H. Conformance vectors (proposed)¶
Add to tests/conformance/vectors/v1/, so the 3-core machine (C++/TS/Rust) validates FWD/FIELD like every other TLV:
fwd-read—FWD{ op=READ, dst=/sensor/temp, src=/reply-ep }(seededsrc).fwd-write-value—FWD{ op=WRITE, dst=/sensor/temp, src=…, VALUE u32 }.fwd-await-timeout—FWD{ op=AWAIT, dst=/sensor/temp, src=…, await_timeout=1e9 }.fwd-write-subscriber-field—FWD{ op=WRITE, dst=/sensor/temp, FIELD :subscribers[], src=…, SUBSCRIBER{...} }.fwd-routed-mount-residual(authored asfwd-routed-multihop; renamed by #419) —FWD{ op=READ, dst=/net/board/can0/ow/sensor, src=/reply-ep }: one strip-K mount (net/board/can0) plus the residual it forwards.fwd-routed-two-mount—FWD{ op=READ, dst=/net/uplink/b/net/uplink/c/sensor/temp, src=/reply-ep }: adstcrossing two mounts, three nodes deep (#419).fwd-src-accumulated— the same op mid-route, after two hops:FWD{ op=READ, dst=/net/uplink/d/sensor/temp, src=/net/downlink/a/net/downlink/cli/reply-ep }.dstshrinks andsrcgrows by wholenet/<module>/<name>mount runs — two hops means sixsrcsegments, not two (the prepend invariant, in its S2a form). Bytes rewritten 2026-08-02 by maintainer ruling on #419; the frame as authored was pre-S2a and did not compose.fwd-reply-result/fwd-reply-error— the reply as the terminus emits it, the request’s routes swapped:FWD{ op=REPLY, dst=/net/downlink/a/net/downlink/cli/reply-ep, src=/sensor/temp, kind=RESULT, VALUE }and the same routes withkind=ERROR, STATUS{ERROR{...}}. Thedstis the accumulated return route in its S2a form — wholenet/<module>/<name>mount runs, so it routes home through the same strip-K descent a request does. Bytes rewritten 2026-08-02 by maintainer ruling (c) on #419; the frames as authored carrieddst=/via_board/via_net/reply-ep, pre-S2a and non-composing.field-indexed,field-nested,field-append— the threeFIELDindex-modes.fwd-wildcard-reject—[*]outside subscriber-path ⇒INVALID_PATH.
Considered options¶
An RPC verb + correlation-id envelope (the #123 framing). Rejected — directly contradicts ADR-0006 (“no connect/subscribe; radical minimalism”) and ADR-0026 (“a subscription is a write”).
FWDcarries the three native primitives (not a new verb layer) and no end-to-end correlation-id, so it is the API on the wire, not RPC.Carry the target in
ROUTER(extend0x0Dwith a forward-path child). Rejected — pulls dedup/MAX_HOPSinto a source-routed request that is loop-free by construction; conflates the request direction with the delivery direction. One overloaded envelope is worse than two purpose-built frames.Fold the
:fieldchain intoPATH(extend0x06). Rejected — breaksPATH’s “children MUST be NAME” invariant and touches every existingPATH/SUBSCRIBER/ROUTERparser. A separateFIELDkeeps the existing 3-core machine unperturbed for one extra type code.Reduce
read/awaitto transient-subscribe writes (soFWDis write-only, noop). Rejected — over-clever: a trivial read would have to mint a reply-target and one-shot flags. An explicitopof the three blessed primitives is simpler and still ADR-0006-clean.A bare
[PATH][payload]transport convention (no new type). Rejected — not self-describing, so invisible to the conformance/cross-core machine that has caught every regression; each transport would re-implement the rule.A separate global destination name / address layer. Rejected — the path already encodes the route and the segment ids; a second naming layer is redundant (the load-bearing insight of this RFC).
Consequences¶
The remote surface is tiny: two structured TLVs (
FWD,FIELD) + the path-as-route rule.read/write/await/subscribe/delivery all reduce to “route a payload+op to a path, hop-by-hop”;subscribe/QoS are field-writes; replies retrace the link.Unblocks the TS client higher ops (#56), browser↔robot (ADR-0031), the reconciler (#58), and remote
:children[](#82/#83).ROUTERand all existing TLVs are unchanged; the 3-core conformance machine extends by adding vectors, not by reshaping anything.Each transport spec must define its own request/reply matching (the one cost of keeping
FWDpure) —transport_wsandtransport_canreference docs gain a “remote-op multiplexing” section.PATHparsers are untouched —:fieldlives only in the newFIELDselector.
Resolved during design (see §B/§D/§E)¶
Reply framing → a thin
FWD{ op=REPLY, kind∈{RESULT,ERROR} }routed back via the accumulatedsrc. Not a bare TLV (we need the route) and not a rich reply-kind (deliveries areWRITEs, not replies).Return route → accumulated in the frame (zero-copy
srcprepend), not per-hop connection state — so forwarders are stateless and replies survive a hop reboot.Unidirectional transport → handled: the reply self-routes via
src; it does not need the inbound link to be bidirectional.Streaming vs one-shot → two planes; streaming never per-sample-remote-writes (producer local-produce + flush + fan-out); a delivery is a
FWD WRITE.Streaming route overhead → the route-handle (§E.1): a per-link, advertise-driven, framing-mode-gated label (header-elision generalized) amortizes the return route on established high-rate subs, keeping forwarders stateless for the one-shot/cold case. Drops the 1 kHz example from ~16× to ~1.5× overhead.
Open questions (for the comment window)¶
await_timeoutcap — resolved by Amendment 3: the deadline is the requester’s, the terminus is not required to enforce one, so no default or cap is normative.Forward-right delegation — does an intermediate hop forward under the original
origin_peer_id(end-to-end identity) or re-originate as itself at each hop? (Affects §F’s first ACL check; leaning end-to-end identity preserved, each hop authorizes by it — notesrcalready records the per-hop forwarder chain.)srcexposure / privacy — the accumulated return route reveals the topology to the destination (and the full source route to the consumer — usually desirable as provenance). Is a redacted/opaque-segment mode ever needed for an untrusted intermediate, or is per-hop ACL sufficient?Stream-tag interop — each transport defines its own request↔reply matching tag; do we want a recommended (non-normative) tag shape so independent transport implementations converge?
Relates¶
ADR-0006 — read/write/await, no connect/subscribe (the verb set
FWDcarries).ADR-0026 — consumer-as-client / subscription is a write.
ADR-0027 — transports/connections are vertices (the mount points).
ADR-0031 — browser↔robot (the consumer of this).
ADR-0034 — the TS client SDK whose higher ops this unblocks.
RFC-0003 — concrete-path delivery (the delivery-side companion).
reference/03 (addressing grammar), reference/04 (flows + mount), reference/05 (TLV registry).
Erratum (2026-07-31): §B required ERROR=INVALID_PATH for a revisiting dst, which no forwarder can emit¶
Follows #445 (with #420 / #444). This was the last normative site; reference/05 was corrected by #490 and ADR-0040 §Context by #721.
A full-tree sweep taken with this erratum found the phantom in three more descriptive places
that #444’s and #721’s sweeps had both missed — CONTEXT.md §loop-freedom
(the canonical glossary, where it self-contradicted the same sentence’s “no dedup state, hop
counter, or depth cap exists anywhere”), reference/README’s index
row for 07-host-embedding, and ADR-0040’s Decision item 4, which the #721 erratum did not reach.
All three are corrected in the same change. The count is the point: a claim repeated in six
places took four passes to retire, because each pass swept the documents it expected to find it in.
What the text said. §B’s last bullet read:
FWDis loop-free by construction (the forwarddstis explicit; adstrevisiting a node is malformed →ERROR=INVALID_PATH). It carries noorigin/hop_count/dedup …
What the behaviour is. fwd_router_t (core/src/fwd_router.cpp) has no visited-set, no hop
counter, and no INVALID_PATH emission of any kind. Every hop does exactly one thing to dst:
resolve its first segment, strip it, re-emit. Loop-freedom is the monotonic shrink, not a check.
Which change made them diverge — none, and that is the point. The two were never together.
The clause was not implementable on the forwarder this same RFC mandates: recognising that a
dst revisits a node requires remembering the nodes already visited, and the very same sentence
forbids carrying that state. §A’s stateless source-router and §B’s revisit ERROR contradicted
each other on the day the RFC was accepted. ERROR=INVALID_PATH here is a requirement no
conforming implementation has ever satisfied, or could.
Why this is an erratum and not an amendment. Per GOVERNANCE.md — “if applying it would change what a conforming implementation does, it is not an erratum.” Applying this changes nothing: no implementation emits the error, so no peer has ever observed it, so no byte on any wire moves. The wire surface is untouched and no MUST that was ever satisfiable is altered. The normative surface is not moving; it is being described accurately for the first time.
Scope. The other INVALID_PATH clauses in this RFC are real, different, and untouched —
§C’s rejection of [*] outside a subscriber-path context (:131) and its fwd-wildcard-reject
vector (:205) are implemented, in core/src/op_resolve_walk.hpp.
Amendment (2026-08-01): §B multi-peer next-hop resolution — a bus link’s NAME is not a routable next-hop¶
RFC-0020 (recording
ADR-0073 §3,
tracked in #741) amends §B’s
forward-resolution step for multi-peer (bus) links: an inbound FWD whose next hop resolves
to a bus link’s own connection NAME with a residual dst below it that names no current peer
MUST be rejected (kind=ERROR, STATUS{ ERROR{ tr::path::invalid (0x0021) } } along the
accumulated src), never emitted over the bus’s shared fan-out endpoint. Only the link’s peer
names route below its mount; a dst naming the mount exactly still addresses the connection
vertex locally. Normative text, rationale, and the two rejected alternatives live in RFC-0020.
Amendment 2 (2026-08-22): §B/§D — a zero-length src is “no reply requested”¶
Records the maintainer ruling on #1491, tracked and specified in #1502. The 14-day comment window is waived by default while the project is solo-maintained (GOVERNANCE.md §Roles); it was not invoked, because the change removes frames rather than adding a surface for a second implementer to build against. (The 2026-08-01 amendment above is Amendment 1 — it predates the numbering.)
Status: accepted. This is an amendment and not an erratum: it changes what a conforming implementation puts on the wire.
The measurement that forced the question¶
A bulk producer streaming into a node over WebSocket (#1491, on an ESP32-C6) solved to ~4.86 ms fixed per batch + ~1.57 µs/byte. At a 2 KB payload the fixed term is 60 % of the total, and throughput peaked at pipeline depth 4 and degraded beyond it — depth 4 being the async TX pool depth of the host’s WebSocket transport, i.e. the queue the replies travel through. The writes were not what saturated. The reporter had already ruled out buffer copies (1.57 µs/byte is 252 cycles/byte, against 1–2 for a word-wise rv32 copy), TCP window (a 4× lwIP window lifted raw HTTP 51 % and moved this path within noise), and batch size (already at the ring’s capacity, worth 2.8× and spent).
The rule¶
A reply is something the origin requests by wiring a return route, never something the
library imposes. The src PATH of a FWD is simultaneously the return route and the
acknowledgement request, so the empty route is the request not made:
§B gains: a request
FWDwhosesrcis a zero-lengthPATHcarries no return route, and the terminus MUST NOT emit any frame in response to it.
Empty, never omitted. The encoding is the present-but-zero-length child. §B’s child run is read positionally, and a
WRITEpayload may itself bePATH-typed (RFC-0024 §7.1 amendment 2 gave that shape back deliberately), so an omittedsrcand aPATHpayload are the same bytes. Omission is ambiguous; emptiness is not. The grammar is unchanged —srcremains a requiredPATHchild — so every shipped parser reads the frame it always read.Empty-
srcWRITE: applied, and silent. The terminus applies the write and emits no frame. A denied or failed empty-srcwrite is dropped silently — no addressedERROR, because there is no address. This invents no drop policy: it is the one a deniedCOMPACTdelivery already runs under (reference/05 §route-handle, recording the #974 ruling — “a denied delivery is dropped like any other unwritable one”). The anti-enumeration property survives for free: a peer that asked for no answer learns nothing about what it may not write.Empty-
srcREAD/AWAIT: malformed. The result has nowhere to go, so the frame asks for an answer and refuses to receive one. Dropped at the terminus, and deliberately not NACKed — for the reason the whole clause exists: there is no route to carry a NACK. Forwarders stay opcode-agnostic; the check sits whereopis already switched, which is the terminus.Empty-
src+ the mint flag (opbit 7): malformed. A bound-path mint answer rides the reply alone (RFC-0024 §7.5), so the pairing is a contradiction on one frame. Dropped at the terminus rather than downgraded to a plain silent write: silently ignoring the flag would leave the origin waiting for a handle it will never be told it cannot have.Empty-
srcremoteSUBSCRIBE(a:subscribers[]WRITE): malformed — derived from the clauses above rather than ruled separately, and stated because they would otherwise leave it open. A subscribe is a standing request for future frames: the edge it binds would carry the empty route as itstarget, and every delivery down it would be the unroutableFWD{WRITE, dst=<empty>}this amendment exists to stop, emitted forever instead of once. Same terminus drop as clause 3.Deliveries stop drawing garbage replies — with no emitter change anywhere. A full-route fan-out delivery is already
FWD{WRITE, dst=<return route>, src=<empty PATH>}(§E; the reference core has emitted exactly that since #136). Before this amendment its receiver assembled aFWD{REPLY}addressed to an empty path — a frame no hop can route, sent for every sample of every stream. Clause 2 fixes that receiver-side, and the emitter is normatively unchanged.
What the origin gives up, stated¶
An empty-src flow has no per-write backpressure feedback: the BACKPRESSURE reply that
tells an origin to slow down is a reply, and there is none. That is inherent to an
unacknowledged channel and it is opt-in — the origin chose it by sending the empty route.
The application’s own sequence counter is its loss detector, which is the mechanism #1491’s
reporter already carries.
Scope boundary: forwarders are untouched, so the marker is not multi-hop¶
§B’s return-route accumulation is unchanged: every forwarding hop still prepends its
inbound-link NAME to src, including when src arrives empty. An empty src therefore
means “no reply requested” only where it is still empty when it reaches the terminus —
which is the two topologies the ruling is about:
a directly attached origin (a WebSocket/TCP client whose frame terminates at the node it is attached to, with zero forwarding hops) — #1491’s own case; and
the delivery leg, where the producer’s terminus emits the empty
srcitself.
An origin behind one or more forwarders cannot express the marker today: the first hop
grows src and the terminus sees a route. Making the hop preserve an empty src was
evaluated and deliberately not taken here — an empty seed src is currently the ordinary
spelling for “name me by the link I arrive on”, relied on by the shipped routing examples, so
preserving it would silently unaddress every one of them. Extending the marker across hops is
therefore a separate proposal that must re-specify the empty seed first, and is not decided by
this amendment.
§D’s tension, stated rather than papered over¶
§D says streaming never per-sample-remote-writes, and that governs the fan-out / consume
direction: delivery stays subscription-driven and flush-driven, and this amendment does not
touch it. The empty-src WRITE sanctions the push-ingest topology — where the producer
is the client and consumer-initiated subscription would invert who initiates. The standing
plane (a SUBSCRIBER plus delivery_compact, §D/§E.1 and
ADR-0030)
remains the documented first answer wherever the topology allows it; the unacknowledged
write is the answer where it does not.
Conformance vectors¶
Six, added to the S-set (core/tests/empty_src_unacked_test.cpp), each asserted as zero
frames emitted rather than as “no reply parsed” — a frame sent to a zero-length route decodes
as a well-formed FWD{REPLY}, so counting sends is the only assertion that catches it. Each is
paired with a positive control carrying a one-segment src, which must still draw exactly one
reply.
vector |
expected |
|
|---|---|---|
a |
empty- |
applied; zero frames |
b |
empty- |
not applied; zero frames ( |
c |
empty- |
dropped; zero frames |
d |
empty- |
dropped; zero frames |
e |
empty- |
dropped; zero frames |
f |
subscription full-route delivery at the consumer |
applied; no reply frame |
Plus the derived clause-5 case (empty-src subscribe: dropped, and no edge bound) and the two
remaining terminus reply sites — an unresolvable dst and a payload-less WRITE — which are
different arms from the ones a–f reach and would otherwise stay free to answer an empty route.
Acceptance still open¶
The on-silicon half of #1491 — a re-run of the reporter’s depth/payload sweep on an ESP32-C6
with the client sending empty-src writes, expecting a large drop in the ~4.86 ms fixed
per-batch term and a shift of the depth-4 knee — is not a merge gate and is not done:
no C6 was attached when this landed. It is recorded as a pending HIL item on #1502 alongside
#1479.
Amendment 3 (2026-10-03): §B — the requester owns an AWAIT’s deadline¶
Records the maintainer ruling of 2026-10-03, recorded with its rationale in ADR-0084. The 14-day comment window is waived by default while the project is solo-maintained (GOVERNANCE.md §Roles), and it was not invoked.
Status: accepted. This is an amendment and not an erratum: it changes what
await_timeout means and what a conforming terminus must send. No wire byte changes: the
child keeps its position, type and encoding, and every shipped frame still parses.
What changes¶
§B gave await_timeout the meaning “the terminus waits this long, then answers
ERROR(TIMEOUT)”, with a 1 s default when the child is absent. That made the terminus hold the
request for the requester’s chosen time. In the reference implementation it held the receive
context of the link the request arrived on, so every later frame on that link waited too.
§B is amended: a remote
AWAIT’s deadline belongs to the requester.await_timeout, when present, is the requester’s own deadline. A terminus MAY use it as a hint and is not required to enforce it. A terminus MUST NOT hold the receive context of the link the request arrived on while theAWAITis pending. It answerskind=RESULTwith the vertex’s value on the next change, orkind=ERRORat once when the request is refused. It MAY end a pendingAWAITwithout a reply when the link goes down or the node tears down.
The requester needs its own deadline anyway. A lost reply is already silence (reference/18 §failure modes): there is no correlation id and no per-hop request state (§D). So every requester must end its own wait. A terminus-side timeout only duplicated that, and the terminus paid for it.
ERROR(TIMEOUT)is no longer a promised answer. A requester that relied on it gets silence instead, and ends the wait at its own deadline, which is the same outcome.Reply order. A terminus that answers later lets the replies to later requests on the same link arrive first, which §D already allows. Replies name no request op, so a requester SHOULD NOT have an
AWAITand aREADoutstanding to the same vertex on one link: the two RESULTs are indistinguishable bysrcsuffix.
Conformance vectors¶
fwd-await-timeout keeps its bytes. Its description now reads the await_timeout child as the
requester’s deadline hint, not as a terminus-enforced timeout.