Reference 00 — Core overview¶

This suite is the libtracer standard, at draft status for v1. It describes what any conforming implementation must do, independently of how the reference implementation does it. Design rationale is recorded in the architecture decision records and in git history.

What this document is¶

libtracer is a wire and addressing protocol for a decentralized graph of endpoints. Hosts publish and subscribe to paths; the underlying transport is whatever is loaded as a module (TCP, UDP, CAN, I²C, SHM, RDMA — all opt-in). The protocol is language-agnostic: the reference implementation is in C++23, but any C++17, Rust, Zig, or Go implementation that honors this spec interoperates byte-for-byte.

The remaining sections of this reference suite specify byte format, graph semantics, addressing, communication flows, protocol-defined TLVs, user-data packing, and how a host’s local view embeds into the global network.

Conformance¶

A conforming implementation SHALL:

Encode and decode TLV frames per 01-data-format.md.
Honor the path syntax of 03-addressing.md and the subtree-subscription semantics of RFC-0005 — subtree subscriptions (every subscription observes its vertex and all descendants).
Implement the read / write / await primitives and the field-write control surface of 04-communication-flows.md.
Reserve type codes per 05-protocol-tlvs.md and treat unknown codes safely.
Maintain the same-substrate invariant of 02-graph-model.md: any sequence of mix/split/concat operations on the in-memory view tree, followed by serialization, MUST produce the same wire bytes as the equivalent fresh-buffer construction.
Forward FWD frames hop-by-hop per 07-host-embedding.md: strip the whole leading dst mount run — net/<module>/<name>[/<peer>] (RFC-0014 S2a) — and prepend the inbound link’s own mount run to src when acting as a forwarder. Loop-freedom is by construction — dst shrinks monotonically, so a route is finite; there is no revisit check.
Provide path handles per ../spec/v1.md §3.1: encode each address used more than once into a PATH TLV exactly once (build-time literal or init-time registration); reuse the encoded bytes on every read / write / await; do not require string parsing or allocation on the hot path.

A conforming node MAY load any subset of transport, discovery, security, and executor modules. A node loading zero transport modules conforms — it is an in-process-only graph. A node loading multiple transport modules SHALL implement the FWD forwarding logic of 07-host-embedding.md.

A conforming TLV SHALL fit in the header layout of 01-data-format.md, use a type code from a defined range, and pass CRC-32C verification when the CR bit is set.

Conformance profiles¶

(Distinct from the architectural layers below — these are build-size profiles, not protocol layers. Named with a P prefix to avoid collision.)

Profile	Required	Typical use
P0 — in-process	required modules only, zero transports	unit tests, in-process pub/sub, the API substrate other layers compose against
P1 — single-transport leaf	required + 1 transport	RC car over UART, sensor over CAN, ESP32 over Wi-Fi
P2 — forwarder	required + ≥2 transports	gateway between buses (CAN ↔ IP), edge router
P3 — full	P2 + discovery + executor + security	production deployment

Higher profiles are strict supersets. A conformance test suite exercises P0 mandatorily; P1+ are exercised against the available transport modules.

The six load-bearing claims¶

Any conforming implementation must honor these. They are what distinguishes libtracer from existing protocols (see the repository README for the comparison).

A TLV in memory IS a graph node IS the wire bytes. No separate serialization layer. The in-memory representation is a tree of refcounted views over real backing memory. Mix/split/concat at the graph level rearranges views without touching bytes. Serialization is a walk of the view tree. (02-graph-model.md, 06-user-data-packing.md)
The API is read/write only. Three calls — read, write, await — plus refcount management. Subscriptions, QoS, ACLs, liveness — every control surface — are writable fields on endpoints, addressed via the : separator. There is no connect / disconnect / subscribe primitive. (04-communication-flows.md)
No fragmentation rules in the wire format. Logically large messages are addressed across child endpoints (ep[0..N]) with a shared timestamp. Each slice is an independently-routable TLV. The wire format never carries reassembly metadata. (03-addressing.md, 06-user-data-packing.md)
Forwarding is core. Any host with two transport modules loaded is a forwarder — a stateless FWD hop between links. The “one address space across CAN + IP + RDMA” claim is structural, not opt-in. From a subscriber’s API view, transport choice is invisible. (07-host-embedding.md)

Statelessness, qualified. “Stateless” describes the bare forward hop, which is the default and the only thing a cold or one-shot flow ever touches. It is already narrower than it reads: the route-handle plane of RFC-0004 §E.1 puts per-link label state on the hops a compact delivery flow crosses (05-protocol-tlvs.md §Route-handle frames). RFC-0027 (accepted 2026-08-15, implemented) narrows it further — a hop that mints a path label learns per-flow state for the flows it minted for — and both mechanisms degrade to the bare stateless hop when the state is absent. Path-label minting is opt-in and off by default: a node with no injected label table mints nothing and stays the bare hop, which is the shape every deployment ships until it chooses otherwise (05-protocol-tlvs.md §0x06 §path label element).
The graph imposes no shape on user data. An endpoint is a name attached to a memory view. The protocol makes no claim about what that memory contains, how it’s chunked, or how endpoints are arranged. A single boolean fits in one byte; a 10-GB camera frame fits across ep[0..N] slices; a memory-mapped GPIO register fits as a one-byte view backed by mmio_base + offset. (06-user-data-packing.md)
Paths are encoded once, used many times. A vertex address is encoded into a PATH TLV at build time (a .rodata literal) or at node init (one allocation), and reused thereafter. The hot-path API takes a path handle, not a string — no snprintf, no parser walk, no allocation per write. This is what makes a 16 KB Cortex-M0 a first-class libtracer node and what lets a publisher write from inside an ISR. (../spec/v1.md §3.1, 03-addressing.md, 05-protocol-tlvs.md)

The six-layer model¶

The protocol stack has six layers of concern, numbered bottom-up from memory at L0 to application semantics at L5. Concepts in this reference suite belong to exactly one layer; conflating them produces design confusion.

Layer	Concern	Specified in
L0 — Memory substrate	Real buffers, MMIO, queues, pools, peripheral FIFOs; allocation, lifetime, cache, DMA	09-memory-substrate.md
L1 — Views and ownership	Memory views, refcounted segments, ropes (view chains), the TLV-as-cast	08-views-and-ownership.md
L2 — Frame envelope	Slice the byte stream into framed units; verify integrity; carry wire-time	01-data-format.md
L3 — TLV semantics	Interpret the type code; recurse into structured (PL=1) containers	05-protocol-tlvs.md
L4 — Graph endpoint logic	Vertices, edges, paths, subscriptions, fan-out, QoS, ACL, FWD forwarding	02-graph-model.md, 03-addressing.md, 04-communication-flows.md
L5 — Application semantics	What the bytes inside `VALUE` mean; control logic	application code

Each adjacent pair of layers communicates through a small contract:

L0 ↔ L1: backend interface (alloc, release, cache hooks). Bytes flow up as segments; lifetime flows down via destroy callbacks per backend.
L1 ↔ L2: views are cast to TLVs. The cast is zero-copy reinterpretation.
L2 ↔ L3: the type byte (carried at L2, meaningful at L3) and opt.PL (signal to recurse).
L3 ↔ L4: the protocol-defined TLV registry (SUBSCRIBER, PATH, POINT, FWD, …) and what each means at the graph layer.
L4 ↔ L5: the path / read / write / await API and the field-write control surface.

        flowchart TB
  L5["L5 — Application semantics<br/>(what bytes inside VALUE mean)"]
  L4["L4 — Graph endpoint logic<br/>vertices, edges, paths, fanout, QoS, ACL, FWD forwarding"]
  L3["L3 — TLV semantics<br/>type byte, structured (PL=1) recursion"]
  L2["L2 — Frame envelope<br/>header + payload + trailer; CRC; wire-time TS"]
  L1["L1 — Views and ownership<br/>refcounted view tree, rope, TLV-as-cast"]
  L0["L0 — Memory substrate<br/>heap, pool, MMIO, DMA, pbuf, skbuff, FIFO"]

  L5 -->|read/write/await + path handle| L4
  L4 -->|PATH, SUBSCRIBER, FWD...| L3
  L3 -->|type byte + opt.PL| L2
  L2 -->|view-cast| L1
  L1 -->|alloc/release/cache hooks| L0

  classDef applic fill:#fdf,stroke:#333
  classDef graphl fill:#dfd,stroke:#333
  classDef tlv    fill:#dff,stroke:#333
  classDef wire   fill:#ddf,stroke:#333
  classDef view   fill:#ffd,stroke:#333
  classDef mem    fill:#fdd,stroke:#333
  class L5 applic
  class L4 graphl
  class L3 tlv
  class L2 wire
  class L1 view
  class L0 mem

The wire format (L2) carries the type byte and opt.PL at fixed positions so routers can decide whether to recurse into nested children without parsing payload. The type byte’s meaning is L3; it sits in the L2 header for routing convenience. Priority is not an L2 concern — it is cached at L4 from the SUBSCRIPTION’s delivery policy (bits 2-4, RFC-0022 §3.A; it was a per-vertex :settings.priority knob until then, and nothing consumed it). The L2 opt byte’s other bits select wire-format variants (LL length width, CW CRC width, TF TS form) — these are framing choices, not semantic information.

Static path handles in the layer model¶

The path-handle mechanism (load-bearing claim 6) sits at the L4 ↔ L5 boundary: applications hold handles, the graph layer consumes them. Encoding a path into a PATH TLV is an L3 act; storing the result in .rodata or a long-lived L0 segment is an L0/L1 act; reusing the bytes on every write keeps L4 dispatch keyed on canonical PATH bytes.

        flowchart LR
  subgraph BuildOrInit["Once — build time or node init"]
    SRC["Path string<br/>&quot;/sensor/temp&quot;"]
    ENC["Encode &amp; validate<br/>(L3 PATH TLV layout)"]
    ROD[".rodata / heap segment<br/>(L0 backing memory)"]
    SRC --> ENC --> ROD
  end

  subgraph Hot["Hot path — once per write"]
    HND["L4 path handle<br/>(pointer into segment)"]
    API["L5 calls tracer_write(handle, value)"]
    DISP["L4 router dispatch<br/>(byte-keyed on PATH bytes)"]
    HND --> API --> DISP
  end

  ROD -.points to.-> HND

The diagram is the visual form of ../spec/v1.md §3.1: encoding is an init-phase concern, dispatch is hot-path-clean, and the bytes that flow through L4’s dispatch table are the same regardless of whether the handle was a build-time literal or a runtime registration.

TLV-at-rest = TLV-in-transit + trailer¶

The L2 frame has a header + payload + optional trailer structure. The payload region is byte-identical across every state of the TLV’s life:

At rest in the graph (stored at a vertex, on disk in a recorder file): header + payload.
In transit on a transport: header + payload + trailer (wire-time TS + CRC).
Re-emitted by a forwarder to another transport: header + payload + new_trailer.

The trailer is append-only at egress, strip-only at ingress. A forwarder or recorder moves a TLV between rest and transit by attaching/stripping the trailer; the payload bytes are never touched. This is what makes the same-substrate insight extend cleanly across multi-hop forwarding and recording. See 01-data-format.md §the trailer is append-only and 02-graph-model.md §the trailer enables payload-bytes invariance.

Graph-data vs in-flight-message: the FWD envelope¶

The TLV substrate plays two distinct roles, structurally distinguished by the FWD TLV (type 0x0F, RFC-0004 — remote operation addressing):

Graph data at a vertex: just the payload, no envelope. Identity = vertex path.
In-flight message crossing a link: an FWD TLV (structured, PL=1) carrying the op VALUE, the dst PATH (the remaining route — it shrinks by a whole mount run per hop), the src PATH (the accumulated return route — it grows by the same run), and the payload TLV as its last child. Identity = the explicit source route the frame carries.

The terminus sheds the FWD envelope: it applies the op locally, stores only the bare payload (trailer-less at rest), and replies with a fresh FWD{REPLY} routed by the accumulated src. Because every remote endpoint is addressed by its explicit source route, forwarding is loop-free by construction (dst shrinks monotonically per hop, so a route is finite — there is no revisit check) and needs no duplicate detection — 0x0D ROUTER is a reserved, decodable wire code with no implemented mechanism. See 04-communication-flows.md and 07-host-embedding.md.

Everything is a module¶

There is no “core” carved out from “modules”. A libtracer node is a chosen set of modules linked together. Some modules are required by every conforming node (frame codec, path resolver, refcount/view machinery, router/dispatcher, FWD forwarding logic) — those are tagged required in the catalog. The rest are tagged by what they bring (a transport, a discovery mechanism, a security wrap, an executor, a memory backend, an I/O view module). A bare-minimum node loads only the required modules; a feature-rich node loads many.

This framing matters because the so-called “core” itself is a bundle of modules separated by clean interfaces — frame codec is one module, path resolver another, dispatcher a third. They happen to be required for every conformance profile, but they are not architecturally privileged.

The full module catalog — everything across L0..L5 — is in 10-module-catalog.md, with a pairing table that says which L0 backends pair with which L1 view modules pair with which transports.

A node’s footprint is the sum of its loaded modules. Each profile therefore carries a footprint target, stated as a design target for the profile — not as a number a conforming implementation is required to hit:

Build	Target
P0, required modules only, stripped, `arm-none-eabi-g++ -std=c++23 -Os -fno-exceptions -fno-rtti`	≤ 16 KB
adding `transport_tcp`	~5 KB on Linux / ~8 KB on Cortex-M (lwIP-dependent)
a single-UART-transport node	under 25 KB
TCP + UDP + mDNS + CAN + TLS	30–50 KB

The reference implementation does not meet the P0 target. A Cortex-M0 footprint sentinel — the required modules plus a send-and-receive fixture, cross-compiled with the profile above, linked and stripped — measures 20,937 B, ~4.5 KiB over the 16 KiB bound. Measured against the std::pmr::memory_resource& form of the arena decoder’s memory seam, the same fixture came in ~0.9 KiB over, decomposed as ~2.7 KiB of soft-float pulled in by std::pmr through that seam, plus ~1.5 KiB of CRC lookup tables. The decoder takes a tr::mem::block_source_t& per ADR-0065 — failable allocation gets its own seam, the ~0.9 KiB figure has not been re-measured against it, and the 20,937 B overage is not attributed. The measured footprints of this implementation, the sentinel that produces them and the build axes that move them are described in the configuration space, which is the record of this implementation’s costs rather than part of the standard.

The module ABI itself is an implementation concern, not a protocol property — two implementations need not share a module ABI; they need to share the wire format, addressing scheme, and flows. See 10-module-catalog.md §module ABI for the reference implementation’s seams.

Implementation-language portability¶

The reference implementation is C++23 (chosen for std::expected, std::span, std::byte, designated initializers, constexpr, three-way comparison, [[nodiscard]], and <atomic>; built -fno-exceptions -fno-rtti for the constrained profile). The choice is pragmatic: a freestanding-friendly C++23 subset produces a small portable binary, has broad MCU toolchain coverage (GCC 13+, Clang 18+, ESP-IDF 5.3+ as a compiler floor, arm-none-eabi-g++ 14.x), and exposes a clean extern "C" FFI surface for higher-level wrappers. The wire format itself is language-neutral, so a C, Rust, or Zig node interoperates byte-for-byte.

The protocol itself is implementable in any language with:

Capability	Why needed	Substitute in C++ / Rust / Go
Atomic refcount with relaxed/acq_rel ordering	Buffer-view ownership	`std::atomic<uint32_t>` (C++); `Arc<T>` or `AtomicUsize` (Rust); `sync/atomic` (Go)
Pointer + length view over external memory	Zero-copy view tree	`std::span<const std::byte>` (C++20); `&[u8]` borrows or `Bytes` (Rust); `[]byte` slices (Go)
Packed struct with explicit endianness	TLV header layout	`#[repr(C, packed)]` (Rust); `#pragma pack` / `[[gnu::packed]]` (C++); `encoding/binary` (Go)
Iterative (non-recursive) parser, its nesting depth bounded by the receiver’s decode resources rather than by a constant	MCU stack safety	trivially portable
Fixed-width little-endian integer load/store (`u16` and `u32`)	TLV length encoding, selected by the header’s `LL` bit	trivially portable
CRC-32C, hardware-accelerated where present	integrity	x86 SSE 4.2, ARMv8 `+crc`, software fallback — present in any language

A portable C binding would be a thin extern "C" layer over the C++23 core — where this suite sketches C-shaped APIs, that is the shape being sketched (a potential binding surface, not the reference core itself). A pure-Rust port is straightforward — Bytes from the bytes crate maps directly to libtracer’s view + refcount; tokio or mio provides the run loop. A pure-Go port would lose the explicit refcount (Go has GC) but could use the same wire format and addressing.

The wire format and the addressing scheme — not the C ABI — are the standard. A future spec audit at the end of the v1 milestones is the gate to declaring this reference suite “frozen for v1,” at which point a second implementation in C++ or Rust becomes the conformance test.

Versioning¶

libtracer v1 is the wire format. It does not version per-frame. There is no version bit in opt. The wire format is a one-shot commitment: get it right, ship it, don’t bump.

Future incompatible changes — should they ever be needed — are versioned at the discovery layer: a different mDNS service name (_libtracer-v2._tcp vs _libtracer._tcp), a different default TCP port, a different CAN-ID prefix, etc. Peers learn each other’s wire-format identity at discovery time.

The forward-extension path within v1 is the type-code registry: new core type codes can be added in the unassigned reaches — 0x15–0x1F in the fast-track range and 0x20–0x7F in the long-term registry — without breaking existing receivers, who gracefully ignore unknown codes per 01-data-format.md §handling unknown type codes. (0x0E–0x14 are assigned: SPEC, FWD, FIELD, ADVERTISE, COMPACT, HANDLE_NACK, PATH_REF — see 05-protocol-tlvs.md §type-code registry.)

Reading order¶

For a first pass at understanding the protocol:

01-data-format.md — what bytes look like.
02-graph-model.md — what those bytes mean structurally, and the same-substrate insight.
03-addressing.md — how to name things.
04-communication-flows.md — how nodes talk.
05-protocol-tlvs.md — every reserved TLV, byte-precise.
06-user-data-packing.md — how the user puts their data into the graph (worked examples spanning 1 byte to 10 GB/s).
07-host-embedding.md — how a host’s local view fits into the network.

For a parser/sender writer in another language: read 01, 03, 05, 06 in that order; then 02 once you start optimizing for zero-copy.

For a forwarder implementer: 02, 03, 04, 07 are mandatory; 06 is illustrative of what you’ll be routing.

For a deployer sizing a node — someone who has the protocol and now has to decide how much RAM each stage gets and where the pipeline is allowed to queue or shed: 22-backpressure-and-sizing.md, with 09-memory-substrate.md and 12-deployment-profiles.md beside it.

Out-of-scope for this reference suite¶

The API/ABI of any specific implementation (header signatures, struct layouts beyond the packed wire header). See the implementation’s own headers — for the reference C++23 core, those land in ../../core/.
The module ABI (the shape of a transport or backend seam, and the version tag a loader would check). See 10-module-catalog.md §module ABI.
The configuration file format (TOML for forwarding/discovery). The catalog deliberately declines to fix it — see 10-module-catalog.md §boundaries of the catalog.
Build options and CMake toggles (defined with the core/ rebuild).
Cluster consensus, CRDTs, distributed transactions — explicit non-goals. See 04-communication-flows.md §coherency.