From 9546ad6c7b0fabeb221f971678bd3df2420fe87d Mon Sep 17 00:00:00 2001 From: Codex Date: Sat, 4 Jul 2026 19:58:47 +0200 Subject: [PATCH] docs: record shared memory formats and addressing --- ..._MEMORY_FORMATS_AND_ADDRESSING_20260704.md | 629 ++++++++++++++++++ 1 file changed, 629 insertions(+) create mode 100644 prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md diff --git a/prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md b/prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md new file mode 100644 index 00000000..05223c6c --- /dev/null +++ b/prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md @@ -0,0 +1,629 @@ +## Shared Memory Formats And Addressing + +Date: 2026-07-04 +Host: `DOLPHIN` +Scope: all shared-memory formats and SHM-adjacent IPC formats directly inspected from source during UV / BLUE-PRIME / DITAv2 work. + +This document is intentionally concrete. It separates: + +1. the **transport container** (`Zinc` region, or `iceoryx2` service), +2. the **payload framing inside that container**, +3. the **semantic payload schema** written by a given subsystem, +4. the **addressing rule** by which a writer and a reader find the same object. + +It also records which format is actually in use for each known subsystem as inspected on this host. + +### Short Answer + +- A **Zinc region is addressed by its logical region name**, passed to `SharedRegion.create(name, ...)` / `SharedRegion.open(name)`. +- On Linux, Zinc materializes that as a POSIX SHM object named **`/zinc_`**, which appears in `/dev/shm` as **`zinc_`**. +- Therefore: + - reader/writer open **`uv_shadow_state`** + - the OS object visible under `/dev/shm` is **`zinc_uv_shadow_state`** + - opening `zinc_uv_shadow_state` through the Zinc API is wrong and fails +- Two algo instances avoid collision by using **different logical prefixes** and deriving all region names from that prefix. + +For `iceoryx2`, the address is not a Zinc region name. It is the **service name** such as `uv/pulse`. + +--- + +## 1. What Exists + +### 1.1 Formats actually encountered + +| Layer | Transport | Address form | Payload framing | Current role | +|---|---|---|---|---| +| Zinc region container | Zinc shared memory | logical name `name`; OS object `/dev/shm/zinc_` | Zinc internal region header, then user data area | base transport for Zinc-backed regions | +| DITAv2 plane packet | Zinc region data area | `_intent`, `_state`, `_control`, `_venue` | `!QQ` = `(seq, json_size)` + UTF-8 JSON | DITAv2 real Zinc plane | +| DITAv2 control packet | Zinc region data area | `_control` | same `!QQ + JSON` envelope | DITAv2 control plane | +| UV / BLUE-PRIME snapshot | Zinc region data area | `uv_shadow_state` | `UVZINC01` + dual-seq seqlock header + UTF-8 JSON | authoritative BLUE-PRIME shadow snapshot, live | +| UV hook frame | Zinc region data area | `uv_shadow_blue_prime_hooks` | same `UVZINC01` + dual-seq + UTF-8 JSON | hook observability frame, live/auxiliary | +| UV test scratch snapshot | Zinc region data area | `uv_t6__state` | same `UVZINC01` + dual-seq + UTF-8 JSON | test / temporary namespaces | +| UV pulse bridge payload | `iceoryx2` publish-subscribe service | service name `uv/pulse` | fixed 40-byte `PulseFrame` POD | optional derived feed for Rust TUI; not authoritative | + +### 1.2 Not shared memory, but easy to confuse with it + +The old file transport in `prod/clean_arch/violet/uv/shm.py`: + +- explicit mode only +- writes JSON files under `UV_SHM_ROOT` / `/dev/shm/uv` +- **not** the runtime BLUE-PRIME contract +- retained for tests / fallback diagnostics only + +That path is not documented further here because the request is specifically about shared memory. + +--- + +## 2. Zinc Region Container Format + +Authoritative sources: + +- `zinc/core/src/header.rs` +- `zinc/core/src/region.rs` +- `zinc/core/src/platform/unix.rs` +- `zinc/adapters/python/zinc/_ffi.py` +- `zinc/adapters/python/zinc/__init__.py` + +### 2.1 Addressing + +Zinc validates and opens a **logical name** such as: + +- `uv_shadow_state` +- `vst_dita_state` +- `dolphin_violet_control` + +The Linux backend then maps that to a POSIX SHM object: + +- logical name: `uv_shadow_state` +- OS shm object: `/zinc_uv_shadow_state` +- visible filesystem entry: `/dev/shm/zinc_uv_shadow_state` + +This mapping is implemented in `zinc/core/src/platform/unix.rs`. + +### 2.2 Name rules + +Valid logical characters are: + +- ASCII alphanumeric +- `_` +- `-` + +Slashes are not allowed in raw Zinc names. Callers that start from path-like prefixes sanitize before opening. + +### 2.3 Region layout + +Every Zinc region is: + +1. one page of Zinc-owned metadata/header +2. followed by the user data area + +The Rust core exposes the user data area by returning `page_size()` bytes past the mapping base. The Python adapter likewise exposes only the user data area through `SharedRegion.as_buffer()`. + +This point matters: + +- the **underlying region really is Zinc** +- but a reader using `as_buffer()` does **not** see the Zinc header at byte 0 +- it sees byte 0 of the **user payload** + +That is why a live UV reader sees `UVZINC01` at the first visible bytes even though the mapped OS object is a Zinc region. + +### 2.4 Zinc internal header + +From `zinc/core/src/header.rs`: + +- `magic: u64` = `"ZINC_REG"` +- `version: u16` = currently `2` +- `flags: u16` +- `notify_seq: AtomicU32` +- `capacity: u64` +- `ref_count: AtomicU32` +- `owner_pid: AtomicI32` +- `created_at: u64` +- `name_hash: u64` +- `ring_head: AtomicU64` +- `ring_tail: AtomicU64` + +This header is aligned to one cache line and occupies the Zinc-managed metadata area, not the caller-visible payload buffer. + +### 2.5 Notify/wait semantics + +Zinc provides: + +- `notify()` +- `wait(timeout_ms)` + +These operate on `notify_seq` in the Zinc header. The payload framing on top of Zinc is owned by the higher-level subsystem. + +--- + +## 3. DITAv2 Real Zinc Plane Format + +Authoritative sources: + +- `prod/clean_arch/dita_v2/real_zinc_plane.py` +- `prod/clean_arch/dita_v2/real_control_plane.py` + +### 3.1 Addressing + +`RealZincPlane(prefix=...)` derives names as: + +- `base = prefix.strip("/").replace("/", "_")` +- `intent_name = f"{base}_intent"` +- `state_name = f"{base}_state"` +- `control_name = f"{base}_control"` +- `venue_name = f"{base}_venue"` + +So if `prefix="vst/dita"`: + +- logical names become: + - `vst_dita_intent` + - `vst_dita_state` + - `vst_dita_control` + - `vst_dita_venue` +- OS objects become: + - `/dev/shm/zinc_vst_dita_intent` + - `/dev/shm/zinc_vst_dita_state` + - `/dev/shm/zinc_vst_dita_control` + - `/dev/shm/zinc_vst_dita_venue` + +### 3.2 Region capacities + +Defaults in `RealZincPlane.__init__`: + +- `intent_capacity = 1 << 20` = 1 MiB +- `state_capacity = 1 << 20` = 1 MiB +- `control_capacity = 1 << 20` = 1 MiB +- `venue_region` also uses `control_capacity` + +### 3.3 Payload framing inside the Zinc data area + +DITAv2 does **not** use the `UVZINC01` header. + +It uses: + +- `struct.pack("!QQ", seq, len(json_bytes))` +- followed by UTF-8 JSON bytes + +That is: + +- bytes `0..8`: `seq` as big-endian `u64` +- bytes `8..16`: JSON length as big-endian `u64` +- bytes `16..16+size`: JSON bytes + +There is no dual-seq torn-read guard here. Readers trust: + +- the header is present +- `size` is sane +- the JSON decodes + +### 3.4 Semantic payloads + +By region: + +- `*_intent`: `{"items": [...]}` where items are serialized `KernelIntent`s +- `*_state`: `{"slots": [...]}` where slots are serialized `TradeSlot`s +- `*_control`: `{"control": {...}}` where value is a `KernelControlSnapshot` +- `*_venue`: `{"venue": {...}}` where value is a `VenueTelemetrySnapshot` + +### 3.5 Write semantics + +Writers: + +- increment a region-local sequence +- build `!QQ + JSON` +- copy packet into the entire visible data buffer +- zero the tail +- call `region.notify()` + +This is a simple packet format, not a seqlock format. + +### 3.6 Current use + +This is the intended real shared-memory format for DITAv2 when `RealZincPlane` / `RealZincControlPlane` are active. + +At the time of this writing, I did **not** find live openable regions under the tested names: + +- `vst_dita_intent` +- `vst_dita_state` +- `vst_dita_control` +- `vst_dita_venue` +- `dolphin_violet_intent` +- `dolphin_violet_state` +- `dolphin_violet_control` +- `dolphin_violet_venue` + +So this format is source-authoritative, but not directly observed live under those tested prefixes at sample time. + +--- + +## 4. UV / BLUE-PRIME Zinc Snapshot Format + +Authoritative source: + +- `prod/clean_arch/violet/uv/shm.py` + +This is the important one for current UV / BLUE-PRIME observability. + +### 4.1 Addressing + +Default prefix: + +- `UV_ZINC_PREFIX`, default `uv_shadow` + +Logical region naming rule: + +- `prefix.strip("/").replace("/", "_")` +- plus `_` +- except slot `"blue_prime"` and slot `"state"` are both normalized to `__state` + +Examples: + +- `publish("blue_prime", ...)` -> logical region `uv_shadow_state` +- `publish("state", ...)` -> logical region `uv_shadow_state` +- `publish("blue_prime_hooks", ...)` -> logical region `uv_shadow_blue_prime_hooks` + +### 4.2 Region capacities + +Default: + +- `UV_ZINC_STATE_BYTES`, default `64 << 20` = 64 MiB + +Observed live regions: + +- `/dev/shm/zinc_uv_shadow_state` +- `/dev/shm/zinc_uv_shadow_blue_prime_hooks` + +Observed test/scratch leftovers: + +- `/dev/shm/zinc_uv_t6_8933e28c84_state` +- `/dev/shm/zinc_uv_t6_a1f7e3254b_state` +- `/dev/shm/zinc_uv_t6_b16924a96a_state` +- `/dev/shm/zinc_uv_t6_eed1be7947_state` + +### 4.3 Payload framing inside the Zinc data area + +Header: + +- magic = `b"UVZINC01"` +- struct = `struct.Struct("!8sQQQ")` + +Visible data-area layout: + +1. bytes `0..8`: magic `"UVZINC01"` +2. bytes `8..16`: `seq_a` big-endian `u64` +3. bytes `16..24`: `seq_b` big-endian `u64` +4. bytes `24..32`: JSON payload size big-endian `u64` +5. bytes `32..32+size`: UTF-8 JSON payload + +### 4.4 Write semantics + +Writer uses a simple seqlock pattern: + +1. compute next logical sequence `seq` +2. derive: + - `seq_even = seq * 2` + - `seq_odd = seq_even - 1` +3. write header with odd/in-flight sequence and `size = 0` +4. copy JSON body +5. optionally zero one byte after body +6. write header again with even/stable sequence and real `size` +7. call `region.notify()` + +### 4.5 Read semantics + +Reader accepts payload only if all are true: + +- magic == `UVZINC01` +- `seq_a != 0` +- `seq_a` is even +- `seq_a == seq_b` +- `size` is in bounds +- after copying the body, rereading the header yields the exact same + - magic + - `seq_a` + - `seq_b` + - `size` + +If any of that fails, the read is treated as torn / not yet initialized. + +### 4.6 Semantic payloads + +This format carries JSON snapshots rather than a fixed struct. Current known uses: + +- `uv_shadow_state` + - authoritative BLUE-PRIME snapshot + - includes domains such as `meta`, `scan`, `live_inputs`, `engine`, `decision`, `efsm`, `dita`, `ram`, `source_trace` +- `uv_shadow_blue_prime_hooks` + - one published hook runner frame per scan + - keys include `scan`, `hooks`, `hook_count`, `ok_count`, `total_us` + +### 4.7 Current use + +This is the **live authoritative BLUE-PRIME shared-memory format** currently observed on host. + +Verification made on host: + +- `SharedRegion.open("uv_shadow_state")` succeeds +- `SharedRegion.open("zinc_uv_shadow_state")` fails +- first bytes of visible region buffer are `55565a494e433031...` = `UVZINC01` + +That is the definitive proof that: + +- the region container is Zinc +- the payload framing currently in use inside the container is `UVZINC01` + +--- + +## 5. UV Hook Frame Region + +Authoritative source: + +- `prod/clean_arch/violet/uv/hooks/runner.py` + +The hook runner publishes: + +- `self.shm.publish("blue_prime_hooks", frame)` + +Because `ShmChannel` defaults to Zinc-backed transport, that becomes: + +- logical region: `uv_shadow_blue_prime_hooks` +- OS shm object: `/dev/shm/zinc_uv_shadow_blue_prime_hooks` + +The payload framing is the same `UVZINC01` seqlock JSON envelope documented above. + +The semantic payload differs: + +- per-scan hook effects +- timing / ordering / success counts + +This is auxiliary observability, not the main state snapshot. + +--- + +## 6. iceoryx2 Pulse Bridge Format + +Authoritative sources: + +- `uv_tui/crates/pulse_frame/src/lib.rs` +- `uv_tui/bridge/src/lib.rs` +- `uv_tui/tui/src/lib.rs` + +### 6.1 What it is + +This is not the authoritative shared-memory snapshot. It is a **derived bridge feed**: + +1. bridge opens Zinc region `uv_shadow_state` +2. bridge decodes the `UVZINC01` payload +3. bridge extracts a reduced pulse contract +4. bridge republishes that reduced contract over `iceoryx2` service `uv/pulse` + +### 6.2 Addressing + +Address is a service name, not a Zinc region name: + +- service: `uv/pulse` + +Current code hardcodes a singleton default service. Unlike Zinc prefixes, this is **not yet namespaced per parallel algo instance**. + +### 6.3 Payload framing + +Fixed `PulseFrame`, length 40 bytes: + +- `observe_mono_ns: u64` +- `scan_number: u64` +- `region_seq: u64` +- `publish_latency_us: f64` +- `has_entry: u8` +- trailing reserved padding + +Serialized little-endian by the shared `pulse_frame` crate. + +### 6.4 Current use + +- optional +- derived +- non-authoritative +- useful for the Rust TUI, especially when decoupling bridge and display + +As of current Rust TUI work, the TUI can also read the Zinc region directly and no longer requires the bridge. + +--- + +## 7. Addressing Rules, Precisely + +### 7.1 Zinc regions + +There are **three names** to keep distinct: + +1. **logical region name** + - what code passes to Zinc + - example: `uv_shadow_state` +2. **POSIX SHM object name** + - what the Linux SHM API sees + - example: `/zinc_uv_shadow_state` +3. **filesystem entry under `/dev/shm`** + - example: `/dev/shm/zinc_uv_shadow_state` + +The reader/writer contract uses **(1)**. + +The operator inspecting `/dev/shm` sees **(3)**. + +Confusing (1) and (3) is the common failure mode. + +### 7.2 How a writer and reader agree on the same region + +They must share: + +- the same transport family + - Zinc or `iceoryx2` +- the same prefix / service namespace +- the same slot derivation rule +- the same payload framing +- the same semantic schema + +For Zinc-backed UV: + +- prefix source: `UV_ZINC_PREFIX` +- slot naming logic: `_slot_region(prefix, slot)` +- payload format: `UVZINC01` +- state slot: `blue_prime` -> `uv_shadow_state` + +For DITAv2: + +- prefix passed to `RealZincPlane(prefix=...)` +- suffixes: `_intent`, `_state`, `_control`, `_venue` +- payload format: `!QQ + JSON` + +For `iceoryx2`: + +- service name must match exactly +- payload struct must match exactly + +### 7.3 How two running algo instances know “their” region + +They do not discover “theirs” by magic. They must be started with a namespace choice. + +Correct pattern: + +- instance A: + - `UV_ZINC_PREFIX=uv_shadow_a` + - state region -> `uv_shadow_a_state` +- instance B: + - `UV_ZINC_PREFIX=uv_shadow_b` + - state region -> `uv_shadow_b_state` + +Then: + +- A writer writes only `uv_shadow_a_*` +- A TUI for A opens only `uv_shadow_a_*` +- B writer/TUI use `uv_shadow_b_*` + +For test runs, this pattern is already used: + +- `uv_t6_` prefixes produce regions like `uv_t6_8933e28c84_state` + +### 7.4 Collision rule + +If two independent algo instances reuse the same Zinc logical region name: + +- they are not isolated +- last writer wins +- readers observe a mixed stream +- observability becomes invalid + +So region naming is not cosmetic. It is the namespace boundary. + +--- + +## 8. What Is In Use For What, Now + +### 8.1 BLUE-PRIME authoritative observability + +- transport container: Zinc region +- logical region: `uv_shadow_state` +- payload framing inside region: `UVZINC01` +- semantic schema: BLUE-PRIME snapshot JSON +- status: live and observed + +### 8.2 BLUE-PRIME hook observability + +- transport container: Zinc region +- logical region: `uv_shadow_blue_prime_hooks` +- payload framing inside region: `UVZINC01` +- semantic schema: hook runner frame JSON +- status: live region observed + +### 8.3 UV Rust TUI direct mode + +- reads: Zinc region directly +- logical region by default: `uv_shadow_state` +- expects payload framing: `UVZINC01` +- status: implemented and verified + +### 8.4 UV Rust TUI bridge mode + +- bridge reads Zinc region `uv_shadow_state` +- bridge republishes `PulseFrame` over `iceoryx2` service `uv/pulse` +- TUI subscribes to `uv/pulse` +- status: implemented and verified, but secondary to direct Zinc reads + +### 8.5 DITAv2 real shared-memory plane + +- transport container: Zinc region +- logical regions: `_{intent,state,control,venue}` +- payload framing: `!QQ + JSON` +- status: source-authoritative, but not directly observed live under tested prefixes during this inspection + +--- + +## 9. Operator Notes + +### 9.1 To inspect the live authoritative UV region + +Use the logical name: + +```python +from zinc import SharedRegion +r = SharedRegion.open("uv_shadow_state") +buf = r.as_buffer() +``` + +Do **not** do: + +```python +SharedRegion.open("zinc_uv_shadow_state") +``` + +That is the `/dev/shm` object name, not the logical Zinc name. + +### 9.2 To tell whether a region uses DITAv2 or UV framing + +Look at the first visible bytes of `as_buffer()`: + +- `UVZINC01` -> UV / BLUE-PRIME seqlock payload +- otherwise, if the first 16 bytes parse as `!QQ` and the JSON decodes, likely DITAv2 packet framing + +You will **not** see `ZINC_REG` there through normal adapter reads, because that Zinc header lives before the exposed data area. + +### 9.3 To run two independent observability stacks + +Assign different prefixes up front. Example: + +```bash +export UV_ZINC_PREFIX=uv_shadow_main +export UV_ZINC_PREFIX=uv_shadow_soak +``` + +Then point each consumer at its matching logical region names. + +If `iceoryx2` bridge mode is used in parallel too, it needs the same kind of namespacing. Current bridge service default is singleton `uv/pulse`; that should be parameterized if simultaneous parallel bridges are required. + +--- + +## 10. Bottom Line + +The current live BLUE-PRIME observability stack is: + +- **container:** Zinc shared memory +- **live authoritative region:** `uv_shadow_state` +- **live payload framing:** `UVZINC01` seqlock JSON +- **live hook side region:** `uv_shadow_blue_prime_hooks` + +DITAv2’s real Zinc plane is a different format: + +- same Zinc container idea +- different region family +- different payload envelope: `!QQ + JSON` + +The Rust `PulseFrame` is yet another layer: + +- `iceoryx2` service payload +- derived from the authoritative Zinc region +- not itself the source of truth + +The rule for addressing is simple and must be followed strictly: + +- **open logical Zinc names through Zinc** +- **inspect `/dev/shm/zinc_*` only as an operator artifact** +- **namespace parallel instances by prefix** +- **match the payload framing to the subsystem**