docs: record shared memory formats and addressing

2026-07-04 19:58:47 +02:00
parent a098962eec
commit 9546ad6c7b
1 changed files with 629 additions and 0 deletions
--- a/prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md
+++ b/prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md
@@ -0,0 +1,629 @@
+## Shared Memory Formats And Addressing
+
+Date: 2026-07-04  
+Host: `DOLPHIN`  
+Scope: all shared-memory formats and SHM-adjacent IPC formats directly inspected from source during UV / BLUE-PRIME / DITAv2 work.
+
+This document is intentionally concrete. It separates:
+
+1. the **transport container** (`Zinc` region, or `iceoryx2` service),
+2. the **payload framing inside that container**,
+3. the **semantic payload schema** written by a given subsystem,
+4. the **addressing rule** by which a writer and a reader find the same object.
+
+It also records which format is actually in use for each known subsystem as inspected on this host.
+
+### Short Answer
+
+- A **Zinc region is addressed by its logical region name**, passed to `SharedRegion.create(name, ...)` / `SharedRegion.open(name)`.
+- On Linux, Zinc materializes that as a POSIX SHM object named **`/zinc_<logical_name>`**, which appears in `/dev/shm` as **`zinc_<logical_name>`**.
+- Therefore:
+  - reader/writer open **`uv_shadow_state`**
+  - the OS object visible under `/dev/shm` is **`zinc_uv_shadow_state`**
+  - opening `zinc_uv_shadow_state` through the Zinc API is wrong and fails
+- Two algo instances avoid collision by using **different logical prefixes** and deriving all region names from that prefix.
+
+For `iceoryx2`, the address is not a Zinc region name. It is the **service name** such as `uv/pulse`.
+
+---
+
+## 1. What Exists
+
+### 1.1 Formats actually encountered
+
+| Layer | Transport | Address form | Payload framing | Current role |
+|---|---|---|---|---|
+| Zinc region container | Zinc shared memory | logical name `name`; OS object `/dev/shm/zinc_<name>` | Zinc internal region header, then user data area | base transport for Zinc-backed regions |
+| DITAv2 plane packet | Zinc region data area | `<prefix>_intent`, `<prefix>_state`, `<prefix>_control`, `<prefix>_venue` | `!QQ` = `(seq, json_size)` + UTF-8 JSON | DITAv2 real Zinc plane |
+| DITAv2 control packet | Zinc region data area | `<prefix>_control` | same `!QQ + JSON` envelope | DITAv2 control plane |
+| UV / BLUE-PRIME snapshot | Zinc region data area | `uv_shadow_state` | `UVZINC01` + dual-seq seqlock header + UTF-8 JSON | authoritative BLUE-PRIME shadow snapshot, live |
+| UV hook frame | Zinc region data area | `uv_shadow_blue_prime_hooks` | same `UVZINC01` + dual-seq + UTF-8 JSON | hook observability frame, live/auxiliary |
+| UV test scratch snapshot | Zinc region data area | `uv_t6_<id>_state` | same `UVZINC01` + dual-seq + UTF-8 JSON | test / temporary namespaces |
+| UV pulse bridge payload | `iceoryx2` publish-subscribe service | service name `uv/pulse` | fixed 40-byte `PulseFrame` POD | optional derived feed for Rust TUI; not authoritative |
+
+### 1.2 Not shared memory, but easy to confuse with it
+
+The old file transport in `prod/clean_arch/violet/uv/shm.py`:
+
+- explicit mode only
+- writes JSON files under `UV_SHM_ROOT` / `/dev/shm/uv`
+- **not** the runtime BLUE-PRIME contract
+- retained for tests / fallback diagnostics only
+
+That path is not documented further here because the request is specifically about shared memory.
+
+---
+
+## 2. Zinc Region Container Format
+
+Authoritative sources:
+
+- `zinc/core/src/header.rs`
+- `zinc/core/src/region.rs`
+- `zinc/core/src/platform/unix.rs`
+- `zinc/adapters/python/zinc/_ffi.py`
+- `zinc/adapters/python/zinc/__init__.py`
+
+### 2.1 Addressing
+
+Zinc validates and opens a **logical name** such as:
+
+- `uv_shadow_state`
+- `vst_dita_state`
+- `dolphin_violet_control`
+
+The Linux backend then maps that to a POSIX SHM object:
+
+- logical name: `uv_shadow_state`
+- OS shm object: `/zinc_uv_shadow_state`
+- visible filesystem entry: `/dev/shm/zinc_uv_shadow_state`
+
+This mapping is implemented in `zinc/core/src/platform/unix.rs`.
+
+### 2.2 Name rules
+
+Valid logical characters are:
+
+- ASCII alphanumeric
+- `_`
+- `-`
+
+Slashes are not allowed in raw Zinc names. Callers that start from path-like prefixes sanitize before opening.
+
+### 2.3 Region layout
+
+Every Zinc region is:
+
+1. one page of Zinc-owned metadata/header
+2. followed by the user data area
+
+The Rust core exposes the user data area by returning `page_size()` bytes past the mapping base. The Python adapter likewise exposes only the user data area through `SharedRegion.as_buffer()`.
+
+This point matters:
+
+- the **underlying region really is Zinc**
+- but a reader using `as_buffer()` does **not** see the Zinc header at byte 0
+- it sees byte 0 of the **user payload**
+
+That is why a live UV reader sees `UVZINC01` at the first visible bytes even though the mapped OS object is a Zinc region.
+
+### 2.4 Zinc internal header
+
+From `zinc/core/src/header.rs`:
+
+- `magic: u64` = `"ZINC_REG"`
+- `version: u16` = currently `2`
+- `flags: u16`
+- `notify_seq: AtomicU32`
+- `capacity: u64`
+- `ref_count: AtomicU32`
+- `owner_pid: AtomicI32`
+- `created_at: u64`
+- `name_hash: u64`
+- `ring_head: AtomicU64`
+- `ring_tail: AtomicU64`
+
+This header is aligned to one cache line and occupies the Zinc-managed metadata area, not the caller-visible payload buffer.
+
+### 2.5 Notify/wait semantics
+
+Zinc provides:
+
+- `notify()`
+- `wait(timeout_ms)`
+
+These operate on `notify_seq` in the Zinc header. The payload framing on top of Zinc is owned by the higher-level subsystem.
+
+---
+
+## 3. DITAv2 Real Zinc Plane Format
+
+Authoritative sources:
+
+- `prod/clean_arch/dita_v2/real_zinc_plane.py`
+- `prod/clean_arch/dita_v2/real_control_plane.py`
+
+### 3.1 Addressing
+
+`RealZincPlane(prefix=...)` derives names as:
+
+- `base = prefix.strip("/").replace("/", "_")`
+- `intent_name = f"{base}_intent"`
+- `state_name = f"{base}_state"`
+- `control_name = f"{base}_control"`
+- `venue_name = f"{base}_venue"`
+
+So if `prefix="vst/dita"`:
+
+- logical names become:
+  - `vst_dita_intent`
+  - `vst_dita_state`
+  - `vst_dita_control`
+  - `vst_dita_venue`
+- OS objects become:
+  - `/dev/shm/zinc_vst_dita_intent`
+  - `/dev/shm/zinc_vst_dita_state`
+  - `/dev/shm/zinc_vst_dita_control`
+  - `/dev/shm/zinc_vst_dita_venue`
+
+### 3.2 Region capacities
+
+Defaults in `RealZincPlane.__init__`:
+
+- `intent_capacity = 1 << 20` = 1 MiB
+- `state_capacity = 1 << 20` = 1 MiB
+- `control_capacity = 1 << 20` = 1 MiB
+- `venue_region` also uses `control_capacity`
+
+### 3.3 Payload framing inside the Zinc data area
+
+DITAv2 does **not** use the `UVZINC01` header.
+
+It uses:
+
+- `struct.pack("!QQ", seq, len(json_bytes))`
+- followed by UTF-8 JSON bytes
+
+That is:
+
+- bytes `0..8`: `seq` as big-endian `u64`
+- bytes `8..16`: JSON length as big-endian `u64`
+- bytes `16..16+size`: JSON bytes
+
+There is no dual-seq torn-read guard here. Readers trust:
+
+- the header is present
+- `size` is sane
+- the JSON decodes
+
+### 3.4 Semantic payloads
+
+By region:
+
+- `*_intent`: `{"items": [...]}` where items are serialized `KernelIntent`s
+- `*_state`: `{"slots": [...]}` where slots are serialized `TradeSlot`s
+- `*_control`: `{"control": {...}}` where value is a `KernelControlSnapshot`
+- `*_venue`: `{"venue": {...}}` where value is a `VenueTelemetrySnapshot`
+
+### 3.5 Write semantics
+
+Writers:
+
+- increment a region-local sequence
+- build `!QQ + JSON`
+- copy packet into the entire visible data buffer
+- zero the tail
+- call `region.notify()`
+
+This is a simple packet format, not a seqlock format.
+
+### 3.6 Current use
+
+This is the intended real shared-memory format for DITAv2 when `RealZincPlane` / `RealZincControlPlane` are active.
+
+At the time of this writing, I did **not** find live openable regions under the tested names:
+
+- `vst_dita_intent`
+- `vst_dita_state`
+- `vst_dita_control`
+- `vst_dita_venue`
+- `dolphin_violet_intent`
+- `dolphin_violet_state`
+- `dolphin_violet_control`
+- `dolphin_violet_venue`
+
+So this format is source-authoritative, but not directly observed live under those tested prefixes at sample time.
+
+---
+
+## 4. UV / BLUE-PRIME Zinc Snapshot Format
+
+Authoritative source:
+
+- `prod/clean_arch/violet/uv/shm.py`
+
+This is the important one for current UV / BLUE-PRIME observability.
+
+### 4.1 Addressing
+
+Default prefix:
+
+- `UV_ZINC_PREFIX`, default `uv_shadow`
+
+Logical region naming rule:
+
+- `prefix.strip("/").replace("/", "_")`
+- plus `_<slot>`
+- except slot `"blue_prime"` and slot `"state"` are both normalized to `_<prefix>_state`
+
+Examples:
+
+- `publish("blue_prime", ...)` -> logical region `uv_shadow_state`
+- `publish("state", ...)` -> logical region `uv_shadow_state`
+- `publish("blue_prime_hooks", ...)` -> logical region `uv_shadow_blue_prime_hooks`
+
+### 4.2 Region capacities
+
+Default:
+
+- `UV_ZINC_STATE_BYTES`, default `64 << 20` = 64 MiB
+
+Observed live regions:
+
+- `/dev/shm/zinc_uv_shadow_state`
+- `/dev/shm/zinc_uv_shadow_blue_prime_hooks`
+
+Observed test/scratch leftovers:
+
+- `/dev/shm/zinc_uv_t6_8933e28c84_state`
+- `/dev/shm/zinc_uv_t6_a1f7e3254b_state`
+- `/dev/shm/zinc_uv_t6_b16924a96a_state`
+- `/dev/shm/zinc_uv_t6_eed1be7947_state`
+
+### 4.3 Payload framing inside the Zinc data area
+
+Header:
+
+- magic = `b"UVZINC01"`
+- struct = `struct.Struct("!8sQQQ")`
+
+Visible data-area layout:
+
+1. bytes `0..8`: magic `"UVZINC01"`
+2. bytes `8..16`: `seq_a` big-endian `u64`
+3. bytes `16..24`: `seq_b` big-endian `u64`
+4. bytes `24..32`: JSON payload size big-endian `u64`
+5. bytes `32..32+size`: UTF-8 JSON payload
+
+### 4.4 Write semantics
+
+Writer uses a simple seqlock pattern:
+
+1. compute next logical sequence `seq`
+2. derive:
+   - `seq_even = seq * 2`
+   - `seq_odd = seq_even - 1`
+3. write header with odd/in-flight sequence and `size = 0`
+4. copy JSON body
+5. optionally zero one byte after body
+6. write header again with even/stable sequence and real `size`
+7. call `region.notify()`
+
+### 4.5 Read semantics
+
+Reader accepts payload only if all are true:
+
+- magic == `UVZINC01`
+- `seq_a != 0`
+- `seq_a` is even
+- `seq_a == seq_b`
+- `size` is in bounds
+- after copying the body, rereading the header yields the exact same
+  - magic
+  - `seq_a`
+  - `seq_b`
+  - `size`
+
+If any of that fails, the read is treated as torn / not yet initialized.
+
+### 4.6 Semantic payloads
+
+This format carries JSON snapshots rather than a fixed struct. Current known uses:
+
+- `uv_shadow_state`
+  - authoritative BLUE-PRIME snapshot
+  - includes domains such as `meta`, `scan`, `live_inputs`, `engine`, `decision`, `efsm`, `dita`, `ram`, `source_trace`
+- `uv_shadow_blue_prime_hooks`
+  - one published hook runner frame per scan
+  - keys include `scan`, `hooks`, `hook_count`, `ok_count`, `total_us`
+
+### 4.7 Current use
+
+This is the **live authoritative BLUE-PRIME shared-memory format** currently observed on host.
+
+Verification made on host:
+
+- `SharedRegion.open("uv_shadow_state")` succeeds
+- `SharedRegion.open("zinc_uv_shadow_state")` fails
+- first bytes of visible region buffer are `55565a494e433031...` = `UVZINC01`
+
+That is the definitive proof that:
+
+- the region container is Zinc
+- the payload framing currently in use inside the container is `UVZINC01`
+
+---
+
+## 5. UV Hook Frame Region
+
+Authoritative source:
+
+- `prod/clean_arch/violet/uv/hooks/runner.py`
+
+The hook runner publishes:
+
+- `self.shm.publish("blue_prime_hooks", frame)`
+
+Because `ShmChannel` defaults to Zinc-backed transport, that becomes:
+
+- logical region: `uv_shadow_blue_prime_hooks`
+- OS shm object: `/dev/shm/zinc_uv_shadow_blue_prime_hooks`
+
+The payload framing is the same `UVZINC01` seqlock JSON envelope documented above.
+
+The semantic payload differs:
+
+- per-scan hook effects
+- timing / ordering / success counts
+
+This is auxiliary observability, not the main state snapshot.
+
+---
+
+## 6. iceoryx2 Pulse Bridge Format
+
+Authoritative sources:
+
+- `uv_tui/crates/pulse_frame/src/lib.rs`
+- `uv_tui/bridge/src/lib.rs`
+- `uv_tui/tui/src/lib.rs`
+
+### 6.1 What it is
+
+This is not the authoritative shared-memory snapshot. It is a **derived bridge feed**:
+
+1. bridge opens Zinc region `uv_shadow_state`
+2. bridge decodes the `UVZINC01` payload
+3. bridge extracts a reduced pulse contract
+4. bridge republishes that reduced contract over `iceoryx2` service `uv/pulse`
+
+### 6.2 Addressing
+
+Address is a service name, not a Zinc region name:
+
+- service: `uv/pulse`
+
+Current code hardcodes a singleton default service. Unlike Zinc prefixes, this is **not yet namespaced per parallel algo instance**.
+
+### 6.3 Payload framing
+
+Fixed `PulseFrame`, length 40 bytes:
+
+- `observe_mono_ns: u64`
+- `scan_number: u64`
+- `region_seq: u64`
+- `publish_latency_us: f64`
+- `has_entry: u8`
+- trailing reserved padding
+
+Serialized little-endian by the shared `pulse_frame` crate.
+
+### 6.4 Current use
+
+- optional
+- derived
+- non-authoritative
+- useful for the Rust TUI, especially when decoupling bridge and display
+
+As of current Rust TUI work, the TUI can also read the Zinc region directly and no longer requires the bridge.
+
+---
+
+## 7. Addressing Rules, Precisely
+
+### 7.1 Zinc regions
+
+There are **three names** to keep distinct:
+
+1. **logical region name**
+   - what code passes to Zinc
+   - example: `uv_shadow_state`
+2. **POSIX SHM object name**
+   - what the Linux SHM API sees
+   - example: `/zinc_uv_shadow_state`
+3. **filesystem entry under `/dev/shm`**
+   - example: `/dev/shm/zinc_uv_shadow_state`
+
+The reader/writer contract uses **(1)**.
+
+The operator inspecting `/dev/shm` sees **(3)**.
+
+Confusing (1) and (3) is the common failure mode.
+
+### 7.2 How a writer and reader agree on the same region
+
+They must share:
+
+- the same transport family
+  - Zinc or `iceoryx2`
+- the same prefix / service namespace
+- the same slot derivation rule
+- the same payload framing
+- the same semantic schema
+
+For Zinc-backed UV:
+
+- prefix source: `UV_ZINC_PREFIX`
+- slot naming logic: `_slot_region(prefix, slot)`
+- payload format: `UVZINC01`
+- state slot: `blue_prime` -> `uv_shadow_state`
+
+For DITAv2:
+
+- prefix passed to `RealZincPlane(prefix=...)`
+- suffixes: `_intent`, `_state`, `_control`, `_venue`
+- payload format: `!QQ + JSON`
+
+For `iceoryx2`:
+
+- service name must match exactly
+- payload struct must match exactly
+
+### 7.3 How two running algo instances know “their” region
+
+They do not discover “theirs” by magic. They must be started with a namespace choice.
+
+Correct pattern:
+
+- instance A:
+  - `UV_ZINC_PREFIX=uv_shadow_a`
+  - state region -> `uv_shadow_a_state`
+- instance B:
+  - `UV_ZINC_PREFIX=uv_shadow_b`
+  - state region -> `uv_shadow_b_state`
+
+Then:
+
+- A writer writes only `uv_shadow_a_*`
+- A TUI for A opens only `uv_shadow_a_*`
+- B writer/TUI use `uv_shadow_b_*`
+
+For test runs, this pattern is already used:
+
+- `uv_t6_<id>` prefixes produce regions like `uv_t6_8933e28c84_state`
+
+### 7.4 Collision rule
+
+If two independent algo instances reuse the same Zinc logical region name:
+
+- they are not isolated
+- last writer wins
+- readers observe a mixed stream
+- observability becomes invalid
+
+So region naming is not cosmetic. It is the namespace boundary.
+
+---
+
+## 8. What Is In Use For What, Now
+
+### 8.1 BLUE-PRIME authoritative observability
+
+- transport container: Zinc region
+- logical region: `uv_shadow_state`
+- payload framing inside region: `UVZINC01`
+- semantic schema: BLUE-PRIME snapshot JSON
+- status: live and observed
+
+### 8.2 BLUE-PRIME hook observability
+
+- transport container: Zinc region
+- logical region: `uv_shadow_blue_prime_hooks`
+- payload framing inside region: `UVZINC01`
+- semantic schema: hook runner frame JSON
+- status: live region observed
+
+### 8.3 UV Rust TUI direct mode
+
+- reads: Zinc region directly
+- logical region by default: `uv_shadow_state`
+- expects payload framing: `UVZINC01`
+- status: implemented and verified
+
+### 8.4 UV Rust TUI bridge mode
+
+- bridge reads Zinc region `uv_shadow_state`
+- bridge republishes `PulseFrame` over `iceoryx2` service `uv/pulse`
+- TUI subscribes to `uv/pulse`
+- status: implemented and verified, but secondary to direct Zinc reads
+
+### 8.5 DITAv2 real shared-memory plane
+
+- transport container: Zinc region
+- logical regions: `<prefix>_{intent,state,control,venue}`
+- payload framing: `!QQ + JSON`
+- status: source-authoritative, but not directly observed live under tested prefixes during this inspection
+
+---
+
+## 9. Operator Notes
+
+### 9.1 To inspect the live authoritative UV region
+
+Use the logical name:
+
+```python
+from zinc import SharedRegion
+r = SharedRegion.open("uv_shadow_state")
+buf = r.as_buffer()
+```
+
+Do **not** do:
+
+```python
+SharedRegion.open("zinc_uv_shadow_state")
+```
+
+That is the `/dev/shm` object name, not the logical Zinc name.
+
+### 9.2 To tell whether a region uses DITAv2 or UV framing
+
+Look at the first visible bytes of `as_buffer()`:
+
+- `UVZINC01` -> UV / BLUE-PRIME seqlock payload
+- otherwise, if the first 16 bytes parse as `!QQ` and the JSON decodes, likely DITAv2 packet framing
+
+You will **not** see `ZINC_REG` there through normal adapter reads, because that Zinc header lives before the exposed data area.
+
+### 9.3 To run two independent observability stacks
+
+Assign different prefixes up front. Example:
+
+```bash
+export UV_ZINC_PREFIX=uv_shadow_main
+export UV_ZINC_PREFIX=uv_shadow_soak
+```
+
+Then point each consumer at its matching logical region names.
+
+If `iceoryx2` bridge mode is used in parallel too, it needs the same kind of namespacing. Current bridge service default is singleton `uv/pulse`; that should be parameterized if simultaneous parallel bridges are required.
+
+---
+
+## 10. Bottom Line
+
+The current live BLUE-PRIME observability stack is:
+
+- **container:** Zinc shared memory
+- **live authoritative region:** `uv_shadow_state`
+- **live payload framing:** `UVZINC01` seqlock JSON
+- **live hook side region:** `uv_shadow_blue_prime_hooks`
+
+DITAv2’s real Zinc plane is a different format:
+
+- same Zinc container idea
+- different region family
+- different payload envelope: `!QQ + JSON`
+
+The Rust `PulseFrame` is yet another layer:
+
+- `iceoryx2` service payload
+- derived from the authoritative Zinc region
+- not itself the source of truth
+
+The rule for addressing is simple and must be followed strictly:
+
+- **open logical Zinc names through Zinc**
+- **inspect `/dev/shm/zinc_*` only as an operator artifact**
+- **namespace parallel instances by prefix**
+- **match the payload framing to the subsystem**