siloqy/prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md

## Shared Memory Formats And Addressing

Date: 2026-07-04  
Host: `DOLPHIN`  
Scope: all shared-memory formats and SHM-adjacent IPC formats directly inspected from source during UV / BLUE-PRIME / DITAv2 work.

This document is intentionally concrete. It separates:

1. the **transport container** (`Zinc` region, or `iceoryx2` service),
2. the **payload framing inside that container**,
3. the **semantic payload schema** written by a given subsystem,
4. the **addressing rule** by which a writer and a reader find the same object.

It also records which format is actually in use for each known subsystem as inspected on this host.

### Short Answer

- A **Zinc region is addressed by its logical region name**, passed to `SharedRegion.create(name, ...)` / `SharedRegion.open(name)`.
- On Linux, Zinc materializes that as a POSIX SHM object named **`/zinc_<logical_name>`**, which appears in `/dev/shm` as **`zinc_<logical_name>`**.
- Therefore:
  - reader/writer open **`uv_shadow_state`**
  - the OS object visible under `/dev/shm` is **`zinc_uv_shadow_state`**
  - opening `zinc_uv_shadow_state` through the Zinc API is wrong and fails
- Two algo instances avoid collision by using **different logical prefixes** and deriving all region names from that prefix.

For `iceoryx2`, the address is not a Zinc region name. It is the **service name** such as `uv/pulse`.

---

## 1. What Exists

### 1.1 Formats actually encountered

| Layer | Transport | Address form | Payload framing | Current role |
|---|---|---|---|---|
| Zinc region container | Zinc shared memory | logical name `name`; OS object `/dev/shm/zinc_<name>` | Zinc internal region header, then user data area | base transport for Zinc-backed regions |
| DITAv2 plane packet | Zinc region data area | `<prefix>_intent`, `<prefix>_state`, `<prefix>_control`, `<prefix>_venue` | `!QQ` = `(seq, json_size)` + UTF-8 JSON | DITAv2 real Zinc plane |
| DITAv2 control packet | Zinc region data area | `<prefix>_control` | same `!QQ + JSON` envelope | DITAv2 control plane |
| UV / BLUE-PRIME snapshot | Zinc region data area | `uv_shadow_state` | `UVZINC01` + dual-seq seqlock header + UTF-8 JSON | authoritative BLUE-PRIME shadow snapshot, live |
| UV hook frame | Zinc region data area | `uv_shadow_blue_prime_hooks` | same `UVZINC01` + dual-seq + UTF-8 JSON | hook observability frame, live/auxiliary |
| UV test scratch snapshot | Zinc region data area | `uv_t6_<id>_state` | same `UVZINC01` + dual-seq + UTF-8 JSON | test / temporary namespaces |
| UV pulse bridge payload | `iceoryx2` publish-subscribe service | service name `uv/pulse` | fixed 40-byte `PulseFrame` POD | optional derived feed for Rust TUI; not authoritative |

### 1.2 Not shared memory, but easy to confuse with it

The old file transport in `prod/clean_arch/violet/uv/shm.py`:

- explicit mode only
- writes JSON files under `UV_SHM_ROOT` / `/dev/shm/uv`
- **not** the runtime BLUE-PRIME contract
- retained for tests / fallback diagnostics only

That path is not documented further here because the request is specifically about shared memory.

---

## 2. Zinc Region Container Format

Authoritative sources:

- `zinc/core/src/header.rs`
- `zinc/core/src/region.rs`
- `zinc/core/src/platform/unix.rs`
- `zinc/adapters/python/zinc/_ffi.py`
- `zinc/adapters/python/zinc/__init__.py`

### 2.1 Addressing

Zinc validates and opens a **logical name** such as:

- `uv_shadow_state`
- `vst_dita_state`
- `dolphin_violet_control`

The Linux backend then maps that to a POSIX SHM object:

- logical name: `uv_shadow_state`
- OS shm object: `/zinc_uv_shadow_state`
- visible filesystem entry: `/dev/shm/zinc_uv_shadow_state`

This mapping is implemented in `zinc/core/src/platform/unix.rs`.

### 2.2 Name rules

Valid logical characters are:

- ASCII alphanumeric
- `_`
- `-`

Slashes are not allowed in raw Zinc names. Callers that start from path-like prefixes sanitize before opening.

### 2.3 Region layout

Every Zinc region is:

1. one page of Zinc-owned metadata/header
2. followed by the user data area

The Rust core exposes the user data area by returning `page_size()` bytes past the mapping base. The Python adapter likewise exposes only the user data area through `SharedRegion.as_buffer()`.

This point matters:

- the **underlying region really is Zinc**
- but a reader using `as_buffer()` does **not** see the Zinc header at byte 0
- it sees byte 0 of the **user payload**

That is why a live UV reader sees `UVZINC01` at the first visible bytes even though the mapped OS object is a Zinc region.

### 2.4 Zinc internal header

From `zinc/core/src/header.rs`:

- `magic: u64` = `"ZINC_REG"`
- `version: u16` = currently `2`
- `flags: u16`
- `notify_seq: AtomicU32`
- `capacity: u64`
- `ref_count: AtomicU32`
- `owner_pid: AtomicI32`
- `created_at: u64`
- `name_hash: u64`
- `ring_head: AtomicU64`
- `ring_tail: AtomicU64`

This header is aligned to one cache line and occupies the Zinc-managed metadata area, not the caller-visible payload buffer.

### 2.5 Notify/wait semantics

Zinc provides:

- `notify()`
- `wait(timeout_ms)`

These operate on `notify_seq` in the Zinc header. The payload framing on top of Zinc is owned by the higher-level subsystem.

---

## 3. DITAv2 Real Zinc Plane Format

Authoritative sources:

- `prod/clean_arch/dita_v2/real_zinc_plane.py`
- `prod/clean_arch/dita_v2/real_control_plane.py`

### 3.1 Addressing

`RealZincPlane(prefix=...)` derives names as:

- `base = prefix.strip("/").replace("/", "_")`
- `intent_name = f"{base}_intent"`
- `state_name = f"{base}_state"`
- `control_name = f"{base}_control"`
- `venue_name = f"{base}_venue"`

So if `prefix="vst/dita"`:

- logical names become:
  - `vst_dita_intent`
  - `vst_dita_state`
  - `vst_dita_control`
  - `vst_dita_venue`
- OS objects become:
  - `/dev/shm/zinc_vst_dita_intent`
  - `/dev/shm/zinc_vst_dita_state`
  - `/dev/shm/zinc_vst_dita_control`
  - `/dev/shm/zinc_vst_dita_venue`

### 3.2 Region capacities

Defaults in `RealZincPlane.__init__`:

- `intent_capacity = 1 << 20` = 1 MiB
- `state_capacity = 1 << 20` = 1 MiB
- `control_capacity = 1 << 20` = 1 MiB
- `venue_region` also uses `control_capacity`

### 3.3 Payload framing inside the Zinc data area

DITAv2 does **not** use the `UVZINC01` header.

It uses:

- `struct.pack("!QQ", seq, len(json_bytes))`
- followed by UTF-8 JSON bytes

That is:

- bytes `0..8`: `seq` as big-endian `u64`
- bytes `8..16`: JSON length as big-endian `u64`
- bytes `16..16+size`: JSON bytes

There is no dual-seq torn-read guard here. Readers trust:

- the header is present
- `size` is sane
- the JSON decodes

### 3.4 Semantic payloads

By region:

- `*_intent`: `{"items": [...]}` where items are serialized `KernelIntent`s
- `*_state`: `{"slots": [...]}` where slots are serialized `TradeSlot`s
- `*_control`: `{"control": {...}}` where value is a `KernelControlSnapshot`
- `*_venue`: `{"venue": {...}}` where value is a `VenueTelemetrySnapshot`

### 3.5 Write semantics

Writers:

- increment a region-local sequence
- build `!QQ + JSON`
- copy packet into the entire visible data buffer
- zero the tail
- call `region.notify()`

This is a simple packet format, not a seqlock format.

### 3.6 Current use

This is the intended real shared-memory format for DITAv2 when `RealZincPlane` / `RealZincControlPlane` are active.

At the time of this writing, I did **not** find live openable regions under the tested names:

- `vst_dita_intent`
- `vst_dita_state`
- `vst_dita_control`
- `vst_dita_venue`
- `dolphin_violet_intent`
- `dolphin_violet_state`
- `dolphin_violet_control`
- `dolphin_violet_venue`

So this format is source-authoritative, but not directly observed live under those tested prefixes at sample time.

---

## 4. UV / BLUE-PRIME Zinc Snapshot Format

Authoritative source:

- `prod/clean_arch/violet/uv/shm.py`

This is the important one for current UV / BLUE-PRIME observability.

### 4.1 Addressing

Default prefix:

- `UV_ZINC_PREFIX`, default `uv_shadow`

Logical region naming rule:

- `prefix.strip("/").replace("/", "_")`
- plus `_<slot>`
- except slot `"blue_prime"` and slot `"state"` are both normalized to `_<prefix>_state`

Examples:

- `publish("blue_prime", ...)` -> logical region `uv_shadow_state`
- `publish("state", ...)` -> logical region `uv_shadow_state`
- `publish("blue_prime_hooks", ...)` -> logical region `uv_shadow_blue_prime_hooks`

### 4.2 Region capacities

Default:

- `UV_ZINC_STATE_BYTES`, default `64 << 20` = 64 MiB

Observed live regions:

- `/dev/shm/zinc_uv_shadow_state`
- `/dev/shm/zinc_uv_shadow_blue_prime_hooks`

Observed test/scratch leftovers:

- `/dev/shm/zinc_uv_t6_8933e28c84_state`
- `/dev/shm/zinc_uv_t6_a1f7e3254b_state`
- `/dev/shm/zinc_uv_t6_b16924a96a_state`
- `/dev/shm/zinc_uv_t6_eed1be7947_state`

### 4.3 Payload framing inside the Zinc data area

Header:

- magic = `b"UVZINC01"`
- struct = `struct.Struct("!8sQQQ")`

Visible data-area layout:

1. bytes `0..8`: magic `"UVZINC01"`
2. bytes `8..16`: `seq_a` big-endian `u64`
3. bytes `16..24`: `seq_b` big-endian `u64`
4. bytes `24..32`: JSON payload size big-endian `u64`
5. bytes `32..32+size`: UTF-8 JSON payload

### 4.4 Write semantics

Writer uses a simple seqlock pattern:

1. compute next logical sequence `seq`
2. derive:
   - `seq_even = seq * 2`
   - `seq_odd = seq_even - 1`
3. write header with odd/in-flight sequence and `size = 0`
4. copy JSON body
5. optionally zero one byte after body
6. write header again with even/stable sequence and real `size`
7. call `region.notify()`

### 4.5 Read semantics

Reader accepts payload only if all are true:

- magic == `UVZINC01`
- `seq_a != 0`
- `seq_a` is even
- `seq_a == seq_b`
- `size` is in bounds
- after copying the body, rereading the header yields the exact same
  - magic
  - `seq_a`
  - `seq_b`
  - `size`

If any of that fails, the read is treated as torn / not yet initialized.

### 4.6 Semantic payloads

This format carries JSON snapshots rather than a fixed struct. Current known uses:

- `uv_shadow_state`
  - authoritative BLUE-PRIME snapshot
  - includes domains such as `meta`, `scan`, `live_inputs`, `engine`, `decision`, `efsm`, `dita`, `ram`, `source_trace`
- `uv_shadow_blue_prime_hooks`
  - one published hook runner frame per scan
  - keys include `scan`, `hooks`, `hook_count`, `ok_count`, `total_us`

### 4.7 Current use

This is the **live authoritative BLUE-PRIME shared-memory format** currently observed on host.

Verification made on host:

- `SharedRegion.open("uv_shadow_state")` succeeds
- `SharedRegion.open("zinc_uv_shadow_state")` fails
- first bytes of visible region buffer are `55565a494e433031...` = `UVZINC01`

That is the definitive proof that:

- the region container is Zinc
- the payload framing currently in use inside the container is `UVZINC01`

---

## 5. UV Hook Frame Region

Authoritative source:

- `prod/clean_arch/violet/uv/hooks/runner.py`

The hook runner publishes:

- `self.shm.publish("blue_prime_hooks", frame)`

Because `ShmChannel` defaults to Zinc-backed transport, that becomes:

- logical region: `uv_shadow_blue_prime_hooks`
- OS shm object: `/dev/shm/zinc_uv_shadow_blue_prime_hooks`

The payload framing is the same `UVZINC01` seqlock JSON envelope documented above.

The semantic payload differs:

- per-scan hook effects
- timing / ordering / success counts

This is auxiliary observability, not the main state snapshot.

---

## 6. iceoryx2 Pulse Bridge Format

Authoritative sources:

- `uv_tui/crates/pulse_frame/src/lib.rs`
- `uv_tui/bridge/src/lib.rs`
- `uv_tui/tui/src/lib.rs`

### 6.1 What it is

This is not the authoritative shared-memory snapshot. It is a **derived bridge feed**:

1. bridge opens Zinc region `uv_shadow_state`
2. bridge decodes the `UVZINC01` payload
3. bridge extracts a reduced pulse contract
4. bridge republishes that reduced contract over `iceoryx2` service `uv/pulse`

### 6.2 Addressing

Address is a service name, not a Zinc region name:

- service: `uv/pulse`

Current code hardcodes a singleton default service. Unlike Zinc prefixes, this is **not yet namespaced per parallel algo instance**.

### 6.3 Payload framing

Fixed `PulseFrame`, length 40 bytes:

- `observe_mono_ns: u64`
- `scan_number: u64`
- `region_seq: u64`
- `publish_latency_us: f64`
- `has_entry: u8`
- trailing reserved padding

Serialized little-endian by the shared `pulse_frame` crate.

### 6.4 Current use

- optional
- derived
- non-authoritative
- useful for the Rust TUI, especially when decoupling bridge and display

As of current Rust TUI work, the TUI can also read the Zinc region directly and no longer requires the bridge.

---

## 7. Addressing Rules, Precisely

### 7.1 Zinc regions

There are **three names** to keep distinct:

1. **logical region name**
   - what code passes to Zinc
   - example: `uv_shadow_state`
2. **POSIX SHM object name**
   - what the Linux SHM API sees
   - example: `/zinc_uv_shadow_state`
3. **filesystem entry under `/dev/shm`**
   - example: `/dev/shm/zinc_uv_shadow_state`

The reader/writer contract uses **(1)**.

The operator inspecting `/dev/shm` sees **(3)**.

Confusing (1) and (3) is the common failure mode.

### 7.2 How a writer and reader agree on the same region

They must share:

- the same transport family
  - Zinc or `iceoryx2`
- the same prefix / service namespace
- the same slot derivation rule
- the same payload framing
- the same semantic schema

For Zinc-backed UV:

- prefix source: `UV_ZINC_PREFIX`
- slot naming logic: `_slot_region(prefix, slot)`
- payload format: `UVZINC01`
- state slot: `blue_prime` -> `uv_shadow_state`

For DITAv2:

- prefix passed to `RealZincPlane(prefix=...)`
- suffixes: `_intent`, `_state`, `_control`, `_venue`
- payload format: `!QQ + JSON`

For `iceoryx2`:

- service name must match exactly
- payload struct must match exactly

### 7.3 How two running algo instances know “their” region

They do not discover “theirs” by magic. They must be started with a namespace choice.

Correct pattern:

- instance A:
  - `UV_ZINC_PREFIX=uv_shadow_a`
  - state region -> `uv_shadow_a_state`
- instance B:
  - `UV_ZINC_PREFIX=uv_shadow_b`
  - state region -> `uv_shadow_b_state`

Then:

- A writer writes only `uv_shadow_a_*`
- A TUI for A opens only `uv_shadow_a_*`
- B writer/TUI use `uv_shadow_b_*`

For test runs, this pattern is already used:

- `uv_t6_<id>` prefixes produce regions like `uv_t6_8933e28c84_state`

### 7.4 Collision rule

If two independent algo instances reuse the same Zinc logical region name:

- they are not isolated
- last writer wins
- readers observe a mixed stream
- observability becomes invalid

So region naming is not cosmetic. It is the namespace boundary.

---

## 8. What Is In Use For What, Now

### 8.1 BLUE-PRIME authoritative observability

- transport container: Zinc region
- logical region: `uv_shadow_state`
- payload framing inside region: `UVZINC01`
- semantic schema: BLUE-PRIME snapshot JSON
- status: live and observed

### 8.2 BLUE-PRIME hook observability

- transport container: Zinc region
- logical region: `uv_shadow_blue_prime_hooks`
- payload framing inside region: `UVZINC01`
- semantic schema: hook runner frame JSON
- status: live region observed

### 8.3 UV Rust TUI direct mode

- reads: Zinc region directly
- logical region by default: `uv_shadow_state`
- expects payload framing: `UVZINC01`
- status: implemented and verified

### 8.4 UV Rust TUI bridge mode

- bridge reads Zinc region `uv_shadow_state`
- bridge republishes `PulseFrame` over `iceoryx2` service `uv/pulse`
- TUI subscribes to `uv/pulse`
- status: implemented and verified, but secondary to direct Zinc reads

### 8.5 DITAv2 real shared-memory plane

- transport container: Zinc region
- logical regions: `<prefix>_{intent,state,control,venue}`
- payload framing: `!QQ + JSON`
- status: source-authoritative, but not directly observed live under tested prefixes during this inspection

---

## 9. Operator Notes

### 9.1 To inspect the live authoritative UV region

Use the logical name:

```python
from zinc import SharedRegion
r = SharedRegion.open("uv_shadow_state")
buf = r.as_buffer()
```

Do **not** do:

```python
SharedRegion.open("zinc_uv_shadow_state")
```

That is the `/dev/shm` object name, not the logical Zinc name.

### 9.2 To tell whether a region uses DITAv2 or UV framing

Look at the first visible bytes of `as_buffer()`:

- `UVZINC01` -> UV / BLUE-PRIME seqlock payload
- otherwise, if the first 16 bytes parse as `!QQ` and the JSON decodes, likely DITAv2 packet framing

You will **not** see `ZINC_REG` there through normal adapter reads, because that Zinc header lives before the exposed data area.

### 9.3 To run two independent observability stacks

Assign different prefixes up front. Example:

```bash
export UV_ZINC_PREFIX=uv_shadow_main
export UV_ZINC_PREFIX=uv_shadow_soak
```

Then point each consumer at its matching logical region names.

If `iceoryx2` bridge mode is used in parallel too, it needs the same kind of namespacing. Current bridge service default is singleton `uv/pulse`; that should be parameterized if simultaneous parallel bridges are required.

---

## 10. Bottom Line

The current live BLUE-PRIME observability stack is:

- **container:** Zinc shared memory
- **live authoritative region:** `uv_shadow_state`
- **live payload framing:** `UVZINC01` seqlock JSON
- **live hook side region:** `uv_shadow_blue_prime_hooks`

DITAv2’s real Zinc plane is a different format:

- same Zinc container idea
- different region family
- different payload envelope: `!QQ + JSON`

The Rust `PulseFrame` is yet another layer:

- `iceoryx2` service payload
- derived from the authoritative Zinc region
- not itself the source of truth

The rule for addressing is simple and must be followed strictly:

- **open logical Zinc names through Zinc**
- **inspect `/dev/shm/zinc_*` only as an operator artifact**
- **namespace parallel instances by prefix**
- **match the payload framing to the subsystem**