siloqy/prod/docs/SHARED_MEMORY_FORMATS_AND_ADDRESSING_20260704.md

Shared Memory Formats And Addressing

Date: 2026-07-04
Host: DOLPHIN
Scope: all shared-memory formats and SHM-adjacent IPC formats directly inspected from source during UV / BLUE-PRIME / DITAv2 work.

This document is intentionally concrete. It separates:

1. the transport container (Zinc region, or iceoryx2 service),
2. the payload framing inside that container,
3. the semantic payload schema written by a given subsystem,
4. the addressing rule by which a writer and a reader find the same object.

It also records which format is actually in use for each known subsystem as inspected on this host.

Short Answer

• A Zinc region is addressed by its logical region name, passed to SharedRegion.create(name, ...) / SharedRegion.open(name).
• On Linux, Zinc materializes that as a POSIX SHM object named /zinc_<logical_name>, which appears in /dev/shm as zinc_<logical_name>.
• Therefore:
  • reader/writer open uv_shadow_state
  • the OS object visible under /dev/shm is zinc_uv_shadow_state
  • opening zinc_uv_shadow_state through the Zinc API is wrong and fails
• Two algo instances avoid collision by using different logical prefixes and deriving all region names from that prefix.

For iceoryx2, the address is not a Zinc region name. It is the service name such as uv/pulse.

---

1. What Exists

1.1 Formats actually encountered

Layer	Transport	Address form	Payload framing	Current role
Zinc region container	Zinc shared memory	logical name name; OS object /dev/shm/zinc_<name>	Zinc internal region header, then user data area	base transport for Zinc-backed regions
DITAv2 plane packet	Zinc region data area	<prefix>_intent, <prefix>_state, <prefix>_control, <prefix>_venue	!QQ = (seq, json_size) + UTF-8 JSON	DITAv2 real Zinc plane
DITAv2 control packet	Zinc region data area	<prefix>_control	same !QQ + JSON envelope	DITAv2 control plane
UV / BLUE-PRIME snapshot	Zinc region data area	uv_shadow_state	UVZINC01 + dual-seq seqlock header + UTF-8 JSON	authoritative BLUE-PRIME shadow snapshot, live
UV hook frame	Zinc region data area	uv_shadow_blue_prime_hooks	same UVZINC01 + dual-seq + UTF-8 JSON	hook observability frame, live/auxiliary
UV test scratch snapshot	Zinc region data area	uv_t6_<id>_state	same UVZINC01 + dual-seq + UTF-8 JSON	test / temporary namespaces
UV pulse bridge payload	iceoryx2 publish-subscribe service	service name uv/pulse	fixed 40-byte PulseFrame POD	optional derived feed for Rust TUI; not authoritative

1.2 Not shared memory, but easy to confuse with it

The old file transport in prod/clean_arch/violet/uv/shm.py:

• explicit mode only
• writes JSON files under UV_SHM_ROOT / /dev/shm/uv
• not the runtime BLUE-PRIME contract
• retained for tests / fallback diagnostics only

That path is not documented further here because the request is specifically about shared memory.

---

2. Zinc Region Container Format

Authoritative sources:

• zinc/core/src/header.rs
• zinc/core/src/region.rs
• zinc/core/src/platform/unix.rs
• zinc/adapters/python/zinc/_ffi.py
• zinc/adapters/python/zinc/__init__.py

2.1 Addressing

Zinc validates and opens a logical name such as:

• uv_shadow_state
• vst_dita_state
• dolphin_violet_control

The Linux backend then maps that to a POSIX SHM object:

• logical name: uv_shadow_state
• OS shm object: /zinc_uv_shadow_state
• visible filesystem entry: /dev/shm/zinc_uv_shadow_state

This mapping is implemented in zinc/core/src/platform/unix.rs.

2.2 Name rules

Valid logical characters are:

• ASCII alphanumeric
• _
• -

Slashes are not allowed in raw Zinc names. Callers that start from path-like prefixes sanitize before opening.

2.3 Region layout

Every Zinc region is:

1. one page of Zinc-owned metadata/header
2. followed by the user data area

The Rust core exposes the user data area by returning page_size() bytes past the mapping base. The Python adapter likewise exposes only the user data area through SharedRegion.as_buffer().

This point matters:

• the underlying region really is Zinc
• but a reader using as_buffer() does not see the Zinc header at byte 0
• it sees byte 0 of the user payload

That is why a live UV reader sees UVZINC01 at the first visible bytes even though the mapped OS object is a Zinc region.

2.4 Zinc internal header

From zinc/core/src/header.rs:

• magic: u64 = "ZINC_REG"
• version: u16 = currently 2
• flags: u16
• notify_seq: AtomicU32
• capacity: u64
• ref_count: AtomicU32
• owner_pid: AtomicI32
• created_at: u64
• name_hash: u64
• ring_head: AtomicU64
• ring_tail: AtomicU64

This header is aligned to one cache line and occupies the Zinc-managed metadata area, not the caller-visible payload buffer.

2.5 Notify/wait semantics

Zinc provides:

• notify()
• wait(timeout_ms)

These operate on notify_seq in the Zinc header. The payload framing on top of Zinc is owned by the higher-level subsystem.

---

3. DITAv2 Real Zinc Plane Format

Authoritative sources:

• prod/clean_arch/dita_v2/real_zinc_plane.py
• prod/clean_arch/dita_v2/real_control_plane.py

3.1 Addressing

RealZincPlane(prefix=...) derives names as:

• base = prefix.strip("/").replace("/", "_")
• intent_name = f"{base}_intent"
• state_name = f"{base}_state"
• control_name = f"{base}_control"
• venue_name = f"{base}_venue"

So if prefix="vst/dita":

• logical names become:
  • vst_dita_intent
  • vst_dita_state
  • vst_dita_control
  • vst_dita_venue
• OS objects become:
  • /dev/shm/zinc_vst_dita_intent
  • /dev/shm/zinc_vst_dita_state
  • /dev/shm/zinc_vst_dita_control
  • /dev/shm/zinc_vst_dita_venue

3.2 Region capacities

Defaults in RealZincPlane.__init__:

• intent_capacity = 1 << 20 = 1 MiB
• state_capacity = 1 << 20 = 1 MiB
• control_capacity = 1 << 20 = 1 MiB
• venue_region also uses control_capacity

3.3 Payload framing inside the Zinc data area

DITAv2 does not use the UVZINC01 header.

It uses:

• struct.pack("!QQ", seq, len(json_bytes))
• followed by UTF-8 JSON bytes

That is:

• bytes 0..8: seq as big-endian u64
• bytes 8..16: JSON length as big-endian u64
• bytes 16..16+size: JSON bytes

There is no dual-seq torn-read guard here. Readers trust:

• the header is present
• size is sane
• the JSON decodes

3.4 Semantic payloads

By region:

• *_intent: {"items": [...]} where items are serialized KernelIntents
• *_state: {"slots": [...]} where slots are serialized TradeSlots
• *_control: {"control": {...}} where value is a KernelControlSnapshot
• *_venue: {"venue": {...}} where value is a VenueTelemetrySnapshot

3.5 Write semantics

Writers:

• increment a region-local sequence
• build !QQ + JSON
• copy packet into the entire visible data buffer
• zero the tail
• call region.notify()

This is a simple packet format, not a seqlock format.

3.6 Current use

This is the intended real shared-memory format for DITAv2 when RealZincPlane / RealZincControlPlane are active.

At the time of this writing, I did not find live openable regions under the tested names:

• vst_dita_intent
• vst_dita_state
• vst_dita_control
• vst_dita_venue
• dolphin_violet_intent
• dolphin_violet_state
• dolphin_violet_control
• dolphin_violet_venue

So this format is source-authoritative, but not directly observed live under those tested prefixes at sample time.

---

4. UV / BLUE-PRIME Zinc Snapshot Format

Authoritative source:

• prod/clean_arch/violet/uv/shm.py

This is the important one for current UV / BLUE-PRIME observability.

4.1 Addressing

Default prefix:

• UV_ZINC_PREFIX, default uv_shadow

Logical region naming rule:

• prefix.strip("/").replace("/", "_")
• plus _<slot>
• except slot "blue_prime" and slot "state" are both normalized to _<prefix>_state

Examples:

• publish("blue_prime", ...) -> logical region uv_shadow_state
• publish("state", ...) -> logical region uv_shadow_state
• publish("blue_prime_hooks", ...) -> logical region uv_shadow_blue_prime_hooks

4.2 Region capacities

Default:

• UV_ZINC_STATE_BYTES, default 64 << 20 = 64 MiB

Observed live regions:

• /dev/shm/zinc_uv_shadow_state
• /dev/shm/zinc_uv_shadow_blue_prime_hooks

Observed test/scratch leftovers:

• /dev/shm/zinc_uv_t6_8933e28c84_state
• /dev/shm/zinc_uv_t6_a1f7e3254b_state
• /dev/shm/zinc_uv_t6_b16924a96a_state
• /dev/shm/zinc_uv_t6_eed1be7947_state

4.3 Payload framing inside the Zinc data area

Header:

• magic = b"UVZINC01"
• struct = struct.Struct("!8sQQQ")

Visible data-area layout:

1. bytes 0..8: magic "UVZINC01"
2. bytes 8..16: seq_a big-endian u64
3. bytes 16..24: seq_b big-endian u64
4. bytes 24..32: JSON payload size big-endian u64
5. bytes 32..32+size: UTF-8 JSON payload

4.4 Write semantics

Writer uses a simple seqlock pattern:

1. compute next logical sequence seq
2. derive:
   • seq_even = seq * 2
   • seq_odd = seq_even - 1
3. write header with odd/in-flight sequence and size = 0
4. copy JSON body
5. optionally zero one byte after body
6. write header again with even/stable sequence and real size
7. call region.notify()

4.5 Read semantics

Reader accepts payload only if all are true:

• magic == UVZINC01
• seq_a != 0
• seq_a is even
• seq_a == seq_b
• size is in bounds
• after copying the body, rereading the header yields the exact same
  • magic
  • seq_a
  • seq_b
  • size

If any of that fails, the read is treated as torn / not yet initialized.

4.6 Semantic payloads

This format carries JSON snapshots rather than a fixed struct. Current known uses:

• uv_shadow_state
  • authoritative BLUE-PRIME snapshot
  • includes domains such as meta, scan, live_inputs, engine, decision, efsm, dita, ram, source_trace
• uv_shadow_blue_prime_hooks
  • one published hook runner frame per scan
  • keys include scan, hooks, hook_count, ok_count, total_us

4.7 Current use

This is the live authoritative BLUE-PRIME shared-memory format currently observed on host.

Verification made on host:

• SharedRegion.open("uv_shadow_state") succeeds
• SharedRegion.open("zinc_uv_shadow_state") fails
• first bytes of visible region buffer are 55565a494e433031... = UVZINC01

That is the definitive proof that:

• the region container is Zinc
• the payload framing currently in use inside the container is UVZINC01

---

5. UV Hook Frame Region

Authoritative source:

• prod/clean_arch/violet/uv/hooks/runner.py

The hook runner publishes:

• self.shm.publish("blue_prime_hooks", frame)

Because ShmChannel defaults to Zinc-backed transport, that becomes:

• logical region: uv_shadow_blue_prime_hooks
• OS shm object: /dev/shm/zinc_uv_shadow_blue_prime_hooks

The payload framing is the same UVZINC01 seqlock JSON envelope documented above.

The semantic payload differs:

• per-scan hook effects
• timing / ordering / success counts

This is auxiliary observability, not the main state snapshot.

---

6. iceoryx2 Pulse Bridge Format

Authoritative sources:

• uv_tui/crates/pulse_frame/src/lib.rs
• uv_tui/bridge/src/lib.rs
• uv_tui/tui/src/lib.rs

6.1 What it is

This is not the authoritative shared-memory snapshot. It is a derived bridge feed:

1. bridge opens Zinc region uv_shadow_state
2. bridge decodes the UVZINC01 payload
3. bridge extracts a reduced pulse contract
4. bridge republishes that reduced contract over iceoryx2 service uv/pulse

6.2 Addressing

Address is a service name, not a Zinc region name:

• service: uv/pulse

Current code hardcodes a singleton default service. Unlike Zinc prefixes, this is not yet namespaced per parallel algo instance.

6.3 Payload framing

Fixed PulseFrame, length 40 bytes:

• observe_mono_ns: u64
• scan_number: u64
• region_seq: u64
• publish_latency_us: f64
• has_entry: u8
• trailing reserved padding

Serialized little-endian by the shared pulse_frame crate.

6.4 Current use

• optional
• derived
• non-authoritative
• useful for the Rust TUI, especially when decoupling bridge and display

As of current Rust TUI work, the TUI can also read the Zinc region directly and no longer requires the bridge.

---

7. Addressing Rules, Precisely

7.1 Zinc regions

There are three names to keep distinct:

1. logical region name
   • what code passes to Zinc
   • example: uv_shadow_state
2. POSIX SHM object name
   • what the Linux SHM API sees
   • example: /zinc_uv_shadow_state
3. filesystem entry under /dev/shm
   • example: /dev/shm/zinc_uv_shadow_state

The reader/writer contract uses (1).

The operator inspecting /dev/shm sees (3).

Confusing (1) and (3) is the common failure mode.

7.2 How a writer and reader agree on the same region

They must share:

• the same transport family
  • Zinc or iceoryx2
• the same prefix / service namespace
• the same slot derivation rule
• the same payload framing
• the same semantic schema

For Zinc-backed UV:

• prefix source: UV_ZINC_PREFIX
• slot naming logic: _slot_region(prefix, slot)
• payload format: UVZINC01
• state slot: blue_prime -> uv_shadow_state

For DITAv2:

• prefix passed to RealZincPlane(prefix=...)
• suffixes: _intent, _state, _control, _venue
• payload format: !QQ + JSON

For iceoryx2:

• service name must match exactly
• payload struct must match exactly

7.3 How two running algo instances know “their” region

They do not discover “theirs” by magic. They must be started with a namespace choice.

Correct pattern:

• instance A:
  • UV_ZINC_PREFIX=uv_shadow_a
  • state region -> uv_shadow_a_state
• instance B:
  • UV_ZINC_PREFIX=uv_shadow_b
  • state region -> uv_shadow_b_state

Then:

• A writer writes only uv_shadow_a_*
• A TUI for A opens only uv_shadow_a_*
• B writer/TUI use uv_shadow_b_*

For test runs, this pattern is already used:

• uv_t6_<id> prefixes produce regions like uv_t6_8933e28c84_state

7.4 Collision rule

If two independent algo instances reuse the same Zinc logical region name:

• they are not isolated
• last writer wins
• readers observe a mixed stream
• observability becomes invalid

So region naming is not cosmetic. It is the namespace boundary.

---

8. What Is In Use For What, Now

8.1 BLUE-PRIME authoritative observability

• transport container: Zinc region
• logical region: uv_shadow_state
• payload framing inside region: UVZINC01
• semantic schema: BLUE-PRIME snapshot JSON
• status: live and observed

8.2 BLUE-PRIME hook observability

• transport container: Zinc region
• logical region: uv_shadow_blue_prime_hooks
• payload framing inside region: UVZINC01
• semantic schema: hook runner frame JSON
• status: live region observed

8.3 UV Rust TUI direct mode

• reads: Zinc region directly
• logical region by default: uv_shadow_state
• expects payload framing: UVZINC01
• status: implemented and verified

8.4 UV Rust TUI bridge mode

• bridge reads Zinc region uv_shadow_state
• bridge republishes PulseFrame over iceoryx2 service uv/pulse
• TUI subscribes to uv/pulse
• status: implemented and verified, but secondary to direct Zinc reads

8.5 DITAv2 real shared-memory plane

• transport container: Zinc region
• logical regions: <prefix>_{intent,state,control,venue}
• payload framing: !QQ + JSON
• status: source-authoritative, but not directly observed live under tested prefixes during this inspection

---

9. Operator Notes

9.1 To inspect the live authoritative UV region

Use the logical name:

from zinc import SharedRegion
r = SharedRegion.open("uv_shadow_state")
buf = r.as_buffer()

Do not do:

SharedRegion.open("zinc_uv_shadow_state")

That is the /dev/shm object name, not the logical Zinc name.

9.2 To tell whether a region uses DITAv2 or UV framing

Look at the first visible bytes of as_buffer():

• UVZINC01 -> UV / BLUE-PRIME seqlock payload
• otherwise, if the first 16 bytes parse as !QQ and the JSON decodes, likely DITAv2 packet framing

You will not see ZINC_REG there through normal adapter reads, because that Zinc header lives before the exposed data area.

9.3 To run two independent observability stacks

Assign different prefixes up front. Example:

export UV_ZINC_PREFIX=uv_shadow_main
export UV_ZINC_PREFIX=uv_shadow_soak

Then point each consumer at its matching logical region names.

If iceoryx2 bridge mode is used in parallel too, it needs the same kind of namespacing. Current bridge service default is singleton uv/pulse; that should be parameterized if simultaneous parallel bridges are required.

---

10. Bottom Line

The current live BLUE-PRIME observability stack is:

• container: Zinc shared memory
• live authoritative region: uv_shadow_state
• live payload framing: UVZINC01 seqlock JSON
• live hook side region: uv_shadow_blue_prime_hooks

DITAv2’s real Zinc plane is a different format:

• same Zinc container idea
• different region family
• different payload envelope: !QQ + JSON

The Rust PulseFrame is yet another layer:

• iceoryx2 service payload
• derived from the authoritative Zinc region
• not itself the source of truth

The rule for addressing is simple and must be followed strictly:

• open logical Zinc names through Zinc
• inspect /dev/shm/zinc_* only as an operator artifact
• namespace parallel instances by prefix
• match the payload framing to the subsystem