Files
siloqy/prod/docs/UV_MASTER_SPEC_20260702.md

224 lines
17 KiB
Markdown
Raw Normal View History

# UV (ULTRAVIOLET) MASTER SPEC — 2026-07-02
**Author:** Fable (Claude Fable 5; read Fable/Claude-4.8 history as one unit).
**Supersedes:** `~/.claude/plans/drifting-knitting-zebra.md` (2026-06-29, the approved C0C9 plan)
— this doc IS that plan, amended with verified 2026-07-02 findings. Commit into `/root/violet.git`
main at T0.
**Companions (unchanged, still binding):** `uv/UV_DEV_LOOP.md` (dev loop), `uv/specs/SPEC_00_*`
+ 11 hook specs + PASS4 WIRE specs, `prod/docs/UV_BLUE_PRIME_SHM_RESHAPE_SPEC.md`,
`prod/docs/UV_BLUE_PRIME_ZINC_SHADOW_SPEC.md`, `prod/docs/AI_DEV_DOCTRINE.md`,
`prod/docs/TESTING_DOCTRINE.md`.
---
## 1. Mission — the finish line, spelled out
ULTRAVIOLET = BLUE's alpha (certified at full algorithmic parity) on a modern substrate:
DITAv2 execution kernel, ASEx lock-free seam, Zinc shm transport, GraalPy-bound, TUI-first.
BLUE-PRIME is the parity **oracle** (instrumented read-only copy of BLUE); UV is certified
against PRIME; PRIME is certified against live BLUE by data diff — never by touching BLUE.
**This spec is DONE only when (end-state acceptance, Wave 4 §9):**
1. UV places and manages **real BingX VST testnet orders through DITAv2 as the exchange-exec
kernel, with the ASEx improvements underneath** (single-writer, lock-free, no async seams).
2. Every UV trade's decision chain is traceable to the **certified-BLUE algorithm at Q=scan**
(journaled in `dolphin_uv.uv_decisions`; the parity differ stays green in shadow while
UV trades).
3. SL/TP and protective market action execute **faster than eigenscan cadence** (rate-parity
stage 2 unlock) — demonstrably, with measured action latency in the journal.
4. Soak-proven on VST: N days, zero orphan positions, reconciler-clean, DARK→armed ladder
respected at every step.
5. Everything built **Graal-ready** (§10) — the Graal migration itself is a separate later
dev-ops spec, but no wave may ship code that violates the §10 constraints.
"UV stood up with BLUE parity" (end of Wave 3) is the midpoint of this spec, not its end.
## 2. Rate-parity doctrine (three stages; Q is a first-class dial)
1. **Parity cert:** UV may run a fast internal clock but the decision surface is downsampled
through a **Q factor** to BLUE's NG7 eigenscan cadence (~6 s), so ALL timing-derived
affordances (bars_held, max_hold_bars, staleness windows, EFSM post-win windows, dedup)
match BLUE **warts and all**. Parity diffs join on **scan identity, never wall clock**.
2. **Post-cert:** SL/TP + market action execute FASTER than eigenscan (DITAv2/ASEx);
alpha decisions stay scan-quantized. This is the economic point of UV.
3. **Far:** Graal substrate; the Q layer itself loosened **gradually**, parity-guarded
(certified-Q config stays runnable as the rollback baseline). Hard ceiling today: NG7
eigenscan generation compute. Operator caution (doctrine): the 6 s @ 15 m correlation
rhythm may itself carry alpha — loosening Q is an experiment, never an assumption.
**Engineering rule:** the Q-quantizer is ONE explicit component with an interface (human-defined
boundary per AI_DEV_DOCTRINE #8), never cadence assumptions scattered through modules.
## 3. Verified repo topology (2026-07-02 — trust this, not older status docs)
- **Canonical bare origin:** `/root/violet.git`. `main` @ `15fb189` = 11 hooks + WIRE.1/2/3 +
WIRE.5 journal + pi's comprehensive hook tests.
- **Unmerged, to integrate at T0:** `uv/wire-4-live-inputs` @ `2fd1f9d` (live_inputs + EFSM
mirror); `docs/uv-blue-prime-shm-reshape` @ `ee4e5eb` (snapshot contract + zinc_shadow +
1078 tests; branched from `ba7e12e`, so it lacks WIRE.4/5).
- **Divergent clone line:** `/root/uv-wt/uv` + `/root/uv-wt/blue-prime` @ `1902e4b` (5 unpushed
commits forked at `5d583e4`): TUI v2, EFSM CH-mirror fix `6156e17`, real /dev/shm zinc region
writer `01e35c4`. Salvage-review at T0. **The live soak (zellij `UV_BLUE-PRIME_TUI`) runs this
line** — real ZINC_REG region `/dev/shm/zinc_uv_shadow_state`, seq advancing at scan cadence.
- **RETIRED as wrong:** `/root/uv-wt/uv/prod/docs/UV_DEV_CURRENT_STATUS_2026-07-01.md`'s claim
that `/root/uv-wt/uv` is canonical/"ahead" — it is a fork missing hooks/WIRE/reshape. Its
UV-core / PRIME-oracle / VIOLET-substrate *conceptual* distinction remains correct and adopted.
- Dev loop stays as `UV_DEV_LOOP.md`: local disk only, clone-per-chunk off the bare, branch
`uv/<name>`, push → scoped CI, **Fable integrates to main**.
## 4. Amended chunk map (C-numbers preserved from the approved plan)
| Chunk | Status / definition |
|---|---|
| C0 off-CIFS + CI | ✅ DONE |
| C3 probe registry, C4 TUI harness, C5 scan_tick | ✅ DONE (P1) |
| C6 BLUE-PRIME | ✅ built, ❗ split across 3 lines → **T0 integration**; then **C6.5 (NEW): pollution neutralization** → T1 |
| **C6.5 pollution guard (NEW)** | PRIME must be provably write-free toward BLUE's namespaces (CH `dolphin.*`, HZ, shared spool). See `UV_TASK_T1`. |
| C7 parity instrument (**REDEFINED**) | Not an entry_signal probe. C7a = journal hardening + differ core (T2). C7b = differ tests + TUI parity panel (T3). C7c = certification reporter (T4). |
| C8 BingX reconciler | Wave 2, SHRUNK: verify/adapt DITAv2's existing reconcile logic under UV's `u-` prefix + rate-budget tests — not a new build (§8). |
| C10 (NEW) DITAv2 exec seam | Wave 2: certified-PRIME decision path promoted to active + KernelIntent → DITAv2 → VST, v4-runner pattern as template (§8). |
| C11 (NEW) SL/TP fast clock | Wave 2, post-canary unlock (§8). **Empirical justification (2026-07-02 forensics):** scan-cadence stop overshoot is real — FET 2,433 exited at 1.61% adverse vs 1.2% stop (+34% overshoot), WIN 1,920 at 1.32% (+10%); ≈$800/wk excess on 3 stops alone. **Millisecond forensics on the FET stop (dolphin.obf_universe @130ms):** price approached the stop visibly for 19s (0.87%→1.20%), breached at 16:08:15.9, next eigenscan not until 16:08:19.6 (measured scan gaps 1112s, not 6s) → entire overshoot lived inside one scan gap; exec latency after the scan was <1s (mechanism healthy). Book stayed orderly throughout (spread pinned 5.4bps, top-5 depth $140200K both-sides) breach-time exit was executable; like-for-like save ≈$500580 on this trade. Faster-than-scan SL action recovers this by construction. NOTE: `dolphin.obf_fast_intrade` (in-trade fast OB feed) has 0 rows dead wiring in BLUE; UV's fast clock must not repeat this (a populated fast feed is the C11 sensor). |
| C1 UV Zinc plane + Q-dial, C2 GraalPy smoke, C9 DITAv2 probe | **DEFERRED** post-first-trade (§9). Graal-readiness is enforced NOW via §10 instead. |
| UV core runtime (separate rewrite) | **CANCELLED as churn** — UV v1 IS the certified PRIME path promoted (§8). `uv/blue_prime/` freezes as oracle at certification. |
## 5. Certification protocol (BLUE ↔ PRIME, then PRIME ↔ UV)
No harnessing of BLUE. PRIME runs read-only beside live BLUE; certification is a
**tick-aligned data diff**:
- PRIME's per-scan record: `dolphin_uv.prime_decisions` (WIRE.5 journal: inputs, 11 hook
effects, decision) + zinc snapshot.
- BLUE's record: `dolphin.trade_events` (+ logs) — entries/exits with asset/side/leverage.
- Continuous signal: per-scan hook/leverage modulation self-consistency; hard gate: **entry
events**, matched on scan identity, **bit-identity** (PASS2.5 standard: mismatch = bug,
not tolerance).
- **Gate (restructured 2026-07-02, operator: compute-bound not calendar-bound):**
- **Gate A — replay-cert (gating):** replay BLUE's RECORDED input history (months of
eigen-scans, all realized trades) through PRIME's decision path, Q-quantized
(scan-sequenced, warts and all), vectorized where the kernel allows. Bit-diff every
decision vs `dolphin.trade_events` — every entry BLUE ever made, not 3. 0 unexplained
diffs. (Synthetic inputs CANNOT gate BLUE-parity — BLUE has no recorded answer for them.)
- **Gate B — faultline assault (gating):** Hypothesis/adversarial/fuzz on breakspots
(vel_div threshold boundary, EFSM transitions incl. post-win LONG overlay, staleness,
poison) — properties: PRIME-internal consistency + PRIME↔UV agreement.
- **Gate C — live plumbing (non-gating):** guarded soak keeps running; live HZ reads,
scan gaps, mirror hydration clean over X hours; any live entry = bonus bit-check.
- Pollution invariant green throughout; operator signs the cert. Then PRIME freezes as
oracle; UV certifies against PRIME with the same instruments (Gate A replay + Gate B),
writing `dolphin_uv.uv_decisions`.
- **Wave-2+ note (operator):** multi-instance PRIME-[n] clone farms (forkd / workdir.dev
-class system forks; VIBRASS bandit meta-gov) — design-in now: per-instance CH namespace
`dolphin_uv_{n}`, per-instance zinc prefix `uv_shadow_{n}`, injectable clock/Q.
## 6. Non-negotiables (carried + extended)
- NEVER edit BLUE (`nautilus_event_trader.py`, kernels, `prod/ch_writer.py` [vendored/shared],
supervisord, HZ contents, `dolphin.*` tables). PRIME reads BLUE's world; writes NOTHING into it.
- `dolphin_uv.*` is UV/PRIME's only CH namespace. Hard-guarded in code + tests
(journal URL guard raises on non-dolphin_uv; T1 no_write_guard diverts every other CH write
to a local audit file; T4 reporter is zero-CH-write; HZ is wrapped read-only).
- **Separate-install direction (operator, 2026-07-02):** before UV's first VST trade (C10),
UV's own writes move to a DEDICATED CH instance (own port/datadir or container) — namespace
isolation is the guard, instance isolation is the wall. UV needs NO HZ writes in wave 1
(reads BLUE's HZ read-only); if UV ever needs its own KV plane, it gets its own instance —
never keys in BLUE's cluster.
- VST only; `ALLOW_MAINNET=0`; DARK until operator arms.
- Testing doctrine: mutation litmus, poison/edges/concurrency, no green-by-pollution,
run your own suite before push. AI_DEV doctrine: one problem/one branch/one PR,
explicit staging, docs > chat.
- Vendored-drift gate: VENDOR.lock components edited only upstream + `vendor_sync.sh`.
## 7. Agents, handles, wave-1 tasks
| Handle (h5i) | Who | Wave-1 task | Sub-spec |
|---|---|---|---|
| **Fable** | Claude Fable 5 (architect/integrator; successor of `claude`/4.8 + `cc-ultrav-1`) | **T0**: git integration pass (main + wire-4 + reshape + clone salvage → main); commit this spec; correct/retire stale status doc | this doc §34 |
| **codex** | Codex 5.4mini | **T1 (CRITICAL)**: pollution forensics + neutralization + PRIME relaunch | `UV_TASK_T1_CODEX_POLLUTION_GUARD.md` |
| **mm_VIOLET1** | mimocode (slow, precise) | **T2**: journal hardening + C7a differ core | `UV_TASK_T2_MM_JOURNAL_DIFFER.md` |
| **pi_nvnemo** | PI harness (Nemotron) | **T3**: C7b differ test suite + TUI parity panel | `UV_TASK_T3_PI_PARITY_TESTS_TUI.md` |
| **cmd-PASS1.1** | Command Code / DeepSeek (operator fires instance) | **T4**: C7c certification reporter | `UV_TASK_T4_CMD_CERT_REPORTER.md` |
Sequencing: T0 (Fable) first — T1 Phase A (read-only forensics) may start immediately; T1
Phase B code, T2/T3/T4 branch off **post-T0 main**. All reporting on the canonical h5i bus
(`/mnt/dolphinng5_predict`); reply to **Fable**. Sub-specs live in
`prod/docs/uv_subspecs/` (committed to the violet repo at T0).
## 8. Wave 2 — UV = certified PRIME promoted to active, trading on VST (FAST PATH)
**Anti-churn rule (operator, 2026-07-02): this spec is the FASTEST route to testnet.**
No rewrites of things that already work. Concretely:
- **NO new UV decision core.** PRIME already runs BLUE's real engine + 11 hooks + EFSM mirror
+ live inputs. The day PRIME certifies, **UV v1 = the certified PRIME decision path promoted
from shadow to active** (new process/config, journals `dolphin_uv.uv_decisions`, DARK) +
a KernelIntent emitter. Zero algorithm code rewritten between certification and first trade.
- **NO new exec layer.** DITAv2 is already VST-proven (PINK burn-in; VIOLET v4 runner
precedent — reuse its KernelIntent→DITAv2 wiring pattern as a NEW instance; never edit the
live v4 runner). DITAv2 stays vendored (VENDOR.lock; improvements upstream + vendor_sync).
ASEx improvements ride along inside DITAv2's existing integration — the single-writer /
no-async-seam guarantees are why it's the kernel; we do not re-plumb them.
- **NO new reconciler from scratch.** Adopt DITAv2's existing reconcile logic (PINK ownership-
filter lineage) under UV's own VST account/clientOrderId prefix (`u-`); C8 shrinks to
"verify + adapt + rate-budget test", not "build".
- **Arming ladder (operator-gated per rung):** DARK (journal-only) → observe-only → VST canary
(min size, single slot) → staged size. `ALLOW_MAINNET=0` throughout; mainnet is out of scope.
- **Decision-vs-execution parity split:** decision layer must bit-match certified-BLUE at
Q=scan (differ green in shadow while UV trades); execution quality is judged by
reconciler-clean + DITAv2 accounting integrity (fill-price PnL doctrine), since BLUE's
"fills" are in-memory bookkeeping and UV's are real VST mechanics.
- **C11 SL/TP fast clock** (rate-parity stage 2 — protective actions faster than eigenscan,
alpha stays at Q=scan): unlocked AFTER first clean canary trades, not before.
- **Exit = §1 end-state acceptance.** That is the whole spec.
## 9. Deferred (post-first-trade; separate specs — do NOT build in this spec's waves)
- C1 UV-own Zinc plane + generalized Q-dial (UV v1 runs at scan cadence natively — Q=1:1 —
so the dial abstraction earns nothing until stage-3 loosening).
- C2 GraalPy smoke rig; C9 DITAv2 innards probe; TUI beyond the existing panels.
- Wave-Graal: the GraalPy/GraalVM/Graal-OS migration dev-ops pass (own spec when VST soak
is running). Q-loosening experiments (rate-parity stage 3) live there or after.
### 9.1 Sketchlog lane (operator-flagged CRITICAL, 2026-07-02)
Source: `prod/docs/VIOLET_TODO_CRITICAL_DISTRIBUTION_TRACKING_IN_CONSTRAINED_MEMORY.md`
(9 signals mapped to BIBLE integration points; sketchlog = DDSketch/HLL/CMS/DriftSketch,
93 KB constant memory, mergeable monoids, WindowedStreamLog realtime windows, optional C++).
- **Now (observability, zero parity risk):** sketch dimensions (vel_div percentiles,
signal breadth HLL, reversal freq) added to PRIME snapshot + journal as OBSERVE-ONLY
columns; T7 replay computes them over full history = instant candidate-feature backtest.
- **Post-cert (alpha, gated):** signals as decision inputs (esp. #3 rolling-MAE tail
detector → adaptive exits = the left-tail killer; #1 widening; #4 breadth; #6 reversal)
— UV-divergence features via the certified-Q baseline + diff-guarded rollout. NEVER BLUE.
- Merge algebra fits PRIME-[n] farms (coordination-free merge); pure-Python path = §10 G1 ok.
### 9.2 Control plane (operator directive 2026-07-02: "NATS/iceoryx2 the hell out of it")
One control plane over the WHOLE system: fleet lifecycle (start/stop/arm PRIME-[n]/UV/soaks),
config + Q-dial distribution, heartbeats, gate-ledger events, kill-switch propagation.
- **Split doctrine:** data plane intra-box = Zinc/iceoryx2 (ADR-1, adopted); CONTROL plane
inter-process/inter-box = message bus. Candidate: NATS (operator-named). NOTE: ADR-2
reserved Zenoh for inter-box DATA — NATS-for-control vs Zenoh-for-data can coexist;
Fable authors the control-plane ADR when wave 2 opens.
- NOT on the critical path to first testnet trade; REQUIRED before the clone farm.
- Design-in now (already true): every long-running process publishes a zinc snapshot and
takes env-injected config — those are the surfaces the control plane will drive.
## 10. Graal-readiness — build constraints binding NOW despite deferred migration (NFR-G)
Cheap guardrails (mostly "don't do X"), enforced in review — so the later migration is a
runtime swap, not a rewrite:
- **G1** No CPython-only C-extensions in UV hot paths (pure-Python, or Rust behind stable FFI).
- **G2** No `__del__`/refcount-timing for correctness — explicit lifecycle (ASEx Drop-reliance
leak = the cautionary tale).
- **G3** No "GIL makes this safe" — cross-thread state only via single-writer/Zinc/ASEx seams.
- **G4** UV code never imports `hazelcast` directly — HZ quarantined behind the existing
reader seams (HZBridge direction).
- **G5** No hardcoded paths/creds; env-injected config.
- **G6** Long-running RSS-stable, jemalloc-compatible processes.
## 11. T6 (ACTIVE, HIGH PRIORITY — promoted 2026-07-02): real-Zinc unification
The clone line (banked as branch `salvage/uv-clone-line-1902e4b`) carries the REAL mmap Zinc
region transport (`ZincShadowChannel`, prefix `uv_shadow`, 18 h live soak) + the Textual
`tui_v2`. The merged main publishes the same snapshot via atomic file (cross-process, works
today) + in-memory zinc. T6 = port `ZincShadowChannel` under the reshape snapshot contract as
the transport, re-home `tui_v2` on it. Small, spec to follow; does NOT gate T1-D relaunch.