From 255722d6fd1e6a24b8ead289069839178da124d1 Mon Sep 17 00:00:00 2001 From: Codex Date: Wed, 17 Jun 2026 11:51:35 +0200 Subject: [PATCH] =?UTF-8?q?VIOLET=20OA:=20partial=20spec=20PASS=209=20?= =?UTF-8?q?=E2=80=94=20economics=20+=20observability=20completeness=20(OA?= =?UTF-8?q?=20backlog=20complete)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VIOLET_PART_SPEC_OA_TODO_PASS9.md — the final parallelizable pass, discharging VIOLET's binding V1 economics/observability inheritance. Six independent units extending contracts_v3 (Provenance/ EconomicsRow/DecisionDivergenceRow/SoakReadiness): 43. Observability/economics contracts. 44. DDL-first exactly-one-row-per-event economics sink (ReplacingMergeTree key, event_seq, wait_for_async_insert=1, reject-at-source) — the PINK duplicate-emission fix. 45. Provenance + per-trade PnL feedback (capital = anchor + Σ trade-realized; never a balance snapshot — shared-account foreign-fill immunity). 46. Factor-plane divergence monitor v2 (per-factor VIOLET-vs-BLUE abs_err, DDL-first). 47. Soak-readiness aggregator (consolidate all reports → SoakReadiness GO/NO_GO). 48. Soak-readiness @gate — GO iff parity≥thr ∧ latency ∧ determinism ∧ one-row ∧ isolation; EXPECTED NO_GO today by design (CRITICAL #1), the gate makes it machine-checked (not forced green). DDL-first + row-set==DDL parity + reject-at-source discipline throughout; never writes a production table (new dolphin_violet tables only); ch_writer.py not edited. OA BACKLOG COMPLETE: PASS 1–9 (~48 independent contracts_v3-keyed units) cover the full parallelizable DARK surface of V0→V6. Remaining work is non-parallelizable + Claude's: parity root-cause, review of all passes, integration, E2E, VBT re-cert, real-key V4 smoke, live execution. Co-Authored-By: Claude Opus 4.8 --- prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md | 173 ++++++++++++++++++++ prod/docs/VIOLET_TODO_CRITICAL.md | 10 ++ 2 files changed, 183 insertions(+) create mode 100644 prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md diff --git a/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md b/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md new file mode 100644 index 00000000..a11b6c09 --- /dev/null +++ b/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md @@ -0,0 +1,173 @@ +# VIOLET — partial spec for another agent, PASS 9 (OA TODO): ECONOMICS + OBSERVABILITY COMPLETENESS + +Date: 2026-06-17. Final parallelizable pass (PASS 1–8 cover the rest of the DARK-buildable surface). +PASS 9 closes the **economics/observability** inheritance the plan binds VIOLET to, and the +**soak-readiness** machinery that decides go/no-go for a DARK soak. All DARK; independent units +sharing `contracts_v3`. + +**Plan obligations this pass discharges (V1 inheritance, BINDING):** +> "VIOLET inherits and must keep: `pnl_source`/`capital_source` provenance, `event_seq` on every CH +> row, `wait_for_async_insert=1` on economics tables, exactly-one-row-per-event (the PINK +> duplicate-emission fix is open spec debt — VIOLET gates on it), DDL-before-code migration +> discipline, per-trade sizer feedback from trade-realized PnL (never capital deltas)." +Plus: capital = anchor + Σ deltas (never last-value) — the zombie-trade lesson (PASS-4 ledger). + +**Read first (authoritative references — READ ONLY):** +- `prod/clean_arch/violet/shadow_journal.py` — the validate-then-sink + reject-at-source pattern + (the model for every PASS-9 sink). `prod/clean_arch/violet/domain.py` — V-TYPES (`DivergenceRow`, + EpochMs/Seq/MonoNs etc.). `prod/clean_arch/violet/divergence.py` — the V1 divergence monitor. +- `prod/clickhouse/violet/*.sql` + `apply_violet_ddl.py` — DDL-first migration discipline; the + `test_apply_violet_ddl.py` pattern; the row-set==DDL-columns parity test in + `test_violet_shadow_journal.py`. +- `prod/ch_writer.py` — READ for the CH insert path / `wait_for_async_insert` convention; **do NOT + edit it** (its design flaws are documented; VIOLET uses its own sink, mirroring the journal). +- PASS-2 `parity_report.py`, PASS-3 `slippage_metric.py`, PASS-8 cadence telemetry — the reports the + soak-readiness aggregator (Task 47) consolidates. +- CH (read-only): `http://localhost:8123`, user `dolphin`/key `dolphin_ch_2026`; VIOLET db + `dolphin_violet`. NEVER write/alter PRODUCTION tables; new VIOLET tables only, DDL-first. + +--- + +## 0. HARD RULES (identical to PASS 1–8 — summarized) +- **Never edit shared files** (`prod/nautilus_event_trader.py`, `clean_arch/dita_v2/**`, + `dita/decision.py`, `nautilus_dolphin/**`, `blue_parity.py`, `prod/bingx/leverage.py`, + `prod/ch_writer.py`). READ only. +- **VIOLET DARK** — sinks write only to NEW `dolphin_violet` tables; no orders, no venue, no + service/HZ control, no PROGREEN. Never write a production/BLUE table. +- **V-TYPES on all new code**; faithful poison-guards; reject-at-source to a counter/dead_letter, + never crash the sink (the bars_held=-106 spool lesson). +- **DDL-FIRST**: every new table ships a `.sql` in `prod/clickhouse/violet/` BEFORE the sink code; + a row-set==DDL-columns parity test is MANDATORY (mirror `test_violet_shadow_journal.py`). +- **NEW-FILE-ONLY** under `prod/clean_arch/violet/obs/` (+ extend `contracts_v3.py` + new `.sql`). + Do NOT modify in-flight files (`shadow_journal.py`, `divergence.py`, `domain.py`, + `22_violet_decisions.sql`, the PASS-5..8 files). READ + IMPORT them. + +## 0a. COMMIT / BRANCH POLICY (3 shared-index collisions on 2026-06-16 — non-negotiable) +Own `git worktree` (`git worktree add ../vp-oa9 -b agent/oa-violet9`) strongly preferred. Else +never `git add -A`; `git commit -F msg -- ` with explicit pathspec; verify +`git show --stat --format="" HEAD` lists ONLY your files. One commit/task, prefix `VIOLET OA:`, +Co-Authored-By trailer. Tests on `/home/dolphin/siloqy_env/bin/python3`. `git grep` only. + +--- + +## I. SHARED INTERFACE EXTENSIONS (add to `contracts_v3.py`; never fork a parallel type) +Reuse PASS-3..8 types (`CapitalState`, `PositionDelta`, `RegimeView`, etc.). ADD: +1. **`Provenance`** — `pnl_source: str`, `capital_source: str`, `source: str`, `event_seq: Seq`, + `ts: EpochMs`. The provenance stamp every economics row carries. +2. **`EconomicsRow`** — a validated `dolphin_violet.violet_economics` row: `session_id: SessionId`, + `event_seq: Seq`, `asset: Symbol`, `realized_pnl: float (finite)`, `fee: float (ge=0)`, + `capital_after: float (ge=0)`, `pnl_source: str`, `capital_source: str`, `ts: EpochMs`, + `mono_ns: MonoNs`. Field set MUST equal its DDL columns (parity test). +3. **`DecisionDivergenceRow`** — extends the V1 divergence idea to the FACTOR plane: `ts: EpochMs`, + `asset: Symbol`, `factor: str`, `violet_value: float`, `blue_value: float`, `abs_err: float`, + `event_seq: Seq`. +4. **`SoakReadiness`** — `parity_pick_match: float`, `latency_p99_ms: float`, + `determinism_ok: bool`, `exactly_one_row_ok: bool`, `namespace_isolation_ok: bool`, + `verdict: str` ("GO"/"NO_GO"), `reasons: list[str]`. + +If a task needs another field, ADD it here and note it. + +--- + +## TASK 43 — Observability/economics contracts +**Why.** The shared provenance/economics/divergence vocabulary. +**Affected files (NEW):** extend `contracts_v3.py`; `prod/clean_arch/violet/obs/__init__.py`; +`prod/clean_arch/violet/obs/test_violet_obs_contracts.py`. +**Pass criteria.** All types construct + poison-reject (non-finite pnl, negative capital/fee +rejected); provenance strings non-empty where required. No edits outside the new/extended files. + +## TASK 44 — DDL-first exactly-one-row-per-event economics sink [the PINK duplicate-emission fix] +**Why.** The binding inheritance: exactly-one-row-per-event, `event_seq` on every row, +`wait_for_async_insert=1`, validate-then-sink, reject-at-source. This is VIOLET paying down the +PINK duplicate-emission debt. +**Affected files (NEW):** `prod/clickhouse/violet/30_violet_economics.sql` (DDL FIRST), +`prod/clean_arch/violet/obs/economics_sink.py`, `prod/clean_arch/violet/obs/test_violet_economics_sink.py`. +**Interface/approach.** DDL: `dolphin_violet.violet_economics` (columns == `EconomicsRow` fields; +`ENGINE = ReplacingMergeTree` keyed on `(session_id, event_seq)` so a re-emit collapses to ONE row; +TTL per convention). Sink: `EconomicsSink(sink_fn, session_id)` mirrors `VioletDecisionJournal`: +validate each `EconomicsRow`, **dedupe by `(session_id, event_seq)`** (in-process guard + the +ReplacingMergeTree key as the durable guard), reject malformed to a counter (never head-of-line), +insert with `wait_for_async_insert=1` on the economics path. NO writes to any production table. +**Pass criteria.** Row-set == DDL columns (parity test, mirror the journal test); duplicate +`event_seq` → exactly ONE emitted (in-process) and ReplacingMergeTree key documented; malformed row +rejected to counter, sink never raises; provenance fields required. No edits outside the new files. + +## TASK 45 — Provenance + per-trade PnL feedback (capital = anchor + Σ trade-realized) +**Why.** Capital must be anchor + Σ TRADE-REALIZED deltas (never capital snapshots / never WS +balance — the shared-account foreign-fill immunity); every economics event carries +`pnl_source`/`capital_source`. +**Affected files (NEW):** `prod/clean_arch/violet/obs/provenance.py`, +`prod/clean_arch/violet/obs/test_violet_provenance.py`. (composes PASS-4 `EconomicsLedger` + +PASS-5/6 `PositionDelta`.) +**Interface/approach.** `stamp_provenance(delta: PositionDelta, *, pnl_source: str, capital_source: +str) -> (EconomicsRow, Provenance)` — derive `realized_pnl` from the TRADE-REALIZED delta (NEVER a +capital-difference), feed the ledger (anchor + Σ), tag provenance. A guard/test asserts the feedback +path uses trade-realized PnL, not capital deltas. +**Pass criteria.** capital_after == anchor + Σ realized deltas; provenance present on every row; a +test proves capital is NOT derived from a balance snapshot; non-finite rejected. No edits outside the 2 files. + +## TASK 46 — Decision/factor divergence monitor v2 +**Why.** Extend the V1 divergence monitor to the FACTOR plane (boost/beta/mc_scale/esof/ob/dc/ +posture VIOLET-vs-BLUE) so divergence is observable per factor, not just per feed. New module (do +NOT edit `divergence.py`). +**Affected files (NEW):** `prod/clickhouse/violet/31_violet_decision_divergence.sql` (DDL FIRST), +`prod/clean_arch/violet/obs/divergence_v2.py`, `prod/clean_arch/violet/obs/test_violet_divergence_v2.py`. +**Interface/approach.** `record_factor_divergence(violet_factors, blue_factors, *, asset, ts, +event_seq) -> list[DecisionDivergenceRow]` — per factor, abs_err; sink via a validate-then-sink to +`violet_decision_divergence`. Reuse the row-guard pattern. Read-only over inputs. +**Pass criteria.** Row-set == DDL columns; per-factor abs_err correct on a fixture; malformed +rejected. No edits to `divergence.py`. No edits outside the new files. + +## TASK 47 — Soak-readiness aggregator (one dashboard from all reports) +**Why.** Consolidate the scattered reports (PASS-2 parity, PASS-3 slippage, PASS-5/6 exec gate, +PASS-7 rank parity, PASS-8 cadence/regime, the latency + determinism gates) into ONE +soak-readiness view, so go/no-go is a single artifact. +**Affected files (NEW):** `prod/clean_arch/violet/obs/soak_readiness.py`, +`prod/clean_arch/violet/obs/test_violet_soak_readiness.py`, report → +`prod/VIOLET_dev/reports/violet_soak_readiness_.{json,md}`. +**Interface/approach.** Read the latest report file per category from `prod/VIOLET_dev/reports/` +(parse the JSONs), extract the key metric each (parity pick-match, latency p99, determinism pass, +mismatch counts, exactly-one-row, namespace isolation), assemble a `SoakReadiness` with a GO/NO_GO +verdict + the failing reasons. Pure aggregation; tolerate missing reports (mark UNKNOWN, not crash). +**Pass criteria.** On a synthetic set of report files the aggregator extracts the right metrics and +verdict (`--self-test`); a real run consolidates the actual reports; missing report → UNKNOWN not +crash. No edits outside the new files. + +## TASK 48 — Soak-readiness @gate (go/no-go prerequisites) +**Why.** A single authoritative gate: a DARK soak is GO only when parity ≥ threshold, latency gate +green, determinism green, exactly-one-row proven, namespace isolation proven (0 violet rows in +`dolphin`/`dolphin_pink`). +**Affected files (NEW):** `prod/clean_arch/violet/obs/test_violet_soak_readiness_gate.py`; report → +`prod/VIOLET_dev/reports/violet_soak_gate_.json`. +**Interface/approach.** Drive Task 47's aggregator; assert `verdict == "GO"` ONLY when all +prerequisites pass. **IMPORTANT:** given CRITICAL #1 (parity pick-match 1.5%), this gate is EXPECTED +to currently return NO_GO — the gate's job is to make that explicit and machine-checked, NOT to be +forced green. A test asserts the gate correctly returns NO_GO when parity is below threshold. +**Pass criteria (`@pytest.mark.gate`).** Gate returns GO iff all prerequisites green; returns NO_GO +with the parity reason when parity < threshold (the current real state); report archived. Do NOT +hardcode GO. No edits outside the new file. + +--- + +## Composition map +``` +contracts_v3 (+Provenance/EconomicsRow/DecisionDivergenceRow/SoakReadiness) + 44 economics_sink ── DDL-first, exactly-one-row (ReplacingMergeTree key), event_seq, async-insert + 45 provenance ── capital = anchor + Σ trade-realized; pnl_source/capital_source stamps + 46 divergence_v2 ── per-factor VIOLET-vs-BLUE abs_err rows (DDL-first) + 47 soak_readiness ── consolidate ALL reports → SoakReadiness(GO/NO_GO) + 48 soak_gate ── GO iff parity≥thr ∧ latency ∧ determinism ∧ one-row ∧ isolation + (currently NO_GO by design — CRITICAL #1) +``` +Integration (wiring the sinks into the live shadow loop, scheduling the aggregator) is the OWNER's +job. The soak GO decision is the operator's, informed by Task 48. + +## Recommended order +**43 (contracts) → 44 (economics sink, DDL-first) → 45 (provenance) → 46 (divergence v2) → 47 +(aggregator) → 48 (gate)**. + +## Still NOT in scope (operator/owner only, or Claude) +- **Starting the DARK soak** — operator decision (Task 48 informs it; currently NO_GO). +- **The live parity root-cause (CRITICAL #1)** — Claude; this pass only MEASURES + gates on it. +- **V4 live execution; HZ-bridge; VIBRISS.** +- Any edit to in-flight / shared files in §0 (especially `ch_writer.py` / production tables). diff --git a/prod/docs/VIOLET_TODO_CRITICAL.md b/prod/docs/VIOLET_TODO_CRITICAL.md index 558c9cf8..0b593aeb 100644 --- a/prod/docs/VIOLET_TODO_CRITICAL.md +++ b/prod/docs/VIOLET_TODO_CRITICAL.md @@ -82,6 +82,16 @@ no invented modulation), regime read-model, cadence shadow-actuation telemetry, parity gate. VIOLET CONSUMES posture/MARAS (MHS owns them); ACB/vol/SL-guard already covered elsewhere. Review when done. +**PASS 9**: `VIOLET_PART_SPEC_OA_TODO_PASS9.md` (issued 2026-06-17) — economics/observability +completeness: DDL-first exactly-one-row economics sink (PINK duplicate-emission fix), provenance + +capital=anchor+Σ-trade-realized, factor divergence v2, soak-readiness aggregator, and the +soak-readiness @gate (currently NO_GO by design — gated on CRITICAL #1). Review when done. + +> **OA BACKLOG COMPLETE (PASS 1–9 issued).** PASS 1–9 carve out the full parallelizable DARK +> surface of the V0→V6 ladder (~48 independent units, contracts_v3-keyed). What remains is NOT +> parallelizable and is Claude's: the parity root-cause (CRITICAL #1), review of all passes, +> integration, E2E, VBT re-cert, the real-key V4 boundary smoke, and live execution. + **Action:** review each pass for correctness, BLUE-algo compliance, V-TYPES, no-shared-edits, real (non-vacuous) tests. **Owner: Claude (me), later.**