siloqy/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md

VIOLET — partial spec for another agent, PASS 9 (OA TODO): ECONOMICS + OBSERVABILITY COMPLETENESS

Date: 2026-06-17. Final parallelizable pass (PASS 1–8 cover the rest of the DARK-buildable surface). PASS 9 closes the economics/observability inheritance the plan binds VIOLET to, and the soak-readiness machinery that decides go/no-go for a DARK soak. All DARK; independent units sharing contracts_v3.

Plan obligations this pass discharges (V1 inheritance, BINDING):

"VIOLET inherits and must keep: pnl_source/capital_source provenance, event_seq on every CH row, wait_for_async_insert=1 on economics tables, exactly-one-row-per-event (the PINK duplicate-emission fix is open spec debt — VIOLET gates on it), DDL-before-code migration discipline, per-trade sizer feedback from trade-realized PnL (never capital deltas)." Plus: capital = anchor + Σ deltas (never last-value) — the zombie-trade lesson (PASS-4 ledger).

Read first (authoritative references — READ ONLY):

• prod/clean_arch/violet/shadow_journal.py — the validate-then-sink + reject-at-source pattern (the model for every PASS-9 sink). prod/clean_arch/violet/domain.py — V-TYPES (DivergenceRow, EpochMs/Seq/MonoNs etc.). prod/clean_arch/violet/divergence.py — the V1 divergence monitor.
• prod/clickhouse/violet/*.sql + apply_violet_ddl.py — DDL-first migration discipline; the test_apply_violet_ddl.py pattern; the row-set==DDL-columns parity test in test_violet_shadow_journal.py.
• prod/ch_writer.py — READ for the CH insert path / wait_for_async_insert convention; do NOT edit it (its design flaws are documented; VIOLET uses its own sink, mirroring the journal).
• PASS-2 parity_report.py, PASS-3 slippage_metric.py, PASS-8 cadence telemetry — the reports the soak-readiness aggregator (Task 47) consolidates.
• CH (read-only): http://localhost:8123, user dolphin/key dolphin_ch_2026; VIOLET db dolphin_violet. NEVER write/alter PRODUCTION tables; new VIOLET tables only, DDL-first.

---

0. HARD RULES (identical to PASS 1–8 — summarized)

• Never edit shared files (prod/nautilus_event_trader.py, clean_arch/dita_v2/**, dita/decision.py, nautilus_dolphin/**, blue_parity.py, prod/bingx/leverage.py, prod/ch_writer.py). READ only.
• VIOLET DARK — sinks write only to NEW dolphin_violet tables; no orders, no venue, no service/HZ control, no PROGREEN. Never write a production/BLUE table.
• V-TYPES on all new code; faithful poison-guards; reject-at-source to a counter/dead_letter, never crash the sink (the bars_held=-106 spool lesson).
• DDL-FIRST: every new table ships a .sql in prod/clickhouse/violet/ BEFORE the sink code; a row-set==DDL-columns parity test is MANDATORY (mirror test_violet_shadow_journal.py).
• NEW-FILE-ONLY under prod/clean_arch/violet/obs/ (+ extend contracts_v3.py + new .sql). Do NOT modify in-flight files (shadow_journal.py, divergence.py, domain.py, 22_violet_decisions.sql, the PASS-5..8 files). READ + IMPORT them.

0a. COMMIT / BRANCH POLICY (3 shared-index collisions on 2026-06-16 — non-negotiable)

Own git worktree (git worktree add ../vp-oa9 -b agent/oa-violet9) strongly preferred. Else never git add -A; git commit -F msg -- <files> with explicit pathspec; verify git show --stat --format="" HEAD lists ONLY your files. One commit/task, prefix VIOLET OA:, Co-Authored-By trailer. Tests on /home/dolphin/siloqy_env/bin/python3. git grep only.

---

I. SHARED INTERFACE EXTENSIONS (add to contracts_v3.py; never fork a parallel type)

Reuse PASS-3..8 types (CapitalState, PositionDelta, RegimeView, etc.). ADD:

1. Provenance — pnl_source: str, capital_source: str, source: str, event_seq: Seq, ts: EpochMs. The provenance stamp every economics row carries.
2. EconomicsRow — a validated dolphin_violet.violet_economics row: session_id: SessionId, event_seq: Seq, asset: Symbol, realized_pnl: float (finite), fee: float (ge=0), capital_after: float (ge=0), pnl_source: str, capital_source: str, ts: EpochMs, mono_ns: MonoNs. Field set MUST equal its DDL columns (parity test).
3. DecisionDivergenceRow — extends the V1 divergence idea to the FACTOR plane: ts: EpochMs, asset: Symbol, factor: str, violet_value: float, blue_value: float, abs_err: float, event_seq: Seq.
4. SoakReadiness — parity_pick_match: float, latency_p99_ms: float, determinism_ok: bool, exactly_one_row_ok: bool, namespace_isolation_ok: bool, verdict: str ("GO"/"NO_GO"), reasons: list[str].

If a task needs another field, ADD it here and note it.

---

TASK 43 — Observability/economics contracts

Why. The shared provenance/economics/divergence vocabulary. Affected files (NEW): extend contracts_v3.py; prod/clean_arch/violet/obs/__init__.py; prod/clean_arch/violet/obs/test_violet_obs_contracts.py. Pass criteria. All types construct + poison-reject (non-finite pnl, negative capital/fee rejected); provenance strings non-empty where required. No edits outside the new/extended files.

TASK 44 — DDL-first exactly-one-row-per-event economics sink [the PINK duplicate-emission fix]

Why. The binding inheritance: exactly-one-row-per-event, event_seq on every row, wait_for_async_insert=1, validate-then-sink, reject-at-source. This is VIOLET paying down the PINK duplicate-emission debt. Affected files (NEW): prod/clickhouse/violet/30_violet_economics.sql (DDL FIRST), prod/clean_arch/violet/obs/economics_sink.py, prod/clean_arch/violet/obs/test_violet_economics_sink.py. Interface/approach. DDL: dolphin_violet.violet_economics (columns == EconomicsRow fields; ENGINE = ReplacingMergeTree keyed on (session_id, event_seq) so a re-emit collapses to ONE row; TTL per convention). Sink: EconomicsSink(sink_fn, session_id) mirrors VioletDecisionJournal: validate each EconomicsRow, dedupe by (session_id, event_seq) (in-process guard + the ReplacingMergeTree key as the durable guard), reject malformed to a counter (never head-of-line), insert with wait_for_async_insert=1 on the economics path. NO writes to any production table. Pass criteria. Row-set == DDL columns (parity test, mirror the journal test); duplicate event_seq → exactly ONE emitted (in-process) and ReplacingMergeTree key documented; malformed row rejected to counter, sink never raises; provenance fields required. No edits outside the new files.

TASK 45 — Provenance + per-trade PnL feedback (capital = anchor + Σ trade-realized)

Why. Capital must be anchor + Σ TRADE-REALIZED deltas (never capital snapshots / never WS balance — the shared-account foreign-fill immunity); every economics event carries pnl_source/capital_source. Affected files (NEW): prod/clean_arch/violet/obs/provenance.py, prod/clean_arch/violet/obs/test_violet_provenance.py. (composes PASS-4 EconomicsLedger + PASS-5/6 PositionDelta.) Interface/approach. stamp_provenance(delta: PositionDelta, *, pnl_source: str, capital_source: str) -> (EconomicsRow, Provenance) — derive realized_pnl from the TRADE-REALIZED delta (NEVER a capital-difference), feed the ledger (anchor + Σ), tag provenance. A guard/test asserts the feedback path uses trade-realized PnL, not capital deltas. Pass criteria. capital_after == anchor + Σ realized deltas; provenance present on every row; a test proves capital is NOT derived from a balance snapshot; non-finite rejected. No edits outside the 2 files.

TASK 46 — Decision/factor divergence monitor v2

Why. Extend the V1 divergence monitor to the FACTOR plane (boost/beta/mc_scale/esof/ob/dc/ posture VIOLET-vs-BLUE) so divergence is observable per factor, not just per feed. New module (do NOT edit divergence.py). Affected files (NEW): prod/clickhouse/violet/31_violet_decision_divergence.sql (DDL FIRST), prod/clean_arch/violet/obs/divergence_v2.py, prod/clean_arch/violet/obs/test_violet_divergence_v2.py. Interface/approach. record_factor_divergence(violet_factors, blue_factors, *, asset, ts, event_seq) -> list[DecisionDivergenceRow] — per factor, abs_err; sink via a validate-then-sink to violet_decision_divergence. Reuse the row-guard pattern. Read-only over inputs. Pass criteria. Row-set == DDL columns; per-factor abs_err correct on a fixture; malformed rejected. No edits to divergence.py. No edits outside the new files.

TASK 47 — Soak-readiness aggregator (one dashboard from all reports)

Why. Consolidate the scattered reports (PASS-2 parity, PASS-3 slippage, PASS-5/6 exec gate, PASS-7 rank parity, PASS-8 cadence/regime, the latency + determinism gates) into ONE soak-readiness view, so go/no-go is a single artifact. Affected files (NEW): prod/clean_arch/violet/obs/soak_readiness.py, prod/clean_arch/violet/obs/test_violet_soak_readiness.py, report → prod/VIOLET_dev/reports/violet_soak_readiness_<UTC>.{json,md}. Interface/approach. Read the latest report file per category from prod/VIOLET_dev/reports/ (parse the JSONs), extract the key metric each (parity pick-match, latency p99, determinism pass, mismatch counts, exactly-one-row, namespace isolation), assemble a SoakReadiness with a GO/NO_GO verdict + the failing reasons. Pure aggregation; tolerate missing reports (mark UNKNOWN, not crash). Pass criteria. On a synthetic set of report files the aggregator extracts the right metrics and verdict (--self-test); a real run consolidates the actual reports; missing report → UNKNOWN not crash. No edits outside the new files.

TASK 48 — Soak-readiness @gate (go/no-go prerequisites)

Why. A single authoritative gate: a DARK soak is GO only when parity ≥ threshold, latency gate green, determinism green, exactly-one-row proven, namespace isolation proven (0 violet rows in dolphin/dolphin_pink). Affected files (NEW): prod/clean_arch/violet/obs/test_violet_soak_readiness_gate.py; report → prod/VIOLET_dev/reports/violet_soak_gate_<UTC>.json. Interface/approach. Drive Task 47's aggregator; assert verdict == "GO" ONLY when all prerequisites pass. IMPORTANT: given CRITICAL #1 (parity pick-match 1.5%), this gate is EXPECTED to currently return NO_GO — the gate's job is to make that explicit and machine-checked, NOT to be forced green. A test asserts the gate correctly returns NO_GO when parity is below threshold. Pass criteria (@pytest.mark.gate). Gate returns GO iff all prerequisites green; returns NO_GO with the parity reason when parity < threshold (the current real state); report archived. Do NOT hardcode GO. No edits outside the new file.

---

Composition map

contracts_v3 (+Provenance/EconomicsRow/DecisionDivergenceRow/SoakReadiness)
   44 economics_sink  ── DDL-first, exactly-one-row (ReplacingMergeTree key), event_seq, async-insert
   45 provenance      ── capital = anchor + Σ trade-realized; pnl_source/capital_source stamps
   46 divergence_v2   ── per-factor VIOLET-vs-BLUE abs_err rows (DDL-first)
   47 soak_readiness  ── consolidate ALL reports → SoakReadiness(GO/NO_GO)
   48 soak_gate       ── GO iff parity≥thr ∧ latency ∧ determinism ∧ one-row ∧ isolation
                          (currently NO_GO by design — CRITICAL #1)

Integration (wiring the sinks into the live shadow loop, scheduling the aggregator) is the OWNER's job. The soak GO decision is the operator's, informed by Task 48.

Recommended order

43 (contracts) → 44 (economics sink, DDL-first) → 45 (provenance) → 46 (divergence v2) → 47 (aggregator) → 48 (gate).

Still NOT in scope (operator/owner only, or Claude)

• Starting the DARK soak — operator decision (Task 48 informs it; currently NO_GO).
• The live parity root-cause (CRITICAL #1) — Claude; this pass only MEASURES + gates on it.
• V4 live execution; HZ-bridge; VIBRISS.
• Any edit to in-flight / shared files in §0 (especially ch_writer.py / production tables).