VIOLET OA: partial spec PASS 9 — economics + observability completeness (OA backlog complete)
VIOLET_PART_SPEC_OA_TODO_PASS9.md — the final parallelizable pass, discharging VIOLET's binding V1
economics/observability inheritance. Six independent units extending contracts_v3 (Provenance/
EconomicsRow/DecisionDivergenceRow/SoakReadiness):
43. Observability/economics contracts.
44. DDL-first exactly-one-row-per-event economics sink (ReplacingMergeTree key, event_seq,
wait_for_async_insert=1, reject-at-source) — the PINK duplicate-emission fix.
45. Provenance + per-trade PnL feedback (capital = anchor + Σ trade-realized; never a balance
snapshot — shared-account foreign-fill immunity).
46. Factor-plane divergence monitor v2 (per-factor VIOLET-vs-BLUE abs_err, DDL-first).
47. Soak-readiness aggregator (consolidate all reports → SoakReadiness GO/NO_GO).
48. Soak-readiness @gate — GO iff parity≥thr ∧ latency ∧ determinism ∧ one-row ∧ isolation;
EXPECTED NO_GO today by design (CRITICAL #1), the gate makes it machine-checked (not forced green).
DDL-first + row-set==DDL parity + reject-at-source discipline throughout; never writes a production
table (new dolphin_violet tables only); ch_writer.py not edited.
OA BACKLOG COMPLETE: PASS 1–9 (~48 independent contracts_v3-keyed units) cover the full
parallelizable DARK surface of V0→V6. Remaining work is non-parallelizable + Claude's: parity
root-cause, review of all passes, integration, E2E, VBT re-cert, real-key V4 smoke, live execution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Codex
173
prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md
Normal file
173
prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS9.md
Normal file
@@ -0,0 +1,173 @@
|
||||
# VIOLET — partial spec for another agent, PASS 9 (OA TODO): ECONOMICS + OBSERVABILITY COMPLETENESS
|
||||
|
||||
Date: 2026-06-17. Final parallelizable pass (PASS 1–8 cover the rest of the DARK-buildable surface).
|
||||
PASS 9 closes the **economics/observability** inheritance the plan binds VIOLET to, and the
|
||||
**soak-readiness** machinery that decides go/no-go for a DARK soak. All DARK; independent units
|
||||
sharing `contracts_v3`.
|
||||
|
||||
**Plan obligations this pass discharges (V1 inheritance, BINDING):**
|
||||
> "VIOLET inherits and must keep: `pnl_source`/`capital_source` provenance, `event_seq` on every CH
|
||||
> row, `wait_for_async_insert=1` on economics tables, exactly-one-row-per-event (the PINK
|
||||
> duplicate-emission fix is open spec debt — VIOLET gates on it), DDL-before-code migration
|
||||
> discipline, per-trade sizer feedback from trade-realized PnL (never capital deltas)."
|
||||
Plus: capital = anchor + Σ deltas (never last-value) — the zombie-trade lesson (PASS-4 ledger).
|
||||
|
||||
**Read first (authoritative references — READ ONLY):**
|
||||
- `prod/clean_arch/violet/shadow_journal.py` — the validate-then-sink + reject-at-source pattern
|
||||
(the model for every PASS-9 sink). `prod/clean_arch/violet/domain.py` — V-TYPES (`DivergenceRow`,
|
||||
EpochMs/Seq/MonoNs etc.). `prod/clean_arch/violet/divergence.py` — the V1 divergence monitor.
|
||||
- `prod/clickhouse/violet/*.sql` + `apply_violet_ddl.py` — DDL-first migration discipline; the
|
||||
`test_apply_violet_ddl.py` pattern; the row-set==DDL-columns parity test in
|
||||
`test_violet_shadow_journal.py`.
|
||||
- `prod/ch_writer.py` — READ for the CH insert path / `wait_for_async_insert` convention; **do NOT
|
||||
edit it** (its design flaws are documented; VIOLET uses its own sink, mirroring the journal).
|
||||
- PASS-2 `parity_report.py`, PASS-3 `slippage_metric.py`, PASS-8 cadence telemetry — the reports the
|
||||
soak-readiness aggregator (Task 47) consolidates.
|
||||
- CH (read-only): `http://localhost:8123`, user `dolphin`/key `dolphin_ch_2026`; VIOLET db
|
||||
`dolphin_violet`. NEVER write/alter PRODUCTION tables; new VIOLET tables only, DDL-first.
|
||||
|
||||
---
|
||||
|
||||
## 0. HARD RULES (identical to PASS 1–8 — summarized)
|
||||
- **Never edit shared files** (`prod/nautilus_event_trader.py`, `clean_arch/dita_v2/**`,
|
||||
`dita/decision.py`, `nautilus_dolphin/**`, `blue_parity.py`, `prod/bingx/leverage.py`,
|
||||
`prod/ch_writer.py`). READ only.
|
||||
- **VIOLET DARK** — sinks write only to NEW `dolphin_violet` tables; no orders, no venue, no
|
||||
service/HZ control, no PROGREEN. Never write a production/BLUE table.
|
||||
- **V-TYPES on all new code**; faithful poison-guards; reject-at-source to a counter/dead_letter,
|
||||
never crash the sink (the bars_held=-106 spool lesson).
|
||||
- **DDL-FIRST**: every new table ships a `.sql` in `prod/clickhouse/violet/` BEFORE the sink code;
|
||||
a row-set==DDL-columns parity test is MANDATORY (mirror `test_violet_shadow_journal.py`).
|
||||
- **NEW-FILE-ONLY** under `prod/clean_arch/violet/obs/` (+ extend `contracts_v3.py` + new `.sql`).
|
||||
Do NOT modify in-flight files (`shadow_journal.py`, `divergence.py`, `domain.py`,
|
||||
`22_violet_decisions.sql`, the PASS-5..8 files). READ + IMPORT them.
|
||||
|
||||
## 0a. COMMIT / BRANCH POLICY (3 shared-index collisions on 2026-06-16 — non-negotiable)
|
||||
Own `git worktree` (`git worktree add ../vp-oa9 -b agent/oa-violet9`) strongly preferred. Else
|
||||
never `git add -A`; `git commit -F msg -- <files>` with explicit pathspec; verify
|
||||
`git show --stat --format="" HEAD` lists ONLY your files. One commit/task, prefix `VIOLET OA:`,
|
||||
Co-Authored-By trailer. Tests on `/home/dolphin/siloqy_env/bin/python3`. `git grep` only.
|
||||
|
||||
---
|
||||
|
||||
## I. SHARED INTERFACE EXTENSIONS (add to `contracts_v3.py`; never fork a parallel type)
|
||||
Reuse PASS-3..8 types (`CapitalState`, `PositionDelta`, `RegimeView`, etc.). ADD:
|
||||
1. **`Provenance`** — `pnl_source: str`, `capital_source: str`, `source: str`, `event_seq: Seq`,
|
||||
`ts: EpochMs`. The provenance stamp every economics row carries.
|
||||
2. **`EconomicsRow`** — a validated `dolphin_violet.violet_economics` row: `session_id: SessionId`,
|
||||
`event_seq: Seq`, `asset: Symbol`, `realized_pnl: float (finite)`, `fee: float (ge=0)`,
|
||||
`capital_after: float (ge=0)`, `pnl_source: str`, `capital_source: str`, `ts: EpochMs`,
|
||||
`mono_ns: MonoNs`. Field set MUST equal its DDL columns (parity test).
|
||||
3. **`DecisionDivergenceRow`** — extends the V1 divergence idea to the FACTOR plane: `ts: EpochMs`,
|
||||
`asset: Symbol`, `factor: str`, `violet_value: float`, `blue_value: float`, `abs_err: float`,
|
||||
`event_seq: Seq`.
|
||||
4. **`SoakReadiness`** — `parity_pick_match: float`, `latency_p99_ms: float`,
|
||||
`determinism_ok: bool`, `exactly_one_row_ok: bool`, `namespace_isolation_ok: bool`,
|
||||
`verdict: str` ("GO"/"NO_GO"), `reasons: list[str]`.
|
||||
|
||||
If a task needs another field, ADD it here and note it.
|
||||
|
||||
---
|
||||
|
||||
## TASK 43 — Observability/economics contracts
|
||||
**Why.** The shared provenance/economics/divergence vocabulary.
|
||||
**Affected files (NEW):** extend `contracts_v3.py`; `prod/clean_arch/violet/obs/__init__.py`;
|
||||
`prod/clean_arch/violet/obs/test_violet_obs_contracts.py`.
|
||||
**Pass criteria.** All types construct + poison-reject (non-finite pnl, negative capital/fee
|
||||
rejected); provenance strings non-empty where required. No edits outside the new/extended files.
|
||||
|
||||
## TASK 44 — DDL-first exactly-one-row-per-event economics sink [the PINK duplicate-emission fix]
|
||||
**Why.** The binding inheritance: exactly-one-row-per-event, `event_seq` on every row,
|
||||
`wait_for_async_insert=1`, validate-then-sink, reject-at-source. This is VIOLET paying down the
|
||||
PINK duplicate-emission debt.
|
||||
**Affected files (NEW):** `prod/clickhouse/violet/30_violet_economics.sql` (DDL FIRST),
|
||||
`prod/clean_arch/violet/obs/economics_sink.py`, `prod/clean_arch/violet/obs/test_violet_economics_sink.py`.
|
||||
**Interface/approach.** DDL: `dolphin_violet.violet_economics` (columns == `EconomicsRow` fields;
|
||||
`ENGINE = ReplacingMergeTree` keyed on `(session_id, event_seq)` so a re-emit collapses to ONE row;
|
||||
TTL per convention). Sink: `EconomicsSink(sink_fn, session_id)` mirrors `VioletDecisionJournal`:
|
||||
validate each `EconomicsRow`, **dedupe by `(session_id, event_seq)`** (in-process guard + the
|
||||
ReplacingMergeTree key as the durable guard), reject malformed to a counter (never head-of-line),
|
||||
insert with `wait_for_async_insert=1` on the economics path. NO writes to any production table.
|
||||
**Pass criteria.** Row-set == DDL columns (parity test, mirror the journal test); duplicate
|
||||
`event_seq` → exactly ONE emitted (in-process) and ReplacingMergeTree key documented; malformed row
|
||||
rejected to counter, sink never raises; provenance fields required. No edits outside the new files.
|
||||
|
||||
## TASK 45 — Provenance + per-trade PnL feedback (capital = anchor + Σ trade-realized)
|
||||
**Why.** Capital must be anchor + Σ TRADE-REALIZED deltas (never capital snapshots / never WS
|
||||
balance — the shared-account foreign-fill immunity); every economics event carries
|
||||
`pnl_source`/`capital_source`.
|
||||
**Affected files (NEW):** `prod/clean_arch/violet/obs/provenance.py`,
|
||||
`prod/clean_arch/violet/obs/test_violet_provenance.py`. (composes PASS-4 `EconomicsLedger` +
|
||||
PASS-5/6 `PositionDelta`.)
|
||||
**Interface/approach.** `stamp_provenance(delta: PositionDelta, *, pnl_source: str, capital_source:
|
||||
str) -> (EconomicsRow, Provenance)` — derive `realized_pnl` from the TRADE-REALIZED delta (NEVER a
|
||||
capital-difference), feed the ledger (anchor + Σ), tag provenance. A guard/test asserts the feedback
|
||||
path uses trade-realized PnL, not capital deltas.
|
||||
**Pass criteria.** capital_after == anchor + Σ realized deltas; provenance present on every row; a
|
||||
test proves capital is NOT derived from a balance snapshot; non-finite rejected. No edits outside the 2 files.
|
||||
|
||||
## TASK 46 — Decision/factor divergence monitor v2
|
||||
**Why.** Extend the V1 divergence monitor to the FACTOR plane (boost/beta/mc_scale/esof/ob/dc/
|
||||
posture VIOLET-vs-BLUE) so divergence is observable per factor, not just per feed. New module (do
|
||||
NOT edit `divergence.py`).
|
||||
**Affected files (NEW):** `prod/clickhouse/violet/31_violet_decision_divergence.sql` (DDL FIRST),
|
||||
`prod/clean_arch/violet/obs/divergence_v2.py`, `prod/clean_arch/violet/obs/test_violet_divergence_v2.py`.
|
||||
**Interface/approach.** `record_factor_divergence(violet_factors, blue_factors, *, asset, ts,
|
||||
event_seq) -> list[DecisionDivergenceRow]` — per factor, abs_err; sink via a validate-then-sink to
|
||||
`violet_decision_divergence`. Reuse the row-guard pattern. Read-only over inputs.
|
||||
**Pass criteria.** Row-set == DDL columns; per-factor abs_err correct on a fixture; malformed
|
||||
rejected. No edits to `divergence.py`. No edits outside the new files.
|
||||
|
||||
## TASK 47 — Soak-readiness aggregator (one dashboard from all reports)
|
||||
**Why.** Consolidate the scattered reports (PASS-2 parity, PASS-3 slippage, PASS-5/6 exec gate,
|
||||
PASS-7 rank parity, PASS-8 cadence/regime, the latency + determinism gates) into ONE
|
||||
soak-readiness view, so go/no-go is a single artifact.
|
||||
**Affected files (NEW):** `prod/clean_arch/violet/obs/soak_readiness.py`,
|
||||
`prod/clean_arch/violet/obs/test_violet_soak_readiness.py`, report →
|
||||
`prod/VIOLET_dev/reports/violet_soak_readiness_<UTC>.{json,md}`.
|
||||
**Interface/approach.** Read the latest report file per category from `prod/VIOLET_dev/reports/`
|
||||
(parse the JSONs), extract the key metric each (parity pick-match, latency p99, determinism pass,
|
||||
mismatch counts, exactly-one-row, namespace isolation), assemble a `SoakReadiness` with a GO/NO_GO
|
||||
verdict + the failing reasons. Pure aggregation; tolerate missing reports (mark UNKNOWN, not crash).
|
||||
**Pass criteria.** On a synthetic set of report files the aggregator extracts the right metrics and
|
||||
verdict (`--self-test`); a real run consolidates the actual reports; missing report → UNKNOWN not
|
||||
crash. No edits outside the new files.
|
||||
|
||||
## TASK 48 — Soak-readiness @gate (go/no-go prerequisites)
|
||||
**Why.** A single authoritative gate: a DARK soak is GO only when parity ≥ threshold, latency gate
|
||||
green, determinism green, exactly-one-row proven, namespace isolation proven (0 violet rows in
|
||||
`dolphin`/`dolphin_pink`).
|
||||
**Affected files (NEW):** `prod/clean_arch/violet/obs/test_violet_soak_readiness_gate.py`; report →
|
||||
`prod/VIOLET_dev/reports/violet_soak_gate_<UTC>.json`.
|
||||
**Interface/approach.** Drive Task 47's aggregator; assert `verdict == "GO"` ONLY when all
|
||||
prerequisites pass. **IMPORTANT:** given CRITICAL #1 (parity pick-match 1.5%), this gate is EXPECTED
|
||||
to currently return NO_GO — the gate's job is to make that explicit and machine-checked, NOT to be
|
||||
forced green. A test asserts the gate correctly returns NO_GO when parity is below threshold.
|
||||
**Pass criteria (`@pytest.mark.gate`).** Gate returns GO iff all prerequisites green; returns NO_GO
|
||||
with the parity reason when parity < threshold (the current real state); report archived. Do NOT
|
||||
hardcode GO. No edits outside the new file.
|
||||
|
||||
---
|
||||
|
||||
## Composition map
|
||||
```
|
||||
contracts_v3 (+Provenance/EconomicsRow/DecisionDivergenceRow/SoakReadiness)
|
||||
44 economics_sink ── DDL-first, exactly-one-row (ReplacingMergeTree key), event_seq, async-insert
|
||||
45 provenance ── capital = anchor + Σ trade-realized; pnl_source/capital_source stamps
|
||||
46 divergence_v2 ── per-factor VIOLET-vs-BLUE abs_err rows (DDL-first)
|
||||
47 soak_readiness ── consolidate ALL reports → SoakReadiness(GO/NO_GO)
|
||||
48 soak_gate ── GO iff parity≥thr ∧ latency ∧ determinism ∧ one-row ∧ isolation
|
||||
(currently NO_GO by design — CRITICAL #1)
|
||||
```
|
||||
Integration (wiring the sinks into the live shadow loop, scheduling the aggregator) is the OWNER's
|
||||
job. The soak GO decision is the operator's, informed by Task 48.
|
||||
|
||||
## Recommended order
|
||||
**43 (contracts) → 44 (economics sink, DDL-first) → 45 (provenance) → 46 (divergence v2) → 47
|
||||
(aggregator) → 48 (gate)**.
|
||||
|
||||
## Still NOT in scope (operator/owner only, or Claude)
|
||||
- **Starting the DARK soak** — operator decision (Task 48 informs it; currently NO_GO).
|
||||
- **The live parity root-cause (CRITICAL #1)** — Claude; this pass only MEASURES + gates on it.
|
||||
- **V4 live execution; HZ-bridge; VIBRISS.**
|
||||
- Any edit to in-flight / shared files in §0 (especially `ch_writer.py` / production tables).
|
||||
Reference in New Issue
Block a user