diff --git a/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md b/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md new file mode 100644 index 00000000..f392fb4f --- /dev/null +++ b/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md @@ -0,0 +1,212 @@ +# VIOLET — partial spec for another agent, PASS 3 (OA TODO) + +Date: 2026-06-17. Follows PASS 1 + PASS 2 (both reported done — review pending, see +`VIOLET_TODO_CRITICAL.md`). PASS 3 is drawn STRICTLY from the existing VIOLET dev plan +(`VIOLET_DEV_SPEC_AND_PLAN.md`, the V0→V6 ladder + its named deferred items) — nothing invented. +Every task is an INDEPENDENT UNIT: it can be built, tested and function on its own, and it +composes with the others ONLY through the explicit shared interfaces in §I. + +--- + +## 0. HARD RULES (identical to PASS 1/2 — re-read those; summarized) + +- **Never edit shared files** (`prod/nautilus_event_trader.py`, `clean_arch/dita_v2/**`, + `dita/decision.py`, `nautilus_dolphin/**`, `blue_parity.py`, `prod/bingx/leverage.py`). READ only. +- **VIOLET stays DARK** — no orders, no execution, no VST keys, no service start/stop, no HZ + restart, no PROGREEN. +- **V-TYPES on all new code** (`StrictModel` / `Annotated[…Field]` / `@typed`); faithful + poison-guards only. +- **NEW-FILE-ONLY**; do NOT modify in-flight files: `live_blue_source.py`, + `shadow_live_factors.py`, `live_factor_source.py`, `live_factors.py`, `decision_engine.py`, + `sizing.py`, `shadow_journal.py`, `cadence.py`, `clock.py`, `divergence.py`, + `22_violet_decisions.sql`. READ + IMPORT them. + +## 0a. COMMIT / BRANCH POLICY (3 shared-index collisions on 2026-06-16 — take this seriously) +Own `git worktree` (`git worktree add ../vp-oa3 -b agent/oa-violet3`) STRONGLY preferred. Else +never `git add -A`; commit with explicit pathspec `git commit -F msg -- ` and verify +`git show --stat --format="" HEAD` lists ONLY your files. One commit per task, prefix +`VIOLET OA:`, Co-Authored-By trailer. Tests on `/home/dolphin/siloqy_env/bin/python3`. Use +`git grep` (recursive grep/find time out on CIFS). + +--- + +## I. SHARED INTERFACES (agreed contracts — define these EXACTLY; the tasks depend on them) + +These are the only coupling points between PASS-3 units. Put the type definitions in ONE new +module `prod/clean_arch/violet/contracts_v3.py` (Task 8 creates it; later tasks import it). +All are `StrictModel` / `Annotated` V-TYPES. + +1. **`VenueTick`** — one normalized venue quote (exchange-agnostic): + `asset: Symbol`, `bid: Px`, `ask: Px`, `mark: Px`, `last: Px`, `mono_ns: MonoNs`, + `venue: str` (e.g. "BINGX","BINANCE"). All prices > 0 finite. This is the SAME normalized + tick named in the plan's "Venue price feed port (CRITICAL gap)". +2. **`OpenPositionView`** — the minimal open-position state an exit/guard needs: + `asset: Symbol`, `side: str` ("SHORT"/"LONG"), `entry_price: Px`, `qty: Qty`, + `entry_ts_ns: MonoNs`, `bars_held: BarsHeld`, `tp_threshold: float` (fixed TP pct), + `sl_threshold: float` (stop pct), `leverage: float`. +3. **`ExitDecision`** — `action: str` ("HOLD"/"EXIT"), `reason: str` + ("FIXED_TP"/"CATASTROPHIC_SL"/"ADVSL"/"NONE"/…), `price: Px`, `priority: int` + (CATASTROPHIC/ADVSL=0 > FIXED_TP=1 > DISCRETIONARY=2 — the plan's mandated order). + +If a later task needs a field not here, ADD it to `contracts_v3.py` (and note it), never fork a +parallel type. + +--- + +## TASK 8 — VenuePriceFeedPort + BingX adapter (DARK, data-only) [plan: "Venue price feed port (CRITICAL gap)"] + +**Why.** The plan flags this as CRITICAL: OBF is Binance-wired (alpha side), but exits fill on +the EXECUTION venue, and FET showed scan-vs-BingX divergence (0.2176 vs 0.1878 = 15%). VIOLET +needs an exchange-agnostic price port. This unit delivers the port interface + a BingX adapter, +data-only (no orders). + +**Affected files (NEW only):** `prod/clean_arch/violet/contracts_v3.py` (the §I types), +`prod/clean_arch/violet/venue_price_feed.py`, `prod/clean_arch/violet/test_violet_venue_price_feed.py`. + +**Interface/approach.** Define `VenuePriceFeedPort` (ABC): `latest(asset) -> Optional[VenueTick]`, +`subscribe(assets)`, `close()`. Implement `BingxSwapPriceFeed(VenuePriceFeedPort)` consuming the +BingX swap WS bookTicker/markPrice (READ the existing BingX WS conventions in the repo; data-only, +no auth needed for public streams — confirm). Implement `MockPriceFeed` (deterministic, for +tests + for the other tasks). All ticks validated into `VenueTick` at ingress (poison-reject +non-finite/≤0). + +**Tests / pass criteria.** Mock feed round-trips ticks as `VenueTick`; poison ticks rejected; +the BingX adapter parses a recorded/sample bookTicker frame into a correct `VenueTick` (use a +captured frame fixture, NOT a live connection in the unit test). DONE when: `VenueTick` contract +finalized, mock + adapter parse-tested, no live WS in unit tests, no edits outside the 3 files. + +--- + +## TASK 9 — MechanicalExitGuard (armed TP/SL) [plan: LINK TP-miss structural fix] + +**Why.** The plan's structural fix for the LINKUSDT −$1,248 TP-miss: TP/SL are MECHANICAL exits +owned by an exit guard with armed price thresholds set at entry; policy layers (V7/MARAS) can +only contribute DISCRETIONARY exits and can NEVER mask mechanical ones. Total priority: +CATASTROPHIC/ADVSL > fixed TP > discretionary. + +**Affected files (NEW only):** `prod/clean_arch/violet/mechanical_exit_guard.py`, +`prod/clean_arch/violet/test_violet_mechanical_exit_guard.py`. (imports `contracts_v3`.) + +**Interface/approach.** `MechanicalExitGuard.evaluate(pos: OpenPositionView, tick: VenueTick) -> +ExitDecision`. Pure function of position + current venue price: compute pnl_pct from +entry/side/price; if it reaches the armed `tp_threshold` → `ExitDecision(EXIT, FIXED_TP, +priority=1)`; if it breaches `sl_threshold` → `ExitDecision(EXIT, CATASTROPHIC_SL, priority=0)`; +else HOLD. Prices off the VENUE tick (not scan). No discretionary logic here — this layer is the +mechanical floor only. + +**Tests / pass criteria.** SHORT + LONG: TP fires exactly at threshold, SL at stop, neither +fires inside the band; priority ordering correct (SL outranks TP if both somehow true). Hypothesis +property: output always finite, priority ∈ {0,1,2}, EXIT only when threshold crossed. DONE when: +deterministic threshold tests + property test pass, no edits outside the 2 files. + +--- + +## TASK 10 — Sub-second catastrophic-SL / ADVSL floor guard [plan: versioned SAFETY DEVIATION] + +**Why.** The plan's one sanctioned sub-second behaviour: a catastrophic-SL/ADVSL floor evaluated +at the fastest cadence against a fast price source, to fix the scan-dark unmanaged-position hazard +(XLM/FET class). "Evaluate at fastest cadence (shadow-log would-be actions), actuate at Q." + +**Affected files (NEW only):** `prod/clean_arch/violet/sl_floor_guard.py`, +`prod/clean_arch/violet/test_violet_sl_floor_guard.py`. (imports `contracts_v3`, composes Task 9.) + +**Interface/approach.** `SLFloorGuard(deadline_ns)` with `on_tick(pos, tick) -> Optional[ExitDecision]` +that EVALUATES every tick (sub-second) and SHADOW-LOGS would-be SL exits, but only RETURNS an +actuation when the configured fast-SL condition holds (catastrophic threshold worse than the +mechanical stop, OR ADVSL trailing breach). Track evaluate-count vs actuate-count (shadow delta). +DARK: returns the decision; never sends orders. + +**Tests / pass criteria.** A tick stream that dips intra-scan triggers the fast SL evaluation and +records the would-be exit; the actuate gate fires only on the catastrophic/ADVSL condition; +evaluate-count > actuate-count proven. DONE when tests pass, no edits outside the 2 files. + +--- + +## TASK 11 — Event-sourced restore from trade_reconstruction [plan: VIOLET restore commitment, Option C] + +**Why.** The plan commits VIOLET to NEVER restore from position_state snapshots; instead replay +the chain-tokened `trade_reconstruction` OPEN/PARTIAL_EXIT/CLOSE journal. Open = ROOT OPEN with no +terminal CLOSE; size = entry − Σ legs; holding from ABSOLUTE entry_ts (kills the bars_held≈0 / +MAX_HOLD-reset class by construction); chain token verified. + +**Affected files (NEW only):** `prod/clean_arch/violet/event_restore.py`, +`prod/clean_arch/violet/test_violet_event_restore.py`. (imports `contracts_v3` → emits +`OpenPositionView`s.) + +**Interface/approach.** `restore_open_positions(journal_rows: list[dict]) -> list[OpenPositionView]` +— pure function over journal rows (asset, chain_root_trade_id, chain_token, leg type, qty, +realized legs, entry_ts, entry_price). Reconstruct each chain; an open position is a ROOT OPEN +with no matching terminal CLOSE; qty = entry − Σ partial-exit legs; bars_held derived from absolute +entry_ts vs now (NEVER from a stored counter). Reject chain-token mismatches to a quarantine list +(returned alongside), never crash. + +**Tests / pass criteria.** Fixtures: clean open, fully-closed (→ not restored), partial-exit chain +(qty correct), chain-token mismatch (→ quarantined), dead-session entry_ts (bars_held computed +from ts, never negative). Hypothesis: qty ≥ 0, bars_held ≥ 0 always. DONE when tests pass, no +edits outside the 2 files. + +--- + +## TASK 12 — Venue lead/lag slippage metric [plan: V1 metrics — venue lead/lag] + +**Why.** The plan's continuous signed SHORT-entry slippage metric: venue fill vs Binance mid at +decision time — answers "is BingX discounting our signal". Rerunnable analysis. + +**Affected files (NEW only):** `prod/clean_arch/violet/slippage_metric.py`, +`prod/clean_arch/violet/test_violet_slippage_metric.py`, report → +`prod/VIOLET_dev/reports/violet_slippage_.json`. + +**Interface/approach.** `signed_entry_slippage(decision_mid: float, venue_fill: float, side: str) +-> float` (signed bps; positive = adverse) + an aggregator over recorded rows producing a +distribution (mean/median/p95). Read-only over recorded data (CH `dolphin`/`dolphin_violet`, no +writes). Pure functions; V-TYPES the rows. + +**Tests / pass criteria.** Sign convention exact for SHORT and LONG on hand-built cases; aggregator +stats correct on a synthetic fixture (`--self-test`). DONE when self-test passes + a report can be +produced on the prod host, no production writes, no edits outside the files. + +--- + +## TASK 13 — Cadence per-action Q schedule + evaluate/actuate telemetry [plan: cadence quantizer] + +**Why.** The plan's cadence quantizer: each action (SL/TP/ENTRY/OBF/ExoF) gets its own +quantization Q; evaluate at fastest cadence (shadow evidence), actuate at Q; step Q down later. +`cadence.py` (the `CadenceControlPlane`) exists; this unit adds a per-action Q-SCHEDULE LOADER + +a shadow-delta TELEMETRY recorder WITHOUT modifying `cadence.py`. + +**Affected files (NEW only):** `prod/clean_arch/violet/cadence_schedule.py`, +`prod/clean_arch/violet/test_violet_cadence_schedule.py`. + +**Interface/approach.** Read `cadence.py` for `Action` + `CadenceControlPlane` API. Provide +`load_q_schedule(mapping) -> dict[Action, int_ns]` (validated; the plan's initial table: SL tight, +TP=scan, ENTRY=scan, OBF~1s) and `CadenceTelemetry` that records per-action evaluate-count vs +actuate-count and emits a shadow-delta summary. Compose with `CadenceControlPlane` by wrapping its +`due()` calls (do not edit it). + +**Tests / pass criteria.** Schedule loader validates/rejects bad Q values; telemetry counts +evaluate > actuate under a synthetic action stream; per-action Q honored. DONE when tests pass, no +edits outside the 2 files. + +--- + +## Composition map (how the units fit — the "well-known interfaces") +``` +Task 8 contracts_v3 (VenueTick, OpenPositionView, ExitDecision) ← the shared vocabulary + │ + ├── Task 8 venue_price_feed → VenueTick stream + ├── Task 9 mechanical_exit_guard(pos, tick) → ExitDecision (consumes VenueTick) + ├── Task 10 sl_floor_guard(pos, tick) → ExitDecision (composes Task 9) + ├── Task 11 event_restore(journal) → [OpenPositionView] (feeds 9/10) + ├── Task 12 slippage_metric(mid, fill, side)→ bps (uses VenueTick mids) + └── Task 13 cadence_schedule/telemetry → per-action Q + deltas +``` +Integration (wiring these into the live reactor + decision/exit path) is the OWNER's job later, +NOT part of these units. Each ships standalone + tested. + +## Recommended order +**8 (contracts + feed) → 9 (mechanical exit) → 11 (restore) → 10 (SL floor) → 13 (cadence) → 12 +(slippage)**. Task 8 first because `contracts_v3.py` is the shared vocabulary everything imports. + +## Still NOT in scope (operator/owner only) +DARK soak start; V4 live execution / BingX ExecutionClient; HZ-bridge refactor; the V3.4c parity +root-cause (CRITICAL #1 — Claude's review job); any edit to the in-flight files in §0. diff --git a/prod/docs/VIOLET_TODO_CRITICAL.md b/prod/docs/VIOLET_TODO_CRITICAL.md new file mode 100644 index 00000000..100e9a19 --- /dev/null +++ b/prod/docs/VIOLET_TODO_CRITICAL.md @@ -0,0 +1,82 @@ +# VIOLET — CRITICAL TODO / review queue (prominent) + +Date: 2026-06-17. Single place for the must-not-forget VIOLET items. Review-later, not now. + +--- + +## 🔴 CRITICAL #1 — VIOLET↔BLUE parity is VERY DISAPPOINTING (review + root-cause BEFORE any soak/V4) + +Report: `prod/VIOLET_dev/reports/violet_parity_20260616_220412.md` (+ `.json`). +Window 2026-06-14 20:15 → 2026-06-15 21:00. Produced by PASS-2 Task 4 (`parity_report.py`). + +**Headline:** +- Pick-match rate: **0.015 (1.5%)** — VIOLET rows 2853, exact pick matches only 43. +- Same-asset rate: **0.136 (13.6%)**; no-pick: **2465 / 2853 (86%)**. +- BLUE rows 25941 (scan_eval 25278 + trade_events 663). + +**KEY NUANCE (where the review should START):** on the 43 rows that DID align, **sizing is +near-identical** — leverage abs error mean 0.016 / **median 0.0** / max 0.135. So the V3.4 +sizing math (boost/beta/mc_scale/ob/esof/compose) is NOT the problem; the divergence is in +**ASSET SELECTION / TIMING / the comparison's ALIGNMENT method**. Candidate causes to +investigate (do not assume — measure): +1. **Apples-to-oranges population.** BLUE `scan_eval` (25278) is likely per-scan-per-asset + *evaluations*, while VIOLET rows are *actuated* decisions — the report may be comparing + different things. Verify the alignment/join semantics in `parity_report.py` first. +2. **Selection divergence.** VIOLET's `VioletAssetSelector` (IRP) vs BLUE's live selection over + the same scan stream — are they fed the same universe/lookback at the same scan index? The + sizing-gap samples (TRX/ATOM/LTC/XLM SHORT with large notional_rel_err) suggest VIOLET fires + on assets BLUE sized very differently or didn't pick. +3. **Cadence/actuation.** VIOLET actuates at Q=scan; if its scan alignment or dedupe differs, + picks land at different scans → counted as no-pick. +4. **The known structural items** (OB single-shot before V3.4d; mc_scale; live-factor sourcing) + — re-run parity AFTER V3.4d's persistent-OB launcher + the bit-identity fixes to see if the + number moves. + +**Action:** full review of `parity_report.py` + a root-cause pass; fix the alignment OR the +selection divergence; re-run. This gates a meaningful DARK soak — a 1.5% pick-match makes the +soak uninterpretable. **Owner: Claude (me), later.** + +--- + +## 🟡 #2 — Review the OA-delegated PASS work (NOT yet reviewed) + +The parallel agent reports these DONE; none reviewed for correctness / BLUE-compliance yet. + +**PASS 1** (`VIOLET_PART_SPEC_OA_TODO.md`): +- `53bdd90` sizing parity-pin tests +- `12b768b` venue OB provider seam (`venue_ob_provider.py`) +- `bae9284` base-fraction sizing study (+ archived report) + +**PASS 2** (`VIOLET_PART_SPEC_OA_TODO_PASS2.md`): +- `parity_report.py` (Task 4 — the report above) + test +- `tradeability.py` (Task 6) + test +- `test_violet_v3_decision_latency_gate.py` (Task 5) → report `violet_v3_decision_latency_2026...` +- `test_violet_replay_determinism_gate.py` (Task 7) → report `violet_replay_determinism_2026...` + +**PASS 3**: `VIOLET_PART_SPEC_OA_TODO_PASS3.md` (issued 2026-06-17) — items below; review when done. + +**Action:** review each pass for correctness, BLUE-algo compliance, V-TYPES, no-shared-edits, +real (non-vacuous) tests. **Owner: Claude (me), later.** + +--- + +## 🟢 #3 — Integration / "sprint" consolidation (my later work) + +Once the passes are reviewed, I will: **(a)** review ALL passes together, **(b)** integrate them +(resolve interfaces, dedupe), **(c)** test them together, **(d)** plug into the operational +system, **(e)** E2E test. + +**Nomenclature note (raised by operator):** a "pass" here = a batch of self-contained tasks +delegated to one agent — smaller than an Agile **sprint** (a time-boxed iteration, typ. 1-4 +weeks, team-scoped, ending in a shippable increment, with planning/review/retro ceremonies). +In this project's existing usage, **"Sprint N" already maps to a V-stage bundle** (Sprint 1 = +V0+V1, Sprint 2 = V2, Sprint 3 = V3) — i.e. an epic/milestone. So the cleanest mapping: +**V-stage = sprint/epic; "pass" = a sub-sprint work-package / task-bundle within it.** Renaming +passes to "sprints" would over-claim scope; keep "pass"/"work-package", or call each an +"increment". Decide at integration time. + +--- + +## Gating rule +The DARK soak (operator-held) and V4 are NOT meaningfully runnable until CRITICAL #1 is +root-caused — a 1.5% pick-match means the shadow is not yet tracking BLUE's decisions.