From 9c068619e1fdae5fa5d75f8673817fca431b1716 Mon Sep 17 00:00:00 2001 From: Codex Date: Wed, 17 Jun 2026 09:01:49 +0200 Subject: [PATCH] VIOLET: CRITICAL parity-disappointment TODO + OA partial spec PASS 3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VIOLET_TODO_CRITICAL.md (prominent review queue): - πŸ”΄ CRITICAL #1: VIOLET↔BLUE parity report (PASS-2 Task 4, violet_parity_20260616_220412.md) is VERY DISAPPOINTING β€” pick-match 1.5%, no-pick 86%, same-asset 13.6%. KEY NUANCE: on the 43 aligned rows sizing is near-identical (lev_err median 0.0) β†’ the divergence is SELECTION / TIMING / comparison-ALIGNMENT, NOT the sizing math. Root-cause before any soak/V4. (Claude.) - #2: OA PASS-1 + PASS-2 reported done, NOT yet reviewed β€” review queue listed. - #3: future integration/E2E + the pass-vs-sprint nomenclature note (pass = sub-sprint work-package; V-stage = sprint/epic, matching existing usage). VIOLET_PART_SPEC_OA_TODO_PASS3.md (lengthy, plan-derived, independent units with shared interfaces β€” nothing invented): contracts_v3 (VenueTick/OpenPositionView/ExitDecision) + VenuePriceFeedPort+BingX adapter, MechanicalExitGuard (armed TP/SL), sub-second SL/ADVSL floor guard, event-sourced restore from trade_reconstruction, venue lead/lag slippage metric, cadence per-action Q schedule + telemetry. Each ships + tests standalone; composition map + order given. Co-Authored-By: Claude Opus 4.8 --- prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md | 212 ++++++++++++++++++++ prod/docs/VIOLET_TODO_CRITICAL.md | 82 ++++++++ 2 files changed, 294 insertions(+) create mode 100644 prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md create mode 100644 prod/docs/VIOLET_TODO_CRITICAL.md diff --git a/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md b/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md new file mode 100644 index 00000000..f392fb4f --- /dev/null +++ b/prod/docs/VIOLET_PART_SPEC_OA_TODO_PASS3.md @@ -0,0 +1,212 @@ +# VIOLET β€” partial spec for another agent, PASS 3 (OA TODO) + +Date: 2026-06-17. Follows PASS 1 + PASS 2 (both reported done β€” review pending, see +`VIOLET_TODO_CRITICAL.md`). PASS 3 is drawn STRICTLY from the existing VIOLET dev plan +(`VIOLET_DEV_SPEC_AND_PLAN.md`, the V0β†’V6 ladder + its named deferred items) β€” nothing invented. +Every task is an INDEPENDENT UNIT: it can be built, tested and function on its own, and it +composes with the others ONLY through the explicit shared interfaces in Β§I. + +--- + +## 0. HARD RULES (identical to PASS 1/2 β€” re-read those; summarized) + +- **Never edit shared files** (`prod/nautilus_event_trader.py`, `clean_arch/dita_v2/**`, + `dita/decision.py`, `nautilus_dolphin/**`, `blue_parity.py`, `prod/bingx/leverage.py`). READ only. +- **VIOLET stays DARK** β€” no orders, no execution, no VST keys, no service start/stop, no HZ + restart, no PROGREEN. +- **V-TYPES on all new code** (`StrictModel` / `Annotated[…Field]` / `@typed`); faithful + poison-guards only. +- **NEW-FILE-ONLY**; do NOT modify in-flight files: `live_blue_source.py`, + `shadow_live_factors.py`, `live_factor_source.py`, `live_factors.py`, `decision_engine.py`, + `sizing.py`, `shadow_journal.py`, `cadence.py`, `clock.py`, `divergence.py`, + `22_violet_decisions.sql`. READ + IMPORT them. + +## 0a. COMMIT / BRANCH POLICY (3 shared-index collisions on 2026-06-16 β€” take this seriously) +Own `git worktree` (`git worktree add ../vp-oa3 -b agent/oa-violet3`) STRONGLY preferred. Else +never `git add -A`; commit with explicit pathspec `git commit -F msg -- ` and verify +`git show --stat --format="" HEAD` lists ONLY your files. One commit per task, prefix +`VIOLET OA:`, Co-Authored-By trailer. Tests on `/home/dolphin/siloqy_env/bin/python3`. Use +`git grep` (recursive grep/find time out on CIFS). + +--- + +## I. SHARED INTERFACES (agreed contracts β€” define these EXACTLY; the tasks depend on them) + +These are the only coupling points between PASS-3 units. Put the type definitions in ONE new +module `prod/clean_arch/violet/contracts_v3.py` (Task 8 creates it; later tasks import it). +All are `StrictModel` / `Annotated` V-TYPES. + +1. **`VenueTick`** β€” one normalized venue quote (exchange-agnostic): + `asset: Symbol`, `bid: Px`, `ask: Px`, `mark: Px`, `last: Px`, `mono_ns: MonoNs`, + `venue: str` (e.g. "BINGX","BINANCE"). All prices > 0 finite. This is the SAME normalized + tick named in the plan's "Venue price feed port (CRITICAL gap)". +2. **`OpenPositionView`** β€” the minimal open-position state an exit/guard needs: + `asset: Symbol`, `side: str` ("SHORT"/"LONG"), `entry_price: Px`, `qty: Qty`, + `entry_ts_ns: MonoNs`, `bars_held: BarsHeld`, `tp_threshold: float` (fixed TP pct), + `sl_threshold: float` (stop pct), `leverage: float`. +3. **`ExitDecision`** β€” `action: str` ("HOLD"/"EXIT"), `reason: str` + ("FIXED_TP"/"CATASTROPHIC_SL"/"ADVSL"/"NONE"/…), `price: Px`, `priority: int` + (CATASTROPHIC/ADVSL=0 > FIXED_TP=1 > DISCRETIONARY=2 β€” the plan's mandated order). + +If a later task needs a field not here, ADD it to `contracts_v3.py` (and note it), never fork a +parallel type. + +--- + +## TASK 8 β€” VenuePriceFeedPort + BingX adapter (DARK, data-only) [plan: "Venue price feed port (CRITICAL gap)"] + +**Why.** The plan flags this as CRITICAL: OBF is Binance-wired (alpha side), but exits fill on +the EXECUTION venue, and FET showed scan-vs-BingX divergence (0.2176 vs 0.1878 = 15%). VIOLET +needs an exchange-agnostic price port. This unit delivers the port interface + a BingX adapter, +data-only (no orders). + +**Affected files (NEW only):** `prod/clean_arch/violet/contracts_v3.py` (the Β§I types), +`prod/clean_arch/violet/venue_price_feed.py`, `prod/clean_arch/violet/test_violet_venue_price_feed.py`. + +**Interface/approach.** Define `VenuePriceFeedPort` (ABC): `latest(asset) -> Optional[VenueTick]`, +`subscribe(assets)`, `close()`. Implement `BingxSwapPriceFeed(VenuePriceFeedPort)` consuming the +BingX swap WS bookTicker/markPrice (READ the existing BingX WS conventions in the repo; data-only, +no auth needed for public streams β€” confirm). Implement `MockPriceFeed` (deterministic, for +tests + for the other tasks). All ticks validated into `VenueTick` at ingress (poison-reject +non-finite/≀0). + +**Tests / pass criteria.** Mock feed round-trips ticks as `VenueTick`; poison ticks rejected; +the BingX adapter parses a recorded/sample bookTicker frame into a correct `VenueTick` (use a +captured frame fixture, NOT a live connection in the unit test). DONE when: `VenueTick` contract +finalized, mock + adapter parse-tested, no live WS in unit tests, no edits outside the 3 files. + +--- + +## TASK 9 β€” MechanicalExitGuard (armed TP/SL) [plan: LINK TP-miss structural fix] + +**Why.** The plan's structural fix for the LINKUSDT βˆ’$1,248 TP-miss: TP/SL are MECHANICAL exits +owned by an exit guard with armed price thresholds set at entry; policy layers (V7/MARAS) can +only contribute DISCRETIONARY exits and can NEVER mask mechanical ones. Total priority: +CATASTROPHIC/ADVSL > fixed TP > discretionary. + +**Affected files (NEW only):** `prod/clean_arch/violet/mechanical_exit_guard.py`, +`prod/clean_arch/violet/test_violet_mechanical_exit_guard.py`. (imports `contracts_v3`.) + +**Interface/approach.** `MechanicalExitGuard.evaluate(pos: OpenPositionView, tick: VenueTick) -> +ExitDecision`. Pure function of position + current venue price: compute pnl_pct from +entry/side/price; if it reaches the armed `tp_threshold` β†’ `ExitDecision(EXIT, FIXED_TP, +priority=1)`; if it breaches `sl_threshold` β†’ `ExitDecision(EXIT, CATASTROPHIC_SL, priority=0)`; +else HOLD. Prices off the VENUE tick (not scan). No discretionary logic here β€” this layer is the +mechanical floor only. + +**Tests / pass criteria.** SHORT + LONG: TP fires exactly at threshold, SL at stop, neither +fires inside the band; priority ordering correct (SL outranks TP if both somehow true). Hypothesis +property: output always finite, priority ∈ {0,1,2}, EXIT only when threshold crossed. DONE when: +deterministic threshold tests + property test pass, no edits outside the 2 files. + +--- + +## TASK 10 β€” Sub-second catastrophic-SL / ADVSL floor guard [plan: versioned SAFETY DEVIATION] + +**Why.** The plan's one sanctioned sub-second behaviour: a catastrophic-SL/ADVSL floor evaluated +at the fastest cadence against a fast price source, to fix the scan-dark unmanaged-position hazard +(XLM/FET class). "Evaluate at fastest cadence (shadow-log would-be actions), actuate at Q." + +**Affected files (NEW only):** `prod/clean_arch/violet/sl_floor_guard.py`, +`prod/clean_arch/violet/test_violet_sl_floor_guard.py`. (imports `contracts_v3`, composes Task 9.) + +**Interface/approach.** `SLFloorGuard(deadline_ns)` with `on_tick(pos, tick) -> Optional[ExitDecision]` +that EVALUATES every tick (sub-second) and SHADOW-LOGS would-be SL exits, but only RETURNS an +actuation when the configured fast-SL condition holds (catastrophic threshold worse than the +mechanical stop, OR ADVSL trailing breach). Track evaluate-count vs actuate-count (shadow delta). +DARK: returns the decision; never sends orders. + +**Tests / pass criteria.** A tick stream that dips intra-scan triggers the fast SL evaluation and +records the would-be exit; the actuate gate fires only on the catastrophic/ADVSL condition; +evaluate-count > actuate-count proven. DONE when tests pass, no edits outside the 2 files. + +--- + +## TASK 11 β€” Event-sourced restore from trade_reconstruction [plan: VIOLET restore commitment, Option C] + +**Why.** The plan commits VIOLET to NEVER restore from position_state snapshots; instead replay +the chain-tokened `trade_reconstruction` OPEN/PARTIAL_EXIT/CLOSE journal. Open = ROOT OPEN with no +terminal CLOSE; size = entry βˆ’ Ξ£ legs; holding from ABSOLUTE entry_ts (kills the bars_heldβ‰ˆ0 / +MAX_HOLD-reset class by construction); chain token verified. + +**Affected files (NEW only):** `prod/clean_arch/violet/event_restore.py`, +`prod/clean_arch/violet/test_violet_event_restore.py`. (imports `contracts_v3` β†’ emits +`OpenPositionView`s.) + +**Interface/approach.** `restore_open_positions(journal_rows: list[dict]) -> list[OpenPositionView]` +β€” pure function over journal rows (asset, chain_root_trade_id, chain_token, leg type, qty, +realized legs, entry_ts, entry_price). Reconstruct each chain; an open position is a ROOT OPEN +with no matching terminal CLOSE; qty = entry βˆ’ Ξ£ partial-exit legs; bars_held derived from absolute +entry_ts vs now (NEVER from a stored counter). Reject chain-token mismatches to a quarantine list +(returned alongside), never crash. + +**Tests / pass criteria.** Fixtures: clean open, fully-closed (β†’ not restored), partial-exit chain +(qty correct), chain-token mismatch (β†’ quarantined), dead-session entry_ts (bars_held computed +from ts, never negative). Hypothesis: qty β‰₯ 0, bars_held β‰₯ 0 always. DONE when tests pass, no +edits outside the 2 files. + +--- + +## TASK 12 β€” Venue lead/lag slippage metric [plan: V1 metrics β€” venue lead/lag] + +**Why.** The plan's continuous signed SHORT-entry slippage metric: venue fill vs Binance mid at +decision time β€” answers "is BingX discounting our signal". Rerunnable analysis. + +**Affected files (NEW only):** `prod/clean_arch/violet/slippage_metric.py`, +`prod/clean_arch/violet/test_violet_slippage_metric.py`, report β†’ +`prod/VIOLET_dev/reports/violet_slippage_.json`. + +**Interface/approach.** `signed_entry_slippage(decision_mid: float, venue_fill: float, side: str) +-> float` (signed bps; positive = adverse) + an aggregator over recorded rows producing a +distribution (mean/median/p95). Read-only over recorded data (CH `dolphin`/`dolphin_violet`, no +writes). Pure functions; V-TYPES the rows. + +**Tests / pass criteria.** Sign convention exact for SHORT and LONG on hand-built cases; aggregator +stats correct on a synthetic fixture (`--self-test`). DONE when self-test passes + a report can be +produced on the prod host, no production writes, no edits outside the files. + +--- + +## TASK 13 β€” Cadence per-action Q schedule + evaluate/actuate telemetry [plan: cadence quantizer] + +**Why.** The plan's cadence quantizer: each action (SL/TP/ENTRY/OBF/ExoF) gets its own +quantization Q; evaluate at fastest cadence (shadow evidence), actuate at Q; step Q down later. +`cadence.py` (the `CadenceControlPlane`) exists; this unit adds a per-action Q-SCHEDULE LOADER + +a shadow-delta TELEMETRY recorder WITHOUT modifying `cadence.py`. + +**Affected files (NEW only):** `prod/clean_arch/violet/cadence_schedule.py`, +`prod/clean_arch/violet/test_violet_cadence_schedule.py`. + +**Interface/approach.** Read `cadence.py` for `Action` + `CadenceControlPlane` API. Provide +`load_q_schedule(mapping) -> dict[Action, int_ns]` (validated; the plan's initial table: SL tight, +TP=scan, ENTRY=scan, OBF~1s) and `CadenceTelemetry` that records per-action evaluate-count vs +actuate-count and emits a shadow-delta summary. Compose with `CadenceControlPlane` by wrapping its +`due()` calls (do not edit it). + +**Tests / pass criteria.** Schedule loader validates/rejects bad Q values; telemetry counts +evaluate > actuate under a synthetic action stream; per-action Q honored. DONE when tests pass, no +edits outside the 2 files. + +--- + +## Composition map (how the units fit β€” the "well-known interfaces") +``` +Task 8 contracts_v3 (VenueTick, OpenPositionView, ExitDecision) ← the shared vocabulary + β”‚ + β”œβ”€β”€ Task 8 venue_price_feed β†’ VenueTick stream + β”œβ”€β”€ Task 9 mechanical_exit_guard(pos, tick) β†’ ExitDecision (consumes VenueTick) + β”œβ”€β”€ Task 10 sl_floor_guard(pos, tick) β†’ ExitDecision (composes Task 9) + β”œβ”€β”€ Task 11 event_restore(journal) β†’ [OpenPositionView] (feeds 9/10) + β”œβ”€β”€ Task 12 slippage_metric(mid, fill, side)β†’ bps (uses VenueTick mids) + └── Task 13 cadence_schedule/telemetry β†’ per-action Q + deltas +``` +Integration (wiring these into the live reactor + decision/exit path) is the OWNER's job later, +NOT part of these units. Each ships standalone + tested. + +## Recommended order +**8 (contracts + feed) β†’ 9 (mechanical exit) β†’ 11 (restore) β†’ 10 (SL floor) β†’ 13 (cadence) β†’ 12 +(slippage)**. Task 8 first because `contracts_v3.py` is the shared vocabulary everything imports. + +## Still NOT in scope (operator/owner only) +DARK soak start; V4 live execution / BingX ExecutionClient; HZ-bridge refactor; the V3.4c parity +root-cause (CRITICAL #1 β€” Claude's review job); any edit to the in-flight files in Β§0. diff --git a/prod/docs/VIOLET_TODO_CRITICAL.md b/prod/docs/VIOLET_TODO_CRITICAL.md new file mode 100644 index 00000000..100e9a19 --- /dev/null +++ b/prod/docs/VIOLET_TODO_CRITICAL.md @@ -0,0 +1,82 @@ +# VIOLET β€” CRITICAL TODO / review queue (prominent) + +Date: 2026-06-17. Single place for the must-not-forget VIOLET items. Review-later, not now. + +--- + +## πŸ”΄ CRITICAL #1 β€” VIOLET↔BLUE parity is VERY DISAPPOINTING (review + root-cause BEFORE any soak/V4) + +Report: `prod/VIOLET_dev/reports/violet_parity_20260616_220412.md` (+ `.json`). +Window 2026-06-14 20:15 β†’ 2026-06-15 21:00. Produced by PASS-2 Task 4 (`parity_report.py`). + +**Headline:** +- Pick-match rate: **0.015 (1.5%)** β€” VIOLET rows 2853, exact pick matches only 43. +- Same-asset rate: **0.136 (13.6%)**; no-pick: **2465 / 2853 (86%)**. +- BLUE rows 25941 (scan_eval 25278 + trade_events 663). + +**KEY NUANCE (where the review should START):** on the 43 rows that DID align, **sizing is +near-identical** β€” leverage abs error mean 0.016 / **median 0.0** / max 0.135. So the V3.4 +sizing math (boost/beta/mc_scale/ob/esof/compose) is NOT the problem; the divergence is in +**ASSET SELECTION / TIMING / the comparison's ALIGNMENT method**. Candidate causes to +investigate (do not assume β€” measure): +1. **Apples-to-oranges population.** BLUE `scan_eval` (25278) is likely per-scan-per-asset + *evaluations*, while VIOLET rows are *actuated* decisions β€” the report may be comparing + different things. Verify the alignment/join semantics in `parity_report.py` first. +2. **Selection divergence.** VIOLET's `VioletAssetSelector` (IRP) vs BLUE's live selection over + the same scan stream β€” are they fed the same universe/lookback at the same scan index? The + sizing-gap samples (TRX/ATOM/LTC/XLM SHORT with large notional_rel_err) suggest VIOLET fires + on assets BLUE sized very differently or didn't pick. +3. **Cadence/actuation.** VIOLET actuates at Q=scan; if its scan alignment or dedupe differs, + picks land at different scans β†’ counted as no-pick. +4. **The known structural items** (OB single-shot before V3.4d; mc_scale; live-factor sourcing) + β€” re-run parity AFTER V3.4d's persistent-OB launcher + the bit-identity fixes to see if the + number moves. + +**Action:** full review of `parity_report.py` + a root-cause pass; fix the alignment OR the +selection divergence; re-run. This gates a meaningful DARK soak β€” a 1.5% pick-match makes the +soak uninterpretable. **Owner: Claude (me), later.** + +--- + +## 🟑 #2 β€” Review the OA-delegated PASS work (NOT yet reviewed) + +The parallel agent reports these DONE; none reviewed for correctness / BLUE-compliance yet. + +**PASS 1** (`VIOLET_PART_SPEC_OA_TODO.md`): +- `53bdd90` sizing parity-pin tests +- `12b768b` venue OB provider seam (`venue_ob_provider.py`) +- `bae9284` base-fraction sizing study (+ archived report) + +**PASS 2** (`VIOLET_PART_SPEC_OA_TODO_PASS2.md`): +- `parity_report.py` (Task 4 β€” the report above) + test +- `tradeability.py` (Task 6) + test +- `test_violet_v3_decision_latency_gate.py` (Task 5) β†’ report `violet_v3_decision_latency_2026...` +- `test_violet_replay_determinism_gate.py` (Task 7) β†’ report `violet_replay_determinism_2026...` + +**PASS 3**: `VIOLET_PART_SPEC_OA_TODO_PASS3.md` (issued 2026-06-17) β€” items below; review when done. + +**Action:** review each pass for correctness, BLUE-algo compliance, V-TYPES, no-shared-edits, +real (non-vacuous) tests. **Owner: Claude (me), later.** + +--- + +## 🟒 #3 β€” Integration / "sprint" consolidation (my later work) + +Once the passes are reviewed, I will: **(a)** review ALL passes together, **(b)** integrate them +(resolve interfaces, dedupe), **(c)** test them together, **(d)** plug into the operational +system, **(e)** E2E test. + +**Nomenclature note (raised by operator):** a "pass" here = a batch of self-contained tasks +delegated to one agent β€” smaller than an Agile **sprint** (a time-boxed iteration, typ. 1-4 +weeks, team-scoped, ending in a shippable increment, with planning/review/retro ceremonies). +In this project's existing usage, **"Sprint N" already maps to a V-stage bundle** (Sprint 1 = +V0+V1, Sprint 2 = V2, Sprint 3 = V3) β€” i.e. an epic/milestone. So the cleanest mapping: +**V-stage = sprint/epic; "pass" = a sub-sprint work-package / task-bundle within it.** Renaming +passes to "sprints" would over-claim scope; keep "pass"/"work-package", or call each an +"increment". Decide at integration time. + +--- + +## Gating rule +The DARK soak (operator-held) and V4 are NOT meaningfully runnable until CRITICAL #1 is +root-caused β€” a 1.5% pick-match means the shadow is not yet tracking BLUE's decisions.