From 1415a65670f633dab35a0ec4f4d0a16b3994c65a Mon Sep 17 00:00:00 2001 From: Codex Date: Tue, 16 Jun 2026 16:54:35 +0200 Subject: [PATCH] VIOLET V3.4c review: fix mc_scale BLUE-parity bug + document re-derivation debt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit REVIEW of the V3.4c/d/e work (other agents) for BLUE-algo compliance found one real bug and one architectural concern. BUG FIXED — mc_scale derivation (live_blue_source.py): The adapter mapped `mc_forewarner_latest.status == "ORANGE"` → 0.5. But BLUE's sizing mc_scale is NOT the MC service's `status` label (that label, from mc_forewarner_flow.py, uses GREEN<0.10/ORANGE<0.30/RED — observability only). The live trader re-derives the haircut in begin_day (esf_alpha_orchestrator.py:956-962) from the SAME published fields `catastrophic_prob` + `envelope_score` with DIFFERENT thresholds: mc_red = cat>0.25 or env<-1.0 mc_orange = (not mc_red) and (env<0 or cat>0.10) mc_scale = 0.5 if mc_orange else 1.0 The two disagree (e.g. cat=0.05/env=-0.5 → label GREEN→1.0 but BLUE orange→0.5; cat=0.28 → label ORANGE→0.5 but BLUE red→1.0). Rewrote `_derive_mc_scale` to mirror begin_day exactly on the source fields. Per operator: there is genuine ambiguity over which surface is "nominal" — we go with the SOURCE FIELDS + begin_day formula (the path that actually drives BLUE's sizing); the ambiguity + decision are noted in the docstring. TESTS — the old fixtures were complicit: they fed `{"status":"ORANGE"}`, a payload shape BLUE never emits, so they "passed" against fiction. Replaced with BLUE's real payload (`catastrophic_prob`/`envelope_score`) and added a parametrized formula test covering the exact divergence cases the status-based code got wrong, plus missing/garbage-field neutral. 15 passed (live-HZ smoke deselected). ARCHITECTURAL DEBT documented — VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md: VIOLET imitates BLUE's computations in a DIFFERENT module/scope structure. Kernels are WRAPPED (safe), but orchestration arithmetic (compose/regime/ob/strength/mc_scale) is HAND-REPLICATED out of the monolithic NDAlphaEngine — making orderly, verifiable BLUE↔VIOLET parity comparison and refactoring hard. Doc inventories every re-derivation with BLUE's file:line authority, the drift risk, and mitigations (parity-pin tests; single canonical surface per ambiguous factor; eventual DITAv2 Rust-backplane convergence). Pointer added to live_blue_source.py's module docstring. Also reviewed (no change needed): launcher launch_dolphin_violet.py is DARK-safe (ObserveOnlyVenue + shadow gated default-OFF, no order path); shadow_live_factors.py glue is correct; trade_slot_compare.py reimplements no BLUE algo. fb34431's launcher tests — which the authoring agent never ran on the slow mount — pass (14 green). violet-only; no shared-file edits. Co-Authored-By: Claude Opus 4.8 --- prod/clean_arch/violet/live_blue_source.py | 54 ++++++++++-- .../violet/test_violet_live_blue_source.py | 45 ++++++++-- ...IOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md | 85 +++++++++++++++++++ 3 files changed, 171 insertions(+), 13 deletions(-) create mode 100644 prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md diff --git a/prod/clean_arch/violet/live_blue_source.py b/prod/clean_arch/violet/live_blue_source.py index 65157a79..b7191a21 100644 --- a/prod/clean_arch/violet/live_blue_source.py +++ b/prod/clean_arch/violet/live_blue_source.py @@ -13,6 +13,18 @@ translates them into ``SizingFactors`` for the shadow path: The remaining DC signal is left neutral here for now. It needs the same live signal-history path BLUE uses and should be added as a separate mirror step. + +PARITY DEBT (see prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md): this module +reconstructs BLUE's factors in a DIFFERENT file/scope structure than BLUE's monolithic +NDAlphaEngine (esf_alpha_orchestrator.py). Kernels here are WRAPPED (OBFeatureEngine, +AlphaSignalGenerator, VioletAssetSelector — single source of truth), but two derivations +are HAND-REPLICATED / surface-substituted and can drift silently from BLUE: + - ``_derive_mc_scale`` transcribes begin_day's mc thresholds (no pin to BLUE's fn); + - ``AlphaSignalGenerator()`` is built with BARE DEFAULTS, not BLUE's threaded params + (esf_alpha_orchestrator.py:180-191) — cosmetic only while dc_leverage_boost==1.0; + - boost/beta read the PUBLISHED ``acb_boost`` (acb_processor_service), not the trader's + ``get_dynamic_boost_from_hz`` recompute — the two surfaces may differ. +Any change to BLUE's corresponding formula REQUIRES a matching change here + a test. """ from __future__ import annotations @@ -72,13 +84,41 @@ def _read_hz_map(client: hazelcast.HazelcastClient, map_name: str, key: str) -> return None -def _map_status_to_mc_scale(payload: Any) -> float: +def _derive_mc_scale(payload: Any) -> float: + """Mirror BLUE ``begin_day``'s ``mc_scale`` EXACTLY (esf_alpha_orchestrator.py:956-962). + + BLUE does NOT use the MC service's published ``status`` label for sizing — that label + (mc_forewarner_flow.py: GREEN<0.10 / ORANGE<0.30 / RED) is observability-only and uses + DIFFERENT thresholds from the engine. The live trader re-derives the size haircut from + the SAME published fields ``catastrophic_prob`` + ``envelope_score`` with the engine's + own thresholds: + + mc_red = catastrophic_prob > 0.25 or envelope_score < -1.0 + mc_orange = (not mc_red) and (envelope_score < 0 or catastrophic_prob > 0.10) + mc_scale = 0.5 if mc_orange else 1.0 # RED → 1.0 here; BLUE halts via regime_dd_halt + + Reading ``status`` instead diverges (e.g. cat=0.05/env=-0.5 → publisher GREEN→1.0 but + BLUE orange→0.5). + + AMBIGUITY (flagged 2026-06-16): the MC service's ``status`` and the engine's begin_day + thresholds genuinely disagree, and from OUTSIDE BLUE there is no way to know which is the + "nominal" intent — they are two independent threshold sets over the same numbers. We go + with the SOURCE FIELDS + begin_day formula because that is the path that actually drives + BLUE's live sizing (the published ``status`` is consumed only by the TUI/observability). + If BLUE's begin_day thresholds change, THIS must change with them. Operator confirmation + of the canonical surface is still desirable. + + Missing/unparseable fields → neutral 1.0 (no haircut).""" data = _jsonish(payload) - if isinstance(data, Mapping): - status = str(data.get("status", "")).upper() - else: - status = str(data).upper() - return 0.5 if status == "ORANGE" else 1.0 + if not isinstance(data, Mapping): + return 1.0 + cat = _coerce_float(data.get("catastrophic_prob"), None) + env = _coerce_float(data.get("envelope_score"), None) + if cat is None or env is None: + return 1.0 + mc_red = cat > 0.25 or env < -1.0 + mc_orange = (not mc_red) and (env < 0.0 or cat > 0.10) + return 0.5 if mc_orange else 1.0 def _extract_acb(payload: Any) -> tuple[float, float]: @@ -302,7 +342,7 @@ def source_live_blue_sizing_factors( acb_boost, acb_beta = _extract_acb(acb_raw) mc_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "mc_forewarner_latest") - mc_scale = _map_status_to_mc_scale(mc_raw) + mc_scale = _derive_mc_scale(mc_raw) scan_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "latest_eigen_scan") scan = scan_history.ingest_scan(scan_raw) diff --git a/prod/clean_arch/violet/test_violet_live_blue_source.py b/prod/clean_arch/violet/test_violet_live_blue_source.py index c1ccfa49..8d92d674 100644 --- a/prod/clean_arch/violet/test_violet_live_blue_source.py +++ b/prod/clean_arch/violet/test_violet_live_blue_source.py @@ -14,6 +14,7 @@ from prod.clean_arch.violet.decision_engine import SizingFactors from prod.clean_arch.violet.live_blue_source import ( HazelcastOBProvider, LiveBlueScanHistory, + _derive_mc_scale, source_live_blue_sizing_factors, ) from prod.clean_arch.violet.alpha_wrappers import VioletAssetSelector @@ -63,7 +64,7 @@ def test_hz_ob_provider_filters_and_parses_latest_payload(): "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} ), "acb_boost": json.dumps({"boost": 1.2, "beta": 0.3}), - "mc_forewarner_latest": json.dumps({"status": "ORANGE"}), + "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.15, "envelope_score": 0.5}), "esof_latest": json.dumps({"advisory_score": 0.4}), }, "DOLPHIN_STATE_BLUE": {"latest_nautilus": json.dumps({"posture": "restored"})}, @@ -107,7 +108,7 @@ def test_source_live_blue_sizing_factors_unit(monkeypatch): "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} ), "acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}), - "mc_forewarner_latest": json.dumps({"status": "ORANGE"}), + "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.15, "envelope_score": 0.5}), "esof_latest": json.dumps({"advisory_score": 0.4}), "latest_eigen_scan": json.dumps( { @@ -171,7 +172,7 @@ def test_source_live_blue_sizing_factors_preserves_skip_contradict(monkeypatch): "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} ), "acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}), - "mc_forewarner_latest": json.dumps({"status": "GREEN"}), + "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}), "esof_latest": json.dumps({"advisory_score": 0.4}), "latest_eigen_scan": json.dumps( { @@ -254,7 +255,7 @@ def test_live_blue_sequence_matches_blue_selector_and_dc_at_each_step(monkeypatc "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} ), "acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}), - "mc_forewarner_latest": json.dumps({"status": "GREEN"}), + "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}), "esof_latest": json.dumps({"advisory_score": 0.3}), "latest_eigen_scan": json.dumps({**scan, "target_asset": "BTCUSDT"}), }, @@ -314,7 +315,7 @@ def test_live_blue_sequence_rejects_anomalous_values_without_poisoning_history(m "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} ), "acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}), - "mc_forewarner_latest": json.dumps({"status": "GREEN"}), + "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}), "esof_latest": json.dumps({"advisory_score": 0.3}), "latest_eigen_scan": json.dumps(scan), }, @@ -351,7 +352,7 @@ def test_source_live_blue_sizing_factors_handles_anomalies(monkeypatch): "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} ), "acb_boost": json.dumps({"boost": -9, "beta": "bad"}), - "mc_forewarner_latest": json.dumps({"status": "GREEN"}), + "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}), "esof_latest": "not json", "latest_eigen_scan": json.dumps({"target_asset": "BTCUSDT", "assets": ["BTCUSDT"], "asset_prices": [0]}), }, @@ -370,6 +371,38 @@ def test_source_live_blue_sizing_factors_handles_anomalies(monkeypatch): assert res.selected_asset == "BTCUSDT" +@pytest.mark.parametrize( + "cat, env, expected", + [ + # mc_orange (→0.5): not red, and (env<0 OR cat>0.10) + (0.15, 0.5, 0.5), # orange via cat>0.10 + (0.05, -0.5, 0.5), # orange via env<0 — OLD status-code returned GREEN→1.0 (the bug) + (0.10, -0.001, 0.5), # orange via env<0 boundary (cat==0.10 not >0.10) + # red (→1.0 here; BLUE halts separately via regime_dd_halt) + (0.28, 0.5, 1.0), # red via cat>0.25 — OLD status-code: 0.28<0.30→ORANGE→0.5 (the bug) + (0.05, -1.5, 1.0), # red via env<-1.0 + (0.30, -2.0, 1.0), # red via both + # ok (→1.0): not red, not orange + (0.05, 0.5, 1.0), # benign + (0.10, 0.0, 1.0), # cat==0.10 (not >0.10), env==0 (not <0) → ok + ], +) +def test_derive_mc_scale_matches_blue_begin_day_formula(cat, env, expected): + payload = json.dumps({"catastrophic_prob": cat, "envelope_score": env, "status": "IGNORED"}) + assert _derive_mc_scale(payload) == expected + + +def test_derive_mc_scale_neutral_when_fields_missing_or_garbage(): + # The published `status` label must NOT be used — payloads lacking the source + # fields derive neutral (1.0), never a haircut from a stale/foreign label. + assert _derive_mc_scale(json.dumps({"status": "ORANGE"})) == 1.0 + assert _derive_mc_scale(json.dumps({"catastrophic_prob": 0.15})) == 1.0 # missing env + assert _derive_mc_scale(json.dumps({"envelope_score": -0.5})) == 1.0 # missing cat + assert _derive_mc_scale("not json") == 1.0 + assert _derive_mc_scale(None) == 1.0 + assert _derive_mc_scale({"catastrophic_prob": "bad", "envelope_score": 0.5}) == 1.0 + + def test_live_hz_smoke_reads_current_state(): client = hazelcast.HazelcastClient(cluster_name="dolphin", cluster_members=["localhost:5701"]) try: diff --git a/prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md b/prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md new file mode 100644 index 00000000..92418ec9 --- /dev/null +++ b/prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md @@ -0,0 +1,85 @@ +# VIOLET ↔ BLUE parity: structural divergence & re-derivation debt + +**Date:** 2026-06-16 +**Status:** ACKNOWLEDGED TRADEOFF / open architectural debt +**Raised by:** operator, during the V3.4c review. + +## The problem, stated plainly + +VIOLET reproduces BLUE's sizing behaviour **bit-for-bit by intent**, but it does so in a +**different module / file / scope structure** than BLUE. BLUE's logic lives in one place — +the monolithic `NDAlphaEngine` (`nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py`), +which holds day-state (`_day_base_boost`, `_day_beta`, `_day_mc_scale`, `_day_posture`), +constructs its own `signal_gen`/`bet_sizer`, and runs `begin_day` / `_try_entry` inline. + +VIOLET re-expresses that same logic spread across: +`sizing.py`, `live_blue_source.py`, `live_factors.py`, `live_factor_source.py`, +`decision_engine.py`, `alpha_wrappers.py`. + +**Consequence (the operator's concern, verbatim intent):** because VIOLET imitates the +computations *while* ending up with a different structure, any *orderly, systemic, +verifiable* BLUE↔VIOLET algo parity comparison — and any future refactor of either side — +is **much harder**. The surfaces do not line up 1:1, so a diff between the two engines is +not mechanical; it requires a human to know which VIOLET fragment mirrors which BLUE line. + +## Two kinds of reuse — and only one is safe + +1. **WRAPPED kernels (safe — single source of truth).** VIOLET imports and calls BLUE's + actual kernel objects. A BLUE change propagates automatically. + - `esof_size_mult_from_score`, `parse_esof_payload`, `esof_score_from_payload` + (`esof_size_gate.py`) — wrapped by `sizing.py` / `live_factor_source.py`. + - `OBFeatureEngine.get_market` (`ob_features.py`) — wrapped by `live_blue_source.py`. + - `AlphaSignalGenerator.generate` (`alpha_signal_generator.py`) — wrapped by `live_blue_source.py`. + - `AlphaAssetSelector` / `AlphaBetSizer` — wrapped by `alpha_wrappers.py`. + - `map_internal_conviction_to_exchange_leverage` (`bingx/leverage.py`) — wrapped by `exchange_leverage.py`. + +2. **HAND-REPLICATED arithmetic (the debt — duplicated formulas, drift-prone).** VIOLET + transcribes BLUE's pure float arithmetic into its own functions. A BLUE change here is + SILENT in VIOLET until someone notices. + +## Re-derivation inventory (the drift liabilities) + +| Computation | BLUE authority (file:line) | VIOLET replica | Parity safety-net today | Drift risk | +|---|---|---|---|---| +| 5-factor compose + caps | `esf_alpha_orchestrator.py:600-619` | `sizing.VioletSizer.compose` | `@gate` Monte-Carlo vs REAL orchestrator (bit-identity) | LOW (gated) | +| `regime_size_mult` = boost·(1+β·s³)·mc | `esf_alpha_orchestrator.py:898-909` | `sizing.VioletSizer.regime_size_mult` | same gate | LOW-MED | +| `strength_cubic` | `esf_alpha_orchestrator.py:872-885` | `sizing.VioletSizer.strength_cubic` | same gate | LOW-MED | +| `market_ob_mult` consensus | `esf_alpha_orchestrator.py:587-595` | `sizing.VioletSizer.market_ob_mult` | same gate | MED | +| `dc_lev_mult` | `esf_alpha_orchestrator.py:575-577` | `sizing.VioletSizer.dc_lev_mult` | unit only | MED (but ≡1.0 while dc_leverage_boost=1.0) | +| **`mc_scale`** | `esf_alpha_orchestrator.py:956-962` (`begin_day`) | `live_blue_source._derive_mc_scale` | **unit only — NO pin to BLUE's fn** | **HIGH** | +| `boost`/`beta` source | trader recompute `acb.get_dynamic_boost_from_hz(exf_latest)` | reads published `DOLPHIN_FEATURES.acb_boost` (acb_processor_service) | none | MED (two surfaces may differ on dynamic-β / OB Sub-4) | +| `dc_status` config | `signal_gen` built with threaded params `:180-191` | `AlphaSignalGenerator()` **bare defaults** | none | MED (cosmetic while dc_lev_mult≡1.0) | +| OB feed shape | live OB accumulation | single-snapshot `HazelcastOBProvider` + `bar_idx=0` | none | MED | + +The worst link is **`mc_scale`**: pure duplication of `begin_day`'s thresholds with no +test that pins it to BLUE's actual function (BLUE computes it inline inside `begin_day`, +which is not callable in isolation). The V3.4c bug fixed on 2026-06-16 (the adapter keyed +off the MC service's `status` label instead of BLUE's `cat`/`env` thresholds) is exactly +the failure mode this structure invites. + +## Why we accept it (for now) + +- The kernels that carry the heavy alpha are WRAPPED, not copied. +- The composition arithmetic IS gated bit-for-bit against the real orchestrator. +- VIOLET must stay a *read-only, DARK* mirror of a *running* BLUE; it cannot import BLUE's + live in-process day-state, so some reconstruction from published HZ surfaces is unavoidable. + +## Mitigations (recommended, not yet done) + +1. **Parity-pin every hand-replicated formula.** For each row above, add a test that + imports BLUE's authoritative function/constant and asserts VIOLET's replica equals it + over a sampled grid — converting silent drift into a red test. Where BLUE's logic is + trapped inside `begin_day` (mc_scale), refactor a *pure* `mc_scale_from(cat, env)` helper + **on the BLUE side** (BLUE-domain change, operator-gated) that BOTH engines call. +2. **Single ambiguity owner.** Surfaces like MC (`status` label vs `begin_day` thresholds) + and ACB (published `acb_boost` vs trader recompute) have two disagreeing sources; pick + ONE canonical per factor and document it (see `_derive_mc_scale` docstring). +3. **Backplane convergence (the real fix).** When the DITAv2 Rust middleware becomes the + shared backplane, BOTH BLUE and VIOLET should consume factors from it rather than each + computing/replicating — collapsing this divergence at the source. Until then, every new + hand-replication MUST be logged in this table. + +## Maintenance rule + +Any change to a BLUE formula in the left column REQUIRES a matching change + test update in +the VIOLET replica in the same PR. Any NEW hand-replication MUST add a row here.