VIOLET V3.4c review: fix mc_scale BLUE-parity bug + document re-derivation debt
REVIEW of the V3.4c/d/e work (other agents) for BLUE-algo compliance found one
real bug and one architectural concern.
BUG FIXED — mc_scale derivation (live_blue_source.py):
The adapter mapped `mc_forewarner_latest.status == "ORANGE"` → 0.5. But BLUE's sizing
mc_scale is NOT the MC service's `status` label (that label, from mc_forewarner_flow.py,
uses GREEN<0.10/ORANGE<0.30/RED — observability only). The live trader re-derives the
haircut in begin_day (esf_alpha_orchestrator.py:956-962) from the SAME published fields
`catastrophic_prob` + `envelope_score` with DIFFERENT thresholds:
mc_red = cat>0.25 or env<-1.0
mc_orange = (not mc_red) and (env<0 or cat>0.10)
mc_scale = 0.5 if mc_orange else 1.0
The two disagree (e.g. cat=0.05/env=-0.5 → label GREEN→1.0 but BLUE orange→0.5;
cat=0.28 → label ORANGE→0.5 but BLUE red→1.0). Rewrote `_derive_mc_scale` to mirror
begin_day exactly on the source fields. Per operator: there is genuine ambiguity over
which surface is "nominal" — we go with the SOURCE FIELDS + begin_day formula (the path
that actually drives BLUE's sizing); the ambiguity + decision are noted in the docstring.
TESTS — the old fixtures were complicit: they fed `{"status":"ORANGE"}`, a payload shape
BLUE never emits, so they "passed" against fiction. Replaced with BLUE's real payload
(`catastrophic_prob`/`envelope_score`) and added a parametrized formula test covering the
exact divergence cases the status-based code got wrong, plus missing/garbage-field neutral.
15 passed (live-HZ smoke deselected).
ARCHITECTURAL DEBT documented — VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md:
VIOLET imitates BLUE's computations in a DIFFERENT module/scope structure. Kernels are
WRAPPED (safe), but orchestration arithmetic (compose/regime/ob/strength/mc_scale) is
HAND-REPLICATED out of the monolithic NDAlphaEngine — making orderly, verifiable BLUE↔VIOLET
parity comparison and refactoring hard. Doc inventories every re-derivation with BLUE's
file:line authority, the drift risk, and mitigations (parity-pin tests; single canonical
surface per ambiguous factor; eventual DITAv2 Rust-backplane convergence). Pointer added to
live_blue_source.py's module docstring.
Also reviewed (no change needed): launcher launch_dolphin_violet.py is DARK-safe
(ObserveOnlyVenue + shadow gated default-OFF, no order path); shadow_live_factors.py glue
is correct; trade_slot_compare.py reimplements no BLUE algo. fb34431's launcher tests —
which the authoring agent never ran on the slow mount — pass (14 green).
violet-only; no shared-file edits.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Codex
@@ -13,6 +13,18 @@ translates them into ``SizingFactors`` for the shadow path:
|
||||
|
||||
The remaining DC signal is left neutral here for now. It needs the same live
|
||||
signal-history path BLUE uses and should be added as a separate mirror step.
|
||||
|
||||
PARITY DEBT (see prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md): this module
|
||||
reconstructs BLUE's factors in a DIFFERENT file/scope structure than BLUE's monolithic
|
||||
NDAlphaEngine (esf_alpha_orchestrator.py). Kernels here are WRAPPED (OBFeatureEngine,
|
||||
AlphaSignalGenerator, VioletAssetSelector — single source of truth), but two derivations
|
||||
are HAND-REPLICATED / surface-substituted and can drift silently from BLUE:
|
||||
- ``_derive_mc_scale`` transcribes begin_day's mc thresholds (no pin to BLUE's fn);
|
||||
- ``AlphaSignalGenerator()`` is built with BARE DEFAULTS, not BLUE's threaded params
|
||||
(esf_alpha_orchestrator.py:180-191) — cosmetic only while dc_leverage_boost==1.0;
|
||||
- boost/beta read the PUBLISHED ``acb_boost`` (acb_processor_service), not the trader's
|
||||
``get_dynamic_boost_from_hz`` recompute — the two surfaces may differ.
|
||||
Any change to BLUE's corresponding formula REQUIRES a matching change here + a test.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -72,13 +84,41 @@ def _read_hz_map(client: hazelcast.HazelcastClient, map_name: str, key: str) ->
|
||||
return None
|
||||
|
||||
|
||||
def _map_status_to_mc_scale(payload: Any) -> float:
|
||||
def _derive_mc_scale(payload: Any) -> float:
|
||||
"""Mirror BLUE ``begin_day``'s ``mc_scale`` EXACTLY (esf_alpha_orchestrator.py:956-962).
|
||||
|
||||
BLUE does NOT use the MC service's published ``status`` label for sizing — that label
|
||||
(mc_forewarner_flow.py: GREEN<0.10 / ORANGE<0.30 / RED) is observability-only and uses
|
||||
DIFFERENT thresholds from the engine. The live trader re-derives the size haircut from
|
||||
the SAME published fields ``catastrophic_prob`` + ``envelope_score`` with the engine's
|
||||
own thresholds:
|
||||
|
||||
mc_red = catastrophic_prob > 0.25 or envelope_score < -1.0
|
||||
mc_orange = (not mc_red) and (envelope_score < 0 or catastrophic_prob > 0.10)
|
||||
mc_scale = 0.5 if mc_orange else 1.0 # RED → 1.0 here; BLUE halts via regime_dd_halt
|
||||
|
||||
Reading ``status`` instead diverges (e.g. cat=0.05/env=-0.5 → publisher GREEN→1.0 but
|
||||
BLUE orange→0.5).
|
||||
|
||||
AMBIGUITY (flagged 2026-06-16): the MC service's ``status`` and the engine's begin_day
|
||||
thresholds genuinely disagree, and from OUTSIDE BLUE there is no way to know which is the
|
||||
"nominal" intent — they are two independent threshold sets over the same numbers. We go
|
||||
with the SOURCE FIELDS + begin_day formula because that is the path that actually drives
|
||||
BLUE's live sizing (the published ``status`` is consumed only by the TUI/observability).
|
||||
If BLUE's begin_day thresholds change, THIS must change with them. Operator confirmation
|
||||
of the canonical surface is still desirable.
|
||||
|
||||
Missing/unparseable fields → neutral 1.0 (no haircut)."""
|
||||
data = _jsonish(payload)
|
||||
if isinstance(data, Mapping):
|
||||
status = str(data.get("status", "")).upper()
|
||||
else:
|
||||
status = str(data).upper()
|
||||
return 0.5 if status == "ORANGE" else 1.0
|
||||
if not isinstance(data, Mapping):
|
||||
return 1.0
|
||||
cat = _coerce_float(data.get("catastrophic_prob"), None)
|
||||
env = _coerce_float(data.get("envelope_score"), None)
|
||||
if cat is None or env is None:
|
||||
return 1.0
|
||||
mc_red = cat > 0.25 or env < -1.0
|
||||
mc_orange = (not mc_red) and (env < 0.0 or cat > 0.10)
|
||||
return 0.5 if mc_orange else 1.0
|
||||
|
||||
|
||||
def _extract_acb(payload: Any) -> tuple[float, float]:
|
||||
@@ -302,7 +342,7 @@ def source_live_blue_sizing_factors(
|
||||
acb_boost, acb_beta = _extract_acb(acb_raw)
|
||||
|
||||
mc_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "mc_forewarner_latest")
|
||||
mc_scale = _map_status_to_mc_scale(mc_raw)
|
||||
mc_scale = _derive_mc_scale(mc_raw)
|
||||
|
||||
scan_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "latest_eigen_scan")
|
||||
scan = scan_history.ingest_scan(scan_raw)
|
||||
|
||||
@@ -14,6 +14,7 @@ from prod.clean_arch.violet.decision_engine import SizingFactors
|
||||
from prod.clean_arch.violet.live_blue_source import (
|
||||
HazelcastOBProvider,
|
||||
LiveBlueScanHistory,
|
||||
_derive_mc_scale,
|
||||
source_live_blue_sizing_factors,
|
||||
)
|
||||
from prod.clean_arch.violet.alpha_wrappers import VioletAssetSelector
|
||||
@@ -63,7 +64,7 @@ def test_hz_ob_provider_filters_and_parses_latest_payload():
|
||||
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
|
||||
),
|
||||
"acb_boost": json.dumps({"boost": 1.2, "beta": 0.3}),
|
||||
"mc_forewarner_latest": json.dumps({"status": "ORANGE"}),
|
||||
"mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.15, "envelope_score": 0.5}),
|
||||
"esof_latest": json.dumps({"advisory_score": 0.4}),
|
||||
},
|
||||
"DOLPHIN_STATE_BLUE": {"latest_nautilus": json.dumps({"posture": "restored"})},
|
||||
@@ -107,7 +108,7 @@ def test_source_live_blue_sizing_factors_unit(monkeypatch):
|
||||
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
|
||||
),
|
||||
"acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}),
|
||||
"mc_forewarner_latest": json.dumps({"status": "ORANGE"}),
|
||||
"mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.15, "envelope_score": 0.5}),
|
||||
"esof_latest": json.dumps({"advisory_score": 0.4}),
|
||||
"latest_eigen_scan": json.dumps(
|
||||
{
|
||||
@@ -171,7 +172,7 @@ def test_source_live_blue_sizing_factors_preserves_skip_contradict(monkeypatch):
|
||||
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
|
||||
),
|
||||
"acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}),
|
||||
"mc_forewarner_latest": json.dumps({"status": "GREEN"}),
|
||||
"mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
|
||||
"esof_latest": json.dumps({"advisory_score": 0.4}),
|
||||
"latest_eigen_scan": json.dumps(
|
||||
{
|
||||
@@ -254,7 +255,7 @@ def test_live_blue_sequence_matches_blue_selector_and_dc_at_each_step(monkeypatc
|
||||
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
|
||||
),
|
||||
"acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}),
|
||||
"mc_forewarner_latest": json.dumps({"status": "GREEN"}),
|
||||
"mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
|
||||
"esof_latest": json.dumps({"advisory_score": 0.3}),
|
||||
"latest_eigen_scan": json.dumps({**scan, "target_asset": "BTCUSDT"}),
|
||||
},
|
||||
@@ -314,7 +315,7 @@ def test_live_blue_sequence_rejects_anomalous_values_without_poisoning_history(m
|
||||
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
|
||||
),
|
||||
"acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}),
|
||||
"mc_forewarner_latest": json.dumps({"status": "GREEN"}),
|
||||
"mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
|
||||
"esof_latest": json.dumps({"advisory_score": 0.3}),
|
||||
"latest_eigen_scan": json.dumps(scan),
|
||||
},
|
||||
@@ -351,7 +352,7 @@ def test_source_live_blue_sizing_factors_handles_anomalies(monkeypatch):
|
||||
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
|
||||
),
|
||||
"acb_boost": json.dumps({"boost": -9, "beta": "bad"}),
|
||||
"mc_forewarner_latest": json.dumps({"status": "GREEN"}),
|
||||
"mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
|
||||
"esof_latest": "not json",
|
||||
"latest_eigen_scan": json.dumps({"target_asset": "BTCUSDT", "assets": ["BTCUSDT"], "asset_prices": [0]}),
|
||||
},
|
||||
@@ -370,6 +371,38 @@ def test_source_live_blue_sizing_factors_handles_anomalies(monkeypatch):
|
||||
assert res.selected_asset == "BTCUSDT"
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"cat, env, expected",
|
||||
[
|
||||
# mc_orange (→0.5): not red, and (env<0 OR cat>0.10)
|
||||
(0.15, 0.5, 0.5), # orange via cat>0.10
|
||||
(0.05, -0.5, 0.5), # orange via env<0 — OLD status-code returned GREEN→1.0 (the bug)
|
||||
(0.10, -0.001, 0.5), # orange via env<0 boundary (cat==0.10 not >0.10)
|
||||
# red (→1.0 here; BLUE halts separately via regime_dd_halt)
|
||||
(0.28, 0.5, 1.0), # red via cat>0.25 — OLD status-code: 0.28<0.30→ORANGE→0.5 (the bug)
|
||||
(0.05, -1.5, 1.0), # red via env<-1.0
|
||||
(0.30, -2.0, 1.0), # red via both
|
||||
# ok (→1.0): not red, not orange
|
||||
(0.05, 0.5, 1.0), # benign
|
||||
(0.10, 0.0, 1.0), # cat==0.10 (not >0.10), env==0 (not <0) → ok
|
||||
],
|
||||
)
|
||||
def test_derive_mc_scale_matches_blue_begin_day_formula(cat, env, expected):
|
||||
payload = json.dumps({"catastrophic_prob": cat, "envelope_score": env, "status": "IGNORED"})
|
||||
assert _derive_mc_scale(payload) == expected
|
||||
|
||||
|
||||
def test_derive_mc_scale_neutral_when_fields_missing_or_garbage():
|
||||
# The published `status` label must NOT be used — payloads lacking the source
|
||||
# fields derive neutral (1.0), never a haircut from a stale/foreign label.
|
||||
assert _derive_mc_scale(json.dumps({"status": "ORANGE"})) == 1.0
|
||||
assert _derive_mc_scale(json.dumps({"catastrophic_prob": 0.15})) == 1.0 # missing env
|
||||
assert _derive_mc_scale(json.dumps({"envelope_score": -0.5})) == 1.0 # missing cat
|
||||
assert _derive_mc_scale("not json") == 1.0
|
||||
assert _derive_mc_scale(None) == 1.0
|
||||
assert _derive_mc_scale({"catastrophic_prob": "bad", "envelope_score": 0.5}) == 1.0
|
||||
|
||||
|
||||
def test_live_hz_smoke_reads_current_state():
|
||||
client = hazelcast.HazelcastClient(cluster_name="dolphin", cluster_members=["localhost:5701"])
|
||||
try:
|
||||
|
||||
85
prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md
Normal file
85
prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# VIOLET ↔ BLUE parity: structural divergence & re-derivation debt
|
||||
|
||||
**Date:** 2026-06-16
|
||||
**Status:** ACKNOWLEDGED TRADEOFF / open architectural debt
|
||||
**Raised by:** operator, during the V3.4c review.
|
||||
|
||||
## The problem, stated plainly
|
||||
|
||||
VIOLET reproduces BLUE's sizing behaviour **bit-for-bit by intent**, but it does so in a
|
||||
**different module / file / scope structure** than BLUE. BLUE's logic lives in one place —
|
||||
the monolithic `NDAlphaEngine` (`nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py`),
|
||||
which holds day-state (`_day_base_boost`, `_day_beta`, `_day_mc_scale`, `_day_posture`),
|
||||
constructs its own `signal_gen`/`bet_sizer`, and runs `begin_day` / `_try_entry` inline.
|
||||
|
||||
VIOLET re-expresses that same logic spread across:
|
||||
`sizing.py`, `live_blue_source.py`, `live_factors.py`, `live_factor_source.py`,
|
||||
`decision_engine.py`, `alpha_wrappers.py`.
|
||||
|
||||
**Consequence (the operator's concern, verbatim intent):** because VIOLET imitates the
|
||||
computations *while* ending up with a different structure, any *orderly, systemic,
|
||||
verifiable* BLUE↔VIOLET algo parity comparison — and any future refactor of either side —
|
||||
is **much harder**. The surfaces do not line up 1:1, so a diff between the two engines is
|
||||
not mechanical; it requires a human to know which VIOLET fragment mirrors which BLUE line.
|
||||
|
||||
## Two kinds of reuse — and only one is safe
|
||||
|
||||
1. **WRAPPED kernels (safe — single source of truth).** VIOLET imports and calls BLUE's
|
||||
actual kernel objects. A BLUE change propagates automatically.
|
||||
- `esof_size_mult_from_score`, `parse_esof_payload`, `esof_score_from_payload`
|
||||
(`esof_size_gate.py`) — wrapped by `sizing.py` / `live_factor_source.py`.
|
||||
- `OBFeatureEngine.get_market` (`ob_features.py`) — wrapped by `live_blue_source.py`.
|
||||
- `AlphaSignalGenerator.generate` (`alpha_signal_generator.py`) — wrapped by `live_blue_source.py`.
|
||||
- `AlphaAssetSelector` / `AlphaBetSizer` — wrapped by `alpha_wrappers.py`.
|
||||
- `map_internal_conviction_to_exchange_leverage` (`bingx/leverage.py`) — wrapped by `exchange_leverage.py`.
|
||||
|
||||
2. **HAND-REPLICATED arithmetic (the debt — duplicated formulas, drift-prone).** VIOLET
|
||||
transcribes BLUE's pure float arithmetic into its own functions. A BLUE change here is
|
||||
SILENT in VIOLET until someone notices.
|
||||
|
||||
## Re-derivation inventory (the drift liabilities)
|
||||
|
||||
| Computation | BLUE authority (file:line) | VIOLET replica | Parity safety-net today | Drift risk |
|
||||
|---|---|---|---|---|
|
||||
| 5-factor compose + caps | `esf_alpha_orchestrator.py:600-619` | `sizing.VioletSizer.compose` | `@gate` Monte-Carlo vs REAL orchestrator (bit-identity) | LOW (gated) |
|
||||
| `regime_size_mult` = boost·(1+β·s³)·mc | `esf_alpha_orchestrator.py:898-909` | `sizing.VioletSizer.regime_size_mult` | same gate | LOW-MED |
|
||||
| `strength_cubic` | `esf_alpha_orchestrator.py:872-885` | `sizing.VioletSizer.strength_cubic` | same gate | LOW-MED |
|
||||
| `market_ob_mult` consensus | `esf_alpha_orchestrator.py:587-595` | `sizing.VioletSizer.market_ob_mult` | same gate | MED |
|
||||
| `dc_lev_mult` | `esf_alpha_orchestrator.py:575-577` | `sizing.VioletSizer.dc_lev_mult` | unit only | MED (but ≡1.0 while dc_leverage_boost=1.0) |
|
||||
| **`mc_scale`** | `esf_alpha_orchestrator.py:956-962` (`begin_day`) | `live_blue_source._derive_mc_scale` | **unit only — NO pin to BLUE's fn** | **HIGH** |
|
||||
| `boost`/`beta` source | trader recompute `acb.get_dynamic_boost_from_hz(exf_latest)` | reads published `DOLPHIN_FEATURES.acb_boost` (acb_processor_service) | none | MED (two surfaces may differ on dynamic-β / OB Sub-4) |
|
||||
| `dc_status` config | `signal_gen` built with threaded params `:180-191` | `AlphaSignalGenerator()` **bare defaults** | none | MED (cosmetic while dc_lev_mult≡1.0) |
|
||||
| OB feed shape | live OB accumulation | single-snapshot `HazelcastOBProvider` + `bar_idx=0` | none | MED |
|
||||
|
||||
The worst link is **`mc_scale`**: pure duplication of `begin_day`'s thresholds with no
|
||||
test that pins it to BLUE's actual function (BLUE computes it inline inside `begin_day`,
|
||||
which is not callable in isolation). The V3.4c bug fixed on 2026-06-16 (the adapter keyed
|
||||
off the MC service's `status` label instead of BLUE's `cat`/`env` thresholds) is exactly
|
||||
the failure mode this structure invites.
|
||||
|
||||
## Why we accept it (for now)
|
||||
|
||||
- The kernels that carry the heavy alpha are WRAPPED, not copied.
|
||||
- The composition arithmetic IS gated bit-for-bit against the real orchestrator.
|
||||
- VIOLET must stay a *read-only, DARK* mirror of a *running* BLUE; it cannot import BLUE's
|
||||
live in-process day-state, so some reconstruction from published HZ surfaces is unavoidable.
|
||||
|
||||
## Mitigations (recommended, not yet done)
|
||||
|
||||
1. **Parity-pin every hand-replicated formula.** For each row above, add a test that
|
||||
imports BLUE's authoritative function/constant and asserts VIOLET's replica equals it
|
||||
over a sampled grid — converting silent drift into a red test. Where BLUE's logic is
|
||||
trapped inside `begin_day` (mc_scale), refactor a *pure* `mc_scale_from(cat, env)` helper
|
||||
**on the BLUE side** (BLUE-domain change, operator-gated) that BOTH engines call.
|
||||
2. **Single ambiguity owner.** Surfaces like MC (`status` label vs `begin_day` thresholds)
|
||||
and ACB (published `acb_boost` vs trader recompute) have two disagreeing sources; pick
|
||||
ONE canonical per factor and document it (see `_derive_mc_scale` docstring).
|
||||
3. **Backplane convergence (the real fix).** When the DITAv2 Rust middleware becomes the
|
||||
shared backplane, BOTH BLUE and VIOLET should consume factors from it rather than each
|
||||
computing/replicating — collapsing this divergence at the source. Until then, every new
|
||||
hand-replication MUST be logged in this table.
|
||||
|
||||
## Maintenance rule
|
||||
|
||||
Any change to a BLUE formula in the left column REQUIRES a matching change + test update in
|
||||
the VIOLET replica in the same PR. Any NEW hand-replication MUST add a row here.
|
||||
Reference in New Issue
Block a user