VIOLET V3.4c review: fix mc_scale BLUE-parity bug + document re-derivation debt

REVIEW of the V3.4c/d/e work (other agents) for BLUE-algo compliance found one
real bug and one architectural concern.

BUG FIXED — mc_scale derivation (live_blue_source.py):
The adapter mapped `mc_forewarner_latest.status == "ORANGE"` → 0.5. But BLUE's sizing
mc_scale is NOT the MC service's `status` label (that label, from mc_forewarner_flow.py,
uses GREEN<0.10/ORANGE<0.30/RED — observability only). The live trader re-derives the
haircut in begin_day (esf_alpha_orchestrator.py:956-962) from the SAME published fields
`catastrophic_prob` + `envelope_score` with DIFFERENT thresholds:
    mc_red    = cat>0.25 or env<-1.0
    mc_orange = (not mc_red) and (env<0 or cat>0.10)
    mc_scale  = 0.5 if mc_orange else 1.0
The two disagree (e.g. cat=0.05/env=-0.5 → label GREEN→1.0 but BLUE orange→0.5;
cat=0.28 → label ORANGE→0.5 but BLUE red→1.0). Rewrote `_derive_mc_scale` to mirror
begin_day exactly on the source fields. Per operator: there is genuine ambiguity over
which surface is "nominal" — we go with the SOURCE FIELDS + begin_day formula (the path
that actually drives BLUE's sizing); the ambiguity + decision are noted in the docstring.

TESTS — the old fixtures were complicit: they fed `{"status":"ORANGE"}`, a payload shape
BLUE never emits, so they "passed" against fiction. Replaced with BLUE's real payload
(`catastrophic_prob`/`envelope_score`) and added a parametrized formula test covering the
exact divergence cases the status-based code got wrong, plus missing/garbage-field neutral.
15 passed (live-HZ smoke deselected).

ARCHITECTURAL DEBT documented — VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md:
VIOLET imitates BLUE's computations in a DIFFERENT module/scope structure. Kernels are
WRAPPED (safe), but orchestration arithmetic (compose/regime/ob/strength/mc_scale) is
HAND-REPLICATED out of the monolithic NDAlphaEngine — making orderly, verifiable BLUE↔VIOLET
parity comparison and refactoring hard. Doc inventories every re-derivation with BLUE's
file:line authority, the drift risk, and mitigations (parity-pin tests; single canonical
surface per ambiguous factor; eventual DITAv2 Rust-backplane convergence). Pointer added to
live_blue_source.py's module docstring.

Also reviewed (no change needed): launcher launch_dolphin_violet.py is DARK-safe
(ObserveOnlyVenue + shadow gated default-OFF, no order path); shadow_live_factors.py glue
is correct; trade_slot_compare.py reimplements no BLUE algo. fb34431's launcher tests —
which the authoring agent never ran on the slow mount — pass (14 green).

violet-only; no shared-file edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-06-16 16:54:35 +02:00
parent 520d911722
commit 1415a65670
3 changed files with 171 additions and 13 deletions

View File

@@ -13,6 +13,18 @@ translates them into ``SizingFactors`` for the shadow path:
The remaining DC signal is left neutral here for now. It needs the same live The remaining DC signal is left neutral here for now. It needs the same live
signal-history path BLUE uses and should be added as a separate mirror step. signal-history path BLUE uses and should be added as a separate mirror step.
PARITY DEBT (see prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md): this module
reconstructs BLUE's factors in a DIFFERENT file/scope structure than BLUE's monolithic
NDAlphaEngine (esf_alpha_orchestrator.py). Kernels here are WRAPPED (OBFeatureEngine,
AlphaSignalGenerator, VioletAssetSelector — single source of truth), but two derivations
are HAND-REPLICATED / surface-substituted and can drift silently from BLUE:
- ``_derive_mc_scale`` transcribes begin_day's mc thresholds (no pin to BLUE's fn);
- ``AlphaSignalGenerator()`` is built with BARE DEFAULTS, not BLUE's threaded params
(esf_alpha_orchestrator.py:180-191) — cosmetic only while dc_leverage_boost==1.0;
- boost/beta read the PUBLISHED ``acb_boost`` (acb_processor_service), not the trader's
``get_dynamic_boost_from_hz`` recompute — the two surfaces may differ.
Any change to BLUE's corresponding formula REQUIRES a matching change here + a test.
""" """
from __future__ import annotations from __future__ import annotations
@@ -72,13 +84,41 @@ def _read_hz_map(client: hazelcast.HazelcastClient, map_name: str, key: str) ->
return None return None
def _map_status_to_mc_scale(payload: Any) -> float: def _derive_mc_scale(payload: Any) -> float:
"""Mirror BLUE ``begin_day``'s ``mc_scale`` EXACTLY (esf_alpha_orchestrator.py:956-962).
BLUE does NOT use the MC service's published ``status`` label for sizing — that label
(mc_forewarner_flow.py: GREEN<0.10 / ORANGE<0.30 / RED) is observability-only and uses
DIFFERENT thresholds from the engine. The live trader re-derives the size haircut from
the SAME published fields ``catastrophic_prob`` + ``envelope_score`` with the engine's
own thresholds:
mc_red = catastrophic_prob > 0.25 or envelope_score < -1.0
mc_orange = (not mc_red) and (envelope_score < 0 or catastrophic_prob > 0.10)
mc_scale = 0.5 if mc_orange else 1.0 # RED → 1.0 here; BLUE halts via regime_dd_halt
Reading ``status`` instead diverges (e.g. cat=0.05/env=-0.5 → publisher GREEN→1.0 but
BLUE orange→0.5).
AMBIGUITY (flagged 2026-06-16): the MC service's ``status`` and the engine's begin_day
thresholds genuinely disagree, and from OUTSIDE BLUE there is no way to know which is the
"nominal" intent — they are two independent threshold sets over the same numbers. We go
with the SOURCE FIELDS + begin_day formula because that is the path that actually drives
BLUE's live sizing (the published ``status`` is consumed only by the TUI/observability).
If BLUE's begin_day thresholds change, THIS must change with them. Operator confirmation
of the canonical surface is still desirable.
Missing/unparseable fields → neutral 1.0 (no haircut)."""
data = _jsonish(payload) data = _jsonish(payload)
if isinstance(data, Mapping): if not isinstance(data, Mapping):
status = str(data.get("status", "")).upper() return 1.0
else: cat = _coerce_float(data.get("catastrophic_prob"), None)
status = str(data).upper() env = _coerce_float(data.get("envelope_score"), None)
return 0.5 if status == "ORANGE" else 1.0 if cat is None or env is None:
return 1.0
mc_red = cat > 0.25 or env < -1.0
mc_orange = (not mc_red) and (env < 0.0 or cat > 0.10)
return 0.5 if mc_orange else 1.0
def _extract_acb(payload: Any) -> tuple[float, float]: def _extract_acb(payload: Any) -> tuple[float, float]:
@@ -302,7 +342,7 @@ def source_live_blue_sizing_factors(
acb_boost, acb_beta = _extract_acb(acb_raw) acb_boost, acb_beta = _extract_acb(acb_raw)
mc_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "mc_forewarner_latest") mc_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "mc_forewarner_latest")
mc_scale = _map_status_to_mc_scale(mc_raw) mc_scale = _derive_mc_scale(mc_raw)
scan_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "latest_eigen_scan") scan_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "latest_eigen_scan")
scan = scan_history.ingest_scan(scan_raw) scan = scan_history.ingest_scan(scan_raw)

View File

@@ -14,6 +14,7 @@ from prod.clean_arch.violet.decision_engine import SizingFactors
from prod.clean_arch.violet.live_blue_source import ( from prod.clean_arch.violet.live_blue_source import (
HazelcastOBProvider, HazelcastOBProvider,
LiveBlueScanHistory, LiveBlueScanHistory,
_derive_mc_scale,
source_live_blue_sizing_factors, source_live_blue_sizing_factors,
) )
from prod.clean_arch.violet.alpha_wrappers import VioletAssetSelector from prod.clean_arch.violet.alpha_wrappers import VioletAssetSelector
@@ -63,7 +64,7 @@ def test_hz_ob_provider_filters_and_parses_latest_payload():
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
), ),
"acb_boost": json.dumps({"boost": 1.2, "beta": 0.3}), "acb_boost": json.dumps({"boost": 1.2, "beta": 0.3}),
"mc_forewarner_latest": json.dumps({"status": "ORANGE"}), "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.15, "envelope_score": 0.5}),
"esof_latest": json.dumps({"advisory_score": 0.4}), "esof_latest": json.dumps({"advisory_score": 0.4}),
}, },
"DOLPHIN_STATE_BLUE": {"latest_nautilus": json.dumps({"posture": "restored"})}, "DOLPHIN_STATE_BLUE": {"latest_nautilus": json.dumps({"posture": "restored"})},
@@ -107,7 +108,7 @@ def test_source_live_blue_sizing_factors_unit(monkeypatch):
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
), ),
"acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}), "acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}),
"mc_forewarner_latest": json.dumps({"status": "ORANGE"}), "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.15, "envelope_score": 0.5}),
"esof_latest": json.dumps({"advisory_score": 0.4}), "esof_latest": json.dumps({"advisory_score": 0.4}),
"latest_eigen_scan": json.dumps( "latest_eigen_scan": json.dumps(
{ {
@@ -171,7 +172,7 @@ def test_source_live_blue_sizing_factors_preserves_skip_contradict(monkeypatch):
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
), ),
"acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}), "acb_boost": json.dumps({"boost": 1.4, "beta": 0.2}),
"mc_forewarner_latest": json.dumps({"status": "GREEN"}), "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
"esof_latest": json.dumps({"advisory_score": 0.4}), "esof_latest": json.dumps({"advisory_score": 0.4}),
"latest_eigen_scan": json.dumps( "latest_eigen_scan": json.dumps(
{ {
@@ -254,7 +255,7 @@ def test_live_blue_sequence_matches_blue_selector_and_dc_at_each_step(monkeypatc
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
), ),
"acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}), "acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}),
"mc_forewarner_latest": json.dumps({"status": "GREEN"}), "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
"esof_latest": json.dumps({"advisory_score": 0.3}), "esof_latest": json.dumps({"advisory_score": 0.3}),
"latest_eigen_scan": json.dumps({**scan, "target_asset": "BTCUSDT"}), "latest_eigen_scan": json.dumps({**scan, "target_asset": "BTCUSDT"}),
}, },
@@ -314,7 +315,7 @@ def test_live_blue_sequence_rejects_anomalous_values_without_poisoning_history(m
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
), ),
"acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}), "acb_boost": json.dumps({"boost": 1.0, "beta": 0.0}),
"mc_forewarner_latest": json.dumps({"status": "GREEN"}), "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
"esof_latest": json.dumps({"advisory_score": 0.3}), "esof_latest": json.dumps({"advisory_score": 0.3}),
"latest_eigen_scan": json.dumps(scan), "latest_eigen_scan": json.dumps(scan),
}, },
@@ -351,7 +352,7 @@ def test_source_live_blue_sizing_factors_handles_anomalies(monkeypatch):
"bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]} "bid_depth": [1, 1, 1, 1, 1], "ask_depth": [1, 1, 1, 1, 1]}
), ),
"acb_boost": json.dumps({"boost": -9, "beta": "bad"}), "acb_boost": json.dumps({"boost": -9, "beta": "bad"}),
"mc_forewarner_latest": json.dumps({"status": "GREEN"}), "mc_forewarner_latest": json.dumps({"catastrophic_prob": 0.02, "envelope_score": 0.9}),
"esof_latest": "not json", "esof_latest": "not json",
"latest_eigen_scan": json.dumps({"target_asset": "BTCUSDT", "assets": ["BTCUSDT"], "asset_prices": [0]}), "latest_eigen_scan": json.dumps({"target_asset": "BTCUSDT", "assets": ["BTCUSDT"], "asset_prices": [0]}),
}, },
@@ -370,6 +371,38 @@ def test_source_live_blue_sizing_factors_handles_anomalies(monkeypatch):
assert res.selected_asset == "BTCUSDT" assert res.selected_asset == "BTCUSDT"
@pytest.mark.parametrize(
"cat, env, expected",
[
# mc_orange (→0.5): not red, and (env<0 OR cat>0.10)
(0.15, 0.5, 0.5), # orange via cat>0.10
(0.05, -0.5, 0.5), # orange via env<0 — OLD status-code returned GREEN→1.0 (the bug)
(0.10, -0.001, 0.5), # orange via env<0 boundary (cat==0.10 not >0.10)
# red (→1.0 here; BLUE halts separately via regime_dd_halt)
(0.28, 0.5, 1.0), # red via cat>0.25 — OLD status-code: 0.28<0.30→ORANGE→0.5 (the bug)
(0.05, -1.5, 1.0), # red via env<-1.0
(0.30, -2.0, 1.0), # red via both
# ok (→1.0): not red, not orange
(0.05, 0.5, 1.0), # benign
(0.10, 0.0, 1.0), # cat==0.10 (not >0.10), env==0 (not <0) → ok
],
)
def test_derive_mc_scale_matches_blue_begin_day_formula(cat, env, expected):
payload = json.dumps({"catastrophic_prob": cat, "envelope_score": env, "status": "IGNORED"})
assert _derive_mc_scale(payload) == expected
def test_derive_mc_scale_neutral_when_fields_missing_or_garbage():
# The published `status` label must NOT be used — payloads lacking the source
# fields derive neutral (1.0), never a haircut from a stale/foreign label.
assert _derive_mc_scale(json.dumps({"status": "ORANGE"})) == 1.0
assert _derive_mc_scale(json.dumps({"catastrophic_prob": 0.15})) == 1.0 # missing env
assert _derive_mc_scale(json.dumps({"envelope_score": -0.5})) == 1.0 # missing cat
assert _derive_mc_scale("not json") == 1.0
assert _derive_mc_scale(None) == 1.0
assert _derive_mc_scale({"catastrophic_prob": "bad", "envelope_score": 0.5}) == 1.0
def test_live_hz_smoke_reads_current_state(): def test_live_hz_smoke_reads_current_state():
client = hazelcast.HazelcastClient(cluster_name="dolphin", cluster_members=["localhost:5701"]) client = hazelcast.HazelcastClient(cluster_name="dolphin", cluster_members=["localhost:5701"])
try: try:

View File

@@ -0,0 +1,85 @@
# VIOLET ↔ BLUE parity: structural divergence & re-derivation debt
**Date:** 2026-06-16
**Status:** ACKNOWLEDGED TRADEOFF / open architectural debt
**Raised by:** operator, during the V3.4c review.
## The problem, stated plainly
VIOLET reproduces BLUE's sizing behaviour **bit-for-bit by intent**, but it does so in a
**different module / file / scope structure** than BLUE. BLUE's logic lives in one place —
the monolithic `NDAlphaEngine` (`nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py`),
which holds day-state (`_day_base_boost`, `_day_beta`, `_day_mc_scale`, `_day_posture`),
constructs its own `signal_gen`/`bet_sizer`, and runs `begin_day` / `_try_entry` inline.
VIOLET re-expresses that same logic spread across:
`sizing.py`, `live_blue_source.py`, `live_factors.py`, `live_factor_source.py`,
`decision_engine.py`, `alpha_wrappers.py`.
**Consequence (the operator's concern, verbatim intent):** because VIOLET imitates the
computations *while* ending up with a different structure, any *orderly, systemic,
verifiable* BLUE↔VIOLET algo parity comparison — and any future refactor of either side —
is **much harder**. The surfaces do not line up 1:1, so a diff between the two engines is
not mechanical; it requires a human to know which VIOLET fragment mirrors which BLUE line.
## Two kinds of reuse — and only one is safe
1. **WRAPPED kernels (safe — single source of truth).** VIOLET imports and calls BLUE's
actual kernel objects. A BLUE change propagates automatically.
- `esof_size_mult_from_score`, `parse_esof_payload`, `esof_score_from_payload`
(`esof_size_gate.py`) — wrapped by `sizing.py` / `live_factor_source.py`.
- `OBFeatureEngine.get_market` (`ob_features.py`) — wrapped by `live_blue_source.py`.
- `AlphaSignalGenerator.generate` (`alpha_signal_generator.py`) — wrapped by `live_blue_source.py`.
- `AlphaAssetSelector` / `AlphaBetSizer` — wrapped by `alpha_wrappers.py`.
- `map_internal_conviction_to_exchange_leverage` (`bingx/leverage.py`) — wrapped by `exchange_leverage.py`.
2. **HAND-REPLICATED arithmetic (the debt — duplicated formulas, drift-prone).** VIOLET
transcribes BLUE's pure float arithmetic into its own functions. A BLUE change here is
SILENT in VIOLET until someone notices.
## Re-derivation inventory (the drift liabilities)
| Computation | BLUE authority (file:line) | VIOLET replica | Parity safety-net today | Drift risk |
|---|---|---|---|---|
| 5-factor compose + caps | `esf_alpha_orchestrator.py:600-619` | `sizing.VioletSizer.compose` | `@gate` Monte-Carlo vs REAL orchestrator (bit-identity) | LOW (gated) |
| `regime_size_mult` = boost·(1+β·s³)·mc | `esf_alpha_orchestrator.py:898-909` | `sizing.VioletSizer.regime_size_mult` | same gate | LOW-MED |
| `strength_cubic` | `esf_alpha_orchestrator.py:872-885` | `sizing.VioletSizer.strength_cubic` | same gate | LOW-MED |
| `market_ob_mult` consensus | `esf_alpha_orchestrator.py:587-595` | `sizing.VioletSizer.market_ob_mult` | same gate | MED |
| `dc_lev_mult` | `esf_alpha_orchestrator.py:575-577` | `sizing.VioletSizer.dc_lev_mult` | unit only | MED (but ≡1.0 while dc_leverage_boost=1.0) |
| **`mc_scale`** | `esf_alpha_orchestrator.py:956-962` (`begin_day`) | `live_blue_source._derive_mc_scale` | **unit only — NO pin to BLUE's fn** | **HIGH** |
| `boost`/`beta` source | trader recompute `acb.get_dynamic_boost_from_hz(exf_latest)` | reads published `DOLPHIN_FEATURES.acb_boost` (acb_processor_service) | none | MED (two surfaces may differ on dynamic-β / OB Sub-4) |
| `dc_status` config | `signal_gen` built with threaded params `:180-191` | `AlphaSignalGenerator()` **bare defaults** | none | MED (cosmetic while dc_lev_mult≡1.0) |
| OB feed shape | live OB accumulation | single-snapshot `HazelcastOBProvider` + `bar_idx=0` | none | MED |
The worst link is **`mc_scale`**: pure duplication of `begin_day`'s thresholds with no
test that pins it to BLUE's actual function (BLUE computes it inline inside `begin_day`,
which is not callable in isolation). The V3.4c bug fixed on 2026-06-16 (the adapter keyed
off the MC service's `status` label instead of BLUE's `cat`/`env` thresholds) is exactly
the failure mode this structure invites.
## Why we accept it (for now)
- The kernels that carry the heavy alpha are WRAPPED, not copied.
- The composition arithmetic IS gated bit-for-bit against the real orchestrator.
- VIOLET must stay a *read-only, DARK* mirror of a *running* BLUE; it cannot import BLUE's
live in-process day-state, so some reconstruction from published HZ surfaces is unavoidable.
## Mitigations (recommended, not yet done)
1. **Parity-pin every hand-replicated formula.** For each row above, add a test that
imports BLUE's authoritative function/constant and asserts VIOLET's replica equals it
over a sampled grid — converting silent drift into a red test. Where BLUE's logic is
trapped inside `begin_day` (mc_scale), refactor a *pure* `mc_scale_from(cat, env)` helper
**on the BLUE side** (BLUE-domain change, operator-gated) that BOTH engines call.
2. **Single ambiguity owner.** Surfaces like MC (`status` label vs `begin_day` thresholds)
and ACB (published `acb_boost` vs trader recompute) have two disagreeing sources; pick
ONE canonical per factor and document it (see `_derive_mc_scale` docstring).
3. **Backplane convergence (the real fix).** When the DITAv2 Rust middleware becomes the
shared backplane, BOTH BLUE and VIOLET should consume factors from it rather than each
computing/replicating — collapsing this divergence at the source. Until then, every new
hand-replication MUST be logged in this table.
## Maintenance rule
Any change to a BLUE formula in the left column REQUIRES a matching change + test update in
the VIOLET replica in the same PR. Any NEW hand-replication MUST add a row here.