hjnormey/siloqy
Public Access
1
0
Fork 0
Code Issues 7 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

VIOLET V3.4c: make boost/beta, signal-gen, OB bit-identical to BLUE (+ exhaustive tests)

Browse Source 
Operator directive: VIOLET must do IDENTICALLY what BLUE does for the three parity
flags — approximation cannot guarantee bit-for-bit functioning. Reworked
live_blue_source.py to call BLUE's OWN code paths, not reconstruct/substitute them.

boost/beta — was reading the published DOLPHIN_FEATURES.acb_boost scalar
(acb_processor_service's daily value). BLUE's trader does NOT use that for sizing; it
recomputes live via acb.get_dynamic_boost_from_hz(exf_latest, w750_velocity, direction)
with a bare AdaptiveCircuitBreaker() and NO ob_engine (nautilus_event_trader.py
on_exf_update:4769 / rollover prewarm:2710). New _source_boost_beta replicates that call
exactly (reads exf_latest + latest_eigen_scan.w750_velocity; 0.0→None like BLUE; on stale
exf ValueError → neutral, mirroring BLUE's "ACB Stale Data Fallback"). The published
acb_boost is never read. Test pins bit-identity against the real ACB.

signal-gen (dc_status) — was AlphaSignalGenerator() bare defaults; coincidentally equal to
BLUE today, but BLUE builds it from ENGINE_KWARGS (trader:128-133, threaded at
esf_alpha_orchestrator.py:180-191), so a champion retune would silently diverge. Now
constructed with BLUE_SIGNAL_GEN_KWARGS (vel_div_* imported from the kernel constants).
Test parses ENGINE_KWARGS from the trader source and asserts each param matches — drift
becomes a red test, not a silent miss.

OB — was a reinvented HazelcastOBProvider reading asset_*_ob with custom parsing. Now uses
BLUE's OWN HZOBProvider + OBFeatureEngine, wired exactly as _wire_obf
(nautilus_event_trader.py:4967-4980): step_live(assets, bar_idx) then get_market. Engine
is injectable + persistent so OB accumulation matches BLUE across scans (caller owns it).

Deleted: HazelcastOBProvider, _extract_acb, the status-label mc path. mc_scale fix
(begin_day cat/env thresholds) retained. Module docstring + structural-divergence doc
updated: all three flags FIXED; only _derive_mc_scale remains hand-replicated (pinned by
formula test). OPEN follow-up: launcher shadow_decision_step should pass a persistent
ob_engine + bar_idx for cross-scan OB history.

34 tests (33 + live-HZ smoke deselected): boost/beta-vs-ACB bit-identity (incl. w750=0→None,
no-exf, stale ValueError, ignores acb_boost), signal-gen ENGINE_KWARGS pin (parametrized),
OB wiring (step_live call, HZ coords, neutral paths), mc_scale formula, sequence dc/selector
parity, anomaly handling. violet-only; no shared-file edits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Codex
2026-06-16 17:33:01 +02:00
parent 1415a65670
commit fac287d678
3 changed files with 470 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files

277
prod/clean_arch/violet/live_blue_source.py
View File

@@ -5,33 +5,38 @@ This is VIOLET-only. BLUE is untouched.
The adapter reads the published BLUE surfaces that already exist in HZ and
translates them into ``SizingFactors`` for the shadow path:
- ``posture`` from ``DOLPHIN_STATE_BLUE.latest_nautilus`` / ``engine_snapshot``
- ``esof_score`` from ``DOLPHIN_FEATURES.esof_latest`` or ``esof_advisor_latest``
- ``acb_boost`` / ``acb_beta`` from ``DOLPHIN_FEATURES.acb_boost``
- ``mc_scale`` from ``DOLPHIN_FEATURES.mc_forewarner_latest``
- OB market consensus from the live ``asset_*_ob`` maps via BLUE's own
- ``esof_score`` from ``DOLPHIN_FEATURES.esof_latest`` or ``esof_advisor_latest`` via
BLUE's own ``parse_esof_payload`` / ``esof_score_from_payload``
- ``boost`` / ``beta`` RECOMPUTED via ``AdaptiveCircuitBreaker.get_dynamic_boost_from_hz``
over ``DOLPHIN_FEATURES.exf_latest`` + ``latest_eigen_scan.w750_velocity`` — IDENTICAL
to the trader's on_exf_update path (NOT the published ``acb_boost`` scalar)
- ``mc_scale`` from ``DOLPHIN_FEATURES.mc_forewarner_latest`` via ``_derive_mc_scale``
(begin_day's cat/env thresholds, not the MC service's status label)
- OB market consensus from the live ``asset_*_ob`` maps via BLUE's own ``HZOBProvider`` +
``OBFeatureEngine``
- ``dc_status`` via BLUE's ``AlphaSignalGenerator`` (params pinned to BLUE's ENGINE_KWARGS)
over the replayed scan price-history
The remaining DC signal is left neutral here for now. It needs the same live
signal-history path BLUE uses and should be added as a separate mirror step.
PARITY DEBT (see prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md): this module
reconstructs BLUE's factors in a DIFFERENT file/scope structure than BLUE's monolithic
NDAlphaEngine (esf_alpha_orchestrator.py). Kernels here are WRAPPED (OBFeatureEngine,
AlphaSignalGenerator, VioletAssetSelector — single source of truth), but two derivations
are HAND-REPLICATED / surface-substituted and can drift silently from BLUE:
- ``_derive_mc_scale`` transcribes begin_day's mc thresholds (no pin to BLUE's fn);
- ``AlphaSignalGenerator()`` is built with BARE DEFAULTS, not BLUE's threaded params
(esf_alpha_orchestrator.py:180-191) — cosmetic only while dc_leverage_boost==1.0;
- boost/beta read the PUBLISHED ``acb_boost`` (acb_processor_service), not the trader's
``get_dynamic_boost_from_hz`` recompute — the two surfaces may differ.
Any change to BLUE's corresponding formula REQUIRES a matching change here + a test.
PARITY (see prod/docs/VIOLET_BLUE_PARITY_STRUCTURAL_DIVERGENCE.md): this module reconstructs
BLUE's factors in a DIFFERENT file/scope structure than BLUE's monolithic NDAlphaEngine.
Every kernel is now WRAPPED, not copied (ACB, OBFeatureEngine+HZOBProvider, AlphaSignalGenerator,
VioletAssetSelector) — so the only hand-replicated arithmetic left is ``_derive_mc_scale``
(begin_day's thresholds; pinned by test). Remaining REAL fidelity caveats:
- OB faithfulness needs a PERSISTENT ``ob_engine`` + per-scan ``bar_idx`` (OBFeatureEngine
accumulates a lookback window); a single-shot engine has no cross-scan history.
- On stale exf (>12h) the ACB raises ValueError; BLUE keeps the prior boost/beta, VIOLET
has no prior → neutral (1.0, 0.0).
Any change to BLUE's ENGINE_KWARGS or begin_day mc thresholds REQUIRES a matching change +
test here (see test_signal_gen_params_match_blue_engine_kwargs / the mc_scale formula test).
"""
from __future__ import annotations
import json
import os as _os
import sys
from collections import deque
from datetime import datetime, timezone
from collections.abc import Mapping
from dataclasses import dataclass
from dataclasses import field
@@ -47,14 +52,43 @@ for _p in (str(_PROJECT_ROOT), str(_PROJECT_ROOT / "nautilus_dolphin")):
sys.path.insert(0, _p)
from nautilus_dolphin.nautilus.ob_features import OBFeatureEngine
from nautilus_dolphin.nautilus.ob_provider import OBSnapshot, OBProvider
from nautilus_dolphin.nautilus.alpha_signal_generator import AlphaSignalGenerator
from nautilus_dolphin.nautilus.hz_ob_provider import HZOBProvider
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.alpha_signal_generator import (
AlphaSignalGenerator,
VEL_DIV_THRESHOLD, VEL_DIV_EXTREME,
LONG_VEL_DIV_THRESHOLD, LONG_VEL_DIV_EXTREME,
)
from .alpha_wrappers import VioletAssetSelector
from .decision_engine import SizingFactors
from .live_factor_source import esof_score_from_features, posture_from_engine_snapshot
from .live_factors import extract_live_sizing_factors
# Hazelcast coordinates — MUST equal nautilus_event_trader.py:107-108 (BLUE's live
# cluster). HZOBProvider opens its own connection to these, exactly as BLUE's _wire_obf.
HZ_CLUSTER = _os.environ.get("HZ_CLUSTER", "dolphin")
HZ_HOST = _os.environ.get("HZ_HOST", "127.0.0.1:5701")
# AlphaSignalGenerator construction — pinned to BLUE's live ENGINE_KWARGS
# (nautilus_event_trader.py:128-133). These equal AlphaSignalGenerator's own defaults
# TODAY, but BLUE constructs it EXPLICITLY from ENGINE_KWARGS, so a future champion
# retune (e.g. dc_lookback_bars→10) would silently diverge a bare AlphaSignalGenerator().
# We pin explicitly and assert the pin in test_signal_gen_params_match_blue_engine_kwargs.
# vel_div_* import the module constants directly so they auto-track the kernel.
BLUE_SIGNAL_GEN_KWARGS = dict(
vel_div_threshold=VEL_DIV_THRESHOLD, # -0.02
vel_div_extreme=VEL_DIV_EXTREME, # -0.05
long_vel_div_threshold=LONG_VEL_DIV_THRESHOLD, # 0.01
long_vel_div_extreme=LONG_VEL_DIV_EXTREME, # 0.04
dc_lookback_bars=7,
dc_min_magnitude_bps=0.75,
dc_skip_contradicts=True,
dc_leverage_boost=1.0,
dc_leverage_reduce=0.5,
use_direction_confirm=True,
)
def _jsonish(value: Any) -> Any:
if isinstance(value, str):
@@ -121,13 +155,48 @@ def _derive_mc_scale(payload: Any) -> float:
return 0.5 if mc_orange else 1.0
def _extract_acb(payload: Any) -> tuple[float, float]:
data = _jsonish(payload)
if isinstance(data, Mapping):
boost = _coerce_float(data.get("boost"), 1.0) or 1.0
beta = _coerce_float(data.get("beta"), 0.0) or 0.0
return max(0.0, boost), max(0.0, beta)
return 1.0, 0.0
def _source_boost_beta(
client: hazelcast.HazelcastClient,
*,
date_str: str,
trade_direction: int,
acb: Optional[AdaptiveCircuitBreaker] = None,
) -> tuple[float, float]:
"""Recompute (boost, beta) EXACTLY as BLUE's live trader does — NOT from acb_boost.
BLUE (nautilus_event_trader.py on_exf_update:4769 / rollover prewarm:2710):
acb = AdaptiveCircuitBreaker() # bare, no args (trader 578/585)
info = acb.get_dynamic_boost_from_hz(
date_str=today,
exf_snapshot=DOLPHIN_FEATURES['exf_latest'],
w750_velocity=latest_eigen_scan['w750_velocity'] or None,
direction=trade_direction, # NO ob_engine in the live path
)
boost, beta = info['boost'], info['beta']
The published ``DOLPHIN_FEATURES['acb_boost']`` is a SEPARATE daily publish
(acb_processor_service) and is NOT what drives BLUE's sizing, so we never read it.
On stale exf (>12h) get_dynamic_boost_from_hz raises ValueError; BLUE logs
'ACB Stale Data Fallback' and keeps the prior boost/beta — VIOLET has no prior, so it
returns the neutral identity (1.0, 0.0)."""
exf = _jsonish(_read_hz_map(client, "DOLPHIN_FEATURES", "exf_latest"))
if not isinstance(exf, Mapping):
return 1.0, 0.0
eigen = _jsonish(_read_hz_map(client, "DOLPHIN_FEATURES", "latest_eigen_scan"))
w750 = _coerce_float(eigen.get("w750_velocity"), None) if isinstance(eigen, Mapping) else None
acb = acb or AdaptiveCircuitBreaker()
try:
info = acb.get_dynamic_boost_from_hz(
date_str=date_str,
exf_snapshot=dict(exf),
w750_velocity=float(w750) if w750 else None, # 0.0 → None, matches BLUE
direction=trade_direction,
)
except ValueError:
return 1.0, 0.0 # ACB Stale Data Fallback (BLUE keeps prior; VIOLET neutral)
boost = _coerce_float(info.get("boost"), 1.0) or 1.0
beta = _coerce_float(info.get("beta"), 0.0) or 0.0
return max(0.0, boost), max(0.0, beta)
def _scan_view(payload: Any) -> Mapping[str, Any]:
@@ -209,7 +278,12 @@ class LiveBlueScanHistory:
history = self.price_history(asset[0])
if not history:
return "NONE"
signal_gen = AlphaSignalGenerator()
# BLUE constructs its signal_gen from ENGINE_KWARGS (the orchestrator threads them,
# esf_alpha_orchestrator.py:180-191). Pin the SAME params — not bare defaults — so a
# champion retune that changes dc_lookback_bars / dc_min_magnitude_bps / thresholds
# diverges loudly (caught by test_signal_gen_params_match_blue_engine_kwargs), never
# silently. dc_status is deterministic in the params + price history (counters unused).
signal_gen = AlphaSignalGenerator(**BLUE_SIGNAL_GEN_KWARGS)
sig = signal_gen.generate(
vel_div=vel_div,
vel_div_history=None,
@@ -221,80 +295,37 @@ class LiveBlueScanHistory:
return sig.dc_status
class HazelcastOBProvider(OBProvider):
"""Read the current BLUE OB shards directly from Hazelcast."""
def _source_ob_market(
ob_assets: list[str],
*,
bar_idx: int,
ob_engine: Optional[OBFeatureEngine] = None,
) -> tuple[Optional[float], Optional[float]]:
"""Derive (median_imbalance, agreement_pct) EXACTLY as BLUE does, via BLUE's HZOBProvider.
def __init__(self, client: hazelcast.HazelcastClient):
self.client = client
BLUE wires OB once in _wire_obf (nautilus_event_trader.py:4967-4980):
live_ob = HZOBProvider(hz_cluster=HZ_CLUSTER, hz_host=HZ_HOST, assets=assets)
ob_eng = OBFeatureEngine(live_ob); eng.set_ob_engine(ob_eng)
then per scan calls ``ob_eng.step_live(assets, bar_idx)`` and the orchestrator reads
``ob_eng.get_market(bar_idx, assets)`` (esf_alpha_orchestrator.py:590). We use BLUE's
OWN HZOBProvider — NOT a reinvented reader — so OB parsing/shard semantics are BLUE's.
def _asset_keys(self) -> list[str]:
try:
keys = self.client.get_map("DOLPHIN_FEATURES").blocking().key_set()
except Exception:
return []
assets = []
for key in keys:
if not isinstance(key, str) or not key.startswith("asset_") or not key.endswith("_ob"):
continue
asset = key[len("asset_"):-len("_ob")]
if asset and asset not in assets:
assets.append(asset)
return sorted(assets)
def _read_snapshot(self, asset: str) -> Optional[OBSnapshot]:
raw = _read_hz_map(self.client, "DOLPHIN_FEATURES", f"asset_{asset}_ob")
data = _jsonish(raw)
if not isinstance(data, Mapping):
return None
bid_notional = np.array(
[_coerce_float(v, 0.0) or 0.0 for v in data.get("bid_notional", [0, 0, 0, 0, 0])][:5],
dtype=np.float64,
)
ask_notional = np.array(
[_coerce_float(v, 0.0) or 0.0 for v in data.get("ask_notional", [0, 0, 0, 0, 0])][:5],
dtype=np.float64,
)
bid_depth = np.array(
[_coerce_float(v, 0.0) or 0.0 for v in data.get("bid_depth", [0, 0, 0, 0, 0])][:5],
dtype=np.float64,
)
ask_depth = np.array(
[_coerce_float(v, 0.0) or 0.0 for v in data.get("ask_depth", [0, 0, 0, 0, 0])][:5],
dtype=np.float64,
)
ts = _coerce_float(data.get("timestamp"), 0.0) or 0.0
if (
bid_notional.shape != (5,) or ask_notional.shape != (5,)
or bid_depth.shape != (5,) or ask_depth.shape != (5,)
):
return None
if np.any(bid_notional < 0) or np.any(ask_notional < 0):
return None
if np.any(bid_depth < 0) or np.any(ask_depth < 0):
return None
return OBSnapshot(
timestamp=ts,
asset=asset,
bid_notional=bid_notional,
ask_notional=ask_notional,
bid_depth=bid_depth,
ask_depth=ask_depth,
)
def get_snapshot(self, asset: str, timestamp: float) -> Optional[OBSnapshot]:
return self._read_snapshot(asset)
def get_assets(self) -> list[str]:
return self._asset_keys()
def get_all_timestamps(self, asset: str) -> np.ndarray:
snap = self._read_snapshot(asset)
if snap is None:
return np.array([], dtype=np.float64)
return np.array([snap.timestamp], dtype=np.float64)
def get_snapshot_count(self, asset: str) -> int:
return 1 if self._read_snapshot(asset) is not None else 0
OBFeatureEngine ACCUMULATES per-asset history across scans (lookback=10), so a faithful
mirror requires a PERSISTENT ``ob_engine`` + per-scan-incrementing ``bar_idx`` (the
caller/shadow loop owns it, exactly as BLUE keeps one ob_eng). When no engine is passed
this builds a single-shot HZOBProvider-backed engine — correct wiring but no cross-scan
history; use only for one-off reads / the live smoke."""
if not ob_assets:
return None, None
if ob_engine is None:
provider = HZOBProvider(hz_cluster=HZ_CLUSTER, hz_host=HZ_HOST, assets=list(ob_assets))
ob_engine = OBFeatureEngine(provider)
try:
ob_engine.step_live(list(ob_assets), bar_idx)
market = ob_engine.get_market(bar_idx, list(ob_assets))
return float(market.median_imbalance), float(market.agreement_pct)
except Exception:
return None, None
@dataclass(frozen=True)
@@ -314,10 +345,22 @@ def source_live_blue_sizing_factors(
assets: Optional[Iterable[str]] = None,
scan_history: Optional[LiveBlueScanHistory] = None,
selector: Optional[VioletAssetSelector] = None,
acb: Optional[AdaptiveCircuitBreaker] = None,
ob_engine: Optional[OBFeatureEngine] = None,
bar_idx: int = 0,
date_str: Optional[str] = None,
) -> LiveBlueSourceResult:
"""Read the BLUE-published live surfaces and return a typed factor plane."""
"""Read BLUE's live surfaces and RECONSTRUCT the factor plane the way BLUE computes it.
boost/beta are recomputed via ``AdaptiveCircuitBreaker.get_dynamic_boost_from_hz`` (NOT
the published acb_boost scalar); OB via BLUE's ``HZOBProvider`` + ``OBFeatureEngine``;
dc_status via ``AlphaSignalGenerator`` pinned to BLUE's ENGINE_KWARGS. For full OB
accumulation faithfulness the caller passes a PERSISTENT ``ob_engine`` + per-scan
``bar_idx`` (BLUE keeps one ob_eng). ``date_str`` defaults to today's UTC date (BLUE
uses self.current_day)."""
scan_history = scan_history or LiveBlueScanHistory()
selector = selector or VioletAssetSelector()
today = date_str or datetime.now(timezone.utc).strftime("%Y-%m-%d")
engine_snapshot_raw = _read_hz_map(client, "DOLPHIN_STATE_BLUE", "latest_nautilus")
if engine_snapshot_raw is None:
@@ -336,11 +379,6 @@ def source_live_blue_sizing_factors(
esof_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "esof_advisor_latest")
esof_score = esof_score_from_features(esof_raw)
acb_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "acb_boost")
if acb_raw is None:
acb_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "acb_boost_short")
acb_boost, acb_beta = _extract_acb(acb_raw)
mc_raw = _read_hz_map(client, "DOLPHIN_FEATURES", "mc_forewarner_latest")
mc_scale = _derive_mc_scale(mc_raw)
@@ -358,26 +396,23 @@ def source_live_blue_sizing_factors(
)
or -1
)
# boost/beta — recompute via the ACB exactly as BLUE's trader does (NOT the published
# acb_boost). Needs trade_direction, so computed after it.
acb_boost, acb_beta = _source_boost_beta(
client, date_str=today, trade_direction=trade_direction, acb=acb,
)
candidate_market = scan_history.market_data(selector.lookback)
pick = selector.pick(candidate_market, regime_direction=trade_direction)
selected_asset = pick.asset if pick is not None else (scan_assets[0] if scan_assets else "")
dc_status = scan_history.dc_status(scan, already_ingested=True)
ob_provider = HazelcastOBProvider(client)
ob_engine = OBFeatureEngine(ob_provider)
ob_assets = list(assets) if assets is not None else (scan_assets or ob_provider.get_assets())
if ob_assets:
try:
ob_engine.step_live(ob_assets, bar_idx=0)
market = ob_engine.get_market(0, ob_assets)
ob_median_imbalance = float(market.median_imbalance)
ob_agreement_pct = float(market.agreement_pct)
except Exception:
ob_median_imbalance = None
ob_agreement_pct = None
else:
ob_median_imbalance = None
ob_agreement_pct = None
# OB market consensus — BLUE's HZOBProvider + OBFeatureEngine (persistent engine if given).
ob_assets = list(assets) if assets is not None else scan_assets
ob_median_imbalance, ob_agreement_pct = _source_ob_market(
ob_assets, bar_idx=bar_idx, ob_engine=ob_engine,
)
hz_snapshot = {
"boost": acb_boost,