Files
DOLPHIN/prod/docs/SYSTEM_BIBLE_v1_MIG7_20260307.md

1105 lines
41 KiB
Markdown
Raw Normal View History

# DOLPHIN-NAUTILUS SYSTEM BIBLE
## Doctrinal Reference — As Running 2026-03-07
**Version**: MIG7 Complete (post-MIG7_MIG8_FIXES)
**CI gate**: 14/14 tests green
**Status**: Paper trading ready. NOT deployed with real capital.
---
## TABLE OF CONTENTS
1. [System Philosophy](#1-system-philosophy)
2. [Physical Architecture](#2-physical-architecture)
3. [Data Layer](#3-data-layer)
4. [Signal Layer — vel_div & DC](#4-signal-layer)
5. [Asset Selection — IRP](#5-asset-selection-irp)
6. [Position Sizing — AlphaBetSizer](#6-position-sizing)
7. [Exit Management](#7-exit-management)
8. [Fee & Slippage Model](#8-fee--slippage-model)
9. [OB Intelligence Layer](#9-ob-intelligence-layer)
10. [ACB v6 — Adaptive Circuit Breaker](#10-acb-v6)
11. [Survival Stack — Posture Control](#11-survival-stack)
12. [MC-Forewarner Envelope Gate](#12-mc-forewarner-envelope-gate)
13. [NDAlphaEngine — Full Bar Loop](#13-ndalpha-engine-full-bar-loop)
14. [DolphinActor — Nautilus Integration](#14-dolphin-actor)
15. [Hazelcast Feature Store — Sharding](#15-hazelcast-feature-store)
16. [Production Daemon Topology](#16-production-daemon-topology)
17. [CI Test Suite](#17-ci-test-suite)
18. [Parameter Reference](#18-parameter-reference)
19. [Known Research TODOs](#19-known-research-todos)
---
## 1. SYSTEM PHILOSOPHY
DOLPHIN-NAUTILUS is a **SHORT-only** (champion configuration) systematic trading engine targeting crypto perpetual futures on Binance.
**Core thesis**: When crypto market correlation matrices show accelerating eigenvalue-velocity divergence (`vel_div < -0.02`), the market is entering an instability regime. Shorting during early instability onset and exiting at fixed take-profit captures the mean-reversion from panic to normalization.
**Design constraints**:
- Zero signal re-implementation in the Nautilus layer. All alpha logic lives in `NDAlphaEngine`.
- 512-bit arithmetic for correlation matrix processing (separate NG3 pipeline; not in hot path of this engine).
- Champion parameters are FROZEN. They were validated via exhaustive VBT backtest on `dolphin_vbt_real.py`.
- The Nautilus actor is a thin wire, not a strategy. It routes parquet data → NDAlphaEngine → HZ result.
**Champion performance** (ACBv6 + IRP + DC + OB, full-stack 55-day Dec31Feb25):
- ROI: +54.67% | PF: 1.141 | Sharpe: 2.84 | Max DD: 15.80% | WR: 49.5% | Trades: 2145
- Log: `run_logs/summary_20260307_163401.json`
> **Data correction note (2026-03-07)**: An earlier reference showed ROI=+57.18%, PF=1.149,
> Sharpe=3.00. Those figures came from a stale `vbt_cache/2026-02-25.parquet` that was built
> mid-day — missing 435 scans and carrying corrupt vel_div on 492 rows for the final day of the
> window. ALGO-3 parity testing caught the mismatch (max_diff=1.22 vs tolerance 1e-10).
> The parquet was rebuilt from live NG3 JSON (`build_parquet_cache(dates=['2026-02-25'], force=True)`).
> The stale file is preserved as `2026-02-25.parquet.STALE_20260307` for replicability.
> The corrected numbers above are the canonical reference. The ~2.5pp ROI drop reflects real
> late-day trades on Feb 25 that the stale parquet had silently omitted.
---
## 2. PHYSICAL ARCHITECTURE
```
┌─────────────────────────────────────────────────────────────────────┐
│ DATA SOURCES │
│ NG3 Scanner → correlation_arb512/ (512-bit eigenvalue matrices) │
│ CCXT Live Feed → 5s OHLCV bars (48+ USDT perpetuals on Binance) │
│ VBT Cache → vbt_cache_klines/*.parquet (pre-built from CCXT) │
└───────────────────┬──────────────────────────────────────────────────┘
┌───────────────────▼──────────────────────────────────────────────────┐
│ HAZELCAST IN-MEMORY GRID (localhost:5701, cluster: "dolphin") │
│ IMaps: │
│ DOLPHIN_SAFETY → AtomicReference / IMap (posture JSON) │
│ DOLPHIN_FEATURES → ACB boost + OB features per bar │
│ DOLPHIN_PNL_BLUE → Per-date trade results (SHORT posture) │
│ DOLPHIN_PNL_GREEN → Per-date trade results (LONG posture) │
│ DOLPHIN_FEATURES_SHARD_00..09 → 400-asset sharded feature store │
└───────────────────┬──────────────────────────────────────────────────┘
┌───────────────────▼──────────────────────────────────────────────────┐
│ PRODUCTION DAEMONS (each runs in its own process/thread) │
│ acb_processor_service.py — Daily ACB boost computation + HZ write │
│ ob_stream_service.py — 500ms OB snapshot ingestion + HZ write │
│ system_watchdog_service.py — Survival Stack Rm computation │
│ mc_forewarner_flow.py — Prefect-orchestrated MC gate │
└───────────────────┬──────────────────────────────────────────────────┘
┌───────────────────▼──────────────────────────────────────────────────┐
│ NAUTILUS TRADING ENGINE │
│ paper_trade_flow.py → DolphinActor → NDAlphaEngine │
│ Receives 5s bars via Nautilus event loop │
│ Calls process_day() once per calendar date │
└─────────────────────────────────────────────────────────────────────┘
```
**Key invariant**: `DolphinActor.on_bar()` fires ~17,280 times per day (every 5 seconds). `process_day()` is called **exactly once per calendar date** — guarded by `_processed_dates` set.
---
## 3. DATA LAYER
### 3.1 vbt_cache_klines Parquet Schema
Location: `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\YYYY-MM-DD.parquet`
| Column | Type | Description |
|--------|------|-------------|
| `vel_div` | float64 | Eigenvalue velocity divergence: `v50_vel v750_vel` (primary signal) |
| `v50_lambda_max_velocity` | float64 | Short-window (50-bar) lambda_max rate of change |
| `v150_lambda_max_velocity` | float64 | 150-bar window lambda velocity |
| `v300_lambda_max_velocity` | float64 | 300-bar window lambda velocity |
| `v750_lambda_max_velocity` | float64 | Long-window (750-bar) macro eigenvalue velocity |
| `instability_50` | float64 | General market instability index (50-bar) |
| `instability_150` | float64 | General market instability index (150-bar) |
| `BTCUSDT``STXUSDT` | float64 | Per-asset close prices (48 assets in current dataset) |
Each file: 1,439 rows (1 per 5-second bar over 24h), 57 columns.
### 3.2 NG3 Eigenvalue Data
Location: `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\`
```
eigenvalues/
YYYY-MM-DD/
scan_NNNNNN__Indicators.npz ← ACBv6 external factors (funding, dvol, fng, taker)
scan_NNNNNN__scan_global.npz ← lambda_vel_w750 for dynamic beta
matrices/
YYYY-MM-DD/
scan_NNNNNN_w50_HHMMSS.arb512.pkl.zst ← 512-bit correlation matrix (unused in hot path)
```
NPZ files loaded by `AdaptiveCircuitBreaker._load_external_factors()` (max 10 scans per date, median-aggregated).
---
## 4. SIGNAL LAYER
### 4.1 Primary Signal: vel_div Threshold Gate
**Source**: `alpha_signal_generator.py`, `AlphaSignalGenerator.generate()`
**SHORT signal condition**:
```
vel_div < VEL_DIV_THRESHOLD (-0.02)
```
**LONG signal condition** (green posture, not champion):
```
vel_div > LONG_THRESHOLD (0.01)
```
**Confidence calculation** (SHORT path):
```python
ratio = clamp((threshold - vel_div) / (threshold - extreme), 0, 1)
= clamp((-0.02 - vel_div) / (-0.02 - (-0.05)), 0, 1)
= clamp((-0.02 - vel_div) / 0.03, 0, 1)
confidence = 0.50 + ratio * 0.40 # range: [0.50, 0.90]
```
`is_extreme = (vel_div <= -0.05)`
### 4.2 Direction Confirmation (DC) — Layer 6
**Source**: `alpha_signal_generator.py`, `check_dc_nb()` (numba JIT)
```python
# Looks back dc_lookback_bars (default 7) bars on the selected asset price
p0 = price[n - lookback - 1]
p1 = price[n - 1]
chg_bps = (p1 - p0) / p0 * 10000.0
if chg_bps < -min_magnitude_bps (-0.75): return CONFIRM # falling price SHORT OK
if chg_bps > min_magnitude_bps (+0.75): return CONTRADICT
else: return NEUTRAL
```
**dc_skip_contradicts = True** (champion): CONTRADICT returns null signal (skip entry).
**Effect on leverage**: DC has `dc_leverage_boost=1.0` (no boost in champion). CONTRADICT kills entry.
### 4.3 OB Sub-2: Per-Asset Imbalance Confirmation
When `ob_engine` is wired in (`use_ob_edge=True`):
```python
eff_imb = -ob_signal.imbalance_ma5 # For SHORT: sell pressure = positive eff_imb
if eff_imb > 0.10: # OB confirms → confidence boost ≤+15%
ob_adj = 1 + min(0.15, eff_imb * persistence * 0.5)
confidence *= ob_adj
elif eff_imb < -0.15 and persistence > 0.60: # Strong persistent OB contradiction → HARD SKIP
return null_signal
elif eff_imb < -0.10: # Moderate soft dampen confidence
ob_adj = max(0.85, 1 - |eff_imb| * persistence * 0.4)
confidence *= ob_adj
```
---
## 5. ASSET SELECTION — IRP
### 5.1 Overview
**Source**: `alpha_asset_selector.py`, `AlphaAssetSelector.rank_assets()` + numba kernels
IRP = **Impulse Response Profiling**. Ranks all available assets by historical behavior over the last 50 bars in the regime direction. Selects the asset with the highest ARS (Asset Ranking Score) that passes all filters.
**Enabled by**: `use_asset_selection=True` (production default).
### 5.2 Numba Kernel: compute_irp_nb
```python
# Input: price_segment (last 50 prices), direction (-1 or +1)
dir_returns[i] = (price[i+1] - price[i]) * direction # directional returns
cumulative = cumsum(dir_returns)
mfe = max(cumulative) # Maximum Favorable Excursion
mae = abs(min(cumulative, 0)) # Maximum Adverse Excursion
efficiency = mfe / (mae + 1e-6)
alignment = count(dir_returns > 0) / n_ret
noise = variance(dir_returns)
latency = bars_to_reach_10pct_of_mfe # (default: 50 if mfe==0)
```
### 5.3 Numba Kernel: compute_ars_nb
```
ARS = 0.5 * log1p(efficiency) + 0.35 * alignment - 0.15 * noise * 1000
```
### 5.4 Numba Kernel: rank_assets_irp_nb
For each asset:
1. Compute IRP in DIRECT direction (regime_direction)
2. Compute IRP in INVERSE direction (-regime_direction)
3. Take whichever gives higher ARS (allows inverse selection)
4. Apply filter gates:
- `noise > 500` → skip
- `latency > 20` → skip (must reach 10% MFE within 20 bars)
- `alignment < 0.20` → skip
5. Bubble-sort by ARS descending (numba nopython)
### 5.5 AlphaAssetSelector Python Wrapper
```python
# Build 2D array (max_len × n_assets), right-aligned
valid = rank_assets_irp_nb(prices_2d, idx=max_len, regime_direction, ...)
# Walk ranked list:
for r in rankings:
if min_irp_alignment > 0 and r.metrics.alignment < min_irp_alignment:
continue # alignment gate (default 0.45)
if r.action != expected_action:
continue # direction gate
if ob_engine and ob_placement.depth_quality < 0.20:
continue # OB depth gate (try next asset)
trade_asset = r.asset
break
# No match → return None (no fallback to BTCUSDT when IRP enabled)
```
**OB Sub-1**: ARS adjusted ±5%/10% by per-asset OB depth quality before sorting.
---
## 6. POSITION SIZING
### 6.1 Signal Strength Score
**Source**: `alpha_bet_sizer.py`, `compute_sizing_nb()` (numba JIT)
```python
# SHORT path (vel_div < threshold):
if vel_div <= extreme (-0.05):
strength_score = 1.0
else:
strength_score = (threshold - vel_div) / (threshold - extreme)
# = (-0.02 - vel_div) / 0.03
strength_score = clamp(strength_score, 0.0, 1.0)
```
### 6.2 Dynamic Leverage (Cubic Convex)
```python
scaled_score = strength_score ** leverage_convexity (3.0)
eff_leverage = min_leverage + scaled_score * (max_leverage - min_leverage)
= 0.5 + scaled_score³ * 4.5 # range: [0.5, 5.0] before ACB
```
### 6.3 Signal Bucket Classification
```python
# Numba: get_signal_bucket_nb(vel_div, threshold=-0.02, extreme=-0.05)
if vel_div <= extreme * 1.5 (-0.075): bucket = 0 # "extreme"
elif vel_div <= extreme (-0.05): bucket = 1 # "strong"
elif vel_div <= (threshold+extreme)/2: bucket = 2 # "moderate" (-0.035)
else: bucket = 3 # "weak"
```
### 6.4 Alpha Layers (Layer 7)
**Bucket Boost** — adaptive win-rate feedback:
```python
# get_bucket_boost_nb: per-bucket win rate → multiplier
wr > 0.60 → 1.3x | wr > 0.55 → 1.1x | wr < 0.40 0.7x | wr < 0.45 0.85x
```
**Streak Multiplier** — recent 5-trade loss streak:
```python
# get_streak_mult_nb
losses_in_last_5 >= 4 → 0.5x | >= 3 → 0.7x | <= 1 → 1.1x
```
**Trend Multiplier** — vel_div acceleration:
```python
# get_trend_mult_nb(vd_trend = vel_div_history[-1] - vel_div_history[-10])
vd_trend < -0.01 1.3x (deepening instability)
vd_trend > 0.01 → 0.7x (recovering)
```
**Effective Fraction computation**:
```python
confidence = 0.70 if is_extreme else 0.55
conf_mult = confidence / 0.95
extreme_boost = 2.0 if is_extreme else 1.0
base_frac = 0.02 + strength_score * (base_fraction - 0.02)
eff_fraction = base_frac * conf_mult * extreme_boost * trend_mult * bucket_boost * streak_mult
eff_fraction = clamp(eff_fraction, 0.02, base_fraction=0.20)
```
**Final notional**:
```python
notional = capital * eff_fraction * final_leverage
```
### 6.5 ACB + MC Size Multiplier
```python
# regime_size_mult is recomputed every bar via _update_regime_size_mult(vel_div)
if day_beta > 0:
strength_cubic = clamp((threshold - vel_div) / (threshold - extreme), 0, 1) ** convexity
regime_size_mult = day_base_boost * (1.0 + day_beta * strength_cubic) * day_mc_scale
else:
regime_size_mult = day_base_boost * day_mc_scale
# Applied to leverage ceiling:
clamped_max_leverage = min(base_max_leverage * regime_size_mult * market_ob_mult, abs_max_leverage=6.0)
raw_leverage = size_result["leverage"] * dc_lev_mult * regime_size_mult * market_ob_mult
# STALKER posture hard cap:
if posture == 'STALKER': clamped_max_leverage = min(clamped_max_leverage, 2.0)
final_leverage = clamp(raw_leverage, min_leverage=0.5, clamped_max_leverage)
```
---
## 7. EXIT MANAGEMENT
### 7.1 Exit Priority Order (champion)
**Source**: `alpha_exit_manager.py`, `AlphaExitManager.evaluate()`
1. **FIXED_TP**: `pnl_pct >= 0.0095` (95 basis points)
2. **STOP_LOSS**: `pnl_pct <= -1.0` (DISABLED in practice — 100% loss never triggers before TP/max_hold)
3. **OB DURESS exits** (when ob_engine != None):
- Cascade Detection: `cascade_count > 0` → widen TP ×1.40, halve max_hold
- Liquidity Withdrawal: `regime_signal == 1` → hard SL 10%, TP ×0.60
4. **vel_div adverse-turn exits** (`vd_enabled=False` by default — disabled pending calibration)
5. **MAX_HOLD**: `bars_held >= 120` (= 600 seconds)
### 7.2 OB Dynamic Exit Parameter Adjustment
```python
if cascade_count > 0:
dynamic_tp_pct *= 1.40
dynamic_max_hold = int(max_hold_bars * 0.50) # take profit fast before snap-back
elif regime_signal == 1: # LIQUIDITY WITHDRAWAL STRESS
dynamic_sl_pct = 0.10 # hard 10% stop (tail protection)
if pnl_pct > 0.0:
dynamic_tp_pct *= 0.60 # take profit sooner under stress
if eff_imb < -0.10: # OB actively opposing
dynamic_max_hold = int(max_hold_bars * 0.40)
elif regime_signal == -1 and eff_imb > 0.15: # CALM + FAVORABLE
dynamic_max_hold = int(max_hold_bars * 1.50) # let winners run
# Per-asset withdrawal (micro-level):
if withdrawal_velocity < -0.20 and not in cascade/stress:
dynamic_max_hold = min(dynamic_max_hold, int(max_hold_bars * 0.40))
if pnl_pct > 0.0: dynamic_tp_pct *= 0.75
```
### 7.3 Sub-day ACB Force Exit
When HZ listener fires an ACB update mid-day:
```python
# In update_acb_boost(boost, beta):
if old_boost >= 1.25 and boost < 1.10:
evaluate_subday_exits() # → _execute_exit("SUBDAY_ACB_NORMALIZATION", ...)
```
Threshold is ARBITRARY (not backtested). Marked research TODO. Safe under pending-flag pattern (fires on next bar, not mid-loop).
### 7.4 Slippage on Exit
```python
# SHORT position exit:
exit_price = current_price * (1.0 + slip) # slippage against us when covering short
# STOP_LOSS: slip = 0.0005 (5 bps — market order fill)
# FIXED_TP: slip = 0.0002 (2 bps — likely limit fill)
# All others: slip = 0.0002
```
---
## 8. FEE & SLIPPAGE MODEL
### 8.1 SmartPlacer Fee Model (Layer 3)
**Source**: `esf_alpha_orchestrator.py`, `_execute_exit()`
Blended taker/maker fee rates based on historical SP fill statistics:
```python
# Entry fee:
entry_fee = (0.0002 * sp_maker_entry_rate + 0.0005 * (1 - sp_maker_entry_rate)) * notional
= (0.0002 * 0.62 + 0.0005 * 0.38) * notional
= (0.0001240 + 0.0001900) * notional
= 0.000314 * notional (31.4 bps)
# Exit fee (non-SL):
exit_fee = (0.0002 * sp_maker_exit_rate + 0.0005 * (1 - sp_maker_exit_rate)) * notional
= (0.0002 * 0.50 + 0.0005 * 0.50) * notional
= 0.000350 * notional (35.0 bps)
# Exit fee (SL):
exit_fee = 0.0005 * notional (50 bps — pure taker)
```
### 8.2 SP Slippage Refund
```python
# Probabilistic refund for maker fills (saves the bid-ask spread):
if rng.random() < sp_maker_entry_rate (0.62):
pnl_pct_raw += 0.0002 # +2 bps refund
if reason != STOP_LOSS and rng.random() < sp_maker_exit_rate (0.50):
pnl_pct_raw += 0.0002 # +2 bps refund
```
### 8.3 OB Edge (Layer 4)
```python
# With real OB engine:
if ob_placement.depth_quality > 0.5:
pnl_pct_raw += ob_placement.fill_probability * ob_edge_bps * 1e-4
# Without OB engine (legacy Monte Carlo fallback):
if rng.random() < ob_confirm_rate (0.40):
pnl_pct_raw += ob_edge_bps * 1e-4 # default: +5 bps
```
**Net PnL**:
```python
gross_pnl = pnl_pct_raw * notional
net_pnl = gross_pnl - entry_fee - exit_fee
capital += net_pnl
```
---
## 9. OB INTELLIGENCE LAYER
**Source**: `ob_features.py`, `ob_provider.py`, `hz_ob_provider.py`
The OB layer is wired in via `engine.set_ob_engine(ob_engine)` which propagates to signal_gen, asset_selector, and exit_manager. It is OPTIONAL — the engine degrades gracefully to legacy Monte Carlo when `ob_engine=None`.
### 9.1 OB Signals Per Asset
```python
ob_signal = ob_engine.get_signal(asset, timestamp)
# Fields:
# imbalance_ma5 — 5-bar MA of bid/ask size imbalance ([-1, +1])
# imbalance_persistence — fraction of last N bars sustaining sign
# withdrawal_velocity — rate of depth decay (negative = book thinning)
```
### 9.2 OB Macro (Market-Wide)
```python
ob_macro = ob_engine.get_macro()
# Fields:
# cascade_count — number of assets in liquidation cascade
# regime_signal — (-1=calm/trending, 0=neutral, +1=withdrawal stress)
```
### 9.3 OB Placement Quality
```python
ob_placement = ob_engine.get_placement(asset, timestamp)
# Fields:
# depth_quality — book depth score ([0, 2+]; >1 = deep book)
# fill_probability — probability of maker fill at entry price
```
### 9.4 OB Sub-Systems Summary
| Sub | Location | Effect |
|-----|----------|--------|
| OB-1 | AlphaAssetSelector | ARS adjusted ±5%/10% by depth quality |
| OB-2 | AlphaSignalGenerator | Confidence boost/dampen by imbalance; hard skip if persistent contradiction |
| OB-3 | NDAlphaEngine._try_entry | Market-wide imbalance multiplier on final leverage (±20%/15%) |
| OB-4 | AdaptiveCircuitBreaker | Macro withdrawal stress modulates ACBv6 dynamic beta (+25% max) |
| OB-5 | AlphaExitManager | Dynamic TP/SL/max_hold based on cascade/withdrawal/calm regime |
---
## 10. ACB v6 — ADAPTIVE CIRCUIT BREAKER
### 10.1 Architecture (3-Scale Confirmation)
**Source**: `adaptive_circuit_breaker.py`, `AdaptiveCircuitBreaker`
```
Scale 1 (Daily): External macro factors → log_0.5 base_boost
Scale 2 (Per-bar): vel_div signal strength → meta-boost multiplier
Scale 3 (Macro): w750 eigenvalue velocity → dynamic beta switch
```
### 10.2 Scale 1 — External Factor Signals
Loaded from NG3 `scan_*__Indicators.npz` files (median of first 10 scans per date):
| Factor | Source | Bearish Threshold | Signal Weight |
|--------|--------|------------------|---------------|
| `funding_btc` | Funding rate | < -0.0001 (very) / < 0.0 (mild) | 1.0 / 0.5 |
| `dvol_btc` | BTC implied vol | > 80 (extreme) / > 55 (elevated) | 1.0 / 0.5 |
| `fng` | Fear & Greed Index | < 25 (extreme fear) / < 40 (fear) | 1.0 / 0.5 (requires confirmation) |
| `taker` | Taker buy ratio | < 0.80 (selling) / < 0.90 (mild) | 1.0 / 0.5 |
**Signal counting**:
```python
signals = sum(individual_signal_weights) # float, e.g. 2.5
```
**Base boost formula**:
```python
if signals >= 1.0:
base_boost = 1.0 + 0.5 * log1p(signals)
# signals=1 → 1.347x | signals=2 → 1.549x | signals=3 → 1.693x
else:
base_boost = 1.0
```
### 10.3 Scale 3 — Dynamic Beta
```python
# Preloaded: w750_threshold = np.percentile(all_w750_vels, 60)
if w750_vel >= w750_threshold:
beta = BETA_HIGH = 0.8 # aggressive meta-boost during macro acceleration
else:
beta = BETA_LOW = 0.2 # conservative during calm macro
```
### 10.4 Scale 2 — Per-Bar Meta-Boost
```python
# Computed every bar inside _update_regime_size_mult(vel_div):
strength_cubic = clamp((threshold - vel_div) / (threshold - extreme), 0, 1) ** 3
# leverage_convexity = 3 → cubic
if day_beta > 0:
regime_size_mult = base_boost * (1.0 + beta * strength_cubic) * mc_scale
else:
regime_size_mult = base_boost * mc_scale
```
### 10.5 Sub-Day ACB Update (HZ Listener)
The `acb_processor_service.py` re-runs ACB computation mid-day when new NG3 scan data arrives and writes `{boost, beta}` to `DOLPHIN_FEATURES` IMap.
`_on_acb_event()` in `DolphinActor` stores the payload in `self._pending_acb` (GIL-safe dict write). Applied at start of next `on_bar()` iteration:
```python
# In on_bar() — BEFORE processing:
if _pending_acb is not None and engine is not None:
engine.update_acb_boost(pending_acb['boost'], pending_acb['beta'])
_pending_acb = None
```
---
## 11. SURVIVAL STACK — POSTURE CONTROL
### 11.1 Overview
**Source**: `survival_stack.py`, `SurvivalStack`
Computes a continuous Risk Multiplier `Rm ∈ [0, 1]` from 5 sensor categories. Maps to discrete posture {APEX, STALKER, TURTLE, HIBERNATE}.
### 11.2 Five Sensor Categories
**Cat1 — Binary Invariant** (kill switch):
```python
if hz_nodes < 1 or heartbeat_age_s > 30:
return 0.0 # Total system failure → HIBERNATE immediately
return 1.0
```
**Cat2 — Structural** (MC-Forewarner + data staleness):
```python
base = {OK: 1.0, ORANGE: 0.5, RED: 0.1}[mc_status]
decay = exp(-max(0, staleness_hours - 6) / 3)
f_structural = base * decay # Exponential decay after 6h stale
```
**Cat3 — Microstructure** (OB depth/fill quality):
```python
if ob_stale:
return 0.5
score = min(depth_quality, fill_prob)
return clamp(0.3 + 0.7 * score, 0.3, 1.0)
```
**Cat4 — Environmental** (DVOL spike impulse):
```python
if dvol_spike and t_since_spike_min == 0:
return 0.3 # Instant degradation at spike
return 0.3 + 0.7 * (1 - exp(-t_since_spike_min / 60)) # 60-min recovery tau
```
**Cat5 — Capital** (sigmoid drawdown constraint):
```python
# Rm5 ≈ 1.0 at DD<5%, ≈ 0.5 at DD=12%, ≈ 0.1 at DD=20%
return 1 / (1 + exp(30 * (drawdown - 0.12)))
```
### 11.3 Hierarchical Combination
```python
f_environment = min(f_structural, f_ext) # worst of Cat2/Cat4
f_execution = f_micro # Cat3
r_target = Cat1 * Cat5 * f_environment * f_execution
# Correlated sensor collapse penalty:
degraded = count([f_structural < 0.8, f_micro < 0.8, f_ext < 0.8])
if degraded >= 2:
r_target *= 0.5
```
### 11.4 Bounded Recovery Dynamics
```python
# Fast attack (instant degradation), slow recovery (5%/minute max):
if r_target < last_r_total:
r_final = r_target # immediate drop
else:
alpha = min(1.0, 0.02 * dt_min)
step = min(alpha * (r_target - last_r_total), 0.05 * dt_min)
r_final = last_r_total + step
```
### 11.5 Posture Mapping
**NOTE: Thresholds are deliberately TIGHTER than mathematical spec (safety buffer).**
```python
if Rm >= 0.90: APEX # Full trading, no constraints
if Rm >= 0.75: STALKER # Max leverage capped at 2.0x
if Rm >= 0.50: TURTLE # regime_dd_halt = True (no new entries)
else: HIBERNATE # Force-close open positions, no new entries
```
### 11.6 Hysteresis
```python
# Down: requires hysteresis_down=2 consecutive bars at lower level
# Up: requires hysteresis_up=5 consecutive bars at higher level
# Prevents flip-flopping around thresholds
```
### 11.7 Posture → Engine Effect
| Posture | Engine Effect |
|---------|--------------|
| APEX | No constraint (max leverage = abs_max=6.0 × regime_size_mult) |
| STALKER | `clamped_max_leverage = min(..., 2.0)` in `_try_entry` |
| TURTLE | `regime_dd_halt = True``process_bar` skips entry block |
| HIBERNATE | `_manage_position` forces EXIT("HIBERNATE_HALT"), `regime_dd_halt = True` |
---
## 12. MC-FOREWARNER ENVELOPE GATE
**Source**: Called via `engine.set_mc_forewarner(forewarner, mc_base_cfg)`
Runs daily at start of `process_day()`:
```python
mc_cfg = {**mc_base_cfg, 'max_leverage': base_max_leverage * day_base_boost}
mc_report = forewarner.assess_config_dict(mc_cfg)
mc_red = mc_report.catastrophic_probability > 0.25 or mc_report.envelope_score < -1.0
mc_orange = (not mc_red) and (envelope_score < 0 or catastrophic_probability > 0.10)
day_mc_status = 'RED' | 'ORANGE' | 'OK'
day_mc_scale = 0.5 if ORANGE else 1.0
if mc_red or posture in ['TURTLE', 'HIBERNATE']:
regime_dd_halt = True # No new entries for the entire day
```
---
## 13. NDAlphaEngine — FULL BAR LOOP
### 13.1 process_day() Flow
```python
def process_day(date_str, df, asset_columns, vol_regime_ok, direction, posture):
# --- Per-day initialization ---
regime_direction = direction or -1 # -1=SHORT champion
regime_dd_halt = False
day_posture = posture
# --- ACB: Scale 1 + Scale 3 (once per day) ---
if acb:
info = acb.get_dynamic_boost_for_date(date_str, ob_engine)
day_base_boost = info['boost'] # log_0.5 curve
day_beta = info['beta'] # dynamic beta (w750 percentile switch)
# --- MC-Forewarner gate (once per day) ---
if forewarner:
mc_report = forewarner.assess_config_dict(...)
if mc_red or posture in [TURTLE, HIBERNATE]:
regime_dd_halt = True # kill new entries for day
# === BAR LOOP ===
for ri, row in df.iterrows():
vd = row['vel_div'] # primary signal
v50_vel = row['v50_lambda_max_velocity']
v750_vel = row['v750_lambda_max_velocity']
prices = {col: row[col] for col in asset_columns if row[col] > 0}
# Append to price_histories (capped at 200; sliding window)
for ac, p in prices.items():
price_histories[ac].append(p)
if len(price_histories[ac]) > 500:
price_histories[ac] = price_histories[ac][-200:]
vol_ok = vol_regime_ok[ri] if vol_regime_ok else (bid >= 100)
# ACB Scale 2: regime_size_mult updated every bar
_update_regime_size_mult(vd)
process_bar(global_bar_idx, vd, prices, vol_ok, price_histories, v50_vel, v750_vel)
global_bar_idx += 1
return {date, pnl, capital, boost, beta, mc_status, trades}
```
### 13.2 process_bar() Flow
```python
def process_bar(bar_idx, vel_div, prices, vol_regime_ok, price_histories, v50_vel, v750_vel):
bar_count += 1
vel_div_history.append(vel_div) # trimmed to 200
# === EXIT MANAGEMENT (always first) ===
if position is not None:
exit_info = _manage_position(bar_idx, prices, vel_div, v50_vel, v750_vel)
# → AlphaExitManager.evaluate() → if EXIT: _execute_exit()
# === ENTRY (only when no position) ===
if position is None AND bar_idx > last_exit_bar AND NOT regime_dd_halt:
if bar_count >= lookback (100) AND vol_regime_ok:
entry_info = _try_entry(bar_idx, vel_div, prices, price_histories, v50_vel, v750_vel)
```
### 13.3 _try_entry() Flow
```python
def _try_entry(bar_idx, vel_div, prices, price_histories, v50_vel, v750_vel):
if capital <= 0: return None
# 1. IRP Asset Selection (Layer 2)
if use_asset_selection:
market_data = {a: history[-50:] for a, history in price_histories if len >= 50}
rankings = asset_selector.rank_assets(market_data, regime_direction)
trade_asset = first_asset_passing_all_gates(rankings)
if trade_asset is None: return None # strict: no fallback
else:
trade_asset = "BTCUSDT" # fallback when IRP disabled
# 2. Signal Generation + DC (Layer 6)
signal = signal_gen.generate(vel_div, vel_div_history,
price_histories[trade_asset],
regime_direction, trade_asset)
if not signal.is_valid: return None # vel_div or DC killed it
# 3. Position Sizing (Layers 7-8)
size = bet_sizer.calculate_size(capital, vel_div, signal.vel_div_trend, regime_direction)
# 4. OB Sub-3: Cross-asset market multiplier
market_ob_mult = ob_engine.get_market_multiplier(...) # ±20%
# 5. ACB leverage ceiling enforcement
clamped_max = min(base_max_leverage * regime_size_mult * market_ob_mult, abs_max_leverage=6.0)
if posture == STALKER: clamped_max = min(clamped_max, 2.0)
final_leverage = clamp(size.leverage * regime_size_mult * market_ob_mult, min_lev, clamped_max)
# 6. Notional and entry
notional = capital * size.fraction * final_leverage
entry_price = prices[trade_asset]
# 7. Create position
position = NDPosition(trade_asset, regime_direction, entry_price,
notional, final_leverage, ...)
exit_manager.setup_position(trade_id, entry_price, direction, bar_idx, v50_vel, v750_vel)
```
---
## 14. DOLPHIN ACTOR — NAUTILUS INTEGRATION
**Source**: `dolphin_actor.py`, `DolphinActor(Strategy)`
### 14.1 Lifecycle
```
__init__: set defaults, _pending_acb = None
on_start: connect HZ, read posture, setup ACB listener, instantiate NDAlphaEngine
on_bar: (fires every 5s)
1. Apply _pending_acb if set (thread-safe drain from HZ listener)
2. Parse bar date
3. On date change: re-read posture from HZ
4. If date already processed: RETURN (strict once-per-day guard)
5. If HIBERNATE: process_day(empty df), mark done, write HZ, RETURN
6. Load parquet: _load_parquet_data(date_str)
7. process_day(date_str, df, asset_cols, vol_regime_ok, direction, posture)
8. _processed_dates.add(date_str)
9. _write_result_to_hz(date_str, result)
on_stop: hz_client.shutdown()
```
### 14.2 Thread Safety: ACB Pending-Flag Pattern
**CRITICAL**: HZ entry listeners run on HZ client pool threads, NOT the Nautilus event loop.
```python
# HZ thread (ONLY writes _pending_acb — no engine access):
def _on_acb_event(event):
if event.value:
self._pending_acb = json.loads(event.value) # GIL-safe dict assignment
# Nautilus event loop thread (only reader + engine caller):
def on_bar(bar):
if _pending_acb is not None and engine is not None:
engine.update_acb_boost(_pending_acb['boost'], _pending_acb['beta'])
_pending_acb = None
# ... rest of on_bar
```
Max ACB latency: 1 bar interval (5 seconds). Acceptable given ACB changes on minute-to-hour timescale.
### 14.3 Data Loading
```python
def _load_parquet_data(date_str):
path = HCM_DIR / "vbt_cache_klines" / f"{date_str}.parquet"
df = pd.read_parquet(path)
meta_cols = {vel_div, scan_number, v50_..., v150_..., v300_..., v750_..., instability_50, instability_150}
asset_columns = [c for c in df.columns if c not in meta_cols]
vol_regime_ok = None # deferred to process_day (bar >= 100 gate)
return df, asset_columns, None
```
### 14.4 Posture Reading
Primary: `HZ CP Subsystem AtomicReference('DOLPHIN_SAFETY')` — linearizable.
Fallback: `HZ IMap('DOLPHIN_SAFETY').get('latest')` — eventually consistent.
### 14.5 Result Writing
```python
def _write_result_to_hz(date_str, result):
imap_pnl = hz_client.get_map('DOLPHIN_PNL_BLUE').blocking()
imap_pnl.put(date_str, json.dumps(result))
# result: {date, pnl, capital, boost, beta, mc_status, trades}
```
---
## 15. HAZELCAST FEATURE STORE — SHARDING
**Source**: `hz_sharded_feature_store.py`, `ShardedFeatureStore`
### 15.1 Shard Routing
```python
SHARD_COUNT = 10
shard = sum(ord(c) for c in symbol) % SHARD_COUNT
map_name = f"DOLPHIN_FEATURES_SHARD_{shard:02d}" # ..._00 through ..._09
```
Hash is stable (sum-of-ord, not Python's hash()), deterministic across process restarts.
### 15.2 API
```python
store = ShardedFeatureStore(hz_client)
store.put('BTCUSDT', 'vel_div', -0.03) # routes to shard based on symbol hash
val = store.get('BTCUSDT', 'vel_div') # same shard
store.delete('BTCUSDT', 'vel_div')
# Key format inside IMap: "vel_div_BTCUSDT"
# Map handles cached after first access
```
### 15.3 Near Cache Configuration
```python
near_cache_config = ShardedFeatureStore.near_cache_config()
# Returns dict for all 10 shards:
# { 'DOLPHIN_FEATURES_SHARD_XX': {
# invalidate_on_change: True,
# time_to_live_seconds: 300,
# max_idle_seconds: 60,
# eviction_policy: LRU,
# max_size: 5000
# }
# }
hz = hazelcast.HazelcastClient(near_caches=near_cache_config, ...)
```
### 15.4 Concurrency Notes
| Risk | Severity | Mitigation |
|------|----------|-----------|
| `_maps` dict not thread-safe | LOW | Only accessed from actor event loop (single thread) |
| Multiple ACBProcessorService writers | MEDIUM | CP FencedLock (only if all writers use it) |
| HZ Near Cache stale | LOW | TTL=300s, invalidate_on_change=True |
---
## 16. PRODUCTION DAEMON TOPOLOGY
### 16.1 ACB Processor Service
**Purpose**: Computes ACBv6 daily boost + dynamic beta and writes to HZ `DOLPHIN_FEATURES` IMap.
**Schedule**: Runs once at market open (08:00 UTC), then re-runs whenever a new NG3 scan batch arrives. Uses CP FencedLock to prevent simultaneous writes.
**Output**: `DOLPHIN_FEATURES.put('acb_boost', json.dumps({boost, beta}))`
### 16.2 OB Stream Service
**Purpose**: Fetches live Binance order book snapshots at 500ms cadence. Computes per-asset imbalance, fills, and macro signals. Writes to HZ.
**Cadence**: 500ms per cycle (2 updates/second).
### 16.3 System Watchdog Service
**Purpose**: Reads HZ heartbeat, OB quality, MC status, drawdown → runs `SurvivalStack.compute_rm()` → writes posture to `DOLPHIN_SAFETY` AtomicReference via `SurvivalStack.write_to_hz()`.
**Cadence**: ~60s per cycle.
### 16.4 MC-Forewarner Flow
**Purpose**: Prefect-orchestrated. Assesses current config envelope daily using correlation matrix statistics. Outcome: OK / ORANGE / RED status in HZ. ORANGE → `day_mc_scale=0.5`. RED → `regime_dd_halt=True`.
### 16.5 paper_trade_flow.py
**Purpose**: Main production entry point. Instantiates `DolphinActor` and wires into Nautilus `BacktestEngine` (paper mode). Subscribes to CCXT 5s bars. Runs indefinitely.
**Direction**: `direction = -1` (SHORT, blue posture). GREEN (LONG) is separate configuration.
### 16.6 Daemon Start Sequence
```
1. Start Hazelcast (port 5701)
2. acb_processor_service.py ← writes initial ACB boost before market open
3. ob_stream_service.py ← start OB snapshots
4. system_watchdog_service.py ← begin posture computation
5. mc_forewarner_flow.py ← Prefect flow (port 4200 UI)
6. paper_trade_flow.py ← Start actor LAST (reads all the above from HZ)
```
### 16.7 Monitoring Endpoints
| Service | URL / Path |
|---------|------------|
| Hazelcast MC Console | `http://localhost:8080` |
| Prefect UI | `http://localhost:4200` |
| Daily PnL | `HZ IMap: DOLPHIN_PNL_BLUE` (key = date string) |
| Posture | `HZ AtomicRef: DOLPHIN_SAFETY` |
| ACB State | `HZ IMap: DOLPHIN_FEATURES` (key = 'acb_boost') |
---
## 17. CI TEST SUITE
**Location**: `ci/` directory. Runner: `pytest ci/ -v`
**Passing gate: 14/14 tests in 7-10s.**
| File | Tests | What It Covers |
|------|-------|----------------|
| `test_13_nautilus_integration.py` | 6 | Actor import, instantiation, on_bar, HIBERNATE posture, once-per-day guard, ACB thread safety |
| `test_14_long_system.py` | 3 | Multi-day run, capital persistence, trade count |
| `test_15_acb_reactive.py` | 1 | ACB boost update applied correctly mid-day |
| `test_16_scaling.py` | 4 | Memory footprint <4GB (50 assets), shard routing (400 symbols), 400-asset no-crash, 400-asset with IRP |
### 17.1 Key Test: test_acb_listener_does_not_call_engine_directly
```python
# Patches engine.update_acb_boost with a mock, fires _on_acb_event
# Asserts: _pending_acb is set, engine method was NOT called
with patch.object(actor.engine, 'update_acb_boost') as mock_update:
actor._on_acb_event(event)
assert actor._pending_acb['boost'] == 1.35
mock_update.assert_not_called()
```
### 17.2 Key Test: test_process_day_called_once_per_day
```python
# Fires 3 bars on same date → assert process_day.call_count == 1
# Fires 1 bar on next date → assert process_day.call_count == 2
```
---
## 18. PARAMETER REFERENCE
### 18.1 Champion Parameters (Frozen)
| Parameter | Value | Layer | Notes |
|-----------|-------|-------|-------|
| `vel_div_threshold` | -0.02 | Signal gate | PRIMARY entry threshold |
| `vel_div_extreme` | -0.05 | Signal/sizing | Extreme regime: full size |
| `min_leverage` | 0.5 | Sizing | Floor leverage |
| `max_leverage` | 5.0 | Sizing | Base ceiling (before ACB) |
| `abs_max_leverage` | 6.0 | Sizing | Hard geometric ceiling |
| `leverage_convexity` | 3.0 | Sizing | Cubic convex curve |
| `fraction` | 0.20 | Sizing | Max capital fraction per trade |
| `fixed_tp_pct` | 0.0095 | Exit | 95 bps take-profit |
| `stop_pct` | 1.0 | Exit | Effectively disabled |
| `max_hold_bars` | 120 | Exit | 600 seconds |
| `dc_lookback_bars` | 7 | DC | 35 seconds price momentum |
| `dc_min_magnitude_bps` | 0.75 | DC | Minimum BTC momentum |
| `dc_skip_contradicts` | True | DC | Hard skip on contradiction |
| `min_irp_alignment` | 0.45 | IRP | Alignment gate |
| `sp_maker_entry_rate` | 0.62 | Fees | 62% maker fill at entry |
| `sp_maker_exit_rate` | 0.50 | Fees | 50% maker fill at exit |
| `ob_edge_bps` | 5.0 | OB | Legacy MC OB edge |
| `ob_confirm_rate` | 0.40 | OB | Legacy MC confirmation rate |
| `lookback` | 100 | Warmup | Bars before first entry allowed |
| `seed` | 42 | RNG | Deterministic numpy RandomState |
### 18.2 ACBv6 Parameters (Frozen — Validated)
| Parameter | Value | Notes |
|-----------|-------|-------|
| `BETA_HIGH` | 0.8 | w750 above p60 threshold |
| `BETA_LOW` | 0.2 | w750 below p60 threshold |
| `W750_THRESHOLD_PCT` | 60 | Percentile switch point |
| `FUNDING_VERY_BEARISH` | -0.0001 | 1.0 signal |
| `DVOL_EXTREME` | 80 | 1.0 signal |
| `FNG_EXTREME_FEAR` | 25 | 1.0 signal (needs confirmation) |
| `TAKER_SELLING` | 0.8 | 1.0 signal |
### 18.3 Survival Stack Thresholds (Deliberately Tight)
| Posture | Rm Threshold | vs. Math Spec |
|---------|-------------|---------------|
| APEX | ≥ 0.90 | Tighter — spec was 0.85 |
| STALKER | ≥ 0.75 | Tighter — spec was 0.70 |
| TURTLE | ≥ 0.50 | Tighter — spec was 0.45 |
| HIBERNATE | < 0.50 | |
**Do NOT loosen these without quantitative justification.**
---
## 19. KNOWN RESEARCH TODOs
| ID | Description | Priority |
|----|-------------|----------|
| TODO-1 | Calibrate `vd_enabled` adverse-turn exits (currently disabled). Requires analysis of trade vel_div distribution at entry vs. subsequent bars. True invalidation threshold likely ~+0.02 sustained for N=3 bars. | MEDIUM |
| TODO-2 | Validate SUBDAY_ACB force-exit threshold (`old_boost >= 1.25 and boost < 1.10`). Currently ARBITRARY — agent-chosen, not backtest-derived. | MEDIUM |
| TODO-3 | MIG8: Binance live adapter (real order execution). OUT OF SCOPE until after 30-day paper trading validation. | LOW |
| TODO-4 | 48-hour chaos test with all daemons running simultaneously. Watch for: KeyError, stale-read anomalies, concurrent HZ writer collisions. | HIGH (before live capital) |
| TODO-5 | Memory profiler with IRP enabled at 400 assets (current 71 MB measurement was without IRP). Projected ~600 MB — verify. | LOW |
| TODO-6 | TF-spread recovery exits (`tf_enabled=False`). Requires sweep of tf_exhaust_ratio and tf_flip_ratio vs. champion backtest. | LOW |
| TODO-7 | GREEN (LONG) posture paper validation. LONG thresholds (long_threshold=0.01, long_extreme=0.04) not yet production-validated. | MEDIUM |
---
*End of DOLPHIN-NAUTILUS System Bible v1.0 — 2026-03-07*
*Champion: SHORT only (APEX posture, blue configuration)*
*Gate: 14/14 CI tests green. Paper trading ready.*
*Do NOT deploy real capital until 30-day paper run is clean.*