DOLPHIN/nautilus_dolphin/SYSTEM_GOLD_SPEC_GUIDE.md

# SYSTEM GOLD SPEC GUIDE
## DOLPHIN NG — D_LIQ_GOLD Production Reference

**Canonical document. Last updated: 2026-03-22.**
**Purpose:** Exhaustive reference for reproducing D_LIQ_GOLD (ROI=181.81%) from scratch.
Covers every layer of the engine stack, every configuration constant, every file path,
every known gotcha, and the complete research history that led to the gold standard.

---

## TABLE OF CONTENTS

1. [Gold Standard Definition](#1-gold-standard-definition)
2. [How to Reproduce Gold — Step by Step](#2-how-to-reproduce-gold)
3. [System Architecture Overview](#3-system-architecture-overview)
4. [Engine Class Hierarchy (MRO)](#4-engine-class-hierarchy)
5. [Layer-by-Layer Deep Dive](#5-layer-by-layer-deep-dive)
6. [Critical Configuration Constants](#6-critical-configuration-constants)
7. [Data Pipeline — Input Files](#7-data-pipeline)
8. [Test Harness Anatomy](#8-test-harness-anatomy)
9. [Known Bugs and Fixes](#9-known-bugs-and-fixes)
10. [Research History (Exp1–Exp15)](#10-research-history)
11. [File Map — All Critical Paths](#11-file-map)
12. [Failure Mode Reference](#12-failure-mode-reference)

---

## 1. GOLD STANDARD DEFINITION

### D_LIQ_GOLD (production default since 2026-03-15)

```
ROI   = +181.81%
DD    = 17.65%  (max drawdown over 56-day window)
Calmar = 10.30  (ROI / max_DD)
PF    = ~1.55   (profit factor)
WR    = ~52%    (win rate)
T     = 2155    (EXACT — deterministic, any deviation is a regression)
liq_stops = 1   (0.05% rate — 1 liquidation floor stop out of 2155)
avg_leverage = 4.09x
MC    = 0 RED / 0 ORANGE across all 56 days
Meta Monitoring = Gold Certified (MHD v1.0 operational)
```

**Engine class:**
`LiquidationGuardEngine(soft=8x, hard=9x, mc_ref=5x, margin_buffer=0.95, adaptive_beta=True)`

**Factory function:**
```python
from nautilus_dolphin.nautilus.proxy_boost_engine import create_d_liq_engine
engine = create_d_liq_engine(**ENGINE_KWARGS)
```

**Data window:** 56 days, 2025-12-31 to 2026-02-26 (5-second scan data)

**Baseline comparison (BRONZE regression floor):**
NDAlphaEngine, no boost: ROI=+88.55%, PF=1.215, DD=15.05%, Sharpe=4.38, T=2155

---

## 2. HOW TO REPRODUCE GOLD

### Prerequisites
- Python 3.11
- SILOQY env or system Python with: numpy, pandas, numba, scipy, sklearn, pyarrow
- VBT parquet cache at `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache\`
  (56 daily parquet files, 2025-12-31 to 2026-02-26)

### Step 1 — Verify the fix is applied

In `esf_alpha_orchestrator.py` at line ~714, confirm:
```python
def set_esoteric_hazard_multiplier(self, hazard_score: float):
    floor_lev = 3.0
    ceiling_lev = getattr(self, '_extended_soft_cap', 6.0)   # ← MUST use getattr, not 6.0
    ...
```

If it says `ceiling_lev = 6.0` (hardcoded), the fix has NOT been applied.
Apply the fix FIRST or all D_LIQ results will be ~145.84% instead of 181.81%.
See §9 for full explanation.

### Step 2 — Verify leverage state after engine creation

```python
import os; os.environ['NUMBA_DISABLE_JIT'] = '1'  # optional for faster check
from nautilus_dolphin.nautilus.proxy_boost_engine import create_d_liq_engine
from dvae.exp_shared import ENGINE_KWARGS  # load without dvae/__init__ (see §9)

eng = create_d_liq_engine(**ENGINE_KWARGS)
assert eng.base_max_leverage == 8.0
assert eng.abs_max_leverage  == 9.0
assert eng.bet_sizer.max_leverage == 8.0

eng.set_esoteric_hazard_multiplier(0.0)
assert eng.base_max_leverage == 8.0,       "FIX NOT APPLIED — stomp is still active"
assert eng.bet_sizer.max_leverage == 8.0,  "FIX NOT APPLIED — sizer stomped"
print("Leverage state OK")
```

### Step 3 — Run the e2e test suite

```bash
cd "C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin"
pytest -m slow tests/test_proxy_boost_production.py -v
```

Expected: **9/9 PASSED** (runtime ~52 minutes)
- `test_e2e_baseline_reproduces_gold` → ROI=88.55% T=2155
- `test_e2e_d_liq_gold_reproduces_exp9b` → ROI=181.81% T=2155 DD≤18.15%
- `test_e2e_mc_silent_all_days` → 0 RED/ORANGE at d_liq leverage
- Plus 6 other mode tests

### Step 4 — Run the painstaking trace backtest (for detailed analysis)

```bash
cd "C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin"
python dvae/run_trace_backtest.py
```

Outputs to `dvae/trace/`:
- `tick_trace.csv`  — one row per bar (~346k rows) — full system state
- `trade_trace.csv` — one row per trade (~2155 rows)
- `daily_trace.csv` — one row per day (56 rows)
- `summary.json`    — final ROI/DD/T/Calmar

### Quick Gold Reproduction (single-run, no full e2e suite)

```bash
cd "C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin"
python dvae/test_dliq_fix_verify.py
```

Expected output (final 3 lines):
```
ROI match: ✓ PASS (diff=~0pp)
DD  match: ✓ PASS (diff=~0pp)
T   match: ✓ PASS (got 2155)
```

---

## 3. SYSTEM ARCHITECTURE OVERVIEW

```
┌──────────────────────────────────────────────────────────────┐
│                    DOLPHIN NG Production                     │
│                                                              │
│  VBT Parquet Cache (5s scans, 56 days)                      │
│       ↓                                                      │
│  Data Loader → float64 df + dvol per bar + vol_p60          │
│       ↓                                                      │
│  AdaptiveCircuitBreaker (ACBv6) → day_base_boost + day_beta │
│       ↓                                                      │
│  OBFeatureEngine (MockOBProvider, 48 assets)                │
│       ↓                                                      │
│  MC-Forewarner → day_mc_status (OK/ORANGE/RED)              │
│       ↓                                                      │
│  [process_day LOOP per day]                                  │
│    ├── begin_day() → ACB boost + MC gate                     │
│    ├── FOR EACH BAR:                                         │
│    │     _update_proxy(inst50, v750) → proxy_B              │
│    │     step_bar(vel_div, prices, vol_ok) →                 │
│    │       process_bar() →                                   │
│    │         ENTRY PATH: _try_entry() → size + leverage      │
│    │         EXIT PATH:  exit_manager.evaluate() → reason    │
│    │         _execute_exit() → pnl + trade record            │
│    └── end_day() → daily summary                             │
│       ↓                                                      │
│  trade_history: List[NDTradeRecord] — 2155 records          │
└──────────────────────────────────────────────────────────────┘
```

### Signal Pathway

```
vel_div (eigenvalue velocity divergence):
  → vel_div < -0.020 (threshold)  → ENTRY signal  (LONG, mean reversion)
  → vel_div < -0.050 (extreme)    → max leverage region
  → vel_div ≥ -0.020              → NO ENTRY (flat or rising eigenspace)

proxy_B = instability_50 - v750_lambda_max_velocity:
  → high proxy_B = eigenspace stress (high MAE risk at entry)
  → percentile rank vs 500-bar history → day_beta
  → day_beta modulates adaptive_beta boost
  → low proxy_B → higher leverage allowed (less stress, better setup)

vol_regime_ok (dvol > vol_p60):
  → bars where BTC realized vol > 60th percentile of dataset
  → filters out ultra-low-vol bars where signal is noisy
```

---

## 4. ENGINE CLASS HIERARCHY

```
NDAlphaEngine                          ← base backtest engine
  └── ProxyBaseEngine                  ← adds proxy_B tracking per bar
        └── AdaptiveBoostEngine        ← scale-boost using proxy_B rank
              └── ExtendedLeverageEngine ← 8x/9x leverage + MC decoupling
                    └── LiquidationGuardEngine ← per-trade liq floor stop
                          = D_LIQ_GOLD ★
```

**MRO (Python method resolution order):**
`LiquidationGuardEngine → ExtendedLeverageEngine → AdaptiveBoostEngine
 → ProxyBaseEngine → NDAlphaEngine → object`

### Which class owns which method

| Method | Owner class | Notes |
|--------|-------------|-------|
| `process_day()` | `ProxyBaseEngine` | Main backtest loop (begin_day + bar loop + end_day) |
| `step_bar()` | `NDAlphaEngine` (streaming API) | Per-bar state update, calls process_bar() |
| `process_bar()` | `NDAlphaEngine` | Core per-bar logic: entry/exit decision |
| `_try_entry()` | `LiquidationGuardEngine` | Sets `_pending_stop_override` before calling super |
| `_execute_exit()` | `LiquidationGuardEngine` | Counts liquidation stops, calls super |
| `begin_day()` | `ExtendedLeverageEngine` | MC lever-swap: saves true caps, sets mc_ref, restores after super |
| `begin_day()` | `NDAlphaEngine` (via super chain) | ACB boost, MC-Forewarner assessment |
| `end_day()` | `NDAlphaEngine` | Returns daily summary dict |
| `_update_proxy()` | `ProxyBaseEngine` | Updates `_current_proxy_b` per bar |
| `set_esoteric_hazard_multiplier()` | `NDAlphaEngine` | **BUG FIXED 2026-03-22** — now respects `_extended_soft_cap` |
| `set_acb()` | `NDAlphaEngine` | Wires AdaptiveCircuitBreaker |
| `set_ob_engine()` | `NDAlphaEngine` | Wires OBFeatureEngine |
| `set_mc_forewarner()` | `NDAlphaEngine` | Wires DolphinForewarner |

---

## 5. LAYER-BY-LAYER DEEP DIVE

### Layer 1: NDAlphaEngine (base)

**File:** `nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py`

Core engine. All signal generation, position management, exit logic, fee/slippage.

**Key parameters (from ENGINE_KWARGS in exp_shared.py):**
```python
initial_capital        = 25000.0
vel_div_threshold      = -0.020    # entry trigger: vel_div < this
vel_div_extreme        = -0.050    # extreme signal → max bet size
min_leverage           = 0.5       # minimum bet size multiplier
max_leverage           = 5.0       # base soft cap (overridden by ExtendedLeverageEngine)
leverage_convexity     = 3.0       # power in sizing curve
fraction               = 0.20      # Kelly-style fraction of capital per trade
fixed_tp_pct           = 0.0095    # take profit = 95 basis points
stop_pct               = 1.0       # stop loss at 100% of capital = effectively OFF
max_hold_bars          = 120       # max bars before forced exit (~10 minutes at 5s)
use_direction_confirm  = True      # direction confirmation gate
dc_lookback_bars       = 7         # bars for DC calculation
dc_min_magnitude_bps   = 0.75      # min move magnitude for DC confirmation
dc_skip_contradicts    = True      # skip entries with contradicting DC
dc_leverage_boost      = 1.0       # DC confirms → leverage multiplier
dc_leverage_reduce     = 0.5       # DC contradicts → reduce leverage
use_asset_selection    = True      # IRP-based asset selection (48 assets)
min_irp_alignment      = 0.45      # min fraction of assets aligned for entry
use_sp_fees            = True      # spread + maker fee model
use_sp_slippage        = True      # slippage model
sp_maker_entry_rate    = 0.62      # entry maker rate (bps)
sp_maker_exit_rate     = 0.50      # exit maker rate (bps)
use_ob_edge            = True      # order book edge gate
ob_edge_bps            = 5.0       # min OB edge required (bps)
ob_confirm_rate        = 0.40      # min OB confirm rate
lookback               = 100       # lookback for various rolling windows
use_alpha_layers       = True      # enable all alpha layers
use_dynamic_leverage   = True      # dynamic leverage calculation
seed                   = 42        # RNG seed for determinism
```

**Leverage formula in `_try_entry()`:**
```python
clamped_max_leverage = min(
    base_max_leverage * regime_size_mult * market_ob_mult,
    abs_max_leverage
)
raw_leverage = size_result["leverage"] * dc_lev_mult * regime_size_mult * market_ob_mult
leverage = min(raw_leverage, clamped_max_leverage)
```

Where:
- `base_max_leverage`: set by engine (5.0 base, 8.0 for D_LIQ)
- `regime_size_mult`: ACBv6-driven, typically 1.0–1.6
- `market_ob_mult`: OB-driven multiplier
- `abs_max_leverage`: hard cap (6.0 base, 9.0 for D_LIQ)

**Exit reasons (exit_manager):**
- `FIXED_TP` — take profit at 95bps
- `MAX_HOLD` — held 120+ bars
- `STOP_LOSS` — stop triggered (only liquidation guard stop in D_LIQ)
- `HIBERNATE_HALT` — MC-RED day halt
- `OB_TAIL_AVOIDANCE` — OB cascade/withdrawal signal (never fires with MockOBProvider)

### Layer 2: ProxyBaseEngine

**File:** `nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py`

Adds per-bar `proxy_B = instability_50 - v750_lambda_max_velocity` tracking.

**Key attribute:** `self._current_proxy_b` — updated by `_update_proxy(inst, v750)` before each `step_bar()` call.

**Key method — `process_day()` loop:**
```python
def process_day(self, date_str, df, asset_columns, vol_regime_ok=None, ...):
    self.begin_day(date_str, ...)
    for ri in range(len(df)):
        row = df.iloc[ri]
        vd = row.get('vel_div')
        if vd is None or not np.isfinite(float(vd)): continue
        v50  = gf('v50_lambda_max_velocity')
        v750 = gf('v750_lambda_max_velocity')
        inst = gf('instability_50')
        self._update_proxy(inst, v750)        # ← proxy_B updated HERE
        prices = {ac: float(row[ac]) for ac in asset_columns if ...}
        vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
        self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices, ...)   # ← bar processed
    return self.end_day()
```

### Layer 3: AdaptiveBoostEngine

**File:** `nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py`

Scale-boost mechanism. After `super()._try_entry()` succeeds, multiplies notional/leverage
by a factor determined by proxy_B percentile rank vs 500-bar history.

With `adaptive_beta=True` (D_LIQ default):
```python
alpha_eff = alpha * (1 + day_beta)     # day_beta from ACBv6 daily amplitude regime
# Lower proxy_B at entry → lower prank → more aggressive boost
# Higher proxy_B → prank near 1.0 → minimal/no boost
scale = max(1.0, 1.0 + alpha_eff * max(0, threshold - prank))
```

`day_beta`: ACBv6 outputs a beta value proportional to amplitude regime. High beta →
more aggressive days get extra boost. Low beta → conservative days, less amplification.

### Layer 4: ExtendedLeverageEngine

**File:** `nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py`

Extends the leverage ceiling from 5x/6x to 8x/9x (D_LIQ config).

**Critical mechanism — MC decoupling in `begin_day()`:**
```python
def begin_day(self, date_str, posture='APEX', direction=None):
    # Save true extended caps
    _true_base  = self.base_max_leverage    # 8.0 for D_LIQ
    _true_abs   = self.abs_max_leverage     # 9.0 for D_LIQ
    _true_sizer = self.bet_sizer.max_leverage  # 8.0 for D_LIQ

    # Temporarily show MC a reference leverage (5.0)
    self.base_max_leverage      = self._mc_leverage_ref   # 5.0
    self.bet_sizer.max_leverage = self._mc_leverage_ref   # 5.0
    self.abs_max_leverage       = self._mc_leverage_ref   # 5.0

    super().begin_day(...)   # MC-Forewarner assesses at 5.0x reference

    # Restore true caps for actual trading
    self.base_max_leverage      = _true_base   # 8.0
    self.bet_sizer.max_leverage = _true_sizer  # 8.0
    self.abs_max_leverage       = _true_abs    # 9.0
```

**Why this matters:** Without MC decoupling, at 8x/9x leverage the MC-Forewarner
would assess catastrophic risk at 8x and potentially return RED/ORANGE, halting trading.
By showing it mc_ref=5.0 (within its trained range), MC stays GREEN every day.
Empirically confirmed: 0 RED / 0 ORANGE across all 56 days at any leverage up to 10x.

### Layer 5: LiquidationGuardEngine (D_LIQ_GOLD)

**File:** `nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py`

Adds a per-trade liquidation floor stop. Before every entry:
```python
def _try_entry(self, bar_idx, vel_div, prices, price_histories, v50_vel=0.0, v750_vel=0.0):
    self._pending_stop_override = self._liq_stop_pct   # = (1/9) * 0.95 = 10.56%
    return super()._try_entry(...)
```

If price moves >10.56% against the position, stop fires before exchange liquidates.
- With 9x leverage: exchange liquidates at ~11.1% adverse move
- Our stop at 10.56% → exits ~0.56% before exchange force-liquidation
- `margin_buffer = 0.95` provides this safety margin

**Result:** 1 stop triggered across 2155 trades = 0.05% rate (negligible). The guard
provides safety without materially impacting returns.

---

## 6. CRITICAL CONFIGURATION CONSTANTS

### ENGINE_KWARGS (test harness gold standard)
**File:** `nautilus_dolphin/dvae/exp_shared.py` line 56–67

```python
ENGINE_KWARGS = dict(
    initial_capital=25000.0, vel_div_threshold=-0.02, vel_div_extreme=-0.05,
    min_leverage=0.5, max_leverage=5.0, leverage_convexity=3.0,
    fraction=0.20, fixed_tp_pct=0.0095, stop_pct=1.0, max_hold_bars=120,
    use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=0.75,
    dc_skip_contradicts=True, dc_leverage_boost=1.0, dc_leverage_reduce=0.5,
    use_asset_selection=True, min_irp_alignment=0.45,
    use_sp_fees=True, use_sp_slippage=True,
    sp_maker_entry_rate=0.62, sp_maker_exit_rate=0.50,
    use_ob_edge=True, ob_edge_bps=5.0, ob_confirm_rate=0.40,
    lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
```

Note: `max_leverage=5.0` is passed but IGNORED for D_LIQ — `ExtendedLeverageEngine.__init__`
overrides it to `D_LIQ_SOFT_CAP=8.0` unconditionally.

### MC_BASE_CFG (MC-Forewarner config)
**File:** `nautilus_dolphin/dvae/exp_shared.py` line 69–81

Key param: `'max_leverage': 5.00` — matches `D_LIQ_MC_REF=5.0` for consistency.

### D_LIQ Constants
**File:** `nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py` line 437–440

```python
D_LIQ_SOFT_CAP   = 8.0    # base_max_leverage: soft ceiling, ACBv6 can push toward hard cap
D_LIQ_ABS_CAP    = 9.0    # abs_max_leverage: hard ceiling, never exceeded
D_LIQ_MC_REF     = 5.0    # MC-Forewarner reference: within GOLD trained range
D_LIQ_MARGIN_BUF = 0.95   # liquidation floor = (1/9) * 0.95 = 10.56% adverse move
```

### Vol P60 (gold calibration method)
**File:** `nautilus_dolphin/dvae/exp_shared.py` `load_data()` function

```
vol_p60 ≈ 0.00009868 (from 2 parquet files, range(60), seg-based stddev, v>0 filter)
```

Method:
```python
for pf in parquet_files[:2]:   # ONLY FIRST 2 FILES
    df = pd.read_parquet(pf)
    pr = df['BTCUSDT'].values
    for i in range(60, len(pr)):   # range(60), NOT range(50)
        seg = pr[max(0, i-50):i]
        if len(seg) < 10: continue
        v = float(np.std(np.diff(seg) / seg[:-1]))
        if v > 0: all_vols.append(v)   # v>0 filter
vol_p60 = float(np.percentile(all_vols, 60))
```

**CRITICAL:** Any deviation from this exact method will change vol_p60 and alter trade
timing, potentially changing T away from 2155. The `run_backtest()` function in
`exp_shared.py` uses a slightly different (rolling) method that may give slightly
different results — the `load_data()` gold method is canonical for gold reproduction.

### MockOBProvider config
**File:** `nautilus_dolphin/dvae/exp_shared.py` `load_data()` function

```python
MockOBProvider(
    imbalance_bias=-0.09, depth_scale=1.0, assets=OB_ASSETS,
    imbalance_biases={
        "BTCUSDT": -0.086, "ETHUSDT": -0.092,
        "BNBUSDT": +0.05,  "SOLUSDT": +0.05,
    },
)
```

OB_ASSETS = all 48 assets from parquet file columns (sorted alphabetically).
`ob_confirm_rate=0.40`: with these biases, most entries pass the OB gate.
`OB_TAIL_AVOIDANCE` (cascade/withdrawal exits) NEVER fire with MockOBProvider
(mock provider generates synthetic data that never crosses the cascade threshold).

---

## 7. DATA PIPELINE

### Primary Data Source (backtest)
**Path:** `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache\`
**Format:** Daily Parquet files, named `YYYY-MM-DD.parquet`
**Window:** 56 files, 2025-12-31 to 2026-02-26
**Columns per file:**
  - 48 asset price columns (BTCUSDT, ETHUSDT, BNBUSDT, etc.) — float64
  - `vel_div` — eigenvalue velocity divergence — core signal
  - `v50_lambda_max_velocity`, `v150_lambda_max_velocity`, `v300_lambda_max_velocity`, `v750_lambda_max_velocity`
  - `instability_50`, `instability_150`
  - `timestamp`, `scan_number` (metadata, excluded from asset columns)

**META_COLS** (excluded from asset price list):
```python
META_COLS = {
    'timestamp', 'scan_number', 'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
    'v300_lambda_max_velocity', 'v750_lambda_max_velocity', 'vel_div',
    'instability_50', 'instability_150'
}
```

**Bar frequency:** 5 seconds (scan_number advances every 5s from NG5 scanner)
**Bars per day:** ~6,192 (5s × 86400s / 5 ≈ 17,280, but market hours vary → ~6k active bars)
**Total bars:** ~346,740 over 56 days

### Eigenvalue Source
**Path:** `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues\`
Used by NG5 scanner to produce eigenvalue features for each 5s scan.

### ExF NPZ (External Factors)
**Path:** `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\external_factors\eso_cache\`
**Latest:** `latest_esoteric_factors.json` — lunar phase, regional session, Fibonacci time
**NPZ backfill:** All 1710 historical dates in `vbt_cache_klines/` have `scan_000001__Indicators.npz`

### MC-Forewarner Models
**Path:** `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin\mc_results\models\`
**Format:** Pickle files (sklearn 1.7.1 — note: loading with 1.8.0 raises InconsistentVersionWarning but works)
**Models:** OneClassSVM, DummyRegressor, DecisionTreeRegressor, GradientBoostingRegressor, StandardScaler
Result: All 56 days return OK (0 RED/ORANGE) when assessed at mc_ref=5.0.

### 5-Year Klines Dataset (for regime research)
**Path:** `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\`
**Format:** 1710 parquet files, 2021-06-15 to 2026-03-05
**Frequency:** 1-minute OHLCV + vel_div + eigenvalue features

---

## 8. TEST HARNESS ANATOMY

### Primary e2e test
**File:** `nautilus_dolphin/tests/test_proxy_boost_production.py` (UNTRACKED)

Contains 9 e2e tests including `test_e2e_d_liq_gold_reproduces_exp9b`.
Run: `pytest -m slow tests/test_proxy_boost_production.py`

**`_run_engine()` function (critical path):**
```python
def _run_engine(eng, d, fw, MC_BASE_CFG):
    acb = AdaptiveCircuitBreaker()
    acb.preload_w750(d['date_strings'])
    eng.set_ob_engine(d['ob_eng'])
    eng.set_acb(acb)
    if fw is not None:
        eng.set_mc_forewarner(fw, MC_BASE_CFG)
    eng.set_esoteric_hazard_multiplier(0.0)   # ← CRITICAL: now correct (calls fixed function)
    for pf_file in d['parquet_files']:
        ds = pf_file.stem
        df, acols, dvol = d['pq_data'][ds]    # ← float64 pq_data from load_data()
        cap_before = eng.capital
        vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
        eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
```

### exp_shared.py functions

**File:** `nautilus_dolphin/dvae/exp_shared.py`

- `load_data()` → gold-standard data loading (float64, seg-based vol_p60, 48 OB assets)
- `run_backtest(engine_factory, name, ...)` → lazy-loading backtest (float32, rolling vol_p60)
  Note: `run_backtest()` internally calls `eng.set_esoteric_hazard_multiplier(0.0)` — now
  correct after the fix. Uses slightly different vol_p60 method (rolling, not gold).
- `ensure_jit()` → triggers numba JIT warmup (calls all numba functions once)

### Import Warning: dvae/__init__.py loads PyTorch
**File:** `nautilus_dolphin/dvae/__init__.py`

```python
from .hierarchical_dvae import HierarchicalDVAE   # ← loads PyTorch!
from .corpus_builder import DolphinCorpus
```

**CONSEQUENCE:** Any `from dvae.exp_shared import ...` statement will import PyTorch
via `dvae/__init__.py`, consuming ~700MB+ RAM and potentially causing OOM.

**CORRECT PATTERN:**
```python
_HERE = Path(__file__).resolve().parent  # dvae/ directory
sys.path.insert(0, str(_HERE))           # add dvae/ to path
from exp_shared import run_backtest, GOLD  # direct import, no __init__.py
```

### Trace Backtest (painstaking per-tick logger)
**File:** `nautilus_dolphin/dvae/run_trace_backtest.py` (created 2026-03-22)

Produces per-tick, per-trade, and per-day CSV trace files.
See §2 Step 4 for usage and output format.

---

## 9. KNOWN BUGS AND FIXES

### BUG 1 (CRITICAL — FIXED 2026-03-22): set_esoteric_hazard_multiplier stomps D_LIQ leverage

**File:** `nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py` line 707–720

**Symptom:** D_LIQ backtests give ROI=145.84% instead of gold 181.81%.

**Root cause:** `ceiling_lev = 6.0` was hardcoded. When called with `hazard_score=0.0`,
the function set `base_max_leverage = 6.0` and `bet_sizer.max_leverage = 6.0`,
overwriting D_LIQ's `_extended_soft_cap = 8.0`. On all non-ACB-boosted days
(~40 of 56 days), this reduced available leverage from 8.0x to 6.0x = **33% less**.
The resulting ROI gap: 181.81% - 145.84% = **35.97pp**.

**Fix:**
```python
# BEFORE (wrong):
ceiling_lev = 6.0

# AFTER (correct):
ceiling_lev = getattr(self, '_extended_soft_cap', 6.0)
```

**Verified:** After fix, `eng.set_esoteric_hazard_multiplier(0.0)` on a D_LIQ engine
gives `base_max_leverage = 8.0` (was 6.0). NDAlphaEngine (no `_extended_soft_cap`)
still gets 6.0 — backwards compatible.

**Callers affected:**
- `exp_shared.py:176` → `run_backtest()` → now correct
- `tests/test_proxy_boost_production.py` → `_run_engine()` → now correct
- `titan_stage1_run.py:130` → now correct
- `dvae/exp9b_liquidation_guard.py`, all exp*.py → now correct
- `run_esoteric_throttled_backtest.py:142` → now correct (correctly scales from 8.0 ceiling)

**Paradox resolved:** `titan_stage1_run.log` shows ROI=181.81% WITH the stomp call.
This was because when titan_stage1 ran (2026-03-15), the function either (a) didn't
exist yet and was added to titan_stage1_run.py after the run, or (b) the function had
a different implementation. Since titan_stage1_run.py is untracked, git history is
unavailable. The 181.81% result is authentic (confirmed by 9/9 e2e tests on 2026-03-15).

### BUG 2 (FIXED): dvae/__init__.py PyTorch OOM

See §8. Fixed by using `sys.path.insert(0, str(_HERE))` before importing from `exp_shared`.

### BUG 3 (FIXED 2026-03-09): NG5 scanner PriceData extraction

NG5 scanner had a PriceData extraction bug causing all-zero eigenvalues since 2026-03-08.
Fixed in `eso_cache/` and backfill scripts. All eigenvalue data now correct.

### BUG 4 (KNOWN): MC model sklearn version mismatch

MC models saved with sklearn 1.7.1, loaded with 1.8.0. Produces warnings but works.
Retraining models with sklearn 1.8.0 would eliminate warnings.

### BUG 5 (KNOWN): OOM during numba JIT warmup on low-RAM systems

With <1.5GB free RAM, `ensure_jit()` (which compiles all numba functions) may fail.
**Workaround:** Use `NUMBA_DISABLE_JIT=1` for quick checks, or wait for RAM to free.
Numba cache (95 `.nbc` files in `nautilus_dolphin/nautilus_dolphin/nautilus/__pycache__/`)
should be warmed from previous runs.

---

## 10. RESEARCH HISTORY (Exp1–Exp15)

All experiments ran on the 56-day (Dec31 2025 – Feb26 2026) dataset, T=2155 baseline.

### Foundation: NDAlphaEngine BRONZE baseline
- ROI=88.55%, PF=1.215, DD=15.05%, Sharpe=4.38, T=2155
- Script: `test_pf_dynamic_beta_validate.py`

### Exp4: proxy_B Signal Characterization
- Signal: `proxy_B = instability_50 - v750_lambda_max_velocity`
- AUC=0.715 for eigenspace stress detection
- r=+0.42 (p=0.003) correlation with intraday MAE
- r=-0.03 (ns) with vel_div — ORTHOGONAL → pure position-sizing layer
- Results: `dvae/exp4_proxy_coupling.py`

### Exp6: Stop Tightening Research (DEAD)
- All 8 configs fail: global stops cause re-entry cascade (1415→2916 trades)
- Best gated: ROI=38.42% DD=19.42% — far worse than baseline
- `stop_pct=1.0` (effectively OFF) confirmed optimal
- Results: `dvae/exp6_stop_test_results.json`

### Exp7: Live Coupling (scale_boost)
- `scale_boost(thr=0.35, alpha=1.0)`: ROI=93.61% DD=14.51% T=2155 (+5.06pp ROI, -0.54pp DD)
- `hold_limit` and `rising_exit`: 91% early exit rate → cascade → DEAD
- Results: `dvae/exp7_live_coupling_results.json`

### Exp8: Adaptive Parameterization → PREV GOLD
- `adaptive_beta(thr=0.35, alpha×(1+day_beta))`: ROI=96.55% DD=14.32% T=2155
- NOT overfitting: both H1+H2 temporal halves improve vs baseline
- **Was GOLD from 2026-03-14 to 2026-03-15**
- Results: `dvae/exp8_boost_robustness_results.json`

### Exp9: Extended Leverage Ceiling (8 configs)
- MC-Forewarner stays GREEN at ALL leverage levels tested (5x through 10x)
- DD plateau after 7x: each +1x costs only ~+0.12pp DD (convex curve)
- Best ROI: E(9/10x)=184.00% DD=18.56% — liquidation risk unmodeled
- MC decoupling via `begin_day` lever swap (mc_ref=5.0) confirmed essential
- Results: `dvae/exp9_leverage_ceiling_results.json`

### Exp9b: Liquidation Guard → D_LIQ_GOLD
- D_liq(8/9x) + liquidation floor at 10.56% (=1/9×0.95):
  **ROI=181.81%, DD=17.65%, Calmar=10.30, T=2155, liq_stops=1**
- E_liq(9/10x): 5 stops → cascade → DEAD (DD=31.79%)
- D_liq is the sweet spot: high leverage, protected against forced liquidation
- **BECAME GOLD 2026-03-15**
- Results: `dvae/exp9b_liquidation_guard_results.json`

### Exp9c: Overfitting Validation
- H1/H2 split: D_LIQ wins ROI in BOTH halves; Calmar dips H2 (-0.23 margin)
- Q1–Q4 split: PASS Q1/Q2, marginal FAIL Q3/Q4 (Q2 carries most outperformance)
- Buffer sweep 0.80–1.00: 0.90–1.00 identical; 0.80 cascades (5 stops)
- Verdict: regime-conditional upgrade — works best in volatile markets (Q1/Q2)
- Results: `dvae/exp9c_overfitting_results.json`

### Exp10: 1m Keyframe Gate (DEAD)
- i150_z1m + TITAN recon_err z-scores — all z_recon signals hurt performance
- Calmar 7.70 vs 7.82 threshold. Killed.

### Exp11: z_recon_inv Baseline (DEAD)
- Noise floor, scale_mean=0.999. No signal. Killed.

### Exp12: ConvNeXt z-Gate (INCONCLUSIVE)
- Used wrong ep=17 model (not v2 BOB). z10_inv +1.41pp ROI but Calmar 7.73 < 7.83.

### Exp13 v2: ZLeverageGateEngine → CONFIRMED SIGNAL ⭐
- Signal: daily z[13] from 1m ConvNeXt v2 BOB (r=+0.933 with proxy_B)
- **9/20 Phase 2 configs PASS** Calmar > 7.83 vs D_LIQ_GOLD baseline
- Best: `A_P5_M2_W1_a0.5` → ROI=186.40% (+4.59pp), Calmar=7.87
- Compounding: +$2.38M on $25k over 1 year (+11.1%) — zero DD cost
- **⚠️ PENDING PRODUCTIONIZATION** (see `memory/todo_productize.md`)
- Results: `dvae/exp13_multiscale_sweep_results.json`

### Exp14: z[13]/z_post_std/delta leverage gate sweep
- Running at time of writing. Results → `dvae/exp14_results.json`

### Exp15: z[13]-gated per-trade stop+TP extension
- Running at time of writing. Results → `dvae/exp15_results.json`

---

## 11. FILE MAP — ALL CRITICAL PATHS

### Core Engine Files

```
nautilus_dolphin/
└── nautilus_dolphin/
    └── nautilus/
        ├── esf_alpha_orchestrator.py     ← NDAlphaEngine (BASE) — ALL core logic
        │   Lines of interest:
        │     68–83:   NDTradeRecord dataclass
        │     86–720:  NDAlphaEngine class
        │     196:     self.trade_history: List[NDTradeRecord]
        │     241–289: step_bar() — streaming API
        │     294–357: process_bar() — per-bar entry/exit
        │     358–450: _execute_exit() — exit finalization
        │     707–720: set_esoteric_hazard_multiplier() ← BUG FIXED 2026-03-22
        │     779–826: begin_day() — streaming API
        │     827–850: end_day() — streaming API
        │
        ├── proxy_boost_engine.py         ← ProxyBase + AdaptiveBoost + ExtendedLev + LiqGuard
        │   Lines of interest:
        │     1–36:    module docstring with gold numbers
        │     47–103:  create_boost_engine() factory
        │     110–203: ProxyBaseEngine — process_day() + _update_proxy()
        │     209–303: AdaptiveBoostEngine — scale-boost logic
        │     311–385: ExtendedLeverageEngine — MC decoupling begin_day()
        │     392–430: LiquidationGuardEngine — _try_entry() + _execute_exit()
        │     437–465: D_LIQ constants + create_d_liq_engine() factory
        │
        ├── adaptive_circuit_breaker.py   ← ACBv6: day_base_boost + day_beta
        ├── alpha_exit_manager.py         ← Exit logic: FIXED_TP, MAX_HOLD, STOP, OB
        ├── alpha_bet_sizer.py            ← compute_sizing_nb() — numba leverage sizing
        ├── alpha_asset_selector.py       ← compute_irp_nb() — IRP asset ranking
        ├── alpha_signal_generator.py     ← check_dc_nb() — direction confirmation
        ├── ob_features.py                ← OB feature computation (numba)
        └── ob_provider.py                ← MockOBProvider / CSVOBProvider
```

### Test & Research Files

```
nautilus_dolphin/
├── tests/
│   └── test_proxy_boost_production.py    ← 9 e2e tests, 42 unit tests (UNTRACKED)
│
├── dvae/
│   ├── exp_shared.py                     ← Shared test infrastructure (gold data, run_backtest)
│   ├── run_trace_backtest.py             ← Painstaking per-tick trace logger (new 2026-03-22)
│   ├── test_dliq_fix_verify.py           ← Quick D_LIQ gold reproduction verify (new 2026-03-22)
│   ├── test_dliq_nostomp.py              ← WITH vs WITHOUT stomp comparison
│   │
│   ├── exp9b_liquidation_guard.py        ← Exp9b original research
│   ├── exp9b_retest.log                  ← Retest attempt (died with sklearn warnings)
│   ├── exp9c_overfitting_validation.py   ← Overfitting validation research
│   ├── exp13_multiscale_sweep.py         ← ConvNeXt z[13] gate sweep
│   ├── exp14_sweep.py                    ← z[13] leverage gate sweep (running)
│   ├── exp15_stop_gate.py                ← z[13] stop+TP gate (running)
│   │
│   ├── trace/                            ← Created by run_trace_backtest.py
│   │   ├── tick_trace.csv                ← Per-bar system state (~69MB)
│   │   ├── trade_trace.csv               ← Per-trade details (~320KB)
│   │   ├── daily_trace.csv               ← Per-day summary
│   │   └── summary.json                  ← Final ROI/DD/T/Calmar
│   │
│   └── convnext_model_v2.json            ← 1m ConvNeXt v2 BOB (z[13]=proxy_B, r=0.933)
│       = convnext_model_1m_bob.json      ← symlink/copy
│
└── run_logs/
    └── REGISTRY.md                       ← CANONICAL inter-agent sync file — READ FIRST
```

### Data Paths

```
C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\
├── vbt_cache\                            ← 56 daily parquet files (5s scans, primary backtest)
│   └── 2025-12-31.parquet .. 2026-02-26.parquet
│
├── vbt_cache_klines\                     ← 1710 daily parquet files (1m OHLCV, 2021–2026)
│
├── external_factors\
│   ├── eso_cache\
│   │   └── latest_esoteric_factors.json  ← Current lunar/astro/session state
│   ├── realtime_exf_service.py           ← Live ExF update service
│   └── backfill_patch_npz.py             ← Historical ExF backfill
│
└── nautilus_dolphin\
    ├── mc_results\models\                ← MC-Forewarner trained models
    └── run_logs\REGISTRY.md              ← Canonical registry
```

### Production Files (live trading)

```
nautilus_dolphin/
├── prod/
│   ├── dolphin_actor.py                  ← Live trading actor (reads boost_mode from config)
│   ├── paper_trade_flow.py               ← Prefect paper trading flow
│   ├── vbt_backtest_flow.py              ← Prefect backtest flow
│   ├── mc_forewarner_flow.py             ← MC update flow (every 4h)
│   └── esof_update_flow.py               ← EsoF update flow (every 6h)
│
└── dolphin_vbt_real.py                   ← (parent dir) Live VBT backtest runner
```

---

## 12. FAILURE MODE REFERENCE

### T ≠ 2155 (wrong trade count)

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| T < 2155 | Missing parquet files / wrong date range | Check vbt_cache, ensure 56 files Dec31–Feb26 |
| T < 2155 | Wrong vol_p60 (too high → too many bars filtered) | Use gold load_data() method |
| T ≠ 2155 | Wrong seed → different DC/IRP randomness | Ensure seed=42 in ENGINE_KWARGS |
| T > 2155 | Liquidation stop cascade (re-entries) | margin_buffer too small or abs_cap too high |

### ROI ≈ 145% instead of 181.81%

**Cause:** `set_esoteric_hazard_multiplier()` stomping `base_max_leverage` to 6.0.
**Fix:** Apply the `getattr(self, '_extended_soft_cap', 6.0)` fix. See §9 Bug 1.

### ROI ≈ 88–96% instead of 181.81%

**Cause:** Using wrong engine. `create_boost_engine(mode='adaptive_beta')` gives 96.55%.
`NDAlphaEngine` gives 88.55%. Must use `create_d_liq_engine()` for 181.81%.

### OOM during test run

**Cause 1:** `from dvae.exp_shared import ...` triggers `dvae/__init__.py` → PyTorch load.
**Fix:** Use `sys.path.insert(0, str(_HERE)); from exp_shared import ...` (direct import).

**Cause 2:** `ensure_jit()` numba compilation with <1.5GB free RAM.
**Fix:** Close memory-hungry apps. Numba cache (95 `.nbc` files) should prevent recompilation.
Check: `ls nautilus_dolphin/nautilus_dolphin/nautilus/__pycache__/*.nbc | wc -l` → 95.

### MC-Forewarner fires RED/ORANGE

**Cause:** `set_mc_forewarner()` called BEFORE `set_esoteric_hazard_multiplier()`.
Or `mc_leverage_ref` not set to 5.0 (mc_ref must match MC trained range ~5x).
**Expected:** 0 RED / 0 ORANGE when mc_ref=5.0. Any RED day halts trading → fewer trades.

### sklearn InconsistentVersionWarning

Not a bug — MC models saved with 1.7.1, loaded with 1.8.0. Warnings are harmless.
Suppress: `import warnings; warnings.filterwarnings('ignore', category=InconsistentVersionWarning)`

---

## APPENDIX: Quick Commands

```bash
# Working directory for all commands:
cd "C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin"

# Verify fix (fast, no numba):
python -c "
import os; os.environ['NUMBA_DISABLE_JIT']='1'
import sys; sys.path.insert(0, '.')
from nautilus_dolphin.nautilus.proxy_boost_engine import create_d_liq_engine
from dvae.exp_shared import ENGINE_KWARGS
e=create_d_liq_engine(**ENGINE_KWARGS)
e.set_esoteric_hazard_multiplier(0.0)
assert e.base_max_leverage==8.0, 'STOMP ACTIVE'
print('FIX OK: base_max_leverage=', e.base_max_leverage)
"

# Quick D_LIQ gold reproduction (~400s with lazy loading):
python dvae/test_dliq_fix_verify.py

# Full painstaking trace with per-tick/trade logging (~400s):
python dvae/run_trace_backtest.py

# Full e2e suite (9 tests, ~52 minutes):
pytest -m slow tests/test_proxy_boost_production.py -v

# Check REGISTRY for latest run history:
type run_logs\REGISTRY.md | more
```