DOLPHIN/nautilus_dolphin/dvae/exp9_leverage_ceiling.py

"""
Exp 9 — Extended Leverage Ceiling Test

Motivation:
  GOLD (adaptive_beta) achieved DD=14.32% (−0.72pp vs Silver=15.05%). The question is whether
  this headroom permits raising the leverage ceiling above the current 5x soft / 6x hard cap
  without exceeding Silver's DD budget — or ideally staying below GOLD's DD while gaining ROI.

Architecture recap (DO NOT MODIFY any of the below):
  bet_sizer.max_leverage  (5.0) → convex curve soft cap — strength_score^3 tops out here
  base_max_leverage        (5.0) → base for ACB regime excursions
  abs_max_leverage         (6.0) → hard ceiling: min(base*regime_mult, abs_max)
  ACB regime_size_mult     (1.0–1.6) → can push raw_leverage above bet_sizer.max_leverage
  → Current actual range: 0.5x – 6.0x (6x hard cap is binding on extreme signals)

Fork strategy:
  ExtendedLeverageEngine(AdaptiveBoostEngine) — subclass only, zero parent modification.
  Sets all three caps consistently: bet_sizer.max_leverage = base_max_leverage = extended_soft_cap,
  abs_max_leverage = extended_abs_cap.
  Example at 7.0/8.0: extreme signal gets bet_sizer→7.0 × regime_mult≈1.2 = 8.4, clamped at 8.0.

MC-Forewarner interaction (CRITICAL):
  begin_day() feeds MC: mc_cfg['max_leverage'] = base_max_leverage * day_base_boost
  MC was trained at max_leverage 5.0–6.0 (champion region). At 8x+ it's outside training
  distribution → One-Class SVM envelope_score likely < -1.0 → RED → regime_dd_halt=True
  → ZERO trades that day.

  Two modes tested:
    mc_ref=5.0  ("decoupled"): MC sees original 5.0x reference → same verdicts as GOLD
                               Pure effect of higher ceiling with MC behavior unchanged
    mc_ref=soft_cap ("coupled"): MC sees actual new cap → may flag RED/ORANGE on more days
                                 Tests "what would MC do if it knew about the new leverage"

  ExtendedLeverageEngine.begin_day() temporarily swaps base_max_leverage to mc_leverage_ref
  before calling super().begin_day(), then restores extended caps for actual trading.
  This is the ONLY divergence from parent logic.

Test configs:
  A: 5.0/6.0  mc_ref=5.0  — GOLD reference (must reproduce 96.55% / 14.32%)
  B: 6.0/7.0  mc_ref=5.0  — +1x soft, MC decoupled
  C: 7.0/8.0  mc_ref=5.0  — +2x soft, MC decoupled
  D: 8.0/9.0  mc_ref=5.0  — +3x soft, MC decoupled
  E: 9.0/10.0 mc_ref=5.0  — +4x soft, MC decoupled
  F: 6.0/7.0  mc_ref=6.0  — +1x soft, MC coupled
  G: 7.0/8.0  mc_ref=7.0  — +2x soft, MC coupled
  H: 8.0/9.0  mc_ref=8.0  — +3x soft, MC coupled
  Total: 8 runs × ~250s ≈ 35 min

Key metrics:
  ROI%, DD%, PF, Trades, avg_leverage (mean realized leverage per trade)
  MC status breakdown: RED days / ORANGE days / OK days / halted days
  Δ vs GOLD: ΔROI, ΔDD

Results → exp9_leverage_ceiling_results.json
"""

import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np

_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))

from exp_shared import (
    ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
    load_data, load_forewarner, log_results,
)
from nautilus_dolphin.nautilus.proxy_boost_engine import AdaptiveBoostEngine, DEFAULT_THRESHOLD, DEFAULT_ALPHA
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker

# ── GOLD reference for this experiment (adaptive_beta, not the old silver) ───
_GOLD_EXP9 = dict(roi=96.55, dd=14.32, trades=2155)  # adaptive_beta GOLD
_SILVER     = dict(roi=88.55, dd=15.05, trades=2155)  # former gold = regression floor


# ── ExtendedLeverageEngine — subclass only, ZERO parent modification ──────────

class ExtendedLeverageEngine(AdaptiveBoostEngine):
    """
    Extended leverage ceiling fork of AdaptiveBoostEngine (adaptive_beta GOLD).

    Three new constructor params vs parent:
      extended_soft_cap  : replaces max_leverage (5.0) and base_max_leverage
      extended_abs_cap   : replaces abs_max_leverage (6.0) — hard ceiling
      mc_leverage_ref    : what to feed the MC-Forewarner (independent of actual caps)
                           If None → MC sees extended_soft_cap (fully coupled)
                           Set to 5.0 → MC always sees original 5x reference (decoupled)

    ALL existing leverage logic (convex curve, ACB regime_size_mult, STALKER 2x cap,
    proxy_B scale_boost, etc.) is completely untouched.  Only the three cap values change.
    """

    def __init__(
        self,
        *args,
        extended_soft_cap: float = 5.0,
        extended_abs_cap: float = 6.0,
        mc_leverage_ref: float = None,
        **kwargs,
    ):
        # Inject extended caps as max_leverage / abs_max_leverage so parent stores them
        kwargs['max_leverage'] = extended_soft_cap
        kwargs['abs_max_leverage'] = extended_abs_cap
        super().__init__(*args, **kwargs)

        # Explicitly ensure all three cap locations are consistent
        # (parent __init__ sets base_max_leverage = max_leverage and bet_sizer.max_leverage
        # from the kwargs, but we set explicitly for clarity)
        self.bet_sizer.max_leverage = extended_soft_cap
        self.base_max_leverage = extended_soft_cap
        self.abs_max_leverage = extended_abs_cap

        self._extended_soft_cap = extended_soft_cap
        self._extended_abs_cap = extended_abs_cap
        # MC reference: 5.0 = decoupled (MC sees original gold reference)
        #               extended_soft_cap = coupled (MC sees actual new cap)
        self._mc_leverage_ref = mc_leverage_ref if mc_leverage_ref is not None else extended_soft_cap

        # Per-day MC verdict monitoring
        self.mc_monitor = dict(red=0, orange=0, ok=0, halted=0, total=0)

    def begin_day(self, date_str: str, posture: str = 'APEX', direction=None) -> None:
        """
        Temporarily expose mc_leverage_ref to the MC-Forewarner assessment,
        then restore the true extended caps for actual trading.

        The parent begin_day() computes:
            mc_cfg['max_leverage'] = self.base_max_leverage * self._day_base_boost
        By setting base_max_leverage = mc_leverage_ref before the call, MC sees the
        reference leverage.  After super().begin_day() returns, regime_dd_halt and
        _day_mc_scale are already set correctly for that reference.  We then restore
        the true extended caps so _try_entry uses the full new ceiling.
        """
        # Save true extended caps
        _true_base  = self.base_max_leverage
        _true_abs   = self.abs_max_leverage
        _true_sizer = self.bet_sizer.max_leverage

        # Temporarily expose mc_leverage_ref to parent's MC assessment
        self.base_max_leverage    = self._mc_leverage_ref
        self.bet_sizer.max_leverage = self._mc_leverage_ref
        self.abs_max_leverage     = self._mc_leverage_ref  # only affects _try_entry, safe to temp-set

        super().begin_day(date_str, posture=posture, direction=direction)

        # Restore true extended caps — all trading this day uses extended ceiling
        self.base_max_leverage    = _true_base
        self.bet_sizer.max_leverage = _true_sizer
        self.abs_max_leverage     = _true_abs

        # Record MC verdict for monitoring
        self.mc_monitor['total'] += 1
        status = self._day_mc_status
        if status == 'RED':
            self.mc_monitor['red'] += 1
        elif status == 'ORANGE':
            self.mc_monitor['orange'] += 1
        else:
            self.mc_monitor['ok'] += 1
        if self.regime_dd_halt:
            self.mc_monitor['halted'] += 1

    def reset(self):
        super().reset()
        # NDAlphaEngine.reset() rebuilds bet_sizer from self.bet_sizer.max_leverage
        # (which at reset time = _extended_soft_cap, so it rebuilds correctly).
        # But we re-apply explicitly to be safe.
        self.bet_sizer.max_leverage = self._extended_soft_cap
        self.base_max_leverage = self._extended_soft_cap
        self.abs_max_leverage = self._extended_abs_cap
        # Reset monitoring
        self.mc_monitor = dict(red=0, orange=0, ok=0, halted=0, total=0)


# ── Run harness ───────────────────────────────────────────────────────────────

def _run(engine_factory, name, d, fw):
    """Full 55-day run + MC monitoring + avg_leverage tracking."""
    kw = ENGINE_KWARGS.copy()
    acb = AdaptiveCircuitBreaker()
    acb.preload_w750(d['date_strings'])

    eng = engine_factory(kw)
    eng.set_ob_engine(d['ob_eng'])
    eng.set_acb(acb)
    if fw is not None:
        eng.set_mc_forewarner(fw, MC_BASE_CFG)
    eng.set_esoteric_hazard_multiplier(0.0)

    daily_caps, daily_pnls = [], []

    for pf in d['parquet_files']:
        ds = pf.stem
        df, acols, dvol = d['pq_data'][ds]
        cap_before = eng.capital
        vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
        eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
        daily_caps.append(eng.capital)
        daily_pnls.append(eng.capital - cap_before)

    tr  = eng.trade_history
    n   = len(tr)
    roi = (eng.capital - 25000.0) / 25000.0 * 100.0

    if n == 0:
        mc_mon = getattr(eng, 'mc_monitor', {})
        return dict(name=name, roi=roi, pf=0.0, dd=0.0, wr=0.0, sharpe=0.0,
                    trades=0, avg_leverage=0.0, sizing_scale_mean=1.0,
                    mc_monitor=mc_mon)

    def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
    wins   = [t for t in tr if _abs(t) > 0]
    losses = [t for t in tr if _abs(t) <= 0]
    wr     = len(wins) / n * 100.0
    pf_val = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)

    peak_cap, max_dd = 25000.0, 0.0
    for cap in daily_caps:
        peak_cap = max(peak_cap, cap)
        max_dd   = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)

    dr     = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
    sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0

    # Realized leverage stats
    lev_vals = [t.leverage for t in tr if hasattr(t, 'leverage') and t.leverage > 0]
    avg_lev  = float(np.mean(lev_vals)) if lev_vals else 0.0
    max_lev  = float(np.max(lev_vals)) if lev_vals else 0.0
    lev_p90  = float(np.percentile(lev_vals, 90)) if lev_vals else 0.0
    lev_at_cap = sum(1 for v in lev_vals if v >= (eng.abs_max_leverage - 0.05))
    pct_at_cap = lev_at_cap / len(lev_vals) * 100.0 if lev_vals else 0.0

    # proxy_B scale stats
    scale_mean = getattr(eng, 'sizing_scale_mean', 1.0)

    # MC monitoring
    mc_mon = getattr(eng, 'mc_monitor', {})

    return dict(
        name=name,
        roi=roi, pf=pf_val, dd=max_dd, wr=wr, sharpe=sharpe, trades=n,
        avg_leverage=avg_lev, max_leverage_realized=max_lev,
        lev_p90=lev_p90,
        pct_at_hard_cap=pct_at_cap,
        sizing_scale_mean=scale_mean,
        mc_monitor=mc_mon,
    )


# ── Main ─────────────────────────────────────────────────────────────────────

def main():
    t_start = time.time()
    print("=" * 76)
    print("Exp 9 — Extended Leverage Ceiling Test (fork of GOLD adaptive_beta)")
    print("=" * 76)
    print(f"  GOLD ref  : ROI={_GOLD_EXP9['roi']:.2f}% DD={_GOLD_EXP9['dd']:.2f}%  (adaptive_beta)")
    print(f"  SILVER ref: ROI={_SILVER['roi']:.2f}% DD={_SILVER['dd']:.2f}%  (former gold, regression floor)")

    ensure_jit()
    d  = load_data()
    fw = load_forewarner()

    # Config: (label, extended_soft_cap, extended_abs_cap, mc_leverage_ref)
    configs = [
        # ── Decoupled (MC sees original 5.0x reference across all runs) ──────────
        ("A_5.0/6.0_mc5.0_GOLD-ref",     5.0,  6.0,  5.0),  # must reproduce GOLD 96.55%
        ("B_6.0/7.0_mc5.0_decoupled",    6.0,  7.0,  5.0),
        ("C_7.0/8.0_mc5.0_decoupled",    7.0,  8.0,  5.0),
        ("D_8.0/9.0_mc5.0_decoupled",    8.0,  9.0,  5.0),
        ("E_9.0/10.0_mc5.0_decoupled",   9.0, 10.0,  5.0),
        # ── Coupled (MC sees actual new cap → tests MC's response) ───────────────
        ("F_6.0/7.0_mc6.0_coupled",      6.0,  7.0,  6.0),
        ("G_7.0/8.0_mc7.0_coupled",      7.0,  8.0,  7.0),
        ("H_8.0/9.0_mc8.0_coupled",      8.0,  9.0,  8.0),
    ]

    # adaptive_beta proxy_B params (match GOLD exactly)
    _PROXY_KWARGS = dict(threshold=DEFAULT_THRESHOLD, alpha=DEFAULT_ALPHA,
                         adaptive_beta=True, adaptive_alpha=False, adaptive_thr=False)

    results = []
    for i, (label, soft, hard, mc_ref) in enumerate(configs):
        t0 = time.time()
        print(f"\n[{i+1}/{len(configs)}] {label} ...")

        def _factory(kw, s=soft, h=hard, r=mc_ref):
            return ExtendedLeverageEngine(
                extended_soft_cap=s,
                extended_abs_cap=h,
                mc_leverage_ref=r,
                **_PROXY_KWARGS,
                **kw,
            )

        res = _run(_factory, label, d, fw)
        elapsed = time.time() - t0

        mc = res['mc_monitor']
        mc_str = (f"  MC: ok={mc.get('ok',0)} orange={mc.get('orange',0)} "
                  f"red={mc.get('red',0)} halted={mc.get('halted',0)}/{mc.get('total',0)}")
        dROI = res['roi'] - _GOLD_EXP9['roi']
        dDD  = res['dd']  - _GOLD_EXP9['dd']
        print(f"  ROI={res['roi']:>7.2f}% (Δ{dROI:+.2f}pp)  DD={res['dd']:>6.2f}% (Δ{dDD:+.2f}pp)  "
              f"PF={res['pf']:.4f}  Trades={res['trades']}")
        print(f"  avg_lev={res['avg_leverage']:.2f}x  p90={res['lev_p90']:.2f}x  "
              f"max={res['max_leverage_realized']:.2f}x  at_hard_cap={res['pct_at_hard_cap']:.1f}%  "
              f"scale_mean={res['sizing_scale_mean']:.4f}")
        print(f"{mc_str}  ({elapsed:.0f}s)")

        results.append(res)

    # ── Verification ─────────────────────────────────────────────────────────
    gold_ref = results[0]
    gold_ok = (abs(gold_ref['roi'] - _GOLD_EXP9['roi']) < 0.5 and
               abs(gold_ref['dd']  - _GOLD_EXP9['dd'])  < 0.5 and
               abs(gold_ref['trades'] - _GOLD_EXP9['trades']) < 10)
    print(f"\n{'='*76}")
    print(f"GOLD REFERENCE VERIFICATION: {'PASS ✓' if gold_ok else 'FAIL ✗'}")
    print(f"  Expected: ROI={_GOLD_EXP9['roi']:.2f}% DD={_GOLD_EXP9['dd']:.2f}%  "
          f"Got: ROI={gold_ref['roi']:.2f}% DD={gold_ref['dd']:.2f}%")

    # ── Results table ─────────────────────────────────────────────────────────
    print(f"\n{'='*76}")
    print("FULL RESULTS (vs GOLD = adaptive_beta 96.55% / 14.32%)")
    print(f"{'Config':<38} {'ROI%':>7} {'ΔROI':>6} {'DD%':>6} {'ΔDD':>6} "
          f"{'Trades':>7} {'avgLev':>7} {'p90Lev':>7} {'@cap%':>6} {'OK?':>4}")
    print('-' * 100)
    for r in results:
        dROI = r['roi'] - _GOLD_EXP9['roi']
        dDD  = r['dd']  - _GOLD_EXP9['dd']
        # Pass: better than GOLD on DD, not worse than SILVER on ROI
        ok = ('Y' if r['dd'] < _GOLD_EXP9['dd'] and r['roi'] >= _SILVER['roi'] else
              'G' if r['dd'] < _SILVER['dd']  and r['roi'] >= _SILVER['roi'] else
              'N')
        print(f"{r['name']:<38} {r['roi']:>7.2f} {dROI:>+6.2f} {r['dd']:>6.2f} {dDD:>+6.2f} "
              f"{r['trades']:>7} {r['avg_leverage']:>7.2f} {r['lev_p90']:>7.2f} "
              f"{r['pct_at_hard_cap']:>6.1f} {ok:>4}")
    print("  OK=Y: beats GOLD on DD + above SILVER ROI | OK=G: beats SILVER on both | OK=N: fail")

    # ── MC interaction table ──────────────────────────────────────────────────
    print(f"\n{'='*76}")
    print("MC-FOREWARNER INTERACTION (per-day breakdown)")
    print(f"{'Config':<38} {'OK':>5} {'ORG':>5} {'RED':>5} {'HALT':>5} {'TOTAL':>6} {'halt%':>7}")
    print('-' * 75)
    for r in results:
        mc = r['mc_monitor']
        total = mc.get('total', 0)
        halt  = mc.get('halted', 0)
        halt_pct = halt / total * 100.0 if total else 0.0
        print(f"{r['name']:<38} {mc.get('ok',0):>5} {mc.get('orange',0):>5} "
              f"{mc.get('red',0):>5} {halt:>5} {total:>6} {halt_pct:>7.1f}%")

    # ── Decoupled vs Coupled delta at each leverage level ────────────────────
    print(f"\n{'='*76}")
    print("MC COUPLING EFFECT (decoupled vs coupled at same leverage level)")
    pairs = [
        ("B_6.0/7.0", "F_6.0/7.0"),
        ("C_7.0/8.0", "G_7.0/8.0"),
        ("D_8.0/9.0", "H_8.0/9.0"),
    ]
    rmap = {r['name'][:10]: r for r in results}
    for dec_label, coup_label in pairs:
        dec = next((r for r in results if dec_label in r['name']), None)
        cop = next((r for r in results if coup_label in r['name']), None)
        if dec and cop:
            halt_dec = dec['mc_monitor'].get('halted', 0)
            halt_cop = cop['mc_monitor'].get('halted', 0)
            print(f"  {dec_label}: decoupled ROI={dec['roi']:.2f}% DD={dec['dd']:.2f}% halted={halt_dec}d")
            print(f"  {coup_label}: coupled   ROI={cop['roi']:.2f}% DD={cop['dd']:.2f}% halted={halt_cop}d")
            print(f"    MC coupling cost: ΔROI={cop['roi']-dec['roi']:+.2f}pp  ΔDD={cop['dd']-dec['dd']:+.2f}pp  "
                  f"Δhalted={halt_cop-halt_dec:+d}d")
            print()

    # ── Best config summary ───────────────────────────────────────────────────
    print(f"{'='*76}")
    decoupled = [r for r in results if 'decoupled' in r['name'] or 'GOLD' in r['name']]
    best_roi = max(decoupled, key=lambda r: r['roi'])
    best_dd  = min(decoupled, key=lambda r: r['dd'])
    best_combined = max(decoupled, key=lambda r: r['roi'] - r['dd'])  # ROI−DD as proxy

    print(f"BEST (decoupled only — pure leverage effect):")
    print(f"  Best ROI: {best_roi['name']}  ROI={best_roi['roi']:.2f}% DD={best_roi['dd']:.2f}%")
    print(f"  Best DD:  {best_dd['name']}   ROI={best_dd['roi']:.2f}% DD={best_dd['dd']:.2f}%")
    print(f"  Best R−D: {best_combined['name']}  ROI={best_combined['roi']:.2f}% DD={best_combined['dd']:.2f}%")

    # ── Log ──────────────────────────────────────────────────────────────────
    outfile = _HERE / "exp9_leverage_ceiling_results.json"
    log_results(results, outfile, gold=_GOLD_EXP9, meta={
        "exp": "exp9",
        "question": "Does adaptive_beta DD headroom permit higher leverage ceiling?",
        "silver": _SILVER,
        "total_elapsed_s": round(time.time() - t_start, 1),
        "gold_ref_ok": gold_ok,
        "note_mc": (
            "Decoupled runs (mc_ref=5.0): MC assesses at original 5x reference — "
            "pure leverage cap test. Coupled runs: MC sees actual new cap — tests MC response."
        ),
    })

    total = time.time() - t_start
    print(f"\nTotal elapsed: {total / 60:.1f} min")
    print("Done.")


if __name__ == "__main__":
    main()