Files
DOLPHIN/nautilus_dolphin/dvae/exp9_leverage_ceiling.py

409 lines
20 KiB
Python
Raw Normal View History

"""
Exp 9 Extended Leverage Ceiling Test
Motivation:
GOLD (adaptive_beta) achieved DD=14.32% (0.72pp vs Silver=15.05%). The question is whether
this headroom permits raising the leverage ceiling above the current 5x soft / 6x hard cap
without exceeding Silver's DD budget — or ideally staying below GOLD's DD while gaining ROI.
Architecture recap (DO NOT MODIFY any of the below):
bet_sizer.max_leverage (5.0) convex curve soft cap strength_score^3 tops out here
base_max_leverage (5.0) base for ACB regime excursions
abs_max_leverage (6.0) hard ceiling: min(base*regime_mult, abs_max)
ACB regime_size_mult (1.01.6) can push raw_leverage above bet_sizer.max_leverage
Current actual range: 0.5x 6.0x (6x hard cap is binding on extreme signals)
Fork strategy:
ExtendedLeverageEngine(AdaptiveBoostEngine) subclass only, zero parent modification.
Sets all three caps consistently: bet_sizer.max_leverage = base_max_leverage = extended_soft_cap,
abs_max_leverage = extended_abs_cap.
Example at 7.0/8.0: extreme signal gets bet_sizer7.0 × regime_mult1.2 = 8.4, clamped at 8.0.
MC-Forewarner interaction (CRITICAL):
begin_day() feeds MC: mc_cfg['max_leverage'] = base_max_leverage * day_base_boost
MC was trained at max_leverage 5.06.0 (champion region). At 8x+ it's outside training
distribution One-Class SVM envelope_score likely < -1.0 RED regime_dd_halt=True
ZERO trades that day.
Two modes tested:
mc_ref=5.0 ("decoupled"): MC sees original 5.0x reference same verdicts as GOLD
Pure effect of higher ceiling with MC behavior unchanged
mc_ref=soft_cap ("coupled"): MC sees actual new cap may flag RED/ORANGE on more days
Tests "what would MC do if it knew about the new leverage"
ExtendedLeverageEngine.begin_day() temporarily swaps base_max_leverage to mc_leverage_ref
before calling super().begin_day(), then restores extended caps for actual trading.
This is the ONLY divergence from parent logic.
Test configs:
A: 5.0/6.0 mc_ref=5.0 GOLD reference (must reproduce 96.55% / 14.32%)
B: 6.0/7.0 mc_ref=5.0 +1x soft, MC decoupled
C: 7.0/8.0 mc_ref=5.0 +2x soft, MC decoupled
D: 8.0/9.0 mc_ref=5.0 +3x soft, MC decoupled
E: 9.0/10.0 mc_ref=5.0 +4x soft, MC decoupled
F: 6.0/7.0 mc_ref=6.0 +1x soft, MC coupled
G: 7.0/8.0 mc_ref=7.0 +2x soft, MC coupled
H: 8.0/9.0 mc_ref=8.0 +3x soft, MC coupled
Total: 8 runs × ~250s 35 min
Key metrics:
ROI%, DD%, PF, Trades, avg_leverage (mean realized leverage per trade)
MC status breakdown: RED days / ORANGE days / OK days / halted days
Δ vs GOLD: ΔROI, ΔDD
Results exp9_leverage_ceiling_results.json
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
load_data, load_forewarner, log_results,
)
from nautilus_dolphin.nautilus.proxy_boost_engine import AdaptiveBoostEngine, DEFAULT_THRESHOLD, DEFAULT_ALPHA
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# ── GOLD reference for this experiment (adaptive_beta, not the old silver) ───
_GOLD_EXP9 = dict(roi=96.55, dd=14.32, trades=2155) # adaptive_beta GOLD
_SILVER = dict(roi=88.55, dd=15.05, trades=2155) # former gold = regression floor
# ── ExtendedLeverageEngine — subclass only, ZERO parent modification ──────────
class ExtendedLeverageEngine(AdaptiveBoostEngine):
"""
Extended leverage ceiling fork of AdaptiveBoostEngine (adaptive_beta GOLD).
Three new constructor params vs parent:
extended_soft_cap : replaces max_leverage (5.0) and base_max_leverage
extended_abs_cap : replaces abs_max_leverage (6.0) hard ceiling
mc_leverage_ref : what to feed the MC-Forewarner (independent of actual caps)
If None MC sees extended_soft_cap (fully coupled)
Set to 5.0 MC always sees original 5x reference (decoupled)
ALL existing leverage logic (convex curve, ACB regime_size_mult, STALKER 2x cap,
proxy_B scale_boost, etc.) is completely untouched. Only the three cap values change.
"""
def __init__(
self,
*args,
extended_soft_cap: float = 5.0,
extended_abs_cap: float = 6.0,
mc_leverage_ref: float = None,
**kwargs,
):
# Inject extended caps as max_leverage / abs_max_leverage so parent stores them
kwargs['max_leverage'] = extended_soft_cap
kwargs['abs_max_leverage'] = extended_abs_cap
super().__init__(*args, **kwargs)
# Explicitly ensure all three cap locations are consistent
# (parent __init__ sets base_max_leverage = max_leverage and bet_sizer.max_leverage
# from the kwargs, but we set explicitly for clarity)
self.bet_sizer.max_leverage = extended_soft_cap
self.base_max_leverage = extended_soft_cap
self.abs_max_leverage = extended_abs_cap
self._extended_soft_cap = extended_soft_cap
self._extended_abs_cap = extended_abs_cap
# MC reference: 5.0 = decoupled (MC sees original gold reference)
# extended_soft_cap = coupled (MC sees actual new cap)
self._mc_leverage_ref = mc_leverage_ref if mc_leverage_ref is not None else extended_soft_cap
# Per-day MC verdict monitoring
self.mc_monitor = dict(red=0, orange=0, ok=0, halted=0, total=0)
def begin_day(self, date_str: str, posture: str = 'APEX', direction=None) -> None:
"""
Temporarily expose mc_leverage_ref to the MC-Forewarner assessment,
then restore the true extended caps for actual trading.
The parent begin_day() computes:
mc_cfg['max_leverage'] = self.base_max_leverage * self._day_base_boost
By setting base_max_leverage = mc_leverage_ref before the call, MC sees the
reference leverage. After super().begin_day() returns, regime_dd_halt and
_day_mc_scale are already set correctly for that reference. We then restore
the true extended caps so _try_entry uses the full new ceiling.
"""
# Save true extended caps
_true_base = self.base_max_leverage
_true_abs = self.abs_max_leverage
_true_sizer = self.bet_sizer.max_leverage
# Temporarily expose mc_leverage_ref to parent's MC assessment
self.base_max_leverage = self._mc_leverage_ref
self.bet_sizer.max_leverage = self._mc_leverage_ref
self.abs_max_leverage = self._mc_leverage_ref # only affects _try_entry, safe to temp-set
super().begin_day(date_str, posture=posture, direction=direction)
# Restore true extended caps — all trading this day uses extended ceiling
self.base_max_leverage = _true_base
self.bet_sizer.max_leverage = _true_sizer
self.abs_max_leverage = _true_abs
# Record MC verdict for monitoring
self.mc_monitor['total'] += 1
status = self._day_mc_status
if status == 'RED':
self.mc_monitor['red'] += 1
elif status == 'ORANGE':
self.mc_monitor['orange'] += 1
else:
self.mc_monitor['ok'] += 1
if self.regime_dd_halt:
self.mc_monitor['halted'] += 1
def reset(self):
super().reset()
# NDAlphaEngine.reset() rebuilds bet_sizer from self.bet_sizer.max_leverage
# (which at reset time = _extended_soft_cap, so it rebuilds correctly).
# But we re-apply explicitly to be safe.
self.bet_sizer.max_leverage = self._extended_soft_cap
self.base_max_leverage = self._extended_soft_cap
self.abs_max_leverage = self._extended_abs_cap
# Reset monitoring
self.mc_monitor = dict(red=0, orange=0, ok=0, halted=0, total=0)
# ── Run harness ───────────────────────────────────────────────────────────────
def _run(engine_factory, name, d, fw):
"""Full 55-day run + MC monitoring + avg_leverage tracking."""
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
for pf in d['parquet_files']:
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
if n == 0:
mc_mon = getattr(eng, 'mc_monitor', {})
return dict(name=name, roi=roi, pf=0.0, dd=0.0, wr=0.0, sharpe=0.0,
trades=0, avg_leverage=0.0, sizing_scale_mean=1.0,
mc_monitor=mc_mon)
def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf_val = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
# Realized leverage stats
lev_vals = [t.leverage for t in tr if hasattr(t, 'leverage') and t.leverage > 0]
avg_lev = float(np.mean(lev_vals)) if lev_vals else 0.0
max_lev = float(np.max(lev_vals)) if lev_vals else 0.0
lev_p90 = float(np.percentile(lev_vals, 90)) if lev_vals else 0.0
lev_at_cap = sum(1 for v in lev_vals if v >= (eng.abs_max_leverage - 0.05))
pct_at_cap = lev_at_cap / len(lev_vals) * 100.0 if lev_vals else 0.0
# proxy_B scale stats
scale_mean = getattr(eng, 'sizing_scale_mean', 1.0)
# MC monitoring
mc_mon = getattr(eng, 'mc_monitor', {})
return dict(
name=name,
roi=roi, pf=pf_val, dd=max_dd, wr=wr, sharpe=sharpe, trades=n,
avg_leverage=avg_lev, max_leverage_realized=max_lev,
lev_p90=lev_p90,
pct_at_hard_cap=pct_at_cap,
sizing_scale_mean=scale_mean,
mc_monitor=mc_mon,
)
# ── Main ─────────────────────────────────────────────────────────────────────
def main():
t_start = time.time()
print("=" * 76)
print("Exp 9 — Extended Leverage Ceiling Test (fork of GOLD adaptive_beta)")
print("=" * 76)
print(f" GOLD ref : ROI={_GOLD_EXP9['roi']:.2f}% DD={_GOLD_EXP9['dd']:.2f}% (adaptive_beta)")
print(f" SILVER ref: ROI={_SILVER['roi']:.2f}% DD={_SILVER['dd']:.2f}% (former gold, regression floor)")
ensure_jit()
d = load_data()
fw = load_forewarner()
# Config: (label, extended_soft_cap, extended_abs_cap, mc_leverage_ref)
configs = [
# ── Decoupled (MC sees original 5.0x reference across all runs) ──────────
("A_5.0/6.0_mc5.0_GOLD-ref", 5.0, 6.0, 5.0), # must reproduce GOLD 96.55%
("B_6.0/7.0_mc5.0_decoupled", 6.0, 7.0, 5.0),
("C_7.0/8.0_mc5.0_decoupled", 7.0, 8.0, 5.0),
("D_8.0/9.0_mc5.0_decoupled", 8.0, 9.0, 5.0),
("E_9.0/10.0_mc5.0_decoupled", 9.0, 10.0, 5.0),
# ── Coupled (MC sees actual new cap → tests MC's response) ───────────────
("F_6.0/7.0_mc6.0_coupled", 6.0, 7.0, 6.0),
("G_7.0/8.0_mc7.0_coupled", 7.0, 8.0, 7.0),
("H_8.0/9.0_mc8.0_coupled", 8.0, 9.0, 8.0),
]
# adaptive_beta proxy_B params (match GOLD exactly)
_PROXY_KWARGS = dict(threshold=DEFAULT_THRESHOLD, alpha=DEFAULT_ALPHA,
adaptive_beta=True, adaptive_alpha=False, adaptive_thr=False)
results = []
for i, (label, soft, hard, mc_ref) in enumerate(configs):
t0 = time.time()
print(f"\n[{i+1}/{len(configs)}] {label} ...")
def _factory(kw, s=soft, h=hard, r=mc_ref):
return ExtendedLeverageEngine(
extended_soft_cap=s,
extended_abs_cap=h,
mc_leverage_ref=r,
**_PROXY_KWARGS,
**kw,
)
res = _run(_factory, label, d, fw)
elapsed = time.time() - t0
mc = res['mc_monitor']
mc_str = (f" MC: ok={mc.get('ok',0)} orange={mc.get('orange',0)} "
f"red={mc.get('red',0)} halted={mc.get('halted',0)}/{mc.get('total',0)}")
dROI = res['roi'] - _GOLD_EXP9['roi']
dDD = res['dd'] - _GOLD_EXP9['dd']
print(f" ROI={res['roi']:>7.2f}% (Δ{dROI:+.2f}pp) DD={res['dd']:>6.2f}% (Δ{dDD:+.2f}pp) "
f"PF={res['pf']:.4f} Trades={res['trades']}")
print(f" avg_lev={res['avg_leverage']:.2f}x p90={res['lev_p90']:.2f}x "
f"max={res['max_leverage_realized']:.2f}x at_hard_cap={res['pct_at_hard_cap']:.1f}% "
f"scale_mean={res['sizing_scale_mean']:.4f}")
print(f"{mc_str} ({elapsed:.0f}s)")
results.append(res)
# ── Verification ─────────────────────────────────────────────────────────
gold_ref = results[0]
gold_ok = (abs(gold_ref['roi'] - _GOLD_EXP9['roi']) < 0.5 and
abs(gold_ref['dd'] - _GOLD_EXP9['dd']) < 0.5 and
abs(gold_ref['trades'] - _GOLD_EXP9['trades']) < 10)
print(f"\n{'='*76}")
print(f"GOLD REFERENCE VERIFICATION: {'PASS ✓' if gold_ok else 'FAIL ✗'}")
print(f" Expected: ROI={_GOLD_EXP9['roi']:.2f}% DD={_GOLD_EXP9['dd']:.2f}% "
f"Got: ROI={gold_ref['roi']:.2f}% DD={gold_ref['dd']:.2f}%")
# ── Results table ─────────────────────────────────────────────────────────
print(f"\n{'='*76}")
print("FULL RESULTS (vs GOLD = adaptive_beta 96.55% / 14.32%)")
print(f"{'Config':<38} {'ROI%':>7} {'ΔROI':>6} {'DD%':>6} {'ΔDD':>6} "
f"{'Trades':>7} {'avgLev':>7} {'p90Lev':>7} {'@cap%':>6} {'OK?':>4}")
print('-' * 100)
for r in results:
dROI = r['roi'] - _GOLD_EXP9['roi']
dDD = r['dd'] - _GOLD_EXP9['dd']
# Pass: better than GOLD on DD, not worse than SILVER on ROI
ok = ('Y' if r['dd'] < _GOLD_EXP9['dd'] and r['roi'] >= _SILVER['roi'] else
'G' if r['dd'] < _SILVER['dd'] and r['roi'] >= _SILVER['roi'] else
'N')
print(f"{r['name']:<38} {r['roi']:>7.2f} {dROI:>+6.2f} {r['dd']:>6.2f} {dDD:>+6.2f} "
f"{r['trades']:>7} {r['avg_leverage']:>7.2f} {r['lev_p90']:>7.2f} "
f"{r['pct_at_hard_cap']:>6.1f} {ok:>4}")
print(" OK=Y: beats GOLD on DD + above SILVER ROI | OK=G: beats SILVER on both | OK=N: fail")
# ── MC interaction table ──────────────────────────────────────────────────
print(f"\n{'='*76}")
print("MC-FOREWARNER INTERACTION (per-day breakdown)")
print(f"{'Config':<38} {'OK':>5} {'ORG':>5} {'RED':>5} {'HALT':>5} {'TOTAL':>6} {'halt%':>7}")
print('-' * 75)
for r in results:
mc = r['mc_monitor']
total = mc.get('total', 0)
halt = mc.get('halted', 0)
halt_pct = halt / total * 100.0 if total else 0.0
print(f"{r['name']:<38} {mc.get('ok',0):>5} {mc.get('orange',0):>5} "
f"{mc.get('red',0):>5} {halt:>5} {total:>6} {halt_pct:>7.1f}%")
# ── Decoupled vs Coupled delta at each leverage level ────────────────────
print(f"\n{'='*76}")
print("MC COUPLING EFFECT (decoupled vs coupled at same leverage level)")
pairs = [
("B_6.0/7.0", "F_6.0/7.0"),
("C_7.0/8.0", "G_7.0/8.0"),
("D_8.0/9.0", "H_8.0/9.0"),
]
rmap = {r['name'][:10]: r for r in results}
for dec_label, coup_label in pairs:
dec = next((r for r in results if dec_label in r['name']), None)
cop = next((r for r in results if coup_label in r['name']), None)
if dec and cop:
halt_dec = dec['mc_monitor'].get('halted', 0)
halt_cop = cop['mc_monitor'].get('halted', 0)
print(f" {dec_label}: decoupled ROI={dec['roi']:.2f}% DD={dec['dd']:.2f}% halted={halt_dec}d")
print(f" {coup_label}: coupled ROI={cop['roi']:.2f}% DD={cop['dd']:.2f}% halted={halt_cop}d")
print(f" MC coupling cost: ΔROI={cop['roi']-dec['roi']:+.2f}pp ΔDD={cop['dd']-dec['dd']:+.2f}pp "
f"Δhalted={halt_cop-halt_dec:+d}d")
print()
# ── Best config summary ───────────────────────────────────────────────────
print(f"{'='*76}")
decoupled = [r for r in results if 'decoupled' in r['name'] or 'GOLD' in r['name']]
best_roi = max(decoupled, key=lambda r: r['roi'])
best_dd = min(decoupled, key=lambda r: r['dd'])
best_combined = max(decoupled, key=lambda r: r['roi'] - r['dd']) # ROIDD as proxy
print(f"BEST (decoupled only — pure leverage effect):")
print(f" Best ROI: {best_roi['name']} ROI={best_roi['roi']:.2f}% DD={best_roi['dd']:.2f}%")
print(f" Best DD: {best_dd['name']} ROI={best_dd['roi']:.2f}% DD={best_dd['dd']:.2f}%")
print(f" Best RD: {best_combined['name']} ROI={best_combined['roi']:.2f}% DD={best_combined['dd']:.2f}%")
# ── Log ──────────────────────────────────────────────────────────────────
outfile = _HERE / "exp9_leverage_ceiling_results.json"
log_results(results, outfile, gold=_GOLD_EXP9, meta={
"exp": "exp9",
"question": "Does adaptive_beta DD headroom permit higher leverage ceiling?",
"silver": _SILVER,
"total_elapsed_s": round(time.time() - t_start, 1),
"gold_ref_ok": gold_ok,
"note_mc": (
"Decoupled runs (mc_ref=5.0): MC assesses at original 5x reference — "
"pure leverage cap test. Coupled runs: MC sees actual new cap — tests MC response."
),
})
total = time.time() - t_start
print(f"\nTotal elapsed: {total / 60:.1f} min")
print("Done.")
if __name__ == "__main__":
main()