hjnormey/DOLPHIN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial: import DOLPHIN baseline 2026-04-21 from dolphinng5_predict working tree

Browse Source 
Includes core prod + GREEN/BLUE subsystems:
- prod/ (BLUE harness, configs, scripts, docs)
- nautilus_dolphin/ (GREEN Nautilus-native impl + dvae/ preserved)
- adaptive_exit/ (AEM engine + models/bucket_assignments.pkl)
- Observability/ (EsoF advisor, TUI, dashboards)
- external_factors/ (EsoF producer)
- mc_forewarning_qlabs_fork/ (MC regime/envelope)

Excludes runtime caches, logs, backups, and reproducible artifacts per .gitignore.
This commit is contained in:
hjnormey
2026-04-21 16:58:38 +02:00
commit 01c19662cb
643 changed files with 260241 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

229
nautilus_dolphin/dvae/PROXY_B_RESEARCH_FILING.md Executable file
View File

@@ -0,0 +1,229 @@
# proxy_B — Research Filing
**Date:** 2026-03-14
**Status:** Closed for direct exploitation; open as modulator candidate
**Gold baseline:** ROI=+88.55%, PF=1.215, DD=15.05%, Sharpe=4.38, Trades=2155
---
## 1. Signal Definition
```
proxy_B = instability_50 - v750_lambda_max_velocity
```
- `instability_50`: short-window (50-bar) eigenvalue instability in correlation matrix
- `v750_lambda_max_velocity`: long-window (750-bar) max-eigenvalue velocity
- Intuition: **short-term stress MINUS long-term momentum**. When this is high, the
eigenspace is rapidly destabilising relative to its recent trend.
- Available in 5s scan parquets. Computed in `ShadowLoggingEngine.process_day()`.
---
## 2. Discovery & Measurement
**Experiment:** Precursor sweep (`e2e_precursor_auc.py`, `flint_precursor_sweep.py`)
**Metric:** AUC for predicting eigenspace stress events at K=5 bars forward
| Window / Signal | AUC (K=5) |
|-------------------|-----------|
| proxy_B (inst50 − v750_vel) | **0.715** |
| instability_50 alone | ~0.65 |
| v750_lambda_max_velocity alone | ~0.61 |
| FlintHDVAE latent z (β=0.1) | 0.6918 |
| vel_div (entry signal) | baseline |
proxy_B leads stress events by ~25 seconds (5-bar horizon on 5s data).
It is NOT the entry signal — it measures a different aspect of the eigenspace.
---
## 3. Orthogonality to System Signals
**Test:** Exp 4 — shadow run, 48/2155 trades had valid aligned pb_entry+vd_entry
(entry_bar alignment bug: only ~2% of trades yield correctly-matched bar-level values;
see Section 6 Technical Note).
| Pair | Pearson r | p-value | Spearman rho | Verdict |
|------|-----------|---------|--------------|---------|
| proxy_B_entry ↔ vel_div_entry | −0.031 | 0.837 | −0.463 | **Orthogonal (ns)** |
| proxy_B_entry ↔ pnl_frac | +0.166 | 0.260 | +0.158 | Not predictive of outcome (ns) |
| **proxy_B_entry ↔ MAE** | **+0.420** | **0.003 \*\*** | +0.149 | **Predicts intraday adversity** |
| proxy_B_entry ↔ hold_bars | −0.054 | 0.717 | −0.171 | Orthogonal (ns) |
| proxy_B_max ↔ pnl_frac | +0.066 | 0.655 | −0.379 | ns |
| proxy_B_max ↔ MAE | +0.047 | 0.750 | −0.280 | ns |
Mann-Whitney (worst-10% pnl vs rest): pb_entry worst=-5.40, rest=+0.27, p=0.183 ns
Mann-Whitney (worst-10% MAE vs rest): pb_entry worst=-5.41, rest=+0.27, p=0.115 ns
**Critical finding:**
- proxy_B IS orthogonal to vel_div (the entry signal) — r≈0, ns ✓
- proxy_B does NOT predict final trade PnL — r=+0.17, ns ✓ (confirms prior findings)
- **proxy_B DOES predict intraday adversity (MAE): r=+0.42, p=0.003** ← KEY
**Mechanistic interpretation:** When proxy_B is high at entry, the trade experiences
a worse intraday adverse excursion (deeper MAE). But final PnL is unaffected because
the engine's exit logic (TP/max_hold/direction-confirm) successfully navigates through
the stress period. This is the complete explanation for why:
1. Gating on proxy_B removes trades that are temporarily stressed but then RECOVER → hurts
2. A proxy-coupled stop would cut those recoveries short → reduces DD but also reduces ROI
3. The signal has genuine information content (AUC=0.715, MAE correlation p=0.003)
but the system is ALREADY correctly managing the trades it tags as stressed
---
## 4. Experiments Performed
### Exp 1 — proxy_B Position Sizing (`exp1_proxy_sizing.py`)
Tests `bet_sizer.base_fraction * scale(proxy_B_at_entry)`.
| Config | ROI% | PF | DD% | Sharpe | scale_mean |
|--------|------|----|-----|--------|------------|
| GOLD | 88.55 | 1.215 | 15.05 | 4.38 | — |
| Baseline (no sizing) | 88.55 | 1.2147 | 15.05 | 4.378 | — |
| S1 [0.5x–1.5x] w500 | 91.48 | 1.1782 | 16.93 | 3.528 | 1.004 |
| S2 [0.25x–2.0x] w500 | 105.51 | 1.1537 | 20.30 | 2.956 | **1.133** |
| S3 [0.5x–1.5x] w1000 | 89.49 | 1.1763 | 16.69 | 3.514 | 1.000 |
| S4 [0.5x–1.5x] clip | 87.13 | 1.1628 | 18.03 | 3.184 | 1.019 |
**Finding:** scale_mean > 1.0 in all configs → proxy_B is more often LOW during trading
activity, meaning the engine sizes UP on average. Higher ROI (S2: +17pp) is a leverage
effect, not signal quality — PF drops and Sharpe collapses. **The signal is anti-correlated
with trade quality per unit capital.**
### Exp 2 — proxy_B Shadow Exit (`exp2_proxy_exit.py`)
Post-hoc test: would exiting when proxy_B < threshold have helped?
| Threshold | Trigger rate | AvgDelta% | Early better | Est. ROI |
|-----------|-------------|-----------|--------------|---------|
| p10 | ~60% of trades (at ≥1 bar) | −0.15% | 37% | −0.96pp |
| p25 | ~69% | +0.04% | 43% | +0.26pp |
| p50 | ~85% | +0.02% | 43% | +0.15pp |
**Note:** High trigger rates are mathematically expected — a 120-bar hold has
~100% chance of *any* bar crossing the p50 level. The signal fires constantly
during holds; using it as an exit trigger is noise, not information.
**Verdict:** Holding to natural exit is better. Early exit is weakly beneficial
in only 37-43% of cases.
### Exp 3 — Longer Window Proxies (`exp3_longer_proxies.py`)
All 5 proxy variants × 3 modes (gate/size/exit) × 3 thresholds. AE validation
of top 10 fast-sweep configs.
| Config | AE ROI% | PF | DD% | Note |
|--------|---------|-----|-----|------|
| GOLD | 88.55 | 1.215 | 15.05 | — |
| V50/gate/p50 | **−21.58** | 0.822 | 31.94 | Catastrophic |
| V150/gate/p50 | **−24.34** | 0.909 | 31.97 | Catastrophic |
| B150/gate/p10 | −17.37 | 0.941 | 29.00 | Catastrophic |
| B150/gate/p25 | −1.26 | 0.996 | 28.25 | Marginal hurt |
| Exit modes | 88.55 (=base) | — | — | 0 early exits |
**Why velocity gates are catastrophic:** V50 = instability_50 − v750_velocity and
V150 = instability_150 − v750_velocity. The velocity divergence short-minus-long is
highly *noisy* at short windows. Gating on it suppresses large fractions of trades
(compound-leverage paradox: each suppressed trade costs more than it saves due to
capital compounding).
**Exit mode 0-triggers in AE:** `_try_entry` is the wrong hook for exits. The AE
exit path goes through `exit_manager.evaluate()`. Fast-sweep exit approximations
are valid; AE validation of exit modes requires `exit_manager` override.
### Exp 5 — Two-pass β DVAE (`exp5_dvae_twopass.py`)
Does high-β first pass → low-β second pass improve latent space?
| Variant | AUC | vs baseline | Active dims |
|---------|-----|-------------|-------------|
| A: single-pass β=0.1 | **0.6918** | — | 8/8 |
| B: β=4→β=0.1 | <0.6918 | −0.006 | collapsed |
| C: β=2→β=0.1 | <0.6918 | negative | partial |
| D: dual concat β=4‖β=0.1 | <0.6918 | negative | mixed |
**Root cause:** High-β collapses z_var to 0.008–0.019 (from 0.1+ single-pass) by
epoch 10 via KL domination. The collapsed posterior is a *worse* initialiser than
random. β=12 was not tested (β=6 already gave 0/20 active dims).
### Exp 4 — Coupling Sweep (`exp4_proxy_coupling.py`) — COMPLETE
155 configs tested in 0.11s (retroactive). Shadow run confirmed: ROI=88.55%, Trades=2155.
**DD < 15.05% AND ROI ≥ 84.1% candidates (19 found):**
| Config | ROI% | DD% | ΔROI | ΔDD | Note |
|--------|------|----|------|-----|------|
| B/pb_entry/thr0.35/a1.0 | 86.93 | **14.89** | −1.62 | −0.15 | scale_boost, smean=1.061 |
| **E/stop_0.003** | **89.90** | **14.91** | **+1.36** | **−0.14** | **pure_stop, 18 triggers** |
| B/pb_entry/thr0.35/a0.5 | 87.74 | 14.97 | −0.81 | −0.08 | scale_boost |
| E/stop_0.005 | 89.29 | 14.97 | +0.74 | −0.07 | pure_stop, 11 triggers |
| E/stop_0.015 | 89.27 | 14.97 | +0.72 | −0.07 | pure_stop, 2 triggers |
| F/stop_0.005/gate_p0.5 | 88.68 | 15.03 | +0.14 | −0.01 | gated_stop, 4 triggers |
Best per mode: scale_suppress → DD worsens; hold_limit → DD worsens; rising_exit → DD worsens;
pure_stop → best legitimate DD reducer; gated_stop → marginal (few triggers).
**IMPORTANT CAVEAT:** entry_bar alignment bug caused 2107/2155 pb_entry to be NaN
(entry_bar appears to store global_bar_idx not per-day ri). The proxy-coupling modes
(A, B, F) used median fill-in for 98% of trades → effectively a null test. Only Mode E
(pure_stop) is fully valid because it uses MAE computed from shadow hold prices.
**Valid conclusion from Exp 4:**
- A 0.3% retroactive stop (`E/stop_0.003`) improves BOTH ROI (+1.36pp) and DD (−0.14pp)
- Only 18 trades triggered → the improvement is modest but directionally sound
- The proxy-coupled stop (Mode F) needs proper entry_bar alignment to test meaningfully
- **Next step**: implement stop_pct parameter in exit_manager for real engine test
---
## 5. Core Findings
### 5.1 The Compound-Leverage Paradox
With dynamic leverage, gating ANY subset of trades (even below-average quality ones)
costs ROI because capital that would have compounded is left idle. The break-even
requires gated trades to have strongly negative expected value — but the 50.5% win
rate means most trades are net-positive.
### 5.2 Why proxy_B Gating Specifically Hurts
scale_mean > 1.0 in position sizing tests = proxy_B is LOWER during most trading
time windows than the neutral baseline. The system naturally avoids high-proxy
periods (or avoids entering during them) already. Gating explicitly on high-proxy
removes the REMAINING high-proxy trades, which happen to be positive on average.
### 5.3 The Unresolved Question: MAE vs Final PnL
proxy_B has AUC=0.715 for eigenspace stress prediction. The signal IS predictive of
something real. The hypothesis (untested until Exp 4): **proxy_B predicts intraday
adversity (MAE) but NOT final trade outcome**, because the engine's exit logic
successfully recovers from intraday stress. If confirmed:
- proxy_B fires during the rough patch mid-trade
- The trade then recovers to its natural TP/exit
- Gating removes trades that look scary but ultimately recover
- **A tighter retroactive stop ONLY during high-proxy periods might reduce DD
without proportionally reducing ROI** — if the recovery is systematic
---
## 6. Open Research Directions
| Priority | Direction | Rationale |
|----------|-----------|-----------|
| HIGH | Exp 4 coupling results | Does gated stop reduce DD without ROI cost? |
| MED | Exit hook override | Implement `exit_manager` proxy gate for proper AE test |
| MED | 5s crossover test | Does vel_div crossover on 5s data escape fee pressure? |
| LOW | Longer proxy windows | B300, B500 (instability_300 not in data) |
| LOW | Combined proxy | B50 × B150 product for sharper stress signal |
---
## 7. Files
| File | Description |
|------|-------------|
| `exp1_proxy_sizing.py` | Position scaling by proxy_B |
| `exp2_proxy_exit.py` | Shadow exit analysis (corrected) |
| `exp3_longer_proxies.py` | All 5 proxies × all 3 modes × 3 thresholds |
| `exp4_proxy_coupling.py` | Coupling sweep + orthogonality test |
| `exp5_dvae_twopass.py` | Two-pass β DVAE test |
| `exp1_proxy_sizing_results.json` | Logged results |
| `exp2_proxy_exit_results.json` | Logged results |
| `exp3_fast_sweep_results.json` | Fast numpy sweep |
| `exp3_alpha_engine_results.json` | AE validation |
| `exp4_proxy_coupling_results.json` | Coupling sweep output |
| `exp5_dvae_twopass_results.json` | Two-pass DVAE output |
| `flint_hd_vae.py` | FlintHDVAE implementation |
| `e2e_precursor_auc.py` | AUC measurement infrastructure |

3
nautilus_dolphin/dvae/__init__.py Executable file
View File

@@ -0,0 +1,3 @@
"""DOLPHIN Hierarchical Disentangled VAE — multi-generation corpus training."""
from .hierarchical_dvae import HierarchicalDVAE
from .corpus_builder import DolphinCorpus

171
nautilus_dolphin/dvae/alpha_signal_generator_flint_gate.py Executable file
View File

@@ -0,0 +1,171 @@
"""
FlintGatedEngine — TEST FORK. Does NOT modify any production code.
Adds proxy_B = instability_50 - v750_lambda_max_velocity entry gate.
Gate: SUPPRESS new entry when proxy_B < proxy_b_threshold
ALLOW new entry when proxy_B >= proxy_b_threshold
proxy_B is a 25-second leading indicator of eigenspace stress (AUC=0.715 at K=5).
High proxy_B = stress event incoming = good mean-reversion entry environment.
Low proxy_B = calm eigenspace = vel_div signal is likely false alarm.
Threshold is computed from the rolling window of observed proxy_B values.
"""
import sys, os
_dvae_dir = os.path.dirname(os.path.abspath(__file__))
_nd_root = os.path.dirname(_dvae_dir) # nautilus_dolphin/ outer dir (package root)
_proj_root = os.path.dirname(_nd_root) # project root
sys.path.insert(0, _nd_root)
sys.path.insert(0, _proj_root)
import numpy as np
from typing import Optional, Dict, List, Any
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
class FlintGatedEngine(NDAlphaEngine):
"""
NDAlphaEngine with a proxy_B entry gate.
Additional constructor parameters:
proxy_b_threshold : float
Minimum proxy_B value required to allow a new entry.
Any scan where instability_50 - v750_lambda_max_velocity < threshold
will suppress the NEW ENTRY only (existing positions still managed).
proxy_b_percentile : float or None
If set (e.g. 0.5 = median), threshold is computed adaptively from a
warm-up window rather than using a fixed value. Overrides
proxy_b_threshold when enough warm-up data is collected.
warmup_bars : int
Number of bars to collect before activating the adaptive gate.
"""
def __init__(
self,
*args,
proxy_b_threshold: float = 0.0, # fixed threshold (default: > 0 = gate on positive proxy_B)
proxy_b_percentile: Optional[float] = None, # e.g. 0.5 for median adaptive gate
warmup_bars: int = 500,
**kwargs,
):
super().__init__(*args, **kwargs)
self._proxy_b_threshold = proxy_b_threshold
self._proxy_b_percentile = proxy_b_percentile
self._warmup_bars = warmup_bars
# Runtime state updated from process_day override
self._current_instability_50: float = 0.0
self._current_v750_vel_for_gate: float = 0.0
# Adaptive threshold history
self._proxy_b_history: List[float] = []
self._gate_active_threshold: float = proxy_b_threshold
# Stats
self.gate_suppressed: int = 0
self.gate_allowed: int = 0
def _compute_proxy_b(self) -> float:
return self._current_instability_50 - self._current_v750_vel_for_gate
def _update_gate_threshold(self, proxy_b: float):
"""Update rolling adaptive threshold if percentile mode is on."""
self._proxy_b_history.append(proxy_b)
if len(self._proxy_b_history) > 2000:
self._proxy_b_history = self._proxy_b_history[-1000:]
if (self._proxy_b_percentile is not None and
len(self._proxy_b_history) >= self._warmup_bars):
self._gate_active_threshold = float(
np.percentile(self._proxy_b_history, self._proxy_b_percentile * 100)
)
def process_day(
self,
date_str: str,
df,
asset_columns,
vol_regime_ok=None,
direction=None,
posture: str = 'APEX',
) -> dict:
"""Override: reads instability_50 from each row before calling parent step_bar."""
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
# 1. Setup day (inherited)
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
v50_raw = row.get('v50_lambda_max_velocity')
v750_raw = row.get('v750_lambda_max_velocity')
v50_val = float(v50_raw) if (v50_raw is not None and np.isfinite(float(v50_raw))) else 0.0
v750_val = float(v750_raw) if (v750_raw is not None and np.isfinite(float(v750_raw))) else 0.0
# ── NEW: capture instability_50 for gate ─────────────────
inst_raw = row.get('instability_50')
inst_val = float(inst_raw) if (inst_raw is not None and np.isfinite(float(inst_raw))) else 0.0
self._current_instability_50 = inst_val
self._current_v750_vel_for_gate = v750_val
proxy_b = self._compute_proxy_b()
self._update_gate_threshold(proxy_b)
# ─────────────────────────────────────────────────────────
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(p):
prices[ac] = float(p)
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self.step_bar(
bar_idx=ri,
vel_div=float(vd),
prices=prices,
vol_regime_ok=vrok,
v50_vel=v50_val,
v750_vel=v750_val,
)
bid += 1
return self.end_day()
def _try_entry(
self,
bar_idx: int,
vel_div: float,
prices: Dict[str, float],
price_histories: Optional[Dict[str, List[float]]],
v50_vel: float = 0.0,
v750_vel: float = 0.0,
) -> Optional[Dict]:
"""Override: apply proxy_B gate before allowing new entry."""
proxy_b = self._compute_proxy_b()
# Gate: suppress entry if proxy_B below threshold
if proxy_b < self._gate_active_threshold:
self.gate_suppressed += 1
return None
self.gate_allowed += 1
return super()._try_entry(bar_idx, vel_div, prices, price_histories, v50_vel, v750_vel)
def get_gate_stats(self) -> dict:
total = self.gate_suppressed + self.gate_allowed
return {
'gate_threshold': self._gate_active_threshold,
'gate_suppressed': self.gate_suppressed,
'gate_allowed': self.gate_allowed,
'gate_total_entry_attempts': total,
'suppression_rate': self.gate_suppressed / max(1, total),
}

44
nautilus_dolphin/dvae/analysis_results.json Executable file
View File

@@ -0,0 +1,44 @@
{
"macro_regime_k": 7,
"macro_regime_silhouettes": {
"3": 1.2346,
"4": 1.3428,
"5": 1.1801,
"6": 1.3357,
"7": 1.3577,
"8": 1.3035
},
"z1_leads_z0": {
"1": 0.0007,
"3": -0.0078,
"5": -0.0086,
"10": 0.0004,
"20": -0.0081
},
"z1_peak_lead_lag": 5,
"active_dims": 0,
"var_per_dim": [
0.00025703103098368845,
0.00028948736020975336,
0.00024541800878152527,
0.0002782960301000939,
0.01298944744570043,
0.008084571955901239,
0.004822071392686802,
0.008369066411074426,
0.0019036063690540516,
0.0120343102415987,
0.011469297945944847,
0.01545028485150473,
5.283625351179199e-10,
4.396326750003878e-08,
4.179382781434889e-09,
2.8163319663137794e-08,
2.004156954585684e-08,
7.610757411291218e-08,
4.65066037228969e-09,
6.595101181957703e-09
],
"recon_err_mean": 0.42131611033692357,
"recon_err_p95": 0.6568199701540868
}

44
nautilus_dolphin/dvae/analysis_results_beta05.json Executable file
View File

@@ -0,0 +1,44 @@
{
"macro_regime_k": 6,
"macro_regime_silhouettes": {
"3": 1.0784,
"4": 1.0369,
"5": 1.0727,
"6": 1.1377,
"7": 1.0712,
"8": 1.0434
},
"z1_leads_z0": {
"1": 0.002,
"3": -0.0007,
"5": 0.006,
"10": 0.0091,
"20": 0.0099
},
"z1_peak_lead_lag": 20,
"active_dims": 0,
"var_per_dim": [
0.00012800445580857145,
8.531996355383793e-05,
5.910444519925593e-05,
7.25133118984212e-05,
5.466937414344175e-06,
2.103502361258361e-07,
1.3829218736923569e-06,
7.117502941433482e-08,
2.696514269214737e-07,
5.3351005165789584e-08,
1.5573366927270033e-07,
6.82847625884739e-07,
2.3622843845548987e-08,
1.391215288082256e-07,
3.228828287565509e-08,
8.832248019511851e-08,
4.565268782682383e-08,
1.773469547532686e-07,
1.0306041981046993e-07,
3.908315856902533e-08
],
"recon_err_mean": 0.4274084299928307,
"recon_err_p95": 0.6560542069479147
}

44
nautilus_dolphin/dvae/analysis_results_beta6.json Executable file
View File

@@ -0,0 +1,44 @@
{
"macro_regime_k": 8,
"macro_regime_silhouettes": {
"3": 1.0825,
"4": 1.164,
"5": 1.1201,
"6": 1.045,
"7": 1.249,
"8": 1.2561
},
"z1_leads_z0": {
"1": 0.0049,
"3": -0.0254,
"5": 0.0066,
"10": 0.0009,
"20": -0.0013
},
"z1_peak_lead_lag": 3,
"active_dims": 0,
"var_per_dim": [
2.891610166279152e-05,
3.0922398955150796e-05,
2.6778158479914613e-05,
3.276855140176666e-05,
7.528364813807103e-06,
3.8058184945766717e-07,
8.343217846104335e-06,
8.610409182091088e-07,
5.164212481888014e-08,
4.2038235764690244e-07,
9.007637297207591e-07,
1.3331745890423244e-06,
8.58313445064302e-09,
2.855058658369656e-08,
7.697479886121003e-09,
3.012180655617942e-08,
1.625070986842035e-08,
7.986989673920088e-08,
5.23765968435441e-09,
1.1474841146856828e-08
],
"recon_err_mean": 0.424669903822917,
"recon_err_p95": 0.6535680228789759
}

141
nautilus_dolphin/dvae/convnext_5s_query.py Executable file
View File

@@ -0,0 +1,141 @@
"""
convnext_5s_query.py — inference query against trained convnext_model_5s.json
Reports:
1. Per-channel reconstruction correlation (orig vs recon)
2. z-dim activity and spread
3. Top z-dims correlated with proxy_B (ch7)
Uses vbt_cache/*.parquet (5s scan corpus, C_in=8, no ExF).
"""
import os, sys, json, io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8', errors='replace')
import numpy as np
import glob
import pandas as pd
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
DVAE_DIR = os.path.join(ROOT, 'nautilus_dolphin', 'dvae')
sys.path.insert(0, DVAE_DIR)
MODEL_PATH = os.path.join(DVAE_DIR, 'convnext_model_5s.json')
SCANS_DIR = os.path.join(ROOT, 'vbt_cache')
FEATURE_COLS = [
'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity',
'vel_div', 'instability_50', 'instability_150',
]
CH_NAMES = FEATURE_COLS + ['proxy_B'] # 8 channels
T_WIN = 32
N_PROBES = 200 # more probes — 56 files, sample ~3-4 per file
# ── load model ──────────────────────────────────────────────────────────────
from convnext_dvae import ConvNeXtVAE
with open(MODEL_PATH) as f:
meta = json.load(f)
arch = meta.get('architecture', {})
model = ConvNeXtVAE(
C_in = arch.get('C_in', 8),
T_in = arch.get('T_in', 32),
z_dim = arch.get('z_dim', 32),
base_ch = arch.get('base_ch', 32),
n_blocks = arch.get('n_blocks', 3),
seed = 42,
)
model.load(MODEL_PATH)
norm_mean = np.array(meta['norm_mean']) if 'norm_mean' in meta else None
norm_std = np.array(meta['norm_std']) if 'norm_std' in meta else None
print(f"Model: epoch={meta.get('epoch')} val_loss={meta.get('val_loss', float('nan')):.5f}")
print(f" C_in={arch.get('C_in')} z_dim={arch.get('z_dim')} base_ch={arch.get('base_ch')}\n")
# ── build probe set ──────────────────────────────────────────────────────────
files = sorted(f for f in glob.glob(os.path.join(SCANS_DIR, '*.parquet'))
if 'catalog' not in f)
step = max(1, len(files) // (N_PROBES // 4)) # ~4 probes per file
probes_raw, proxy_B_vals = [], []
rng = np.random.default_rng(42)
for f in files[::step]:
try:
df = pd.read_parquet(f, columns=FEATURE_COLS).dropna()
if len(df) < T_WIN + 4: continue
# sample multiple starting positions per file
positions = rng.integers(0, len(df) - T_WIN, size=4)
for pos in positions:
arr = df[FEATURE_COLS].values[pos:pos+T_WIN].astype(np.float64) # (T, 7)
proxy_B = (arr[:, 5] - arr[:, 3]).reshape(-1, 1) # instability_50 - v750
arr8 = np.concatenate([arr, proxy_B], axis=1) # (T, 8)
if not np.isfinite(arr8).all(): continue
probes_raw.append(arr8.T) # (8, T)
proxy_B_vals.append(float(proxy_B.mean()))
if len(probes_raw) >= N_PROBES: break
except Exception:
pass
if len(probes_raw) >= N_PROBES: break
probes_raw = np.stack(probes_raw) # (N, 8, T)
proxy_B_arr = np.array(proxy_B_vals) # (N,)
print(f"Probe set: {probes_raw.shape} ({len(probes_raw)} windows × {probes_raw.shape[1]} ch × {T_WIN} steps)\n")
# ── normalise ────────────────────────────────────────────────────────────────
probes = probes_raw.copy()
if norm_mean is not None:
probes = (probes - norm_mean[None, :, None]) / norm_std[None, :, None]
np.clip(probes, -6.0, 6.0, out=probes)
# ── encode / decode ──────────────────────────────────────────────────────────
z_mu, z_logvar = model.encode(probes)
x_recon = model.decode(z_mu)
# ── 1. Per-channel reconstruction correlation ────────────────────────────────
print("── Per-channel reconstruction r (orig vs recon) ──────────────────")
for c, name in enumerate(CH_NAMES):
rs = []
for b in range(len(probes)):
o, r = probes[b, c], x_recon[b, c]
if o.std() > 1e-6 and r.std() > 1e-6:
rv = float(np.corrcoef(o, r)[0, 1])
if np.isfinite(rv): rs.append(rv)
mean_r = np.mean(rs) if rs else float('nan')
bar = '█' * int(max(0, mean_r) * 20)
print(f" ch{c:2d} {name:<30s} r={mean_r:+.3f} {bar}")
# ── 2. z-dim activity ────────────────────────────────────────────────────────
z_std_per_dim = z_mu.std(0) # (D,)
z_active = int((z_std_per_dim > 0.01).sum())
z_post_std = float(np.exp(0.5 * z_logvar).mean())
print(f"\n── Latent space ──────────────────────────────────────────────────")
print(f" z_active_dims : {z_active} / {z_mu.shape[1]}")
print(f" z_post_std : {z_post_std:.4f} (>1 = posterior wider than prior)")
z_stds_sorted = sorted(enumerate(z_std_per_dim), key=lambda x: -x[1])
print(f" Top z-dim stds: " + " ".join(f"z[{i}]={s:.3f}" for i, s in z_stds_sorted[:8]))
# ── 3. z-dim × proxy_B correlation ──────────────────────────────────────────
print(f"\n── z-dim correlation with proxy_B (all active dims) ─────────────")
corrs = []
for d in range(z_mu.shape[1]):
if z_std_per_dim[d] > 0.01:
r = float(np.corrcoef(z_mu[:, d], proxy_B_arr)[0, 1])
if np.isfinite(r): corrs.append((abs(r), r, d))
corrs.sort(reverse=True)
print(f" (proxy_B mean={proxy_B_arr.mean():+.4f} std={proxy_B_arr.std():.4f})")
for _, r, d in corrs[:15]:
bar = '█' * int(abs(r) * 30)
print(f" z[{d:2d}] r={r:+.4f} {bar}")
# ── 4. z-dim statistics ──────────────────────────────────────────────────────
print(f"\n── z-dim statistics (z_mu) ──────────────────────────────────────")
print(f" {'dim':>4} {'mean':>8} {'std':>8} {'min':>8} {'max':>8} {'r_proxyB':>10}")
for i, s in z_stds_sorted[:16]:
r_pb = float(np.corrcoef(z_mu[:, i], proxy_B_arr)[0, 1]) if s > 0.01 else float('nan')
print(f" z[{i:2d}] {z_mu[:, i].mean():>+8.4f} {s:>8.4f} "
f"{z_mu[:, i].min():>+8.4f} {z_mu[:, i].max():>+8.4f} {r_pb:>+10.4f}")
print(f"\nDone.")

172
nautilus_dolphin/dvae/convnext_5s_sensor.py Executable file
View File

@@ -0,0 +1,172 @@
"""
convnext_5s_sensor.py — Inference wrapper for the 5s ConvNeXt-1D β-TCVAE.
Usage
-----
sensor = ConvNext5sSensor(model_path)
z_mu, z_post_std = sensor.encode_raw(arr)
# arr: (C_IN, T_WIN) float64
# z_mu: (z_dim,) float64 — latent mean
# z_post_std: float — mean posterior std (>1 = wide/uncertain)
z_mu, z_post_std = sensor.encode_scan_window(arr2d)
# arr2d: (T_WIN, C_IN) or (T_WIN, 7) — from scan parquet rows
# If 7 columns, proxy_B is appended as ch7.
Key differences from the 1m sensor (convnext_sensor.py):
- Model path: convnext_model_5s.json
- C_IN = 8 (7 FEATURE + proxy_B — NO ExF channels)
- No dvol_btc, fng, funding_btc channels
- FEATURE_COLS are the same 7 features as the 1m sensor
- proxy_B = instability_50 - v750_lambda_max_velocity (ch7, same formula as 1m)
Architecture: ConvNeXtVAE C_in=8 T_in=32 z_dim=32 base_ch=32 n_blocks=3
Input channels:
ch0 v50_lambda_max_velocity
ch1 v150_lambda_max_velocity
ch2 v300_lambda_max_velocity
ch3 v750_lambda_max_velocity
ch4 vel_div
ch5 instability_50
ch6 instability_150
ch7 proxy_B (= instability_50 - v750_lambda_max_velocity)
"""
import os
import sys
import json
import numpy as np
_DVAE_DIR = os.path.dirname(os.path.abspath(__file__))
if _DVAE_DIR not in sys.path:
sys.path.insert(0, _DVAE_DIR)
from convnext_dvae import ConvNeXtVAE
FEATURE_COLS = [
'v50_lambda_max_velocity',
'v150_lambda_max_velocity',
'v300_lambda_max_velocity',
'v750_lambda_max_velocity',
'vel_div',
'instability_50',
'instability_150',
]
T_WIN = 32
C_IN = 8 # 7 FEATURE + proxy_B (no ExF)
class ConvNext5sSensor:
"""
Stateless inference wrapper for the 5s ConvNeXt model.
No ExF channels — 8-channel input only.
Thread-safe (model weights are read-only numpy).
"""
def __init__(self, model_path: str):
with open(model_path) as f:
meta = json.load(f)
arch = meta.get('architecture', {})
self.model = ConvNeXtVAE(
C_in = arch.get('C_in', C_IN),
T_in = arch.get('T_in', T_WIN),
z_dim = arch.get('z_dim', 32),
base_ch = arch.get('base_ch', 32),
n_blocks = arch.get('n_blocks', 3),
seed = 42,
)
self.model.load(model_path)
self.norm_mean = np.array(meta['norm_mean'], dtype=np.float64) if 'norm_mean' in meta else None
self.norm_std = np.array(meta['norm_std'], dtype=np.float64) if 'norm_std' in meta else None
self.epoch = meta.get('epoch', '?')
self.val_loss = meta.get('val_loss', float('nan'))
self.z_dim = arch.get('z_dim', 32)
# ── low-level: encode a (C_IN, T_WIN) array ──────────────────────────────
def encode_raw(self, arr: np.ndarray):
"""
arr: (C_IN, T_WIN) float64, already in raw (un-normalised) units.
Returns z_mu (z_dim,), z_post_std float.
"""
x = arr[np.newaxis].astype(np.float64) # (1, C, T)
if self.norm_mean is not None:
x = (x - self.norm_mean[None, :, None]) / self.norm_std[None, :, None]
np.clip(x, -6.0, 6.0, out=x)
z_mu, z_logvar = self.model.encode(x) # (1, D)
z_post_std = float(np.exp(0.5 * z_logvar).mean())
return z_mu[0], z_post_std
# ── high-level: encode from a 2D scan array ───────────────────────────────
def encode_scan_window(self, arr2d: np.ndarray):
"""
arr2d: (T_WIN, C_IN) or (T_WIN, 7) — rows from scan parquet.
If arr2d has 7 columns, proxy_B (instability_50 - v750_lambda_max_velocity)
is appended as ch7 before encoding.
Returns
-------
z_mu : (z_dim,) float64
z_post_std : float (>1 suggests OOD regime)
"""
arr2d = np.asarray(arr2d, dtype=np.float64)
T_actual, n_cols = arr2d.shape
if n_cols == 7:
# Append proxy_B = instability_50 (col5) - v750_lambda_max_velocity (col3)
proxy_b = arr2d[:, 5] - arr2d[:, 3]
arr2d = np.concatenate([arr2d, proxy_b[:, np.newaxis]], axis=1) # (T, 8)
# Pad / trim to T_WIN rows (zero-pad at the start if shorter)
if T_actual < T_WIN:
pad = np.zeros((T_WIN - T_actual, C_IN), dtype=np.float64)
arr2d = np.concatenate([pad, arr2d], axis=0)
else:
arr2d = arr2d[-T_WIN:] # keep the most recent T_WIN rows
return self.encode_raw(arr2d.T) # (C_IN, T_WIN)
# ── find proxy_B dimension via correlation probe ──────────────────────────
def find_proxy_b_dim(self, probe_windows: np.ndarray):
"""
Given probe_windows of shape (N, C_IN, T_WIN), find the z-dim most
correlated with the mean proxy_B value (ch7 mean) across windows.
Parameters
----------
probe_windows : (N, C_IN, T_WIN) float64
Returns
-------
dim_idx : int — z-dim index with highest |r|
corr : float — Pearson r with proxy_B mean
"""
N = len(probe_windows)
if N == 0:
return 0, 0.0
proxy_b_means = probe_windows[:, 7, :].mean(axis=1) # (N,) — mean of ch7 per window
z_mus = []
for i in range(N):
z_mu, _ = self.encode_raw(probe_windows[i])
z_mus.append(z_mu)
z_mus = np.stack(z_mus, axis=0) # (N, z_dim)
# Pearson r between each z-dim and proxy_B mean
pb_centered = proxy_b_means - proxy_b_means.mean()
pb_std = pb_centered.std() + 1e-12
best_dim = 0
best_corr = 0.0
for d in range(z_mus.shape[1]):
zd = z_mus[:, d]
zd_c = zd - zd.mean()
zd_std = zd_c.std() + 1e-12
r = float((pb_centered * zd_c).mean() / (pb_std * zd_std))
if abs(r) > abs(best_corr):
best_corr = r
best_dim = d
return best_dim, best_corr

1048
nautilus_dolphin/dvae/convnext_dvae.py Executable file
View File

File diff suppressed because it is too large Load Diff

1
nautilus_dolphin/dvae/convnext_model.json Executable file
View File

File diff suppressed because one or more lines are too long

1
nautilus_dolphin/dvae/convnext_model_1m_bob.json Executable file
View File

File diff suppressed because one or more lines are too long

1
nautilus_dolphin/dvae/convnext_model_5s.json Executable file
View File

File diff suppressed because one or more lines are too long

1
nautilus_dolphin/dvae/convnext_model_ep17_bak.json Executable file
View File

File diff suppressed because one or more lines are too long

1
nautilus_dolphin/dvae/convnext_model_ep17_prod_bak.json Executable file
View File

File diff suppressed because one or more lines are too long

1
nautilus_dolphin/dvae/convnext_model_v2.json Executable file
View File

File diff suppressed because one or more lines are too long

1
nautilus_dolphin/dvae/convnext_model_v2_ep13_snap.json Executable file
View File

File diff suppressed because one or more lines are too long

1
nautilus_dolphin/dvae/convnext_model_v3.json Executable file
View File

File diff suppressed because one or more lines are too long

145
nautilus_dolphin/dvae/convnext_query.py Executable file
View File

@@ -0,0 +1,145 @@
"""
convnext_query.py — inference query against trained convnext_model.json
Reports:
1. Per-channel reconstruction correlation (orig vs recon)
2. z-dim activity and spread
3. Top z-dims correlated with proxy_B (ch7)
"""
import os, sys, json
import numpy as np
import glob
import pandas as pd
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
DVAE_DIR = os.path.join(ROOT, 'nautilus_dolphin', 'dvae')
sys.path.insert(0, DVAE_DIR)
MODEL_PATH = os.path.join(DVAE_DIR, 'convnext_model.json')
KLINES_DIR = os.path.join(ROOT, 'vbt_cache_klines')
EIGENVALUES_PATH = r"C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues"
EXF_NPZ_NAME = "scan_000001__Indicators.npz"
FEATURE_COLS = [
'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity',
'vel_div', 'instability_50', 'instability_150',
]
EXF_COLS = ['dvol_btc', 'fng', 'funding_btc']
CH_NAMES = FEATURE_COLS + ['proxy_B'] + EXF_COLS # 11 channels
T_WIN = 32
N_PROBES = 100
# ── load model ──────────────────────────────────────────────────────────────
from convnext_dvae import ConvNeXtVAE
with open(MODEL_PATH) as f:
meta = json.load(f)
arch = meta.get('architecture', {})
model = ConvNeXtVAE(C_in=arch.get('C_in', 11), T_in=arch.get('T_in', 32),
z_dim=arch.get('z_dim', 32), base_ch=arch.get('base_ch', 32),
n_blocks=3, seed=42)
model.load(MODEL_PATH)
norm_mean = np.array(meta['norm_mean']) if 'norm_mean' in meta else None
norm_std = np.array(meta['norm_std']) if 'norm_std' in meta else None
print(f"Model: epoch={meta.get('epoch')} val_loss={meta.get('val_loss', float('nan')):.4f}")
print(f" C_in={arch.get('C_in')} z_dim={arch.get('z_dim')} base_ch={arch.get('base_ch')}\n")
# ── build probe set ──────────────────────────────────────────────────────────
_exf_idx = None
def get_exf_indices():
global _exf_idx
if _exf_idx is not None: return _exf_idx
for ds in sorted(os.listdir(EIGENVALUES_PATH)):
p = os.path.join(EIGENVALUES_PATH, ds, EXF_NPZ_NAME)
if os.path.exists(p):
try:
d = np.load(p, allow_pickle=True)
_exf_idx = {n: i for i, n in enumerate(d['api_names'])}
return _exf_idx
except Exception: continue
return {}
files = sorted(glob.glob(os.path.join(KLINES_DIR, '*.parquet')))
step = max(1, len(files) // N_PROBES)
idx_map = get_exf_indices()
probes_raw, proxy_B_vals = [], []
for f in files[::step]:
try:
df = pd.read_parquet(f, columns=FEATURE_COLS).dropna()
if len(df) < T_WIN + 10: continue
pos = len(df) // 2
arr = df[FEATURE_COLS].values[pos:pos+T_WIN].astype(np.float64)
proxy_B = (arr[:, 5] - arr[:, 3]).reshape(-1, 1)
arr = np.concatenate([arr, proxy_B], axis=1) # (T, 8)
exf = np.zeros((T_WIN, len(EXF_COLS)), dtype=np.float64)
date_str = os.path.basename(f).replace('.parquet', '')
npz_p = os.path.join(EIGENVALUES_PATH, date_str, EXF_NPZ_NAME)
if os.path.exists(npz_p) and idx_map:
d = np.load(npz_p, allow_pickle=True)
for ci, col in enumerate(EXF_COLS):
fi = idx_map.get(col, -1)
if fi >= 0 and bool(d['api_success'][fi]):
exf[:, ci] = float(d['api_indicators'][fi])
arr = np.concatenate([arr, exf], axis=1).T # (11, T)
probes_raw.append(arr)
proxy_B_vals.append(float(proxy_B.mean()))
except Exception:
pass
if len(probes_raw) >= N_PROBES: break
probes_raw = np.stack(probes_raw) # (N, 11, T)
proxy_B_arr = np.array(proxy_B_vals) # (N,)
print(f"Probe set: {probes_raw.shape} ({len(probes_raw)} windows × {probes_raw.shape[1]} ch × {T_WIN} steps)\n")
# ── normalise ────────────────────────────────────────────────────────────────
probes = probes_raw.copy()
if norm_mean is not None:
probes = (probes - norm_mean[None, :, None]) / norm_std[None, :, None]
np.clip(probes, -6.0, 6.0, out=probes)
# ── encode / decode ──────────────────────────────────────────────────────────
z_mu, z_logvar = model.encode(probes)
x_recon = model.decode(z_mu)
# ── 1. Per-channel reconstruction correlation ────────────────────────────────
print("── Per-channel reconstruction r (orig vs recon) ──────────────────")
for c, name in enumerate(CH_NAMES):
rs = []
for b in range(len(probes)):
o, r = probes[b, c], x_recon[b, c]
if o.std() > 1e-6 and r.std() > 1e-6:
rv = float(np.corrcoef(o, r)[0, 1])
if np.isfinite(rv): rs.append(rv)
mean_r = np.mean(rs) if rs else float('nan')
bar = '█' * int(max(0, mean_r) * 20)
print(f" ch{c:2d} {name:<30s} r={mean_r:+.3f} {bar}")
# ── 2. z-dim activity ────────────────────────────────────────────────────────
z_std_per_dim = z_mu.std(0) # (D,)
z_active = int((z_std_per_dim > 0.01).sum())
z_post_std = float(np.exp(0.5 * z_logvar).mean())
print(f"\n── Latent space ──────────────────────────────────────────────────")
print(f" z_active_dims : {z_active} / {z_mu.shape[1]}")
print(f" z_post_std : {z_post_std:.4f} (>1 = posterior wider than prior)")
# ── 3. z-dim × proxy_B correlation ──────────────────────────────────────────
print(f"\n── z-dim correlation with proxy_B (top 10) ──────────────────────")
corrs = []
for d in range(z_mu.shape[1]):
if z_std_per_dim[d] > 0.01:
r = float(np.corrcoef(z_mu[:, d], proxy_B_arr)[0, 1])
if np.isfinite(r): corrs.append((abs(r), r, d))
corrs.sort(reverse=True)
for _, r, d in corrs[:10]:
bar = '█' * int(abs(r) * 20)
print(f" z[{d:2d}] r={r:+.3f} {bar}")
print(f"\nDone.")

140
nautilus_dolphin/dvae/convnext_sensor.py Executable file
View File

@@ -0,0 +1,140 @@
"""
convnext_sensor.py — Inference wrapper for the trained ConvNeXt-1D β-TCVAE.
Usage
-----
sensor = ConvNextSensor(model_path)
z_mu, z_post_std = sensor.encode_window(df_1m, end_row)
# z_mu: (32,) float64 — latent mean for the 32-bar window ending at end_row
# z_post_std: float — mean posterior std (OOD indicator, >1 = wide/uncertain)
Key z-dim assignments (from convnext_query.py, ep=17 checkpoint):
z[10] r=+0.973 proxy_B (instability_50 - v750_velocity)
z[30] r=-0.968 proxy_B (anti-correlated)
z[24] r=+0.942 proxy_B
...10+ dims encoding proxy_B trajectory at >0.86
Architecture: ConvNeXtVAE C_in=11 T_in=32 z_dim=32 base_ch=32 n_blocks=3
Input channels:
ch0-3 v50/v150/v300/v750 lambda_max_velocity
ch4 vel_div
ch5 instability_50
ch6 instability_150
ch7 proxy_B (= instability_50 - v750_lambda_max_velocity)
ch8 dvol_btc (ExF, broadcast constant)
ch9 fng (ExF, broadcast constant)
ch10 funding_btc (ExF, broadcast constant)
"""
import os
import sys
import json
import numpy as np
_DVAE_DIR = os.path.dirname(os.path.abspath(__file__))
if _DVAE_DIR not in sys.path:
sys.path.insert(0, _DVAE_DIR)
from convnext_dvae import ConvNeXtVAE
FEATURE_COLS = [
'v50_lambda_max_velocity',
'v150_lambda_max_velocity',
'v300_lambda_max_velocity',
'v750_lambda_max_velocity',
'vel_div',
'instability_50',
'instability_150',
]
EXF_COLS = ['dvol_btc', 'fng', 'funding_btc']
T_WIN = 32
N_CH = 11 # 7 FEATURE + proxy_B + 3 ExF
# z-dim index of the primary proxy_B encoding (r=+0.973)
PROXY_B_DIM = 10
class ConvNextSensor:
"""
Stateless inference wrapper. Thread-safe (model weights are read-only numpy).
"""
def __init__(self, model_path: str):
with open(model_path) as f:
meta = json.load(f)
arch = meta.get('architecture', {})
self.model = ConvNeXtVAE(
C_in = arch.get('C_in', N_CH),
T_in = arch.get('T_in', T_WIN),
z_dim = arch.get('z_dim', 32),
base_ch = arch.get('base_ch', 32),
n_blocks = arch.get('n_blocks', 3),
seed = 42,
)
self.model.load(model_path)
self.norm_mean = np.array(meta['norm_mean'], dtype=np.float64) if 'norm_mean' in meta else None
self.norm_std = np.array(meta['norm_std'], dtype=np.float64) if 'norm_std' in meta else None
self.epoch = meta.get('epoch', '?')
self.val_loss = meta.get('val_loss', float('nan'))
self.z_dim = arch.get('z_dim', 32)
# ── low-level: encode a (1, N_CH, T_WIN) array ──────────────────────────
def encode_raw(self, arr: np.ndarray):
"""
arr: (N_CH, T_WIN) float64, already in raw (un-normalised) units.
Returns z_mu (z_dim,), z_post_std float.
"""
x = arr[np.newaxis].astype(np.float64) # (1, C, T)
if self.norm_mean is not None:
x = (x - self.norm_mean[None, :, None]) / self.norm_std[None, :, None]
np.clip(x, -6.0, 6.0, out=x)
z_mu, z_logvar = self.model.encode(x) # (1, D)
z_post_std = float(np.exp(0.5 * z_logvar).mean())
return z_mu[0], z_post_std
# ── high-level: encode from a 1m DataFrame row ──────────────────────────
def encode_window(self, df_1m, end_row: int,
exf_dvol: float = 0., exf_fng: float = 0.,
exf_funding: float = 0.):
"""
Build a (N_CH, T_WIN) window ending at end_row (inclusive) from df_1m.
Missing columns are treated as zero.
Parameters
----------
df_1m : DataFrame with FEATURE_COLS as columns
end_row : integer row index (loc-style), window = [end_row-T_WIN+1 : end_row+1]
exf_* : ExF scalars broadcast across the window (set to 0 if unavailable)
Returns
-------
z_mu : (z_dim,) float64
z_post_std : float (>1 suggests OOD regime)
"""
start = max(0, end_row - T_WIN + 1)
rows = df_1m.iloc[start : end_row + 1]
T_actual = len(rows)
arr = np.zeros((T_WIN, N_CH - 3), dtype=np.float64) # (T_WIN, 8)
for i, col in enumerate(FEATURE_COLS):
if col in rows.columns:
vals = rows[col].values.astype(np.float64)
arr[T_WIN - T_actual:, i] = vals
# proxy_B = instability_50 - v750_lambda_max_velocity (ch7)
arr[:, 7] = arr[:, 5] - arr[:, 3]
# ExF channels broadcast as scalar across T_WIN
exf = np.array([exf_dvol, exf_fng, exf_funding], dtype=np.float64)
full = np.concatenate([arr, np.tile(exf, (T_WIN, 1))], axis=1) # (T_WIN, 11)
return self.encode_raw(full.T) # (N_CH, T_WIN)
# ── convenience scalar: primary proxy_B z-dim ────────────────────────────
def z_proxy_b(self, df_1m, end_row: int, **exf_kwargs) -> float:
"""Return scalar z[PROXY_B_DIM] for the window ending at end_row."""
z_mu, _ = self.encode_window(df_1m, end_row, **exf_kwargs)
return float(z_mu[PROXY_B_DIM])

548
nautilus_dolphin/dvae/corpus_builder.py Executable file
View File

@@ -0,0 +1,548 @@
"""
DOLPHIN Multi-Generation Corpus Builder (Memory-Efficient, 5-Tier)
====================================================================
Loads ALL available Dolphin data into a unified feature matrix.
TIERS (distinct, layered, can be frozen/trained independently):
Tier 0 (8 dims) ALWAYS — breadth (bull/bear), cyclic time, has_eigen flag
Tier 1 (20 dims) NG3+ — eigenvalue structure: 4 windows × 5 features
Tier 2 (50 dims) NG3+ — per-asset volatility cross-section (50 symbols)
Tier 3 (25 dims) NG3+ — ExF macro indicators (dvol, fng, funding, OI, etc.)
Tier 4 (8 dims) ALWAYS — EsoF: lunar, fibonacci, session, cycle (computed)
Total: 111 dims. Missing tiers are zero-filled; mask tracks availability.
Memory strategy:
- NEVER accumulate raw JSON dicts — parse → extract → discard immediately
- Write to memory-mapped numpy array (np.memmap) in fixed-size chunks
- Per-date ExF NPZ loaded once and reused for all scans of that day
- Pre-allocate output array based on estimated sample count
"""
import json
import re
import math
import numpy as np
from pathlib import Path
from datetime import datetime
from typing import Optional, Iterator, Tuple
# ── Paths ──────────────────────────────────────────────────────────────────
BASE = Path(r"C:\Users\Lenovo\Documents")
NG1_DIR = BASE / "- Dolphin NG"
NG2_DIR = BASE / "- Dolphin NG2"
NG4_DIR = BASE / "- DOLPHIN NG4" / "- Results"
NG5_DIR = BASE / "- Dolphin NG5"
NG3_EIGEN = BASE / "- Dolphin NG HD (NG3)" / "correlation_arb512" / "eigenvalues"
HERE = Path(__file__).parent
# ── Tier dimensions ────────────────────────────────────────────────────────
T0 = 8 # breadth + time + flag
T1 = 20 # eigenvalues (4 windows × 5)
T2 = 50 # per-asset volatility
T3 = 25 # ExF macro indicators
T4 = 8 # EsoF esoteric
DIMS = [T0, T1, T2, T3, T4]
TOTAL = sum(DIMS) # 111
OFF = [0, T0, T0+T1, T0+T1+T2, T0+T1+T2+T3] # slice offsets
WINDOWS = [50, 150, 300, 750]
EPS = 1e-8
# ── ExF indicator selection (from the 85-field NPZ, keep reliable ones) ───
EXF_FIELDS = [
'dvol_btc', 'dvol_eth', # implied vol
'fng', 'fng_prev', # fear & greed
'btc_dom', 'eth_dom', # dominance
'chg24_btc', 'chg24_eth', # 24h returns
'dispersion', 'correlation', # cross-market
'imbal_btc', 'imbal_eth', # OB imbalance
'funding_btc', 'funding_eth', # perp funding
'mvrv', # on-chain
'tvl', # DeFi
'pcr_vol', 'pcr_oi', # options
'basis', 'liq_proxy', # futures
'spread', 'vol24', # microstructure
'hashrate', # mining
'btc_price', # price level
'fng_vol', # FnG volatility component
]
assert len(EXF_FIELDS) == T3, f"EXF_FIELDS len={len(EXF_FIELDS)} != T3={T3}"
# ExF normalisation constants (robust: divide by median absolute scale)
EXF_SCALE = {
'dvol_btc': 50.0, 'dvol_eth': 50.0,
'fng': 50.0, 'fng_prev': 50.0,
'btc_dom': 50.0, 'eth_dom': 10.0,
'chg24_btc': 5.0, 'chg24_eth': 5.0,
'dispersion': 5.0, 'correlation': 1.0,
'imbal_btc': 1.0, 'imbal_eth': 1.0,
'funding_btc': 0.001, 'funding_eth': 0.001,
'mvrv': 3.0,
'tvl': 1e11,
'pcr_vol': 1.0, 'pcr_oi': 1.0,
'basis': 0.1, 'liq_proxy': 1.0,
'spread': 0.01, 'vol24': 1e10,
'hashrate': 1e9,
'btc_price': 1e5,
}
# ── Time helpers ───────────────────────────────────────────────────────────
def _parse_ts(s: str) -> Optional[datetime]:
for fmt in ("%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%dT%H:%M:%S",
"%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S"):
try:
return datetime.strptime(str(s)[:26], fmt)
except ValueError:
continue
return None
def _tier0(bull_pct: float, bear_pct: float, ts: datetime, has_eigen: bool) -> np.ndarray:
bull = np.clip(bull_pct / 100.0, 0, 1)
bear = np.clip(bear_pct / 100.0, 0, 1)
side = max(0.0, 1.0 - bull - bear)
h = ts.hour + ts.minute / 60.0
d = ts.weekday()
return np.array([
bull, bear, side,
math.sin(2 * math.pi * h / 24),
math.cos(2 * math.pi * h / 24),
math.sin(2 * math.pi * d / 7),
math.cos(2 * math.pi * d / 7),
1.0 if has_eigen else 0.0,
], dtype=np.float32)
def _tier1(windows: dict) -> Tuple[np.ndarray, bool]:
vec = np.zeros(T1, dtype=np.float32)
if not windows:
return vec, False
valid = False
for i, w in enumerate(WINDOWS):
wdata = windows.get(w) or windows.get(str(w)) or {}
td = wdata.get('tracking_data') or wdata
rs = wdata.get('regime_signals') or {}
lmax = float(td.get('lambda_max', 0) or 0)
if lmax > 0:
valid = True
vel = float(td.get('lambda_max_velocity', 0) or 0)
gap = float(td.get('eigenvalue_gap', 0) or 0)
inst = float(rs.get('instability_score', 0) or 0)
rtp = float(rs.get('regime_transition_probability', 0) or 0)
log_lmax = math.log(max(lmax, 1e-6))
vel_norm = np.clip(vel / (abs(lmax) + EPS), -5, 5)
gap_ratio = np.clip(gap / (lmax + EPS), 0, 10)
base = i * 5
vec[base] = np.float32(np.clip(log_lmax / 10.0, -3, 3))
vec[base+1] = np.float32(vel_norm)
vec[base+2] = np.float32(gap_ratio)
vec[base+3] = np.float32(np.clip(inst, 0, 1))
vec[base+4] = np.float32(np.clip(rtp, 0, 1))
return vec, valid
def _tier2(pricing: dict) -> Tuple[np.ndarray, bool]:
vec = np.zeros(T2, dtype=np.float32)
vol = (pricing or {}).get('volatility') or {}
if not vol:
return vec, False
vals = np.array(list(vol.values())[:T2], dtype=np.float32)
if len(vals) == 0:
return vec, False
mu, sd = vals.mean(), vals.std() + EPS
vals = np.clip((vals - mu) / sd, -5, 5)
n = min(T2, len(vals))
vec[:n] = vals[:n]
return vec, True
def _tier3(exf_lookup: Optional[dict]) -> np.ndarray:
"""Extract ExF Tier-3 vector from per-date indicator dict."""
vec = np.zeros(T3, dtype=np.float32)
if not exf_lookup:
return vec
for i, field in enumerate(EXF_FIELDS):
v = exf_lookup.get(field, 0.0) or 0.0
scale = EXF_SCALE.get(field, 1.0)
vec[i] = np.float32(np.clip(float(v) / scale, -10, 10))
return vec
def _tier4(ts) -> np.ndarray:
"""
EsoF Tier-4: 8 computed esoteric features from timestamp alone.
Accepts Unix float timestamp OR datetime object.
No external data needed — all derived from ts.
"""
import calendar as cal_mod
# Normalise to both float-seconds and datetime
if isinstance(ts, (int, float)):
ts_f = float(ts)
dt = datetime.utcfromtimestamp(ts_f)
else:
dt = ts
ts_f = dt.timestamp()
# Moon illumination approx (simplified Meeus formula)
# JD of Unix epoch (1970-01-01 00:00 UTC) = 2440587.5
jd = 2440587.5 + ts_f / 86400.0
D = jd - 2451545.0 # days since J2000.0
# Moon phase angle (degrees)
moon_age = (D % 29.53058867) / 29.53058867 # 0=new, 0.5=full
moon_illum = 0.5 * (1 - math.cos(2 * math.pi * moon_age))
# Mercury retrograde cycles (~3x/year, each ~21 days) — simplified
merc_cycle = (D % 115.88) / 115.88
merc_retro = 1.0 if 0.82 < merc_cycle < 1.0 else 0.0 # last ~18/115 of cycle
# Fibonacci time: minutes into day
mins = dt.hour * 60 + dt.minute
fib_mins = [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1440]
dists = [abs(mins - f) for f in fib_mins]
fib_proximity = 1.0 / (1.0 + min(dists) / 60.0) # 1=at fib, 0=far
# Session (0=Asia, 0.33=London, 0.67=NY, 1=Close)
h = dt.hour + dt.minute / 60.0
if 0 <= h < 7: session = 0.0
elif 7 <= h < 13: session = 0.33
elif 13 <= h < 21: session = 0.67
else: session = 1.0
# Market cycle position (annual)
doy = dt.timetuple().tm_yday
days_in_year = 366 if cal_mod.isleap(dt.year) else 365
cycle_pos = doy / days_in_year
# Day of week sin/cos (weekly cycle)
dow_sin = math.sin(2 * math.pi * dt.weekday() / 7)
dow_cos = math.cos(2 * math.pi * dt.weekday() / 7)
return np.array([
moon_illum, # lunar phase
moon_age, # 0=new, 0.5=full, 1=new
merc_retro, # binary: Mercury Rx
fib_proximity, # nearness to Fibonacci time
session, # liquidity session
cycle_pos, # annual cycle position
dow_sin, dow_cos, # weekly cycle
], dtype=np.float32)
# ── ExF NPZ loader (per-date, cached) ─────────────────────────────────────
class ExFCache:
"""Loads ExF NPZ once per date directory, provides field lookup."""
def __init__(self, eigen_base: Path):
self._base = eigen_base
self._current_date: Optional[str] = None
self._lookup: Optional[dict] = None
def get(self, date_str: str) -> Optional[dict]:
if date_str == self._current_date:
return self._lookup
self._current_date = date_str
self._lookup = None
date_dir = self._base / date_str
# Find ANY __Indicators.npz in this dir
npz_files = list(date_dir.glob('*__Indicators.npz'))
if not npz_files:
return None
try:
d = np.load(npz_files[0], allow_pickle=True)
names = list(d['api_names'])
vals = d['api_indicators']
ok = d['api_success']
self._lookup = {n: float(v) for n, v, s in zip(names, vals, ok) if s and float(v) != 0}
except Exception:
self._lookup = None
return self._lookup
# ── Streaming generators (memory-efficient) ───────────────────────────────
def _stream_ng1_ng2() -> Iterator[np.ndarray]:
import os
for ng_dir in [NG1_DIR, NG2_DIR]:
if not ng_dir.exists():
continue
# Use os.scandir (non-sorted) — much faster than sorted(rglob) on 300K+ files
# NG1/NG2 files are all at the top level
for entry in os.scandir(str(ng_dir)):
f = Path(entry.path)
if not (entry.name.startswith('regime_result_') and entry.name.endswith('.json')):
continue
try:
txt = f.read_text(encoding='utf-8', errors='replace')
d = json.loads(txt)
ts = _parse_ts(d.get('timestamp', ''))
if ts is None:
continue
bull = float(d.get('up_ratio', 0)) * 100
bear = float(d.get('down_ratio', 0)) * 100
t0 = _tier0(bull, bear, ts, False)
t4 = _tier4(ts)
row = np.zeros(TOTAL, dtype=np.float32)
row[OFF[0]:OFF[0]+T0] = t0
row[OFF[4]:OFF[4]+T4] = t4
yield row
except Exception:
continue
def _stream_ng4() -> Iterator[np.ndarray]:
if not NG4_DIR.exists():
return
log_re = re.compile(
r'(\d{4}-\d{2}-\d{2}T[\d:.]+Z).*REGIME STATUS: \w+ \| Bull: ([\d.]+)% Bear: ([\d.]+)%'
)
for f in sorted(NG4_DIR.glob('*.txt')):
try:
for line in f.read_text(encoding='utf-8', errors='replace').splitlines():
m = log_re.search(line)
if not m:
continue
ts = _parse_ts(m.group(1).replace('T', ' ').rstrip('Z'))
if ts is None:
continue
bull, bear = float(m.group(2)), float(m.group(3))
t0 = _tier0(bull, bear, ts, False)
t4 = _tier4(ts)
row = np.zeros(TOTAL, dtype=np.float32)
row[OFF[0]:OFF[0]+T0] = t0
row[OFF[4]:OFF[4]+T4] = t4
yield row
except Exception:
continue
def _stream_ng5_local() -> Iterator[np.ndarray]:
import os
if not NG5_DIR.exists():
return
for entry in os.scandir(str(NG5_DIR)):
f = Path(entry.path)
if not (entry.name.startswith('regime_result_') and entry.name.endswith('.json')):
continue
try:
d = json.loads(f.read_text(encoding='utf-8', errors='replace'))
ts = _parse_ts(str(d.get('timestamp', '')))
if ts is None:
continue
bull = float(d.get('bull_pct', 50))
bear = float(d.get('bear_pct', 50))
mwr = d.get('multi_window_results') or {}
pricing = d.get('pricing_data') or {}
t1, has_eigen = _tier1(mwr)
t2, has_price = _tier2(pricing)
t0 = _tier0(bull, bear, ts, has_eigen)
t4 = _tier4(ts)
row = np.zeros(TOTAL, dtype=np.float32)
row[OFF[0]:OFF[0]+T0] = t0
row[OFF[1]:OFF[1]+T1] = t1
row[OFF[2]:OFF[2]+T2] = t2
row[OFF[4]:OFF[4]+T4] = t4
# No ExF for NG5 local (no companion NPZ per scan)
yield row
except Exception:
continue
def _stream_ng3_scans(exf_cache: ExFCache,
date_from: str = '2025-12-31',
max_per_day: Optional[int] = None) -> Iterator[np.ndarray]:
"""
Stream NG3/NG5 scan JSONs one at a time — never accumulates in memory.
ExF loaded once per date from companion NPZ.
max_per_day: limit scans per day (subsample for very long training days).
"""
if not NG3_EIGEN.exists():
return
date_dirs = sorted(
d for d in NG3_EIGEN.iterdir()
if d.is_dir() and not d.name.endswith('_SKIP') and d.name >= date_from
)
for date_dir in date_dirs:
exf = exf_cache.get(date_dir.name)
t3 = _tier3(exf)
day_count = 0
for f in sorted(date_dir.glob('scan_*.json')):
if '__Indicators' in f.name:
continue
if max_per_day and day_count >= max_per_day:
break
try:
# Read and immediately parse — don't accumulate
txt = f.read_text(encoding='utf-8', errors='replace')
d = json.loads(txt)
ts = _parse_ts(str(d.get('timestamp', '')))
if ts is None:
continue
windows = d.get('windows') or d.get('multi_window_results') or {}
pricing = d.get('pricing_data') or {}
pc = pricing.get('price_changes', {})
if pc:
vs = list(pc.values())
bull = 100.0 * sum(1 for v in vs if float(v) > 0) / max(len(vs), 1)
bear = 100.0 * sum(1 for v in vs if float(v) < 0) / max(len(vs), 1)
else:
bull, bear = 50.0, 50.0
t1, has_eigen = _tier1(windows)
t2, _ = _tier2(pricing)
t0 = _tier0(bull, bear, ts, has_eigen)
t4 = _tier4(ts)
row = np.zeros(TOTAL, dtype=np.float32)
row[OFF[0]:OFF[0]+T0] = t0
row[OFF[1]:OFF[1]+T1] = t1
row[OFF[2]:OFF[2]+T2] = t2
row[OFF[3]:OFF[3]+T3] = t3 # same ExF for all scans of this day
row[OFF[4]:OFF[4]+T4] = t4
yield row
day_count += 1
del d, txt # explicit release
except Exception:
continue
# ── Master corpus builder ──────────────────────────────────────────────────
class DolphinCorpus:
"""
Unified DOLPHIN corpus across all generations, 5 tiers, 111 dims.
Attributes:
X : (N, 111) float32 — the feature matrix
mask : (N, 5) bool — [t0, t1_eigen, t2_price, t3_exf, t4_esof]
sources : (N,) int8 — 0=NG1/2, 1=NG4, 2=NG5-local, 3=NG3-scan
"""
DIMS = DIMS
TOTAL = TOTAL
OFF = OFF
def __init__(self):
self.X = None
self.mask = None
self.sources = None
def build(self,
ng3_date_from: str = '2025-12-31',
max_scans_per_day: Optional[int] = None,
max_per_source: Optional[int] = None,
max_ng5: int = 3_000,
chunk_size: int = 50_000,
verbose: bool = True) -> 'DolphinCorpus':
"""
Memory-efficient build using streaming generators.
chunk_size: accumulate this many rows before extending array.
max_per_source: cap rows from NG1/NG2/NG4 (breadth-only sources).
max_ng5: separate cap for NG5-local (files are larger, reads ~26/s).
"""
print("Building DOLPHIN multi-generation corpus (streaming)...", flush=True)
exf_cache = ExFCache(NG3_EIGEN)
# Per-source caps: NG5-local is separately capped (slow reads)
_caps = {
0: max_per_source, # NG1/NG2
1: max_per_source, # NG4
2: max_ng5, # NG5-local — separate low cap
3: None, # NG3-scan — limited by max_scans_per_day
}
sources_list = [
(0, _stream_ng1_ng2(), "NG1/NG2"),
(1, _stream_ng4(), "NG4"),
(2, _stream_ng5_local(), "NG5-local"),
(3, _stream_ng3_scans(exf_cache, ng3_date_from, max_scans_per_day), "NG3-scan"),
]
all_chunks, all_src = [], []
buf_rows, buf_src = [], []
total = 0
for src_id, gen, name in sources_list:
src_count = 0
cap = _caps.get(src_id)
for row in gen:
buf_rows.append(row)
buf_src.append(src_id)
src_count += 1
total += 1
if len(buf_rows) >= chunk_size:
all_chunks.append(np.array(buf_rows, dtype=np.float32))
all_src.extend(buf_src)
buf_rows.clear(); buf_src.clear()
if verbose:
print(f" {name}: {src_count:,} (total so far: {total:,})", flush=True)
if cap and src_count >= cap:
break
if verbose:
print(f" {name}: {src_count:,} samples", flush=True)
# Flush remainder
if buf_rows:
all_chunks.append(np.array(buf_rows, dtype=np.float32))
all_src.extend(buf_src)
self.X = np.vstack(all_chunks) if all_chunks else np.empty((0, TOTAL), dtype=np.float32)
self.sources = np.array(all_src, dtype=np.int8)
np.nan_to_num(self.X, copy=False, nan=0.0, posinf=0.0, neginf=0.0)
# Build mask from has_eigen flag (bit7 of T0) and non-zero tiers
has_eigen = self.X[:, OFF[0] + 7] > 0.5 # T0[-1]
has_price = np.any(self.X[:, OFF[2]:OFF[2]+T2] != 0, axis=1)
has_exf = np.any(self.X[:, OFF[3]:OFF[3]+T3] != 0, axis=1)
self.mask = np.column_stack([
np.ones(len(self.X), dtype=bool), # T0 always
has_eigen,
has_price,
has_exf,
np.ones(len(self.X), dtype=bool), # T4 always (computed)
])
if verbose:
print(f"\nCorpus summary:")
print(f" Total : {len(self.X):,}")
print(f" Shape : {self.X.shape} ({self.X.nbytes/1e6:.0f} MB)")
print(f" T1 eigen : {self.mask[:,1].sum():,} ({100*self.mask[:,1].mean():.1f}%)")
print(f" T2 price : {self.mask[:,2].sum():,} ({100*self.mask[:,2].mean():.1f}%)")
print(f" T3 exf : {self.mask[:,3].sum():,} ({100*self.mask[:,3].mean():.1f}%)")
return self
def save(self, path: str):
p = path if path.endswith('.npz') else path + '.npz'
np.savez_compressed(p, X=self.X, mask=self.mask, sources=self.sources)
print(f"Corpus saved: {p} ({self.X.nbytes/1e6:.0f} MB uncompressed, compressed ~10x)")
@classmethod
def load(cls, path: str) -> 'DolphinCorpus':
c = cls()
p = path if path.endswith('.npz') else path + '.npz'
d = np.load(p)
c.X, c.mask, c.sources = d['X'], d['mask'], d['sources']
print(f"Corpus loaded: {len(c.X):,} samples, {c.X.shape[1]} dims")
return c
# ── Tier slices ─────────────────────────────────────────────────────
def t0(self): return self.X[:, OFF[0]:OFF[0]+T0]
def t1(self): return self.X[:, OFF[1]:OFF[1]+T1]
def t2(self): return self.X[:, OFF[2]:OFF[2]+T2]
def t3(self): return self.X[:, OFF[3]:OFF[3]+T3]
def t4(self): return self.X[:, OFF[4]:OFF[4]+T4]
def tier_names(self):
return ['breadth+time', 'eigenvalues', 'per-asset-vol', 'ExF-macro', 'EsoF']
def describe(self):
print(f"Corpus: N={len(self.X):,} dims={TOTAL} ({self.X.nbytes/1e6:.0f}MB)")
print(f"Tiers: {list(zip(self.tier_names(), DIMS))}")
print(f"Masks: {[(t, self.mask[:,i].sum()) for i, t in enumerate(self.tier_names())]}")
src_names = {0: 'NG1/2', 1: 'NG4', 2: 'NG5-local', 3: 'NG3-scan'}
for sid, name in src_names.items():
n = (self.sources == sid).sum()
if n > 0:
print(f" {name:12s}: {n:,}")
if __name__ == '__main__':
import sys
max_per_day = int(sys.argv[1]) if len(sys.argv) > 1 else None
corpus = DolphinCorpus().build(verbose=True, max_scans_per_day=max_per_day)
corpus.save(str(HERE / 'corpus_cache'))
corpus.describe()

BIN
nautilus_dolphin/dvae/corpus_cache.npz Executable file
View File

Binary file not shown.

88
nautilus_dolphin/dvae/data_range_archaeology.py Executable file
View File

@@ -0,0 +1,88 @@
import json
import re
import os
from pathlib import Path
from datetime import datetime
BASE = Path(r"C:\Users\Lenovo\Documents")
DIRS = {
"NG1": BASE / "- Dolphin NG",
"NG2": BASE / "- Dolphin NG2",
"NG4": BASE / "- DOLPHIN NG4" / "- Results",
"NG5": BASE / "- Dolphin NG5",
"NG3": BASE / "- Dolphin NG HD (NG3)" / "correlation_arb512" / "eigenvalues"
}
def parse_ts(s):
for fmt in ("%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%dT%H:%M:%S",
"%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S",
"%Y-%m-%dT%H:%M:%SZ"):
try:
return datetime.strptime(str(s)[:26].replace('Z', ''), fmt)
except ValueError:
continue
return None
def scan_dir_json(d, pattern):
ts_list = []
if not d.exists(): return ts_list
for f in d.glob(pattern):
try:
with open(f, 'r', encoding='utf-8', errors='replace') as fb:
data = json.load(fb)
ts_str = data.get('timestamp')
if ts_str:
ts = parse_ts(ts_str)
if ts: ts_list.append(ts)
except: continue
return ts_list
def scan_ng4(d):
ts_list = []
if not d.exists(): return ts_list
log_re = re.compile(r'(\d{4}-\d{2}-\d{2}T[\d:.]+Z)')
for f in d.glob('*.txt'):
try:
with open(f, 'r', encoding='utf-8', errors='replace') as fb:
for line in fb:
m = log_re.search(line)
if m:
ts = parse_ts(m.group(1))
if ts: ts_list.append(ts)
except: continue
return ts_list
def scan_ng3(d):
ts_list = []
if not d.exists(): return ts_list
# Just check the first and last date directories to save time
subdirs = sorted([s for s in d.iterdir() if s.is_dir() and not s.name.endswith('_SKIP')])
if not subdirs: return ts_list
for subdir in [subdirs[0], subdirs[-1]]:
for f in subdir.glob('scan_*.json'):
if '__Indicators' in f.name: continue
try:
with open(f, 'r', encoding='utf-8', errors='replace') as fb:
data = json.load(fb)
ts_str = data.get('timestamp')
if ts_str:
ts = parse_ts(ts_str)
if ts: ts_list.append(ts)
except: continue
return ts_list
print("--- Data Archaeology Result ---")
for name, d in DIRS.items():
print(f"Checking {name} in {d}...")
if name in ["NG1", "NG2", "NG5"]:
times = scan_dir_json(d, 'regime_result_*.json')
elif name == "NG4":
times = scan_ng4(d)
elif name == "NG3":
times = scan_ng3(d)
if times:
print(f" {name}: {min(times)} to {max(times)} ({len(times)} samples found in scan)")
else:
print(f" {name}: No data found.")

153
nautilus_dolphin/dvae/diagnose_latents.py Executable file
View File

@@ -0,0 +1,153 @@
"""
What did the encoder actually learn?
Correlate z0/z1 latent dims with raw input features.
"""
import sys
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
import numpy as np
from pathlib import Path
HERE = Path(__file__).parent
sys.path.insert(0, str(HERE))
from corpus_builder import DolphinCorpus, WINDOWS
from hierarchical_dvae import HierarchicalDVAE, T_OFF, TIER0_DIM, TIER1_DIM, TIER3_DIM
# ── Feature names ─────────────────────────────────────────────────────────
T0_NAMES = ['bull_pct', 'bear_pct', 'side_pct', 'sin_hour', 'cos_hour', 'sin_day', 'cos_day', 'has_eigen']
T1_NAMES = []
for w in WINDOWS:
for feat in ['log_lmax', 'vel_norm', 'gap_ratio', 'instability', 'rtp']:
T1_NAMES.append(f"w{w}_{feat}")
EXF_FIELDS = [
'dvol_btc', 'dvol_eth', 'fng', 'fng_prev', 'btc_dom', 'eth_dom',
'chg24_btc', 'chg24_eth', 'dispersion', 'correlation', 'imbal_btc', 'imbal_eth',
'funding_btc', 'funding_eth', 'mvrv', 'tvl', 'pcr_vol', 'pcr_oi',
'basis', 'liq_proxy', 'spread', 'vol24', 'hashrate', 'btc_price', 'fng_vol',
]
# ── Load ───────────────────────────────────────────────────────────────────
print("Loading corpus...")
corpus = DolphinCorpus.load(str(HERE / 'corpus_cache.npz'))
idx = corpus.mask[:, 1] # 16K eigen samples
X_e = corpus.X[idx]
mask_e = corpus.mask[idx]
print(f"Eigen subset: {len(X_e):,} samples")
print("Loading model...")
model = HierarchicalDVAE(hidden=128, beta=0.5, gamma=1.0, lam=1.0, seed=42)
model.fit_normaliser(corpus.X, corpus.mask)
# Load weights
d = np.load(str(HERE / 'hdvae_checkpoint.npz'), allow_pickle=True)
def load_enc(enc, name):
for i, layer in enumerate(enc.mlp.layers):
layer.W = d[f'{name}_mlp{i}_W']; layer.b = d[f'{name}_mlp{i}_b']
enc.mu_head.W = d[f'{name}_mu_W']; enc.mu_head.b = d[f'{name}_mu_b']
enc.lv_head.W = d[f'{name}_lv_W']; enc.lv_head.b = d[f'{name}_lv_b']
def load_dec(dec, name):
for i, layer in enumerate(dec.mlp.layers):
layer.W = d[f'{name}_mlp{i}_W']; layer.b = d[f'{name}_mlp{i}_b']
for n, e in [('enc0',model.enc0),('enc1',model.enc1),('enc2',model.enc2)]:
load_enc(e, n)
for n, dc in [('dec0',model.dec0),('dec1',model.dec1),('dec2',model.dec2)]:
load_dec(dc, n)
# ── Encode all 16K samples ─────────────────────────────────────────────────
print("Encoding...")
rng = np.random.RandomState(0)
BATCH = 512
mu0_all, mu1_all = [], []
for start in range(0, len(X_e), BATCH):
Xb = X_e[start:start+BATCH]
mb = mask_e[start:start+BATCH]
enc = model.encode(Xb, mb, rng)
mu0_all.append(enc['mu0'])
mu1_all.append(enc['mu1'])
mu0 = np.concatenate(mu0_all) # (N, 4)
mu1 = np.concatenate(mu1_all) # (N, 8)
print(f"\nmu0 stats: mean={mu0.mean(0).round(4)} std={mu0.std(0).round(4)}")
print(f"mu1 stats: mean={mu1.mean(0).round(4)} std={mu1.std(0).round(4)}")
# ── Raw features ──────────────────────────────────────────────────────────
t0_raw = X_e[:, T_OFF[0]:T_OFF[0]+TIER0_DIM]
t1_raw = X_e[:, T_OFF[1]:T_OFF[1]+TIER1_DIM]
t3_raw = X_e[:, T_OFF[3]:T_OFF[3]+TIER3_DIM]
# ── Correlation: z1 dims vs T1 features ───────────────────────────────────
print("\n" + "="*70)
print("z1 DIMS vs T1 FEATURES (top correlations per z1 dim)")
print("="*70)
for zd in range(8):
corrs = [np.corrcoef(mu1[:,zd], t1_raw[:,fd])[0,1] for fd in range(TIER1_DIM)]
corrs = np.array(corrs)
top3 = np.argsort(np.abs(corrs))[-3:][::-1]
var = mu1[:,zd].var()
print(f"z1[{zd}] var={var:.4f}: " + " ".join(f"{T1_NAMES[i]}={corrs[i]:+.3f}" for i in top3))
# ── Correlation: z0 dims vs T0 features ───────────────────────────────────
print("\n" + "="*70)
print("z0 DIMS vs T0 FEATURES (top correlations per z0 dim)")
print("="*70)
for zd in range(4):
corrs = [np.corrcoef(mu0[:,zd], t0_raw[:,fd])[0,1] for fd in range(TIER0_DIM)]
corrs = np.array(corrs)
top3 = np.argsort(np.abs(corrs))[-3:][::-1]
var = mu0[:,zd].var()
print(f"z0[{zd}] var={var:.5f}: " + " ".join(f"{T0_NAMES[i]}={corrs[i]:+.3f}" for i in top3))
# ── What is z1 actually distinguishing? ───────────────────────────────────
print("\n" + "="*70)
print("z1[4] (highest var=0.015): value distribution vs T1 raw ranges")
print("="*70)
z1_4 = mu1[:, 4] # dim with highest var (index 4 = z1[4])
# Actually find which z1 dim has highest variance
best_z1 = np.argmax(mu1.var(0))
z1_best = mu1[:, best_z1]
print(f"Best z1 dim: {best_z1}, var={mu1[:,best_z1].var():.4f}")
# Split into top/bottom 20% by z1 value
lo_mask = z1_best < np.percentile(z1_best, 20)
hi_mask = z1_best > np.percentile(z1_best, 80)
print(f"\nT1 feature means: LOW z1[{best_z1}] (bot20%) vs HIGH z1[{best_z1}] (top20%)")
print(f"{'Feature':<20} {'LOW':>8} {'HIGH':>8} {'diff':>8}")
for fd, name in enumerate(T1_NAMES):
lo_val = t1_raw[lo_mask, fd].mean()
hi_val = t1_raw[hi_mask, fd].mean()
diff = hi_val - lo_val
if abs(diff) > 0.02:
print(f" {name:<18} {lo_val:8.4f} {hi_val:8.4f} {diff:+8.4f}")
# ── ExF correlation check ─────────────────────────────────────────────────
print("\n" + "="*70)
print(f"z1[{best_z1}] vs ExF (T3) features")
print("="*70)
exf_corrs = [(np.corrcoef(z1_best, t3_raw[:,i])[0,1], EXF_FIELDS[i]) for i in range(min(25, t3_raw.shape[1]))]
exf_corrs.sort(key=lambda x: abs(x[0]), reverse=True)
for r, name in exf_corrs[:10]:
print(f" {name:<20} r={r:+.4f}")
# ── z0 clustering: what do the 7 clusters look like? ─────────────────────
print("\n" + "="*70)
print("z0 cluster analysis (k=7, what separates them?)")
print("="*70)
from scipy.cluster.vq import kmeans2
try:
centroids, labels = kmeans2(mu0, 7, seed=42, minit='points')
print(f"Cluster sizes: {np.bincount(labels)}")
print(f"\nCluster centroids (z0 space):")
for k in range(7):
c = centroids[k]
# What T0 features distinguish this cluster?
mask_k = labels == k
t0_k = t0_raw[mask_k].mean(0)
t1_k = t1_raw[mask_k].mean(0)
print(f"\n Cluster {k} (N={mask_k.sum():,}): z0={c.round(3)}")
print(f" T0: bull={t0_k[0]:.3f} bear={t0_k[1]:.3f} side={t0_k[2]:.3f}")
print(f" T1: log_lmax_w50={t1_k[0]:.3f} vel_norm_w50={t1_k[1]:+.3f} gap_ratio_w50={t1_k[2]:.3f} inst={t1_k[3]:.3f} rtp={t1_k[4]:.3f}")
except Exception as e:
print(f"Clustering failed: {e}")
print("\nDone.")

151
nautilus_dolphin/dvae/e2e_precursor_auc.py Executable file
View File

@@ -0,0 +1,151 @@
"""
Task 3: E2E Precursor AUC Test.
Train FlintHDVAE (beta=0.1) on 80% of 16K T1 corpus.
Encode all samples → z (8-dim latent).
Build eigenspace stress labels at K=5 scans (25s): inst>p90 AND gap<p10.
Test logistic regression on z → stress labels (chronological OOS split).
Compare against proxy_B baseline (AUC=0.715 from flint_precursor_sweep.py).
Gate: AUC ≥ 0.65 → proceed to Task 4.
"""
import sys, os
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict")
import numpy as np
from pathlib import Path
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score
HERE = Path(__file__).parent
# ── Load 16K eigen corpus ─────────────────────────────────────────
print("Loading 16K eigen corpus...")
from corpus_builder import DolphinCorpus, OFF, T1 as T1_DIM
corpus = DolphinCorpus.load(str(HERE / 'corpus_cache.npz'))
idx_mask = corpus.mask[:, 1]
X_e = corpus.X[idx_mask]
T1 = X_e[:, OFF[1]:OFF[1] + T1_DIM].copy() # (16607, 20)
N = len(T1)
print(f" N={N} T1 shape={T1.shape}")
# ── Feature shortcuts for proxy_B baseline ────────────────────────
inst_w50 = T1[:, 3]
vel_w750 = T1[:, 16]
gap_w50 = T1[:, 2]
proxy_B = inst_w50 - vel_w750
# ── Build stress labels: K=5 scans (25s), inst>p90, gap<p10 ──────
print("\nBuilding stress labels (K=5, inst>p90, gap<p10)...")
K = 5
inst_p90 = np.percentile(inst_w50, 90)
gap_p10 = np.percentile(gap_w50, 10)
print(f" inst_p90={inst_p90:.4f} gap_p10={gap_p10:.4f}")
labels = np.zeros(N, dtype=np.float32)
for i in range(N - K):
fi = inst_w50[i+1:i+1+K]
fg = gap_w50 [i+1:i+1+K]
if np.any(fi > inst_p90) and np.any(fg < gap_p10):
labels[i] = 1.0
pos_rate = labels.mean()
print(f" Positive rate: {pos_rate*100:.1f}% Positive count: {labels.sum():.0f}")
# ── Proxy B baseline (no model) ───────────────────────────────────
print("\n" + "="*55)
print("PROXY_B BASELINE (direct, no model)")
print("="*55)
pB_vals = proxy_B[:-K]
y_vals = labels[:-K]
valid = np.isfinite(pB_vals) & np.isfinite(y_vals)
# Chronological split (same as Task 3 below)
n_test = len(pB_vals) // 4
pB_test = pB_vals[-n_test:]
y_test = y_vals[-n_test:]
auc_pB = roc_auc_score(y_test, pB_test)
auc_pB = max(auc_pB, 1 - auc_pB)
print(f" proxy_B OOS AUC = {auc_pB:.4f}")
# ── Train FlintHDVAE ──────────────────────────────────────────────
print("\n" + "="*55)
print("TRAINING FlintHDVAE (beta=0.1, 40 epochs)")
print("="*55)
from flint_hd_vae import FlintHDVAE
# Chronological 80/20 train split for the VAE itself
n_vae_train = int(N * 0.8)
T1_vae_train = T1[:n_vae_train]
model = FlintHDVAE(input_dim=20, hd_dim=512, latent_dim=8,
beta=0.1, seed=42, use_flint_norm=False)
model.fit(T1_vae_train, epochs=40, lr=1e-3, batch_size=256,
verbose=True, warmup_frac=0.3)
# ── Encode full corpus → z ────────────────────────────────────────
print("\nEncoding full 16K corpus → z (8-dim)...")
z_all = model.encode(T1) # (16607, 8)
print(f" z shape: {z_all.shape}")
print(f" z range: [{z_all.min():.3f}, {z_all.max():.3f}]")
print(f" z var per dim: {z_all.var(0).round(3)}")
# ── Logistic regression: z → stress labels ───────────────────────
print("\n" + "="*55)
print("LOGISTIC REGRESSION: z_regime → stress labels (K=5)")
print("="*55)
X_lr = z_all[:-K]
y_lr = labels[:-K]
valid_lr = np.isfinite(X_lr).all(1) & np.isfinite(y_lr)
X_lr, y_lr = X_lr[valid_lr], y_lr[valid_lr]
n_test_lr = len(X_lr) // 4
X_train_lr = X_lr[:-n_test_lr]
X_test_lr = X_lr[-n_test_lr:]
y_train_lr = y_lr[:-n_test_lr]
y_test_lr = y_lr[-n_test_lr:]
print(f" Train: {len(X_train_lr)} Test: {len(X_test_lr)}")
print(f" Test pos rate: {y_test_lr.mean()*100:.1f}%")
lr_clf = LogisticRegression(class_weight='balanced', max_iter=500, C=0.1)
lr_clf.fit(X_train_lr, y_train_lr)
preds = lr_clf.predict_proba(X_test_lr)[:, 1]
auc_z = roc_auc_score(y_test_lr, preds)
auc_z = max(auc_z, 1 - auc_z)
ap_z = average_precision_score(y_test_lr, preds)
print(f" z-regime LogReg OOS AUC={auc_z:.4f} AvgPrecision={ap_z:.4f}")
# ── Combined: z + proxy_B ─────────────────────────────────────────
print("\n" + "="*55)
print("COMBINED: z_regime + proxy_B")
print("="*55)
X_comb = np.column_stack([z_all[:-K], proxy_B[:-K].reshape(-1,1)])[valid_lr]
X_c_train = X_comb[:-n_test_lr]
X_c_test = X_comb[-n_test_lr:]
lr_comb = LogisticRegression(class_weight='balanced', max_iter=500, C=0.1)
lr_comb.fit(X_c_train, y_train_lr)
preds_c = lr_comb.predict_proba(X_c_test)[:, 1]
auc_c = roc_auc_score(y_test_lr, preds_c)
auc_c = max(auc_c, 1 - auc_c)
print(f" Combined OOS AUC={auc_c:.4f}")
# ── Summary and Gate ──────────────────────────────────────────────
print("\n" + "="*55)
print("SUMMARY")
print("="*55)
print(f" proxy_B direct: AUC = {auc_pB:.4f}")
print(f" z_regime (VAE): AUC = {auc_z:.4f}")
print(f" z + proxy_B: AUC = {auc_c:.4f}")
GATE_AUC = 0.65
best_auc = max(auc_pB, auc_z, auc_c)
print(f"\n Gate threshold: AUC ≥ {GATE_AUC}")
if best_auc >= GATE_AUC:
print(f" GATE PASS: best AUC={best_auc:.4f} ≥ {GATE_AUC}")
print(" → Proceed to Task 4: fork AlphaSignalGenerator with proxy_B gate")
else:
print(f" GATE FAIL: best AUC={best_auc:.4f} < {GATE_AUC}")
print(" → Do NOT proceed with gate integration")

403
nautilus_dolphin/dvae/exp10_1m_keyframe.py Executable file
View File

@@ -0,0 +1,403 @@
"""
exp10_1m_keyframe.py — 1m trajectory keyframe gate test
=========================================================
Tests whether instability_150 z-score at 1m timescale (and/or D-VAE recon_err
at 1m) provides actionable scale modulation ON TOP OF D_LIQ_GOLD (proxy_B boost
already baked in).
Signal forms tested (7 configs):
0. Baseline — D_LIQ_GOLD unmodified (control)
1. A_hard — hard threshold on rolling i150 z-score
2. B_analogue — continuous tanh on rolling i150 z-score
3. C_hard — hard threshold on VAE recon_err z-score at 1m
4. D_analogue — continuous tanh on VAE recon_err z-score at 1m
5. AC_hard — A × C (both hard, multiplicative)
6. BD_analogue — B × D (both analogue, multiplicative)
Analogue formula (split tanh, asymmetric):
z >= 0: scale = 1 + UP_STRENGTH * tanh(z / K) → [1.0, ~1.15)
z < 0: scale = 1 + DOWN_STRENGTH * tanh(z / K) → (~0.50, 1.0)
At z=-1.22: ~0.64; z=-3: ~0.50 floor; z=+1.11: ~1.12; z=+3: ~1.15 ceiling.
Zero changes to production code. D_LIQ_GOLD engine forked via subclass.
"""
import sys, time, json, warnings
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
warnings.filterwarnings('ignore')
from pathlib import Path
import numpy as np
import pandas as pd
ROOT = Path(__file__).parent.parent
sys.path.insert(0, str(ROOT))
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from nautilus_dolphin.nautilus.ob_features import (
OBFeatureEngine, compute_imbalance_nb, compute_depth_1pct_nb,
compute_depth_quality_nb, compute_fill_probability_nb, compute_spread_proxy_nb,
compute_depth_asymmetry_nb, compute_imbalance_persistence_nb,
compute_withdrawal_velocity_nb, compute_market_agreement_nb, compute_cascade_signal_nb,
)
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.proxy_boost_engine import LiquidationGuardEngine, create_d_liq_engine
from mc.mc_ml import DolphinForewarner
from dvae.titan_sensor import TitanSensor, build_feature_vector
# ── JIT warmup ────────────────────────────────────────────────────────────────
print("Warming up JIT...")
_p = np.array([1.,2.,3.], dtype=np.float64)
compute_irp_nb(_p,-1); compute_ars_nb(1.,.5,.01)
rank_assets_irp_nb(np.ones((10,2),dtype=np.float64),8,-1,5,500.,20,0.20)
compute_sizing_nb(-.03,-.02,-.05,3.,.5,5.,.20,True,True,0.,
np.zeros(4,dtype=np.int64),np.zeros(4,dtype=np.int64),
np.zeros(5,dtype=np.float64),0,-1,.01,.04)
check_dc_nb(_p,3,1,.75)
_b=np.array([100.,200.,300.,400.,500.],dtype=np.float64)
_a=np.array([110.,190.,310.,390.,510.],dtype=np.float64)
compute_imbalance_nb(_b,_a); compute_depth_1pct_nb(_b,_a)
compute_depth_quality_nb(210.,200.); compute_fill_probability_nb(1.)
compute_spread_proxy_nb(_b,_a); compute_depth_asymmetry_nb(_b,_a)
compute_imbalance_persistence_nb(np.array([.1,-.1],dtype=np.float64),2)
compute_withdrawal_velocity_nb(np.array([100.,110.],dtype=np.float64),1)
compute_market_agreement_nb(np.array([.1,-.05],dtype=np.float64),2)
compute_cascade_signal_nb(np.array([-.05,-.15],dtype=np.float64),2,-.10)
print(" JIT ready.")
# ── Paths ─────────────────────────────────────────────────────────────────────
VBT5s = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
VBT1m = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines")
MODEL_PATH = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\dvae_regime_model_TITAN_ULTRA_GD.json")
MC_MODELS = str(ROOT / "mc_results" / "models")
OUT_FILE = Path(__file__).parent / "exp10_1m_keyframe_results.json"
META_COLS = {'timestamp','scan_number','v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity','vel_div',
'instability_50','instability_150'}
# Base kwargs accepted by NDAlphaEngine (no boost/leverage-guard params)
BASE_ENGINE_KWARGS = dict(
initial_capital=25000., vel_div_threshold=-.02, vel_div_extreme=-.05,
min_leverage=.5, max_leverage=5., leverage_convexity=3.,
fraction=.20, fixed_tp_pct=.0095, stop_pct=1., max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=.75,
dc_skip_contradicts=True, dc_leverage_boost=1., dc_leverage_reduce=.5,
use_asset_selection=True, min_irp_alignment=.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=.62, sp_maker_exit_rate=.50,
use_ob_edge=True, ob_edge_bps=5., ob_confirm_rate=.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
# D_LIQ_GOLD-specific params passed explicitly to KeyframeGateEngine / create_d_liq_engine
D_LIQ_KWARGS = dict(
extended_soft_cap=8., extended_abs_cap=9., mc_leverage_ref=5.,
margin_buffer=.95, threshold=.35, alpha=1., adaptive_beta=True,
)
MC_BASE_CFG = {
'trial_id':0, 'vel_div_threshold':-.020, 'vel_div_extreme':-.050,
'use_direction_confirm':True, 'dc_lookback_bars':7, 'dc_min_magnitude_bps':.75,
'dc_skip_contradicts':True, 'dc_leverage_boost':1.00, 'dc_leverage_reduce':.50,
'vd_trend_lookback':10, 'min_leverage':.50, 'max_leverage':5.00,
'leverage_convexity':3.00, 'fraction':.20, 'use_alpha_layers':True,
'use_dynamic_leverage':True, 'fixed_tp_pct':.0095, 'stop_pct':1.00,
'max_hold_bars':120, 'use_sp_fees':True, 'use_sp_slippage':True,
'sp_maker_entry_rate':.62, 'sp_maker_exit_rate':.50, 'use_ob_edge':True,
'ob_edge_bps':5.00, 'ob_confirm_rate':.40, 'ob_imbalance_bias':-.09,
'ob_depth_scale':1.00, 'use_asset_selection':True, 'min_irp_alignment':.45,
'lookback':100, 'acb_beta_high':.80, 'acb_beta_low':.20, 'acb_w750_threshold_pct':60,
}
WINDOW_Z = 30 # rolling bars for z-score (30-min at 1m)
K_TANH = 1.5 # tanh curvature
UP_STR = 0.15 # max upside boost
DOWN_STR = 0.50 # max downside cut
# ── Scale functions ───────────────────────────────────────────────────────────
def hard_scale(z: float) -> float:
if z < 0.: return 0.5
if z > 1.11: return 1.15
return 1.0
def analogue_scale(z: float) -> float:
t = np.tanh(float(z) / K_TANH)
if z >= 0.:
return 1.0 + UP_STR * t
else:
return 1.0 + DOWN_STR * t
# ── Pre-compute 1m signals ────────────────────────────────────────────────────
def precompute_1m_signals(parquet_files_5s, sensor):
"""
For each day, build per-5s-bar arrays of:
z_roll[bar] : rolling z-score of instability_150 at 1m
z_recon[bar] : rolling z-score of VAE recon_err at 1m
Returns dict[date_str] -> {'z_roll': np.ndarray, 'z_recon': np.ndarray}
"""
print("Pre-computing 1m signals...")
signals = {}
for pf5 in parquet_files_5s:
ds = pf5.stem
pf1 = VBT1m / f"{ds}.parquet"
if not pf1.exists():
signals[ds] = None
continue
df5 = pd.read_parquet(pf5)
df1 = pd.read_parquet(pf1).replace([np.inf,-np.inf], np.nan).fillna(0.)
n5, n1 = len(df5), len(df1)
assets = [c for c in df1.columns if c not in META_COLS]
# 1m instability_150 rolling z-score
i150 = df1['instability_150'].values.copy() if 'instability_150' in df1.columns else np.zeros(n1)
z_roll_1m = np.zeros(n1)
for j in range(WINDOW_Z, n1):
seg = i150[max(0,j-WINDOW_Z):j]
mu, sigma = np.mean(seg), np.std(seg)
z_roll_1m[j] = (i150[j] - mu) / max(sigma, 1e-10)
# 1m VAE recon_err rolling z-score
recon_1m = np.zeros(n1)
for j in range(n1):
feat = build_feature_vector(df1, j, assets)
_, recon_err, _ = sensor.encode(feat)
recon_1m[j] = recon_err
# z-score recon_err
z_recon_1m = np.zeros(n1)
for j in range(WINDOW_Z, n1):
seg = recon_1m[max(0,j-WINDOW_Z):j]
mu, sigma = np.mean(seg), np.std(seg)
z_recon_1m[j] = (recon_1m[j] - mu) / max(sigma, 1e-10)
# Map 1m arrays to 5s bar indices (proportional)
z_roll_5s = np.zeros(n5)
z_recon_5s = np.zeros(n5)
for i in range(n5):
j = int(i * n1 / n5)
j = min(j, n1-1)
z_roll_5s[i] = z_roll_1m[j]
z_recon_5s[i] = z_recon_1m[j]
signals[ds] = {'z_roll': z_roll_5s, 'z_recon': z_recon_5s}
print(f" {ds}: {n1} 1m bars -> {n5} 5s bars "
f"z_roll=[{z_roll_5s.min():.2f},{z_roll_5s.max():.2f}] "
f"z_recon=[{z_recon_5s.min():.2f},{z_recon_5s.max():.2f}]")
return signals
# ── Keyframe engine subclass ──────────────────────────────────────────────────
class KeyframeGateEngine(LiquidationGuardEngine):
"""
Adds a 1m keyframe scale multiplier on top of D_LIQ_GOLD.
_pending_1m_scale is set per bar by the backtest loop before _try_entry fires.
"""
def __init__(self, scale_fn, **kwargs):
super().__init__(**kwargs)
self._scale_fn = scale_fn # callable(z_roll, z_recon) -> float
self._bar_z_roll : np.ndarray | None = None
self._bar_z_recon : np.ndarray | None = None
self._1m_scale_history: list = []
def set_1m_signals(self, z_roll: np.ndarray, z_recon: np.ndarray):
self._bar_z_roll = z_roll
self._bar_z_recon = z_recon
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0., v750_vel=0.):
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
if result and self.position is not None:
zr = float(self._bar_z_roll[bar_idx]) if (self._bar_z_roll is not None and bar_idx < len(self._bar_z_roll)) else 0.
ze = float(self._bar_z_recon[bar_idx]) if (self._bar_z_recon is not None and bar_idx < len(self._bar_z_recon)) else 0.
s = float(self._scale_fn(zr, ze))
s = max(0.2, min(2.0, s)) # hard clip, safety
self.position.notional *= s
self._1m_scale_history.append(s)
return result
def reset(self):
super().reset()
self._1m_scale_history = []
# ── Scale function definitions ─────────────────────────────────────────────────
CONFIGS = {
"0_baseline": None, # no gate
"1_A_hard": lambda zr, ze: hard_scale(zr),
"2_B_analogue": lambda zr, ze: analogue_scale(zr),
"3_C_hard": lambda zr, ze: hard_scale(ze),
"4_D_analogue": lambda zr, ze: analogue_scale(ze),
"5_AC_hard": lambda zr, ze: hard_scale(zr) * hard_scale(ze),
"6_BD_analogue":lambda zr, ze: analogue_scale(zr) * analogue_scale(ze),
}
# ── Backtest runner ───────────────────────────────────────────────────────────
def run_one(config_name, scale_fn, parquet_files, pq_data, signals, vol_p60):
"""Run one backtest config. Returns result dict."""
OB_ASSETS = sorted({a for ds,(df,ac,_) in pq_data.items() for a in ac})
_mock_ob = MockOBProvider(
imbalance_bias=-.09, depth_scale=1., assets=OB_ASSETS,
imbalance_biases={"BTCUSDT":-.086,"ETHUSDT":-.092,"BNBUSDT":+.05,"SOLUSDT":+.05},
)
ob_eng = OBFeatureEngine(_mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
forewarner = DolphinForewarner(models_dir=MC_MODELS)
acb = AdaptiveCircuitBreaker()
acb.preload_w750([pf.stem for pf in parquet_files])
if scale_fn is None:
engine = create_d_liq_engine(**BASE_ENGINE_KWARGS)
else:
engine = KeyframeGateEngine(scale_fn=scale_fn, **BASE_ENGINE_KWARGS, **D_LIQ_KWARGS)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.)
t0 = time.time()
bar_global = 0
for pf in parquet_files:
ds = pf.stem
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
if scale_fn is not None and isinstance(engine, KeyframeGateEngine):
sig = signals.get(ds)
if sig:
engine.set_1m_signals(sig['z_roll'], sig['z_recon'])
else:
engine.set_1m_signals(np.zeros(len(df)), np.zeros(len(df)))
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
bar_global += len(df)
elapsed = time.time() - t0
trades = engine.trade_history
roi = (engine.capital - 25000.) / 25000. * 100.
# drawdown
cap_curve = [25000.]
for t_ in sorted(trades, key=lambda x: getattr(x,'exit_bar',0)):
cap_curve.append(cap_curve[-1] + getattr(t_,'pnl_absolute',0.))
cap_arr = np.array(cap_curve)
peak = np.maximum.accumulate(cap_arr)
dd = float(np.max((peak - cap_arr) / (peak + 1e-10)) * 100.)
calmar = roi / max(dd, 1e-4)
scale_hist = getattr(engine, '_1m_scale_history', [])
res = {
'config': config_name,
'T': len(trades),
'ROI': round(roi, 4),
'DD': round(dd, 4),
'Calmar': round(calmar, 4),
'elapsed_s': round(elapsed, 1),
'scale_mean': round(np.mean(scale_hist), 4) if scale_hist else 1.0,
'scale_min': round(np.min(scale_hist), 4) if scale_hist else 1.0,
'scale_max': round(np.max(scale_hist), 4) if scale_hist else 1.0,
'n_scale_applied': len(scale_hist),
}
return res
# ── Main ──────────────────────────────────────────────────────────────────────
def main():
parquet_files = sorted(VBT5s.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
print(f"Dataset: {len(parquet_files)} days ({parquet_files[0].stem} to {parquet_files[-1].stem})")
print("Loading parquet data...")
pq_data = {}
all_assets = set()
for pf in parquet_files:
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
all_assets.update(ac)
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0,i-50):i]
if len(seg)>=10:
dv[i] = float(np.std(np.diff(seg)/seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
# vol_p60
all_vols=[]
for pf in parquet_files[:2]:
df=pd.read_parquet(pf)
if 'BTCUSDT' not in df.columns: continue
pr=df['BTCUSDT'].values
for i in range(60,len(pr)):
seg=pr[max(0,i-50):i]
if len(seg)>=10:
v=float(np.std(np.diff(seg)/seg[:-1]))
if v>0: all_vols.append(v)
vol_p60 = float(np.percentile(all_vols,60)) if all_vols else 0.
# Load sensor
print(f"\nLoading TitanSensor...")
sensor = TitanSensor(str(MODEL_PATH))
# Pre-compute 1m signals
signals = precompute_1m_signals(parquet_files, sensor)
n_missing = sum(1 for v in signals.values() if v is None)
print(f" 1m signals ready: {len(signals)-n_missing}/{len(signals)} days")
# Run all configs
print()
results = []
for name, fn in CONFIGS.items():
print(f"Running [{name}]...", flush=True)
r = run_one(name, fn, parquet_files, pq_data, signals, vol_p60)
results.append(r)
baseline_roi = results[0]['ROI'] if results else 0.
delta_roi = r['ROI'] - baseline_roi
delta_dd = r['DD'] - results[0]['DD'] if len(results)>1 else 0.
print(f" T={r['T']} ROI={r['ROI']:+.2f}% DD={r['DD']:.2f}% "
f"Calmar={r['Calmar']:.2f} "
f"(dROI={delta_roi:+.2f}pp dDD={delta_dd:+.2f}pp) "
f"scale_mean={r['scale_mean']:.3f} {r['elapsed_s']:.0f}s")
# Summary table
print()
print("="*90)
print(f"{'Config':<20} {'T':>5} {'ROI%':>8} {'DD%':>7} {'Calmar':>7} "
f"{'dROI':>7} {'dDD':>6} {'s_mean':>7}")
print("-"*90)
base = results[0]
for r in results:
dr = r['ROI'] - base['ROI']
dd = r['DD'] - base['DD']
print(f"{r['config']:<20} {r['T']:>5} {r['ROI']:>8.2f} {r['DD']:>7.2f} "
f"{r['Calmar']:>7.2f} {dr:>+7.2f} {dd:>+6.2f} {r['scale_mean']:>7.3f}")
# Save
with open(OUT_FILE,'w') as f:
json.dump({'baseline': base, 'results': results}, f, indent=2)
print(f"\nResults: {OUT_FILE}")
# Verdict
print("\n=== VERDICT ===")
best = max(results[1:], key=lambda x: x['Calmar'])
print(f"Best config: [{best['config']}] Calmar={best['Calmar']:.2f} "
f"ROI={best['ROI']:+.2f}% DD={best['DD']:.2f}%")
if best['Calmar'] > base['Calmar'] * 1.02:
print(" PROCEED: meaningful Calmar improvement vs baseline")
else:
print(" MARGINAL or NO improvement over D_LIQ_GOLD")
if __name__ == '__main__':
main()

100
nautilus_dolphin/dvae/exp10_1m_keyframe_results.json Executable file
View File

@@ -0,0 +1,100 @@
{
"baseline": {
"config": "0_baseline",
"T": 2155,
"ROI": 181.8069,
"DD": 23.6916,
"Calmar": 7.6739,
"elapsed_s": 339.8,
"scale_mean": 1.0,
"scale_min": 1.0,
"scale_max": 1.0,
"n_scale_applied": 0
},
"results": [
{
"config": "0_baseline",
"T": 2155,
"ROI": 181.8069,
"DD": 23.6916,
"Calmar": 7.6739,
"elapsed_s": 339.8,
"scale_mean": 1.0,
"scale_min": 1.0,
"scale_max": 1.0,
"n_scale_applied": 0
},
{
"config": "1_A_hard",
"T": 2155,
"ROI": 179.3283,
"DD": 23.6916,
"Calmar": 7.5693,
"elapsed_s": 312.4,
"scale_mean": 0.997,
"scale_min": 0.5,
"scale_max": 1.15,
"n_scale_applied": 2156
},
{
"config": "2_B_analogue",
"T": 2155,
"ROI": 182.3446,
"DD": 23.6916,
"Calmar": 7.6966,
"elapsed_s": 308.1,
"scale_mean": 0.9991,
"scale_min": 0.5181,
"scale_max": 1.1437,
"n_scale_applied": 2156
},
{
"config": "3_C_hard",
"T": 2155,
"ROI": 181.0274,
"DD": 23.6916,
"Calmar": 7.641,
"elapsed_s": 304.1,
"scale_mean": 0.9954,
"scale_min": 0.5,
"scale_max": 1.15,
"n_scale_applied": 2156
},
{
"config": "4_D_analogue",
"T": 2155,
"ROI": 181.2387,
"DD": 23.6916,
"Calmar": 7.6499,
"elapsed_s": 307.7,
"scale_mean": 0.9989,
"scale_min": 0.5532,
"scale_max": 1.1458,
"n_scale_applied": 2156
},
{
"config": "5_AC_hard",
"T": 2155,
"ROI": 178.0085,
"DD": 23.6916,
"Calmar": 7.5136,
"elapsed_s": 308.0,
"scale_mean": 0.9931,
"scale_min": 0.25,
"scale_max": 1.3225,
"n_scale_applied": 2156
},
{
"config": "6_BD_analogue",
"T": 2155,
"ROI": 181.7092,
"DD": 23.6916,
"Calmar": 7.6698,
"elapsed_s": 309.2,
"scale_mean": 0.9981,
"scale_min": 0.3919,
"scale_max": 1.2704,
"n_scale_applied": 2156
}
]
}

371
nautilus_dolphin/dvae/exp11_zrecon_inv.py Executable file
View File

@@ -0,0 +1,371 @@
"""
exp11_zrecon_inv.py — z_recon direction inversion test
=======================================================
Exp10 showed z_recon now works (C: −0.78pp vs −8.65pp broken, D: −0.57pp vs −1.32pp)
but the direction is WRONG: the current setup boosts on HIGH z_recon (OOD bars) and
cuts on LOW z_recon (normal bars), but OOD bars at entry should be cut (not boosted).
This experiment tests the INVERTED direction: use −ze instead of ze for the recon signal.
D_inv: analogue_scale(−ze) → CUT on OOD (high z_recon), BOOST on quiet bars
BD_inv: B × D_inv → combine z_roll analogue with inverted z_recon
Configs (4 total):
0. Baseline — D_LIQ_GOLD unmodified (control)
1. B_analogue — z_roll analogue (positive control, same as exp10 config 2)
2. D_inv_analogue — z_recon analogue, INVERTED direction (−ze)
3. BD_inv — B_analogue × D_inv_analogue
Signal from exp10 that was POSITIVE: B_analogue (z_roll i150, +0.54pp)
Signal from exp10 that was NEGATIVE: D_analogue (z_recon, −0.57pp)
Hypothesis: D_inv (−ze) should be positive since OOD = bad entry condition.
No production code changes. Same model path, same data.
"""
import sys, time, json, warnings
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
warnings.filterwarnings('ignore')
from pathlib import Path
import numpy as np
import pandas as pd
ROOT = Path(__file__).parent.parent
sys.path.insert(0, str(ROOT))
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from nautilus_dolphin.nautilus.ob_features import (
OBFeatureEngine, compute_imbalance_nb, compute_depth_1pct_nb,
compute_depth_quality_nb, compute_fill_probability_nb, compute_spread_proxy_nb,
compute_depth_asymmetry_nb, compute_imbalance_persistence_nb,
compute_withdrawal_velocity_nb, compute_market_agreement_nb, compute_cascade_signal_nb,
)
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.proxy_boost_engine import LiquidationGuardEngine, create_d_liq_engine
from mc.mc_ml import DolphinForewarner
from dvae.titan_sensor import TitanSensor, build_feature_vector
# ── JIT warmup ────────────────────────────────────────────────────────────────
print("Warming up JIT...")
_p = np.array([1.,2.,3.], dtype=np.float64)
compute_irp_nb(_p,-1); compute_ars_nb(1.,.5,.01)
rank_assets_irp_nb(np.ones((10,2),dtype=np.float64),8,-1,5,500.,20,0.20)
compute_sizing_nb(-.03,-.02,-.05,3.,.5,5.,.20,True,True,0.,
np.zeros(4,dtype=np.int64),np.zeros(4,dtype=np.int64),
np.zeros(5,dtype=np.float64),0,-1,.01,.04)
check_dc_nb(_p,3,1,.75)
_b=np.array([100.,200.,300.,400.,500.],dtype=np.float64)
_a=np.array([110.,190.,310.,390.,510.],dtype=np.float64)
compute_imbalance_nb(_b,_a); compute_depth_1pct_nb(_b,_a)
compute_depth_quality_nb(210.,200.); compute_fill_probability_nb(1.)
compute_spread_proxy_nb(_b,_a); compute_depth_asymmetry_nb(_b,_a)
compute_imbalance_persistence_nb(np.array([.1,-.1],dtype=np.float64),2)
compute_withdrawal_velocity_nb(np.array([100.,110.],dtype=np.float64),1)
compute_market_agreement_nb(np.array([.1,-.05],dtype=np.float64),2)
compute_cascade_signal_nb(np.array([-.05,-.15],dtype=np.float64),2,-.10)
print(" JIT ready.")
# ── Paths ─────────────────────────────────────────────────────────────────────
VBT5s = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
VBT1m = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines")
MODEL_PATH = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\dvae_regime_model_TITAN_ULTRA_GD.json")
MC_MODELS = str(ROOT / "mc_results" / "models")
OUT_FILE = Path(__file__).parent / "exp11_zrecon_inv_results.json"
META_COLS = {'timestamp','scan_number','v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity','vel_div',
'instability_50','instability_150'}
BASE_ENGINE_KWARGS = dict(
initial_capital=25000., vel_div_threshold=-.02, vel_div_extreme=-.05,
min_leverage=.5, max_leverage=5., leverage_convexity=3.,
fraction=.20, fixed_tp_pct=.0095, stop_pct=1., max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=.75,
dc_skip_contradicts=True, dc_leverage_boost=1., dc_leverage_reduce=.5,
use_asset_selection=True, min_irp_alignment=.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=.62, sp_maker_exit_rate=.50,
use_ob_edge=True, ob_edge_bps=5., ob_confirm_rate=.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
D_LIQ_KWARGS = dict(
extended_soft_cap=8., extended_abs_cap=9., mc_leverage_ref=5.,
margin_buffer=.95, threshold=.35, alpha=1., adaptive_beta=True,
)
MC_BASE_CFG = {
'trial_id':0, 'vel_div_threshold':-.020, 'vel_div_extreme':-.050,
'use_direction_confirm':True, 'dc_lookback_bars':7, 'dc_min_magnitude_bps':.75,
'dc_skip_contradicts':True, 'dc_leverage_boost':1.00, 'dc_leverage_reduce':.50,
'vd_trend_lookback':10, 'min_leverage':.50, 'max_leverage':5.00,
'leverage_convexity':3.00, 'fraction':.20, 'use_alpha_layers':True,
'use_dynamic_leverage':True, 'fixed_tp_pct':.0095, 'stop_pct':1.00,
'max_hold_bars':120, 'use_sp_fees':True, 'use_sp_slippage':True,
'sp_maker_entry_rate':.62, 'sp_maker_exit_rate':.50, 'use_ob_edge':True,
'ob_edge_bps':5.00, 'ob_confirm_rate':.40, 'ob_imbalance_bias':-.09,
'ob_depth_scale':1.00, 'use_asset_selection':True, 'min_irp_alignment':.45,
'lookback':100, 'acb_beta_high':.80, 'acb_beta_low':.20, 'acb_w750_threshold_pct':60,
}
WINDOW_Z = 30
K_TANH = 1.5
UP_STR = 0.15
DOWN_STR = 0.50
def analogue_scale(z: float) -> float:
t = np.tanh(float(z) / K_TANH)
if z >= 0.:
return 1.0 + UP_STR * t
else:
return 1.0 + DOWN_STR * t
# ── Pre-compute 1m signals (same as exp10) ─────────────────────────────────
def precompute_1m_signals(parquet_files_5s, sensor):
print("Pre-computing 1m signals...")
signals = {}
for pf5 in parquet_files_5s:
ds = pf5.stem
pf1 = VBT1m / f"{ds}.parquet"
if not pf1.exists():
signals[ds] = None
continue
df5 = pd.read_parquet(pf5)
df1 = pd.read_parquet(pf1).replace([np.inf,-np.inf], np.nan).fillna(0.)
n5, n1 = len(df5), len(df1)
assets = [c for c in df1.columns if c not in META_COLS]
i150 = df1['instability_150'].values.copy() if 'instability_150' in df1.columns else np.zeros(n1)
z_roll_1m = np.zeros(n1)
for j in range(WINDOW_Z, n1):
seg = i150[max(0,j-WINDOW_Z):j]
mu, sigma = np.mean(seg), np.std(seg)
z_roll_1m[j] = (i150[j] - mu) / max(sigma, 1e-10)
recon_1m = np.zeros(n1)
for j in range(n1):
feat = build_feature_vector(df1, j, assets)
_, recon_err, _ = sensor.encode(feat)
recon_1m[j] = recon_err
z_recon_1m = np.zeros(n1)
for j in range(WINDOW_Z, n1):
seg = recon_1m[max(0,j-WINDOW_Z):j]
mu, sigma = np.mean(seg), np.std(seg)
z_recon_1m[j] = (recon_1m[j] - mu) / max(sigma, 1e-10)
z_roll_5s = np.zeros(n5)
z_recon_5s = np.zeros(n5)
for i in range(n5):
j = min(int(i * n1 / n5), n1-1)
z_roll_5s[i] = z_roll_1m[j]
z_recon_5s[i] = z_recon_1m[j]
signals[ds] = {'z_roll': z_roll_5s, 'z_recon': z_recon_5s}
print(f" {ds}: z_roll=[{z_roll_5s.min():.2f},{z_roll_5s.max():.2f}] "
f"z_recon=[{z_recon_5s.min():.2f},{z_recon_5s.max():.2f}]")
return signals
# ── Engine subclass (same as exp10) ──────────────────────────────────────────
class KeyframeGateEngine(LiquidationGuardEngine):
def __init__(self, scale_fn, **kwargs):
super().__init__(**kwargs)
self._scale_fn = scale_fn
self._bar_z_roll = None
self._bar_z_recon = None
self._1m_scale_history = []
def set_1m_signals(self, z_roll, z_recon):
self._bar_z_roll = z_roll
self._bar_z_recon = z_recon
def _try_entry(self, bar_idx, vel_div, prices, price_histories, v50_vel=0., v750_vel=0.):
result = super()._try_entry(bar_idx, vel_div, prices, price_histories, v50_vel, v750_vel)
if result and self.position is not None:
zr = float(self._bar_z_roll[bar_idx]) if (self._bar_z_roll is not None and bar_idx < len(self._bar_z_roll)) else 0.
ze = float(self._bar_z_recon[bar_idx]) if (self._bar_z_recon is not None and bar_idx < len(self._bar_z_recon)) else 0.
s = float(self._scale_fn(zr, ze))
s = max(0.2, min(2.0, s))
self.position.notional *= s
self._1m_scale_history.append(s)
return result
def reset(self):
super().reset()
self._1m_scale_history = []
# ── Configs: focus on inverted z_recon direction ──────────────────────────────
CONFIGS = {
"0_baseline": None,
"1_B_analogue": lambda zr, ze: analogue_scale(zr), # positive control
"2_D_inv_analogue":lambda zr, ze: analogue_scale(-ze), # INVERTED z_recon
"3_BD_inv": lambda zr, ze: analogue_scale(zr) * analogue_scale(-ze), # B × D_inv
}
def run_one(config_name, scale_fn, parquet_files, pq_data, signals, vol_p60):
OB_ASSETS = sorted({a for ds,(df,ac,_) in pq_data.items() for a in ac})
_mock_ob = MockOBProvider(
imbalance_bias=-.09, depth_scale=1., assets=OB_ASSETS,
imbalance_biases={"BTCUSDT":-.086,"ETHUSDT":-.092,"BNBUSDT":+.05,"SOLUSDT":+.05},
)
ob_eng = OBFeatureEngine(_mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
forewarner = DolphinForewarner(models_dir=MC_MODELS)
acb = AdaptiveCircuitBreaker()
acb.preload_w750([pf.stem for pf in parquet_files])
if scale_fn is None:
engine = create_d_liq_engine(**BASE_ENGINE_KWARGS)
else:
engine = KeyframeGateEngine(scale_fn=scale_fn, **BASE_ENGINE_KWARGS, **D_LIQ_KWARGS)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.)
t0 = time.time()
for pf in parquet_files:
ds = pf.stem
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
if scale_fn is not None and isinstance(engine, KeyframeGateEngine):
sig = signals.get(ds)
if sig:
engine.set_1m_signals(sig['z_roll'], sig['z_recon'])
else:
engine.set_1m_signals(np.zeros(len(df)), np.zeros(len(df)))
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
elapsed = time.time() - t0
trades = engine.trade_history
roi = (engine.capital - 25000.) / 25000. * 100.
cap_curve = [25000.]
for t_ in sorted(trades, key=lambda x: getattr(x,'exit_bar',0)):
cap_curve.append(cap_curve[-1] + getattr(t_,'pnl_absolute',0.))
cap_arr = np.array(cap_curve)
peak = np.maximum.accumulate(cap_arr)
dd = float(np.max((peak - cap_arr) / (peak + 1e-10)) * 100.)
calmar = roi / max(dd, 1e-4)
scale_hist = getattr(engine, '_1m_scale_history', [])
return {
'config': config_name,
'T': len(trades),
'ROI': round(roi, 4),
'DD': round(dd, 4),
'Calmar': round(calmar, 4),
'elapsed_s': round(elapsed, 1),
'scale_mean': round(np.mean(scale_hist), 4) if scale_hist else 1.0,
'scale_min': round(np.min(scale_hist), 4) if scale_hist else 1.0,
'scale_max': round(np.max(scale_hist), 4) if scale_hist else 1.0,
'n_scale_applied': len(scale_hist),
}
def main():
parquet_files = sorted(VBT5s.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
print(f"Dataset: {len(parquet_files)} days")
print("Loading parquet data...")
pq_data = {}
for pf in parquet_files:
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0,i-50):i]
if len(seg) >= 10:
dv[i] = float(np.std(np.diff(seg)/seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
all_vols = []
for pf in parquet_files[:2]:
df = pd.read_parquet(pf)
if 'BTCUSDT' not in df.columns: continue
pr = df['BTCUSDT'].values
for i in range(60, len(pr)):
seg = pr[max(0,i-50):i]
if len(seg) >= 10:
v = float(np.std(np.diff(seg)/seg[:-1]))
if v > 0: all_vols.append(v)
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.
print(f"\nLoading TitanSensor (GD-v2 with normalization)...")
sensor = TitanSensor(str(MODEL_PATH))
print(f" lstm_weights_valid={sensor.lstm_weights_valid} "
f"norm_mean is {'present' if sensor.norm_mean is not None else 'MISSING'}")
signals = precompute_1m_signals(parquet_files, sensor)
n_missing = sum(1 for v in signals.values() if v is None)
print(f" 1m signals ready: {len(signals)-n_missing}/{len(signals)} days")
print()
results = []
for name, fn in CONFIGS.items():
print(f"Running [{name}]...", flush=True)
r = run_one(name, fn, parquet_files, pq_data, signals, vol_p60)
results.append(r)
baseline_roi = results[0]['ROI']
delta_roi = r['ROI'] - baseline_roi
delta_dd = r['DD'] - results[0]['DD'] if len(results) > 1 else 0.
print(f" T={r['T']} ROI={r['ROI']:+.2f}% DD={r['DD']:.2f}% "
f"Calmar={r['Calmar']:.2f} "
f"(dROI={delta_roi:+.2f}pp dDD={delta_dd:+.2f}pp) "
f"scale_mean={r['scale_mean']:.3f} {r['elapsed_s']:.0f}s")
print()
print("=" * 90)
print(f"{'Config':<22} {'T':>5} {'ROI%':>8} {'DD%':>7} {'Calmar':>7} "
f"{'dROI':>7} {'dDD':>6} {'s_mean':>7}")
print("-" * 90)
base = results[0]
for r in results:
dr = r['ROI'] - base['ROI']
dd = r['DD'] - base['DD']
print(f"{r['config']:<22} {r['T']:>5} {r['ROI']:>8.2f} {r['DD']:>7.2f} "
f"{r['Calmar']:>7.2f} {dr:>+7.2f} {dd:>+6.2f} {r['scale_mean']:>7.3f}")
with open(OUT_FILE, 'w') as f:
json.dump({'baseline': base, 'results': results}, f, indent=2)
print(f"\nResults: {OUT_FILE}")
print("\n=== VERDICT ===")
best = max(results[1:], key=lambda x: x['Calmar'])
threshold = base['Calmar'] * 1.02
print(f"Best config: [{best['config']}] Calmar={best['Calmar']:.2f} "
f"ROI={best['ROI']:+.2f}% DD={best['DD']:.2f}%")
print(f"Threshold: Calmar > {threshold:.2f} (1.02× baseline {base['Calmar']:.2f})")
if best['Calmar'] > threshold:
print(" PROCEED: meaningful Calmar improvement vs baseline")
else:
print(" MARGINAL or NO improvement over D_LIQ_GOLD")
# Direction analysis
print("\n=== DIRECTION ANALYSIS ===")
d_fwd = next((r for r in results if '4_D_analogue' in r['config']), None)
d_inv = next((r for r in results if 'D_inv' in r['config']), None)
if d_inv:
print(f" D_inv_analogue (−ze): ROI={d_inv['ROI']:+.2f}% "
f"Calmar={d_inv['Calmar']:.2f} dROI={d_inv['ROI']-base['ROI']:+.2f}pp")
print(f" vs exp10 D_analogue (+ze): approx dROI=−0.57pp")
if d_inv['ROI'] - base['ROI'] > 0:
print(" CONFIRMED: inverted z_recon direction is POSITIVE — OOD = cut is right")
else:
print(" INCONCLUSIVE: inverted direction still negative")
if __name__ == '__main__':
main()

64
nautilus_dolphin/dvae/exp11_zrecon_inv_results.json Executable file
View File

@@ -0,0 +1,64 @@
{
"baseline": {
"config": "0_baseline",
"T": 2155,
"ROI": 181.8069,
"DD": 23.6916,
"Calmar": 7.6739,
"elapsed_s": 338.6,
"scale_mean": 1.0,
"scale_min": 1.0,
"scale_max": 1.0,
"n_scale_applied": 0
},
"results": [
{
"config": "0_baseline",
"T": 2155,
"ROI": 181.8069,
"DD": 23.6916,
"Calmar": 7.6739,
"elapsed_s": 338.6,
"scale_mean": 1.0,
"scale_min": 1.0,
"scale_max": 1.0,
"n_scale_applied": 0
},
{
"config": "1_B_analogue",
"T": 2155,
"ROI": 182.3446,
"DD": 23.6916,
"Calmar": 7.6966,
"elapsed_s": 318.3,
"scale_mean": 0.9991,
"scale_min": 0.5181,
"scale_max": 1.1437,
"n_scale_applied": 2156
},
{
"config": "2_D_inv_analogue",
"T": 2155,
"ROI": 180.6862,
"DD": 23.6916,
"Calmar": 7.6266,
"elapsed_s": 311.9,
"scale_mean": 0.9978,
"scale_min": 0.5139,
"scale_max": 1.134,
"n_scale_applied": 2156
},
{
"config": "3_BD_inv",
"T": 2155,
"ROI": 181.1797,
"DD": 23.6916,
"Calmar": 7.6474,
"elapsed_s": 313.8,
"scale_mean": 0.997,
"scale_min": 0.3503,
"scale_max": 1.2128,
"n_scale_applied": 2156
}
]
}

347
nautilus_dolphin/dvae/exp12_convnext_gate.py Executable file
View File

@@ -0,0 +1,347 @@
"""
exp12_convnext_gate.py — ConvNeXt z-signal gate on D_LIQ_GOLD
==============================================================
exp10/11 used TitanSensor with broken LSTM weights → z_recon ~10^14 → noise.
This experiment uses the newly trained ConvNeXt-1D β-TCVAE (ep=17, val=19.26):
z[10] r=+0.973 with proxy_B (32-bar trajectory encoding)
z_post_std OOD indicator (>1 = uncertain/unusual regime)
The z[10] signal captures the 32-bar TRAJECTORY of proxy_B, whereas the current
D_LIQ_GOLD engine uses only the instantaneous proxy_B at entry. If trajectory
adds information, z[10] should produce a measurable Calmar improvement.
Configs (5 runs):
0. baseline — D_LIQ_GOLD unmodified (control)
1. z10_analogue — analogue_scale(z10) → boost high proxy_B, cut low
2. z10_inv — analogue_scale(-z10) → inverse direction test
3. zstd_gate — notional × max(0.4, 1 - z_post_std/4) OOD cut
4. z10_x_zstd — z10_analogue × zstd_gate combined
analogue_scale (same as exp10/11):
z >= 0 : 1 + UP_STR * tanh(z / K) ceiling ~1.15
z < 0 : 1 + DOWN_STR * tanh(z / K) floor ~0.50
Threshold test: Calmar > baseline × 1.02 (same as exp10/11)
Gold baseline: ROI=181.81% DD=17.65% Calmar=10.30 (D_LIQ_GOLD memory)
"""
import sys, time, json, warnings
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
warnings.filterwarnings('ignore')
from pathlib import Path
import numpy as np
import pandas as pd
ROOT = Path(__file__).parent.parent
sys.path.insert(0, str(ROOT))
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from nautilus_dolphin.nautilus.ob_features import (
OBFeatureEngine, compute_imbalance_nb, compute_depth_1pct_nb,
compute_depth_quality_nb, compute_fill_probability_nb, compute_spread_proxy_nb,
compute_depth_asymmetry_nb, compute_imbalance_persistence_nb,
compute_withdrawal_velocity_nb, compute_market_agreement_nb, compute_cascade_signal_nb,
)
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.proxy_boost_engine import LiquidationGuardEngine, create_d_liq_engine
from mc.mc_ml import DolphinForewarner
from dvae.convnext_sensor import ConvNextSensor, PROXY_B_DIM
# ── JIT warmup ────────────────────────────────────────────────────────────────
print("Warming up JIT...")
_p = np.array([1.,2.,3.], dtype=np.float64)
compute_irp_nb(_p,-1); compute_ars_nb(1.,.5,.01)
rank_assets_irp_nb(np.ones((10,2),dtype=np.float64),8,-1,5,500.,20,0.20)
compute_sizing_nb(-.03,-.02,-.05,3.,.5,5.,.20,True,True,0.,
np.zeros(4,dtype=np.int64),np.zeros(4,dtype=np.int64),
np.zeros(5,dtype=np.float64),0,-1,.01,.04)
check_dc_nb(_p,3,1,.75)
_b=np.array([100.,200.,300.,400.,500.],dtype=np.float64)
_a=np.array([110.,190.,310.,390.,510.],dtype=np.float64)
compute_imbalance_nb(_b,_a); compute_depth_1pct_nb(_b,_a)
compute_depth_quality_nb(210.,200.); compute_fill_probability_nb(1.)
compute_spread_proxy_nb(_b,_a); compute_depth_asymmetry_nb(_b,_a)
compute_imbalance_persistence_nb(np.array([.1,-.1],dtype=np.float64),2)
compute_withdrawal_velocity_nb(np.array([100.,110.],dtype=np.float64),1)
compute_market_agreement_nb(np.array([.1,-.05],dtype=np.float64),2)
compute_cascade_signal_nb(np.array([-.05,-.15],dtype=np.float64),2,-.10)
print(" JIT ready.")
# ── Paths ─────────────────────────────────────────────────────────────────────
VBT5s = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
VBT1m = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines")
MODEL_PATH = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin\dvae\convnext_model.json")
MC_MODELS = str(ROOT / "mc_results" / "models")
OUT_FILE = Path(__file__).parent / "exp12_convnext_gate_results.json"
META_COLS = {'timestamp','scan_number','v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity','vel_div',
'instability_50','instability_150'}
BASE_ENGINE_KWARGS = dict(
initial_capital=25000., vel_div_threshold=-.02, vel_div_extreme=-.05,
min_leverage=.5, max_leverage=5., leverage_convexity=3.,
fraction=.20, fixed_tp_pct=.0095, stop_pct=1., max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=.75,
dc_skip_contradicts=True, dc_leverage_boost=1., dc_leverage_reduce=.5,
use_asset_selection=True, min_irp_alignment=.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=.62, sp_maker_exit_rate=.50,
use_ob_edge=True, ob_edge_bps=5., ob_confirm_rate=.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
D_LIQ_KWARGS = dict(
extended_soft_cap=8., extended_abs_cap=9., mc_leverage_ref=5.,
margin_buffer=.95, threshold=.35, alpha=1., adaptive_beta=True,
)
MC_BASE_CFG = {
'trial_id':0, 'vel_div_threshold':-.020, 'vel_div_extreme':-.050,
'use_direction_confirm':True, 'dc_lookback_bars':7, 'dc_min_magnitude_bps':.75,
'dc_skip_contradicts':True, 'dc_leverage_boost':1.00, 'dc_leverage_reduce':.50,
'vd_trend_lookback':10, 'min_leverage':.50, 'max_leverage':5.00,
'leverage_convexity':3.00, 'fraction':.20, 'use_alpha_layers':True,
'use_dynamic_leverage':True, 'fixed_tp_pct':.0095, 'stop_pct':1.00,
'max_hold_bars':120, 'use_sp_fees':True, 'use_sp_slippage':True,
'sp_maker_entry_rate':.62, 'sp_maker_exit_rate':.50, 'use_ob_edge':True,
'ob_edge_bps':5.00, 'ob_confirm_rate':.40, 'ob_imbalance_bias':-.09,
'ob_depth_scale':1.00, 'use_asset_selection':True, 'min_irp_alignment':.45,
'lookback':100, 'acb_beta_high':.80, 'acb_beta_low':.20, 'acb_w750_threshold_pct':60,
}
K_TANH = 1.5
UP_STR = 0.15
DOWN_STR = 0.50
def analogue_scale(z: float) -> float:
t = np.tanh(float(z) / K_TANH)
return 1.0 + (UP_STR if z >= 0. else DOWN_STR) * t
def zstd_scale(zstd: float) -> float:
"""OOD cut: normal zstd~0.94, high zstd = uncertain regime → cut notional."""
return float(max(0.4, 1.0 - (zstd - 0.94) / 4.0))
# ── Pre-compute signals per day ───────────────────────────────────────────────
def precompute_signals(parquet_files_5s, sensor: ConvNextSensor):
print("Pre-computing ConvNext z signals from 1m data...")
signals = {}
for pf5 in parquet_files_5s:
ds = pf5.stem
pf1 = VBT1m / f"{ds}.parquet"
if not pf1.exists():
signals[ds] = None
continue
df1 = pd.read_parquet(pf1).replace([np.inf, -np.inf], np.nan).fillna(0.)
n1 = len(df1)
n5 = len(pd.read_parquet(pf5, columns=['timestamp']))
z10_1m = np.zeros(n1, dtype=np.float64)
zstd_1m = np.zeros(n1, dtype=np.float64)
for j in range(n1):
z_mu, z_post_std = sensor.encode_window(df1, j)
z10_1m[j] = z_mu[PROXY_B_DIM]
zstd_1m[j] = z_post_std
# Map 1m → 5s by nearest index
z10_5s = np.array([z10_1m[min(int(i * n1 / n5), n1-1)] for i in range(n5)])
zstd_5s = np.array([zstd_1m[min(int(i * n1 / n5), n5-1)] for i in range(n5)])
signals[ds] = {'z10': z10_5s, 'zstd': zstd_5s}
print(f" {ds}: z10=[{z10_5s.min():.2f},{z10_5s.max():.2f}] "
f"zstd=[{zstd_5s.min():.3f},{zstd_5s.max():.3f}]")
n_ok = sum(1 for v in signals.values() if v is not None)
print(f" Signals ready: {n_ok}/{len(signals)} days\n")
return signals
# ── Engine subclass ───────────────────────────────────────────────────────────
class ConvNextGateEngine(LiquidationGuardEngine):
def __init__(self, scale_fn, **kwargs):
super().__init__(**kwargs)
self._scale_fn = scale_fn
self._bar_z10 = None
self._bar_zstd = None
self._scale_history = []
def set_signals(self, z10: np.ndarray, zstd: np.ndarray):
self._bar_z10 = z10
self._bar_zstd = zstd
def _try_entry(self, bar_idx, vel_div, prices, price_histories, v50_vel=0., v750_vel=0.):
result = super()._try_entry(bar_idx, vel_div, prices, price_histories, v50_vel, v750_vel)
if result and self.position is not None:
z10 = float(self._bar_z10[bar_idx]) if (self._bar_z10 is not None and bar_idx < len(self._bar_z10)) else 0.
zstd = float(self._bar_zstd[bar_idx]) if (self._bar_zstd is not None and bar_idx < len(self._bar_zstd)) else 0.94
s = float(self._scale_fn(z10, zstd))
s = max(0.2, min(2.0, s))
self.position.notional *= s
self._scale_history.append(s)
return result
def reset(self):
super().reset()
self._scale_history = []
# ── Scale functions ───────────────────────────────────────────────────────────
CONFIGS = {
"0_baseline": None,
"1_z10_analogue": lambda z10, zstd: analogue_scale(z10),
"2_z10_inv": lambda z10, zstd: analogue_scale(-z10),
"3_zstd_gate": lambda z10, zstd: zstd_scale(zstd),
"4_z10_x_zstd": lambda z10, zstd: analogue_scale(z10) * zstd_scale(zstd),
}
def run_one(config_name, scale_fn, parquet_files, pq_data, signals, vol_p60):
OB_ASSETS = sorted({a for ds, (df, ac, _) in pq_data.items() for a in ac})
_mock_ob = MockOBProvider(
imbalance_bias=-.09, depth_scale=1., assets=OB_ASSETS,
imbalance_biases={"BTCUSDT":-.086,"ETHUSDT":-.092,"BNBUSDT":+.05,"SOLUSDT":+.05},
)
ob_eng = OBFeatureEngine(_mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
forewarner = DolphinForewarner(models_dir=MC_MODELS)
acb = AdaptiveCircuitBreaker()
acb.preload_w750([pf.stem for pf in parquet_files])
if scale_fn is None:
engine = create_d_liq_engine(**BASE_ENGINE_KWARGS)
else:
engine = ConvNextGateEngine(scale_fn=scale_fn, **BASE_ENGINE_KWARGS, **D_LIQ_KWARGS)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.)
t0 = time.time()
for pf in parquet_files:
ds = pf.stem
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
if scale_fn is not None and isinstance(engine, ConvNextGateEngine):
sig = signals.get(ds)
if sig:
engine.set_signals(sig['z10'], sig['zstd'])
else:
engine.set_signals(np.zeros(len(df)), np.full(len(df), 0.94))
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
elapsed = time.time() - t0
trades = engine.trade_history
roi = (engine.capital - 25000.) / 25000. * 100.
cap_curve = [25000.]
for t_ in sorted(trades, key=lambda x: getattr(x, 'exit_bar', 0)):
cap_curve.append(cap_curve[-1] + getattr(t_, 'pnl_absolute', 0.))
cap_arr = np.array(cap_curve)
peak = np.maximum.accumulate(cap_arr)
dd = float(np.max((peak - cap_arr) / (peak + 1e-10)) * 100.)
calmar = roi / max(dd, 1e-4)
sh = getattr(engine, '_scale_history', [])
return {
'config': config_name,
'T': len(trades),
'ROI': round(roi, 4),
'DD': round(dd, 4),
'Calmar': round(calmar, 4),
'elapsed_s': round(elapsed, 1),
'scale_mean': round(float(np.mean(sh)), 4) if sh else 1.0,
'scale_min': round(float(np.min(sh)), 4) if sh else 1.0,
'scale_max': round(float(np.max(sh)), 4) if sh else 1.0,
'n_scaled': len(sh),
}
def main():
parquet_files = sorted(VBT5s.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
print(f"Dataset: {len(parquet_files)} days (5s scans)\n")
print("Loading ConvNextSensor...")
sensor = ConvNextSensor(str(MODEL_PATH))
print(f" epoch={sensor.epoch} val_loss={sensor.val_loss:.4f} z_dim={sensor.z_dim}\n")
signals = precompute_signals(parquet_files, sensor)
print("Loading 5s parquet data...")
pq_data = {}
for pf in parquet_files:
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0, i-50):i]
if len(seg) >= 10:
dv[i] = float(np.std(np.diff(seg) / seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
all_vols = []
for pf in parquet_files[:2]:
df = pd.read_parquet(pf)
if 'BTCUSDT' not in df.columns: continue
pr = df['BTCUSDT'].values
for i in range(60, len(pr)):
seg = pr[max(0, i-50):i]
if len(seg) >= 10:
v = float(np.std(np.diff(seg) / seg[:-1]))
if v > 0: all_vols.append(v)
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.
print(f"\nRunning {len(CONFIGS)} configs...\n")
results = []
for name, fn in CONFIGS.items():
print(f"[{name}]", flush=True)
r = run_one(name, fn, parquet_files, pq_data, signals, vol_p60)
results.append(r)
base = results[0]
dr = r['ROI'] - base['ROI']
dd = r['DD'] - base['DD']
print(f" T={r['T']} ROI={r['ROI']:+.2f}% DD={r['DD']:.2f}% "
f"Calmar={r['Calmar']:.2f} dROI={dr:+.2f}pp dDD={dd:+.2f}pp "
f"s_mean={r['scale_mean']:.3f} ({r['elapsed_s']:.0f}s)\n")
print("=" * 95)
print(f"{'Config':<22} {'T':>5} {'ROI%':>8} {'DD%':>7} {'Calmar':>7} "
f"{'dROI':>7} {'dDD':>6} {'s_mean':>7}")
print("-" * 95)
base = results[0]
for r in results:
dr = r['ROI'] - base['ROI']
dd = r['DD'] - base['DD']
print(f"{r['config']:<22} {r['T']:>5} {r['ROI']:>8.2f} {r['DD']:>7.2f} "
f"{r['Calmar']:>7.2f} {dr:>+7.2f} {dd:>+6.2f} {r['scale_mean']:>7.3f}")
with open(OUT_FILE, 'w') as f:
json.dump({'baseline': base, 'results': results,
'model_epoch': sensor.epoch, 'model_val_loss': sensor.val_loss}, f, indent=2)
print(f"\nResults -> {OUT_FILE}")
print("\n=== VERDICT ===")
best = max(results[1:], key=lambda x: x['Calmar'])
threshold = base['Calmar'] * 1.02
print(f"Best: [{best['config']}] Calmar={best['Calmar']:.2f} "
f"ROI={best['ROI']:.2f}% DD={best['DD']:.2f}%")
print(f"Threshold: Calmar > {threshold:.2f} (1.02x baseline {base['Calmar']:.2f})")
if best['Calmar'] > threshold:
print(" SIGNAL CONFIRMED — proceed to exp13 (productionization)")
else:
print(" MARGINAL / NO improvement over D_LIQ_GOLD")
if __name__ == '__main__':
main()

78
nautilus_dolphin/dvae/exp12_convnext_gate_results.json Executable file
View File

@@ -0,0 +1,78 @@
{
"baseline": {
"config": "0_baseline",
"T": 2155,
"ROI": 181.8069,
"DD": 23.6916,
"Calmar": 7.6739,
"elapsed_s": 411.0,
"scale_mean": 1.0883,
"scale_min": 1.0,
"scale_max": 1.63,
"n_scaled": 2156
},
"results": [
{
"config": "0_baseline",
"T": 2155,
"ROI": 181.8069,
"DD": 23.6916,
"Calmar": 7.6739,
"elapsed_s": 411.0,
"scale_mean": 1.0883,
"scale_min": 1.0,
"scale_max": 1.63,
"n_scaled": 2156
},
{
"config": "1_z10_analogue",
"T": 2155,
"ROI": 177.1364,
"DD": 23.6916,
"Calmar": 7.4768,
"elapsed_s": 429.5,
"scale_mean": 1.041,
"scale_min": 0.6597,
"scale_max": 1.63,
"n_scaled": 4312
},
{
"config": "2_z10_inv",
"T": 2155,
"ROI": 183.2151,
"DD": 23.6916,
"Calmar": 7.7333,
"elapsed_s": 517.9,
"scale_mean": 1.0451,
"scale_min": 1.0,
"scale_max": 1.63,
"n_scaled": 4312
},
{
"config": "3_zstd_gate",
"T": 2155,
"ROI": 181.7825,
"DD": 23.6916,
"Calmar": 7.6729,
"elapsed_s": 489.5,
"scale_mean": 1.0442,
"scale_min": 0.9955,
"scale_max": 1.63,
"n_scaled": 4312
},
{
"config": "4_z10_x_zstd",
"T": 2155,
"ROI": 177.1174,
"DD": 23.6916,
"Calmar": 7.476,
"elapsed_s": 504.2,
"scale_mean": 1.041,
"scale_min": 0.6568,
"scale_max": 1.63,
"n_scaled": 4312
}
],
"model_epoch": 17,
"model_val_loss": 19.260574247143158
}

363
nautilus_dolphin/dvae/exp13_model_sweep.py Executable file
View File

@@ -0,0 +1,363 @@
"""
exp13_model_sweep.py — Multi-model exp13 test harness.
For each model in the registry:
1. Auto-identify proxy_B dim (highest |r| correlation with raw proxy_B signal)
2. Validate calibration (always-positive in 56-day window required for exp13 scaling)
3. Run exp13 Phase 1 (14-day screening) + Phase 2 (full 56-day, top-k configs)
4. Save per-model results to exp13_sweep_<tag>_results.json
5. Print final comparison table across all models
All tests use IDENTICAL configs/window/threshold to exp13 v2 (the CONFIRMED baseline).
Threshold: Calmar > 7.83 (102% of D_LIQ_GOLD baseline 7.67 in the 56-day window)
Reference: v2 BOB — 9/20 PASS, best dROI=+4.59pp, best Calmar=7.87
Usage (from nautilus_dolphin/ dir):
python dvae/exp13_model_sweep.py # all available models in registry
python dvae/exp13_model_sweep.py --models v4 # single model
python dvae/exp13_model_sweep.py --models v4 v5 v6 # explicit list
python dvae/exp13_model_sweep.py --probe_only # dim probe only, no backtest
python dvae/exp13_model_sweep.py --subset 14 --top_k 20 # explicit Phase 1/2 params
Adding new models (v5, v6, v7, v8):
1. Transfer model JSON from DOLPHIN Linux to models/convnext_dvae_ML/
2. Uncomment (or add) the entry in MODEL_REGISTRY below
3. Re-run this script
"""
import sys, os, time, json, importlib, argparse
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace', line_buffering=True)
import numpy as np
import pandas as pd
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent.parent
DVAE_DIR = ROOT / 'nautilus_dolphin' / 'dvae'
sys.path.insert(0, str(ROOT / 'nautilus_dolphin'))
# ── Model registry ─────────────────────────────────────────────────────────────
# Slot in v5/v6/v7/v8 when transferred from DOLPHIN Linux — just uncomment.
MODEL_REGISTRY = {
'v2_bob': ROOT / 'nautilus_dolphin' / 'dvae' / 'convnext_model_v2.json',
'v4': ROOT / 'models' / 'convnext_dvae_ML' / 'convnext_model_v4_ep22_best.json',
'v5': ROOT / 'models' / 'dolphin_training' / 'winning_models' / 'v5_ep28_best_total_loss.json',
'v6': ROOT / 'models' / 'dolphin_training' / 'winning_models' / 'v6_ep8_best_val_loss.json',
'v7': ROOT / 'models' / 'dolphin_training' / 'winning_models' / 'v7_ep10_best_generalization.json',
# v8: step=2 training (1.2M windows), only 2 epochs — val=34.92. Experimental.
'v8': ROOT / 'models' / 'dolphin_training' / 'dvae' / 'v8_step2.json',
# v6.5 (ANOMALOUS — DO NOT USE: broken during training per researcher note)
}
KLINES_DIR = ROOT / 'vbt_cache_klines'
DATE_START = '2025-12-31'
DATE_END = '2026-02-25'
CALMAR_THR = 7.83 # 102% of D_LIQ_GOLD baseline 7.67
FEATURE_COLS = [
'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity',
'vel_div', 'instability_50', 'instability_150',
]
T_WIN = 32
# ── Proxy_B dim identification ─────────────────────────────────────────────────
from dvae.convnext_dvae import ConvNeXtVAE
def _load_model(path: Path):
with open(path) as f:
meta = json.load(f)
arch = meta['architecture']
m = ConvNeXtVAE(
C_in=arch['C_in'], T_in=arch['T_in'],
z_dim=arch['z_dim'], base_ch=arch['base_ch'],
n_blocks=arch.get('n_blocks', 3), seed=42,
)
m.load(str(path))
nm = np.array(meta['norm_mean']) if 'norm_mean' in meta else None
ns = np.array(meta['norm_std']) if 'norm_std' in meta else None
return m, nm, ns, meta
def _build_probe_set():
"""Sample probe windows from 56-day window; shared across all models."""
files = sorted(KLINES_DIR.glob('*.parquet'))
period = [f for f in files if DATE_START <= f.stem[:10] <= DATE_END]
rng = np.random.default_rng(42)
probes_raw, proxy_B_vals = [], []
step = max(1, len(period) // 60)
for f in period[::step]:
try:
df = pd.read_parquet(f, columns=FEATURE_COLS).dropna()
if len(df) < T_WIN + 10: continue
mid = len(df) // 2
pos = int(rng.integers(max(0, mid-30), min(len(df)-T_WIN, mid+30)))
arr = df[FEATURE_COLS].values[pos:pos+T_WIN].astype(np.float64)
proxy_B = (arr[:, 5] - arr[:, 3]).reshape(-1, 1)
exf = np.zeros((T_WIN, 3), dtype=np.float64)
arr11 = np.concatenate([arr, proxy_B, exf], axis=1).T # (11, T)
if not np.isfinite(arr11).all(): continue
probes_raw.append(arr11)
proxy_B_vals.append(float(proxy_B.mean()))
except Exception:
pass
return np.stack(probes_raw), np.array(proxy_B_vals)
def probe_model(tag: str, path: Path, probes_raw: np.ndarray,
proxy_B_arr: np.ndarray) -> dict:
"""Identify proxy_B dim and report calibration for one model."""
print(f"\n{'='*60}")
print(f" PROBE: {tag} ({path.name})")
print(f"{'='*60}")
model, nm, ns, meta = _load_model(path)
ep = meta.get('epoch', '?')
val = meta.get('val_loss', 0.0)
print(f" epoch={ep} val_loss={val:.5f}")
probes = probes_raw.copy()
if nm is not None:
probes = (probes - nm[None, :, None]) / ns[None, :, None]
np.clip(probes, -6., 6., out=probes)
z_mu, z_logvar = model.encode(probes)
z_std = z_mu.std(0)
corrs = []
for d in range(z_mu.shape[1]):
if z_std[d] > 0.01:
r = float(np.corrcoef(z_mu[:, d], proxy_B_arr)[0, 1])
if np.isfinite(r):
corrs.append((abs(r), r, d))
corrs.sort(reverse=True)
best_abs_r, best_r, best_dim = corrs[0] if corrs else (0.0, 0.0, -1)
z_best = z_mu[:, best_dim]
z_min, z_max = float(z_best.min()), float(z_best.max())
always_pos = z_min > 0
always_neg = z_max < 0
if always_pos: calib = 'ALWAYS_POSITIVE'
elif always_neg: calib = 'ALWAYS_NEGATIVE'
else: calib = f'MIXED[{z_min:+.3f},{z_max:+.3f}]'
q75, q25 = np.percentile(proxy_B_arr, 75), np.percentile(proxy_B_arr, 25)
z_hi = float(z_best[proxy_B_arr >= q75].mean())
z_lo = float(z_best[proxy_B_arr <= q25].mean())
sep = abs(z_hi - z_lo)
# Also find best POSITIVELY correlated dim (same sign as v2 z[13])
pos_corrs = [(abs_r, r, d) for abs_r, r, d in corrs if r > 0]
pos_dim = pos_corrs[0][2] if pos_corrs else best_dim
pos_r = pos_corrs[0][1] if pos_corrs else best_r
usable = always_pos and best_abs_r > 0.5
print(f" proxy_B dim : z[{best_dim}] r={best_r:+.4f} "
f"(best |r|) best positive: z[{pos_dim}] r={pos_r:+.4f}")
print(f" Top-5 : " + ' '.join(f'z[{d}]={r:+.3f}' for _,r,d in corrs[:5]))
print(f" Calibration : {calib} sep={sep:.4f}")
print(f" Usable : {'YES ✓' if usable else 'CAUTION ⚠ (will skip exp13 sweep)'}")
return {
'tag': tag, 'path': str(path),
'epoch': ep, 'val_loss': val,
'proxy_B_dim': best_dim, 'proxy_B_r': best_r,
'proxy_B_dim_pos': pos_dim, 'proxy_B_r_pos': pos_r,
'calibration': calib, 'always_positive': always_pos,
'separation': sep, 'top5': [(r,d) for _,r,d in corrs[:5]],
}
# ── exp13 runner per model ─────────────────────────────────────────────────────
def run_exp13_for_model(probe: dict, subset_days: int, top_k: int,
only_config: str = None) -> dict:
"""Patch MODEL_1M + PROXY_B_DIM into exp13, call main(), read results JSON."""
import dvae.exp13_multiscale_sweep as e13
importlib.reload(e13) # fresh state: clears cached signals from prior model
# Patch module-level constants AFTER reload
e13.MODEL_1M = Path(probe['path'])
e13.PROXY_B_DIM = probe['proxy_B_dim']
out_file = ROOT / f"exp13_sweep_{probe['tag']}_results.json"
e13.OUT_FILE = out_file
# Patch sys.argv so argparse inside main() picks up our params
sys.argv = [
'exp13_multiscale_sweep.py',
'--subset', str(subset_days),
'--top_k', str(top_k),
'--skip_sets', 'B,Bp', # skip 5s sets (no 5s model per model variant)
]
if only_config:
sys.argv += ['--only_config', only_config, '--skip_5s']
print(f"\n{'━'*60}")
print(f" EXP13: {probe['tag']} z[{probe['proxy_B_dim']}] r={probe['proxy_B_r']:+.4f}")
print(f" subset={subset_days}d top_k={top_k} out={out_file.name}")
print(f"{'━'*60}")
t0 = time.time()
e13.main()
elapsed = time.time() - t0
# Read saved results
if not out_file.exists():
print(f"[ERROR] Results file not written: {out_file}")
return {'tag': probe['tag'], 'error': 'no results file'}
with open(out_file) as f:
raw = json.load(f)
baseline_full = raw.get('phase2', {}).get('baseline_full') or raw.get('phase1_results', [{}])[0]
p2_list = raw.get('phase2', {}).get('results', [])
if not p2_list:
# full run (subset=0): use phase1_results as p2
p2_list = raw.get('phase1_results', [])
baseline_cal = (raw.get('phase2', {}).get('baseline_full') or {}).get('Calmar', 0.0)
n_pass = sum(1 for r in p2_list if r.get('Calmar', 0) > CALMAR_THR)
best = max(p2_list, key=lambda r: r.get('Calmar', 0)) if p2_list else {}
base_roi = (raw.get('phase2', {}).get('baseline_full') or {}).get('ROI', 0.0)
return {
'tag': probe['tag'],
'val_loss': probe['val_loss'],
'proxy_B_dim': probe['proxy_B_dim'],
'proxy_B_r': probe['proxy_B_r'],
'calibration': probe['calibration'],
'baseline_calmar': baseline_cal,
'baseline_roi': base_roi,
'n_phase2': len(p2_list),
'n_pass': n_pass,
'best_config': best.get('name', '?'),
'best_roi': best.get('ROI', 0.0),
'best_calmar': best.get('Calmar', 0.0),
'best_droi': best.get('ROI', 0.0) - base_roi,
'best_ddd': best.get('DD', 0.0) - (raw.get('phase2', {}).get('baseline_full') or {}).get('DD', 0.0),
'elapsed_s': round(elapsed),
'results_file': str(out_file),
}
# ── Comparison tables ──────────────────────────────────────────────────────────
def _print_probe_table(probes: dict):
print(f"\n{'='*72}")
print(f" MODEL PROBE SUMMARY")
print(f"{'='*72}")
print(f" {'Tag':10s} {'ValLoss':>8s} {'Dim':>5s} {'r':>7s} {'Sep':>6s} {'Calibration':22s} OK?")
print(f" {'-'*72}")
for tag, p in probes.items():
ok = '✓' if p['always_positive'] and abs(p['proxy_B_r']) > 0.5 else '⚠'
print(f" {tag:10s} {p['val_loss']:8.4f} "
f"z[{p['proxy_B_dim']:2d}] {p['proxy_B_r']:+7.4f} "
f"{p['separation']:6.4f} {p['calibration']:22s} {ok}")
def _print_comparison_table(results: list):
if not results:
return
print(f"\n{'='*84}")
print(f" EXP13 MULTI-MODEL FINAL COMPARISON")
print(f" Threshold: Calmar > {CALMAR_THR} | Reference: v2_BOB — 9/20 PASS dROI=+4.59pp Cal=7.87")
print(f"{'='*84}")
print(f" {'Tag':10s} {'ValLoss':>8s} {'Dim':>5s} {'r':>7s} "
f"{'Pass':>6s} {'BestCal':>8s} {'BestdROI':>9s} {'BestdDD':>8s} Best Config")
print(f" {'-'*84}")
for r in sorted(results, key=lambda x: x.get('best_calmar', 0), reverse=True):
pass_str = f"{r['n_pass']:2d}/{r['n_phase2']}"
flag = '✓' if r['n_pass'] > 0 else ' '
print(f" {r['tag']:10s} {r['val_loss']:8.4f} "
f"z[{r['proxy_B_dim']:2d}] {r['proxy_B_r']:+7.4f} "
f"{pass_str:>6s} {flag} {r['best_calmar']:8.3f} "
f"{r['best_droi']:+9.2f}pp {r['best_ddd']:+8.2f}pp {r['best_config']}")
print(f" {'─'*84}")
print(f" {'v2_bob REF':10s} {'18.0024':>8s} z[13] {'+0.9332':>7s} "
f"{'9/20':>6s} ✓ {'7.870':>8s} {'+4.59':>9s}pp {'0.00':>8s}pp A_P5_M2_W1_a0.5")
# ── Main ───────────────────────────────────────────────────────────────────────
def main():
parser = argparse.ArgumentParser(description='Multi-model exp13 test harness')
parser.add_argument('--models', nargs='+', default=None,
help='Model tags to run (default: all available)')
parser.add_argument('--probe_only', action='store_true',
help='Dim probe only — skip exp13 sweep')
parser.add_argument('--subset', type=int, default=14,
help='Phase 1 days (default 14)')
parser.add_argument('--top_k', type=int, default=20,
help='Phase 2 top-k configs (default 20)')
parser.add_argument('--fast_check', type=str, default='',
help='Skip Phase 1; run just this config on full window. '
'Default known-winner: A_P5_M2_W1_a0.5')
args = parser.parse_args()
if args.fast_check == 'winner':
args.fast_check = 'A_P5_M2_W1_a0.5' # shorthand
# Select models
tags = args.models or list(MODEL_REGISTRY.keys())
available = {t: MODEL_REGISTRY[t] for t in tags
if t in MODEL_REGISTRY and Path(MODEL_REGISTRY[t]).exists()}
skipped = [t for t in tags if t not in MODEL_REGISTRY]
missing = [t for t in MODEL_REGISTRY if t not in available and
t in (args.models or list(MODEL_REGISTRY.keys()))]
if skipped: print(f"[WARN] Unknown tags: {skipped}")
if missing: print(f"[WARN] File not found (transfer from DOLPHIN Linux): {missing}")
if not available:
print("No model files found. Check MODEL_REGISTRY or --models flag."); return
print(f"\n{'━'*60}")
print(f" EXP13 MULTI-MODEL SWEEP")
print(f" Models : {list(available.keys())}")
print(f" Window : {DATE_START} → {DATE_END}")
print(f" Threshold: Calmar > {CALMAR_THR}")
print(f" Phase1 : {args.subset} days Phase2 top-k: {args.top_k}")
print(f"{'━'*60}")
# Build shared probe set
print(f"\nBuilding probe set ({DATE_START}→{DATE_END})...")
probes_raw, proxy_B_arr = _build_probe_set()
print(f" {len(probes_raw)} windows proxy_B: μ={proxy_B_arr.mean():+.4f} σ={proxy_B_arr.std():.4f}")
# Step 1: probe all models
probe_reports = {tag: probe_model(tag, path, probes_raw, proxy_B_arr)
for tag, path in available.items()}
_print_probe_table(probe_reports)
if args.probe_only:
return
# Step 2: run exp13 for each usable model
sweep_results = []
for tag, probe in probe_reports.items():
if not probe['always_positive']:
print(f"\n[SKIP] {tag}: calib={probe['calibration']} — not always-positive")
continue
try:
summary = run_exp13_for_model(probe, args.subset, args.top_k,
only_config=args.fast_check or None)
sweep_results.append(summary)
except Exception as ex:
import traceback
print(f"\n[ERROR] {tag}: {ex}")
traceback.print_exc()
_print_comparison_table(sweep_results)
# Save combined summary
out = ROOT / 'exp13_model_sweep_results.json'
with open(out, 'w') as f:
json.dump({
'probes': probe_reports,
'sweep': sweep_results,
'threshold': CALMAR_THR,
'window': {'start': DATE_START, 'end': DATE_END},
'ref_v2_bob': {'n_pass': 9, 'best_droi': 4.59, 'best_calmar': 7.87},
}, f, indent=2, default=str)
print(f"\nSummary → {out}")
if __name__ == '__main__':
main()

1116
nautilus_dolphin/dvae/exp13_multiscale_sweep.py Executable file
View File

File diff suppressed because it is too large Load Diff

4675
nautilus_dolphin/dvae/exp13_multiscale_sweep_results.json Executable file
View File

File diff suppressed because it is too large Load Diff

46
nautilus_dolphin/dvae/exp13_v2_launcher.py Executable file
View File

@@ -0,0 +1,46 @@
"""
exp13_v2_launcher.py — Run exp13 multiscale sweep with convnext_model_v2.json (ep=13).
Changes vs default exp13:
- MODEL_1M → convnext_model_v2.json (calibration-fixed, always-positive z[13])
- PROXY_B_DIM → 13 (z[13] is proxy_B dim in v2, vs z[10] in ep=17)
Everything else identical to exp13_multiscale_sweep.py.
Results logged to ../../exp13_v2_screening_run.log
Usage (from nautilus_dolphin/ dir):
python dvae/exp13_v2_launcher.py --subset 14 --top_k 20 # Phase 1 screening
python dvae/exp13_v2_launcher.py --subset 0 --top_k 0 # Full 56-day run
Why v2 should be better:
ep=17 z[10]: ALWAYS NEGATIVE [-1.24, -0.30] → direct scaling configs hurt
v2 z[13]: ALWAYS POSITIVE [+0.17, +1.46] → direct scaling configs work correctly
Separation (proxy_B quartiles): 0.46 (ep=17) → 0.61 (v2) — 32% improvement
"""
import sys, os
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent.parent
sys.path.insert(0, str(ROOT / 'nautilus_dolphin'))
# ── patch model path and proxy_B dim before importing main ───────────────────
MODEL_V2 = ROOT / 'nautilus_dolphin' / 'dvae' / 'convnext_model_v2.json'
PROXY_B_V2 = 13 # z[13] r=+0.9332 for v2 ep=13 (auto-confirmed by proto_v2_query.py)
assert MODEL_V2.exists(), f"v2 model not found: {MODEL_V2}"
import dvae.exp13_multiscale_sweep as e13
import dvae.convnext_sensor as cs
# Patch module-level constants before main() runs
e13.MODEL_1M = MODEL_V2
e13.PROXY_B_DIM = PROXY_B_V2
cs.PROXY_B_DIM = PROXY_B_V2 # sensor module also exports this
print(f"[v2-launcher] MODEL_1M → {MODEL_V2.name}")
print(f"[v2-launcher] PROXY_B_DIM → {PROXY_B_V2} (was 10)")
print(f"[v2-launcher] v2 ep=13 val=18.002 z[13] r=+0.933 calibration=ALWAYS_POSITIVE")
print()
if __name__ == '__main__':
e13.main()

684
nautilus_dolphin/dvae/exp14_sweep.py Executable file
View File

@@ -0,0 +1,684 @@
"""
exp14_sweep.py — z[13] / z_post_std / resonance-delta leverage gate sweep.
Tests three signal families against D_LIQ_GOLD baseline (56-day 5s scan data):
Family A — z[13] leverage gate:
When z[13] (proxy_B dim, always-positive in Dec-Jan 2026) exceeds a threshold,
cap the effective soft leverage below the D_LIQ 8x default.
High z[13] = high proxy_B context = expect adverse excursion → be smaller.
Family B — z_post_std OOD gate:
When z_post_std exceeds a threshold (market window is OOD / unusual),
cap leverage conservatively regardless of direction.
Family C — 2D combined gate (z[13] AND z_post_std):
Both signals gate simultaneously; take the min of their implied caps.
Family D — Resonance delta gate:
delta = live_proxy_B − implied_proxy_B(z[13]) [fitted linear map]
Three scenarios (see TODO.md exp14):
delta > thr → MORE turbulent than model expects → scale DOWN
delta < -thr → calmer than model expects → scale UP (carefully)
|delta| < resonance_thr → RESONANCE: two sensors agree → MAX CONFIDENCE
(disproportionately large weight: if both say danger → strong cap,
if both say calm → allow near-full leverage)
Family E — Combined best from A + B + D.
Baseline: D_LIQ_GOLD (soft=8x, hard=9x, mc_ref=5x, margin_buffer=0.95)
ROI=181.81% DD=17.65% Calmar=10.30 T=2155
Usage:
cd nautilus_dolphin/
python dvae/exp14_sweep.py --subset 14 --top_k 20 # Phase 1 (14-day screening)
python dvae/exp14_sweep.py --subset 0 --top_k 0 # Phase 2 (full 56 days)
"""
import sys, os, time, json, warnings, argparse
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent.parent
ND_ROOT = ROOT / 'nautilus_dolphin'
sys.path.insert(0, str(ND_ROOT))
from dvae.convnext_sensor import ConvNextSensor
from nautilus_dolphin.nautilus.proxy_boost_engine import (
LiquidationGuardEngine,
D_LIQ_SOFT_CAP, D_LIQ_ABS_CAP, D_LIQ_MC_REF, D_LIQ_MARGIN_BUF,
create_d_liq_engine,
)
from nautilus_dolphin.nautilus.ob_features import (
OBFeatureEngine, compute_imbalance_nb, compute_depth_1pct_nb,
compute_depth_quality_nb, compute_fill_probability_nb, compute_spread_proxy_nb,
compute_depth_asymmetry_nb, compute_imbalance_persistence_nb,
compute_withdrawal_velocity_nb, compute_market_agreement_nb, compute_cascade_signal_nb,
)
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from mc.mc_ml import DolphinForewarner
# ── JIT warmup ────────────────────────────────────────────────────────────────
print("Warming up JIT...")
_p = np.array([1., 2., 3.], dtype=np.float64)
compute_irp_nb(_p, -1); compute_ars_nb(1., .5, .01)
rank_assets_irp_nb(np.ones((10, 2), dtype=np.float64), 8, -1, 5, 500., 20, 0.20)
compute_sizing_nb(-.03, -.02, -.05, 3., .5, 5., .20, True, True, 0.,
np.zeros(4, dtype=np.int64), np.zeros(4, dtype=np.int64),
np.zeros(5, dtype=np.float64), 0, -1, .01, .04)
check_dc_nb(_p, 3, 1, .75)
_b = np.array([100., 200., 300., 400., 500.], dtype=np.float64)
_a = np.array([110., 190., 310., 390., 510.], dtype=np.float64)
compute_imbalance_nb(_b, _a); compute_depth_1pct_nb(_b, _a)
compute_depth_quality_nb(210., 200.); compute_fill_probability_nb(1.)
compute_spread_proxy_nb(_b, _a); compute_depth_asymmetry_nb(_b, _a)
compute_imbalance_persistence_nb(np.array([.1, -.1], dtype=np.float64), 2)
compute_withdrawal_velocity_nb(np.array([100., 110.], dtype=np.float64), 1)
compute_market_agreement_nb(np.array([.1, -.05], dtype=np.float64), 2)
compute_cascade_signal_nb(np.array([-.05, -.15], dtype=np.float64), 2, -.10)
print(" JIT ready.")
MODEL_V2 = ND_ROOT / 'dvae' / 'convnext_model_v2.json'
SCANS_DIR = ROOT / 'vbt_cache'
KLINES_DIR = ROOT / 'vbt_cache_klines'
MC_MODELS = str(ROOT / 'nautilus_dolphin' / 'mc_results' / 'models')
OUT_FILE = ROOT / 'exp14_results.json'
META_COLS = {
'timestamp', 'scan_number',
'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity',
'vel_div', 'instability_50', 'instability_150',
}
FEATURE_COLS = [
'v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity',
'vel_div','instability_50','instability_150',
]
BASE_ENGINE_KWARGS = dict(
initial_capital=25000., vel_div_threshold=-.02, vel_div_extreme=-.05,
min_leverage=.5, max_leverage=5., leverage_convexity=3.,
fraction=.20, fixed_tp_pct=.0095, stop_pct=1., max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=.75,
dc_skip_contradicts=True, dc_leverage_boost=1., dc_leverage_reduce=.5,
use_asset_selection=True, min_irp_alignment=.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=.62, sp_maker_exit_rate=.50,
use_ob_edge=True, ob_edge_bps=5., ob_confirm_rate=.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
D_LIQ_KWARGS = dict(
extended_soft_cap=D_LIQ_SOFT_CAP, extended_abs_cap=D_LIQ_ABS_CAP,
mc_leverage_ref=D_LIQ_MC_REF, margin_buffer=D_LIQ_MARGIN_BUF,
threshold=.35, alpha=1., adaptive_beta=True,
)
MC_BASE_CFG = {
'trial_id': 0, 'vel_div_threshold': -.020, 'vel_div_extreme': -.050,
'use_direction_confirm': True, 'dc_lookback_bars': 7, 'dc_min_magnitude_bps': .75,
'dc_skip_contradicts': True, 'dc_leverage_boost': 1.00, 'dc_leverage_reduce': .50,
'vd_trend_lookback': 10, 'min_leverage': .50, 'max_leverage': 5.00,
'leverage_convexity': 3.00, 'fraction': .20, 'use_alpha_layers': True,
'use_dynamic_leverage': True, 'fixed_tp_pct': .0095, 'stop_pct': 1.00,
'max_hold_bars': 120, 'use_sp_fees': True, 'use_sp_slippage': True,
'sp_maker_entry_rate': .62, 'sp_maker_exit_rate': .50, 'use_ob_edge': True,
'ob_edge_bps': 5.00, 'ob_confirm_rate': .40, 'ob_imbalance_bias': -.09,
'ob_depth_scale': 1.00, 'use_asset_selection': True, 'min_irp_alignment': .45,
'lookback': 100, 'acb_beta_high': .80, 'acb_beta_low': .20,
'acb_w750_threshold_pct': 60,
}
T_WIN = 32
PROXY_B_DIM = 13 # z[13] = proxy_B dim for v2 ep=13 (r=+0.933)
# ── ZLeverageGateEngine ──────────────────────────────────────────────────────
class ZLeverageGateEngine(LiquidationGuardEngine):
"""
LiquidationGuardEngine subclass that modulates the effective soft leverage
cap based on daily z[13], z_post_std, and resonance-delta signals.
Call set_day_signals(z13, z_post_std, delta) before each begin_day().
"""
def __init__(self, *args,
z13_thr: float = 1.0, # z[13] above → reduce
z13_scale: float = 0.75, # scale when z13 > thr
std_thr: float = 99.0, # z_post_std above → reduce
std_scale: float = 0.75, # scale when std > thr
delta_thr: float = 99.0, # |delta| threshold (std units)
delta_danger_scale: float = 0.80, # scale when delta > thr (danger)
delta_calm_scale: float = 1.00, # scale when delta < -thr (calm)
resonance_thr: float = 99.0, # |delta| < this → resonance
resonance_scale: float= 1.00, # scale at resonance
**kwargs):
super().__init__(*args, **kwargs)
self.z13_thr = z13_thr
self.z13_scale = z13_scale
self.std_thr = std_thr
self.std_scale = std_scale
self.delta_thr = delta_thr
self.delta_danger_scale = delta_danger_scale
self.delta_calm_scale = delta_calm_scale
self.resonance_thr = resonance_thr
self.resonance_scale = resonance_scale
# daily signals (set before each begin_day)
self._z13_today = 0.0
self._z_std_today = 1.0
self._delta_today = 0.0
self._active_scale = 1.0
def set_day_signals(self, z13: float, z_post_std: float, delta: float):
self._z13_today = z13
self._z_std_today = z_post_std
self._delta_today = delta
def begin_day(self, date_str: str, posture: str = 'APEX', direction=None) -> None:
super().begin_day(date_str, posture, direction)
# compute effective scale
scale = 1.0
z13 = self._z13_today
std = self._z_std_today
d = self._delta_today
# Family A: z[13] gate
if z13 > self.z13_thr:
scale = min(scale, self.z13_scale)
# Family B: OOD gate
if std > self.std_thr:
scale = min(scale, self.std_scale)
# Family D: resonance-delta gate (three scenarios)
if self.resonance_thr < 99.0 and abs(d) < self.resonance_thr:
# RESONANCE: both sensors agree → apply resonance_scale
# (if resonance_scale < 1 and both say danger, this amplifies caution;
# if resonance_scale >= 1 both say calm, this restores confidence)
if z13 > self.z13_thr: # resonance confirms danger
scale = min(scale, self.resonance_scale)
else: # resonance confirms calm
scale = max(scale, self.resonance_scale)
elif self.delta_thr < 99.0:
if d > self.delta_thr:
scale = min(scale, self.delta_danger_scale) # Scenario 1: danger
elif d < -self.delta_thr:
scale = max(scale, self.delta_calm_scale) # Scenario 2: calm
self._active_scale = scale
gated = max(self._extended_soft_cap * scale, 1.0)
self.bet_sizer.max_leverage = gated
self.base_max_leverage = gated
def reset(self):
super().reset()
self._z13_today = 0.0
self._z_std_today = 1.0
self._delta_today = 0.0
self._active_scale = 1.0
# ── Config generation ────────────────────────────────────────────────────────
def build_configs():
cfgs = []
# Family A: z[13] gate only
for z13_thr in [0.5, 0.8, 1.0, 1.2]:
for scale in [0.60, 0.70, 0.80, 0.90]:
cfgs.append(dict(
name=f'A_z13t{z13_thr}_s{scale}',
z13_thr=z13_thr, z13_scale=scale,
std_thr=99.0, std_scale=1.0,
delta_thr=99.0, resonance_thr=99.0,
))
# Family B: z_post_std OOD gate only
for std_thr in [0.90, 1.00, 1.10, 1.20, 1.50]:
for scale in [0.60, 0.70, 0.80, 0.90]:
cfgs.append(dict(
name=f'B_stdt{std_thr}_s{scale}',
z13_thr=99.0, z13_scale=1.0,
std_thr=std_thr, std_scale=scale,
delta_thr=99.0, resonance_thr=99.0,
))
# Family C: 2D combined (z[13] + z_post_std)
for z13_thr in [0.8, 1.0]:
for std_thr in [1.0, 1.2]:
for scale in [0.70, 0.80]:
cfgs.append(dict(
name=f'C_z13t{z13_thr}_stdt{std_thr}_s{scale}',
z13_thr=z13_thr, z13_scale=scale,
std_thr=std_thr, std_scale=scale,
delta_thr=99.0, resonance_thr=99.0,
))
# Family D: delta gate (3 scenarios)
for delta_thr in [0.25, 0.50, 1.00]:
for dscale in [0.70, 0.80, 0.90]:
# With resonance: |delta| < 0.2*delta_thr → resonance
res_thr = delta_thr * 0.25
cfgs.append(dict(
name=f'D_dt{delta_thr}_ds{dscale}_res{res_thr:.2f}',
z13_thr=99.0, z13_scale=1.0,
std_thr=99.0, std_scale=1.0,
delta_thr=delta_thr,
delta_danger_scale=dscale,
delta_calm_scale=1.0,
resonance_thr=res_thr,
resonance_scale=dscale, # resonance confirms danger → same scale
))
# Without resonance distinction
cfgs.append(dict(
name=f'D_dt{delta_thr}_ds{dscale}_nores',
z13_thr=99.0, z13_scale=1.0,
std_thr=99.0, std_scale=1.0,
delta_thr=delta_thr,
delta_danger_scale=dscale,
delta_calm_scale=1.0,
resonance_thr=99.0,
))
# Family E: combined A + B + D (best-guess params)
for z13_thr, std_thr, delta_thr, scale in [
(0.8, 1.0, 0.5, 0.75),
(1.0, 1.2, 0.5, 0.80),
(0.8, 1.0, 0.25, 0.70),
(1.0, 1.0, 0.5, 0.75),
]:
cfgs.append(dict(
name=f'E_z{z13_thr}_s{std_thr}_d{delta_thr}_sc{scale}',
z13_thr=z13_thr, z13_scale=scale,
std_thr=std_thr, std_scale=scale,
delta_thr=delta_thr,
delta_danger_scale=scale,
delta_calm_scale=1.0,
resonance_thr=delta_thr * 0.25,
resonance_scale=scale,
))
return cfgs
# ── Signal precomputation ────────────────────────────────────────────────────
def precompute_daily_signals(parquet_files_1m, sensor: ConvNextSensor):
"""
For each daily klines file, compute:
z13_mean — mean z[13] over all T_WIN windows in the day
z_std_mean — mean z_post_std
proxy_b_mean — mean raw proxy_B (instability_50 - v750_lambda_max_velocity)
Returns dict: date_str → {z13, z_post_std, proxy_b_raw}
"""
daily = {}
for f in parquet_files_1m:
date_str = Path(f).stem[:10]
try:
df = pd.read_parquet(f, columns=FEATURE_COLS).dropna()
if len(df) < T_WIN + 5:
continue
z13_vals, std_vals, pb_vals = [], [], []
for start in range(0, len(df) - T_WIN, T_WIN // 2):
window = df.iloc[start:start + T_WIN]
if len(window) < T_WIN:
continue
pb_val = float((window['instability_50'] - window['v750_lambda_max_velocity']).mean())
try:
z_mu, z_post_std = sensor.encode_window(df, start + T_WIN)
z13_vals.append(float(z_mu[PROXY_B_DIM]))
std_vals.append(float(z_post_std))
pb_vals.append(pb_val)
except Exception:
pass
if z13_vals:
daily[date_str] = {
'z13': float(np.mean(z13_vals)),
'z_post_std': float(np.mean(std_vals)),
'proxy_b_raw': float(np.mean(pb_vals)),
}
except Exception as e:
pass
return daily
def fit_delta_regression(daily_signals: dict):
"""
Fit linear map: proxy_b_raw = a * z13 + b
Returns (a, b, delta_std) — delta_std used to normalise delta to z-score units.
"""
dates = sorted(daily_signals.keys())
z13 = np.array([daily_signals[d]['z13'] for d in dates])
pb = np.array([daily_signals[d]['proxy_b_raw'] for d in dates])
# OLS
A = np.column_stack([z13, np.ones(len(z13))])
result = np.linalg.lstsq(A, pb, rcond=None)
a, b = result[0]
pb_hat = a * z13 + b
delta_raw = pb - pb_hat
delta_std = float(np.std(delta_raw)) if len(delta_raw) > 2 else 1.0
print(f" Delta regression: proxy_B = {a:.4f}*z[13] + {b:.4f} "
f"r={float(np.corrcoef(z13, pb)[0,1]):.4f} delta_std={delta_std:.4f}")
return float(a), float(b), delta_std
def add_delta(daily_signals: dict, a: float, b: float, delta_std: float):
"""Add normalised delta (z-score units) to each day's signals."""
for d, v in daily_signals.items():
raw_delta = v['proxy_b_raw'] - (a * v['z13'] + b)
v['delta'] = raw_delta / (delta_std + 1e-9) # normalised
# ── pq_data / vol helpers (same pattern as exp13) ────────────────────────────
def _load_pq_data(parquet_files):
"""Load all 5s parquet files into pq_data dict (date_str → (df, acols, dvol))."""
print("Loading 5s parquet data...")
pq_data = {}
for pf in parquet_files:
pf = Path(pf)
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0, i - 50):i]
if len(seg) >= 10:
dv[i] = float(np.std(np.diff(seg) / seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
print(f" Loaded {len(pq_data)} days")
return pq_data
def _compute_vol_p60(parquet_files):
pq = _load_pq_data(parquet_files[:2]) if parquet_files else {}
vols = []
for _, (_, _, dv) in pq.items():
vols.extend(dv[np.isfinite(dv)].tolist())
return float(np.percentile(vols, 60)) if vols else 0.0
def _make_ob_acb(parquet_files_paths, pq_data: dict):
"""Create fresh OBFeatureEngine + ACB + Forewarner combo for one run."""
pf_list = [Path(p) for p in parquet_files_paths]
OB_ASSETS = sorted({a for ds, (_, ac, _) in pq_data.items() for a in ac})
if not OB_ASSETS:
OB_ASSETS = ['BTCUSDT', 'ETHUSDT', 'BNBUSDT', 'SOLUSDT']
mock_ob = MockOBProvider(
imbalance_bias=-.09, depth_scale=1., assets=OB_ASSETS,
imbalance_biases={
"BTCUSDT": -.086, "ETHUSDT": -.092,
"BNBUSDT": +.05, "SOLUSDT": +.05,
},
)
ob_eng = OBFeatureEngine(mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
forewarner = DolphinForewarner(models_dir=MC_MODELS)
acb = AdaptiveCircuitBreaker()
acb.preload_w750([pf.stem for pf in pf_list])
return ob_eng, acb, forewarner
def _compute_metrics(engine, elapsed):
"""Extract ROI/DD/Calmar/T from a finished engine."""
trades = engine.trade_history
roi = (engine.capital - 25000.) / 25000. * 100.
cap_curve = [25000.]
for t_ in sorted(trades, key=lambda x: getattr(x, 'exit_bar', 0)):
cap_curve.append(cap_curve[-1] + getattr(t_, 'pnl_absolute', 0.))
cap_arr = np.array(cap_curve)
peak = np.maximum.accumulate(cap_arr)
dd = float(np.max((peak - cap_arr) / (peak + 1e-10)) * 100.)
calmar = roi / max(dd, 1e-4)
sh = getattr(engine, '_scale_history', [])
return {
'T': len(trades),
'roi': round(roi, 4),
'dd': round(dd, 4),
'calmar': round(calmar, 4),
'elapsed_s': round(elapsed, 1),
'scale_mean': round(float(np.mean(sh)), 4) if sh else 1.0,
}
# ── Single config runner ──────────────────────────────────────────────────────
def run_one(cfg: dict, daily_signals: dict, pq_data: dict,
parquet_files: list, vol_p60: float,
subset_days: int = 0) -> dict:
"""Run ZLeverageGateEngine for one config on pre-loaded pq_data."""
files = [Path(f) for f in parquet_files]
if subset_days > 0:
files = files[:subset_days]
ob_eng, acb, forewarner = _make_ob_acb([str(f) for f in files], pq_data)
engine = ZLeverageGateEngine(
**BASE_ENGINE_KWARGS,
**D_LIQ_KWARGS,
z13_thr = cfg.get('z13_thr', 99.0),
z13_scale = cfg.get('z13_scale', 1.0),
std_thr = cfg.get('std_thr', 99.0),
std_scale = cfg.get('std_scale', 1.0),
delta_thr = cfg.get('delta_thr', 99.0),
delta_danger_scale = cfg.get('delta_danger_scale', 1.0),
delta_calm_scale = cfg.get('delta_calm_scale', 1.0),
resonance_thr = cfg.get('resonance_thr', 99.0),
resonance_scale = cfg.get('resonance_scale', 1.0),
)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.)
t0 = time.time()
for pf in files:
ds = pf.stem
if ds not in pq_data:
continue
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
sig = daily_signals.get(ds, {})
engine.set_day_signals(
z13 = sig.get('z13', 0.0),
z_post_std = sig.get('z_post_std', 1.0),
delta = sig.get('delta', 0.0),
)
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
return _compute_metrics(engine, time.time() - t0)
# ── Baseline runner ───────────────────────────────────────────────────────────
def run_baseline(pq_data: dict, parquet_files: list, vol_p60: float,
subset_days: int = 0) -> dict:
"""Run D_LIQ_GOLD baseline (no gate) on pre-loaded pq_data."""
files = [Path(f) for f in parquet_files]
if subset_days > 0:
files = files[:subset_days]
ob_eng, acb, forewarner = _make_ob_acb([str(f) for f in files], pq_data)
engine = create_d_liq_engine(**BASE_ENGINE_KWARGS)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.)
t0 = time.time()
for pf in files:
ds = pf.stem
if ds not in pq_data:
continue
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
return _compute_metrics(engine, time.time() - t0)
# ── Main ─────────────────────────────────────────────────────────────────────
def main():
ap = argparse.ArgumentParser()
ap.add_argument('--subset', type=int, default=14,
help='Days for Phase 1 screening (0=full 56 days)')
ap.add_argument('--top_k', type=int, default=20,
help='Top-K configs to validate in Phase 2 (0=skip Phase 2)')
args = ap.parse_args()
print(f"exp14_sweep model=v2(ep=13) subset={args.subset} top_k={args.top_k}")
print(f" PROXY_B_DIM={PROXY_B_DIM} Families: A(z13) B(std) C(2D) D(delta) E(combined)")
# ── Load sensor ──────────────────────────────────────────────────────────
print(f"\nLoading v2 sensor from {MODEL_V2.name}...")
assert MODEL_V2.exists(), f"Model not found: {MODEL_V2}"
sensor = ConvNextSensor(str(MODEL_V2))
print(f" ep={sensor.epoch} val={sensor.val_loss:.4f} z_dim={sensor.z_dim}")
# ── Load data ─────────────────────────────────────────────────────────────
print("\nLoading data files...")
scans_5s = sorted(Path(SCANS_DIR).glob('*.parquet'))
klines_1m = sorted(Path(KLINES_DIR).glob('*.parquet'))
# align to same 56-day window (2025-12-31 to 2026-02-25)
scans_5s = [f for f in scans_5s if '2025-12-31' <= f.stem[:10] <= '2026-02-25']
klines_1m = [f for f in klines_1m if '2025-12-31' <= f.stem[:10] <= '2026-02-25']
print(f" 5s scans: {len(scans_5s)} 1m klines: {len(klines_1m)}")
# ── Pre-load pq_data (once, reused for every run) ─────────────────────────
print("\nPre-loading 5s parquet data (done once for all runs)...")
pq_data_full = _load_pq_data([str(f) for f in scans_5s])
# vol_p60 from full dataset
all_vols = []
for _, (_, _, dv) in pq_data_full.items():
all_vols.extend(dv[np.isfinite(dv)].tolist())
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.0
print(f" vol_p60={vol_p60:.6f}")
# ── Precompute daily signals ──────────────────────────────────────────────
print("\nPrecomputing daily z[13] / z_post_std signals from 1m klines...")
t0 = time.time()
daily_sigs = precompute_daily_signals([str(f) for f in klines_1m], sensor)
print(f" {len(daily_sigs)} days with signals ({time.time()-t0:.0f}s)")
print("\nFitting delta regression (proxy_B = a*z[13] + b)...")
a, b, delta_std = fit_delta_regression(daily_sigs)
add_delta(daily_sigs, a, b, delta_std)
deltas = [v['delta'] for v in daily_sigs.values()]
print(f" delta stats: mean={np.mean(deltas):+.3f} std={np.std(deltas):.3f} "
f"min={np.min(deltas):+.3f} max={np.max(deltas):+.3f}")
# ── Build configs ─────────────────────────────────────────────────────────
configs = build_configs()
print(f"\n{len(configs)} configs across 5 families")
# ── Baseline ──────────────────────────────────────────────────────────────
print("\nRunning baseline (D_LIQ_GOLD)...")
t0 = time.time()
baseline = run_baseline(pq_data_full, [str(f) for f in scans_5s], vol_p60, args.subset)
bROI = baseline.get('roi', 0.0)
bDD = baseline.get('dd', 0.0)
bCal = baseline.get('calmar', 0.0)
bT = baseline.get('T', 0)
print(f" Baseline: T={bT} ROI={bROI:.2f}% DD={bDD:.2f}% Calmar={bCal:.2f} "
f"({time.time()-t0:.0f}s)")
# ── Phase 1: screening ───────────────────────────────────────────────────
print(f"\n{'='*65}")
print(f"Phase 1 — screening {len(configs)} configs on {args.subset or 56}-day window")
print(f"{'='*65}")
results = []
for i, cfg in enumerate(configs):
t0 = time.time()
res = run_one(cfg, daily_sigs, pq_data_full, [str(f) for f in scans_5s],
vol_p60, args.subset)
roi = res.get('roi', 0.0)
dd = res.get('dd', 0.0)
cal = res.get('calmar', 0.0)
T = res.get('T', 0)
dROI = roi - bROI
dDD = dd - bDD
dCal = cal - bCal
elapsed = time.time() - t0
print(f"[{i+1:3d}/{len(configs)}] {cfg['name']}")
print(f" T={T} ROI={roi:.2f}% DD={dd:.2f}% Calmar={cal:.2f} "
f"dROI={dROI:+.2f}pp dDD={dDD:+.2f}pp dCal={dCal:+.2f} "
f"({elapsed:.0f}s)")
results.append({**cfg, 'roi': roi, 'dd': dd, 'calmar': cal, 'trades': T,
'dROI': dROI, 'dDD': dDD, 'dCal': dCal})
results.sort(key=lambda x: x['dROI'], reverse=True)
print(f"\nPhase 1 Top 10:")
for r in results[:10]:
print(f" dROI={r['dROI']:+.2f}pp ROI={r['roi']:.2f}% "
f"Cal={r['calmar']:.2f} {r['name']}")
# ── Phase 2: full validation ─────────────────────────────────────────────
if args.top_k > 0 and args.subset > 0:
top_cfgs = results[:args.top_k]
print(f"\n{'='*65}")
print(f"Phase 2 — validating top {len(top_cfgs)} configs on FULL 56 days")
print(f"{'='*65}")
print("\nRunning baseline (full 56 days)...")
t0 = time.time()
base_full = run_baseline(pq_data_full, [str(f) for f in scans_5s], vol_p60, 0)
bROI_f = base_full.get('roi', 0.0)
bDD_f = base_full.get('dd', 0.0)
bCal_f = base_full.get('calmar', 0.0)
bT_f = base_full.get('T', 0)
print(f" Baseline full: T={bT_f} ROI={bROI_f:.2f}% DD={bDD_f:.2f}% "
f"Calmar={bCal_f:.2f} ({time.time()-t0:.0f}s)")
p2_results = []
for i, cfg in enumerate(top_cfgs):
t0 = time.time()
res = run_one(cfg, daily_sigs, pq_data_full, [str(f) for f in scans_5s],
vol_p60, 0)
roi = res.get('roi', 0.0)
dd = res.get('dd', 0.0)
cal = res.get('calmar', 0.0)
T = res.get('T', 0)
dROI = roi - bROI_f
dDD = dd - bDD_f
dCal = cal - bCal_f
elapsed = time.time() - t0
print(f"[P2 {i+1:2d}/{len(top_cfgs)}] {cfg['name']}")
print(f" T={T} ROI={roi:.2f}% DD={dd:.2f}% Calmar={cal:.2f} "
f"dROI={dROI:+.2f}pp dDD={dDD:+.2f}pp dCal={dCal:+.2f} "
f"({elapsed:.0f}s)")
p2_results.append({**cfg, 'roi': roi, 'dd': dd, 'calmar': cal,
'trades': T, 'dROI': dROI, 'dDD': dDD, 'dCal': dCal})
p2_results.sort(key=lambda x: x['dROI'], reverse=True)
print(f"\nPhase 2 Final Ranking:")
for r in p2_results[:10]:
beat = r['calmar'] > bCal_f * 1.02
print(f" dROI={r['dROI']:+.2f}pp dCal={r['dCal']:+.2f} "
f"{'✓ BEATS' if beat else '✗'} baseline {r['name']}")
# Save results
out = {
'baseline_full': {'roi': bROI_f, 'dd': bDD_f, 'calmar': bCal_f, 'trades': bT_f},
'phase2': p2_results,
'delta_regression': {'a': a, 'b': b, 'delta_std': delta_std},
}
out_path = ROOT / 'exp14_results.json'
json.dump(out, open(out_path, 'w'), indent=2)
print(f"\nResults saved to {out_path}")
print(f"\n[DONE]")
if __name__ == '__main__':
main()

605
nautilus_dolphin/dvae/exp15_stop_gate.py Executable file
View File

@@ -0,0 +1,605 @@
"""
exp15_stop_gate.py — z[13]-gated per-trade stop tightening AND TP extension.
Tests whether per-trade exit overrides based on daily z[13] (proxy_B dim from v2 model)
can improve the D_LIQ_GOLD baseline.
Families:
A — Stop tightening only (high z13 → tight stop) [12 configs]
B — TP extension only (low z13 → higher TP) [20 configs]
C — Hold extension only (low z13 → more bars) [12 configs]
D — TP + Hold combined (low z13 → both) [12 configs]
E — Asymmetric bidirectional (HIGH→tight stop, LOW→higher TP) [6 configs]
Baseline: D_LIQ_GOLD (soft=8x, hard=9x, mc_ref=5x, margin_buffer=0.95)
Usage:
cd nautilus_dolphin/
python dvae/exp15_stop_gate.py --subset 14 --top_k 20 # Phase 1 (14-day screening)
python dvae/exp15_stop_gate.py --subset 0 --top_k 0 # Phase 2 (full 56 days)
"""
import sys, os, time, json, warnings, argparse
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace', line_buffering=True)
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent.parent
ND_ROOT = ROOT / 'nautilus_dolphin'
sys.path.insert(0, str(ND_ROOT))
from dvae.convnext_sensor import ConvNextSensor
from nautilus_dolphin.nautilus.proxy_boost_engine import (
LiquidationGuardEngine,
D_LIQ_SOFT_CAP, D_LIQ_ABS_CAP, D_LIQ_MC_REF, D_LIQ_MARGIN_BUF,
create_d_liq_engine,
)
from nautilus_dolphin.nautilus.ob_features import (
OBFeatureEngine, compute_imbalance_nb, compute_depth_1pct_nb,
compute_depth_quality_nb, compute_fill_probability_nb, compute_spread_proxy_nb,
compute_depth_asymmetry_nb, compute_imbalance_persistence_nb,
compute_withdrawal_velocity_nb, compute_market_agreement_nb, compute_cascade_signal_nb,
)
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from mc.mc_ml import DolphinForewarner
# ── JIT warmup ────────────────────────────────────────────────────────────────
print("Warming up JIT...")
_p = np.array([1., 2., 3.], dtype=np.float64)
compute_irp_nb(_p, -1); compute_ars_nb(1., .5, .01)
rank_assets_irp_nb(np.ones((10, 2), dtype=np.float64), 8, -1, 5, 500., 20, 0.20)
compute_sizing_nb(-.03, -.02, -.05, 3., .5, 5., .20, True, True, 0.,
np.zeros(4, dtype=np.int64), np.zeros(4, dtype=np.int64),
np.zeros(5, dtype=np.float64), 0, -1, .01, .04)
check_dc_nb(_p, 3, 1, .75)
_b = np.array([100., 200., 300., 400., 500.], dtype=np.float64)
_a = np.array([110., 190., 310., 390., 510.], dtype=np.float64)
compute_imbalance_nb(_b, _a); compute_depth_1pct_nb(_b, _a)
compute_depth_quality_nb(210., 200.); compute_fill_probability_nb(1.)
compute_spread_proxy_nb(_b, _a); compute_depth_asymmetry_nb(_b, _a)
compute_imbalance_persistence_nb(np.array([.1, -.1], dtype=np.float64), 2)
compute_withdrawal_velocity_nb(np.array([100., 110.], dtype=np.float64), 1)
compute_market_agreement_nb(np.array([.1, -.05], dtype=np.float64), 2)
compute_cascade_signal_nb(np.array([-.05, -.15], dtype=np.float64), 2, -.10)
print(" JIT ready.")
MODEL_V2 = ND_ROOT / 'dvae' / 'convnext_model_v2.json'
SCANS_DIR = ROOT / 'vbt_cache'
KLINES_DIR = ROOT / 'vbt_cache_klines'
MC_MODELS = str(ROOT / 'nautilus_dolphin' / 'mc_results' / 'models')
OUT_FILE = ROOT / 'exp15_results.json'
META_COLS = {
'timestamp', 'scan_number',
'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity',
'vel_div', 'instability_50', 'instability_150',
}
FEATURE_COLS = [
'v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity',
'vel_div','instability_50','instability_150',
]
BASE_ENGINE_KWARGS = dict(
initial_capital=25000., vel_div_threshold=-.02, vel_div_extreme=-.05,
min_leverage=.5, max_leverage=5., leverage_convexity=3.,
fraction=.20, fixed_tp_pct=.0099, stop_pct=1., max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=.75,
dc_skip_contradicts=True, dc_leverage_boost=1., dc_leverage_reduce=.5,
use_asset_selection=True, min_irp_alignment=.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=.62, sp_maker_exit_rate=.50,
use_ob_edge=True, ob_edge_bps=5., ob_confirm_rate=.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
D_LIQ_KWARGS = dict(
extended_soft_cap=D_LIQ_SOFT_CAP, extended_abs_cap=D_LIQ_ABS_CAP,
mc_leverage_ref=D_LIQ_MC_REF, margin_buffer=D_LIQ_MARGIN_BUF,
threshold=.35, alpha=1., adaptive_beta=True,
)
MC_BASE_CFG = {
'trial_id': 0, 'vel_div_threshold': -.020, 'vel_div_extreme': -.050,
'use_direction_confirm': True, 'dc_lookback_bars': 7, 'dc_min_magnitude_bps': .75,
'dc_skip_contradicts': True, 'dc_leverage_boost': 1.00, 'dc_leverage_reduce': .50,
'vd_trend_lookback': 10, 'min_leverage': .50, 'max_leverage': 5.00,
'leverage_convexity': 3.00, 'fraction': .20, 'use_alpha_layers': True,
'use_dynamic_leverage': True, 'fixed_tp_pct': .0099, 'stop_pct': 1.00,
'max_hold_bars': 120, 'use_sp_fees': True, 'use_sp_slippage': True,
'sp_maker_entry_rate': .62, 'sp_maker_exit_rate': .50, 'use_ob_edge': True,
'ob_edge_bps': 5.00, 'ob_confirm_rate': .40, 'ob_imbalance_bias': -.09,
'ob_depth_scale': 1.00, 'use_asset_selection': True, 'min_irp_alignment': .45,
'lookback': 100, 'acb_beta_high': .80, 'acb_beta_low': .20,
'acb_w750_threshold_pct': 60,
}
T_WIN = 32
PROXY_B_DIM = 13 # z[13] = proxy_B dim for v2 ep=13 (r=+0.933)
# ── ZExitGateEngine ───────────────────────────────────────────────────────────
class ZExitGateEngine(LiquidationGuardEngine):
"""
Per-trade TP extension (low z13) and/or stop tightening (high z13).
Uses z[13] (proxy_B dim from v2 model) as a day-level regime signal:
HIGH z13 (> high_thr) = high adversity → tight stop (defense)
LOW z13 (< low_thr) = calm/trending → higher TP + extended hold (offense)
MID z13 = no override (baseline exit logic)
The _try_entry() override ensures overrides apply to EVERY entry on that day,
not just the first (which is what _pending_* would do if set only once).
"""
def __init__(self, *args,
# Stop tightening (high adversity)
high_thr: float = 99.0, # z13 > this → tight stop
tight_stop_pct: float = 0.005,
# TP extension (calm/trending)
low_thr: float = -99.0, # z13 < this → higher TP
wide_tp_pct: float = None, # None = no TP override
extended_hold: int = None, # None = no hold override
**kwargs):
super().__init__(*args, **kwargs)
self.high_thr = high_thr
self.tight_stop_pct = tight_stop_pct
self.low_thr = low_thr
self.wide_tp_pct = wide_tp_pct
self.extended_hold = extended_hold
self._z13_today = 0.0
self._n_stop_triggered = 0
self._n_tp_triggered = 0
self._n_hold_triggered = 0
def set_day_z13(self, z13: float):
self._z13_today = z13
def _try_entry(self, *args, **kwargs):
z = self._z13_today
# Set overrides fresh before EVERY entry (not just the first)
if z > self.high_thr:
self._pending_stop_override = self.tight_stop_pct
self._pending_tp_override = None
self._pending_max_hold_override = None
self._n_stop_triggered += 1
elif z < self.low_thr:
self._pending_stop_override = None
self._pending_tp_override = self.wide_tp_pct
self._pending_max_hold_override = self.extended_hold
self._n_tp_triggered += 1
if self.extended_hold:
self._n_hold_triggered += 1
else:
self._pending_stop_override = None
self._pending_tp_override = None
self._pending_max_hold_override = None
return super()._try_entry(*args, **kwargs)
def get_trigger_counts(self):
return {
'n_stop_triggered': self._n_stop_triggered,
'n_tp_triggered': self._n_tp_triggered,
'n_hold_triggered': self._n_hold_triggered,
}
# ── Config generation ─────────────────────────────────────────────────────────
def generate_configs():
"""Generate all 62 configs for exp15."""
configs = []
# FAMILY A — Stop tightening only [12 configs]
high_thrs = [0.5, 0.8, 1.0, 1.2]
tight_stops = [0.003, 0.005, 0.010]
for high_thr in high_thrs:
for tight_stop in tight_stops:
name = f'A_ht{high_thr}_stop{tight_stop}'
configs.append({
'name': name,
'family': 'A',
'high_thr': high_thr,
'tight_stop_pct': tight_stop,
'low_thr': -99.0,
'wide_tp_pct': None,
'extended_hold': None,
})
# FAMILY B — TP extension only [20 configs]
low_thrs = [-99.0, 0.3, 0.0, -0.3, -0.5]
wide_tps = [0.0110, 0.0120, 0.0130, 0.0150]
for low_thr in low_thrs:
for wide_tp in wide_tps:
name = f'B_lt{low_thr}_tp{wide_tp:.4f}'
configs.append({
'name': name,
'family': 'B',
'high_thr': 99.0,
'tight_stop_pct': 0.005,
'low_thr': low_thr,
'wide_tp_pct': wide_tp,
'extended_hold': None,
})
# FAMILY C — Hold extension only [12 configs]
low_thrs = [-99.0, 0.3, 0.0, -0.3]
extended_holds = [150, 180, 240]
for low_thr in low_thrs:
for hold in extended_holds:
name = f'C_lt{low_thr}_hold{hold}'
configs.append({
'name': name,
'family': 'C',
'high_thr': 99.0,
'tight_stop_pct': 0.005,
'low_thr': low_thr,
'wide_tp_pct': None,
'extended_hold': hold,
})
# FAMILY D — TP + Hold combined [12 configs]
combos = [
(-99.0, 0.0120, 150), (-99.0, 0.0130, 150), (-99.0, 0.0150, 180),
(-99.0, 0.0120, 180), (-99.0, 0.0130, 180), (-99.0, 0.0150, 240),
(0.3, 0.0120, 150), (0.3, 0.0130, 150), (0.3, 0.0150, 180),
(0.3, 0.0120, 180), (0.3, 0.0130, 180), (0.3, 0.0150, 240),
]
for low_thr, wide_tp, hold in combos:
name = f'D_lt{low_thr}_tp{wide_tp:.4f}_hold{hold}'
configs.append({
'name': name,
'family': 'D',
'high_thr': 99.0,
'tight_stop_pct': 0.005,
'low_thr': low_thr,
'wide_tp_pct': wide_tp,
'extended_hold': hold,
})
# FAMILY E — Asymmetric bidirectional [6 configs]
combos = [
(1.0, 0.005, 0.0, 0.0120, None),
(1.0, 0.005, 0.0, 0.0130, None),
(1.0, 0.005, -0.3, 0.0120, None),
(1.0, 0.005, -0.3, 0.0130, None),
(1.0, 0.005, 0.0, 0.0120, 150),
(1.0, 0.005, -0.3, 0.0130, 150),
]
for high_thr, tight_stop, low_thr, wide_tp, hold in combos:
name = f'E_ht{high_thr}_stop{tight_stop}_lt{low_thr}_tp{wide_tp:.4f}'
if hold:
name += f'_hold{hold}'
configs.append({
'name': name,
'family': 'E',
'high_thr': high_thr,
'tight_stop_pct': tight_stop,
'low_thr': low_thr,
'wide_tp_pct': wide_tp,
'extended_hold': hold,
})
return configs
# ── Data helpers (process_day pattern — same as exp14) ────────────────────────
def _load_pq_data(parquet_files):
"""Load all 5s parquet files into pq_data dict (date_str → (df, acols, dvol))."""
print("Loading 5s parquet data...")
pq_data = {}
for pf in parquet_files:
pf = Path(pf)
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0, i - 50):i]
if len(seg) >= 10:
dv[i] = float(np.std(np.diff(seg) / seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
print(f" Loaded {len(pq_data)} days")
return pq_data
def _make_ob_acb(parquet_files_paths, pq_data: dict):
"""Create fresh OBFeatureEngine + ACB + Forewarner combo for one run."""
pf_list = [Path(p) for p in parquet_files_paths]
OB_ASSETS = sorted({a for ds, (_, ac, _) in pq_data.items() for a in ac})
if not OB_ASSETS:
OB_ASSETS = ['BTCUSDT', 'ETHUSDT', 'BNBUSDT', 'SOLUSDT']
mock_ob = MockOBProvider(
imbalance_bias=-.09, depth_scale=1., assets=OB_ASSETS,
imbalance_biases={
"BTCUSDT": -.086, "ETHUSDT": -.092,
"BNBUSDT": +.05, "SOLUSDT": +.05,
},
)
ob_eng = OBFeatureEngine(mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
forewarner = DolphinForewarner(models_dir=MC_MODELS)
acb = AdaptiveCircuitBreaker()
acb.preload_w750([pf.stem for pf in pf_list])
return ob_eng, acb, forewarner
def _compute_metrics(engine, elapsed):
"""Extract ROI/DD/Calmar/T from a finished engine."""
trades = engine.trade_history
roi = (engine.capital - 25000.) / 25000. * 100.
cap_curve = [25000.]
for t_ in sorted(trades, key=lambda x: getattr(x, 'exit_bar', 0)):
cap_curve.append(cap_curve[-1] + getattr(t_, 'pnl_absolute', 0.))
cap_arr = np.array(cap_curve)
peak = np.maximum.accumulate(cap_arr)
dd = float(np.max((peak - cap_arr) / (peak + 1e-10)) * 100.)
calmar = roi / max(dd, 1e-4)
sh = getattr(engine, '_scale_history', [])
return {
'T': len(trades),
'roi': round(roi, 4),
'dd': round(dd, 4),
'calmar': round(calmar, 4),
'elapsed_s': round(elapsed, 1),
'scale_mean': round(float(np.mean(sh)), 4) if sh else 1.0,
}
def precompute_z13_per_day(parquet_files_1m, sensor):
"""
Compute daily mean z[13] from 1m klines files.
Returns dict: date_str → float (mean z[13] over T_WIN windows in that day)
"""
print("Precomputing daily z[13] from 1m klines...")
z13_by_date = {}
for f in parquet_files_1m:
date_str = Path(f).stem[:10]
try:
df = pd.read_parquet(f, columns=FEATURE_COLS).dropna()
if len(df) < T_WIN + 5:
continue
z13_vals = []
for start in range(0, len(df) - T_WIN, T_WIN // 2):
try:
z_mu, _ = sensor.encode_window(df, start + T_WIN)
z13_vals.append(float(z_mu[PROXY_B_DIM]))
except Exception:
pass
if z13_vals:
z13_by_date[date_str] = float(np.mean(z13_vals))
except Exception:
pass
print(f" {len(z13_by_date)} days with z[13]")
return z13_by_date
# ── Single config runner ───────────────────────────────────────────────────────
def run_one(cfg: dict, z13_by_date: dict, pq_data: dict,
parquet_files: list, vol_p60: float,
subset_days: int = 0) -> dict:
"""Run ZExitGateEngine for one config using process_day API."""
files = [Path(f) for f in parquet_files]
if subset_days > 0:
files = files[:subset_days]
ob_eng, acb, forewarner = _make_ob_acb([str(f) for f in files], pq_data)
engine = ZExitGateEngine(
**BASE_ENGINE_KWARGS,
**D_LIQ_KWARGS,
high_thr = cfg['high_thr'],
tight_stop_pct = cfg['tight_stop_pct'],
low_thr = cfg['low_thr'],
wide_tp_pct = cfg['wide_tp_pct'],
extended_hold = cfg['extended_hold'],
)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.)
t0 = time.time()
for pf in files:
ds = pf.stem
if ds not in pq_data:
continue
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
engine.set_day_z13(z13_by_date.get(ds, 0.0))
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
result = _compute_metrics(engine, time.time() - t0)
result.update(engine.get_trigger_counts())
return result
def run_baseline(pq_data: dict, parquet_files: list, vol_p60: float,
subset_days: int = 0) -> dict:
"""Run D_LIQ_GOLD baseline (no override) on pre-loaded pq_data."""
files = [Path(f) for f in parquet_files]
if subset_days > 0:
files = files[:subset_days]
ob_eng, acb, forewarner = _make_ob_acb([str(f) for f in files], pq_data)
engine = create_d_liq_engine(**BASE_ENGINE_KWARGS)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.)
t0 = time.time()
for pf in files:
ds = pf.stem
if ds not in pq_data:
continue
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
return _compute_metrics(engine, time.time() - t0)
# ── Main ─────────────────────────────────────────────────────────────────────
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--subset', type=int, default=14, help='Days for Phase 1 (0=all)')
parser.add_argument('--top_k', type=int, default=20, help='Top configs for Phase 2')
args = parser.parse_args()
print("=" * 80)
print("exp15 — z[13]-Gated Exit Manager: Stop Tightening AND TP Extension")
print("=" * 80)
# ── Load sensor ──────────────────────────────────────────────────────────
print(f"\nLoading v2 model from {MODEL_V2}...")
assert MODEL_V2.exists(), f"Model not found: {MODEL_V2}"
sensor = ConvNextSensor(str(MODEL_V2))
print(f" Loaded: epoch={sensor.epoch} val_loss={sensor.val_loss:.4f} z_dim={sensor.z_dim}")
# ── Load data files ───────────────────────────────────────────────────────
print("\nLoading data files...")
scans_5s = sorted(Path(SCANS_DIR).glob('*.parquet'))
klines_1m = sorted(Path(KLINES_DIR).glob('*.parquet'))
scans_5s = [f for f in scans_5s if '2025-12-31' <= f.stem[:10] <= '2026-02-25']
klines_1m = [f for f in klines_1m if '2025-12-31' <= f.stem[:10] <= '2026-02-25']
print(f" 5s scans: {len(scans_5s)} 1m klines: {len(klines_1m)}")
# ── Pre-load pq_data (once, reused for every run) ─────────────────────────
print("\nPre-loading 5s parquet data (done once for all runs)...")
pq_data_full = _load_pq_data([str(f) for f in scans_5s])
all_vols = []
for _, (_, _, dv) in pq_data_full.items():
all_vols.extend(dv[np.isfinite(dv)].tolist())
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.0
print(f" vol_p60={vol_p60:.6f}")
# ── Precompute z[13] per day ──────────────────────────────────────────────
z13_by_date = precompute_z13_per_day([str(f) for f in klines_1m], sensor)
# ── Generate configs ──────────────────────────────────────────────────────
configs = generate_configs()
print(f"\nTotal configs: {len(configs)}")
for family in ['A', 'B', 'C', 'D', 'E']:
n = len([c for c in configs if c['family'] == family])
print(f" Family {family}: {n} configs")
# ── Baseline ──────────────────────────────────────────────────────────────
print("\nRunning BASELINE (D_LIQ_GOLD)...")
t0 = time.time()
baseline = run_baseline(pq_data_full, [str(f) for f in scans_5s], vol_p60, args.subset)
bROI = baseline.get('roi', 0.0)
bDD = baseline.get('dd', 0.0)
bCal = baseline.get('calmar', 0.0)
bT = baseline.get('T', 0)
print(f" Baseline: T={bT} ROI={bROI:.2f}% DD={bDD:.2f}% Calmar={bCal:.2f} ({time.time()-t0:.0f}s)")
# ── Phase 1: screening ────────────────────────────────────────────────────
print(f"\n{'='*65}")
print(f"Phase 1 — screening {len(configs)} configs on {args.subset or 56}-day window")
print(f"{'='*65}")
results = []
for i, cfg in enumerate(configs):
t0 = time.time()
res = run_one(cfg, z13_by_date, pq_data_full, [str(f) for f in scans_5s],
vol_p60, args.subset)
roi = res.get('roi', 0.0)
dd = res.get('dd', 0.0)
cal = res.get('calmar', 0.0)
T = res.get('T', 0)
n_stop = res.get('n_stop_triggered', 0)
n_tp = res.get('n_tp_triggered', 0)
n_hold = res.get('n_hold_triggered', 0)
dROI = roi - bROI
dDD = dd - bDD
dCal = cal - bCal
elapsed = time.time() - t0
print(f"[{i+1:3d}/{len(configs)}] {cfg['name']}")
print(f" T={T} ROI={roi:.2f}% DD={dd:.2f}% Calmar={cal:.2f} "
f"dROI={dROI:+.2f}pp dDD={dDD:+.2f}pp dCal={dCal:+.2f} "
f"stop={n_stop} tp={n_tp} hold={n_hold} ({elapsed:.0f}s)")
results.append({**cfg, 'roi': roi, 'dd': dd, 'calmar': cal, 'trades': T,
'dROI': dROI, 'dDD': dDD, 'dCal': dCal,
'n_stop_triggered': n_stop, 'n_tp_triggered': n_tp,
'n_hold_triggered': n_hold})
results.sort(key=lambda x: x['dROI'], reverse=True)
print(f"\nPhase 1 Top 10:")
for r in results[:10]:
print(f" dROI={r['dROI']:+.2f}pp ROI={r['roi']:.2f}% "
f"Cal={r['calmar']:.2f} stop={r['n_stop_triggered']} {r['name']}")
# ── Phase 2: full validation ──────────────────────────────────────────────
p2_results = []
if args.top_k > 0 and args.subset > 0:
top_cfgs = [c for c in results[:args.top_k]]
print(f"\n{'='*65}")
print(f"Phase 2 — validating top {len(top_cfgs)} configs on FULL 56 days")
print(f"{'='*65}")
print("\nRunning baseline (full 56 days)...")
t0 = time.time()
base_full = run_baseline(pq_data_full, [str(f) for f in scans_5s], vol_p60, 0)
bROI_f = base_full.get('roi', 0.0)
bDD_f = base_full.get('dd', 0.0)
bCal_f = base_full.get('calmar', 0.0)
bT_f = base_full.get('T', 0)
print(f" Baseline full: T={bT_f} ROI={bROI_f:.2f}% DD={bDD_f:.2f}% "
f"Calmar={bCal_f:.2f} ({time.time()-t0:.0f}s)")
for i, cfg in enumerate(top_cfgs):
t0 = time.time()
res = run_one(cfg, z13_by_date, pq_data_full,
[str(f) for f in scans_5s], vol_p60, 0)
roi = res.get('roi', 0.0)
dd = res.get('dd', 0.0)
cal = res.get('calmar', 0.0)
T = res.get('T', 0)
n_stop = res.get('n_stop_triggered', 0)
n_tp = res.get('n_tp_triggered', 0)
dROI = roi - bROI_f
dDD = dd - bDD_f
dCal = cal - bCal_f
print(f"[P2 {i+1:2d}/{len(top_cfgs)}] {cfg['name']}")
print(f" T={T} ROI={roi:.2f}% DD={dd:.2f}% Calmar={cal:.2f} "
f"dROI={dROI:+.2f}pp dDD={dDD:+.2f}pp dCal={dCal:+.2f} "
f"stop={n_stop} tp={n_tp} ({time.time()-t0:.0f}s)")
p2_results.append({**cfg, 'roi': roi, 'dd': dd, 'calmar': cal, 'trades': T,
'dROI': dROI, 'dDD': dDD, 'dCal': dCal,
'n_stop_triggered': n_stop, 'n_tp_triggered': n_tp})
# ── Save results ──────────────────────────────────────────────────────────
output = {
'baseline_p1': baseline,
'p1_results': results,
'p2_results': p2_results,
'phase': '1+2' if p2_results else '1',
'n_configs': len(configs),
}
with open(OUT_FILE, 'w') as f:
json.dump(output, f, indent=2, default=str)
print(f"\nResults saved to {OUT_FILE}")
if p2_results:
p2_sorted = sorted(p2_results, key=lambda x: x['dROI'], reverse=True)
print(f"\nPhase 2 Top 5 by ROI delta:")
for r in p2_sorted[:5]:
print(f" dROI={r['dROI']:+.2f}pp DD={r['dd']:.2f}% Cal={r['calmar']:.2f} "
f"stop={r['n_stop_triggered']} {r['name']}")
print("\n[DONE]")
if __name__ == '__main__':
main()

197
nautilus_dolphin/dvae/exp1_proxy_sizing.py Executable file
View File

@@ -0,0 +1,197 @@
"""
Exp 1 — proxy_B-driven position sizing.
Instead of binary gating, scale bet_sizer.base_fraction proportionally to
the proxy_B percentile in a rolling window.
High proxy_B (stress incoming) → scale UP (better mean-reversion environment)
Low proxy_B (calm market) → scale DOWN (weaker signal)
Variants tested:
S1: [0.50x, 1.50x] linear, window=500
S2: [0.25x, 2.00x] linear, window=500 (more aggressive)
S3: [0.50x, 1.50x] linear, window=1000 (slower adaptation)
S4: [0.50x, 1.50x] clipped at p25/p75 (only extreme ends change)
Results logged to exp1_proxy_sizing_results.json.
"""
import sys, time
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD,
load_data, load_forewarner, run_backtest, print_table, log_results
)
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
# ── ProxyBSizedEngine ─────────────────────────────────────────────────────────
class ProxyBSizedEngine(NDAlphaEngine):
"""
NDAlphaEngine that scales base_fraction by rolling proxy_B percentile.
Parameters
----------
proxy_b_min_scale : float Minimum fraction multiplier (at p0 of proxy_B)
proxy_b_max_scale : float Maximum fraction multiplier (at p100 of proxy_B)
proxy_b_clip_low : float Percentile below which use min_scale (0=linear, 0.25=clip p25)
proxy_b_clip_high : float Percentile above which use max_scale
proxy_b_window : int Rolling history length for percentile
"""
def __init__(self, *args,
proxy_b_min_scale: float = 0.5,
proxy_b_max_scale: float = 1.5,
proxy_b_clip_low: float = 0.0,
proxy_b_clip_high: float = 1.0,
proxy_b_window: int = 500,
**kwargs):
super().__init__(*args, **kwargs)
self._pb_min = proxy_b_min_scale
self._pb_max = proxy_b_max_scale
self._pb_clip_lo = proxy_b_clip_low
self._pb_clip_hi = proxy_b_clip_high
self._pb_window = proxy_b_window
self._pb_history = []
self._current_inst50 = 0.0
self._current_v750 = 0.0
# Stats
self.sizing_scales = []
self.sizing_scale_mean = 1.0
def _proxy_b(self): return self._current_inst50 - self._current_v750
def _compute_scale(self):
pb = self._proxy_b()
self._pb_history.append(pb)
if len(self._pb_history) > self._pb_window * 2:
self._pb_history = self._pb_history[-self._pb_window:]
if len(self._pb_history) < 20:
return 1.0 # neutral until enough history
hist = np.array(self._pb_history[-self._pb_window:])
pct = float(np.mean(hist <= pb)) # empirical percentile of current pb
# Clip
pct = max(self._pb_clip_lo, min(self._pb_clip_hi, pct))
# Normalize pct into [0,1] between clip boundaries
span = self._pb_clip_hi - self._pb_clip_lo
if span < 1e-9: return 1.0
t = (pct - self._pb_clip_lo) / span
scale = self._pb_min + t * (self._pb_max - self._pb_min)
return float(scale)
def process_day(self, date_str, df, asset_columns,
vol_regime_ok=None, direction=None, posture='APEX'):
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
v50_raw = row.get('v50_lambda_max_velocity')
v750_raw = row.get('v750_lambda_max_velocity')
inst_raw = row.get('instability_50')
v50_val = float(v50_raw) if (v50_raw is not None and np.isfinite(float(v50_raw))) else 0.0
v750_val = float(v750_raw) if (v750_raw is not None and np.isfinite(float(v750_raw))) else 0.0
inst_val = float(inst_raw) if (inst_raw is not None and np.isfinite(float(inst_raw))) else 0.0
self._current_inst50 = inst_val
self._current_v750 = v750_val
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(p):
prices[ac] = float(p)
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices,
vol_regime_ok=vrok, v50_vel=v50_val, v750_vel=v750_val)
bid += 1
# Update mean scale stat
if self.sizing_scales:
self.sizing_scale_mean = float(np.mean(self.sizing_scales))
return self.end_day()
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
scale = self._compute_scale()
self.sizing_scales.append(scale)
# Temporarily scale fraction
orig = self.bet_sizer.base_fraction
self.bet_sizer.base_fraction = orig * scale
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
self.bet_sizer.base_fraction = orig
return result
# ── Experiment configs ────────────────────────────────────────────────────────
SIZING_VARIANTS = [
# (name, min_scale, max_scale, clip_lo, clip_hi, window)
('S1: [0.5x–1.5x] lin w500', 0.50, 1.50, 0.0, 1.0, 500),
('S2: [0.25x–2.0x] lin w500', 0.25, 2.00, 0.0, 1.0, 500),
('S3: [0.5x–1.5x] lin w1000', 0.50, 1.50, 0.0, 1.0, 1000),
('S4: [0.5x–1.5x] clip p25-p75', 0.50, 1.50, 0.25, 0.75, 500),
]
def main():
ensure_jit()
print("\nLoading data & forewarner...")
load_data()
fw = load_forewarner()
results = []
# Baseline (no sizing mod) — confirms alignment with gold
print("\n" + "="*60)
print("BASELINE (no proxy sizing)")
t0 = time.time()
r = run_backtest(lambda kw: NDAlphaEngine(**kw), 'Baseline (no sizing)', forewarner=fw)
r['elapsed'] = time.time() - t0
results.append(r)
print(f" {r['roi']:.2f}% PF={r['pf']:.4f} DD={r['dd']:.2f}% T={r['trades']} ({r['elapsed']:.0f}s)")
# Sizing variants
for vname, mn, mx, clo, chi, win in SIZING_VARIANTS:
print(f"\n{'='*60}\n{vname}")
t0 = time.time()
def factory(kw, mn=mn, mx=mx, clo=clo, chi=chi, win=win):
return ProxyBSizedEngine(**kw,
proxy_b_min_scale=mn, proxy_b_max_scale=mx,
proxy_b_clip_low=clo, proxy_b_clip_high=chi,
proxy_b_window=win)
r = run_backtest(factory, vname, forewarner=fw)
r['elapsed'] = time.time() - t0
if r.get('sizing_scale_mean'):
print(f" scale_mean={r['sizing_scale_mean']:.3f}")
print(f" {r['roi']:.2f}% PF={r['pf']:.4f} DD={r['dd']:.2f}% T={r['trades']} ({r['elapsed']:.0f}s)")
results.append(r)
print("\n" + "="*83)
print("EXP 1 — proxy_B POSITION SIZING RESULTS")
print("="*83)
print_table(results, gold=GOLD)
log_results(results, _HERE / 'exp1_proxy_sizing_results.json', meta={
'experiment': 'proxy_B position sizing',
'description': 'Scale base_fraction by rolling proxy_B percentile',
'proxy': 'instability_50 - v750_lambda_max_velocity',
})
if __name__ == '__main__':
main()

71
nautilus_dolphin/dvae/exp1_proxy_sizing_results.json Executable file
View File

@@ -0,0 +1,71 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"name": "Baseline (no sizing)",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"elapsed": 268.88467717170715
},
{
"name": "S1: [0.5x\u20131.5x] lin w500",
"roi": 91.48189487501385,
"pf": 1.1782078938827591,
"dd": 16.927834492523125,
"wr": 50.48723897911833,
"sharpe": 3.527834707408158,
"trades": 2155,
"sizing_scale_mean": 1.004486056844373,
"elapsed": 243.42204928398132
},
{
"name": "S2: [0.25x\u20132.0x] lin w500",
"roi": 105.5096611068238,
"pf": 1.1536536843959095,
"dd": 20.295829395125175,
"wr": 50.48723897911833,
"sharpe": 2.955778991462272,
"trades": 2155,
"sizing_scale_mean": 1.1328034858097293,
"elapsed": 244.23685836791992
},
{
"name": "S3: [0.5x\u20131.5x] lin w1000",
"roi": 89.49343433628508,
"pf": 1.176302347754926,
"dd": 16.690286456683094,
"wr": 50.48723897911833,
"sharpe": 3.5136533668662375,
"trades": 2155,
"sizing_scale_mean": 1.0001217537235905,
"elapsed": 231.25820136070251
},
{
"name": "S4: [0.5x\u20131.5x] clip p25-p75",
"roi": 87.12541575348651,
"pf": 1.1628357230291246,
"dd": 18.02585930450568,
"wr": 50.48723897911833,
"sharpe": 3.184294285082965,
"trades": 2155,
"sizing_scale_mean": 1.0189395626318845,
"elapsed": 225.79217648506165
}
],
"meta": {
"experiment": "proxy_B position sizing",
"description": "Scale base_fraction by rolling proxy_B percentile",
"proxy": "instability_50 - v750_lambda_max_velocity"
}
}

314
nautilus_dolphin/dvae/exp2_proxy_exit.py Executable file
View File

@@ -0,0 +1,314 @@
"""
Exp 2 — proxy_B as premature exit signal, with shadow trades.
Post-hoc "what-if" analysis on the baseline trade set.
1. Run baseline engine; log per-day proxy_B and per-asset prices keyed by
(date_str, bar_idx) — the composite key that matches trade.entry_bar.
2. For each trade: find which day it was on (tracked by engine override),
then check if proxy_B dropped below threshold during the hold.
3. Compute early-exit PnL at the trigger bar using the CORRECT asset price.
4. Compare vs actual PnL.
Shadow insight: avg_pnl_delta = early_exit_pnl - actual_pnl
Positive → early exit would have been better
Negative → holding to natural exit was better (proxy_B is NOT a useful exit signal)
Thresholds tested (rolling percentile of proxy_B, window=500):
T1: exit if proxy_B < p10 (rare trigger)
T2: exit if proxy_B < p25 (moderate)
T3: exit if proxy_B < p50 (aggressive)
Logged to exp2_proxy_exit_results.json.
"""
import sys, time, json
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, load_data, load_forewarner, log_results
)
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# ── Engine that logs per-day proxy_B + asset prices + trade dates ─────────────
class ShadowLoggingEngine(NDAlphaEngine):
"""
NDAlphaEngine that captures:
- day_proxy[date][ri] = proxy_b value
- day_prices[date][ri][asset] = price
- trade_dates[trade_idx] = date_str of entry
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.day_proxy = {} # date_str → {ri: proxy_b}
self.day_prices = {} # date_str → {ri: {asset: price}}
self._cur_date = None
self._n_trades_before = 0
self.trade_dates = [] # parallel list to trade_history, entry date per trade
def process_day(self, date_str, df, asset_columns,
vol_regime_ok=None, direction=None, posture='APEX'):
self._cur_date = date_str
self.day_proxy[date_str] = {}
self.day_prices[date_str] = {}
self._n_trades_before = len(self.trade_history)
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
def gf(col):
v = row.get(col)
if v is None: return 0.0
try: f = float(v); return f if np.isfinite(f) else 0.0
except: return 0.0
v50 = gf('v50_lambda_max_velocity')
v750 = gf('v750_lambda_max_velocity')
inst = gf('instability_50')
pb = inst - v750
self.day_proxy[date_str][ri] = pb
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(p):
prices[ac] = float(p)
self.day_prices[date_str][ri] = dict(prices)
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices,
vol_regime_ok=vrok, v50_vel=v50, v750_vel=v750)
bid += 1
result = self.end_day()
# Tag new trades with this date
new_trades = self.trade_history[self._n_trades_before:]
for _ in new_trades:
self.trade_dates.append(date_str)
return result
# ── Shadow analysis ───────────────────────────────────────────────────────────
def shadow_analysis(eng, threshold_pct, window=500):
"""
For each trade, check if proxy_B dropped below rolling threshold
during hold period (same-day bars between entry_bar and exit_bar).
Uses the correct asset price for PnL computation.
"""
tr = eng.trade_history
dates = eng.trade_dates
if len(dates) < len(tr):
# Pad if any trades weren't tagged (shouldn't happen)
dates = dates + [None] * (len(tr) - len(dates))
# Build rolling proxy_B history across all days (chronological)
# We need a global chronological sequence for percentile computation
all_proxy_seq = []
for pf_stem in sorted(eng.day_proxy.keys()):
day_d = eng.day_proxy[pf_stem]
for ri in sorted(day_d.keys()):
all_proxy_seq.append((pf_stem, ri, day_d[ri]))
results = []
proxy_hist = [] # rolling window of ALL bars seen so far
# Build per-day sorted bar sequences for efficient lookup
day_bars = {d: sorted(eng.day_proxy[d].keys()) for d in eng.day_proxy}
# Build lookup: (date, ri) → index in all_proxy_seq (for rolling history)
seq_idx = {(s, r): i for i, (s, r, _) in enumerate(all_proxy_seq)}
for t, date in zip(tr, dates):
if date is None:
results.append(dict(triggered=False, actual_pnl=t.pnl_pct))
continue
entry_bar = int(t.entry_bar) if hasattr(t, 'entry_bar') else 0
exit_bar = int(t.exit_bar) if hasattr(t, 'exit_bar') else entry_bar
actual_pnl = float(t.pnl_pct) if hasattr(t, 'pnl_pct') else 0.0
entry_price = float(t.entry_price) if hasattr(t, 'entry_price') and t.entry_price else 0.0
direction = int(t.direction) if hasattr(t, 'direction') else -1
asset = t.asset if hasattr(t, 'asset') else 'BTCUSDT'
# Rolling threshold: use all bars BEFORE entry on this day
eidx = seq_idx.get((date, entry_bar), 0)
hist_window = [pb for (_, _, pb) in all_proxy_seq[max(0, eidx-window):eidx]]
if len(hist_window) < 20:
results.append(dict(triggered=False, actual_pnl=actual_pnl)); continue
threshold = float(np.percentile(hist_window, threshold_pct * 100))
# Find hold bars on the same day
if date not in day_bars:
results.append(dict(triggered=False, actual_pnl=actual_pnl)); continue
hold_bars = [ri for ri in day_bars[date]
if entry_bar < ri <= exit_bar]
triggered_bar = None
for ri in hold_bars:
if eng.day_proxy[date].get(ri, 999) < threshold:
triggered_bar = ri
break
if triggered_bar is None:
results.append(dict(triggered=False, actual_pnl=actual_pnl)); continue
# Correct early-exit price: same asset, triggered bar on same day
early_price = eng.day_prices[date].get(triggered_bar, {}).get(asset, 0.0)
if entry_price > 0 and early_price > 0:
early_pnl = direction * (early_price - entry_price) / entry_price
else:
results.append(dict(triggered=False, actual_pnl=actual_pnl)); continue
bars_saved = exit_bar - triggered_bar
results.append(dict(
triggered=True,
date=date, entry_bar=entry_bar, exit_bar=exit_bar,
triggered_bar=triggered_bar, bars_saved=bars_saved,
asset=asset, direction=direction,
entry_price=entry_price, early_price=early_price,
actual_pnl=actual_pnl,
early_exit_pnl=early_pnl,
pnl_delta=early_pnl - actual_pnl,
))
triggered = [r for r in results if r['triggered']]
if not triggered:
return dict(n_triggered=0, n_total=len(results), pct_triggered=0,
avg_actual_pnl_pct=0, avg_early_pnl_pct=0, avg_delta_pct=0,
early_better_rate=0, roi_impact_pp=0)
avg_actual = float(np.mean([r['actual_pnl'] for r in triggered]))
avg_early = float(np.mean([r['early_exit_pnl'] for r in triggered]))
avg_delta = float(np.mean([r['pnl_delta'] for r in triggered]))
early_better = float(np.mean([r['pnl_delta'] > 0 for r in triggered]))
avg_bars_saved = float(np.mean([r['bars_saved'] for r in triggered]))
# Estimated ROI impact (sum of pnl deltas × fraction × 100)
roi_impact = float(sum(r['pnl_delta'] for r in triggered) * 0.20 * 100)
# Per-exit-reason breakdown if available
return dict(
n_triggered=len(triggered),
n_total=len(results),
pct_triggered=len(triggered) / max(1, len(results)) * 100,
avg_actual_pnl_pct=avg_actual * 100,
avg_early_exit_pnl_pct=avg_early * 100,
avg_pnl_delta_pct=avg_delta * 100,
early_better_rate=early_better * 100,
avg_bars_saved=avg_bars_saved,
roi_impact_estimate_pp=roi_impact,
)
def main():
ensure_jit()
print("\nLoading data & forewarner...")
d = load_data()
fw = load_forewarner()
from exp_shared import ENGINE_KWARGS, MC_BASE_CFG
import math
print("\nRunning baseline with shadow logging...")
t0 = time.time()
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = ShadowLoggingEngine(**kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw: eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
for pf in d['parquet_files']:
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
print(f" Done in {time.time()-t0:.0f}s Trades={len(tr)} "
f"Tagged={len(eng.trade_dates)}")
# Confirm baseline metrics match gold
def _abs(t): return t.pnl_absolute if hasattr(t,'pnl_absolute') else t.pnl_pct*250.
wins = [t for t in tr if _abs(t) > 0]
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in [x for x in tr if _abs(x)<=0])),1e-9)
roi = (eng.capital - 25000) / 25000 * 100
print(f" Baseline: ROI={roi:.2f}% PF={pf:.4f} (gold: 88.55% / 1.215)")
THRESHOLDS = [
('T1: exit if proxy_B < p10', 0.10),
('T2: exit if proxy_B < p25', 0.25),
('T3: exit if proxy_B < p50', 0.50),
]
all_results = []
for tname, tpct in THRESHOLDS:
print(f"\n{tname}")
res = shadow_analysis(eng, threshold_pct=tpct, window=500)
res['name'] = tname
all_results.append(res)
print(f" Triggered: {res['n_triggered']}/{res['n_total']} "
f"({res['pct_triggered']:.1f}%)")
if res['n_triggered'] > 0:
print(f" Avg actual PnL: {res['avg_actual_pnl_pct']:+.4f}%")
print(f" Avg early-exit PnL: {res['avg_early_exit_pnl_pct']:+.4f}%")
print(f" Avg delta: {res['avg_pnl_delta_pct']:+.4f}% "
f"(+ = early exit BETTER)")
print(f" Early exit better: {res['early_better_rate']:.1f}% of triggered")
print(f" Avg bars saved: {res['avg_bars_saved']:.1f}")
print(f" Est. ROI impact: {res['roi_impact_estimate_pp']:+.2f}pp")
print("\n" + "="*75)
print("EXP 2 — SHADOW EXIT SUMMARY")
print("="*75)
print(f"{'Threshold':<35} {'Trig%':>6} {'AvgDelta%':>11} "
f"{'EarlyBetter%':>13} {'ROI_pp':>8}")
print('-'*75)
for r in all_results:
if r['n_triggered'] > 0:
print(f" {r['name']:<33} {r['pct_triggered']:>6.1f}% "
f"{r['avg_pnl_delta_pct']:>10.4f}% "
f"{r['early_better_rate']:>12.1f}% "
f"{r['roi_impact_estimate_pp']:>8.2f}pp")
else:
print(f" {r['name']:<33} (no triggers)")
verdict = all_results[0] if all_results else {}
if verdict.get('avg_pnl_delta_pct', -1) > 0:
print("\n → VERDICT: Early exit is BENEFICIAL (delta > 0)")
else:
print("\n → VERDICT: Holding to natural exit is BETTER (early exit hurts)")
log_results(all_results, _HERE / 'exp2_proxy_exit_results.json',
meta={'experiment': 'proxy_B exit shadow (corrected)',
'proxy': 'instability_50 - v750_lambda_max_velocity',
'n_trades': len(tr),
'baseline_roi': roi, 'baseline_pf': pf})
if __name__ == '__main__':
main()

55
nautilus_dolphin/dvae/exp2_proxy_exit_results.json Executable file
View File

@@ -0,0 +1,55 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"n_triggered": 32,
"n_total": 2155,
"pct_triggered": 1.4849187935034802,
"avg_actual_pnl_pct": 0.16971558008744564,
"avg_early_exit_pnl_pct": 0.0195519860418085,
"avg_pnl_delta_pct": -0.15016359404563714,
"early_better_rate": 37.5,
"avg_bars_saved": 91.6875,
"roi_impact_estimate_pp": -0.9610470018920778,
"name": "T1: exit if proxy_B < p10"
},
{
"n_triggered": 37,
"n_total": 2155,
"pct_triggered": 1.716937354988399,
"avg_actual_pnl_pct": -0.01204806624716285,
"avg_early_exit_pnl_pct": 0.023025551555410688,
"avg_pnl_delta_pct": 0.03507361780257347,
"early_better_rate": 40.54054054054054,
"avg_bars_saved": 102.70270270270271,
"roi_impact_estimate_pp": 0.2595447717390437,
"name": "T2: exit if proxy_B < p25"
},
{
"n_triggered": 46,
"n_total": 2155,
"pct_triggered": 2.134570765661253,
"avg_actual_pnl_pct": -0.0036840609630585178,
"avg_early_exit_pnl_pct": 0.012711500756466773,
"avg_pnl_delta_pct": 0.016395561719525237,
"early_better_rate": 43.47826086956522,
"avg_bars_saved": 103.47826086956522,
"roi_impact_estimate_pp": 0.1508391678196324,
"name": "T3: exit if proxy_B < p50"
}
],
"meta": {
"experiment": "proxy_B exit shadow (corrected)",
"proxy": "instability_50 - v750_lambda_max_velocity",
"n_trades": 2155,
"baseline_roi": 88.54671933603525,
"baseline_pf": 1.21470506157439
}
}

177
nautilus_dolphin/dvae/exp3_alpha_engine_results.json Executable file
View File

@@ -0,0 +1,177 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"name": "Baseline",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"elapsed": 201.09715700149536
},
{
"name": "V50/gate/p50",
"roi": -21.583410235049012,
"pf": 0.8221680494935062,
"dd": 31.94312195232001,
"wr": 51.162790697674424,
"sharpe": -1.8251030771486862,
"trades": 473,
"gate_suppressed": 106766,
"gate_allowed": 106941,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 338.78401374816895
},
{
"name": "B50/exit/p25",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"gate_suppressed": 0,
"gate_allowed": 0,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 225.31486630439758
},
{
"name": "V50/exit/p10",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"gate_suppressed": 0,
"gate_allowed": 0,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 234.80287218093872
},
{
"name": "B150/gate/p10",
"roi": -17.365740668643976,
"pf": 0.9411606181351053,
"dd": 28.995905550060424,
"wr": 49.43655071043606,
"sharpe": -1.7018177827882233,
"trades": 2041,
"gate_suppressed": 19655,
"gate_allowed": 41671,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 203.65413403511047
},
{
"name": "B50/exit/p50",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"gate_suppressed": 0,
"gate_allowed": 0,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 220.37168407440186
},
{
"name": "B150/gate/p25",
"roi": -1.26185543931782,
"pf": 0.9961437285140536,
"dd": 28.25006238791456,
"wr": 49.69574036511156,
"sharpe": -0.10250195122285347,
"trades": 1972,
"gate_suppressed": 28717,
"gate_allowed": 38549,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 198.88452696800232
},
{
"name": "V150/exit/p50",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"gate_suppressed": 0,
"gate_allowed": 0,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 219.81028938293457
},
{
"name": "V150/gate/p50",
"roi": -24.339020886800313,
"pf": 0.9079395481793519,
"dd": 31.97276290743111,
"wr": 49.57458876914351,
"sharpe": -1.6903434532243515,
"trades": 1763,
"gate_suppressed": 43805,
"gate_allowed": 43807,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 210.28945112228394
},
{
"name": "V300/exit/p50",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"gate_suppressed": 0,
"gate_allowed": 0,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 277.53983092308044
},
{
"name": "V300/exit/p10",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"gate_suppressed": 0,
"gate_allowed": 0,
"early_exits": 0,
"sizing_scale_mean": 1.0,
"elapsed": 331.95733642578125
}
],
"meta": {
"experiment": "exp3 longer proxies alpha engine validation",
"proxies_tested": [
"B50",
"B150",
"V50",
"V150",
"V300"
],
"modes_tested": [
"gate",
"size"
],
"note": "Top-2 per proxy from fast sweep, validated with full Alpha Engine"
}
}

487
nautilus_dolphin/dvae/exp3_fast_sweep_results.json Executable file
View File

@@ -0,0 +1,487 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"roi": 4.1611704733443,
"n_trades": 980,
"wr": 49.28571428571429,
"sharpe": 1.93441354271401,
"key": "V50/gate/p50",
"proxy": "V50",
"mode": "gate",
"threshold_pct": 0.5
},
{
"roi": 5.987937897605655,
"n_trades": 23891,
"wr": 44.81603951278724,
"sharpe": 1.868070095035376,
"key": "B50/exit/p25",
"proxy": "B50",
"mode": "exit",
"threshold_pct": 0.25
},
{
"roi": 4.203067324110643,
"n_trades": 21010,
"wr": 47.42503569728701,
"sharpe": 1.384370226310417,
"key": "V50/exit/p10",
"proxy": "V50",
"mode": "exit",
"threshold_pct": 0.1
},
{
"roi": 5.58761012089275,
"n_trades": 2701,
"wr": 51.05516475379489,
"sharpe": 1.2814598696709858,
"key": "B150/gate/p10",
"proxy": "B150",
"mode": "gate",
"threshold_pct": 0.1
},
{
"roi": 3.4221336926343993,
"n_trades": 35591,
"wr": 43.66272372228934,
"sharpe": 1.2616307481096092,
"key": "B50/exit/p50",
"proxy": "B50",
"mode": "exit",
"threshold_pct": 0.5
},
{
"roi": 5.284493475889573,
"n_trades": 2513,
"wr": 50.33824114604059,
"sharpe": 1.2409796685266818,
"key": "V50/gate/p10",
"proxy": "V50",
"mode": "gate",
"threshold_pct": 0.1
},
{
"roi": 5.03184471791609,
"n_trades": 2666,
"wr": 50.90022505626407,
"sharpe": 1.185462004289883,
"key": "B150/gate/p25",
"proxy": "B150",
"mode": "gate",
"threshold_pct": 0.25
},
{
"roi": 2.3342379236781508,
"n_trades": 46747,
"wr": 43.26480843690504,
"sharpe": 1.1764578378224364,
"key": "V150/exit/p50",
"proxy": "V150",
"mode": "exit",
"threshold_pct": 0.5
},
{
"roi": 4.952775652124575,
"n_trades": 2508,
"wr": 50.75757575757576,
"sharpe": 1.1752617362993856,
"key": "V150/gate/p50",
"proxy": "V150",
"mode": "gate",
"threshold_pct": 0.5
},
{
"roi": 5.018065726503962,
"n_trades": 2636,
"wr": 51.32776934749621,
"sharpe": 1.169177602635925,
"key": "V150/gate/p25",
"proxy": "V150",
"mode": "gate",
"threshold_pct": 0.25
},
{
"roi": 2.4957565643144886,
"n_trades": 47409,
"wr": 43.352528001012466,
"sharpe": 1.1535924003772087,
"key": "V300/exit/p50",
"proxy": "V300",
"mode": "exit",
"threshold_pct": 0.5
},
{
"roi": 3.1810450835647153,
"n_trades": 35753,
"wr": 45.64372220512964,
"sharpe": 1.139549016505284,
"key": "V50/exit/p25",
"proxy": "V50",
"mode": "exit",
"threshold_pct": 0.25
},
{
"roi": 4.723458959017712,
"n_trades": 2638,
"wr": 50.98559514783927,
"sharpe": 1.1183448102112654,
"key": "B150/gate/p50",
"proxy": "B150",
"mode": "gate",
"threshold_pct": 0.5
},
{
"roi": 2.467370511701228,
"n_trades": 45738,
"wr": 43.90878481787573,
"sharpe": 1.1130988979290424,
"key": "V50/exit/p50",
"proxy": "V50",
"mode": "exit",
"threshold_pct": 0.5
},
{
"roi": 4.786060351579402,
"n_trades": 2710,
"wr": 50.29520295202951,
"sharpe": 1.101965763536016,
"key": "B50/gate/p25",
"proxy": "B50",
"mode": "gate",
"threshold_pct": 0.25
},
{
"roi": 3.277471365843976,
"n_trades": 21040,
"wr": 47.010456273764255,
"sharpe": 1.0787267995098386,
"key": "V150/exit/p10",
"proxy": "V150",
"mode": "exit",
"threshold_pct": 0.1
},
{
"roi": 4.606708671848225,
"n_trades": 2707,
"wr": 50.794237162910974,
"sharpe": 1.0621148741933593,
"key": "V150/gate/p10",
"proxy": "V150",
"mode": "gate",
"threshold_pct": 0.1
},
{
"roi": 4.509796198592064,
"n_trades": 2676,
"wr": 50.26158445440957,
"sharpe": 1.0469315281956466,
"key": "B50/gate/p50",
"proxy": "B50",
"mode": "gate",
"threshold_pct": 0.5
},
{
"roi": 3.329600505098229,
"n_trades": 20777,
"wr": 46.69105260624729,
"sharpe": 1.0358050478322955,
"key": "V300/exit/p10",
"proxy": "V300",
"mode": "exit",
"threshold_pct": 0.1
},
{
"roi": 4.4105891591622814,
"n_trades": 2727,
"wr": 49.98166483314998,
"sharpe": 1.0171663918985434,
"key": "B50/gate/p10",
"proxy": "B50",
"mode": "gate",
"threshold_pct": 0.1
},
{
"roi": 4.330032781297288,
"n_trades": 2518,
"wr": 50.35742652899127,
"sharpe": 1.010921034579114,
"key": "V300/gate/p50",
"proxy": "V300",
"mode": "gate",
"threshold_pct": 0.5
},
{
"roi": 4.367799680215123,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "BASELINE",
"proxy": "-",
"mode": "-",
"threshold_pct": 0
},
{
"roi": 4.367799680215123,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "B50/size/p10",
"proxy": "B50",
"mode": "size",
"threshold_pct": 0.1
},
{
"roi": 4.367799680215123,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "B50/size/p25",
"proxy": "B50",
"mode": "size",
"threshold_pct": 0.25
},
{
"roi": 4.367799680215123,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "B50/size/p50",
"proxy": "B50",
"mode": "size",
"threshold_pct": 0.5
},
{
"roi": 3.605522815282547,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "B150/size/p10",
"proxy": "B150",
"mode": "size",
"threshold_pct": 0.1
},
{
"roi": 3.605522815282547,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "B150/size/p25",
"proxy": "B150",
"mode": "size",
"threshold_pct": 0.25
},
{
"roi": 3.605522815282547,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "B150/size/p50",
"proxy": "B150",
"mode": "size",
"threshold_pct": 0.5
},
{
"roi": 2.95312814355122,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V50/size/p10",
"proxy": "V50",
"mode": "size",
"threshold_pct": 0.1
},
{
"roi": 2.95312814355122,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V50/size/p25",
"proxy": "V50",
"mode": "size",
"threshold_pct": 0.25
},
{
"roi": 2.95312814355122,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V50/size/p50",
"proxy": "V50",
"mode": "size",
"threshold_pct": 0.5
},
{
"roi": 4.472629706358244,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V150/size/p10",
"proxy": "V150",
"mode": "size",
"threshold_pct": 0.1
},
{
"roi": 4.472629706358244,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V150/size/p25",
"proxy": "V150",
"mode": "size",
"threshold_pct": 0.25
},
{
"roi": 4.472629706358244,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V150/size/p50",
"proxy": "V150",
"mode": "size",
"threshold_pct": 0.5
},
{
"roi": 3.999178962700034,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V300/size/p10",
"proxy": "V300",
"mode": "size",
"threshold_pct": 0.1
},
{
"roi": 3.999178962700034,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V300/size/p25",
"proxy": "V300",
"mode": "size",
"threshold_pct": 0.25
},
{
"roi": 3.999178962700034,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "V300/size/p50",
"proxy": "V300",
"mode": "size",
"threshold_pct": 0.5
},
{
"roi": 3.626451614638837,
"n_trades": 18495,
"wr": 45.909705325763724,
"sharpe": 0.9461650652992963,
"key": "B50/exit/p10",
"proxy": "B50",
"mode": "exit",
"threshold_pct": 0.1
},
{
"roi": 3.2466809104833683,
"n_trades": 2704,
"wr": 50.18491124260355,
"sharpe": 0.7527502723318003,
"key": "V300/gate/p10",
"proxy": "V300",
"mode": "gate",
"threshold_pct": 0.1
},
{
"roi": 1.9983873882996273,
"n_trades": 36697,
"wr": 44.91647818622776,
"sharpe": 0.7217401163231677,
"key": "V300/exit/p25",
"proxy": "V300",
"mode": "exit",
"threshold_pct": 0.25
},
{
"roi": 2.673177338382371,
"n_trades": 2629,
"wr": 50.741726892354514,
"sharpe": 0.6360878704943932,
"key": "V300/gate/p25",
"proxy": "V300",
"mode": "gate",
"threshold_pct": 0.25
},
{
"roi": 1.3332860487727194,
"n_trades": 1630,
"wr": 48.527607361963184,
"sharpe": 0.46969397860867373,
"key": "V50/gate/p25",
"proxy": "V50",
"mode": "gate",
"threshold_pct": 0.25
},
{
"roi": 1.050754188104408,
"n_trades": 23222,
"wr": 44.74205494789424,
"sharpe": 0.36190958694118786,
"key": "B150/exit/p10",
"proxy": "B150",
"mode": "exit",
"threshold_pct": 0.1
},
{
"roi": 0.40765391258732464,
"n_trades": 36596,
"wr": 44.81910591321456,
"sharpe": 0.14824077173202238,
"key": "V150/exit/p25",
"proxy": "V150",
"mode": "exit",
"threshold_pct": 0.25
},
{
"roi": 0.30344434330278336,
"n_trades": 31821,
"wr": 43.88297036548192,
"sharpe": 0.13013963603284184,
"key": "B150/exit/p50",
"proxy": "B150",
"mode": "exit",
"threshold_pct": 0.5
},
{
"roi": -0.4172667101121519,
"n_trades": 27212,
"wr": 44.11656622078495,
"sharpe": -0.13889774059718565,
"key": "B150/exit/p25",
"proxy": "B150",
"mode": "exit",
"threshold_pct": 0.25
}
],
"meta": {
"experiment": "exp3 fast numpy sweep",
"n_bars": 346740,
"baseline": {
"roi": 4.367799680215123,
"n_trades": 2758,
"wr": 49.41986947063089,
"sharpe": 1.0021989361716817,
"key": "BASELINE",
"proxy": "-",
"mode": "-",
"threshold_pct": 0
},
"note": "simplified SHORT-only, no fees, no leverage"
}
}

419
nautilus_dolphin/dvae/exp3_longer_proxies.py Executable file
View File

@@ -0,0 +1,419 @@
"""
Exp 3 — Longer-window proxies × three modes (gate / size / exit).
Available proxy signals from scan parquets:
proxy_B50 = instability_50 - v750_lambda_max_velocity (original)
proxy_B150 = instability_150 - v750_lambda_max_velocity (longer instability window)
proxy_V50 = v50_lambda_max_velocity - v750_lambda_max_velocity (vel divergence short)
proxy_V150 = v150_lambda_max_velocity - v750_lambda_max_velocity (vel divergence medium)
proxy_V300 = v300_lambda_max_velocity - v750_lambda_max_velocity (vel divergence long)
For each proxy, test:
MODE_GATE: binary suppress entry when proxy < rolling threshold
MODE_SIZE: scale fraction [0.5x, 1.5x] by proxy percentile
MODE_EXIT: (shadow analysis) early exit when proxy < rolling threshold
Run order:
Step 1 — fast numpy sweep across all proxy × mode × threshold
(no Alpha Engine, simplified TP/max_hold model, ~seconds per config)
Step 2 — top-2 configs per proxy validated with full Alpha Engine (~200s each)
Results: exp3_fast_sweep_results.json + exp3_alpha_engine_results.json
"""
import sys, time, math, json
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
import pandas as pd
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, VBT_DIR, META_COLS,
load_data, load_forewarner, run_backtest, print_table, log_results
)
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
# ── Proxy definitions ─────────────────────────────────────────────────────────
PROXY_DEFS = {
'B50': lambda row: _get(row,'instability_50') - _get(row,'v750_lambda_max_velocity'),
'B150': lambda row: _get(row,'instability_150') - _get(row,'v750_lambda_max_velocity'),
'V50': lambda row: _get(row,'v50_lambda_max_velocity') - _get(row,'v750_lambda_max_velocity'),
'V150': lambda row: _get(row,'v150_lambda_max_velocity') - _get(row,'v750_lambda_max_velocity'),
'V300': lambda row: _get(row,'v300_lambda_max_velocity') - _get(row,'v750_lambda_max_velocity'),
}
def _get(row, col, default=0.0):
v = row.get(col)
if v is None: return default
try:
f = float(v)
return f if np.isfinite(f) else default
except Exception:
return default
# ── Step 1: Fast numpy sweep ──────────────────────────────────────────────────
def fast_sweep():
"""
Vectorized sweep across all proxies × modes × thresholds.
Uses simplified backtest: vel_div < -0.02 entry, fixed 0.95% TP, 120-bar max hold.
Single asset (BTCUSDT), no fees, no leverage dynamics.
~100x faster than Alpha Engine.
"""
print("\n" + "="*65)
print("STEP 1 — FAST NUMPY SWEEP (simplified, no Alpha Engine)")
print("="*65)
d = load_data()
TP = 0.0095 # 99bps take profit
MH = 120 # max hold bars
VDT = -0.02 # vel_div entry threshold
# Build concatenated scan data across all days
all_rows = []
for pf in d['parquet_files']:
df, _, _ = d['pq_data'][pf.stem]
for ri in range(len(df)):
row = df.iloc[ri]
r = {c: row.get(c) for c in ['vel_div','BTCUSDT',
'v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity',
'instability_50','instability_150']}
all_rows.append(r)
N = len(all_rows)
vd = np.array([_get(r,'vel_div',np.nan) for r in all_rows])
price = np.array([_get(r,'BTCUSDT',np.nan) for r in all_rows])
# Precompute all proxy arrays
proxy_arrays = {}
for pname, pfn in PROXY_DEFS.items():
proxy_arrays[pname] = np.array([pfn(r) for r in all_rows])
def simplified_backtest(entry_mask, proxy_arr, mode, threshold_pct, window=500):
"""
mode: 'gate' | 'size' | 'exit'
entry_mask: boolean array of candidate entries
Returns: ROI, n_trades, win_rate
"""
capital = 1.0
in_position = False
entry_bar = 0
entry_p = 0.0
pb_hist = []
trades = []
scale = 1.0
for i in range(N):
pb = proxy_arr[i]
if np.isfinite(pb):
pb_hist.append(pb)
hist_window = pb_hist[-window:] if len(pb_hist) >= window else pb_hist
# Rolling threshold
if len(hist_window) >= 20:
thr = float(np.percentile(hist_window, threshold_pct * 100))
else:
thr = -999.0
if in_position:
if np.isnan(price[i]) or entry_p <= 0:
in_position = False; continue
ret = (price[i] - entry_p) / entry_p # LONG direction (for backtest)
# direction=-1 (SHORT) — vel_div < 0 = eigenspace stress = SHORT signal
pnl_pct = -ret # SHORT
bars_held = i - entry_bar
exited = False
# Proxy-based exit (mode='exit')
if mode == 'exit' and np.isfinite(pb) and pb < thr:
exited = True
# Natural exits
if not exited and pnl_pct >= TP:
exited = True
if not exited and bars_held >= MH:
exited = True
if exited:
pos_size = scale * 0.20
trade_pnl = capital * pos_size * pnl_pct
capital += trade_pnl
trades.append(pnl_pct)
in_position = False
else:
if (not np.isnan(vd[i]) and entry_mask[i] and
not np.isnan(price[i]) and price[i] > 0):
# Gate mode: skip if proxy below threshold
if mode == 'gate' and np.isfinite(pb) and pb < thr:
continue
# Sizing mode: compute scale
if mode == 'size' and len(hist_window) >= 20:
pct = float(np.mean(np.array(hist_window) <= pb)) if np.isfinite(pb) else 0.5
scale = 0.5 + pct * 1.0 # linear [0.5, 1.5]
else:
scale = 1.0
in_position = True
entry_bar = i
entry_p = price[i]
n = len(trades)
if n == 0: return dict(roi=0, n_trades=0, wr=0, sharpe=0)
roi = (capital - 1.0) * 100.0
arr = np.array(trades)
wr = float(np.mean(arr > 0)) * 100
sh = float(arr.mean() / (arr.std() + 1e-9) * math.sqrt(n))
return dict(roi=roi, n_trades=n, wr=wr, sharpe=sh)
entry_mask = (np.isfinite(vd)) & (vd < VDT)
MODES = ['gate', 'size', 'exit']
THRESHOLDS = [0.10, 0.25, 0.50]
sweep_results = []
best_by_proxy = {}
for pname in PROXY_DEFS:
parr = proxy_arrays[pname]
for mode in MODES:
for tpct in THRESHOLDS:
key = f"{pname}/{mode}/p{int(tpct*100)}"
res = simplified_backtest(entry_mask, parr, mode, tpct)
res['key'] = key; res['proxy'] = pname
res['mode'] = mode; res['threshold_pct'] = tpct
sweep_results.append(res)
# Baseline (no proxy modification)
base = simplified_backtest(entry_mask, proxy_arrays['B50'], 'size', 0.0)
base['key'] = 'BASELINE'; base['proxy'] = '-'; base['mode'] = '-'; base['threshold_pct'] = 0
sweep_results.insert(0, base)
# Sort by Sharpe
ranked = sorted(sweep_results, key=lambda r: r.get('sharpe', -999), reverse=True)
print(f"\n{'Key':<30} {'ROI%':>7} {'Trades':>7} {'WR%':>6} {'Sharpe':>8}")
print('-'*60)
print(f"{'BASELINE':<30} {base['roi']:>7.2f} {base['n_trades']:>7d} "
f"{base['wr']:>6.1f}% {base['sharpe']:>8.4f}")
print('-'*60)
for r in ranked[:20]:
if r['key'] == 'BASELINE': continue
marker = ' ◄ TOP' if ranked.index(r) <= 5 else ''
print(f"{r['key']:<30} {r['roi']:>7.2f} {r['n_trades']:>7d} "
f"{r['wr']:>6.1f}% {r['sharpe']:>8.4f}{marker}")
log_results(
ranked,
_HERE / 'exp3_fast_sweep_results.json',
gold=None,
meta={'experiment': 'exp3 fast numpy sweep', 'n_bars': N,
'baseline': base, 'note': 'simplified SHORT-only, no fees, no leverage'}
)
# Return top configs for Alpha Engine validation (top 2 per proxy)
top_configs = []
seen_proxies = {}
for r in ranked:
if r['key'] == 'BASELINE': continue
pn = r['proxy']
if pn not in seen_proxies:
seen_proxies[pn] = 0
if seen_proxies[pn] < 2:
top_configs.append(r)
seen_proxies[pn] += 1
return top_configs, ranked[0] # top_configs for AE validation, baseline ref
# ── Step 2: Alpha Engine validation of top configs ────────────────────────────
class MultiProxyEngine(NDAlphaEngine):
"""Generic engine parameterised by proxy function + mode."""
def __init__(self, *args, proxy_name='B50', mode='gate',
threshold_pct=0.25, window=500,
size_min=0.5, size_max=1.5, **kwargs):
super().__init__(*args, **kwargs)
self._proxy_name = proxy_name
self._mode = mode
self._threshold_pct = threshold_pct
self._window = window
self._size_min = size_min
self._size_max = size_max
self._pb_history = []
self._current_vals = {}
# Stats
self.gate_suppressed = 0
self.gate_allowed = 0
self.early_exits = 0
self.sizing_scales = []
def _proxy(self):
v = self._current_vals
if self._proxy_name == 'B50':
return v.get('i50',0.) - v.get('v750',0.)
elif self._proxy_name == 'B150':
return v.get('i150',0.) - v.get('v750',0.)
elif self._proxy_name == 'V50':
return v.get('v50',0.) - v.get('v750',0.)
elif self._proxy_name == 'V150':
return v.get('v150',0.) - v.get('v750',0.)
elif self._proxy_name == 'V300':
return v.get('v300',0.) - v.get('v750',0.)
return 0.0
def _rolling_threshold(self):
h = self._pb_history[-self._window:]
if len(h) < 20: return -999.0
return float(np.percentile(h, self._threshold_pct * 100))
def _rolling_pct(self, pb):
h = np.array(self._pb_history[-self._window:])
if len(h) < 20: return 0.5
return float(np.mean(h <= pb))
def process_day(self, date_str, df, asset_columns,
vol_regime_ok=None, direction=None, posture='APEX'):
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
def gf(col):
v = row.get(col)
if v is None: return 0.0
try:
f = float(v)
return f if np.isfinite(f) else 0.0
except Exception: return 0.0
self._current_vals = dict(
i50=gf('instability_50'), i150=gf('instability_150'),
v50=gf('v50_lambda_max_velocity'), v150=gf('v150_lambda_max_velocity'),
v300=gf('v300_lambda_max_velocity'), v750=gf('v750_lambda_max_velocity'),
)
pb = self._proxy()
self._pb_history.append(pb)
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(p):
prices[ac] = float(p)
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices,
vol_regime_ok=vrok,
v50_vel=self._current_vals['v50'],
v750_vel=self._current_vals['v750'])
bid += 1
return self.end_day()
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
pb = self._proxy()
thr = self._rolling_threshold()
if self._mode == 'gate':
if pb < thr:
self.gate_suppressed += 1
return None
self.gate_allowed += 1
elif self._mode == 'size':
pct = self._rolling_pct(pb)
scale = self._size_min + pct * (self._size_max - self._size_min)
self.sizing_scales.append(scale)
orig = self.bet_sizer.base_fraction
self.bet_sizer.base_fraction = orig * scale
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
self.bet_sizer.base_fraction = orig
return result
return super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
@property
def sizing_scale_mean(self):
return float(np.mean(self.sizing_scales)) if self.sizing_scales else 1.0
def validate_with_alpha_engine(top_configs, forewarner):
print("\n" + "="*65)
print("STEP 2 — ALPHA ENGINE VALIDATION (top configs)")
print("="*65)
ae_results = []
# Baseline first
print("\nBaseline...")
t0 = time.time()
r = run_backtest(lambda kw: NDAlphaEngine(**kw), 'Baseline', forewarner=forewarner)
r['elapsed'] = time.time() - t0
ae_results.append(r)
print(f" {r['roi']:.2f}% PF={r['pf']:.4f} DD={r['dd']:.2f}% ({r['elapsed']:.0f}s)")
for cfg in top_configs:
pn = cfg['proxy']
mode = cfg['mode']
tpct = cfg['threshold_pct']
name = f"{pn}/{mode}/p{int(tpct*100)}"
print(f"\n{name} (sweep rank: Sharpe={cfg['sharpe']:.4f})")
t0 = time.time()
def factory(kw, pn=pn, mode=mode, tpct=tpct):
return MultiProxyEngine(**kw, proxy_name=pn, mode=mode,
threshold_pct=tpct, window=500)
r = run_backtest(factory, name, forewarner=forewarner)
r['elapsed'] = time.time() - t0
ae_results.append(r)
print(f" {r['roi']:.2f}% PF={r['pf']:.4f} DD={r['dd']:.2f}% ({r['elapsed']:.0f}s)")
print("\n" + "="*83)
print("EXP 3 — ALPHA ENGINE RESULTS")
print_table(ae_results, gold=GOLD)
return ae_results
def main():
ensure_jit()
print("\nLoading data & forewarner...")
load_data()
fw = load_forewarner()
top_configs, baseline_ref = fast_sweep()
print(f"\nFast sweep done. Top {len(top_configs)} configs selected for AE validation.")
print(f"Fast baseline: ROI={baseline_ref['roi']:.2f}% Sharpe={baseline_ref['sharpe']:.4f}")
ae_results = validate_with_alpha_engine(top_configs, fw)
log_results(
ae_results,
_HERE / 'exp3_alpha_engine_results.json',
meta={
'experiment': 'exp3 longer proxies alpha engine validation',
'proxies_tested': list(PROXY_DEFS.keys()),
'modes_tested': ['gate','size'], # exit=shadow only, done in exp2
'note': 'Top-2 per proxy from fast sweep, validated with full Alpha Engine',
}
)
if __name__ == '__main__':
main()

505
nautilus_dolphin/dvae/exp4_proxy_coupling.py Executable file
View File

@@ -0,0 +1,505 @@
"""
Exp 4 — proxy_B Coupling & Orthogonality Sweep
Research questions:
1. Is proxy_B orthogonal to the entry signal (vel_div) and other system state?
2. Can proxy_B be *coupled* to existing system parameters to reduce DD without
reducing ROI? (position scale, hold limit, stop gate, rising-proxy exit)
3. Does proxy_B predict trades that will hit large adverse excursions (MAE)?
Method: retroactive shadow analysis on the full 2155-trade baseline.
- One full AE run with extended logging (per-bar proxy_B + vel_div + prices)
- All coupling tests applied post-hoc: O(N_trades) per config → < 1s for 150+ configs
- Focus metric: DD reduction with ROI >= gold * 0.95
Note on stop_pct=1.0 in gold config:
The engine has stop_pct=1.0 (100% — effectively no stop). Trades exit via:
fixed_tp (0.95%), max_hold_bars (120), or direction-reversal signal.
This means MAE can be large before trades recover → proxy-gated stop is meaningful.
Coupling modes:
A. scale_suppress: scale down position when proxy_B high at entry
B. scale_boost: scale up position when proxy_B low at entry
C. hold_limit: exit at fraction of natural hold when proxy_B_max exceeds threshold
D. rising_exit: exit early when proxy_B trajectory during hold is strongly rising
E. pure_stop: retroactive stop simulation (benchmark, no proxy coupling)
F. gated_stop: stop applies ONLY when proxy_B at entry exceeds threshold
Statistical tests:
- Pearson + Spearman: proxy_B vs vel_div, pnl, MAE
- Mann-Whitney U: worst-10% trades vs rest on proxy_B_entry
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
from collections import defaultdict
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
load_data, load_forewarner, log_results
)
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# ── Extended shadow engine ────────────────────────────────────────────────────
class CouplingEngine(NDAlphaEngine):
"""Runs baseline + captures per-bar: proxy_B, vel_div, asset prices."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.day_proxy = {} # date_str → {ri: proxy_B}
self.day_veldiv = {} # date_str → {ri: vel_div}
self.day_prices = {} # date_str → {ri: {asset: price}}
self._n_before = 0
self.trade_dates = [] # parallel to trade_history
def process_day(self, date_str, df, asset_columns,
vol_regime_ok=None, direction=None, posture='APEX'):
self.day_proxy[date_str] = {}
self.day_veldiv[date_str] = {}
self.day_prices[date_str] = {}
self._n_before = len(self.trade_history)
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
def gf(col):
v = row.get(col)
if v is None: return 0.0
try: return float(v) if np.isfinite(float(v)) else 0.0
except: return 0.0
v50 = gf('v50_lambda_max_velocity')
v750 = gf('v750_lambda_max_velocity')
inst = gf('instability_50')
pb = inst - v750
self.day_proxy[date_str][ri] = pb
self.day_veldiv[date_str][ri] = float(vd)
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(float(p)):
prices[ac] = float(p)
self.day_prices[date_str][ri] = prices
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices,
vol_regime_ok=vrok, v50_vel=v50, v750_vel=v750)
bid += 1
self.end_day()
for _ in self.trade_history[self._n_before:]:
self.trade_dates.append(date_str)
# ── Build shadow data ─────────────────────────────────────────────────────────
def build_shadow(d, fw):
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = CouplingEngine(**kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw: eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
for pf in d['parquet_files']:
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
tr = eng.trade_history
roi = (eng.capital - 25000) / 25000 * 100
print(f" Shadow run: ROI={roi:.2f}% Trades={len(tr)}"
f" Tagged={len(eng.trade_dates)}")
return eng, tr
# ── Feature extraction ────────────────────────────────────────────────────────
def extract_features(eng, tr):
"""Per-trade features for coupling analysis."""
feats = []
for t, date in zip(tr, eng.trade_dates):
if date is None:
continue
entry_bar = int(t.entry_bar)
exit_bar = int(getattr(t, 'exit_bar', entry_bar))
direction = int(t.direction)
asset = t.asset
pnl_frac = float(t.pnl_pct) # fraction (not %)
pnl_abs = float(t.pnl_absolute) if hasattr(t, 'pnl_absolute') else pnl_frac * 250.
entry_price = float(getattr(t, 'entry_price', 0) or 0)
pb_entry = eng.day_proxy.get(date, {}).get(entry_bar, np.nan)
vd_entry = eng.day_veldiv.get(date, {}).get(entry_bar, np.nan)
# Hold bars (in-trade, exclusive of entry)
hold_bars = sorted(ri for ri in eng.day_proxy.get(date, {})
if entry_bar < ri <= exit_bar)
pb_hold = [eng.day_proxy[date][ri] for ri in hold_bars]
pb_max = max(pb_hold) if pb_hold else (pb_entry if np.isfinite(pb_entry) else 0.0)
pb_traj = (pb_hold[-1] - pb_hold[0]) if len(pb_hold) > 1 else 0.0
# Max adverse excursion (MAE) — negative = loss
mae = 0.0
if entry_price > 0:
for ri in hold_bars:
p = eng.day_prices.get(date, {}).get(ri, {}).get(asset, 0.0)
if p > 0:
exc = direction * (p - entry_price) / entry_price
if exc < mae:
mae = exc
# Early exit prices at hold fraction 0.25, 0.50, 0.75
early = {}
for frac in (0.25, 0.50, 0.75):
target = entry_bar + max(1, int(frac * (exit_bar - entry_bar)))
avail = [ri for ri in hold_bars if ri >= target]
if avail and entry_price > 0:
p = eng.day_prices.get(date, {}).get(avail[0], {}).get(asset, 0.0)
if p > 0:
early[frac] = direction * (p - entry_price) / entry_price
continue
early[frac] = pnl_frac # fallback: no change
feats.append(dict(
date=date,
hold_bars=exit_bar - entry_bar,
direction=direction,
pnl_frac=pnl_frac,
pnl_abs=pnl_abs,
pb_entry=pb_entry,
vd_entry=vd_entry,
pb_max=pb_max,
pb_traj=pb_traj,
mae=mae,
e25=early[0.25],
e50=early[0.50],
e75=early[0.75],
))
return feats
# ── Orthogonality analysis ────────────────────────────────────────────────────
def orthogonality_analysis(feats):
from scipy.stats import pearsonr, spearmanr, mannwhitneyu
valid = [f for f in feats if np.isfinite(f['pb_entry']) and np.isfinite(f['vd_entry'])]
pb_e = np.array([f['pb_entry'] for f in valid])
vd_e = np.array([f['vd_entry'] for f in valid])
pnl = np.array([f['pnl_frac'] for f in valid])
mae = np.array([f['mae'] for f in valid])
pb_mx = np.array([f['pb_max'] for f in valid])
hold = np.array([f['hold_bars'] for f in valid])
print(f"\n N valid (finite pb_entry + vd_entry): {len(valid)}/{len(feats)}")
print(f" proxy_B stats: mean={pb_e.mean():.4f} std={pb_e.std():.4f} "
f"p10={np.percentile(pb_e,10):.4f} p90={np.percentile(pb_e,90):.4f}")
print(f" vel_div stats: mean={vd_e.mean():.4f} std={vd_e.std():.4f}")
print()
pairs = [
('pb_entry', pb_e, 'vel_div_entry', vd_e),
('pb_entry', pb_e, 'pnl_frac', pnl),
('pb_entry', pb_e, 'mae', mae),
('pb_entry', pb_e, 'hold_bars', hold),
('pb_max', pb_mx, 'pnl_frac', pnl),
('pb_max', pb_mx, 'mae', mae),
]
corr_res = {}
for na, a, nb, b in pairs:
pr, pp = pearsonr(a, b)
sr, sp = spearmanr(a, b)
sig = '***' if pp < 0.001 else '**' if pp < 0.01 else '*' if pp < 0.05 else 'ns'
print(f" corr({na}, {nb}): Pearson r={pr:+.4f} p={pp:.4f} {sig:3s}"
f" Spearman rho={sr:+.4f}")
corr_res[f'{na}_vs_{nb}'] = dict(pearson=float(pr), p=float(pp),
spearman=float(sr), sig=sig)
# Mann-Whitney: is proxy_B different for worst-10% trades vs rest?
print()
for label, metric in [('worst_pnl_10pct', pnl), ('worst_mae_10pct', mae)]:
cut = np.percentile(metric, 10)
mask_w = metric <= cut
pb_w = pb_e[mask_w]
pb_r = pb_e[~mask_w]
stat, p = mannwhitneyu(pb_w, pb_r, alternative='two-sided')
sig = '***' if p < 0.001 else '**' if p < 0.01 else '*' if p < 0.05 else 'ns'
print(f" MW {label}: pb_entry worst={pb_w.mean():.4f} rest={pb_r.mean():.4f} "
f"p={p:.4f} {sig}")
corr_res[f'mw_{label}'] = dict(stat=float(stat), p=float(p),
mean_worst=float(pb_w.mean()),
mean_rest=float(pb_r.mean()), sig=sig)
return corr_res
# ── Coupling sweep ────────────────────────────────────────────────────────────
def _dd_roi(new_pnl_abs, date_order, date_to_trades):
"""Retroactive DD and ROI from modified per-trade PnL array."""
cap, peak, max_dd = 25000.0, 25000.0, 0.0
total = 0.0
for d in date_order:
for i in date_to_trades[d]:
cap += new_pnl_abs[i]
total += new_pnl_abs[i]
if cap > peak: peak = cap
dd = (peak - cap) / peak * 100.0
if dd > max_dd: max_dd = dd
return total / 25000. * 100., max_dd
def coupling_sweep(feats, n_max=None):
N = len(feats)
if n_max: feats = feats[:n_max]
# ---- Arrays ----
pnl_abs = np.array([f['pnl_abs'] for f in feats])
pnl_frac = np.array([f['pnl_frac'] for f in feats])
pb_entry = np.array([f['pb_entry'] for f in feats])
pb_max = np.array([f['pb_max'] for f in feats])
pb_traj = np.array([f['pb_traj'] for f in feats])
mae = np.array([f['mae'] for f in feats])
e25 = np.array([f['e25'] for f in feats])
e50 = np.array([f['e50'] for f in feats])
e75 = np.array([f['e75'] for f in feats])
# Replace NaN pb with median
pb_med = float(np.nanmedian(pb_entry))
pb_entry_c = np.where(np.isfinite(pb_entry), pb_entry, pb_med)
pb_max_c = np.where(np.isfinite(pb_max), pb_max, pb_med)
# Percentile ranks (0=low, 1=high)
def prank(x):
r = np.argsort(np.argsort(x)).astype(float)
return r / max(len(r) - 1, 1)
rk_e = prank(pb_entry_c)
rk_mx = prank(pb_max_c)
rk_tr = prank(pb_traj)
# Date ordering for DD computation
dates_list = [f['date'] for f in feats]
date_order = sorted(set(dates_list))
date_to_trades = defaultdict(list)
for i, d in enumerate(dates_list):
date_to_trades[d].append(i)
base_roi, base_dd = _dd_roi(pnl_abs, date_order, date_to_trades)
# Helper: new_pnl_abs from early exit fraction
def _early_abs(early_frac_arr):
ratio = np.where(np.abs(pnl_frac) > 1e-9,
early_frac_arr / pnl_frac, 1.0)
ratio = np.clip(ratio, -5.0, 5.0)
return pnl_abs * ratio
configs = []
def add(name, new_pnl, **meta):
roi, dd = _dd_roi(new_pnl, date_order, date_to_trades)
configs.append(dict(name=name, roi=roi, dd=dd,
roi_delta=roi - base_roi,
dd_delta=dd - base_dd,
**meta))
# ─── Mode A: scale_suppress — scale down when proxy_B high ───────────────
for sig_name, rk in [('pb_entry', rk_e), ('pb_max', rk_mx), ('pb_traj', rk_tr)]:
for thr in [0.50, 0.65, 0.75, 0.85]:
for alpha in [0.5, 1.0, 2.0]:
for s_min in [0.0, 0.25, 0.5]:
scales = np.maximum(s_min, 1.0 - alpha * np.maximum(0, rk - thr))
add(f'A/{sig_name}/thr{thr}/a{alpha}/min{s_min}',
pnl_abs * scales,
mode='scale_suppress', signal=sig_name,
thr=thr, alpha=alpha, s_min=s_min,
scale_mean=float(scales.mean()))
# ─── Mode B: scale_boost — scale up when proxy_B low ─────────────────────
for sig_name, rk in [('pb_entry', rk_e)]:
for thr in [0.25, 0.35, 0.50]:
for alpha in [0.5, 1.0]:
scales = 1.0 + alpha * np.maximum(0, thr - rk)
add(f'B/{sig_name}/thr{thr}/a{alpha}',
pnl_abs * scales,
mode='scale_boost', signal=sig_name,
thr=thr, alpha=alpha,
scale_mean=float(scales.mean()))
# ─── Mode C: hold_limit — exit early when pb_max high ────────────────────
for frac, early_arr in [(0.25, e25), (0.50, e50), (0.75, e75)]:
for thr_pct in [0.65, 0.75, 0.85, 0.90]:
thr_abs = np.percentile(pb_max_c, thr_pct * 100)
trigger = pb_max_c > thr_abs
new_pnl_f = np.where(trigger, early_arr, pnl_frac)
n_trig = int(trigger.sum())
add(f'C/frac{frac}/pbmax_p{thr_pct}',
_early_abs(new_pnl_f),
mode='hold_limit', frac=frac, thr_pct=thr_pct, n_triggered=n_trig)
# ─── Mode D: rising_exit — exit early when pb trajectory strongly up ──────
for frac, early_arr in [(0.25, e25), (0.50, e50)]:
for thr_pct in [0.70, 0.80, 0.90]:
thr_abs = np.percentile(pb_traj, thr_pct * 100)
trigger = pb_traj > thr_abs
new_pnl_f = np.where(trigger, early_arr, pnl_frac)
n_trig = int(trigger.sum())
add(f'D/frac{frac}/traj_p{thr_pct}',
_early_abs(new_pnl_f),
mode='rising_exit', frac=frac, thr_pct=thr_pct, n_triggered=n_trig)
# ─── Mode E: pure_stop — retroactive stop (no proxy, benchmark) ──────────
for stop_p in [0.003, 0.005, 0.008, 0.010, 0.015, 0.020, 0.030]:
# mae < -stop_p → exit was stopped; clamp pnl_frac to -stop_p
stopped = mae < -stop_p
new_pnl_f = np.where(stopped, -stop_p, pnl_frac)
n_trig = int(stopped.sum())
add(f'E/stop_{stop_p:.3f}',
_early_abs(new_pnl_f),
mode='pure_stop', stop_pct=stop_p, n_triggered=n_trig)
# ─── Mode F: gated_stop — stop applies only when pb_entry high ───────────
for stop_p in [0.005, 0.008, 0.010, 0.015]:
for gate_pct in [0.50, 0.60, 0.75, 0.85]:
gate_thr = np.percentile(pb_entry_c, gate_pct * 100)
gated = pb_entry_c > gate_thr
stopped = gated & (mae < -stop_p)
new_pnl_f = np.where(stopped, -stop_p, pnl_frac)
n_trig = int(stopped.sum())
add(f'F/stop_{stop_p:.3f}/gate_p{gate_pct}',
_early_abs(new_pnl_f),
mode='gated_stop', stop_pct=stop_p, gate_pct=gate_pct,
n_triggered=n_trig)
return base_roi, base_dd, configs
# ── Main ──────────────────────────────────────────────────────────────────────
def main():
ensure_jit()
print("\nLoading data & forewarner...")
d = load_data()
fw = load_forewarner()
print("\nBuilding shadow data (one full AE run)...")
t0 = time.time()
eng, tr = build_shadow(d, fw)
print(f" Built in {time.time()-t0:.0f}s")
print("\nExtracting per-trade features...")
feats = extract_features(eng, tr)
print(f" {len(feats)} trades with valid features")
# ── Orthogonality ─────────────────────────────────────────────────────────
print("\n" + "="*60)
print("ORTHOGONALITY ANALYSIS")
print("="*60)
corr_res = orthogonality_analysis(feats)
# ── Coupling sweep ────────────────────────────────────────────────────────
print("\n" + "="*60)
print(f"COUPLING SWEEP (N={len(feats)} trades)")
print("="*60)
t1 = time.time()
base_roi, base_dd, configs = coupling_sweep(feats)
print(f" Tested {len(configs)} configs in {time.time()-t1:.2f}s")
print(f" Baseline: ROI={base_roi:.2f}% DD={base_dd:.2f}%")
# ── Find DD-reduction candidates ──────────────────────────────────────────
GOLD_ROI = GOLD['roi']
GOLD_DD = GOLD['dd']
ROI_FLOOR = GOLD_ROI * 0.95 # allow at most -5% ROI cost
candidates = [c for c in configs
if c['dd'] < GOLD_DD and c['roi'] >= ROI_FLOOR]
candidates.sort(key=lambda c: (c['dd_delta'], -c['roi_delta']))
print(f"\n Configs with DD < {GOLD_DD:.2f}% AND ROI >= {ROI_FLOOR:.1f}%: "
f"{len(candidates)}")
# Also find absolute best DD reduction regardless of ROI
by_dd = sorted(configs, key=lambda c: c['dd'])[:10]
# Print tables
def hdr():
print(f"\n {'Config':<45} {'ROI%':>7} {'DD%':>6} {'ΔROI':>7} {'ΔDD':>7}"
f" {'mode':<14}")
print(' ' + '-'*90)
def row(c):
extra = ''
if 'n_triggered' in c: extra = f" trig={c['n_triggered']}"
if 'scale_mean' in c: extra = f" smean={c['scale_mean']:.3f}"
print(f" {c['name']:<45} {c['roi']:>7.2f} {c['dd']:>6.2f} "
f"{c['roi_delta']:>+7.2f} {c['dd_delta']:>+7.2f} "
f"{c.get('mode',''):<14}{extra}")
print(f"\n *** GOLD ***: ROI={GOLD_ROI:.2f}% DD={GOLD_DD:.2f}%")
if candidates:
print("\n ── DD < gold AND ROI >= 95% gold ──")
hdr()
for c in candidates[:20]:
row(c)
else:
print("\n (no configs meet both criteria)")
print("\n ── Top 10 by lowest DD (regardless of ROI) ──")
hdr()
for c in by_dd:
row(c)
# ── Summary by mode ───────────────────────────────────────────────────────
from itertools import groupby
print("\n ── Best config per mode (by DD delta, ROI >= floor) ──")
hdr()
by_mode = defaultdict(list)
for c in configs:
by_mode[c.get('mode', 'other')].append(c)
for mode, cs in sorted(by_mode.items()):
best = min(cs, key=lambda c: c['dd'])
row(best)
# ── Log results ───────────────────────────────────────────────────────────
out = _HERE / 'exp4_proxy_coupling_results.json'
payload = {
'gold': GOLD,
'baseline': dict(roi=base_roi, dd=base_dd),
'orthogonality': corr_res,
'n_configs_tested': len(configs),
'dd_reduction_candidates': candidates[:20],
'top10_by_dd': by_dd,
'best_per_mode': {
mode: min(cs, key=lambda c: c['dd'])
for mode, cs in by_mode.items()
},
'all_configs': configs,
}
out.parent.mkdir(parents=True, exist_ok=True)
with open(out, 'w', encoding='utf-8') as f:
json.dump(payload, f, indent=2)
print(f"\n Logged → {out}")
if __name__ == '__main__':
main()

2377
nautilus_dolphin/dvae/exp4_proxy_coupling_results.json Executable file
View File

File diff suppressed because it is too large Load Diff

212
nautilus_dolphin/dvae/exp5_dvae_twopass.py Executable file
View File

@@ -0,0 +1,212 @@
"""
Exp 5 — Two-pass β VAE training.
The question: does high-β pass (β=4) to "map features" followed by low-β pass
(β=0.1) for "fidelity" outperform single-pass β=0.1?
Theory:
Pass 1 (high β): forces encoder to compress — ideally clusters similar market
states together, even at cost of reconstruction quality.
Acts as a structured initializer.
Pass 2 (low β): fine-tunes with more fidelity, starting from the structured
initializer rather than random weights.
We test three variants:
A. Single-pass β=0.1 (baseline, AUC≈0.6918 from flint_precursor_sweep)
B. Two-pass sequential: β=4 (20ep) → β=0.1 (20ep) on same model
C. Two-pass sequential: β=2 (20ep) → β=0.1 (20ep) (softer first pass)
D. Dual encoder: β=4 encoder + β=0.1 encoder, z concatenated (16-dim total)
Metric: OOS AUC for eigenspace stress prediction (K=5, same as e2e_precursor_auc.py).
Gate: if two-pass AUC > single-pass AUC + 0.02 → meaningful improvement.
Note on β=12 (the user's original suggestion):
β=12 would cause complete posterior collapse even with warmup (β=6 collapsed at 0/20 dims).
β=4 is the practical upper bound where some structure survives.
We test β=2 and β=4 to find the sweet spot.
"""
import sys
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE))
_CORPUS_PATH = str(_HERE / 'corpus_cache.npz')
# ── Load T1 corpus ────────────────────────────────────────────────────────────
print("Loading 16K eigen corpus...")
from corpus_builder import DolphinCorpus, OFF, T1 as T1_DIM
corpus = DolphinCorpus.load(_CORPUS_PATH)
mask = corpus.mask[:, 1]
X_e = corpus.X[mask]
T1_data = X_e[:, OFF[1]:OFF[1]+T1_DIM].copy() # (16607, 20)
N = len(T1_data)
print(f" N={N} T1 shape={T1_data.shape}")
# ── Stress labels (K=5) ───────────────────────────────────────────────────────
K = 5
inst_w50 = T1_data[:, 3]
gap_w50 = T1_data[:, 2]
vel_w750 = T1_data[:, 16]
inst_p90 = np.percentile(inst_w50, 90)
gap_p10 = np.percentile(gap_w50, 10)
labels = np.zeros(N, dtype=np.float32)
for i in range(N - K):
if np.any(inst_w50[i+1:i+1+K] > inst_p90) and np.any(gap_w50[i+1:i+1+K] < gap_p10):
labels[i] = 1.0
print(f" Stress labels: {labels.mean()*100:.1f}% positive")
# Chronological split
n_test = N // 4
idx_tr = slice(0, N - n_test)
idx_te = slice(N - n_test, N)
# ── AUC helpers ───────────────────────────────────────────────────────────────
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
def eval_auc(z_all, labels, n_test):
X_lr = z_all[:-K]; y_lr = labels[:-K]
valid = np.isfinite(X_lr).all(1) & np.isfinite(y_lr)
X_lr, y_lr = X_lr[valid], y_lr[valid]
n = len(X_lr) // 4
X_tr, X_te = X_lr[:-n], X_lr[-n:]
y_tr, y_te = y_lr[:-n], y_lr[-n:]
clf = LogisticRegression(class_weight='balanced', max_iter=500, C=0.1)
clf.fit(X_tr, y_tr)
preds = clf.predict_proba(X_te)[:,1]
auc = roc_auc_score(y_te, preds)
return max(auc, 1-auc)
# ── Import FlintHDVAE ─────────────────────────────────────────────────────────
from flint_hd_vae import FlintHDVAE
def build_model(seed=42):
return FlintHDVAE(input_dim=20, hd_dim=512, latent_dim=8,
beta=0.1, seed=seed, use_flint_norm=False)
n_vae_train = int(N * 0.8)
T1_train = T1_data[:n_vae_train]
results = {}
# ── Variant A: Single-pass β=0.1 (baseline) ──────────────────────────────────
print("\n" + "="*55)
print("A. SINGLE-PASS β=0.1 (baseline)")
print("="*55)
m_a = build_model(seed=42)
m_a.fit(T1_train, epochs=40, lr=1e-3, batch_size=256, verbose=True, warmup_frac=0.3)
z_a = m_a.encode(T1_data)
print(f" z var per dim: {z_a.var(0).round(3)}")
print(f" Active dims (var>0.1): {int((z_a.var(0)>0.1).sum())}/8")
auc_a = eval_auc(z_a, labels, n_test)
print(f" OOS AUC = {auc_a:.4f}")
results['A_single_pass_b0.1'] = dict(auc=auc_a, active_dims=int((z_a.var(0)>0.1).sum()),
z_var=z_a.var(0).tolist())
# ── Variant B: Two-pass β=4 → β=0.1 ─────────────────────────────────────────
print("\n" + "="*55)
print("B. TWO-PASS β=4 (20ep) → β=0.1 (20ep)")
print("="*55)
m_b = build_model(seed=42)
print(" Pass 1: β=4, 20 epochs")
m_b.beta = 4.0
m_b.fit(T1_train, epochs=20, lr=1e-3, batch_size=256, verbose=True, warmup_frac=0.3)
print(" Pass 2: β=0.1, 20 epochs (continuing from Pass 1 weights)")
m_b.beta = 0.1
m_b.fit(T1_train, epochs=20, lr=5e-4, batch_size=256, verbose=True, warmup_frac=0.1)
z_b = m_b.encode(T1_data)
print(f" z var per dim: {z_b.var(0).round(3)}")
print(f" Active dims (var>0.1): {int((z_b.var(0)>0.1).sum())}/8")
auc_b = eval_auc(z_b, labels, n_test)
print(f" OOS AUC = {auc_b:.4f} (vs A: {auc_b-auc_a:+.4f})")
results['B_twopass_b4_b0.1'] = dict(auc=auc_b, active_dims=int((z_b.var(0)>0.1).sum()),
z_var=z_b.var(0).tolist())
# ── Variant C: Two-pass β=2 → β=0.1 ─────────────────────────────────────────
print("\n" + "="*55)
print("C. TWO-PASS β=2 (20ep) → β=0.1 (20ep)")
print("="*55)
m_c = build_model(seed=42)
print(" Pass 1: β=2, 20 epochs")
m_c.beta = 2.0
m_c.fit(T1_train, epochs=20, lr=1e-3, batch_size=256, verbose=True, warmup_frac=0.3)
print(" Pass 2: β=0.1, 20 epochs")
m_c.beta = 0.1
m_c.fit(T1_train, epochs=20, lr=5e-4, batch_size=256, verbose=True, warmup_frac=0.1)
z_c = m_c.encode(T1_data)
print(f" z var per dim: {z_c.var(0).round(3)}")
print(f" Active dims (var>0.1): {int((z_c.var(0)>0.1).sum())}/8")
auc_c = eval_auc(z_c, labels, n_test)
print(f" OOS AUC = {auc_c:.4f} (vs A: {auc_c-auc_a:+.4f})")
results['C_twopass_b2_b0.1'] = dict(auc=auc_c, active_dims=int((z_c.var(0)>0.1).sum()),
z_var=z_c.var(0).tolist())
# ── Variant D: Dual encoder (β=4 ‖ β=0.1, z concatenated) ───────────────────
print("\n" + "="*55)
print("D. DUAL ENCODER: β=4 encoder ‖ β=0.1 encoder (z concat → 16-dim)")
print("="*55)
m_d_hi = build_model(seed=42)
m_d_hi.beta = 4.0
print(" Training β=4 encoder (20 epochs)...")
m_d_hi.fit(T1_train, epochs=20, lr=1e-3, batch_size=256, verbose=False, warmup_frac=0.3)
m_d_lo = build_model(seed=123)
m_d_lo.beta = 0.1
print(" Training β=0.1 encoder (40 epochs)...")
m_d_lo.fit(T1_train, epochs=40, lr=1e-3, batch_size=256, verbose=False, warmup_frac=0.3)
z_hi = m_d_hi.encode(T1_data) # (N, 8)
z_lo = m_d_lo.encode(T1_data) # (N, 8)
z_d = np.concatenate([z_hi, z_lo], axis=1) # (N, 16)
print(f" β=4 z var: {z_hi.var(0).round(3)}")
print(f" β=0.1 z var: {z_lo.var(0).round(3)}")
print(f" Combined z shape: {z_d.shape}")
auc_d = eval_auc(z_d, labels, n_test)
print(f" OOS AUC = {auc_d:.4f} (vs A: {auc_d-auc_a:+.4f})")
results['D_dual_b4_b0.1'] = dict(auc=auc_d,
active_dims_hi=int((z_hi.var(0)>0.1).sum()),
active_dims_lo=int((z_lo.var(0)>0.1).sum()),
z_var_hi=z_hi.var(0).tolist(), z_var_lo=z_lo.var(0).tolist())
# ── Summary ───────────────────────────────────────────────────────────────────
GATE = 0.02 # improvement threshold
print("\n" + "="*55)
print("EXP 5 — TWO-PASS β SUMMARY")
print("="*55)
print(f"{'Variant':<35} {'AUC':>8} {'vs A':>8} {'ActiveDims':>11}")
print('-'*65)
for k, v in results.items():
ad = v.get('active_dims', v.get('active_dims_lo', '?'))
delta = v['auc'] - auc_a
flag = ' ◄ GAIN' if delta >= GATE else (' △' if delta > 0 else '')
print(f" {k:<33} {v['auc']:>8.4f} {delta:>+8.4f} {str(ad):>11}{flag}")
best = max(results, key=lambda k: results[k]['auc'])
best_auc = results[best]['auc']
print(f"\n Best: {best} AUC={best_auc:.4f}")
if best_auc - auc_a >= GATE:
print(f" GATE PASS: improvement {best_auc-auc_a:+.4f} ≥ {GATE}")
print(f" → Two-pass training IS beneficial. Adopt for FlintHDVAE.")
else:
print(f" GATE FAIL: best improvement {best_auc-auc_a:+.4f} < {GATE}")
print(f" → Two-pass training offers NO meaningful gain on this dataset.")
# Save
import json
out = _HERE / 'exp5_dvae_twopass_results.json'
with open(out, 'w', encoding='utf-8') as f:
json.dump({'results': results, 'baseline_auc': float(auc_a),
'gate_threshold': GATE, 'winner': best,
'note': 'beta=12 not tested (collapses; beta=6 already showed 0/20 active dims)'}, f, indent=2)
print(f"\n Logged → {out}")

75
nautilus_dolphin/dvae/exp5_dvae_twopass_results.json Executable file
View File

@@ -0,0 +1,75 @@
{
"results": {
"A_single_pass_b0.1": {
"auc": 0.6918441493851568,
"active_dims": 8,
"z_var": [
0.33613626115314765,
0.4036545839396876,
0.46812200154628386,
0.7897261382354528,
0.30868460378586354,
0.6207298112610948,
0.654486990717734,
0.6487686368882809
]
},
"B_twopass_b4_b0.1": {
"auc": 0.685815978891897,
"active_dims": 8,
"z_var": [
0.6463562800257637,
0.794515080808024,
0.6627231420838106,
0.3360878581568057,
0.5536564847674447,
0.47145868605069,
0.6024979573116871,
0.15243441499985075
]
},
"C_twopass_b2_b0.1": {
"auc": 0.6876680380817412,
"active_dims": 8,
"z_var": [
0.13887391434276264,
0.6846345552164088,
0.5075111792873924,
0.6646689437418141,
0.1307330087594524,
0.5100345972756644,
0.7035295234945932,
0.7401098727154677
]
},
"D_dual_b4_b0.1": {
"auc": 0.6771870187460258,
"active_dims_hi": 1,
"active_dims_lo": 8,
"z_var_hi": [
0.016414329885322456,
0.13325390881853802,
0.00914966636326114,
0.015184221145285375,
0.012848108781067335,
0.00813838316387418,
0.023348222620207693,
0.027649003388626463
],
"z_var_lo": [
0.5964793155326009,
0.506480565733225,
0.14274180282824042,
0.3696797264506598,
0.7152758659328332,
0.4816811876862482,
0.45584810995992486,
0.7347258490907227
]
}
},
"baseline_auc": 0.6918441493851568,
"gate_threshold": 0.02,
"winner": "A_single_pass_b0.1",
"note": "beta=12 not tested (collapses; beta=6 already showed 0/20 active dims)"
}

323
nautilus_dolphin/dvae/exp6_stop_test.py Executable file
View File

@@ -0,0 +1,323 @@
"""
Exp 6 — Stop-Tightening: Global vs proxy_B-Gated Stop Test
Context:
proxy_B = instability_50 − v750_lambda_max_velocity
AUC=0.715 for eigenspace stress, r=+0.42 (p=0.003) with intraday MAE.
Retroactive pure_stop benchmark (stop=0.003) showed +1.36pp ROI, −0.14pp DD, 18 triggers.
Hypothesis: gated stop (only when proxy_B_entry > gate_pct percentile) should
preferentially protect worst-MAE trades while leaving the rest alone.
Configs tested:
A. Baseline — stop_pct=1.0 (gold control; must reproduce gold metrics)
B. Global stop sweep — stop_pct ∈ [0.003, 0.005, 0.010] (3 configs)
C. proxy_B gated — gate_pct × tight_stop: [0.65, 0.75] × [0.005, 0.010] = 4 configs
Total: 8 full AE runs.
Focus metric: DD < 15.05% AND ROI >= 84.1% (95% of gold=88.55%).
Also tracked: n_override_set (entries that received tight stop), stop_exits, exit_reason breakdown.
Results logged to exp6_stop_test_results.json.
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
load_data, load_forewarner, log_results, print_table,
)
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# ── proxy_B Gated Stop Engine ─────────────────────────────────────────────────
class ProxyStopGatedEngine(NDAlphaEngine):
"""
Extends NDAlphaEngine with proxy_B-gated per-trade stop logic.
At each bar, proxy_B = instability_50 - v750_lambda_max_velocity is computed
and tracked in a rolling 500-bar history.
At entry, if proxy_B_at_entry > percentile(history, gate_pct):
→ sets _pending_stop_override = tight_stop
Otherwise _pending_stop_override = None (global stop_pct=1.0 applies).
Efficiency hypothesis: gated stop should reduce DD with less ROI cost per unit
of DD reduction vs global stop (since it preferentially targets high-MAE trades).
"""
def __init__(self, *args, gate_pct: float = 0.75, tight_stop: float = 0.005, **kwargs):
super().__init__(*args, **kwargs)
self.gate_pct = gate_pct
self.tight_stop = tight_stop
self._current_proxy_b: float = 0.0
self._proxy_b_history: list = [] # rolling 500-bar window
self.n_override_set: int = 0 # entries where tight stop was applied
def process_day(self, date_str, df, asset_columns,
vol_regime_ok=None, direction=None, posture='APEX'):
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
def gf(col):
v = row.get(col)
if v is None: return 0.0
try: return float(v) if np.isfinite(float(v)) else 0.0
except: return 0.0
v50 = gf('v50_lambda_max_velocity')
v750 = gf('v750_lambda_max_velocity')
inst = gf('instability_50')
pb = inst - v750
# Update proxy_B state before step_bar so _try_entry can read it
self._current_proxy_b = pb
self._proxy_b_history.append(pb)
if len(self._proxy_b_history) > 500:
self._proxy_b_history = self._proxy_b_history[-500:]
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(float(p)):
prices[ac] = float(p)
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices,
vol_regime_ok=vrok, v50_vel=v50, v750_vel=v750)
bid += 1
return self.end_day()
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
# Gate: set _pending_stop_override when proxy_B is elevated
if len(self._proxy_b_history) >= 20:
threshold = np.percentile(self._proxy_b_history, self.gate_pct * 100.0)
if self._current_proxy_b > threshold:
self._pending_stop_override = self.tight_stop
self.n_override_set += 1
else:
self._pending_stop_override = None
else:
self._pending_stop_override = None
return super()._try_entry(bar_idx, vel_div, prices, price_histories, v50_vel, v750_vel)
def reset(self):
super().reset()
self._current_proxy_b = 0.0
self._proxy_b_history = []
self.n_override_set = 0
# ── Run harness ───────────────────────────────────────────────────────────────
def _run(engine_factory, name, d, fw):
"""
Full 55-day backtest using the shared data dict `d`.
Returns metrics dict with additional stop-specific fields.
"""
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
import pandas as pd
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
for pf in d['parquet_files']:
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
if n == 0:
return dict(name=name, roi=roi, pf=0.0, dd=0.0, wr=0.0, sharpe=0.0, trades=0,
stop_exits=0, n_override_set=0, exit_reasons={})
def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
# Exit reason breakdown
exit_reasons = {}
for t in tr:
r = t.exit_reason if hasattr(t, 'exit_reason') else 'UNKNOWN'
exit_reasons[r] = exit_reasons.get(r, 0) + 1
# Stop-specific stats
stop_exits = getattr(eng, 'stop_exits', 0)
n_override_set = getattr(eng, 'n_override_set', 0)
# Efficiency: ROI cost per DD unit reduced vs baseline
# (computed in main after baseline is known)
return dict(
name=name,
roi=roi, pf=pf, dd=max_dd, wr=wr, sharpe=sharpe, trades=n,
stop_exits=stop_exits,
n_override_set=n_override_set,
stop_trigger_rate=stop_exits / n if n > 0 else 0.0,
override_trigger_rate=n_override_set / n if n > 0 else 0.0,
exit_reasons=exit_reasons,
)
# ── Main ──────────────────────────────────────────────────────────────────────
def main():
t_start = time.time()
print("=" * 70)
print("Exp 6 — Stop-Tightening: Global vs proxy_B-Gated Stop Test")
print("=" * 70)
ensure_jit()
d = load_data()
fw = load_forewarner()
configs = []
# A. Baseline
configs.append(("A_baseline", lambda kw: NDAlphaEngine(**kw)))
# B. Global stop sweep
for sp in [0.003, 0.005, 0.010]:
sp_str = f"{sp:.3f}"
def _make_global(stop_val):
def _factory(kw):
k2 = dict(kw); k2['stop_pct'] = stop_val
return NDAlphaEngine(**k2)
return _factory
configs.append((f"B_global_stop={sp_str}", _make_global(sp)))
# C. proxy_B gated stop
for gate_pct in [0.65, 0.75]:
for tight_stop in [0.005, 0.010]:
tag = f"C_gated_gate={gate_pct:.2f}_stop={tight_stop:.3f}"
def _make_gated(gp, ts):
def _factory(kw):
return ProxyStopGatedEngine(gate_pct=gp, tight_stop=ts, **kw)
return _factory
configs.append((tag, _make_gated(gate_pct, tight_stop)))
results = []
for i, (name, factory) in enumerate(configs):
t0 = time.time()
print(f"\n[{i+1}/{len(configs)}] {name} ...")
res = _run(factory, name, d, fw)
elapsed = time.time() - t0
print(f" ROI={res['roi']:.2f}% PF={res['pf']:.4f} DD={res['dd']:.2f}% "
f"WR={res['wr']:.2f}% Sharpe={res['sharpe']:.3f} Trades={res['trades']} "
f"stop_exits={res['stop_exits']} n_override={res['n_override_set']} "
f"({elapsed:.0f}s)")
results.append(res)
# Verification check
baseline = results[0]
print(f"\n{'='*70}")
print(f"VERIFICATION — Baseline vs Gold:")
print(f" ROI: {baseline['roi']:.2f}% (gold={GOLD['roi']:.2f}%)")
print(f" PF: {baseline['pf']:.4f} (gold={GOLD['pf']:.4f})")
print(f" DD: {baseline['dd']:.2f}% (gold={GOLD['dd']:.2f}%)")
print(f" Trades: {baseline['trades']} (gold={GOLD['trades']})")
gold_match = (
abs(baseline['roi'] - GOLD['roi']) < 0.5 and
abs(baseline['pf'] - GOLD['pf']) < 0.005 and
abs(baseline['dd'] - GOLD['dd']) < 0.5 and
abs(baseline['trades'] - GOLD['trades']) < 10
)
print(f" Match: {'PASS ✓' if gold_match else 'FAIL ✗ — check engine state'}")
# Efficiency analysis vs baseline
base_roi = baseline['roi']
base_dd = baseline['dd']
target_roi = GOLD['roi'] * 0.95 # 84.1%
target_dd = GOLD['dd'] # 15.05%
print(f"\n{'='*70}")
print(f"EFFICIENCY TABLE (target: DD<{target_dd:.2f}% AND ROI>={target_roi:.1f}%)")
print(f"{'Config':<42} {'ROI%':>7} {'PF':>6} {'DD%':>6} {'ΔDD':>6} {'ΔROI':>6} {'stops':>6} {'ovrd':>6} {'OK':>4}")
print('-' * 90)
for r in results:
delta_roi = r['roi'] - base_roi
delta_dd = r['dd'] - base_dd
ok = 'Y' if (r['dd'] < target_dd and r['roi'] >= target_roi) else 'N'
stops_str = str(r['stop_exits'])
ovrd_str = str(r['n_override_set']) if r['n_override_set'] > 0 else '-'
print(f"{r['name']:<42} {r['roi']:>7.2f} {r['pf']:>6.4f} {r['dd']:>6.2f} "
f"{delta_dd:>+6.2f} {delta_roi:>+6.2f} {stops_str:>6} {ovrd_str:>6} {ok:>4}")
# Gated vs global efficiency: ROI cost per DD-point gained
print(f"\n{'='*70}")
print("EFFICIENCY RATIO (|ΔROI| / |ΔDD|) — lower = better DD reduction per ROI cost")
for r in results[1:]: # skip baseline
delta_roi = r['roi'] - base_roi
delta_dd = r['dd'] - base_dd
if abs(delta_dd) > 0.01:
eff = abs(delta_roi) / abs(delta_dd)
print(f" {r['name']}: |ΔROI/ΔDD| = {eff:.3f} (ΔROI={delta_roi:+.2f}%, ΔDD={delta_dd:+.2f}%)")
else:
print(f" {r['name']}: ΔDD≈0, no ratio")
# Exit reason breakdown for all configs
print(f"\n{'='*70}")
print("EXIT REASON BREAKDOWN:")
for r in results:
reasons = r.get('exit_reasons', {})
parts = ', '.join(f"{k}={v}" for k, v in sorted(reasons.items()))
print(f" {r['name']}: {parts}")
# Save results
outfile = _HERE / "exp6_stop_test_results.json"
log_results(results, outfile, gold=GOLD, meta={
"exp": "exp6",
"hypothesis": "proxy_B gated stop reduces DD with less ROI cost per unit vs global stop",
"total_elapsed_s": round(time.time() - t_start, 1),
"gold_match": gold_match,
})
total = time.time() - t_start
print(f"\nTotal elapsed: {total/60:.1f} min")
print("Done.")
if __name__ == "__main__":
main()

161
nautilus_dolphin/dvae/exp6_stop_test_results.json Executable file
View File

@@ -0,0 +1,161 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"name": "A_baseline",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"stop_exits": 0,
"n_override_set": 0,
"stop_trigger_rate": 0.0,
"override_trigger_rate": 0.0,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "B_global_stop=0.003",
"roi": -15.311007798855163,
"pf": 0.9421959562747195,
"dd": 23.306320544359586,
"wr": 38.9917695473251,
"sharpe": -1.9388716865680473,
"trades": 2916,
"stop_exits": 1415,
"n_override_set": 0,
"stop_trigger_rate": 0.48525377229080935,
"override_trigger_rate": 0.0,
"exit_reasons": {
"MAX_HOLD": 1139,
"STOP_LOSS": 1415,
"FIXED_TP": 362
}
},
{
"name": "B_global_stop=0.005",
"roi": 13.487121498751875,
"pf": 1.0360002475944843,
"dd": 22.29505159453218,
"wr": 46.31410256410257,
"sharpe": 0.9740585229386011,
"trades": 2496,
"stop_exits": 760,
"n_override_set": 0,
"stop_trigger_rate": 0.30448717948717946,
"override_trigger_rate": 0.0,
"exit_reasons": {
"MAX_HOLD": 1401,
"STOP_LOSS": 760,
"FIXED_TP": 335
}
},
{
"name": "B_global_stop=0.010",
"roi": 25.89578156145128,
"pf": 1.0719795562785166,
"dd": 15.20336058967519,
"wr": 49.137549756744804,
"sharpe": 1.9134302248896795,
"trades": 2261,
"stop_exits": 266,
"n_override_set": 0,
"stop_trigger_rate": 0.11764705882352941,
"override_trigger_rate": 0.0,
"exit_reasons": {
"MAX_HOLD": 1672,
"STOP_LOSS": 266,
"FIXED_TP": 323
}
},
{
"name": "C_gated_gate=0.65_stop=0.005",
"roi": 16.24755153825665,
"pf": 1.0443608125156127,
"dd": 17.778859717952795,
"wr": 49.166666666666664,
"sharpe": 0.9174633345434939,
"trades": 2280,
"stop_exits": 252,
"n_override_set": 18283,
"stop_trigger_rate": 0.11052631578947368,
"override_trigger_rate": 8.018859649122806,
"exit_reasons": {
"MAX_HOLD": 1688,
"STOP_LOSS": 252,
"FIXED_TP": 340
}
},
{
"name": "C_gated_gate=0.65_stop=0.010",
"roi": 35.43917845607325,
"pf": 1.0942200211535686,
"dd": 19.418041939977915,
"wr": 49.954337899543376,
"sharpe": 2.108930895745746,
"trades": 2190,
"stop_exits": 97,
"n_override_set": 17216,
"stop_trigger_rate": 0.04429223744292238,
"override_trigger_rate": 7.861187214611872,
"exit_reasons": {
"MAX_HOLD": 1771,
"FIXED_TP": 322,
"STOP_LOSS": 97
}
},
{
"name": "C_gated_gate=0.75_stop=0.005",
"roi": 21.775022456874765,
"pf": 1.0614653153826534,
"dd": 18.745303697036412,
"wr": 49.20424403183024,
"sharpe": 1.2948006583469642,
"trades": 2262,
"stop_exits": 219,
"n_override_set": 15232,
"stop_trigger_rate": 0.09681697612732096,
"override_trigger_rate": 6.733863837312113,
"exit_reasons": {
"MAX_HOLD": 1705,
"STOP_LOSS": 219,
"FIXED_TP": 338
}
},
{
"name": "C_gated_gate=0.75_stop=0.010",
"roi": 38.422502318822396,
"pf": 1.101746374983589,
"dd": 19.416920166163333,
"wr": 49.70278920896205,
"sharpe": 2.2247675548672463,
"trades": 2187,
"stop_exits": 85,
"n_override_set": 14320,
"stop_trigger_rate": 0.038866026520347506,
"override_trigger_rate": 6.547782350251486,
"exit_reasons": {
"MAX_HOLD": 1776,
"FIXED_TP": 326,
"STOP_LOSS": 85
}
}
],
"meta": {
"exp": "exp6",
"hypothesis": "proxy_B gated stop reduces DD with less ROI cost per unit vs global stop",
"total_elapsed_s": 1876.4,
"gold_match": true
}
}

487
nautilus_dolphin/dvae/exp7_live_coupling.py Executable file
View File

@@ -0,0 +1,487 @@
"""
Exp 7 — Live proxy_B Coupling: Scale, Hold-Limit, Rising-Exit
Context:
proxy_B = instability_50 - v750_lambda_max_velocity
AUC=0.715 for eigenspace stress, r=+0.42 (p=0.003) with intraday MAE.
Orthogonal to vel_div (r=-0.03, ns).
Exp6 confirmed: stop-tightening is DEAD (re-entry cascade, worse DD always).
Exp4 retroactive modes A/B/C/D were 98% invalid (entry_bar alignment bug —
entry_bar = global_bar_idx but day_proxy keyed by local ri).
This experiment is the FIRST valid live test of those four coupling modes.
Modes tested:
B. scale_boost: size UP when proxy_B is LOW at entry (calm = better quality)
A. scale_suppress: size DOWN when proxy_B is HIGH at entry (stress = worse quality)
C. hold_limit: exit early when proxy_B_max during hold exceeds threshold
D. rising_exit: exit early when current proxy_B rises above threshold during hold
Implementation notes:
- Modes B and A: post-entry notional scaling. No timing change → no re-entry dynamics.
ProxyScaleEngine._try_entry calls super() then scales self.position.notional.
- Modes C and D: in process_day loop, per-bar condition check BEFORE step_bar.
If triggered: _execute_exit(force reason) → position cleared → step_bar enters fresh.
No re-entry on same bar (process_bar checks bar_idx > _last_exit_bar).
- proxy_B history: rolling 500-bar window, minimum 20 samples before any gating activates.
- threshold at entry: np.percentile(history, thr_pct * 100) computed from rolling window.
Configs:
Baseline: 1
scale_boost (B): 4 (thr in {0.25, 0.35} x alpha in {0.5, 1.0})
scale_suppress (A): 4 (thr in {0.75, 0.85} x alpha in {0.5, 1.0}, s_min=0.25)
hold_limit (C): 3 (frac=0.5, thr_pct in {0.65, 0.75, 0.85})
rising_exit (D): 3 (frac=0.5, thr_pct in {0.65, 0.75, 0.85})
Total: 15 configs x ~240s = ~60 min
Focus metric: DD < 15.05% AND ROI >= 84.1% (95% of gold=88.55%)
Results logged to exp7_live_coupling_results.json.
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
load_data, load_forewarner, log_results, print_table,
)
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# ── Base: proxy_B per-bar tracking ───────────────────────────────────────────
class ProxyBaseEngine(NDAlphaEngine):
"""Tracks proxy_B = instability_50 - v750_lambda_max_velocity per bar."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._current_proxy_b: float = 0.0
self._proxy_b_history: list = [] # rolling 500-bar window
def _update_proxy(self, inst: float, v750: float) -> float:
pb = inst - v750
self._current_proxy_b = pb
self._proxy_b_history.append(pb)
if len(self._proxy_b_history) > 500:
self._proxy_b_history = self._proxy_b_history[-500:]
return pb
def _proxy_prank(self) -> float:
"""Percentile rank of current proxy_B in history (0=low, 1=high)."""
if not self._proxy_b_history:
return 0.5
n = len(self._proxy_b_history)
return sum(v < self._current_proxy_b for v in self._proxy_b_history) / n
def _proxy_threshold(self, thr_pct: float) -> float:
"""Absolute threshold: percentile thr_pct of rolling history."""
if len(self._proxy_b_history) < 20:
return float('inf')
return float(np.percentile(self._proxy_b_history, thr_pct * 100.0))
def process_day(self, date_str, df, asset_columns,
vol_regime_ok=None, direction=None, posture='APEX'):
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
def gf(col):
v = row.get(col)
if v is None: return 0.0
try: return float(v) if np.isfinite(float(v)) else 0.0
except: return 0.0
v50 = gf('v50_lambda_max_velocity')
v750 = gf('v750_lambda_max_velocity')
inst = gf('instability_50')
self._update_proxy(inst, v750)
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(float(p)):
prices[ac] = float(p)
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self._per_bar_hook(ri, float(vd), prices, v50, v750, vrok)
self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices,
vol_regime_ok=vrok, v50_vel=v50, v750_vel=v750)
bid += 1
return self.end_day()
def _per_bar_hook(self, ri, vd, prices, v50, v750, vrok):
"""Override in subclasses to inject pre-step_bar logic."""
pass
# ── Mode B/A: position scaling ────────────────────────────────────────────────
class ProxyScaleEngine(ProxyBaseEngine):
"""
Mode B (scale_boost): size UP when proxy_B is LOW (calm entry conditions).
Mode A (scale_suppress): size DOWN when proxy_B is HIGH (stress entry conditions).
Scale factor computed from percentile rank of current proxy_B in rolling history.
boost: scale = 1.0 + alpha * max(0, threshold - prank) [rk < thr → boost]
suppress: scale = max(s_min, 1.0 - alpha * max(0, prank - threshold)) [rk > thr → reduce]
Applied post-entry by multiplying self.position.notional directly.
Does NOT change timing → no re-entry cascade dynamics.
"""
def __init__(self, *args, mode: str = 'boost', threshold: float = 0.35,
alpha: float = 1.0, s_min: float = 0.25, **kwargs):
super().__init__(*args, **kwargs)
assert mode in ('boost', 'suppress')
self.mode = mode
self.threshold = threshold
self.alpha = alpha
self.s_min = s_min
self._scale_history: list = []
@property
def sizing_scale_mean(self) -> float:
return float(np.mean(self._scale_history)) if self._scale_history else 1.0
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
if result and self.position:
prank = self._proxy_prank()
if self.mode == 'boost':
scale = 1.0 + self.alpha * max(0.0, self.threshold - prank)
else: # suppress
scale = max(self.s_min, 1.0 - self.alpha * max(0.0, prank - self.threshold))
self.position.notional *= scale
self._scale_history.append(scale)
return result
def reset(self):
super().reset()
self._scale_history = []
# ── Mode C: hold_limit ────────────────────────────────────────────────────────
class ProxyHoldLimitEngine(ProxyBaseEngine):
"""
Exit early when:
- proxy_B during hold has exceeded thr_pct-percentile threshold
- AND bars_held >= frac * max_hold_bars
Threshold computed at entry time from rolling proxy_B history.
Tracks max proxy_B seen since entry; fires as soon as condition is met.
"""
def __init__(self, *args, frac: float = 0.5, thr_pct: float = 0.75, **kwargs):
super().__init__(*args, **kwargs)
self.frac = frac
self.thr_pct = thr_pct
self._pb_hold_threshold: float = float('inf') # set at entry
self._pb_max_so_far: float = 0.0 # tracked during hold
self.early_exits: int = 0
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
if result:
self._pb_hold_threshold = self._proxy_threshold(self.thr_pct)
self._pb_max_so_far = self._current_proxy_b
return result
def _per_bar_hook(self, ri, vd, prices, v50, v750, vrok):
if self.position is None:
return
# Track proxy_B max during hold
self._pb_max_so_far = max(self._pb_max_so_far, self._current_proxy_b)
bars_held = self._global_bar_idx - self.position.entry_bar
min_bars = int(self.frac * self.exit_manager.max_hold_bars)
if (bars_held >= min_bars
and self._pb_max_so_far > self._pb_hold_threshold
and self.position.asset in prices):
self.position.current_price = prices[self.position.asset]
self._execute_exit("PROXY_HOLD_LIMIT", self._global_bar_idx,
bars_held=bars_held)
self._pb_max_so_far = 0.0
self._pb_hold_threshold = float('inf')
self.early_exits += 1
def reset(self):
super().reset()
self._pb_hold_threshold = float('inf')
self._pb_max_so_far = 0.0
self.early_exits = 0
# ── Mode D: rising_exit ───────────────────────────────────────────────────────
class ProxyRisingExitEngine(ProxyBaseEngine):
"""
Exit early when:
- current proxy_B exceeds thr_pct-percentile of rolling history
- AND bars_held >= frac * max_hold_bars
Interpretation: proxy_B has risen to an elevated level during hold →
eigenspace stress is actively high → exit before MAE deepens further.
Threshold computed at entry time from rolling proxy_B history.
"""
def __init__(self, *args, frac: float = 0.5, thr_pct: float = 0.80, **kwargs):
super().__init__(*args, **kwargs)
self.frac = frac
self.thr_pct = thr_pct
self._pb_entry_threshold: float = float('inf') # set at entry
self.early_exits: int = 0
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
if result:
self._pb_entry_threshold = self._proxy_threshold(self.thr_pct)
return result
def _per_bar_hook(self, ri, vd, prices, v50, v750, vrok):
if self.position is None:
return
bars_held = self._global_bar_idx - self.position.entry_bar
min_bars = int(self.frac * self.exit_manager.max_hold_bars)
if (bars_held >= min_bars
and self._current_proxy_b > self._pb_entry_threshold
and self.position.asset in prices):
self.position.current_price = prices[self.position.asset]
self._execute_exit("PROXY_RISING_EXIT", self._global_bar_idx,
bars_held=bars_held)
self._pb_entry_threshold = float('inf')
self.early_exits += 1
def reset(self):
super().reset()
self._pb_entry_threshold = float('inf')
self.early_exits = 0
# ── Run harness ───────────────────────────────────────────────────────────────
def _run(engine_factory, name, d, fw):
"""Full 55-day backtest. Returns gold-comparable metrics + coupling-specific stats."""
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
for pf in d['parquet_files']:
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
if n == 0:
return dict(name=name, roi=roi, pf=0.0, dd=0.0, wr=0.0, sharpe=0.0,
trades=0, early_exits=0, sizing_scale_mean=1.0, exit_reasons={})
def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
exit_reasons = {}
for t in tr:
r = t.exit_reason if hasattr(t, 'exit_reason') else 'UNKNOWN'
exit_reasons[r] = exit_reasons.get(r, 0) + 1
early_exits = getattr(eng, 'early_exits', 0)
sizing_scale_mean = getattr(eng, 'sizing_scale_mean', 1.0)
return dict(
name=name, roi=roi, pf=pf, dd=max_dd, wr=wr, sharpe=sharpe, trades=n,
early_exits=early_exits,
early_exit_rate=early_exits / n if n > 0 else 0.0,
sizing_scale_mean=sizing_scale_mean,
exit_reasons=exit_reasons,
)
# ── Main ──────────────────────────────────────────────────────────────────────
def main():
t_start = time.time()
print("=" * 72)
print("Exp 7 — Live proxy_B Coupling: Scale + Hold-Limit + Rising-Exit")
print("Note: First valid live test — exp4 retroactive was 98% median-filled.")
print("=" * 72)
ensure_jit()
d = load_data()
fw = load_forewarner()
configs = []
# Baseline
configs.append(("A00_baseline", lambda kw: NDAlphaEngine(**kw)))
# Mode B: scale_boost — size UP when proxy_B is LOW
for thr in [0.25, 0.35]:
for alpha in [0.5, 1.0]:
tag = f"B_boost_thr={thr:.2f}_a={alpha:.1f}"
def _make_boost(t, a):
return lambda kw: ProxyScaleEngine(mode='boost', threshold=t, alpha=a, **kw)
configs.append((tag, _make_boost(thr, alpha)))
# Mode A: scale_suppress — size DOWN when proxy_B is HIGH
for thr in [0.75, 0.85]:
for alpha in [0.5, 1.0]:
tag = f"A_suppress_thr={thr:.2f}_a={alpha:.1f}_smin=0.25"
def _make_suppress(t, a):
return lambda kw: ProxyScaleEngine(mode='suppress', threshold=t, alpha=a, s_min=0.25, **kw)
configs.append((tag, _make_suppress(thr, alpha)))
# Mode C: hold_limit — exit early when pb_max during hold exceeds threshold
for thr_pct in [0.65, 0.75, 0.85]:
tag = f"C_hold_limit_frac=0.5_thr={thr_pct:.2f}"
def _make_hold(tp):
return lambda kw: ProxyHoldLimitEngine(frac=0.5, thr_pct=tp, **kw)
configs.append((tag, _make_hold(thr_pct)))
# Mode D: rising_exit — exit early when current pb exceeds threshold during hold
for thr_pct in [0.65, 0.75, 0.85]:
tag = f"D_rising_exit_frac=0.5_thr={thr_pct:.2f}"
def _make_rising(tp):
return lambda kw: ProxyRisingExitEngine(frac=0.5, thr_pct=tp, **kw)
configs.append((tag, _make_rising(thr_pct)))
results = []
for i, (name, factory) in enumerate(configs):
t0 = time.time()
print(f"\n[{i+1}/{len(configs)}] {name} ...")
res = _run(factory, name, d, fw)
elapsed = time.time() - t0
extra = ""
if res['early_exits'] > 0:
extra = f" early_exits={res['early_exits']}"
if abs(res['sizing_scale_mean'] - 1.0) > 0.001:
extra += f" scale_mean={res['sizing_scale_mean']:.4f}"
print(f" ROI={res['roi']:.2f}% PF={res['pf']:.4f} DD={res['dd']:.2f}% "
f"WR={res['wr']:.2f}% Sharpe={res['sharpe']:.3f} Trades={res['trades']}"
f"{extra} ({elapsed:.0f}s)")
results.append(res)
# Verification
baseline = results[0]
gold_match = (
abs(baseline['roi'] - GOLD['roi']) < 0.5 and
abs(baseline['pf'] - GOLD['pf']) < 0.005 and
abs(baseline['dd'] - GOLD['dd']) < 0.5 and
abs(baseline['trades'] - GOLD['trades']) < 10
)
print(f"\n{'='*72}")
print(f"VERIFICATION: baseline ROI={baseline['roi']:.2f}% DD={baseline['dd']:.2f}%"
f" Trades={baseline['trades']} Match={'PASS ✓' if gold_match else 'FAIL ✗'}")
base_roi = baseline['roi']
base_dd = baseline['dd']
target_roi = GOLD['roi'] * 0.95 # 84.12%
target_dd = GOLD['dd'] # 15.05%
print(f"\n{'='*72}")
print(f"RESULTS TABLE (target: DD<{target_dd:.2f}% AND ROI>={target_roi:.1f}%)")
hdr = f"{'Config':<46} {'ROI%':>7} {'PF':>6} {'DD%':>6} {'ΔDD':>6} {'ΔROI':>6} {'extra':>20} {'OK':>4}"
print(hdr)
print('-' * 100)
for r in results:
delta_roi = r['roi'] - base_roi
delta_dd = r['dd'] - base_dd
ok = 'Y' if (r['dd'] < target_dd and r['roi'] >= target_roi) else 'N'
extra = ''
if r['early_exits'] > 0:
extra = f"early={r['early_exits']}"
elif abs(r['sizing_scale_mean'] - 1.0) > 0.001:
extra = f"scale={r['sizing_scale_mean']:.4f}"
print(f"{r['name']:<46} {r['roi']:>7.2f} {r['pf']:>6.4f} {r['dd']:>6.2f} "
f"{delta_dd:>+6.2f} {delta_roi:>+6.2f} {extra:>20} {ok:>4}")
# Summary by mode
print(f"\n{'='*72}")
print("BEST PER MODE:")
def best_dd(group):
return min(group, key=lambda r: r['dd'])
groups = {'baseline': [], 'B_boost': [], 'A_suppress': [],
'C_hold_limit': [], 'D_rising_exit': []}
for r in results:
for k in groups:
if r['name'].startswith(k.split('_')[0]) or r['name'].startswith(k):
groups[k].append(r); break
for mode, rs in groups.items():
if not rs: continue
b = best_dd(rs)
delta_roi = b['roi'] - base_roi
delta_dd = b['dd'] - base_dd
ok = 'Y' if (b['dd'] < target_dd and b['roi'] >= target_roi) else 'N'
print(f" {mode:<16} best: {b['name']:<46} "
f"DD={b['dd']:.2f}% ({delta_dd:+.2f}) ROI={b['roi']:.2f}% ({delta_roi:+.2f}) [{ok}]")
# Exit breakdown
print(f"\n{'='*72}")
print("EXIT REASON BREAKDOWN:")
for r in results:
reasons = r.get('exit_reasons', {})
parts = ', '.join(f"{k}={v}" for k, v in sorted(reasons.items()))
print(f" {r['name']:<46}: {parts}")
# Save
outfile = _HERE / "exp7_live_coupling_results.json"
log_results(results, outfile, gold=GOLD, meta={
"exp": "exp7",
"note": "First valid live test of modes A/B/C/D — exp4 retroactive was invalid (entry_bar bug)",
"total_elapsed_s": round(time.time() - t_start, 1),
"gold_match": gold_match,
"modes_tested": ["B_scale_boost", "A_scale_suppress", "C_hold_limit", "D_rising_exit"],
})
total = time.time() - t_start
print(f"\nTotal elapsed: {total/60:.1f} min")
print("Done.")
if __name__ == "__main__":
main()

268
nautilus_dolphin/dvae/exp7_live_coupling_results.json Executable file
View File

@@ -0,0 +1,268 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"name": "A00_baseline",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 1.0,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "B_boost_thr=0.25_a=0.5",
"roi": 90.29753941684339,
"pf": 1.2160924709583105,
"dd": 14.821932254796943,
"wr": 50.48723897911833,
"sharpe": 4.4424955513000555,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 1.0192115027829314,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "B_boost_thr=0.25_a=1.0",
"roi": 92.0514388467092,
"pf": 1.2174432679504197,
"dd": 14.597213085690708,
"wr": 50.48723897911833,
"sharpe": 4.503982033834149,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 1.0384230055658628,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "B_boost_thr=0.35_a=0.5",
"roi": 91.07372858237454,
"pf": 1.2163247655700182,
"dd": 14.779598850291558,
"wr": 50.48723897911833,
"sharpe": 4.468643300574364,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 1.0309155844155844,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "B_boost_thr=0.35_a=1.0",
"roi": 93.60513448956341,
"pf": 1.2178720236994938,
"dd": 14.512635180033598,
"wr": 50.48723897911833,
"sharpe": 4.55213195481462,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 1.0618311688311688,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "A_suppress_thr=0.75_a=0.5_smin=0.25",
"roi": 85.79811260804453,
"pf": 1.219430737394839,
"dd": 15.170218464648395,
"wr": 50.48723897911833,
"sharpe": 4.408411370917985,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 0.9757196349867308,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "A_suppress_thr=0.75_a=1.0_smin=0.25",
"roi": 83.04374070648592,
"pf": 1.2243226504262263,
"dd": 15.294576696634937,
"wr": 50.48723897911833,
"sharpe": 4.4289241379844855,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 0.9514392699734614,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "A_suppress_thr=0.85_a=0.5_smin=0.25",
"roi": 87.17621721501192,
"pf": 1.2165304302812532,
"dd": 15.089608437921182,
"wr": 50.48723897911833,
"sharpe": 4.367963006765593,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 0.9898321240014754,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "A_suppress_thr=0.85_a=1.0_smin=0.25",
"roi": 85.80516356740421,
"pf": 1.2183755554509148,
"dd": 15.133102187143244,
"wr": 50.48723897911833,
"sharpe": 4.355216195176627,
"trades": 2155,
"early_exits": 0,
"early_exit_rate": 0.0,
"sizing_scale_mean": 0.9796642480029509,
"exit_reasons": {
"MAX_HOLD": 1831,
"FIXED_TP": 324
}
},
{
"name": "C_hold_limit_frac=0.5_thr=0.65",
"roi": 0.9592460335863725,
"pf": 1.0026219317176224,
"dd": 18.646234282343325,
"wr": 48.607446496628555,
"sharpe": 0.0679900506250511,
"trades": 3411,
"early_exits": 3119,
"early_exit_rate": 0.9143946056874817,
"sizing_scale_mean": 1.0,
"exit_reasons": {
"PROXY_HOLD_LIMIT": 3119,
"FIXED_TP": 292
}
},
{
"name": "C_hold_limit_frac=0.5_thr=0.75",
"roi": 0.7052670847762929,
"pf": 1.0019280264886439,
"dd": 18.646234282343325,
"wr": 48.54881266490765,
"sharpe": 0.04992241140682829,
"trades": 3411,
"early_exits": 3118,
"early_exit_rate": 0.9141014365288772,
"sizing_scale_mean": 1.0,
"exit_reasons": {
"PROXY_HOLD_LIMIT": 3118,
"FIXED_TP": 292,
"MAX_HOLD": 1
}
},
{
"name": "C_hold_limit_frac=0.5_thr=0.85",
"roi": 6.409886467879581,
"pf": 1.0168573807902554,
"dd": 18.6546936527647,
"wr": 48.31031442844549,
"sharpe": 0.43074844271123575,
"trades": 3403,
"early_exits": 3106,
"early_exit_rate": 0.9127240669997061,
"sizing_scale_mean": 1.0,
"exit_reasons": {
"PROXY_HOLD_LIMIT": 3106,
"FIXED_TP": 291,
"MAX_HOLD": 6
}
},
{
"name": "D_rising_exit_frac=0.5_thr=0.65",
"roi": -10.113161555686325,
"pf": 0.97085496374074,
"dd": 20.80672837099466,
"wr": 48.660042155977116,
"sharpe": -0.8631426863593179,
"trades": 3321,
"early_exits": 3030,
"early_exit_rate": 0.912375790424571,
"sizing_scale_mean": 1.0,
"exit_reasons": {
"PROXY_RISING_EXIT": 3030,
"FIXED_TP": 291
}
},
{
"name": "D_rising_exit_frac=0.5_thr=0.75",
"roi": -8.07049307300517,
"pf": 0.9758642421190082,
"dd": 21.290045839782977,
"wr": 48.26336454243431,
"sharpe": -0.7095429390652619,
"trades": 3311,
"early_exits": 3011,
"early_exit_rate": 0.9093929326487467,
"sizing_scale_mean": 1.0,
"exit_reasons": {
"PROXY_RISING_EXIT": 3011,
"FIXED_TP": 298,
"MAX_HOLD": 2
}
},
{
"name": "D_rising_exit_frac=0.5_thr=0.85",
"roi": -10.447002613929028,
"pf": 0.9700434423059277,
"dd": 21.932685654212428,
"wr": 48.518284993694834,
"sharpe": -0.8193432726522849,
"trades": 3172,
"early_exits": 2843,
"early_exit_rate": 0.8962799495586381,
"sizing_scale_mean": 1.0,
"exit_reasons": {
"PROXY_RISING_EXIT": 2843,
"FIXED_TP": 287,
"MAX_HOLD": 42
}
}
],
"meta": {
"exp": "exp7",
"note": "First valid live test of modes A/B/C/D \u2014 exp4 retroactive was invalid (entry_bar bug)",
"total_elapsed_s": 3650.6,
"gold_match": true,
"modes_tested": [
"B_scale_boost",
"A_scale_suppress",
"C_hold_limit",
"D_rising_exit"
]
}
}

425
nautilus_dolphin/dvae/exp8_boost_robustness.py Executable file
View File

@@ -0,0 +1,425 @@
"""
Exp 8 — scale_boost Robustness & Adaptive Parameterization
Two questions from exp7 scale_boost winner (thr=0.35, a=1.0):
Q1. Is it overfitting? (+5pp ROI AND -0.54pp DD on same 55 days it was found)
Test: temporal split — first-half (days 1–27) vs second-half (days 28–55)
If improvement holds in BOTH halves independently, it's structurally real.
If only one half drives it, the result is temporally fragile.
Q2. Are threshold and alpha regime-dependent?
Hypothesis: proxy_B is more discriminating in high-eigenvalue-regime days
(high ACB beta). On those days, "calm" entries should receive stronger boost,
and the threshold for "what qualifies as calm" should be tighter.
Adaptive formulas (using ACB state available in _try_entry as self._day_base_boost
and self._day_beta):
alpha_eff = alpha * day_base_boost (more boost on stressed days)
thr_eff = threshold / day_base_boost (tighter gate on stressed days)
Both together: combine both adjustments
Also test dvol-proxy adaptation: use day_beta directly as a continuous scaler.
Configs:
0. Baseline
1. Fixed: thr=0.35, a=1.0 (exp7 winner — must reproduce exp7 results)
2. Adaptive-alpha: alpha_eff = 1.0 * day_base_boost, thr fixed at 0.35
3. Adaptive-threshold: thr_eff = 0.35 / day_base_boost, alpha fixed at 1.0
4. Adaptive-both: both formulas combined
5. Beta-scaled alpha: alpha_eff = 1.0 * (1 + day_beta), thr fixed at 0.35
(day_beta is the ACB eigenvalue signal; more direct than base_boost)
Results include:
- Full 55-day metrics (standard)
- First-half (days 1–27) and second-half (days 28–55) metrics split out
to test temporal stability of the DD reduction
- Per-day scale distribution analysis
Results logged to exp8_boost_robustness_results.json
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
load_data, load_forewarner, log_results,
)
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# ── Re-use ProxyBaseEngine from exp7 (copy-minimal) ──────────────────────────
class ProxyBaseEngine(NDAlphaEngine):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._current_proxy_b: float = 0.0
self._proxy_b_history: list = []
def _update_proxy(self, inst: float, v750: float) -> float:
pb = inst - v750
self._current_proxy_b = pb
self._proxy_b_history.append(pb)
if len(self._proxy_b_history) > 500:
self._proxy_b_history = self._proxy_b_history[-500:]
return pb
def _proxy_prank(self) -> float:
if not self._proxy_b_history:
return 0.5
n = len(self._proxy_b_history)
return sum(v < self._current_proxy_b for v in self._proxy_b_history) / n
def process_day(self, date_str, df, asset_columns,
vol_regime_ok=None, direction=None, posture='APEX'):
self.begin_day(date_str, posture=posture, direction=direction)
bid = 0
for ri in range(len(df)):
row = df.iloc[ri]
vd = row.get('vel_div')
if vd is None or not np.isfinite(float(vd)):
self._global_bar_idx += 1; bid += 1; continue
def gf(col):
v = row.get(col)
if v is None: return 0.0
try: return float(v) if np.isfinite(float(v)) else 0.0
except: return 0.0
v50 = gf('v50_lambda_max_velocity')
v750 = gf('v750_lambda_max_velocity')
inst = gf('instability_50')
self._update_proxy(inst, v750)
prices = {}
for ac in asset_columns:
p = row.get(ac)
if p is not None and p > 0 and np.isfinite(float(p)):
prices[ac] = float(p)
if not prices:
self._global_bar_idx += 1; bid += 1; continue
vrok = bool(vol_regime_ok[ri]) if vol_regime_ok is not None else (bid >= 100)
self.step_bar(bar_idx=ri, vel_div=float(vd), prices=prices,
vol_regime_ok=vrok, v50_vel=v50, v750_vel=v750)
bid += 1
return self.end_day()
# ── Adaptive scale_boost engine ───────────────────────────────────────────────
class AdaptiveBoostEngine(ProxyBaseEngine):
"""
scale_boost with optionally regime-adaptive threshold and alpha.
Fixed mode (adaptive_alpha=False, adaptive_thr=False, adaptive_beta=False):
scale = 1 + alpha * max(0, threshold - prank)
Identical to exp7 ProxyScaleEngine(mode='boost').
Adaptive modes use ACB state (self._day_base_boost, self._day_beta)
which is set by begin_day() before any _try_entry calls in that day:
adaptive_alpha: alpha_eff = alpha * day_base_boost
→ High-boost day (stressed eigenspace regime) → stronger boost on calm entries
→ Low-boost day → modest boost
adaptive_thr: thr_eff = threshold / day_base_boost
→ High-boost day → lower threshold → more selective (only very calm entries qualify)
→ Low-boost day → higher threshold → more entries qualify
adaptive_beta: alpha_eff = alpha * (1 + day_beta)
→ day_beta is the ACB's direct eigenvalue signal (0 when inactive)
→ More discriminating on days where eigenvalue regime is active
Parameters can be combined freely.
"""
def __init__(self, *args,
threshold: float = 0.35,
alpha: float = 1.0,
adaptive_alpha: bool = False,
adaptive_thr: bool = False,
adaptive_beta: bool = False,
**kwargs):
super().__init__(*args, **kwargs)
self.threshold = threshold
self.alpha = alpha
self.adaptive_alpha = adaptive_alpha
self.adaptive_thr = adaptive_thr
self.adaptive_beta = adaptive_beta
self._scale_history: list = []
self._alpha_eff_history: list = []
self._thr_eff_history: list = []
@property
def sizing_scale_mean(self) -> float:
return float(np.mean(self._scale_history)) if self._scale_history else 1.0
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
if result and self.position:
boost = max(1.0, getattr(self, '_day_base_boost', 1.0))
beta = max(0.0, getattr(self, '_day_beta', 0.0))
# Effective parameters
alpha_eff = self.alpha
if self.adaptive_alpha:
alpha_eff *= boost # more boost on stressed-regime days
if self.adaptive_beta:
alpha_eff *= (1.0 + beta) # beta signal scales aggression
thr_eff = self.threshold
if self.adaptive_thr:
# High boost → lower threshold → be more selective about "calm"
thr_eff = self.threshold / max(1.0, boost)
prank = self._proxy_prank()
scale = 1.0 + alpha_eff * max(0.0, thr_eff - prank)
self.position.notional *= scale
self._scale_history.append(scale)
self._alpha_eff_history.append(alpha_eff)
self._thr_eff_history.append(thr_eff)
return result
def reset(self):
super().reset()
self._scale_history = []
self._alpha_eff_history = []
self._thr_eff_history = []
# ── Run harness with half-split ───────────────────────────────────────────────
def _run(engine_factory, name, d, fw):
"""Full run + temporal split (first vs second half of days)."""
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
pf_list = d['parquet_files']
n_days = len(pf_list)
half = n_days // 2 # split point
daily_caps, daily_pnls = [], []
half_caps = [[], []] # [first_half, second_half]
half_pnls = [[], []]
half_trades_n = [0, 0]
for i, pf in enumerate(pf_list):
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
cap_after = eng.capital
daily_caps.append(cap_after)
daily_pnls.append(cap_after - cap_before)
h = 0 if i < half else 1
half_caps[h].append(cap_after)
half_pnls[h].append(cap_after - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
def _metrics(caps, pnls, start_cap=25000.0):
"""Compute metrics for a sub-period given daily capitals and a starting capital."""
if not caps:
return dict(roi=0.0, dd=0.0, sharpe=0.0)
peak = start_cap
max_dd = 0.0
for c in caps:
peak = max(peak, c)
max_dd = max(max_dd, (peak - c) / peak * 100.0)
total_pnl = sum(pnls)
roi_sub = total_pnl / start_cap * 100.0
dr = np.array([p / start_cap * 100.0 for p in pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
return dict(roi=roi_sub, dd=max_dd, sharpe=sharpe, n_days=len(caps))
if n == 0:
return dict(name=name, roi=roi, pf=0.0, dd=0.0, wr=0.0, sharpe=0.0,
trades=0, sizing_scale_mean=1.0)
def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf_val = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
# First/second half split — using capital at end of first-half as baseline for second half
cap_at_halftime = half_caps[0][-1] if half_caps[0] else 25000.0
h1 = _metrics(half_caps[0], half_pnls[0], start_cap=25000.0)
h2 = _metrics(half_caps[1], half_pnls[1], start_cap=cap_at_halftime)
sizing_scale_mean = getattr(eng, 'sizing_scale_mean', 1.0)
# Alpha/threshold eff distributions for adaptive engines
alpha_mean = 1.0
thr_mean = 0.35
eng_ae = eng if isinstance(eng, AdaptiveBoostEngine) else None
if eng_ae:
if eng_ae._alpha_eff_history:
alpha_mean = float(np.mean(eng_ae._alpha_eff_history))
if eng_ae._thr_eff_history:
thr_mean = float(np.mean(eng_ae._thr_eff_history))
return dict(
name=name,
roi=roi, pf=pf_val, dd=max_dd, wr=wr, sharpe=sharpe, trades=n,
sizing_scale_mean=sizing_scale_mean,
alpha_eff_mean=alpha_mean,
thr_eff_mean=thr_mean,
# Temporal split
h1_roi=h1['roi'], h1_dd=h1['dd'], h1_sharpe=h1['sharpe'],
h2_roi=h2['roi'], h2_dd=h2['dd'], h2_sharpe=h2['sharpe'],
split_days=(half, n_days - half),
)
# ── Main ──────────────────────────────────────────────────────────────────────
def main():
t_start = time.time()
print("=" * 74)
print("Exp 8 — scale_boost Robustness & Adaptive Parameterization")
print("=" * 74)
ensure_jit()
d = load_data()
fw = load_forewarner()
configs = [
("0_baseline",
lambda kw: NDAlphaEngine(**kw)),
("1_fixed_thr035_a1.0",
lambda kw: AdaptiveBoostEngine(threshold=0.35, alpha=1.0, **kw)),
("2_adaptive_alpha__thr035_a1.0xboost",
lambda kw: AdaptiveBoostEngine(threshold=0.35, alpha=1.0,
adaptive_alpha=True, **kw)),
("3_adaptive_thr__thr035/boost_a1.0",
lambda kw: AdaptiveBoostEngine(threshold=0.35, alpha=1.0,
adaptive_thr=True, **kw)),
("4_adaptive_both__thr/boost_axboost",
lambda kw: AdaptiveBoostEngine(threshold=0.35, alpha=1.0,
adaptive_alpha=True, adaptive_thr=True, **kw)),
("5_adaptive_beta__thr035_ax(1+beta)",
lambda kw: AdaptiveBoostEngine(threshold=0.35, alpha=1.0,
adaptive_beta=True, **kw)),
]
results = []
for i, (name, factory) in enumerate(configs):
t0 = time.time()
print(f"\n[{i+1}/{len(configs)}] {name} ...")
res = _run(factory, name, d, fw)
elapsed = time.time() - t0
print(f" ROI={res['roi']:.2f}% PF={res['pf']:.4f} DD={res['dd']:.2f}%"
f" WR={res['wr']:.2f}% Sharpe={res['sharpe']:.3f} Trades={res['trades']}"
f" scale={res['sizing_scale_mean']:.4f} alpha_eff={res['alpha_eff_mean']:.4f}"
f" ({elapsed:.0f}s)")
print(f" H1(days 1-{res['split_days'][0]}): ROI={res['h1_roi']:.2f}%"
f" DD={res['h1_dd']:.2f}% Sharpe={res['h1_sharpe']:.3f}")
print(f" H2(days {res['split_days'][0]+1}-{sum(res['split_days'])}): ROI={res['h2_roi']:.2f}%"
f" DD={res['h2_dd']:.2f}% Sharpe={res['h2_sharpe']:.3f}")
results.append(res)
# Baseline verification
b = results[0]
fixed = results[1]
gold_match = (abs(b['roi'] - GOLD['roi']) < 0.5 and abs(b['dd'] - GOLD['dd']) < 0.5
and abs(b['trades'] - GOLD['trades']) < 10)
fixed_match = (abs(fixed['roi'] - 93.61) < 0.5 and abs(fixed['dd'] - 14.51) < 0.5)
print(f"\n{'='*74}")
print(f"VERIFICATION:")
print(f" Baseline vs gold: {'PASS ✓' if gold_match else 'FAIL ✗'} "
f"(ROI={b['roi']:.2f}% DD={b['dd']:.2f}%)")
print(f" Fixed vs exp7 winner: {'PASS ✓' if fixed_match else 'FAIL ✗'} "
f"(ROI={fixed['roi']:.2f}% DD={fixed['dd']:.2f}%)")
print(f"\n{'='*74}")
print(f"FULL-PERIOD RESULTS (target: DD<15.05% AND ROI>=84.1%)")
hdr = f"{'Config':<46} {'ROI%':>7} {'PF':>6} {'DD%':>6} {'ΔDD':>6} {'ΔROI':>6} {'scale':>7} {'alpha':>7} {'OK':>4}"
print(hdr); print('-' * 98)
base_roi = b['roi']; base_dd = b['dd']
for r in results:
dROI = r['roi'] - base_roi; dDD = r['dd'] - base_dd
ok = 'Y' if (r['dd'] < GOLD['dd'] and r['roi'] >= GOLD['roi'] * 0.95) else 'N'
print(f"{r['name']:<46} {r['roi']:>7.2f} {r['pf']:>6.4f} {r['dd']:>6.2f} "
f"{dDD:>+6.2f} {dROI:>+6.2f} {r['sizing_scale_mean']:>7.4f} "
f"{r['alpha_eff_mean']:>7.4f} {ok:>4}")
print(f"\n{'='*74}")
print("TEMPORAL SPLIT — Overfitting check (does improvement hold in both halves?)")
h_days = results[0]['split_days']
print(f"Split: H1=days 1–{h_days[0]}, H2=days {h_days[0]+1}–{sum(h_days)}")
print(f"{'Config':<46} {'H1 ROI':>8} {'H1 DD':>7} {'H2 ROI':>8} {'H2 DD':>7} "
f"{'ΔH1DD':>7} {'ΔH2DD':>7}")
print('-' * 98)
b_h1dd = b['h1_dd']; b_h2dd = b['h2_dd']
for r in results:
dH1 = r['h1_dd'] - b_h1dd; dH2 = r['h2_dd'] - b_h2dd
print(f"{r['name']:<46} {r['h1_roi']:>8.2f} {r['h1_dd']:>7.2f} "
f"{r['h2_roi']:>8.2f} {r['h2_dd']:>7.2f} {dH1:>+7.2f} {dH2:>+7.2f}")
print(f"\n{'='*74}")
print("OVERFITTING VERDICT:")
for r in results[1:]:
h1_better = r['h1_dd'] < b_h1dd
h2_better = r['h2_dd'] < b_h2dd
both = h1_better and h2_better
neither = (not h1_better) and (not h2_better)
verdict = "BOTH halves improve DD ✓" if both else \
"NEITHER half improves DD ✗" if neither else \
f"Mixed: H1={'↓' if h1_better else '↑'} H2={'↓' if h2_better else '↑'}"
print(f" {r['name']:<46}: {verdict}")
# Adaptive summary
print(f"\n{'='*74}")
print("ADAPTIVE PARAMETERIZATION — alpha_eff and thr_eff distributions:")
for r in results[2:]:
print(f" {r['name']:<46}: alpha_eff_mean={r['alpha_eff_mean']:.4f}"
f" thr_eff_mean={r['thr_eff_mean']:.4f}")
outfile = _HERE / "exp8_boost_robustness_results.json"
log_results(results, outfile, gold=GOLD, meta={
"exp": "exp8",
"question": "Is scale_boost overfitting? Are threshold/alpha regime-dependent?",
"total_elapsed_s": round(time.time() - t_start, 1),
"gold_match": gold_match,
"fixed_match": fixed_match,
})
total = time.time() - t_start
print(f"\nTotal elapsed: {total/60:.1f} min")
print("Done.")
if __name__ == "__main__":
main()

151
nautilus_dolphin/dvae/exp8_boost_robustness_results.json Executable file
View File

@@ -0,0 +1,151 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"name": "0_baseline",
"roi": 88.54671933603525,
"pf": 1.21470506157439,
"dd": 15.046245386522427,
"wr": 50.48723897911833,
"sharpe": 4.378300370204196,
"trades": 2155,
"sizing_scale_mean": 1.0,
"alpha_eff_mean": 1.0,
"thr_eff_mean": 0.35,
"h1_roi": 37.632584444436795,
"h1_dd": 15.046245386522427,
"h1_sharpe": 3.9603631793850136,
"h2_roi": 36.992791421534946,
"h2_dd": 9.830290816036223,
"h2_sharpe": 4.769183844351268,
"split_days": [
28,
28
]
},
{
"name": "1_fixed_thr035_a1.0",
"roi": 93.60513448956341,
"pf": 1.2178720236994938,
"dd": 14.512635180033598,
"wr": 50.48723897911833,
"sharpe": 4.55213195481462,
"trades": 2155,
"sizing_scale_mean": 1.0618311688311688,
"alpha_eff_mean": 1.0,
"thr_eff_mean": 0.3500000000000001,
"h1_roi": 37.87397445144304,
"h1_dd": 14.512635180033598,
"h1_sharpe": 3.8950967482396135,
"h2_roi": 40.42181293449837,
"h2_dd": 9.52090561815603,
"h2_sharpe": 5.164864302890685,
"split_days": [
28,
28
]
},
{
"name": "2_adaptive_alpha__thr035_a1.0xboost",
"roi": 93.40183251473115,
"pf": 1.21653391966412,
"dd": 14.512635180033612,
"wr": 50.48723897911833,
"sharpe": 4.538463370694225,
"trades": 2155,
"sizing_scale_mean": 1.069598136684172,
"alpha_eff_mean": 1.1152435704345516,
"thr_eff_mean": 0.3500000000000001,
"h1_roi": 37.863444837785416,
"h1_dd": 14.512635180033612,
"h1_sharpe": 3.863961184840122,
"h2_roi": 40.285071755093604,
"h2_dd": 9.653060108073454,
"h2_sharpe": 5.171823779134293,
"split_days": [
28,
28
]
},
{
"name": "3_adaptive_thr__thr035/boost_a1.0",
"roi": 94.13033277329988,
"pf": 1.2198467641016866,
"dd": 14.512635180033612,
"wr": 50.48723897911833,
"sharpe": 4.577159191163701,
"trades": 2155,
"sizing_scale_mean": 1.0548980466896074,
"alpha_eff_mean": 1.0,
"thr_eff_mean": 0.32186136027605455,
"h1_roi": 38.028871469969275,
"h1_dd": 14.512635180033612,
"h1_sharpe": 3.934631717815155,
"h2_roi": 40.6447294003534,
"h2_dd": 9.419616583596177,
"h2_sharpe": 5.173301424346994,
"split_days": [
28,
28
]
},
{
"name": "4_adaptive_both__thr/boost_axboost",
"roi": 94.11394527938835,
"pf": 1.2192367820323635,
"dd": 14.51263518003362,
"wr": 50.48723897911833,
"sharpe": 4.5739830546529925,
"trades": 2155,
"sizing_scale_mean": 1.0596575120382845,
"alpha_eff_mean": 1.1152435704345516,
"thr_eff_mean": 0.32186136027605455,
"h1_roi": 38.07558919307389,
"h1_dd": 14.51263518003362,
"h1_sharpe": 3.9200708724235827,
"h2_roi": 40.58527391685063,
"h2_dd": 9.471525859621199,
"h2_sharpe": 5.182956035866156,
"split_days": [
28,
28
]
},
{
"name": "5_adaptive_beta__thr035_ax(1+beta)",
"roi": 96.55083977704362,
"pf": 1.2201203038704769,
"dd": 14.321954772270876,
"wr": 50.48723897911833,
"sharpe": 4.631491961453057,
"trades": 2155,
"sizing_scale_mean": 1.0883296846011132,
"alpha_eff_mean": 1.428756957328386,
"thr_eff_mean": 0.3500000000000001,
"h1_roi": 38.80519726574197,
"h1_dd": 14.321954772270876,
"h1_sharpe": 3.9158586921478613,
"h2_roi": 41.60193108673579,
"h2_dd": 9.569030250306442,
"h2_sharpe": 5.302551495714016,
"split_days": [
28,
28
]
}
],
"meta": {
"exp": "exp8",
"question": "Is scale_boost overfitting? Are threshold/alpha regime-dependent?",
"total_elapsed_s": 1401.0,
"gold_match": true,
"fixed_match": true
}
}

408
nautilus_dolphin/dvae/exp9_leverage_ceiling.py Executable file
View File

@@ -0,0 +1,408 @@
"""
Exp 9 — Extended Leverage Ceiling Test
Motivation:
GOLD (adaptive_beta) achieved DD=14.32% (−0.72pp vs Silver=15.05%). The question is whether
this headroom permits raising the leverage ceiling above the current 5x soft / 6x hard cap
without exceeding Silver's DD budget — or ideally staying below GOLD's DD while gaining ROI.
Architecture recap (DO NOT MODIFY any of the below):
bet_sizer.max_leverage (5.0) → convex curve soft cap — strength_score^3 tops out here
base_max_leverage (5.0) → base for ACB regime excursions
abs_max_leverage (6.0) → hard ceiling: min(base*regime_mult, abs_max)
ACB regime_size_mult (1.0–1.6) → can push raw_leverage above bet_sizer.max_leverage
→ Current actual range: 0.5x – 6.0x (6x hard cap is binding on extreme signals)
Fork strategy:
ExtendedLeverageEngine(AdaptiveBoostEngine) — subclass only, zero parent modification.
Sets all three caps consistently: bet_sizer.max_leverage = base_max_leverage = extended_soft_cap,
abs_max_leverage = extended_abs_cap.
Example at 7.0/8.0: extreme signal gets bet_sizer→7.0 × regime_mult≈1.2 = 8.4, clamped at 8.0.
MC-Forewarner interaction (CRITICAL):
begin_day() feeds MC: mc_cfg['max_leverage'] = base_max_leverage * day_base_boost
MC was trained at max_leverage 5.0–6.0 (champion region). At 8x+ it's outside training
distribution → One-Class SVM envelope_score likely < -1.0 → RED → regime_dd_halt=True
→ ZERO trades that day.
Two modes tested:
mc_ref=5.0 ("decoupled"): MC sees original 5.0x reference → same verdicts as GOLD
Pure effect of higher ceiling with MC behavior unchanged
mc_ref=soft_cap ("coupled"): MC sees actual new cap → may flag RED/ORANGE on more days
Tests "what would MC do if it knew about the new leverage"
ExtendedLeverageEngine.begin_day() temporarily swaps base_max_leverage to mc_leverage_ref
before calling super().begin_day(), then restores extended caps for actual trading.
This is the ONLY divergence from parent logic.
Test configs:
A: 5.0/6.0 mc_ref=5.0 — GOLD reference (must reproduce 96.55% / 14.32%)
B: 6.0/7.0 mc_ref=5.0 — +1x soft, MC decoupled
C: 7.0/8.0 mc_ref=5.0 — +2x soft, MC decoupled
D: 8.0/9.0 mc_ref=5.0 — +3x soft, MC decoupled
E: 9.0/10.0 mc_ref=5.0 — +4x soft, MC decoupled
F: 6.0/7.0 mc_ref=6.0 — +1x soft, MC coupled
G: 7.0/8.0 mc_ref=7.0 — +2x soft, MC coupled
H: 8.0/9.0 mc_ref=8.0 — +3x soft, MC coupled
Total: 8 runs × ~250s ≈ 35 min
Key metrics:
ROI%, DD%, PF, Trades, avg_leverage (mean realized leverage per trade)
MC status breakdown: RED days / ORANGE days / OK days / halted days
Δ vs GOLD: ΔROI, ΔDD
Results → exp9_leverage_ceiling_results.json
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
load_data, load_forewarner, log_results,
)
from nautilus_dolphin.nautilus.proxy_boost_engine import AdaptiveBoostEngine, DEFAULT_THRESHOLD, DEFAULT_ALPHA
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# ── GOLD reference for this experiment (adaptive_beta, not the old silver) ───
_GOLD_EXP9 = dict(roi=96.55, dd=14.32, trades=2155) # adaptive_beta GOLD
_SILVER = dict(roi=88.55, dd=15.05, trades=2155) # former gold = regression floor
# ── ExtendedLeverageEngine — subclass only, ZERO parent modification ──────────
class ExtendedLeverageEngine(AdaptiveBoostEngine):
"""
Extended leverage ceiling fork of AdaptiveBoostEngine (adaptive_beta GOLD).
Three new constructor params vs parent:
extended_soft_cap : replaces max_leverage (5.0) and base_max_leverage
extended_abs_cap : replaces abs_max_leverage (6.0) — hard ceiling
mc_leverage_ref : what to feed the MC-Forewarner (independent of actual caps)
If None → MC sees extended_soft_cap (fully coupled)
Set to 5.0 → MC always sees original 5x reference (decoupled)
ALL existing leverage logic (convex curve, ACB regime_size_mult, STALKER 2x cap,
proxy_B scale_boost, etc.) is completely untouched. Only the three cap values change.
"""
def __init__(
self,
*args,
extended_soft_cap: float = 5.0,
extended_abs_cap: float = 6.0,
mc_leverage_ref: float = None,
**kwargs,
):
# Inject extended caps as max_leverage / abs_max_leverage so parent stores them
kwargs['max_leverage'] = extended_soft_cap
kwargs['abs_max_leverage'] = extended_abs_cap
super().__init__(*args, **kwargs)
# Explicitly ensure all three cap locations are consistent
# (parent __init__ sets base_max_leverage = max_leverage and bet_sizer.max_leverage
# from the kwargs, but we set explicitly for clarity)
self.bet_sizer.max_leverage = extended_soft_cap
self.base_max_leverage = extended_soft_cap
self.abs_max_leverage = extended_abs_cap
self._extended_soft_cap = extended_soft_cap
self._extended_abs_cap = extended_abs_cap
# MC reference: 5.0 = decoupled (MC sees original gold reference)
# extended_soft_cap = coupled (MC sees actual new cap)
self._mc_leverage_ref = mc_leverage_ref if mc_leverage_ref is not None else extended_soft_cap
# Per-day MC verdict monitoring
self.mc_monitor = dict(red=0, orange=0, ok=0, halted=0, total=0)
def begin_day(self, date_str: str, posture: str = 'APEX', direction=None) -> None:
"""
Temporarily expose mc_leverage_ref to the MC-Forewarner assessment,
then restore the true extended caps for actual trading.
The parent begin_day() computes:
mc_cfg['max_leverage'] = self.base_max_leverage * self._day_base_boost
By setting base_max_leverage = mc_leverage_ref before the call, MC sees the
reference leverage. After super().begin_day() returns, regime_dd_halt and
_day_mc_scale are already set correctly for that reference. We then restore
the true extended caps so _try_entry uses the full new ceiling.
"""
# Save true extended caps
_true_base = self.base_max_leverage
_true_abs = self.abs_max_leverage
_true_sizer = self.bet_sizer.max_leverage
# Temporarily expose mc_leverage_ref to parent's MC assessment
self.base_max_leverage = self._mc_leverage_ref
self.bet_sizer.max_leverage = self._mc_leverage_ref
self.abs_max_leverage = self._mc_leverage_ref # only affects _try_entry, safe to temp-set
super().begin_day(date_str, posture=posture, direction=direction)
# Restore true extended caps — all trading this day uses extended ceiling
self.base_max_leverage = _true_base
self.bet_sizer.max_leverage = _true_sizer
self.abs_max_leverage = _true_abs
# Record MC verdict for monitoring
self.mc_monitor['total'] += 1
status = self._day_mc_status
if status == 'RED':
self.mc_monitor['red'] += 1
elif status == 'ORANGE':
self.mc_monitor['orange'] += 1
else:
self.mc_monitor['ok'] += 1
if self.regime_dd_halt:
self.mc_monitor['halted'] += 1
def reset(self):
super().reset()
# NDAlphaEngine.reset() rebuilds bet_sizer from self.bet_sizer.max_leverage
# (which at reset time = _extended_soft_cap, so it rebuilds correctly).
# But we re-apply explicitly to be safe.
self.bet_sizer.max_leverage = self._extended_soft_cap
self.base_max_leverage = self._extended_soft_cap
self.abs_max_leverage = self._extended_abs_cap
# Reset monitoring
self.mc_monitor = dict(red=0, orange=0, ok=0, halted=0, total=0)
# ── Run harness ───────────────────────────────────────────────────────────────
def _run(engine_factory, name, d, fw):
"""Full 55-day run + MC monitoring + avg_leverage tracking."""
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
for pf in d['parquet_files']:
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
if n == 0:
mc_mon = getattr(eng, 'mc_monitor', {})
return dict(name=name, roi=roi, pf=0.0, dd=0.0, wr=0.0, sharpe=0.0,
trades=0, avg_leverage=0.0, sizing_scale_mean=1.0,
mc_monitor=mc_mon)
def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf_val = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
# Realized leverage stats
lev_vals = [t.leverage for t in tr if hasattr(t, 'leverage') and t.leverage > 0]
avg_lev = float(np.mean(lev_vals)) if lev_vals else 0.0
max_lev = float(np.max(lev_vals)) if lev_vals else 0.0
lev_p90 = float(np.percentile(lev_vals, 90)) if lev_vals else 0.0
lev_at_cap = sum(1 for v in lev_vals if v >= (eng.abs_max_leverage - 0.05))
pct_at_cap = lev_at_cap / len(lev_vals) * 100.0 if lev_vals else 0.0
# proxy_B scale stats
scale_mean = getattr(eng, 'sizing_scale_mean', 1.0)
# MC monitoring
mc_mon = getattr(eng, 'mc_monitor', {})
return dict(
name=name,
roi=roi, pf=pf_val, dd=max_dd, wr=wr, sharpe=sharpe, trades=n,
avg_leverage=avg_lev, max_leverage_realized=max_lev,
lev_p90=lev_p90,
pct_at_hard_cap=pct_at_cap,
sizing_scale_mean=scale_mean,
mc_monitor=mc_mon,
)
# ── Main ─────────────────────────────────────────────────────────────────────
def main():
t_start = time.time()
print("=" * 76)
print("Exp 9 — Extended Leverage Ceiling Test (fork of GOLD adaptive_beta)")
print("=" * 76)
print(f" GOLD ref : ROI={_GOLD_EXP9['roi']:.2f}% DD={_GOLD_EXP9['dd']:.2f}% (adaptive_beta)")
print(f" SILVER ref: ROI={_SILVER['roi']:.2f}% DD={_SILVER['dd']:.2f}% (former gold, regression floor)")
ensure_jit()
d = load_data()
fw = load_forewarner()
# Config: (label, extended_soft_cap, extended_abs_cap, mc_leverage_ref)
configs = [
# ── Decoupled (MC sees original 5.0x reference across all runs) ──────────
("A_5.0/6.0_mc5.0_GOLD-ref", 5.0, 6.0, 5.0), # must reproduce GOLD 96.55%
("B_6.0/7.0_mc5.0_decoupled", 6.0, 7.0, 5.0),
("C_7.0/8.0_mc5.0_decoupled", 7.0, 8.0, 5.0),
("D_8.0/9.0_mc5.0_decoupled", 8.0, 9.0, 5.0),
("E_9.0/10.0_mc5.0_decoupled", 9.0, 10.0, 5.0),
# ── Coupled (MC sees actual new cap → tests MC's response) ───────────────
("F_6.0/7.0_mc6.0_coupled", 6.0, 7.0, 6.0),
("G_7.0/8.0_mc7.0_coupled", 7.0, 8.0, 7.0),
("H_8.0/9.0_mc8.0_coupled", 8.0, 9.0, 8.0),
]
# adaptive_beta proxy_B params (match GOLD exactly)
_PROXY_KWARGS = dict(threshold=DEFAULT_THRESHOLD, alpha=DEFAULT_ALPHA,
adaptive_beta=True, adaptive_alpha=False, adaptive_thr=False)
results = []
for i, (label, soft, hard, mc_ref) in enumerate(configs):
t0 = time.time()
print(f"\n[{i+1}/{len(configs)}] {label} ...")
def _factory(kw, s=soft, h=hard, r=mc_ref):
return ExtendedLeverageEngine(
extended_soft_cap=s,
extended_abs_cap=h,
mc_leverage_ref=r,
**_PROXY_KWARGS,
**kw,
)
res = _run(_factory, label, d, fw)
elapsed = time.time() - t0
mc = res['mc_monitor']
mc_str = (f" MC: ok={mc.get('ok',0)} orange={mc.get('orange',0)} "
f"red={mc.get('red',0)} halted={mc.get('halted',0)}/{mc.get('total',0)}")
dROI = res['roi'] - _GOLD_EXP9['roi']
dDD = res['dd'] - _GOLD_EXP9['dd']
print(f" ROI={res['roi']:>7.2f}% (Δ{dROI:+.2f}pp) DD={res['dd']:>6.2f}% (Δ{dDD:+.2f}pp) "
f"PF={res['pf']:.4f} Trades={res['trades']}")
print(f" avg_lev={res['avg_leverage']:.2f}x p90={res['lev_p90']:.2f}x "
f"max={res['max_leverage_realized']:.2f}x at_hard_cap={res['pct_at_hard_cap']:.1f}% "
f"scale_mean={res['sizing_scale_mean']:.4f}")
print(f"{mc_str} ({elapsed:.0f}s)")
results.append(res)
# ── Verification ─────────────────────────────────────────────────────────
gold_ref = results[0]
gold_ok = (abs(gold_ref['roi'] - _GOLD_EXP9['roi']) < 0.5 and
abs(gold_ref['dd'] - _GOLD_EXP9['dd']) < 0.5 and
abs(gold_ref['trades'] - _GOLD_EXP9['trades']) < 10)
print(f"\n{'='*76}")
print(f"GOLD REFERENCE VERIFICATION: {'PASS ✓' if gold_ok else 'FAIL ✗'}")
print(f" Expected: ROI={_GOLD_EXP9['roi']:.2f}% DD={_GOLD_EXP9['dd']:.2f}% "
f"Got: ROI={gold_ref['roi']:.2f}% DD={gold_ref['dd']:.2f}%")
# ── Results table ─────────────────────────────────────────────────────────
print(f"\n{'='*76}")
print("FULL RESULTS (vs GOLD = adaptive_beta 96.55% / 14.32%)")
print(f"{'Config':<38} {'ROI%':>7} {'ΔROI':>6} {'DD%':>6} {'ΔDD':>6} "
f"{'Trades':>7} {'avgLev':>7} {'p90Lev':>7} {'@cap%':>6} {'OK?':>4}")
print('-' * 100)
for r in results:
dROI = r['roi'] - _GOLD_EXP9['roi']
dDD = r['dd'] - _GOLD_EXP9['dd']
# Pass: better than GOLD on DD, not worse than SILVER on ROI
ok = ('Y' if r['dd'] < _GOLD_EXP9['dd'] and r['roi'] >= _SILVER['roi'] else
'G' if r['dd'] < _SILVER['dd'] and r['roi'] >= _SILVER['roi'] else
'N')
print(f"{r['name']:<38} {r['roi']:>7.2f} {dROI:>+6.2f} {r['dd']:>6.2f} {dDD:>+6.2f} "
f"{r['trades']:>7} {r['avg_leverage']:>7.2f} {r['lev_p90']:>7.2f} "
f"{r['pct_at_hard_cap']:>6.1f} {ok:>4}")
print(" OK=Y: beats GOLD on DD + above SILVER ROI | OK=G: beats SILVER on both | OK=N: fail")
# ── MC interaction table ──────────────────────────────────────────────────
print(f"\n{'='*76}")
print("MC-FOREWARNER INTERACTION (per-day breakdown)")
print(f"{'Config':<38} {'OK':>5} {'ORG':>5} {'RED':>5} {'HALT':>5} {'TOTAL':>6} {'halt%':>7}")
print('-' * 75)
for r in results:
mc = r['mc_monitor']
total = mc.get('total', 0)
halt = mc.get('halted', 0)
halt_pct = halt / total * 100.0 if total else 0.0
print(f"{r['name']:<38} {mc.get('ok',0):>5} {mc.get('orange',0):>5} "
f"{mc.get('red',0):>5} {halt:>5} {total:>6} {halt_pct:>7.1f}%")
# ── Decoupled vs Coupled delta at each leverage level ────────────────────
print(f"\n{'='*76}")
print("MC COUPLING EFFECT (decoupled vs coupled at same leverage level)")
pairs = [
("B_6.0/7.0", "F_6.0/7.0"),
("C_7.0/8.0", "G_7.0/8.0"),
("D_8.0/9.0", "H_8.0/9.0"),
]
rmap = {r['name'][:10]: r for r in results}
for dec_label, coup_label in pairs:
dec = next((r for r in results if dec_label in r['name']), None)
cop = next((r for r in results if coup_label in r['name']), None)
if dec and cop:
halt_dec = dec['mc_monitor'].get('halted', 0)
halt_cop = cop['mc_monitor'].get('halted', 0)
print(f" {dec_label}: decoupled ROI={dec['roi']:.2f}% DD={dec['dd']:.2f}% halted={halt_dec}d")
print(f" {coup_label}: coupled ROI={cop['roi']:.2f}% DD={cop['dd']:.2f}% halted={halt_cop}d")
print(f" MC coupling cost: ΔROI={cop['roi']-dec['roi']:+.2f}pp ΔDD={cop['dd']-dec['dd']:+.2f}pp "
f"Δhalted={halt_cop-halt_dec:+d}d")
print()
# ── Best config summary ───────────────────────────────────────────────────
print(f"{'='*76}")
decoupled = [r for r in results if 'decoupled' in r['name'] or 'GOLD' in r['name']]
best_roi = max(decoupled, key=lambda r: r['roi'])
best_dd = min(decoupled, key=lambda r: r['dd'])
best_combined = max(decoupled, key=lambda r: r['roi'] - r['dd']) # ROI−DD as proxy
print(f"BEST (decoupled only — pure leverage effect):")
print(f" Best ROI: {best_roi['name']} ROI={best_roi['roi']:.2f}% DD={best_roi['dd']:.2f}%")
print(f" Best DD: {best_dd['name']} ROI={best_dd['roi']:.2f}% DD={best_dd['dd']:.2f}%")
print(f" Best R−D: {best_combined['name']} ROI={best_combined['roi']:.2f}% DD={best_combined['dd']:.2f}%")
# ── Log ──────────────────────────────────────────────────────────────────
outfile = _HERE / "exp9_leverage_ceiling_results.json"
log_results(results, outfile, gold=_GOLD_EXP9, meta={
"exp": "exp9",
"question": "Does adaptive_beta DD headroom permit higher leverage ceiling?",
"silver": _SILVER,
"total_elapsed_s": round(time.time() - t_start, 1),
"gold_ref_ok": gold_ok,
"note_mc": (
"Decoupled runs (mc_ref=5.0): MC assesses at original 5x reference — "
"pure leverage cap test. Coupled runs: MC sees actual new cap — tests MC response."
),
})
total = time.time() - t_start
print(f"\nTotal elapsed: {total / 60:.1f} min")
print("Done.")
if __name__ == "__main__":
main()

189
nautilus_dolphin/dvae/exp9_leverage_ceiling_results.json Executable file
View File

@@ -0,0 +1,189 @@
{
"gold": {
"roi": 96.55,
"dd": 14.32,
"trades": 2155
},
"results": [
{
"name": "A_5.0/6.0_mc5.0_GOLD-ref",
"roi": 96.55083977704362,
"pf": 1.2201203038704769,
"dd": 14.321954772270876,
"wr": 50.48723897911833,
"sharpe": 4.631491961453057,
"trades": 2155,
"avg_leverage": 3.1901002888648518,
"max_leverage_realized": 6.0,
"lev_p90": 6.0,
"pct_at_hard_cap": 40.324825986078885,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
},
{
"name": "B_6.0/7.0_mc5.0_decoupled",
"roi": 119.33687073886617,
"pf": 1.2185272940196565,
"dd": 16.916071652089094,
"wr": 50.48723897911833,
"sharpe": 4.550677484437249,
"trades": 2155,
"avg_leverage": 3.588114025875353,
"max_leverage_realized": 7.0,
"lev_p90": 7.0,
"pct_at_hard_cap": 39.443155452436194,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
},
{
"name": "C_7.0/8.0_mc5.0_decoupled",
"roi": 141.79672282249857,
"pf": 1.2193254974796508,
"dd": 18.318263045357885,
"wr": 50.48723897911833,
"sharpe": 4.517317168361109,
"trades": 2155,
"avg_leverage": 3.8905395845230593,
"max_leverage_realized": 8.0,
"lev_p90": 8.0,
"pct_at_hard_cap": 20.51044083526682,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
},
{
"name": "D_8.0/9.0_mc5.0_decoupled",
"roi": 162.27551845884568,
"pf": 1.2215406999889398,
"dd": 18.43672091284768,
"wr": 50.48723897911833,
"sharpe": 4.490863883629943,
"trades": 2155,
"avg_leverage": 4.093742876621359,
"max_leverage_realized": 9.0,
"lev_p90": 9.0,
"pct_at_hard_cap": 20.139211136890953,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
},
{
"name": "E_9.0/10.0_mc5.0_decoupled",
"roi": 183.99742787632874,
"pf": 1.222401072093006,
"dd": 18.556892309537158,
"wr": 50.48723897911833,
"sharpe": 4.427325011926956,
"trades": 2155,
"avg_leverage": 4.293314913419678,
"max_leverage_realized": 10.0,
"lev_p90": 10.0,
"pct_at_hard_cap": 19.814385150812065,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
},
{
"name": "F_6.0/7.0_mc6.0_coupled",
"roi": 119.33687073886617,
"pf": 1.2185272940196565,
"dd": 16.916071652089094,
"wr": 50.48723897911833,
"sharpe": 4.550677484437249,
"trades": 2155,
"avg_leverage": 3.588114025875353,
"max_leverage_realized": 7.0,
"lev_p90": 7.0,
"pct_at_hard_cap": 39.443155452436194,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
},
{
"name": "G_7.0/8.0_mc7.0_coupled",
"roi": 141.79672282249857,
"pf": 1.2193254974796508,
"dd": 18.318263045357885,
"wr": 50.48723897911833,
"sharpe": 4.517317168361109,
"trades": 2155,
"avg_leverage": 3.8905395845230593,
"max_leverage_realized": 8.0,
"lev_p90": 8.0,
"pct_at_hard_cap": 20.51044083526682,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
},
{
"name": "H_8.0/9.0_mc8.0_coupled",
"roi": 162.27551845884568,
"pf": 1.2215406999889398,
"dd": 18.43672091284768,
"wr": 50.48723897911833,
"sharpe": 4.490863883629943,
"trades": 2155,
"avg_leverage": 4.093742876621359,
"max_leverage_realized": 9.0,
"lev_p90": 9.0,
"pct_at_hard_cap": 20.139211136890953,
"sizing_scale_mean": 1.0883296846011132,
"mc_monitor": {
"red": 0,
"orange": 0,
"ok": 56,
"halted": 0,
"total": 56
}
}
],
"meta": {
"exp": "exp9",
"question": "Does adaptive_beta DD headroom permit higher leverage ceiling?",
"silver": {
"roi": 88.55,
"dd": 15.05,
"trades": 2155
},
"total_elapsed_s": 2697.7,
"gold_ref_ok": true,
"note_mc": "Decoupled runs (mc_ref=5.0): MC assesses at original 5x reference \u2014 pure leverage cap test. Coupled runs: MC sees actual new cap \u2014 tests MC response."
}
}

279
nautilus_dolphin/dvae/exp9b_liquidation_guard.py Executable file
View File

@@ -0,0 +1,279 @@
"""
Exp 9b — Liquidation Guard on Extended Leverage Configs
Exp9 found a DD plateau after 7x soft cap: each additional leverage unit above 7x
costs only +0.12pp DD while adding ~20pp ROI. However, exp9 does NOT model exchange
liquidation: the existing stop_pct=1.0 is effectively disabled, and a position at
10x leverage would be force-closed by the exchange after a 10% adverse move.
This experiment adds a per-trade liquidation floor using the _pending_stop_override hook:
stop_override = (1.0 / abs_cap) * 0.95 (95% of exchange margin = early warning floor)
6/7x config: fires if price moves >13.6% against position in the hold window
7/8x config: >11.9%
8/9x config: >10.6%
9/10x config: >9.5%
All of these thresholds are wide relative to a 10-min BTC hold — the 55-day gold dataset
almost certainly contains 0 such events for B/C, possibly 0-1 for D/E.
The goal is to CONFIRM that exp9 results are not artefacts of missing liquidation model.
If results are identical to exp9: liquidation risk is immaterial for this dataset.
If results differ: we know exactly which config/day triggered it and the cost.
Configs run (skip A = GOLD reference, already verified in exp9):
B_liq: 6/7x mc_ref=5.0 + liquidation guard at 13.6%
C_liq: 7/8x mc_ref=5.0 + liquidation guard at 11.9%
D_liq: 8/9x mc_ref=5.0 + liquidation guard at 10.6%
E_liq: 9/10x mc_ref=5.0 + liquidation guard at 9.5%
Total: 4 runs × ~250s ≈ 17 min
Results → exp9b_liquidation_guard_results.json
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, GOLD, MC_BASE_CFG,
load_data, load_forewarner, log_results,
)
from exp9_leverage_ceiling import ExtendedLeverageEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.proxy_boost_engine import DEFAULT_THRESHOLD, DEFAULT_ALPHA
# Known exp9 results for comparison
_EXP9 = {
'B': dict(roi=119.34, dd=16.92, trades=2155),
'C': dict(roi=141.80, dd=18.32, trades=2155),
'D': dict(roi=162.28, dd=18.44, trades=2155),
'E': dict(roi=184.00, dd=18.56, trades=2155),
}
_GOLD_EXP9 = dict(roi=96.55, dd=14.32, trades=2155)
# ── LiquidationGuardEngine ────────────────────────────────────────────────────
class LiquidationGuardEngine(ExtendedLeverageEngine):
"""
Adds an exchange-liquidation floor stop to ExtendedLeverageEngine.
For each entry, sets _pending_stop_override = (1/abs_cap) * margin_buffer
BEFORE calling super()._try_entry(). The NDAlphaEngine._try_entry() consumes
this and passes it to exit_manager.setup_position() as stop_pct_override.
The exit_manager then monitors: if pnl_pct < -stop_pct_override → EXIT (liquidation).
Using abs_cap (hard ceiling) slightly underestimates the actual liquidation price,
i.e. we exit slightly before the exchange would — a conservative model.
liquidation_stops counter tracks how many times this fires.
"""
def __init__(self, *args, margin_buffer: float = 0.95, **kwargs):
super().__init__(*args, **kwargs)
self.margin_buffer = margin_buffer
self._liq_stop_pct = (1.0 / self._extended_abs_cap) * margin_buffer
self.liquidation_stops = 0
def _try_entry(self, bar_idx, vel_div, prices, price_histories,
v50_vel=0.0, v750_vel=0.0):
# Arm the liquidation floor before parent entry logic consumes it
self._pending_stop_override = self._liq_stop_pct
result = super()._try_entry(bar_idx, vel_div, prices, price_histories,
v50_vel, v750_vel)
return result
def _execute_exit(self, reason, bar_idx, pnl_pct_raw=0.0, bars_held=0):
if reason == 'STOP_LOSS':
self.liquidation_stops += 1
return super()._execute_exit(reason, bar_idx, pnl_pct_raw, bars_held)
def reset(self):
super().reset()
self._liq_stop_pct = (1.0 / self._extended_abs_cap) * self.margin_buffer
self.liquidation_stops = 0
# ── Run harness ───────────────────────────────────────────────────────────────
def _run(engine_factory, name, d, fw):
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
for pf in d['parquet_files']:
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
liq_stops = getattr(eng, 'liquidation_stops', 0)
if n == 0:
return dict(name=name, roi=roi, pf=0.0, dd=0.0, wr=0.0, sharpe=0.0,
trades=0, avg_leverage=0.0, liq_stops=liq_stops,
liq_stop_pct=getattr(eng, '_liq_stop_pct', 0.0))
def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf_val = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
lev_vals = [t.leverage for t in tr if hasattr(t, 'leverage') and t.leverage > 0]
avg_lev = float(np.mean(lev_vals)) if lev_vals else 0.0
return dict(
name=name, roi=roi, pf=pf_val, dd=max_dd, wr=wr, sharpe=sharpe,
trades=n, avg_leverage=avg_lev,
liq_stops=liq_stops,
liq_stop_pct=getattr(eng, '_liq_stop_pct', 0.0),
liq_stop_rate_pct=liq_stops / n * 100.0 if n else 0.0,
)
# ── Main ─────────────────────────────────────────────────────────────────────
def main():
t_start = time.time()
print("=" * 76)
print("Exp 9b — Liquidation Guard on Extended Leverage Configs")
print("=" * 76)
print(" Adding exchange-liquidation floor stop to exp9 B/C/D/E configs.")
print(" Stop fires if adverse move exceeds (1/abs_cap)*0.95 of position.")
ensure_jit()
d = load_data()
fw = load_forewarner()
_PROXY = dict(threshold=DEFAULT_THRESHOLD, alpha=DEFAULT_ALPHA,
adaptive_beta=True, adaptive_alpha=False, adaptive_thr=False)
# (label, soft_cap, abs_cap, mc_ref, exp9_key)
configs = [
("B_liq_6/7x_liqstop13.6%", 6.0, 7.0, 5.0, 'B'),
("C_liq_7/8x_liqstop11.9%", 7.0, 8.0, 5.0, 'C'),
("D_liq_8/9x_liqstop10.6%", 8.0, 9.0, 5.0, 'D'),
("E_liq_9/10x_liqstop9.5%", 9.0, 10.0, 5.0, 'E'),
]
results = []
for i, (label, soft, hard, mc_ref, exp9_key) in enumerate(configs):
t0 = time.time()
liq_pct = (1.0 / hard) * 0.95 * 100
print(f"\n[{i+1}/{len(configs)}] {label} (floor={liq_pct:.1f}%) ...")
def _factory(kw, s=soft, h=hard, r=mc_ref):
return LiquidationGuardEngine(
extended_soft_cap=s, extended_abs_cap=h, mc_leverage_ref=r,
margin_buffer=0.95,
**_PROXY, **kw,
)
res = _run(_factory, label, d, fw)
elapsed = time.time() - t0
ref = _EXP9[exp9_key]
dROI = res['roi'] - ref['roi']
dDD = res['dd'] - ref['dd']
print(f" ROI={res['roi']:>8.2f}% (vs exp9: {dROI:+.2f}pp) "
f"DD={res['dd']:>6.2f}% (vs exp9: {dDD:+.2f}pp) "
f"Trades={res['trades']}")
print(f" avg_lev={res['avg_leverage']:.2f}x "
f"liquidation_stops={res['liq_stops']} "
f"liq_rate={res['liq_stop_rate_pct']:.2f}% ({elapsed:.0f}s)")
results.append(res)
# ── Summary ──────────────────────────────────────────────────────────────
print(f"\n{'='*76}")
print("LIQUIDATION GUARD IMPACT vs EXP9 (unguarded)")
print(f"{'Config':<34} {'ROI(9b)':>8} {'ROI(9)':>8} {'ΔROI':>7} "
f"{'DD(9b)':>7} {'DD(9)':>7} {'ΔDD':>6} {'LiqStops':>9}")
print('-' * 90)
for r in results:
key = r['name'][0] # B/C/D/E
ref = _EXP9[key]
dROI = r['roi'] - ref['roi']
dDD = r['dd'] - ref['dd']
identical = (abs(dROI) < 0.01 and abs(dDD) < 0.01 and r['liq_stops'] == 0)
flag = '✓ identical' if identical else '✗ CHANGED'
print(f"{r['name']:<34} {r['roi']:>8.2f} {ref['roi']:>8.2f} {dROI:>+7.2f} "
f"{r['dd']:>7.2f} {ref['dd']:>7.2f} {dDD:>+6.2f} "
f"{r['liq_stops']:>6} {flag}")
# ── Compounding with liquidation-adjusted numbers ─────────────────────────
print(f"\n{'='*76}")
print("COMPOUNDING TABLE — liquidation-adjusted (starting $25k, 56-day periods)")
all_configs = [
("GOLD 5/6x", _GOLD_EXP9['roi'], _GOLD_EXP9['dd']),
] + [
(r['name'][:10], r['roi'], r['dd']) for r in results
]
print(f"{'Config':<20} {'ROI%':>7} {'DD%':>6} {'Calmar':>7} "
f"{'3per(~5mo)':>12} {'6per(~1yr)':>12} {'12per(~2yr)':>13}")
print('-' * 85)
for label, roi, dd in all_configs:
mult = 1.0 + roi / 100.0
calmar = roi / dd if dd > 0 else 0
v3 = 25000 * mult**3
v6 = 25000 * mult**6
v12 = 25000 * mult**12
print(f"{label:<20} {roi:>7.2f} {dd:>6.2f} {calmar:>7.2f} "
f"${v3:>11,.0f} ${v6:>11,.0f} ${v12:>12,.0f}")
print(f"\n{'='*76}")
any_changed = any(r['liq_stops'] > 0 for r in results)
if not any_changed:
print("VERDICT: Zero liquidation stops fired across all configs.")
print(" Exp9 results are accurate — liquidation risk is immaterial for this 55-day dataset.")
print(" The compounding numbers from exp9 stand as-is.")
else:
changed = [r for r in results if r['liq_stops'] > 0]
print(f"VERDICT: {sum(r['liq_stops'] for r in results)} liquidation stop(s) fired.")
for r in changed:
print(f" {r['name']}: {r['liq_stops']} stops, ΔROI={r['roi']-_EXP9[r['name'][0]]['roi']:+.2f}pp")
outfile = _HERE / "exp9b_liquidation_guard_results.json"
log_results(results, outfile, gold=_GOLD_EXP9, meta={
"exp": "exp9b",
"question": "Do liquidation stops fire in 55-day dataset? Are exp9 results accurate?",
"exp9_reference": _EXP9,
"total_elapsed_s": round(time.time() - t_start, 1),
})
total = time.time() - t_start
print(f"\nTotal elapsed: {total / 60:.1f} min")
print("Done.")
if __name__ == "__main__":
main()

88
nautilus_dolphin/dvae/exp9b_liquidation_guard_results.json Executable file
View File

@@ -0,0 +1,88 @@
{
"gold": {
"roi": 96.55,
"dd": 14.32,
"trades": 2155
},
"results": [
{
"name": "B_liq_6/7x_liqstop13.6%",
"roi": 117.60890852798336,
"pf": 1.2199931287037944,
"dd": 15.800313739426617,
"wr": 50.672853828306266,
"sharpe": 4.592967533169852,
"trades": 2155,
"avg_leverage": 3.501351353769574,
"liq_stops": 1,
"liq_stop_pct": 0.1357142857142857,
"liq_stop_rate_pct": 0.04640371229698376
},
{
"name": "C_liq_7/8x_liqstop11.9%",
"roi": 139.72300151188082,
"pf": 1.2286392091015288,
"dd": 16.7760059114392,
"wr": 50.672853828306266,
"sharpe": 4.844497641085453,
"trades": 2155,
"avg_leverage": 3.7301880355925894,
"liq_stops": 1,
"liq_stop_pct": 0.11875,
"liq_stop_rate_pct": 0.04640371229698376
},
{
"name": "D_liq_8/9x_liqstop10.6%",
"roi": 147.2832528932534,
"pf": 1.233340905262188,
"dd": 16.776005911439114,
"wr": 50.672853828306266,
"sharpe": 4.995057984165502,
"trades": 2155,
"avg_leverage": 3.795150347589037,
"liq_stops": 1,
"liq_stop_pct": 0.10555555555555554,
"liq_stop_rate_pct": 0.04640371229698376
},
{
"name": "E_liq_9/10x_liqstop9.5%",
"roi": 112.16063020117491,
"pf": 1.1952757754808747,
"dd": 30.685728293170744,
"wr": 50.69573283858998,
"sharpe": 3.9564143841036885,
"trades": 2156,
"avg_leverage": 3.854065962587942,
"liq_stops": 5,
"liq_stop_pct": 0.095,
"liq_stop_rate_pct": 0.2319109461966605
}
],
"meta": {
"exp": "exp9b",
"question": "Do liquidation stops fire in 55-day dataset? Are exp9 results accurate?",
"exp9_reference": {
"B": {
"roi": 119.34,
"dd": 16.92,
"trades": 2155
},
"C": {
"roi": 141.8,
"dd": 18.32,
"trades": 2155
},
"D": {
"roi": 162.28,
"dd": 18.44,
"trades": 2155
},
"E": {
"roi": 184.0,
"dd": 18.56,
"trades": 2155
}
},
"total_elapsed_s": 1877.3
}
}

202
nautilus_dolphin/dvae/exp9c_overfitting_results.json Executable file
View File

@@ -0,0 +1,202 @@
{
"gold": {
"roi": 88.55,
"pf": 1.215,
"dd": 15.05,
"sharpe": 4.38,
"wr": 50.5,
"trades": 2155
},
"results": [
{
"name": "d_liq_H1 (days 0-27)",
"roi": 76.14224142505569,
"dd": 16.77600591143919,
"calmar": 4.538758619126139,
"trades": 1050,
"liq_stops": 1,
"days": 28,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "abeta_H1 (days 0-27)",
"roi": 38.57335970751784,
"dd": 14.321954772270908,
"calmar": 2.693302717461493,
"trades": 1050,
"liq_stops": 0,
"days": 28,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "d_liq_H2 (days 28-55)",
"roi": 59.928495551638946,
"dd": 15.381215784261638,
"calmar": 3.896213172755756,
"trades": 1103,
"liq_stops": 0,
"days": 28,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "abeta_H2 (days 28-55)",
"roi": 42.578652854670715,
"dd": 10.236414991759771,
"calmar": 4.159527812124281,
"trades": 1103,
"liq_stops": 0,
"days": 28,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "d_liq_Q1",
"roi": 3.353675329427657,
"dd": 16.77600591143919,
"calmar": 0.199909045522025,
"trades": 532,
"liq_stops": 0,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "abeta_Q1",
"roi": 1.1727921993959172,
"dd": 13.884446888829856,
"calmar": 0.08446805326753329,
"trades": 532,
"liq_stops": 0,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "d_liq_Q2",
"roi": 70.82653734416076,
"dd": 12.810959705505464,
"calmar": 5.528589502449477,
"trades": 517,
"liq_stops": 1,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "abeta_Q2",
"roi": 38.781648637136215,
"dd": 14.437901577945617,
"calmar": 2.686100083711368,
"trades": 517,
"liq_stops": 0,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "d_liq_Q3",
"roi": 55.34503692873527,
"dd": 15.381215784261757,
"calmar": 3.598222514072325,
"trades": 465,
"liq_stops": 0,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "abeta_Q3",
"roi": 38.41268117672217,
"dd": 10.236414991759771,
"calmar": 3.7525521588997766,
"trades": 465,
"liq_stops": 0,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "d_liq_Q4",
"roi": 3.1074038301600084,
"dd": 12.866819616903355,
"calmar": 0.2415051988509857,
"trades": 637,
"liq_stops": 0,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "abeta_Q4",
"roi": 3.1287720381663533,
"dd": 8.984276378639036,
"calmar": 0.3482497539373681,
"trades": 637,
"liq_stops": 0,
"days": 14,
"mc_red": 0,
"mc_halted": 0
},
{
"name": "d_liq_buf0.80",
"roi": 169.47614903222345,
"dd": 20.78163843773938,
"calmar": 8.15509082885666,
"trades": 2156,
"liq_stops": 5,
"days": 56,
"mc_red": 0,
"mc_halted": 0,
"margin_buffer": 0.8
},
{
"name": "d_liq_buf0.90",
"roi": 167.655698815112,
"dd": 16.77600591143919,
"calmar": 9.993779192745233,
"trades": 2155,
"liq_stops": 2,
"days": 56,
"mc_red": 0,
"mc_halted": 0,
"margin_buffer": 0.9
},
{
"name": "d_liq_buf0.95",
"roi": 166.83246014782176,
"dd": 16.77600591143919,
"calmar": 9.944706804976885,
"trades": 2155,
"liq_stops": 1,
"days": 56,
"mc_red": 0,
"mc_halted": 0,
"margin_buffer": 0.95
},
{
"name": "d_liq_buf1.00",
"roi": 166.83246014782176,
"dd": 16.77600591143919,
"calmar": 9.944706804976885,
"trades": 2155,
"liq_stops": 1,
"days": 56,
"mc_red": 0,
"mc_halted": 0,
"margin_buffer": 1.0
}
],
"meta": {
"exp": "exp9c",
"question": "Is D_LIQ_GOLD robust across time windows and parameter perturbations?",
"split_passes": "1/2",
"quarter_passes": "2/4",
"buf_roi_range_pp": 2.644,
"buf_dd_range_pp": 4.006,
"all_pass": false,
"total_elapsed_s": 5657.9
}
}

296
nautilus_dolphin/dvae/exp9c_overfitting_validation.py Executable file
View File

@@ -0,0 +1,296 @@
"""
Exp 9c — Overfitting Validation for D_LIQ_GOLD
Battery of tests designed to expose any period-specific bias in the D_LIQ_GOLD result
(8x/9x + liquidation guard, exp9b: ROI=181.81%, DD=17.65%, Calmar=10.30).
Three test families:
1. TEMPORAL SPLIT (H1/H2)
Same split as exp8 adaptive_beta validation (days 0-27 vs days 28-55).
Each half: fresh engine, fresh capital=$25k, cold start.
Pass criterion: Calmar(d_liq) > Calmar(adaptive_beta) in BOTH halves.
If d_liq only wins in one half → period-specific, do NOT flip default.
2. QUARTERLY SPLIT (Q1/Q2/Q3/Q4)
Four independent ~14-day windows.
Finer-grained: reveals if any single quarter is carrying the full result.
Pass criterion: d_liq Calmar consistently above adaptive_beta across quarters.
3. MARGIN BUFFER SENSITIVITY
Test margin_buffer = 0.80, 0.90, 0.95 (gold), 1.00 on the full period.
Confirms the specific 10.6% floor is not cherry-picked.
Pass criterion: ROI/DD metrics stable across ±0.15 variation in buffer.
Reference benchmarks:
D_LIQ_GOLD (full period): ROI=181.81%, DD=17.65%, Calmar=10.30
adaptive_beta (full): ROI= 96.55%, DD=14.32%, Calmar= 6.74
Results → exp9c_overfitting_results.json
"""
import sys, time, json, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import (
ensure_jit, ENGINE_KWARGS, MC_BASE_CFG,
load_data, load_forewarner, log_results,
)
from nautilus_dolphin.nautilus.proxy_boost_engine import (
AdaptiveBoostEngine, LiquidationGuardEngine,
DEFAULT_THRESHOLD, DEFAULT_ALPHA,
D_LIQ_SOFT_CAP, D_LIQ_ABS_CAP, D_LIQ_MC_REF,
)
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
_D_LIQ_FULL = dict(roi=181.81, dd=17.65, calmar=10.30, trades=2155)
_ABETA_FULL = dict(roi= 96.55, dd=14.32, calmar= 6.74, trades=2155)
_PROXY = dict(threshold=DEFAULT_THRESHOLD, alpha=DEFAULT_ALPHA,
adaptive_beta=True, adaptive_alpha=False, adaptive_thr=False)
# ── Engine factories ──────────────────────────────────────────────────────────
def _make_dliq(kw, margin_buffer=0.95):
return LiquidationGuardEngine(
extended_soft_cap=D_LIQ_SOFT_CAP,
extended_abs_cap=D_LIQ_ABS_CAP,
mc_leverage_ref=D_LIQ_MC_REF,
margin_buffer=margin_buffer,
**_PROXY, **kw,
)
def _make_abeta(kw):
return AdaptiveBoostEngine(**_PROXY, **kw)
# ── Run harness (window-aware) ────────────────────────────────────────────────
def _run_window(engine_factory, name, d, fw, day_indices):
"""Run a sub-period backtest over the given day index slice."""
kw = ENGINE_KWARGS.copy()
acb = AdaptiveCircuitBreaker()
# Preload full date list for proper w750 context even in sub-period runs
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
pf_list = d['parquet_files']
for idx in day_indices:
pf = pf_list[idx]
ds = pf.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
liq_stops = getattr(eng, 'liquidation_stops', 0)
mc_mon = getattr(eng, 'mc_monitor', {})
if n == 0:
return dict(name=name, roi=roi, dd=0.0, calmar=0.0, trades=0,
liq_stops=liq_stops, days=len(day_indices))
def _abs(t): return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
calmar = roi / max_dd if max_dd > 0 else 0.0
return dict(
name=name, roi=roi, dd=max_dd, calmar=calmar, trades=n,
liq_stops=liq_stops, days=len(day_indices),
mc_red=mc_mon.get('red', 0), mc_halted=mc_mon.get('halted', 0),
)
def _compare(dliq_r, abeta_r, window_label):
"""Print head-to-head for one window."""
d_roi = dliq_r['roi'] - abeta_r['roi']
d_dd = dliq_r['dd'] - abeta_r['dd']
d_cal = dliq_r['calmar'] - abeta_r['calmar']
liq = dliq_r.get('liq_stops', 0)
verdict = 'PASS' if dliq_r['calmar'] > abeta_r['calmar'] else 'FAIL'
print(f" {window_label:<18} d_liq {dliq_r['roi']:>7.2f}% / {dliq_r['dd']:>5.2f}% "
f"cal={dliq_r['calmar']:.2f} | abeta {abeta_r['roi']:>7.2f}% / {abeta_r['dd']:>5.2f}% "
f"cal={abeta_r['calmar']:.2f} | ΔROI={d_roi:+.2f} ΔDD={d_dd:+.2f} ΔCal={d_cal:+.2f} "
f"liq={liq} [{verdict}]")
return verdict == 'PASS'
# ── Main ─────────────────────────────────────────────────────────────────────
def main():
t_start = time.time()
print("=" * 80)
print("Exp 9c — D_LIQ_GOLD Overfitting Validation")
print("=" * 80)
ensure_jit()
d = load_data()
fw = load_forewarner()
n_days = len(d['parquet_files'])
print(f" Dataset: {n_days} trading days")
# Day index windows
all_idx = list(range(n_days))
mid = n_days // 2
h1_idx = all_idx[:mid]
h2_idx = all_idx[mid:]
q_size = n_days // 4
q_idx = [all_idx[i*q_size : (i+1)*q_size] for i in range(4)]
# Last quarter gets any remainder
q_idx[3] = all_idx[3*q_size:]
print(f" H1: days 0–{mid-1} ({len(h1_idx)}d) "
f"H2: days {mid}–{n_days-1} ({len(h2_idx)}d)")
print(f" Q1:{len(q_idx[0])}d Q2:{len(q_idx[1])}d "
f"Q3:{len(q_idx[2])}d Q4:{len(q_idx[3])}d")
results_all = []
pass_counts = {'split': 0, 'split_total': 0,
'quarter': 0, 'quarter_total': 0}
# ── FAMILY 1: Temporal split H1/H2 ───────────────────────────────────────
print(f"\n{'='*80}")
print("FAMILY 1 — Temporal Split H1/H2")
print(f"{'='*80}")
for label, idx in [('H1 (days 0-27)', h1_idx), ('H2 (days 28-55)', h2_idx)]:
t0 = time.time()
print(f"\n {label}:")
dliq_r = _run_window(lambda kw: _make_dliq(kw), f'd_liq_{label}', d, fw, idx)
abeta_r = _run_window(lambda kw: _make_abeta(kw), f'abeta_{label}', d, fw, idx)
elapsed = time.time() - t0
passed = _compare(dliq_r, abeta_r, label)
print(f" trades: d_liq={dliq_r['trades']} abeta={abeta_r['trades']} ({elapsed:.0f}s)")
results_all += [dliq_r, abeta_r]
pass_counts['split'] += int(passed)
pass_counts['split_total'] += 1
split_verdict = ('PASS ✓' if pass_counts['split'] == pass_counts['split_total']
else f"PARTIAL ({pass_counts['split']}/{pass_counts['split_total']})")
print(f"\n H1/H2 SPLIT VERDICT: {split_verdict}")
# ── FAMILY 2: Quarterly split ─────────────────────────────────────────────
print(f"\n{'='*80}")
print("FAMILY 2 — Quarterly Split (Q1/Q2/Q3/Q4)")
print(f"{'='*80}")
for qi, idx in enumerate(q_idx, 1):
label = f'Q{qi} (days {idx[0]}-{idx[-1]})'
t0 = time.time()
print(f"\n {label}:")
dliq_r = _run_window(lambda kw: _make_dliq(kw), f'd_liq_Q{qi}', d, fw, idx)
abeta_r = _run_window(lambda kw: _make_abeta(kw), f'abeta_Q{qi}', d, fw, idx)
elapsed = time.time() - t0
passed = _compare(dliq_r, abeta_r, label)
print(f" trades: d_liq={dliq_r['trades']} abeta={abeta_r['trades']} ({elapsed:.0f}s)")
results_all += [dliq_r, abeta_r]
pass_counts['quarter'] += int(passed)
pass_counts['quarter_total'] += 1
quarter_verdict = ('PASS ✓' if pass_counts['quarter'] == pass_counts['quarter_total']
else f"PARTIAL ({pass_counts['quarter']}/{pass_counts['quarter_total']})")
print(f"\n QUARTERLY VERDICT: {quarter_verdict}")
# ── FAMILY 3: Margin buffer sensitivity (full period) ─────────────────────
print(f"\n{'='*80}")
print("FAMILY 3 — Margin Buffer Sensitivity (full period, d_liq only)")
print(f"{'='*80}")
print(f" Floor = (1/abs_cap) * buffer | abs_cap=9.0")
print(f" {'Buffer':>8} {'Floor%':>7} {'ROI%':>8} {'DD%':>6} {'Calmar':>7} "
f"{'liq_stops':>10} {'ΔROI vs gold':>13}")
buf_results = []
for buf in [0.80, 0.90, 0.95, 1.00]:
t0 = time.time()
floor_pct = (1.0 / D_LIQ_ABS_CAP) * buf * 100
r = _run_window(lambda kw, b=buf: _make_dliq(kw, margin_buffer=b),
f'd_liq_buf{buf:.2f}', d, fw, all_idx)
elapsed = time.time() - t0
d_roi = r['roi'] - _D_LIQ_FULL['roi']
marker = ' ← GOLD' if abs(buf - 0.95) < 0.001 else ''
print(f" {buf:>8.2f} {floor_pct:>6.1f}% {r['roi']:>8.2f} {r['dd']:>6.2f} "
f"{r['calmar']:>7.2f} {r['liq_stops']:>10} {d_roi:>+13.2f}pp ({elapsed:.0f}s){marker}")
r['margin_buffer'] = buf
buf_results.append(r)
results_all.append(r)
# Stability check: ROI range across buffers
buf_rois = [r['roi'] for r in buf_results]
roi_range = max(buf_rois) - min(buf_rois)
buf_dds = [r['dd'] for r in buf_results]
dd_range = max(buf_dds) - min(buf_dds)
buf_stable = roi_range < 10.0 and dd_range < 2.0
print(f"\n ROI range across buffers: {roi_range:.2f}pp "
f"DD range: {dd_range:.2f}pp "
f"['STABLE ✓' if buf_stable else 'UNSTABLE ✗']")
# ── SUMMARY ───────────────────────────────────────────────────────────────
total_passes = pass_counts['split'] + pass_counts['quarter']
total_tests = pass_counts['split_total'] + pass_counts['quarter_total']
print(f"\n{'='*80}")
print("OVERFITTING VALIDATION SUMMARY")
print(f"{'='*80}")
print(f" Temporal split (H1/H2): {pass_counts['split']}/{pass_counts['split_total']} {split_verdict}")
print(f" Quarterly split (Q1-Q4): {pass_counts['quarter']}/{pass_counts['quarter_total']} {quarter_verdict}")
print(f" Margin buffer stability: {'STABLE ✓' if buf_stable else 'UNSTABLE ✗'} "
f"(ROI range={roi_range:.1f}pp, DD range={dd_range:.1f}pp)")
print()
all_pass = (total_passes == total_tests and buf_stable)
if all_pass:
print(" VERDICT: ALL TESTS PASS ✓")
print(" D_LIQ_GOLD is robust. Calmar advantage holds across all time windows.")
print(" Margin buffer choice is not critical. Safe to set as DEFAULT.")
else:
print(" VERDICT: SOME TESTS FAIL ✗")
print(f" {total_passes}/{total_tests} split windows passed, "
f"buffer stable={buf_stable}.")
print(" Do NOT flip default until failures are investigated.")
outfile = _HERE / "exp9c_overfitting_results.json"
log_results(results_all, outfile, meta={
"exp": "exp9c",
"question": "Is D_LIQ_GOLD robust across time windows and parameter perturbations?",
"split_passes": f"{pass_counts['split']}/{pass_counts['split_total']}",
"quarter_passes": f"{pass_counts['quarter']}/{pass_counts['quarter_total']}",
"buf_roi_range_pp": round(roi_range, 3),
"buf_dd_range_pp": round(dd_range, 3),
"all_pass": all_pass,
"total_elapsed_s": round(time.time() - t_start, 1),
})
print(f"\nTotal elapsed: {(time.time()-t_start)/60:.1f} min")
print("Done.")
if __name__ == "__main__":
main()

277
nautilus_dolphin/dvae/exp_shared.py Executable file
View File

@@ -0,0 +1,277 @@
"""
Shared infrastructure for proxy-B experiments (exp1–exp3, fast sweep).
Provides: data loading, run_backtest() with gold-matching metrics, log_results().
Gold baseline (2026-03-14 confirmed):
ROI=+88.55%, PF=1.215, DD=15.05%, Sharpe=4.38, WR=50.5%, Trades=2155
"""
import sys, time, math, json
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
import pandas as pd
_HERE = Path(__file__).resolve().parent
_ND_ROOT = _HERE.parent
sys.path.insert(0, str(_ND_ROOT))
# ── Lazy JIT warmup (done once per process) ──────────────────────────────────
_jit_done = False
def ensure_jit():
global _jit_done
if _jit_done: return
print("JIT warmup...")
t0 = time.time()
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from nautilus_dolphin.nautilus.ob_features import (
compute_imbalance_nb, compute_depth_1pct_nb, compute_market_agreement_nb,
compute_cascade_signal_nb,
)
_p = np.array([1.0, 2.0, 3.0], dtype=np.float64)
compute_irp_nb(_p, -1); compute_ars_nb(1.0, 0.5, 0.01)
rank_assets_irp_nb(np.ones((10, 2), dtype=np.float64), 8, -1, 5, 500.0, 20, 0.20)
compute_sizing_nb(-0.03,-0.02,-0.05,3.0,0.5,5.0,0.20,True,True,0.0,
np.zeros(4,dtype=np.int64),np.zeros(4,dtype=np.int64),
np.zeros(5,dtype=np.float64),0,-1,0.01,0.04)
check_dc_nb(_p, 3, 1, 0.75)
_b = np.array([100.,200.,300.,400.,500.], dtype=np.float64)
_a = np.array([110.,190.,310.,390.,510.], dtype=np.float64)
compute_imbalance_nb(_b,_a); compute_depth_1pct_nb(_b,_a)
compute_market_agreement_nb(np.array([0.1,-0.05],dtype=np.float64),2)
compute_cascade_signal_nb(np.array([-0.05,-0.15],dtype=np.float64),2,-0.10)
print(f" JIT: {time.time()-t0:.1f}s")
_jit_done = True
# ── Paths ─────────────────────────────────────────────────────────────────────
VBT_DIR = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
MC_MODELS_DIR= str(Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin\mc_results\models"))
META_COLS = {'timestamp','scan_number','v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity','vel_div',
'instability_50','instability_150'}
ENGINE_KWARGS = dict(
initial_capital=25000.0, vel_div_threshold=-0.02, vel_div_extreme=-0.05,
min_leverage=0.5, max_leverage=5.0, leverage_convexity=3.0,
fraction=0.20, fixed_tp_pct=0.0095, stop_pct=1.0, max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=0.75,
dc_skip_contradicts=True, dc_leverage_boost=1.0, dc_leverage_reduce=0.5,
use_asset_selection=True, min_irp_alignment=0.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=0.62, sp_maker_exit_rate=0.50,
use_ob_edge=True, ob_edge_bps=5.0, ob_confirm_rate=0.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
MC_BASE_CFG = {
'trial_id':0, 'vel_div_threshold':-0.020, 'vel_div_extreme':-0.050,
'use_direction_confirm':True, 'dc_lookback_bars':7, 'dc_min_magnitude_bps':0.75,
'dc_skip_contradicts':True, 'dc_leverage_boost':1.00, 'dc_leverage_reduce':0.50,
'vd_trend_lookback':10, 'min_leverage':0.50, 'max_leverage':5.00,
'leverage_convexity':3.00, 'fraction':0.20, 'use_alpha_layers':True,
'use_dynamic_leverage':True, 'fixed_tp_pct':0.0095, 'stop_pct':1.00,
'max_hold_bars':120, 'use_sp_fees':True, 'use_sp_slippage':True,
'sp_maker_entry_rate':0.62, 'sp_maker_exit_rate':0.50, 'use_ob_edge':True,
'ob_edge_bps':5.00, 'ob_confirm_rate':0.40, 'ob_imbalance_bias':-0.09,
'ob_depth_scale':1.00, 'use_asset_selection':True, 'min_irp_alignment':0.45,
'lookback':100, 'acb_beta_high':0.80, 'acb_beta_low':0.20, 'acb_w750_threshold_pct':60,
}
GOLD = dict(roi=88.55, pf=1.215, dd=15.05, sharpe=4.38, wr=50.5, trades=2155)
# ── Data loading (cached per process) ────────────────────────────────────────
_data_cache = {}
def load_data():
"""Load gold-standard data: float64 pq_data, correct vol_p60 (2-file, offset-60), 48 OB assets."""
from nautilus_dolphin.nautilus.ob_features import OBFeatureEngine
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
parquet_files = sorted(VBT_DIR.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
date_strings = [p.stem for p in parquet_files]
# GOLD vol_p60: 2 files, range(60), seg-based, v>0 filter
all_vols = []
for pf in parquet_files[:2]:
df = pd.read_parquet(pf)
if 'BTCUSDT' in df.columns:
pr = df['BTCUSDT'].values
for i in range(60, len(pr)):
seg = pr[max(0, i-50):i]
if len(seg) < 10: continue
v = float(np.std(np.diff(seg) / seg[:-1]))
if v > 0: all_vols.append(v)
del df
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.0002
print(f" Calibrated vol_p60 (gold method): {vol_p60:.8f}")
# GOLD pq_data: float64, all assets, dvol per bar
pq_data = {}
all_assets = set()
for pf in parquet_files:
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
all_assets.update(ac)
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0, i-50):i]
if len(seg) < 10: continue
dv[i] = float(np.std(np.diff(seg) / seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
OB_ASSETS = sorted(list(all_assets))
_mock_ob = MockOBProvider(
imbalance_bias=-0.09, depth_scale=1.0, assets=OB_ASSETS,
imbalance_biases={"BTCUSDT": -0.086, "ETHUSDT": -0.092,
"BNBUSDT": +0.05, "SOLUSDT": +0.05},
)
ob_eng = OBFeatureEngine(_mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
print(f" OB_ASSETS={len(OB_ASSETS)}, vol_p60={vol_p60:.8f}, days={len(parquet_files)}")
return dict(
parquet_files=parquet_files, date_strings=date_strings,
vol_p60=vol_p60, ob_eng=ob_eng, OB_ASSETS=OB_ASSETS,
pq_data=pq_data,
)
def load_forewarner():
try:
from mc.mc_ml import DolphinForewarner
fw = DolphinForewarner(models_dir=MC_MODELS_DIR)
print(" MC-Forewarner loaded (5 models)")
return fw
except Exception as e:
print(f" MC-Forewarner unavailable: {e}")
return None
def run_backtest(engine_factory, name, forewarner=None, extra_kwargs=None):
"""
Run full 55-day backtest with gold-matching metrics (Lazy loading).
"""
import gc
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
d = load_data()
kw = ENGINE_KWARGS.copy()
if extra_kwargs: kw.update(extra_kwargs)
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if forewarner is not None:
eng.set_mc_forewarner(forewarner, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
all_vols = []
for i, pf in enumerate(d['parquet_files']):
ds = pf.stem
# Lazy Load and cast to float32 to save RAM
df = pd.read_parquet(pf)
for c in df.columns:
if df[c].dtype == 'float64':
df[c] = df[c].astype('float32')
acols = [c for c in df.columns if c not in META_COLS]
# Per-day OB Preloading (Crucial for 230MB RAM)
if eng.ob_engine is not None:
eng.ob_engine.preload_date(ds, d['OB_ASSETS'])
# Optimized 5s dvol approximation
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dvol = np.zeros(len(df), dtype=np.float32)
if bp is not None:
rets = np.diff(bp.astype('float64')) / (bp[:-1].astype('float64') + 1e-9)
for j in range(50, len(rets)):
v = np.std(rets[j-50:j])
dvol[j+1] = v
if v > 0: all_vols.append(v)
cap_before = eng.capital
vp60 = np.percentile(all_vols, 60) if len(all_vols) > 1000 else d['vol_p60']
vol_ok = np.where(dvol > 0, dvol > vp60, False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
# CLEAR OB CACHE FOR DAY
if eng.ob_engine is not None:
eng.ob_engine._preloaded_placement.clear()
eng.ob_engine._preloaded_signal.clear()
eng.ob_engine._preloaded_market.clear()
eng.ob_engine._ts_to_idx.clear()
del df
gc.collect()
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
if n == 0:
return dict(name=name, roi=roi, pf=0, dd=0, wr=0, sharpe=0, trades=0)
def _abs(t): return t.pnl_absolute if hasattr(t,'pnl_absolute') else t.pnl_pct*250.
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p/25000.*100. for p in daily_pnls])
sharpe = float(dr.mean()/(dr.std()+1e-9)*math.sqrt(365)) if len(dr)>1 else 0.
# Gather any engine-specific extra stats
extra = {}
for attr in ('gate_suppressed','gate_allowed','early_exits','sizing_scale_mean'):
v = getattr(eng, attr, None)
if v is not None: extra[attr] = v
return dict(name=name, roi=roi, pf=pf, dd=max_dd, wr=wr, sharpe=sharpe,
trades=n, **extra)
def print_table(results, gold=None):
hdr = f"{'Config':<42} {'ROI%':>7} {'PF':>6} {'DD%':>6} {'WR%':>6} {'Sharpe':>7} {'Trades':>7}"
print(hdr); print('-'*83)
if gold:
g = gold
print(f"{'*** GOLD ***':<42} {g['roi']:>7.2f} {g['pf']:>6.4f} {g['dd']:>6.2f} "
f"{g['wr']:>6.2f} {g['sharpe']:>7.3f} {g['trades']:>7d}")
print('-'*83)
for r in results:
extra = ''
if 'suppression_rate' in r: extra += f" gate_supp={r['suppression_rate']:.1f}%"
if 'early_exits' in r: extra += f" early_exits={r['early_exits']}"
if 'sizing_scale_mean' in r: extra += f" scale_mean={r['sizing_scale_mean']:.3f}"
print(f"{r['name']:<42} {r['roi']:>7.2f} {r['pf']:>6.4f} {r['dd']:>6.2f} "
f"{r['wr']:>6.2f} {r['sharpe']:>7.3f} {r['trades']:>7d}{extra}")
def log_results(results, outfile, gold=None, meta=None):
payload = {'gold': gold or GOLD, 'results': results}
if meta: payload['meta'] = meta
outfile = Path(outfile)
outfile.parent.mkdir(parents=True, exist_ok=True)
with open(outfile, 'w', encoding='utf-8') as f:
json.dump(payload, f, indent=2)
print(f"\n Logged → {outfile}")

259
nautilus_dolphin/dvae/exp_shared_AGENT_fork.py Executable file
View File

@@ -0,0 +1,259 @@
"""
Shared infrastructure for proxy-B experiments (exp1–exp3, fast sweep).
Provides: data loading, run_backtest() with gold-matching metrics, log_results().
Gold baseline (2026-03-14 confirmed):
ROI=+88.55%, PF=1.215, DD=15.05%, Sharpe=4.38, WR=50.5%, Trades=2155
"""
import sys, time, math, json
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
import pandas as pd
_HERE = Path(__file__).resolve().parent
_ND_ROOT = _HERE.parent
sys.path.insert(0, str(_ND_ROOT))
# ── Lazy JIT warmup (done once per process) ──────────────────────────────────
_jit_done = False
def ensure_jit():
global _jit_done
if _jit_done: return
print("JIT warmup...")
t0 = time.time()
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from nautilus_dolphin.nautilus.ob_features import (
compute_imbalance_nb, compute_depth_1pct_nb, compute_market_agreement_nb,
compute_cascade_signal_nb,
)
_p = np.array([1.0, 2.0, 3.0], dtype=np.float64)
compute_irp_nb(_p, -1); compute_ars_nb(1.0, 0.5, 0.01)
rank_assets_irp_nb(np.ones((10, 2), dtype=np.float64), 8, -1, 5, 500.0, 20, 0.20)
compute_sizing_nb(-0.03,-0.02,-0.05,3.0,0.5,5.0,0.20,True,True,0.0,
np.zeros(4,dtype=np.int64),np.zeros(4,dtype=np.int64),
np.zeros(5,dtype=np.float64),0,-1,0.01,0.04)
check_dc_nb(_p, 3, 1, 0.75)
_b = np.array([100.,200.,300.,400.,500.], dtype=np.float64)
_a = np.array([110.,190.,310.,390.,510.], dtype=np.float64)
compute_imbalance_nb(_b,_a); compute_depth_1pct_nb(_b,_a)
compute_market_agreement_nb(np.array([0.1,-0.05],dtype=np.float64),2)
compute_cascade_signal_nb(np.array([-0.05,-0.15],dtype=np.float64),2,-0.10)
print(f" JIT: {time.time()-t0:.1f}s")
_jit_done = True
# ── Paths ─────────────────────────────────────────────────────────────────────
VBT_DIR = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
MC_MODELS_DIR= str(Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin\mc_results\models"))
META_COLS = {'timestamp','scan_number','v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity','vel_div',
'instability_50','instability_150'}
ENGINE_KWARGS = dict(
initial_capital=25000.0, vel_div_threshold=-0.02, vel_div_extreme=-0.05,
min_leverage=0.5, max_leverage=5.0, leverage_convexity=3.0,
fraction=0.20, fixed_tp_pct=0.0095, stop_pct=1.0, max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=0.75,
dc_skip_contradicts=True, dc_leverage_boost=1.0, dc_leverage_reduce=0.5,
use_asset_selection=True, min_irp_alignment=0.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=0.62, sp_maker_exit_rate=0.50,
use_ob_edge=True, ob_edge_bps=5.0, ob_confirm_rate=0.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
MC_BASE_CFG = {
'trial_id':0, 'vel_div_threshold':-0.020, 'vel_div_extreme':-0.050,
'use_direction_confirm':True, 'dc_lookback_bars':7, 'dc_min_magnitude_bps':0.75,
'dc_skip_contradicts':True, 'dc_leverage_boost':1.00, 'dc_leverage_reduce':0.50,
'vd_trend_lookback':10, 'min_leverage':0.50, 'max_leverage':5.00,
'leverage_convexity':3.00, 'fraction':0.20, 'use_alpha_layers':True,
'use_dynamic_leverage':True, 'fixed_tp_pct':0.0095, 'stop_pct':1.00,
'max_hold_bars':120, 'use_sp_fees':True, 'use_sp_slippage':True,
'sp_maker_entry_rate':0.62, 'sp_maker_exit_rate':0.50, 'use_ob_edge':True,
'ob_edge_bps':5.00, 'ob_confirm_rate':0.40, 'ob_imbalance_bias':-0.09,
'ob_depth_scale':1.00, 'use_asset_selection':True, 'min_irp_alignment':0.45,
'lookback':100, 'acb_beta_high':0.80, 'acb_beta_low':0.20, 'acb_w750_threshold_pct':60,
}
GOLD = dict(roi=88.55, pf=1.215, dd=15.05, sharpe=4.38, wr=50.5, trades=2155)
# ── Data loading (cached per process) ────────────────────────────────────────
_data_cache = {}
def load_data():
"""Returns metadata only; actual data loaded lazily in run_backtest."""
from nautilus_dolphin.nautilus.ob_features import OBFeatureEngine
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
parquet_files = sorted(VBT_DIR.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
date_strings = [p.stem for p in parquet_files]
# Sample a few files to get vol_p60
all_vols = []
for pf in parquet_files[:3]:
tmp = pd.read_parquet(pf)
if 'BTCUSDT' in tmp.columns:
bp = tmp['BTCUSDT'].values
diffs = np.diff(bp) / bp[:-1]
for i in range(50, len(diffs)):
all_vols.append(np.std(diffs[i-50:i]))
del tmp
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.0002
print(f" Calibrated vol_p60: {vol_p60:.8f}")
OB_ASSETS = ["BTCUSDT", "ETHUSDT"]
_mock_ob = MockOBProvider(
imbalance_bias=-0.09, depth_scale=1.0, assets=OB_ASSETS,
imbalance_biases={"BTCUSDT":-0.086,"ETHUSDT":-0.092,
"BNBUSDT":+0.05, "SOLUSDT":+0.05},
)
ob_eng = OBFeatureEngine(_mock_ob)
# Preload only the core assets for 56 days (memory-safe now)
ob_eng.preload_date("mock", OB_ASSETS)
return dict(
parquet_files=parquet_files, date_strings=date_strings,
vol_p60=vol_p60, ob_eng=ob_eng, OB_ASSETS=OB_ASSETS,
)
def load_forewarner():
try:
from mc.mc_ml import DolphinForewarner
fw = DolphinForewarner(models_dir=MC_MODELS_DIR)
print(" MC-Forewarner loaded (5 models)")
return fw
except Exception as e:
print(f" MC-Forewarner unavailable: {e}")
return None
def run_backtest(engine_factory, name, forewarner=None, extra_kwargs=None):
"""
Run full 55-day backtest with gold-matching metrics (Lazy loading).
"""
import gc
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
d = load_data()
kw = ENGINE_KWARGS.copy()
if extra_kwargs: kw.update(extra_kwargs)
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng = engine_factory(kw)
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if forewarner is not None:
eng.set_mc_forewarner(forewarner, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps, daily_pnls = [], []
all_vols = []
for i, pf in enumerate(d['parquet_files']):
ds = pf.stem
# Lazy Load and cast to float32 to save RAM
df = pd.read_parquet(pf)
for c in df.columns:
if df[c].dtype == 'float64':
df[c] = df[c].astype('float32')
acols = [c for c in df.columns if c not in META_COLS]
# Per-day OB Preloading (Crucial for 230MB RAM)
if eng.ob_engine is not None:
eng.ob_engine.preload_date(ds, d['OB_ASSETS'])
# Optimized 5s dvol approximation
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dvol = np.zeros(len(df), dtype=np.float32)
if bp is not None:
rets = np.diff(bp.astype('float64')) / (bp[:-1].astype('float64') + 1e-9)
for j in range(50, len(rets)):
v = np.std(rets[j-50:j])
dvol[j+1] = v
if v > 0: all_vols.append(v)
cap_before = eng.capital
vp60 = np.percentile(all_vols, 60) if len(all_vols) > 1000 else d['vol_p60']
vol_ok = np.where(dvol > 0, dvol > vp60, False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
# CLEAR OB CACHE FOR DAY
if eng.ob_engine is not None:
eng.ob_engine._preloaded_placement.clear()
eng.ob_engine._preloaded_signal.clear()
eng.ob_engine._preloaded_market.clear()
eng.ob_engine._ts_to_idx.clear()
del df
gc.collect()
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
if n == 0:
return dict(name=name, roi=roi, pf=0, dd=0, wr=0, sharpe=0, trades=0)
def _abs(t): return t.pnl_absolute if hasattr(t,'pnl_absolute') else t.pnl_pct*250.
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
peak_cap, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
max_dd = max(max_dd, (peak_cap - cap) / peak_cap * 100.0)
dr = np.array([p/25000.*100. for p in daily_pnls])
sharpe = float(dr.mean()/(dr.std()+1e-9)*math.sqrt(365)) if len(dr)>1 else 0.
# Gather any engine-specific extra stats
extra = {}
for attr in ('gate_suppressed','gate_allowed','early_exits','sizing_scale_mean'):
v = getattr(eng, attr, None)
if v is not None: extra[attr] = v
return dict(name=name, roi=roi, pf=pf, dd=max_dd, wr=wr, sharpe=sharpe,
trades=n, **extra)
def print_table(results, gold=None):
hdr = f"{'Config':<42} {'ROI%':>7} {'PF':>6} {'DD%':>6} {'WR%':>6} {'Sharpe':>7} {'Trades':>7}"
print(hdr); print('-'*83)
if gold:
g = gold
print(f"{'*** GOLD ***':<42} {g['roi']:>7.2f} {g['pf']:>6.4f} {g['dd']:>6.2f} "
f"{g['wr']:>6.2f} {g['sharpe']:>7.3f} {g['trades']:>7d}")
print('-'*83)
for r in results:
extra = ''
if 'suppression_rate' in r: extra += f" gate_supp={r['suppression_rate']:.1f}%"
if 'early_exits' in r: extra += f" early_exits={r['early_exits']}"
if 'sizing_scale_mean' in r: extra += f" scale_mean={r['sizing_scale_mean']:.3f}"
print(f"{r['name']:<42} {r['roi']:>7.2f} {r['pf']:>6.4f} {r['dd']:>6.2f} "
f"{r['wr']:>6.2f} {r['sharpe']:>7.3f} {r['trades']:>7d}{extra}")
def log_results(results, outfile, gold=None, meta=None):
payload = {'gold': gold or GOLD, 'results': results}
if meta: payload['meta'] = meta
outfile = Path(outfile)
outfile.parent.mkdir(parents=True, exist_ok=True)
with open(outfile, 'w', encoding='utf-8') as f:
json.dump(payload, f, indent=2)
print(f"\n Logged → {outfile}")

351
nautilus_dolphin/dvae/flint_dvae_kernel.py Executable file
View File

@@ -0,0 +1,351 @@
"""
flint_dvae_kernel.py
Implementation of a SILOQY-compatible Temporal Disentangled VAE that leverages
the 550-bit arbitrary precision of FLINT via TailPreservingEDAIN_KL.
This script extracts Proxy A, B, C, D, and E from the T1 corpus,
normalizes them using the exact 550-bit TailPreservingEDAIN_KL,
and models the temporal dynamics to predict eigenspace stress precursors.
"""
import sys, os
import numpy as np
from pathlib import Path
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
# Ensure the project root is in the path
HERE = Path(__file__).parent
PROJECT_ROOT = r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict"
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
# Import the 550-bit kernel components
from SILOQY_NN_Kernel_COMPLETE6 import MCDAINLayer, FlintTensor, arb_mat, FLINT_AVAILABLE, with_precision, arb, safe_float
from nautilus_dolphin.dvae.corpus_builder import DolphinCorpus, OFF, T1 as T1_DIM
# --- 1. D-VAE Implementation ---
class SiloqyTemporalDVAE:
"""
Temporal Disentangled VAE adapted for the SILOQY framework.
Uses MCDAINLayer internally for robust normalization of highly kurtotic features.
"""
def __init__(self, input_dim=5, hidden_dim=64, latent_dim=8, regime_dim=4, beta=1.0, seq_len=10, precision_bits=550):
self.input_dim = input_dim
self.hidden_dim = hidden_dim
self.latent_dim = latent_dim
self.regime_dim = regime_dim
self.noise_dim = latent_dim - regime_dim
self.beta = beta
self.seq_len = seq_len
self.precision_bits = precision_bits
self.is_trained = False
self._init_weights()
# Instantiate the 550-bit precise normalizer
print(f"Initializing 550-bit MCDAINLayer for dimension {input_dim}...")
self.edain = MCDAINLayer(
input_dim=input_dim,
precision_bits=precision_bits,
use_ball_arithmetic=True
)
def _init_weights(self):
rng = np.random.RandomState(42)
scale = 0.1
self.W_ih = rng.randn(self.input_dim, self.hidden_dim * 4) * scale
self.W_hh = rng.randn(self.hidden_dim, self.hidden_dim * 4) * scale
self.b_h = np.zeros(self.hidden_dim * 4)
self.W_mu = rng.randn(self.hidden_dim, self.latent_dim) * scale
self.W_logvar = rng.randn(self.hidden_dim, self.latent_dim) * scale
self.b_mu = np.zeros(self.latent_dim)
self.b_logvar = np.zeros(self.latent_dim)
self.W_dec = rng.randn(self.latent_dim, self.hidden_dim) * scale
self.W_out = rng.randn(self.hidden_dim, self.input_dim) * scale
self.b_dec = np.zeros(self.hidden_dim)
self.b_out = np.zeros(self.input_dim)
self.regime_centroid = None
self.regime_threshold = None
def _sigmoid(self, x):
return 1.0 / (1.0 + np.exp(-np.clip(x, -500, 500)))
def _lstm_step(self, x, h, c):
gates = x @ self.W_ih + h @ self.W_hh + self.b_h
i, f, g, o = np.split(gates, 4, axis=-1)
i, f, o = self._sigmoid(i), self._sigmoid(f), self._sigmoid(o)
g = np.tanh(g)
c_new = f * c + i * g
h_new = o * np.tanh(c_new)
return h_new, c_new
def _encode_sequence(self, X_seq):
batch = X_seq.shape[0]
h = np.zeros((batch, self.hidden_dim))
c = np.zeros((batch, self.hidden_dim))
for t in range(X_seq.shape[1]):
h, c = self._lstm_step(X_seq[:, t, :], h, c)
mu = h @ self.W_mu + self.b_mu
logvar = h @ self.W_logvar + self.b_logvar
return mu, logvar
def _reparameterize(self, mu, logvar, rng=None):
if rng is None:
rng = np.random.RandomState()
std = np.exp(0.5 * logvar)
eps = rng.randn(*mu.shape)
return mu + eps * std
def _decode(self, z):
h = np.tanh(z @ self.W_dec + self.b_dec)
return h @ self.W_out + self.b_out
def _tc_decomposition(self, mu, logvar):
batch = mu.shape[0]
kl_dim = 0.5 * np.sum(mu**2 + np.exp(logvar) - logvar - 1, axis=1)
var = np.exp(logvar)
cov = np.zeros((self.latent_dim, self.latent_dim))
for i in range(batch):
cov += np.diag(var[i])
cov /= batch
cov += np.outer(mu.mean(0), mu.mean(0))
det = np.linalg.det(cov + 1e-6 * np.eye(self.latent_dim))
tc = 0.5 * (np.sum(np.log(np.diag(cov) + 1e-6)) - np.log(det + 1e-6))
return np.mean(kl_dim), max(0, tc)
def _loss(self, X_seq, X_target, mu, logvar, z):
recon = self._decode(z)
recon_loss = np.mean((recon - X_target) ** 2)
kl_dim, tc = self._tc_decomposition(mu, logvar)
total = recon_loss + kl_dim + self.beta * tc
return total, recon_loss, kl_dim, tc
def _build_sequences(self, X):
n = len(X) - self.seq_len
if n <= 0:
return None, None
seqs = np.array([X[i:i+self.seq_len] for i in range(n)])
targets = X[self.seq_len:]
return seqs, targets
def _convert_arb_to_float(self, matrix_arb) -> np.ndarray:
"""Safely convert FLINT arb_mat to float64 numpy array."""
rows, cols = matrix_arb.nrows(), matrix_arb.ncols()
out = np.zeros((rows, cols), dtype=np.float64)
for i in range(rows):
for j in range(cols):
out[i, j] = safe_float(matrix_arb[i, j])
return out
def _normalize_native(self, X):
"""Native FLINT arbitrary precision MCDAIN logic (bypassing PyTorch)."""
rows, cols = X.shape
X_norm = np.zeros_like(X, dtype=np.float64)
with with_precision(self.precision_bits):
for j in range(cols):
# Calculate vector magnitude for this proxy
sum_sq = arb(0)
for i in range(rows):
sum_sq += arb(str(X[i, j])) ** 2
magnitude = sum_sq.sqrt()
# MCDAIN Analytical params (activation='log')
log_mag = magnitude.log()
mean = magnitude * arb("0.1")
scale = arb("1.0") / (log_mag + arb("1e-8"))
gate = arb("1.0") / (arb("1.0") + (-log_mag).exp())
# Apply normalization
for i in range(rows):
val = arb(str(X[i, j]))
val_centered = val - mean
val_scaled = val_centered * scale
val_gated = val_scaled * gate
X_norm[i, j] = safe_float(val_gated)
# Replace remaining NaNs from float conversion limit if any
X_norm = np.nan_to_num(X_norm, nan=0.0, posinf=5.0, neginf=-5.0)
return X_norm
def fit(self, X, epochs=10, lr=0.001, batch_size=64, verbose=True):
print(f"Normalizing input (shape {X.shape}) natively with MCDAIN Analytical Math (550-bit precision)...")
# 1. Normalize with 550-bit MCDAIN Logic bypassing PyTorch
X_norm = self._normalize_native(X)
# 2. Sequence building
seqs, targets = self._build_sequences(X_norm)
if seqs is None:
return self
n = len(seqs)
rng = np.random.RandomState(42)
losses = []
print(f"Training Temporal D-VAE on normalized sequence space (n={n})...")
for epoch in range(epochs):
idx = rng.permutation(n)
epoch_loss = 0
for start in range(0, n, batch_size):
batch_idx = idx[start:start+batch_size]
X_seq = seqs[batch_idx]
X_tgt = targets[batch_idx]
mu, logvar = self._encode_sequence(X_seq)
z = self._reparameterize(mu, logvar, rng)
loss, rl, kl, tc = self._loss(X_seq, X_tgt, mu, logvar, z)
epoch_loss += loss * len(batch_idx)
grad_scale = lr * 0.1
noise = rng.randn(*self.W_mu.shape) * grad_scale
self.W_mu -= noise * (loss - 1.0)
self.W_logvar -= rng.randn(*self.W_logvar.shape) * grad_scale * (loss - 1.0)
self.W_dec -= rng.randn(*self.W_dec.shape) * grad_scale * rl
self.W_out -= rng.randn(*self.W_out.shape) * grad_scale * rl
epoch_loss /= n
losses.append(epoch_loss)
if verbose and epoch % 2 == 0:
print(f" Epoch {epoch}/{epochs}: loss={epoch_loss:.4f}")
# Finalize
mu_all, _ = self._encode_sequence(seqs)
z_regime = mu_all[:, :self.regime_dim]
self.regime_centroid = z_regime.mean(axis=0)
dists = np.linalg.norm(z_regime - self.regime_centroid, axis=1)
self.regime_threshold = np.mean(dists) + 2.0 * np.std(dists)
self.is_trained = True
return self
def encode(self, X):
if not self.is_trained:
return None, None
X_norm = self._normalize_native(X)
seqs, _ = self._build_sequences(X_norm)
if seqs is None:
return None, None
mu, logvar = self._encode_sequence(seqs)
z_regime = mu[:, :self.regime_dim]
z_noise = mu[:, self.regime_dim:]
return z_regime, z_noise
# --- 2. Main Execution Script ---
def run_analysis():
if not FLINT_AVAILABLE:
print("CRITICAL ERROR: FLINT library is required but not loaded.")
sys.exit(1)
print("Loading corpus...", flush=True)
corpus_path = str(HERE / 'corpus_cache.npz')
if not os.path.exists(corpus_path):
print(f"Corpus not found at {corpus_path}. Make sure to build it.")
sys.exit(1)
corpus = DolphinCorpus.load(corpus_path)
idx = corpus.mask[:, 1] # T1 availability
X_e = corpus.X[idx]
t1 = X_e[:, OFF[1]:OFF[1]+T1_DIM].copy()
# Feature extraction
vel_w50 = t1[:, 1]
vel_w300 = t1[:, 11]
vel_w750 = t1[:, 16]
inst_w50 = t1[:, 3]
inst_w300= t1[:, 13]
gap_w50 = t1[:, 2]
print("\n--- Generating Proxies ---")
proxy_A = -0.674*vel_w750 - 0.357*vel_w300 + 0.421*inst_w50
proxy_B = inst_w50 - vel_w750
proxy_C = vel_w50 - vel_w750
proxy_D = inst_w50 * (-vel_w750)
proxy_E = (inst_w50 - inst_w300) - (vel_w50 - vel_w750)
# Stack proxies into single float array for EDAIN
X_proxies = np.column_stack([proxy_A, proxy_B, proxy_C, proxy_D, proxy_E])
print(f"Proxies Stacked. Shape: {X_proxies.shape}")
for i, name in enumerate(['Proxy A', 'Proxy B', 'Proxy C (kurt=3798)', 'Proxy D', 'Proxy E']):
p = X_proxies[:, i]
kurt = float(((p - p.mean())**4).mean() / (p.std()**4 + 1e-8))
print(f" {name}: skew={float(((p - p.mean())**3).mean() / (p.std()**3 + 1e-8)):.2f}, kurt={kurt:.2f}")
# Initialize SILOQY D-VAE
dvae = SiloqyTemporalDVAE(input_dim=5, hidden_dim=32, latent_dim=8, regime_dim=4, beta=1.0, seq_len=10, precision_bits=550)
print("\n--- Fitting SILOQY D-VAE ---")
# Subsample data to make 550-bit EDAIN training tractable for testing
N_sub = min(2000, len(X_proxies))
sub_idx = np.arange(N_sub)
X_sub = X_proxies[sub_idx]
# Fit the normalizer and temporal D-VAE
dvae.fit(X_sub, epochs=8, batch_size=64, verbose=True)
print("\n--- Evaluating Predictive Power ---")
# Build Precursor Labels
# Goal: Did eigenspace stress follow within N scans?
# We define "stress" as gap_ratio collapse AND instability spike in the FUTURE (T + 2 to T + 12 scans)
# Since we use 10-step sequences, the output of sequence ending at time T corresponds to T.
seqs_len = len(X_sub) - dvae.seq_len
future_horizon = 10
# Labels for the end of sequence `T` looking forward to `T + future_horizon`
labels = np.zeros(seqs_len)
for idx_seq in range(seqs_len):
cur_t = idx_seq + dvae.seq_len
if cur_t + future_horizon >= len(X_sub):
continue
future_inst = inst_w50[cur_t:cur_t+future_horizon]
future_gap = gap_w50[cur_t:cur_t+future_horizon]
# Stress condition
inst_spike = np.any(future_inst > 0.40)
gap_collapse = np.any(future_gap < 0.60)
if inst_spike and gap_collapse:
labels[idx_seq] = 1.0
print(f"Precursor Labels Generated. Positive Class: {labels.mean()*100:.1f}%")
# Encode full validation space
z_regime, z_noise = dvae.encode(X_sub)
if z_regime is not None:
# Align labels
valid_idx = np.arange(len(labels))
valid_z = z_regime[valid_idx]
valid_y = labels[valid_idx]
# Regress
model = LogisticRegression(class_weight='balanced')
model.fit(valid_z, valid_y)
preds = model.predict_proba(valid_z)[:, 1]
auc = roc_auc_score(valid_y, preds)
print("\n--- RESULTS ---")
print(f"Logistic Regression AUC predicting future stress: {auc:.4f}")
print("Coefficients on Latent `z_regime` factors:")
for dim, coef in enumerate(model.coef_[0]):
print(f" z_regime[{dim}]: {coef:+.4f}")
# Correlate transformed proxies with target using normalized arb floats
print("\nDirect predictivity of 550-bit Normalized Proxies:")
X_norm_float = dvae._normalize_native(X_sub)
for k, proxy_name in enumerate(['Proxy A', 'Proxy B', 'Proxy C', 'Proxy D', 'Proxy E']):
px = X_norm_float[dvae.seq_len:, k]
mask = ~np.isnan(px) & ~np.isnan(labels)
if mask.sum() > 0:
corr = np.corrcoef(px[mask], labels[mask])[0, 1]
print(f" {proxy_name}: r = {corr:+.4f}")
else:
print("Failed to encode latent variables.")
if __name__ == '__main__':
run_analysis()

274
nautilus_dolphin/dvae/flint_hd_vae.py Executable file
View File

@@ -0,0 +1,274 @@
"""
flint_hd_vae.py
===============
SILOQY-compatible HD-VAE with inverse projection decoder.
Architecture:
Encoder:
T1 (20-dim)
→ MCDAIN 550-bit normalisation (no upstream modification — read-only call)
→ HD random projection W_enc (20×512), ReLU → h (512)
→ Linear bottleneck: W_mu (512×8), W_lv (512×8) → mu, logvar (8)
→ reparameterisation → z (8)
Decoder (inverse projection — THE NEW PIECE):
z (8)
→ Linear W_dec (8×512), ReLU → h_hat (512) *inverse of bottleneck*
→ Linear W_out (512×20) → T1_hat (20) *pseudo-inverse of HD proj*
Loss:
recon = MSE(T1_hat, T1_norm)
KL = -0.5 * sum(1 + logvar - mu^2 - exp(logvar)) [standard VAE KL]
total = recon + beta * KL
No upstream files are modified. All SILOQY calls are read-only.
"""
import sys, os
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict")
import numpy as np
from pathlib import Path
from SILOQY_NN_Kernel_COMPLETE6 import arb, safe_float, FLINT_AVAILABLE, with_precision
EPS = 1e-8
# ── MCDAIN 550-bit normalisation (read-only logic, no upstream changes) ────
def mcdain_550bit(X_raw: np.ndarray) -> np.ndarray:
"""Apply MCDAIN analytical normalisation at 550-bit precision."""
rows, cols = X_raw.shape
X_norm = np.zeros_like(X_raw, dtype=np.float64)
with with_precision(550):
for j in range(cols):
col = X_raw[:, j]
col_abs = np.abs(col[np.isfinite(col)])
if len(col_abs) == 0 or col_abs.mean() < 1e-12:
continue
magnitude = arb(str(float(col_abs.mean())))
log_mag = magnitude.log()
mean_val = magnitude * arb("0.1")
scale_val = arb("1.0") / (log_mag + arb("1e-8"))
gate_val = arb("1.0") / (arb("1.0") + (-log_mag).exp())
m = safe_float(mean_val)
s = safe_float(scale_val)
g = safe_float(gate_val)
X_norm[:, j] = np.clip((X_raw[:, j] - m) * s * g, -10, 10)
return np.nan_to_num(X_norm, nan=0.0, posinf=5.0, neginf=-5.0)
# ── Adam optimiser state ───────────────────────────────────────────────────
class AdamParam:
def __init__(self, shape, seed=0):
rng = np.random.RandomState(seed)
scale = np.sqrt(2.0 / shape[0])
self.W = rng.randn(*shape).astype(np.float64) * scale
self.m = np.zeros_like(self.W)
self.v = np.zeros_like(self.W)
self.t = 0
def step(self, grad, lr=1e-3, b1=0.9, b2=0.999):
self.t += 1
self.m = b1 * self.m + (1 - b1) * grad
self.v = b2 * self.v + (1 - b2) * grad**2
m_hat = self.m / (1 - b1**self.t)
v_hat = self.v / (1 - b2**self.t)
self.W -= lr * m_hat / (np.sqrt(v_hat) + EPS)
# ── FlintHDVAE ────────────────────────────────────────────────────────────
class FlintHDVAE:
"""
HD-VAE with 550-bit MCDAIN encoder normalisation.
Inverse projection decoder: z(8) → Linear+ReLU(512) → Linear(20).
"""
def __init__(self, input_dim=20, hd_dim=512, latent_dim=8,
beta=0.5, seed=42, use_flint_norm=True):
self.input_dim = input_dim
self.hd_dim = hd_dim
self.latent_dim = latent_dim
self.beta = beta
self.use_flint = use_flint_norm and FLINT_AVAILABLE
rng = np.random.RandomState(seed)
# Fixed random HD projection (encoder side, non-trainable)
self.W_hd = rng.randn(input_dim, hd_dim).astype(np.float64) * np.sqrt(2.0/input_dim)
# Trainable parameters — encoder bottleneck
self.P_mu = AdamParam((hd_dim, latent_dim), seed=seed+1)
self.P_lv = AdamParam((hd_dim, latent_dim), seed=seed+2)
# Trainable parameters — DECODER (inverse projection, THE NEW PIECE)
self.P_dec = AdamParam((latent_dim, hd_dim), seed=seed+3) # z→h_hat
self.P_out = AdamParam((hd_dim, input_dim), seed=seed+4) # h_hat→T1_hat
# Normaliser stats (fitted once)
self._norm_fitted = False
self._norm_mu = np.zeros(input_dim)
self._norm_sd = np.ones(input_dim)
self.train_losses = []
# ── Normalisation ──────────────────────────────────────────────────────
def fit_normaliser(self, X: np.ndarray):
"""Fit normaliser stats from the FULL training set (called once).
For MCDAIN: computes global per-column m/s/g and stores them so that
all subsequent _normalise() calls are deterministic (no batch-dependency).
Falls back to z-score if FLINT unavailable."""
self._norm_mu = X.mean(0)
self._norm_sd = X.std(0) + EPS
if self.use_flint:
# Compute MCDAIN params column-wise on full X, store as fixed stats
X_norm_full = mcdain_550bit(X)
# Store the effective per-column shift/scale as z-score of the MCDAIN output
self._mcdain_mu = X_norm_full.mean(0)
self._mcdain_sd = X_norm_full.std(0) + EPS
# Also store the raw MCDAIN params by fitting a passthrough
self._mcdain_fitted = True
self._X_norm_ref = X_norm_full # kept for diagnostics only (not used in loops)
self._norm_fitted = True
def _normalise(self, X: np.ndarray) -> np.ndarray:
if self.use_flint and self._norm_fitted and hasattr(self, '_mcdain_fitted'):
# Apply MCDAIN then standardise using TRAINING statistics
# This makes normalisation deterministic regardless of batch size
raw = mcdain_550bit(X)
return (raw - self._mcdain_mu) / self._mcdain_sd
return (X - self._norm_mu) / self._norm_sd
# ── Forward pass ──────────────────────────────────────────────────────
def _encode(self, X_norm, rng):
"""X_norm (B,20) → h (B,512) → mu,logvar (B,8) → z (B,8)"""
h = np.maximum(0, X_norm @ self.W_hd) # (B, 512) ReLU
mu = h @ self.P_mu.W # (B, 8)
lv = np.clip(h @ self.P_lv.W, -4, 4) # (B, 8)
eps = rng.randn(*mu.shape)
z = mu + np.exp(0.5 * lv) * eps # reparam
return h, mu, lv, z
def _decode(self, z):
"""z (B,8) → h_hat (B,512) → T1_hat (B,20) — INVERSE PROJECTION"""
h_hat = np.maximum(0, z @ self.P_dec.W) # (B, 512) ReLU
T1_hat = h_hat @ self.P_out.W # (B, 20) linear
return h_hat, T1_hat
# ── Loss ──────────────────────────────────────────────────────────────
def _loss(self, T1_norm, T1_hat, mu, lv):
B = len(T1_norm)
recon = np.mean((T1_hat - T1_norm)**2)
kl = -0.5 * np.mean(1 + lv - mu**2 - np.exp(lv))
total = recon + self.beta * kl
return total, recon, kl
# ── Backward (analytical gradients) ───────────────────────────────────
def _backward(self, T1_norm, T1_hat, h, h_hat, mu, lv, z, lr):
B = len(T1_norm)
# ── Decoder gradients ────────────────────────────────────────────
# dL/dT1_hat = 2*(T1_hat - T1_norm) / (B*D)
dT1 = 2.0 * (T1_hat - T1_norm) / (B * self.input_dim)
# W_out: h_hat.T @ dT1
dW_out = h_hat.T @ dT1 # (512, 20)
self.P_out.step(dW_out, lr)
# Back through ReLU of h_hat
dh_hat = (dT1 @ self.P_out.W.T) * (h_hat > 0) # (B, 512)
# W_dec: z.T @ dh_hat
dW_dec = z.T @ dh_hat # (8, 512)
self.P_dec.step(dW_dec, lr)
# dz from decoder
dz_dec = dh_hat @ self.P_dec.W.T # (B, 8)
# ── KL gradients (standard VAE) ──────────────────────────────────
# dKL/dmu = mu/B; dKL/dlv = 0.5*(exp(lv)-1)/B
dmu_kl = self.beta * mu / B
dlv_kl = self.beta * 0.5 * (np.exp(lv) - 1) / B
# ── Reparameterisation: dz flows back to mu and lv ───────────────
# z = mu + exp(0.5*lv)*eps → dmu = dz, dlv = dz*0.5*z (approx)
dmu = dz_dec + dmu_kl
dlv = dz_dec * 0.5 * (z - mu) + dlv_kl # chain rule
# ── Encoder bottleneck gradients ─────────────────────────────────
dW_mu = h.T @ dmu # (512, 8)
dW_lv = h.T @ dlv
self.P_mu.step(dW_mu, lr)
self.P_lv.step(dW_lv, lr)
# (W_hd is fixed, no gradient needed for it)
# ── Training ──────────────────────────────────────────────────────────
def fit(self, X: np.ndarray, epochs=30, lr=1e-3,
batch_size=256, verbose=True, warmup_frac=0.3):
"""
warmup_frac: fraction of epochs over which beta ramps 0 → self.beta.
Prevents KL from dominating before the decoder learns to reconstruct.
"""
rng = np.random.RandomState(42)
self.fit_normaliser(X) # computes global MCDAIN stats once
X_norm = self._normalise(X) # normalise full dataset once; stable across batches
N = len(X_norm)
target_beta = self.beta
warmup_epochs = max(1, int(epochs * warmup_frac))
for epoch in range(1, epochs + 1):
# KL warmup: ramp beta from 0 to target over first warmup_epochs
if epoch <= warmup_epochs:
self.beta = target_beta * (epoch / warmup_epochs)
else:
self.beta = target_beta
idx = rng.permutation(N)
ep_loss = ep_recon = ep_kl = 0.0
n_batches = 0
for start in range(0, N, batch_size):
bi = idx[start:start + batch_size]
Xb = X_norm[bi] # already normalised with global stats
h, mu, lv, z = self._encode(Xb, rng)
h_hat, T1_hat = self._decode(z)
loss, recon, kl = self._loss(Xb, T1_hat, mu, lv)
self._backward(Xb, T1_hat, h, h_hat, mu, lv, z, lr)
ep_loss += loss; ep_recon += recon; ep_kl += kl
n_batches += 1
ep_loss /= n_batches; ep_recon /= n_batches; ep_kl /= n_batches
self.train_losses.append(ep_loss)
if verbose and (epoch % 5 == 0 or epoch == 1):
# Anti-collapse diagnostic: encode a fixed held-out sample
sample_norm = X_norm[:min(1000, N)]
_, mu_s, _, _ = self._encode(sample_norm, rng)
var_per_dim = mu_s.var(0)
print(f" ep{epoch:3d}/{epochs} beta={self.beta:.3f} "
f"loss={ep_loss:.4f} recon={ep_recon:.4f} kl={ep_kl:.4f} "
f"z_var=[{' '.join(f'{v:.3f}' for v in var_per_dim)}]")
self.beta = target_beta # restore after training
return self
# ── Encode for downstream use ─────────────────────────────────────────
def encode(self, X: np.ndarray) -> np.ndarray:
"""Return deterministic mu (B, latent_dim) for all samples.
Normalisation is deterministic (global MCDAIN stats from fit_normaliser)."""
rng = np.random.RandomState(0)
STEP = 512
mus = []
for s in range(0, len(X), STEP):
Xb = self._normalise(X[s:s+STEP])
_, mu, _, _ = self._encode(Xb, rng)
mus.append(mu)
return np.concatenate(mus)
def reconstruct(self, X: np.ndarray) -> np.ndarray:
"""Returns (T1_hat, X_norm) both in the same normalised space.
Normalisation is deterministic (global MCDAIN stats from fit_normaliser)."""
rng = np.random.RandomState(0)
Xn = self._normalise(X)
STEP = 512
hats = []
for s in range(0, len(Xn), STEP):
_, mu, _, _ = self._encode(Xn[s:s+STEP], rng)
_, T1_hat = self._decode(mu)
hats.append(T1_hat)
return np.concatenate(hats), Xn

225
nautilus_dolphin/dvae/flint_precursor_sweep.py Executable file
View File

@@ -0,0 +1,225 @@
"""
SILOQY 550-bit Precursor Sweep — NO MODIFICATIONS TO UPSTREAM CODE.
Runs on full 16K eigen corpus, tests multiple:
- Precursor label thresholds (rare extreme events)
- Horizons (K=5, 10, 20, 50 scans ahead)
- ML approaches: Logistic, Ridge, k-NN, threshold-only baseline
Reports AUC, Precision@TopDecile, and direct proxy predictivity.
"""
import sys, os
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict")
import numpy as np
from pathlib import Path
from sklearn.linear_model import LogisticRegression, Ridge
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import roc_auc_score, average_precision_score
from sklearn.model_selection import TimeSeriesSplit
HERE = Path(__file__).parent
# ── Load corpus ────────────────────────────────────────────────────────────
print("Loading corpus (16K eigen samples)...")
from corpus_builder import DolphinCorpus, OFF, T1 as T1_DIM
corpus = DolphinCorpus.load(str(HERE / 'corpus_cache.npz'))
idx_mask = corpus.mask[:, 1]
X_e = corpus.X[idx_mask]
t1 = X_e[:, OFF[1]:OFF[1]+T1_DIM].copy()
N = len(t1)
print(f"N={N} samples")
# ── Feature extraction ─────────────────────────────────────────────────────
vel_w50 = t1[:, 1]
vel_w150 = t1[:, 6]
vel_w300 = t1[:, 11]
vel_w750 = t1[:, 16]
inst_w50 = t1[:, 3]
inst_w150= t1[:, 8]
inst_w300= t1[:, 13]
gap_w50 = t1[:, 2]
gap_w300 = t1[:, 12]
lmax_w50 = t1[:, 0]
proxy_A = -0.674*vel_w750 - 0.357*vel_w300 + 0.421*inst_w50
proxy_B = inst_w50 - vel_w750
proxy_C = vel_w50 - vel_w750
proxy_D = inst_w50 * (-vel_w750)
proxy_E = (inst_w50 - inst_w300) - (vel_w50 - vel_w750)
X_proxies = np.column_stack([proxy_A, proxy_B, proxy_C, proxy_D, proxy_E])
proxy_names = ['A(linear)', 'B(inst-vel750)', 'C(vel50-vel750,k=3798)', 'D(inst*-vel750)', 'E(dinst-dvel)']
# ── 550-bit MCDAIN normalization (from flint_dvae_kernel.py, read-only) ────
print("\nApplying 550-bit MCDAIN normalization to proxies...")
from SILOQY_NN_Kernel_COMPLETE6 import arb, safe_float, FLINT_AVAILABLE, with_precision
def mcdain_550bit(X_raw):
"""Read-only implementation of MCDAIN analytical logic at 550-bit."""
rows, cols = X_raw.shape
X_norm = np.zeros_like(X_raw, dtype=np.float64)
with with_precision(550):
for j in range(cols):
col = X_raw[:, j]
col_abs = np.abs(col[np.isfinite(col)])
if len(col_abs) == 0 or col_abs.mean() < 1e-12:
continue
magnitude = arb(str(float(col_abs.mean())))
log_mag = magnitude.log()
mean_val = magnitude * arb("0.1")
scale_val = arb("1.0") / (log_mag + arb("1e-8"))
gate_val = arb("1.0") / (arb("1.0") + (-log_mag).exp())
m = safe_float(mean_val)
s = safe_float(scale_val)
g = safe_float(gate_val)
X_norm[:, j] = np.clip((X_raw[:, j] - m) * s * g, -10, 10)
X_norm = np.nan_to_num(X_norm, nan=0.0, posinf=5.0, neginf=-5.0)
return X_norm
X_norm = mcdain_550bit(X_proxies)
print(f" Normalized. std per proxy: {X_norm.std(0).round(4)}")
print(f" Kurtosis after normalization: {[round(float(((X_norm[:,j]-X_norm[:,j].mean())**4).mean()/(X_norm[:,j].std()**4+1e-8)),2) for j in range(5)]}")
# ── Build precursor labels at multiple thresholds and horizons ─────────────
print("\n" + "="*65)
print("PRECURSOR LABEL SWEEP")
print("="*65)
# inst_w50 thresholds (what percentile constitutes "stress"?)
inst_p80 = np.percentile(inst_w50, 80) # lenient
inst_p90 = np.percentile(inst_w50, 90) # moderate
inst_p95 = np.percentile(inst_w50, 95) # strict
gap_p20 = np.percentile(gap_w50, 20) # gap collapse (low = collapse)
gap_p10 = np.percentile(gap_w50, 10) # strict gap collapse
print(f"inst_w50 thresholds: p80={inst_p80:.4f} p90={inst_p90:.4f} p95={inst_p95:.4f}")
print(f"gap_w50 thresholds: p20={gap_p20:.4f} p10={gap_p10:.4f}")
def build_labels(horizon, inst_thresh, gap_thresh):
"""Did eigenspace stress (inst spike AND gap collapse) occur in next K scans?"""
labels = np.zeros(N, dtype=np.float32)
for i in range(N - horizon):
future_inst = inst_w50[i+1:i+1+horizon]
future_gap = gap_w50[i+1:i+1+horizon]
if np.any(future_inst > inst_thresh) and np.any(future_gap < gap_thresh):
labels[i] = 1.0
return labels
configs = [
('K=10 lenient', 10, inst_p80, gap_p20),
('K=10 moderate', 10, inst_p90, gap_p10),
('K=20 moderate', 20, inst_p90, gap_p10),
('K=20 strict', 20, inst_p95, gap_p10),
('K=50 strict', 50, inst_p95, gap_p10),
]
results = []
for cfg_name, K, it, gt in configs:
y = build_labels(K, it, gt)
pos_rate = y.mean()
print(f"\n [{cfg_name}] K={K} inst>{it:.3f} gap<{gt:.3f} pos_rate={pos_rate*100:.1f}%")
# Skip degenerate
if pos_rate < 0.02 or pos_rate > 0.60:
print(f" Skipping (pos_rate out of range)")
continue
# ── Evaluate each proxy directly ─────────────────────────────────────
print(f" Direct proxy AUC (no model):")
best_proxy_auc = 0
for j, pname in enumerate(proxy_names):
px = X_norm[:-K, j] if K > 0 else X_norm[:, j]
yy = y[:-K] if K > 0 else y
valid = np.isfinite(px) & np.isfinite(yy)
if valid.sum() < 100:
continue
try:
auc = roc_auc_score(yy[valid], px[valid])
auc = max(auc, 1-auc) # flip if < 0.5
best_proxy_auc = max(best_proxy_auc, auc)
if auc > 0.52:
print(f" {pname:<30} AUC={auc:.4f} *")
else:
print(f" {pname:<30} AUC={auc:.4f}")
except Exception:
pass
# ── Logistic regression on all proxies ───────────────────────────────
Xf = X_norm[:-K]
yf = y[:-K]
valid = np.isfinite(Xf).all(1) & np.isfinite(yf)
Xf, yf = Xf[valid], yf[valid]
if len(Xf) < 200:
continue
try:
# Chronological 3-fold split
n_val = len(Xf) // 4
X_train, X_val = Xf[:-n_val], Xf[-n_val:]
y_train, y_val = yf[:-n_val], yf[-n_val:]
lr = LogisticRegression(class_weight='balanced', max_iter=500, C=0.1)
lr.fit(X_train, y_train)
preds = lr.predict_proba(X_val)[:, 1]
auc_lr = roc_auc_score(y_val, preds)
auc_lr = max(auc_lr, 1-auc_lr)
ap_lr = average_precision_score(y_val, preds)
print(f" LogReg (OOS): AUC={auc_lr:.4f} AvgPrecision={ap_lr:.4f}")
except Exception as ex:
print(f" LogReg failed: {ex}")
# ── k-NN (captures non-linear manifold structure) ─────────────────────
try:
knn = KNeighborsClassifier(n_neighbors=15, metric='euclidean')
knn.fit(X_train, y_train)
preds_knn = knn.predict_proba(X_val)[:, 1]
auc_knn = roc_auc_score(y_val, preds_knn)
auc_knn = max(auc_knn, 1-auc_knn)
print(f" k-NN (k=15): AUC={auc_knn:.4f}")
except Exception as ex:
print(f" kNN failed: {ex}")
results.append((cfg_name, K, pos_rate, best_proxy_auc,
auc_lr if 'auc_lr' in dir() else 0,
auc_knn if 'auc_knn' in dir() else 0))
# ── Temporal structure: HOW MANY SCANS AHEAD does the signal lead? ─────────
print("\n" + "="*65)
print("TEMPORAL LEAD STRUCTURE: proxy_B vs future inst/gap (by horizon)")
print("="*65)
print(f" {'Horizon':>10} {'AUC(B)':>8} {'AUC(C)':>8} {'pos_rate':>9}")
for K in [1, 2, 5, 10, 20, 30, 50, 100]:
y_k = build_labels(K, inst_p90, gap_p10)
if y_k.mean() < 0.01 or y_k.mean() > 0.80:
continue
pB = X_norm[:-K, 1] # proxy_B normalized
pC = X_norm[:-K, 2] # proxy_C normalized
yy = y_k[:-K]
valid = np.isfinite(pB) & np.isfinite(pC) & np.isfinite(yy)
if valid.sum() < 100:
continue
try:
aB = roc_auc_score(yy[valid], pB[valid])
aB = max(aB, 1-aB)
aC = roc_auc_score(yy[valid], pC[valid])
aC = max(aC, 1-aC)
print(f" K={K:>3} scans ahead: AUC(B)={aB:.4f} AUC(C)={aC:.4f} pos={y_k.mean()*100:.1f}%")
except Exception:
pass
# ── 512-bit DVAE question: variance per proxy before/after normalization ───
print("\n" + "="*65)
print("550-BIT FLINT EFFECT: variance recovery in heavy-tailed proxies")
print("="*65)
for j, pname in enumerate(proxy_names):
raw = X_proxies[:, j]
norm = X_norm[:, j]
kurt_raw = float(((raw-raw.mean())**4).mean() / (raw.std()**4 + 1e-8))
kurt_norm = float(((norm-norm.mean())**4).mean() / (norm.std()**4 + 1e-8))
# Fraction of samples that would be clipped at ±3σ in float64 z-score
z64 = (raw - raw.mean()) / (raw.std() + 1e-8)
clip_pct = (np.abs(z64) > 3).mean() * 100
print(f" {pname:<32} kurt_raw={kurt_raw:8.1f} kurt_norm={kurt_norm:6.2f} "
f"tail_samples={clip_pct:.1f}%_beyond_3sigma")
print("\nDone.")

88
nautilus_dolphin/dvae/flint_vs_float_analysis.py Executable file
View File

@@ -0,0 +1,88 @@
"""
flint_vs_float_analysis.py
Differential analysis of 550-bit FLINT vs 64-bit float64 precision
specifically for Proxy C (kurtosis = 3798).
"""
import numpy as np
import sys
from pathlib import Path
PROJECT_ROOT = r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict"
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
from SILOQY_NN_Kernel_COMPLETE6 import arb, with_precision, safe_float, FLINT_AVAILABLE
from nautilus_dolphin.dvae.corpus_builder import DolphinCorpus, OFF, T1 as T1_DIM
def compare_precision():
corpus_path = str(Path(PROJECT_ROOT) / 'nautilus_dolphin' / 'dvae' / 'corpus_cache.npz')
corpus = DolphinCorpus.load(corpus_path)
X_e = corpus.X[corpus.mask[:, 1]]
t1 = X_e[:, OFF[1]:OFF[1]+T1_DIM]
# Proxy C: vel_w50 - vel_w750
proxy_c = t1[:, 1] - t1[:, 16]
# Select extreme tails
tails_idx = np.where(np.abs(proxy_c) > np.percentile(np.abs(proxy_c), 99))[0]
sample_tails = proxy_c[tails_idx]
kurt = float(((proxy_c - proxy_c.mean())**4).mean() / (proxy_c.std()**4 + 1e-8))
print(f"Analyzing {len(sample_tails)} extreme samples from Proxy C (kurt={kurt:.2f})")
# MCDAIN logic: y = (x - mean) * scale * gate
# scale = 1.0 / (log(mag) + eps)
mag_f64 = np.sqrt(np.mean(proxy_c**2))
log_mag_f64 = np.log(mag_f64 + 1e-8)
scale_f64 = 1.0 / (log_mag_f64 + 1e-8)
results = []
with with_precision(550):
# Calc 550-bit magnitude
sum_sq_arb = arb(0)
for val in proxy_c:
sum_sq_arb += arb(str(val))**2
mag_arb = (sum_sq_arb / arb(len(proxy_c))).sqrt()
log_mag_arb = mag_arb.log()
scale_arb = arb(1) / (log_mag_arb + arb("1e-8"))
for x in sample_tails[:10]:
# 64-bit path
y_f64 = x * scale_f64
# 550-bit path
x_arb = arb(str(x))
y_arb = x_arb * scale_arb
y_arb_to_f = safe_float(y_arb)
diff = abs(y_f64 - y_arb_to_f)
results.append((x, y_f64, y_arb_to_f, diff))
print("\n| Input (Extreme) | Float64 Norm | 550-bit Norm | Delta |")
print("|-----------------|--------------|--------------|-------|")
for x, f, a, d in results:
print(f"| {x:15.10f} | {f:12.8f} | {a:12.8f} | {d:.2e} |")
# Gradient Stability Mock
# (x + eps) - (x) / eps
eps_range = [1e-8, 1e-15, 1e-30]
print("\nNumerical Gradient Stability (Finite Difference):")
x_test = sample_tails[0]
for e in eps_range:
# Float64
g_f64 = ((x_test + e) * scale_f64 - (x_test) * scale_f64) / e
# 550 bit
with with_precision(550):
e_arb = arb(str(e))
x_arb = arb(str(x_test))
g_arb = ((x_arb + e_arb) * scale_arb - (x_arb) * scale_arb) / e_arb
g_arb_f = safe_float(g_arb)
print(f" eps={e:.1e}: Float64 Grad={g_f64:.8f}, 550-bit Grad={g_arb_f:.8f}")
if __name__ == "__main__":
compare_precision()

BIN
nautilus_dolphin/dvae/hdvae_checkpoint.npz Executable file
View File

Binary file not shown.

559
nautilus_dolphin/dvae/hierarchical_dvae.py Executable file
View File

@@ -0,0 +1,559 @@
"""
Hierarchical Disentangled VAE (H-D-VAE) for DOLPHIN
=====================================================
State-of-the-art β-TCVAE with:
- 3-level HIERARCHY of disentangled latent codes
- Real backpropagation (analytical gradients, no random noise)
- Masking for missing tiers (pre-eigenvalue era)
- Curriculum-aware: can freeze/thaw tiers during training
- Spectral features with 512-bit-aware log/ratio encoding
Latent structure:
z0 (dim=4) "macro regime" ← Tier-0 breadth + time
z1 (dim=8) "eigenstructure" ← Tier-1 eigenvalues, conditioned on z0
z2 (dim=8) "cross-section" ← Tier-2 per-asset, conditioned on z0+z1
Total latent dim: 20.
β-TCVAE decomposition (Chen et al. 2018):
KL = MI(x;z) + TC(z) + dim_KL(z)
Loss = Recon + γ·MI + β·TC + λ·dim_KL
Key insight: TC penalises factorial structure violation, encouraging
each dimension of z to capture an INDEPENDENT factor of variation.
With β >> 1 the model is forced to find the minimal sufficient encoding.
"""
import numpy as np
from typing import Tuple, Dict, Optional, List
# ── Constants ──────────────────────────────────────────────────────────────
# Must match corpus_builder.py DIMS = [8, 20, 50, 25, 8]
TIER0_DIM = 8
TIER1_DIM = 20
TIER2_DIM = 50
TIER3_DIM = 25 # ExF macro
TIER4_DIM = 8 # EsoF
# Tier offsets into 111-dim feature vector
T_OFF = [0, 8, 28, 78, 103]
Z0_DIM = 4 # macro regime
Z1_DIM = 8 # eigenstructure
Z2_DIM = 8 # cross-section + ExF
Z3_DIM = 4 # esoteric
Z_TOTAL = Z0_DIM + Z1_DIM + Z2_DIM + Z3_DIM # 24
EPS = 1e-8
# ── Numerics ───────────────────────────────────────────────────────────────
def _relu(x): return np.maximum(0, x)
def _drelu(x): return (x > 0).astype(x.dtype)
def _tanh(x): return np.tanh(x)
def _dtanh(x): return 1.0 - np.tanh(x)**2
def _softplus(x): return np.log1p(np.exp(np.clip(x, -20, 20)))
# ── Linear layer (weights + bias) with gradient ───────────────────────────
class Linear:
"""Affine layer with Adam optimiser state."""
def __init__(self, in_dim: int, out_dim: int, seed: int = 42):
rng = np.random.RandomState(seed)
scale = np.sqrt(2.0 / in_dim)
self.W = rng.randn(in_dim, out_dim).astype(np.float32) * scale
self.b = np.zeros(out_dim, dtype=np.float32)
# Adam state
self.mW = np.zeros_like(self.W)
self.vW = np.zeros_like(self.W)
self.mb = np.zeros_like(self.b)
self.vb = np.zeros_like(self.b)
def forward(self, x: np.ndarray) -> np.ndarray:
self._x = x
return x @ self.W + self.b
def backward(self, dout: np.ndarray) -> np.ndarray:
self.dW = self._x.T @ dout
self.db = dout.sum(axis=0)
return dout @ self.W.T
def step(self, lr: float, t: int, beta1=0.9, beta2=0.999):
bc1 = 1 - beta1**t
bc2 = 1 - beta2**t
for (p, m, v, g) in [(self.W, self.mW, self.vW, self.dW),
(self.b, self.mb, self.vb, self.db)]:
m[:] = beta1 * m + (1 - beta1) * g
v[:] = beta2 * v + (1 - beta2) * g**2
p -= lr * (m / bc1) / (np.sqrt(v / bc2) + 1e-8)
# ── Simple 2-layer MLP with ReLU ──────────────────────────────────────────
class MLP:
def __init__(self, dims: List[int], seed: int = 42):
self.layers = []
for i, (d_in, d_out) in enumerate(zip(dims[:-1], dims[1:])):
self.layers.append(Linear(d_in, d_out, seed=seed + i))
# Activation cache
self._acts = []
def forward(self, x: np.ndarray, final_linear=True) -> np.ndarray:
self._acts = []
h = x
for i, layer in enumerate(self.layers):
h = layer.forward(h)
if i < len(self.layers) - 1 or not final_linear:
self._acts.append(h.copy())
h = _relu(h)
else:
self._acts.append(None)
return h
def backward(self, dout: np.ndarray) -> np.ndarray:
dh = dout
for i in range(len(self.layers) - 1, -1, -1):
if i < len(self.layers) - 1:
dh = dh * _drelu(self._acts[i])
dh = self.layers[i].backward(dh)
return dh
def step(self, lr: float, t: int):
for layer in self.layers:
layer.step(lr, t)
# ── Encoder: MLP → (mu, logvar) ───────────────────────────────────────────
class VAEEncoder:
"""Encodes input (optionally concatenated with conditioning z) to (mu, logvar)."""
def __init__(self, in_dim: int, hidden: int, z_dim: int, seed: int = 0):
self.mlp = MLP([in_dim, hidden, hidden], seed=seed)
self.mu_head = Linear(hidden, z_dim, seed=seed + 100)
self.lv_head = Linear(hidden, z_dim, seed=seed + 200)
self.z_dim = z_dim
def forward(self, x: np.ndarray) -> Tuple[np.ndarray, np.ndarray]:
h = self.mlp.forward(x) # (B, hidden)
h = _relu(h)
self._h = h
mu = self.mu_head.forward(h)
logvar = self.lv_head.forward(h)
logvar = np.clip(logvar, -10, 4) # keep variance sane
return mu, logvar
def backward(self, dmu: np.ndarray, dlv: np.ndarray) -> np.ndarray:
dh_mu = self.mu_head.backward(dmu)
dh_lv = self.lv_head.backward(dlv)
dh = (dh_mu + dh_lv) * _drelu(self._h)
return self.mlp.backward(dh)
def step(self, lr: float, t: int):
self.mlp.step(lr, t)
self.mu_head.step(lr, t)
self.lv_head.step(lr, t)
# ── Decoder: z → x_hat ────────────────────────────────────────────────────
class VAEDecoder:
def __init__(self, z_dim: int, hidden: int, out_dim: int, seed: int = 0):
self.mlp = MLP([z_dim, hidden, hidden, out_dim], seed=seed + 300)
self.out_dim = out_dim
def forward(self, z: np.ndarray) -> np.ndarray:
return self.mlp.forward(z, final_linear=True)
def backward(self, dout: np.ndarray) -> np.ndarray:
return self.mlp.backward(dout)
def step(self, lr: float, t: int):
self.mlp.step(lr, t)
# ── β-TCVAE loss (Chen et al. 2018, proper dataset-size corrected form) ───
def btcvae_kl(mu: np.ndarray, logvar: np.ndarray,
N_dataset: int, beta: float = 4.0,
gamma: float = 1.0, lam: float = 1.0
) -> Tuple[float, np.ndarray, np.ndarray]:
"""
Decompose KL into: MI + beta*TC + lambda*dim_KL
Returns (total_kl, dmu, dlogvar).
MI = E_q[log q(z|x)] - E_q[log q(z)]
TC = E_q[log q(z)] - E_q[sum_j log q(z_j)]
dKL = E_q[sum_j log q(z_j) - log p(z_j)]
Minibatch-weighted estimator — numerically stable.
"""
B, D = mu.shape
var = np.exp(logvar)
# Per-sample KL (closed form vs N(0,1) prior)
kl_per_dim = 0.5 * (mu**2 + var - logvar - 1) # (B, D)
# Log q(z|x_i) for sample z_i ~ q(z|x_i) (diagonal Gaussian)
# We use the mean as the sample point (reparametrization noise ≈ 0 for gradient estimation)
# log q(z_i | x_i) = sum_d -0.5*(log(2π) + logvar + (z-mu)^2/var)
# At z = mu: (z-mu)=0, so log q(z|x) = -0.5*sum(log(2π) + logvar)
log_q_z_given_x = -0.5 * (D * np.log(2 * np.pi) + logvar.sum(axis=1)) # (B,)
# Minibatch estimate of log q(z): E_data[sum_x log q(z|x)] / N
# log q(z_i) ≈ log(1/N * sum_j q(z_i | x_j))
# Using pairwise: log q(z_i) ≈ logsumexp over j of log q(z_i | x_j) - log(N)
# q(z_i | x_j) for each pair: diffs = z_i[none,:] - mu[none,:] shape (B,B,D)
z_sample = mu # (B, D) — use mean as point estimate
# (B_i, B_j, D) pairwise differences
diff = z_sample[:, None, :] - mu[None, :, :] # (B, B, D)
log_q_z_pair = -0.5 * (
D * np.log(2 * np.pi)
+ logvar[None, :, :].sum(axis=2)
+ (diff**2 / (var[None, :, :] + EPS)).sum(axis=2)
) # (B, B)
log_q_z = np.log(np.sum(np.exp(log_q_z_pair - log_q_z_pair.max(axis=1, keepdims=True)), axis=1) + EPS) \
+ log_q_z_pair.max(axis=1) - np.log(N_dataset + EPS) # (B,)
# log q(z_j) marginal — independence assumption for TC
# log q(z_i_j) for each dim j: logsumexp over data points
log_q_z_product = np.zeros(B, dtype=np.float64)
for j in range(D):
diff_j = z_sample[:, j:j+1] - mu[:, j:j+1].T # (B, B)
log_q_zj = -0.5 * (np.log(2 * np.pi) + logvar[None, :, j] + diff_j**2 / (var[None, :, j] + EPS)) # (B,B)
log_q_zj_marginal = np.log(np.sum(np.exp(log_q_zj - log_q_zj.max(axis=1, keepdims=True)), axis=1) + EPS) \
+ log_q_zj.max(axis=1) - np.log(N_dataset + EPS) # (B,)
log_q_z_product += log_q_zj_marginal
# log p(z) = N(0,1)
log_p_z = -0.5 * (D * np.log(2 * np.pi) + (z_sample**2).sum(axis=1)) # (B,)
mi_loss = np.mean(log_q_z_given_x - log_q_z)
tc_loss = np.mean(log_q_z - log_q_z_product)
dkl_loss = np.mean(log_q_z_product - log_p_z)
total_kl = gamma * mi_loss + beta * max(tc_loss, 0) + lam * dkl_loss
# Gradients w.r.t. mu and logvar: use standard closed-form KL gradient
# ∂KL/∂mu = mu, ∂KL/∂logvar = 0.5*(exp(logvar) - 1)
scale = (gamma + beta + lam) / (3.0 * B + EPS)
dmu = scale * mu
dlogvar = scale * 0.5 * (var - 1.0)
return float(total_kl), dmu.astype(np.float32), dlogvar.astype(np.float32)
# ── Reparametrization ─────────────────────────────────────────────────────
def reparametrize(mu, logvar, rng):
std = np.exp(0.5 * logvar)
eps = rng.randn(*mu.shape).astype(np.float32)
return mu + eps * std
# ── Hierarchical D-VAE ────────────────────────────────────────────────────
class HierarchicalDVAE:
"""
4-level Hierarchical Disentangled VAE (111-dim input, 24-dim latent).
Latent hierarchy:
z0 (4) = enc0(T0) macro regime (breadth + time)
z1 (8) = enc1(T1 + T4 + z0) eigenstructure + esoteric, cond on z0
z2 (8) = enc2(T2 + T3 + z0 + z1) cross-section + ExF, cond on z0,z1
z3 (4) = enc3(T3 + T4 + z0) ExF+EsoF regime, cond on z0
Decoding:
T0_hat = dec0(z0)
T1_hat = dec1(z0, z1)
T2_hat = dec2(z0, z1, z2)
T3_hat = dec3(z0, z3)
T4_hat = dec4(z0) (EsoF is deterministic, serves as auxiliary)
Training phases:
0: enc0/dec0 only — full 500K+ corpus (even pre-eigen NG1/NG2)
1: + enc1/dec1 — eigen-tier (NG3+)
2: + enc2/dec2 — pricing + ExF (NG3+ with pricing)
3: + enc3/dec3 — ExF regime
4: joint fine-tune all
"""
def __init__(self, hidden: int = 64, beta: float = 4.0,
gamma: float = 1.0, lam: float = 1.0, seed: int = 42):
self.hidden = hidden
self.beta = beta
self.gamma = gamma
self.lam = lam
self.seed = seed
# Encoders (each conditioned on upstream z)
self.enc0 = VAEEncoder(TIER0_DIM, hidden, Z0_DIM, seed=seed)
self.enc1 = VAEEncoder(TIER1_DIM + TIER4_DIM + Z0_DIM, hidden, Z1_DIM, seed=seed+10)
self.enc2 = VAEEncoder(TIER2_DIM + TIER3_DIM + Z0_DIM + Z1_DIM, hidden, Z2_DIM, seed=seed+20)
self.enc3 = VAEEncoder(TIER3_DIM + TIER4_DIM + Z0_DIM, hidden//2, Z3_DIM, seed=seed+30)
# Decoders
self.dec0 = VAEDecoder(Z0_DIM, hidden, TIER0_DIM, seed=seed)
self.dec1 = VAEDecoder(Z0_DIM + Z1_DIM, hidden, TIER1_DIM, seed=seed+10)
self.dec2 = VAEDecoder(Z0_DIM + Z1_DIM + Z2_DIM, hidden, TIER2_DIM, seed=seed+20)
self.dec3 = VAEDecoder(Z0_DIM + Z3_DIM, hidden//2, TIER3_DIM, seed=seed+30)
self.dec4 = VAEDecoder(Z0_DIM, hidden//2, TIER4_DIM, seed=seed+40)
# Normalisation statistics (fit on training data)
self._mu_t0 = np.zeros(TIER0_DIM, dtype=np.float32)
self._sd_t0 = np.ones(TIER0_DIM, dtype=np.float32)
self._mu_t1 = np.zeros(TIER1_DIM, dtype=np.float32)
self._sd_t1 = np.ones(TIER1_DIM, dtype=np.float32)
self._mu_t2 = np.zeros(TIER2_DIM, dtype=np.float32)
self._sd_t2 = np.ones(TIER2_DIM, dtype=np.float32)
self._mu_t3 = np.zeros(TIER3_DIM, dtype=np.float32)
self._sd_t3 = np.ones(TIER3_DIM, dtype=np.float32)
self._mu_t4 = np.zeros(TIER4_DIM, dtype=np.float32)
self._sd_t4 = np.ones(TIER4_DIM, dtype=np.float32)
self.step_t = 0 # Adam time step
self.train_losses: List[Dict] = []
# ── Normalisation ──────────────────────────────────────────────────────
def fit_normaliser(self, X: np.ndarray, mask: np.ndarray):
"""Fit per-tier normalisation on a representative sample."""
def _fit(rows, dim):
if len(rows) < 2:
return np.zeros(dim, dtype=np.float32), np.ones(dim, dtype=np.float32)
return rows.mean(0).astype(np.float32), (rows.std(0) + EPS).astype(np.float32)
self._mu_t0, self._sd_t0 = _fit(X[:, T_OFF[0]:T_OFF[0]+TIER0_DIM], TIER0_DIM)
idx1 = mask[:, 1]
self._mu_t1, self._sd_t1 = _fit(X[idx1, T_OFF[1]:T_OFF[1]+TIER1_DIM], TIER1_DIM)
idx2 = mask[:, 2]
self._mu_t2, self._sd_t2 = _fit(X[idx2, T_OFF[2]:T_OFF[2]+TIER2_DIM], TIER2_DIM)
idx3 = mask[:, 3]
self._mu_t3, self._sd_t3 = _fit(X[idx3, T_OFF[3]:T_OFF[3]+TIER3_DIM], TIER3_DIM)
self._mu_t4, self._sd_t4 = _fit(X[:, T_OFF[4]:T_OFF[4]+TIER4_DIM], TIER4_DIM)
def _norm(self, x, mu, sd): return (x - mu) / sd
def _norm0(self, x): return self._norm(x, self._mu_t0, self._sd_t0)
def _norm1(self, x): return self._norm(x, self._mu_t1, self._sd_t1)
def _norm2(self, x): return self._norm(x, self._mu_t2, self._sd_t2)
def _norm3(self, x): return self._norm(x, self._mu_t3, self._sd_t3)
def _norm4(self, x): return self._norm(x, self._mu_t4, self._sd_t4)
# ── Forward pass ──────────────────────────────────────────────────────
def _split(self, X):
"""Split 111-dim row into normalised tier vectors."""
t0 = self._norm0(X[:, T_OFF[0]:T_OFF[0]+TIER0_DIM])
t1 = self._norm1(X[:, T_OFF[1]:T_OFF[1]+TIER1_DIM])
t2 = self._norm2(X[:, T_OFF[2]:T_OFF[2]+TIER2_DIM])
t3 = self._norm3(X[:, T_OFF[3]:T_OFF[3]+TIER3_DIM])
t4 = self._norm4(X[:, T_OFF[4]:T_OFF[4]+TIER4_DIM])
return t0, t1, t2, t3, t4
def encode(self, X: np.ndarray, mask: np.ndarray, rng) -> Dict:
t0, t1, t2, t3, t4 = self._split(X)
mu0, lv0 = self.enc0.forward(t0)
z0 = reparametrize(mu0, lv0, rng)
mu1, lv1 = self.enc1.forward(np.concatenate([t1, t4, z0], axis=1))
z1 = reparametrize(mu1, lv1, rng)
mu2, lv2 = self.enc2.forward(np.concatenate([t2, t3, z0, z1], axis=1))
z2 = reparametrize(mu2, lv2, rng)
mu3, lv3 = self.enc3.forward(np.concatenate([t3, t4, z0], axis=1))
z3 = reparametrize(mu3, lv3, rng)
return dict(t0=t0, t1=t1, t2=t2, t3=t3, t4=t4,
mu0=mu0, lv0=lv0, z0=z0,
mu1=mu1, lv1=lv1, z1=z1,
mu2=mu2, lv2=lv2, z2=z2,
mu3=mu3, lv3=lv3, z3=z3)
def decode(self, enc: Dict) -> Dict:
z0, z1, z2, z3 = enc['z0'], enc['z1'], enc['z2'], enc['z3']
x0_hat = self.dec0.forward(z0)
x1_hat = self.dec1.forward(np.concatenate([z0, z1], axis=1))
x2_hat = self.dec2.forward(np.concatenate([z0, z1, z2], axis=1))
x3_hat = self.dec3.forward(np.concatenate([z0, z3], axis=1))
x4_hat = self.dec4.forward(z0) # EsoF decoded from macro only
return dict(x0_hat=x0_hat, x1_hat=x1_hat, x2_hat=x2_hat,
x3_hat=x3_hat, x4_hat=x4_hat)
# ── Loss and gradients ─────────────────────────────────────────────────
def loss_and_grads(self, enc: Dict, dec: Dict, mask: np.ndarray,
N_dataset: int, phase: int) -> Dict:
"""Compute total loss and all gradients via chain rule."""
t0, t1, t2 = enc['t0'], enc['t1'], enc['t2']
x0_hat = dec['x0_hat']
x1_hat = dec['x1_hat']
x2_hat = dec['x2_hat']
B = len(t0)
# Mask weights: per-sample, per-tier
w1 = mask[:, 1].astype(np.float32)[:, None] # (B,1)
w2 = mask[:, 2].astype(np.float32)[:, None]
# Reconstruction losses
recon0 = np.mean((x0_hat - t0)**2)
recon1 = np.mean(w1 * (x1_hat - t1)**2)
recon2 = np.mean(w2 * (x2_hat - t2)**2)
# Reconstruction gradients
dr0 = 2 * (x0_hat - t0) / (B * TIER0_DIM)
dr1 = 2 * w1 * (x1_hat - t1) / (B * TIER1_DIM)
dr2 = 2 * w2 * (x2_hat - t2) / (B * TIER2_DIM)
# KL losses
kl0, dmu0_kl, dlv0_kl = btcvae_kl(enc['mu0'], enc['lv0'], N_dataset, self.beta, self.gamma, self.lam)
kl1, dmu1_kl, dlv1_kl = btcvae_kl(enc['mu1'], enc['lv1'], N_dataset, self.beta, self.gamma, self.lam)
kl2, dmu2_kl, dlv2_kl = btcvae_kl(enc['mu2'], enc['lv2'], N_dataset, self.beta, self.gamma, self.lam)
# Phase-based scaling: only activate eigen/pricing in correct phase
if phase < 1:
kl1 = recon1 = 0.0; dmu1_kl[:] = 0; dlv1_kl[:] = 0; dr1[:] = 0
if phase < 2:
kl2 = recon2 = 0.0; dmu2_kl[:] = 0; dlv2_kl[:] = 0; dr2[:] = 0
total = recon0 + recon1 + recon2 + kl0 + kl1 + kl2
return dict(
total=total, recon0=recon0, recon1=recon1, recon2=recon2,
kl0=kl0, kl1=kl1, kl2=kl2,
dr0=dr0, dr1=dr1, dr2=dr2,
dmu0_kl=dmu0_kl, dlv0_kl=dlv0_kl,
dmu1_kl=dmu1_kl, dlv1_kl=dlv1_kl,
dmu2_kl=dmu2_kl, dlv2_kl=dlv2_kl,
)
def backward(self, enc: Dict, dec: Dict, grads: Dict, phase: int, lr: float):
"""Backpropagate through all encoders and decoders, then Adam step."""
z0, z1, z2 = enc['z0'], enc['z1'], enc['z2']
t = self.step_t
# ── Decoder 2 (z0+z1+z2 → x2) ──────────────────────────────────
if phase >= 2:
dz_all_2 = self.dec2.backward(grads['dr2']) # (B, Z0+Z1+Z2)
dz0_from_d2 = dz_all_2[:, :Z0_DIM]
dz1_from_d2 = dz_all_2[:, Z0_DIM:Z0_DIM + Z1_DIM]
dz2_from_d2 = dz_all_2[:, Z0_DIM + Z1_DIM:]
else:
dz0_from_d2 = np.zeros_like(z0)
dz1_from_d2 = np.zeros_like(z1)
dz2_from_d2 = np.zeros_like(z2)
self.dec2.backward(np.zeros_like(grads['dr2']))
# ── Encoder 2 ─────────────────────────────────────────────────
# enc2 input = concat(T2, T3, z0, z1) dims = TIER2+TIER3+Z0+Z1
if phase >= 2:
dmu2 = dz2_from_d2 + grads['dmu2_kl']
dlv2 = dz2_from_d2 * 0.5 + grads['dlv2_kl']
dinp2 = self.enc2.backward(dmu2, dlv2) # (B, T2+T3+Z0+Z1)
_off = TIER2_DIM + TIER3_DIM
dz0_from_e2 = dinp2[:, _off:_off + Z0_DIM]
dz1_from_e2 = dinp2[:, _off + Z0_DIM:]
self.dec2.step(lr, t)
self.enc2.step(lr, t)
else:
dz0_from_e2 = np.zeros_like(z0)
dz1_from_e2 = np.zeros_like(z1)
# ── Decoder 1 (z0+z1 → x1) ──────────────────────────────────────
if phase >= 1:
dz_all_1 = self.dec1.backward(grads['dr1']) # (B, Z0+Z1)
dz0_from_d1 = dz_all_1[:, :Z0_DIM]
dz1_from_d1 = dz_all_1[:, Z0_DIM:]
else:
dz0_from_d1 = np.zeros_like(z0)
dz1_from_d1 = np.zeros_like(z1)
self.dec1.backward(np.zeros_like(grads['dr1']))
# ── Encoder 1 ─────────────────────────────────────────────────
# enc1 input = concat(T1, T4, z0) dims = TIER1+TIER4+Z0
if phase >= 1:
dmu1 = dz1_from_d1 + dz1_from_d2 + dz1_from_e2 + grads['dmu1_kl']
dlv1 = dz1_from_d1 * 0.5 + grads['dlv1_kl']
dinp1 = self.enc1.backward(dmu1, dlv1) # (B, T1+T4+Z0)
dz0_from_e1 = dinp1[:, TIER1_DIM + TIER4_DIM:] # correct: skip T1+T4
self.dec1.step(lr, t)
self.enc1.step(lr, t)
else:
dz0_from_e1 = np.zeros_like(z0)
# ── Decoder 0 (z0 → x0) ─────────────────────────────────────────
dz0_from_d0 = self.dec0.backward(grads['dr0']) # (B, Z0)
# ── Encoder 0 ────────────────────────────────────────────────────
dmu0 = dz0_from_d0 + dz0_from_d1 + dz0_from_d2 + dz0_from_e1 + dz0_from_e2 + grads['dmu0_kl']
dlv0 = dz0_from_d0 * 0.5 + grads['dlv0_kl']
self.enc0.backward(dmu0, dlv0)
self.enc0.step(lr, t)
self.dec0.step(lr, t)
# ── Training ──────────────────────────────────────────────────────────
def train_epoch(self, X: np.ndarray, mask: np.ndarray,
lr: float, batch_size: int, phase: int, rng) -> Dict:
N = len(X)
idx = rng.permutation(N)
epoch_stats = dict(total=0, recon0=0, recon1=0, recon2=0, kl0=0, kl1=0, kl2=0, n=0)
for start in range(0, N, batch_size):
bi = idx[start:start + batch_size]
Xb = X[bi]
mb = mask[bi]
self.step_t += 1 # Adam global step (cumulative across all epochs)
enc_out = self.encode(Xb, mb, rng)
dec_out = self.decode(enc_out)
grads = self.loss_and_grads(enc_out, dec_out, mb, N, phase)
self.backward(enc_out, dec_out, grads, phase, lr)
for k in ['total', 'recon0', 'recon1', 'recon2', 'kl0', 'kl1', 'kl2']:
epoch_stats[k] += float(grads[k]) * len(bi)
epoch_stats['n'] += len(bi)
return {k: v / max(epoch_stats['n'], 1) for k, v in epoch_stats.items() if k != 'n'}
# ── Inference ─────────────────────────────────────────────────────────
def get_latents(self, X: np.ndarray, mask: np.ndarray) -> Dict[str, np.ndarray]:
"""Return disentangled latent codes for analysis."""
rng = np.random.RandomState(0) # deterministic for inference
enc = self.encode(X, mask, rng)
return {
'z0_macro': enc['mu0'], # (N, 4) deterministic mean
'z1_eigen': enc['mu1'], # (N, 8)
'z2_xsection': enc['mu2'], # (N, 8)
'z_all': np.concatenate([enc['mu0'], enc['mu1'], enc['mu2']], axis=1), # (N, 20)
'sigma0': np.exp(0.5 * enc['lv0']), # uncertainty
'sigma1': np.exp(0.5 * enc['lv1']),
'sigma2': np.exp(0.5 * enc['lv2']),
}
# ── Persistence ───────────────────────────────────────────────────────
def save(self, path: str):
data = {
'norm': dict(mu_t0=self._mu_t0, sd_t0=self._sd_t0,
mu_t1=self._mu_t1, sd_t1=self._sd_t1,
mu_t2=self._mu_t2, sd_t2=self._sd_t2),
'step_t': self.step_t,
'losses': self.train_losses,
}
# Save all weight matrices
for name, enc in [('enc0', self.enc0), ('enc1', self.enc1), ('enc2', self.enc2)]:
for i, layer in enumerate(enc.mlp.layers):
data[f'{name}_mlp{i}_W'] = layer.W
data[f'{name}_mlp{i}_b'] = layer.b
data[f'{name}_mu_W'] = enc.mu_head.W; data[f'{name}_mu_b'] = enc.mu_head.b
data[f'{name}_lv_W'] = enc.lv_head.W; data[f'{name}_lv_b'] = enc.lv_head.b
for name, dec in [('dec0', self.dec0), ('dec1', self.dec1), ('dec2', self.dec2)]:
for i, layer in enumerate(dec.mlp.layers):
data[f'{name}_mlp{i}_W'] = layer.W
data[f'{name}_mlp{i}_b'] = layer.b
np.savez_compressed(path, **data)
print(f"Model saved: {path}.npz")

193
nautilus_dolphin/dvae/proto_v2_query.py Executable file
View File

@@ -0,0 +1,193 @@
"""
proto_v2_query.py — Back-of-envelope z-space probe for convnext_model_v2.json
Queries the current best checkpoint against the 56-day backtest period
(Dec 31 2025 – Feb 25 2026) to assess signal quality vs the ep=17 baseline.
Reports:
1. z_active, z_post_std (latent health)
2. proxy_B dim + r (encoding quality)
3. Calibration: is z_proxy_B still always negative for this period?
4. Split test: top-25% vs bottom-25% proxy_B days — does z separate them?
ExF columns (dvol_btc, fng, funding_btc) are zero-filled — same as exp13 fallback.
Safe to run while training is still in progress (read-only, no GPU).
"""
import os, sys, json, glob
import io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')
import numpy as np
import pandas as pd
ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
DVAE_DIR = os.path.join(ROOT, 'nautilus_dolphin', 'dvae')
sys.path.insert(0, DVAE_DIR)
MODEL_V2 = os.path.join(DVAE_DIR, 'convnext_model_v2.json')
MODEL_EP17 = os.path.join(DVAE_DIR, 'convnext_model.json')
KLINES_DIR = os.path.join(ROOT, 'vbt_cache_klines')
FEATURE_COLS = [
'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity',
'vel_div', 'instability_50', 'instability_150',
]
EXF_COLS = ['dvol_btc', 'fng', 'funding_btc'] # zero-filled
T_WIN = 32
# 56-day backtest window
DATE_START = '2025-12-31'
DATE_END = '2026-02-25'
# ── load model ───────────────────────────────────────────────────────────────
from convnext_dvae import ConvNeXtVAE
def load_model(path):
with open(path) as f:
meta = json.load(f)
arch = meta.get('architecture', {})
m = ConvNeXtVAE(
C_in=arch['C_in'], T_in=arch['T_in'],
z_dim=arch['z_dim'], base_ch=arch['base_ch'],
n_blocks=arch.get('n_blocks', 3), seed=42,
)
m.load(path)
nm = np.array(meta['norm_mean']) if 'norm_mean' in meta else None
ns = np.array(meta['norm_std']) if 'norm_std' in meta else None
return m, nm, ns, meta
print(f"Loading v2 checkpoint...")
model_v2, nm_v2, ns_v2, meta_v2 = load_model(MODEL_V2)
print(f" ep={meta_v2.get('epoch')} val_loss={meta_v2.get('val_loss',0):.5f}")
ep17_exists = os.path.exists(MODEL_EP17)
if ep17_exists:
print(f"Loading ep=17 baseline for comparison...")
model_17, nm_17, ns_17, meta_17 = load_model(MODEL_EP17)
print(f" ep={meta_17.get('epoch')} val_loss={meta_17.get('val_loss',0):.5f}")
# ── build probe set from 56-day window ───────────────────────────────────────
print(f"\nBuilding probe windows from {DATE_START} to {DATE_END}...")
files = sorted(f for f in glob.glob(os.path.join(KLINES_DIR, '*.parquet')))
# filter to date range
period_files = [f for f in files
if DATE_START <= os.path.basename(f)[:10] <= DATE_END]
print(f" {len(period_files)} klines files in period")
rng = np.random.default_rng(42)
probes_raw, proxy_B_vals, file_dates = [], [], []
step = max(1, len(period_files) // 60) # ~60 probes across period
for f in period_files[::step]:
try:
df = pd.read_parquet(f, columns=FEATURE_COLS).dropna()
if len(df) < T_WIN + 10: continue
# sample from middle of each day (avoid open/close noise)
mid = len(df) // 2
pos = int(rng.integers(max(0, mid - 30), min(len(df) - T_WIN, mid + 30)))
arr = df[FEATURE_COLS].values[pos:pos+T_WIN].astype(np.float64) # (T, 7)
proxy_B = (arr[:, 5] - arr[:, 3]).reshape(-1, 1) # instability_50 - v750
exf = np.zeros((T_WIN, 3), dtype=np.float64) # zero-fill ExF
arr11 = np.concatenate([arr, proxy_B, exf], axis=1).T # (11, T)
if not np.isfinite(arr11).all(): continue
probes_raw.append(arr11)
proxy_B_vals.append(float(proxy_B.mean()))
file_dates.append(os.path.basename(f)[:10])
except Exception as e:
pass
probes_raw = np.stack(probes_raw) # (N, 11, T)
proxy_B_arr = np.array(proxy_B_vals)
print(f" Probe set: {probes_raw.shape} ({len(probes_raw)} windows)")
def normalise(probes, nm, ns):
if nm is None: return probes
p = (probes - nm[None, :, None]) / ns[None, :, None]
np.clip(p, -6., 6., out=p)
return p
def run_query(model, nm, ns, label):
probes = normalise(probes_raw, nm, ns)
z_mu, z_logvar = model.encode(probes)
x_recon = model.decode(z_mu)
# 1. Latent health
z_std_per_dim = z_mu.std(0)
z_active = int((z_std_per_dim > 0.01).sum())
z_post_std = float(np.exp(0.5 * z_logvar).mean())
z_mean_all = float(z_mu.mean())
# 2. Reconstruction
recon_err = ((probes - x_recon) ** 2).mean(axis=(-1, -2))
recon_p50 = float(np.median(recon_err))
# 3. proxy_B correlation — find best dim
corrs = []
for d in range(z_mu.shape[1]):
if z_std_per_dim[d] > 0.01:
r = float(np.corrcoef(z_mu[:, d], proxy_B_arr)[0, 1])
if np.isfinite(r): corrs.append((abs(r), r, d))
corrs.sort(reverse=True)
best_abs_r, best_r, best_dim = corrs[0] if corrs else (0, 0, -1)
# 4. Calibration: is best_dim always negative for this period?
z_best = z_mu[:, best_dim]
z_min, z_max = float(z_best.min()), float(z_best.max())
always_neg = z_max < 0
always_pos = z_min > 0
calib = "ALWAYS NEGATIVE" if always_neg else ("ALWAYS POSITIVE" if always_pos else
f"MIXED [{z_min:+.3f}, {z_max:+.3f}]")
# 5. Split test: top-25% vs bottom-25% proxy_B days
q75 = np.percentile(proxy_B_arr, 75)
q25 = np.percentile(proxy_B_arr, 25)
hi_mask = proxy_B_arr >= q75
lo_mask = proxy_B_arr <= q25
z_hi = float(z_best[hi_mask].mean()) if hi_mask.sum() > 2 else float('nan')
z_lo = float(z_best[lo_mask].mean()) if lo_mask.sum() > 2 else float('nan')
sep = abs(z_hi - z_lo)
print(f"\n{'='*60}")
print(f" {label}")
print(f"{'='*60}")
print(f" z_active : {z_active} / {z_mu.shape[1]}")
print(f" z_post_std : {z_post_std:.4f} (healthy: 0.6–1.2)")
print(f" recon_p50 : {recon_p50:.4f} (ep17 baseline: 0.2999)")
print(f"\n proxy_B dim : z[{best_dim}] r={best_r:+.4f} (ep17 had z[10] r=+0.973)")
print(f" Top-5 z×proxy_B corrs:")
for _, r, d in corrs[:5]:
bar = '#' * int(abs(r) * 30)
print(f" z[{d:2d}] r={r:+.4f} {bar}")
print(f"\n Calibration : {calib}")
print(f" z[{best_dim}] range : [{z_min:+.4f}, {z_max:+.4f}]")
print(f" z[{best_dim}] mean : {z_best.mean():+.4f}")
print(f"\n Split test (proxy_B quartiles):")
print(f" top-25% proxy_B → z[{best_dim}] mean = {z_hi:+.4f}")
print(f" bot-25% proxy_B → z[{best_dim}] mean = {z_lo:+.4f}")
print(f" separation = {sep:.4f} (>0.3 useful, >0.6 good)")
return z_mu, corrs
print("\n" + "="*60)
print(f"proxy_B stats over {len(probes_raw)} probes:")
print(f" mean={proxy_B_arr.mean():+.4f} std={proxy_B_arr.std():.4f} "
f"min={proxy_B_arr.min():+.4f} max={proxy_B_arr.max():+.4f}")
z_v2, corrs_v2 = run_query(model_v2, nm_v2, ns_v2,
f"v2 ep={meta_v2.get('epoch')} val={meta_v2.get('val_loss',0):.5f} [CURRENT BEST]")
if ep17_exists:
z_17, corrs_17 = run_query(model_17, nm_17, ns_17,
f"ep17 val={meta_17.get('val_loss',0):.5f} [PRODUCTION BASELINE]")
# Side-by-side summary
print(f"\n{'='*60}")
print(" COMPARISON SUMMARY")
print(f"{'='*60}")
r_v2 = corrs_v2[0][1] if corrs_v2 else 0
r_17 = corrs_17[0][1] if corrs_17 else 0
print(f" proxy_B r : v2={r_v2:+.4f} vs ep17={r_17:+.4f} "
f"({'BETTER' if abs(r_v2) > abs(r_17) else 'WORSE' if abs(r_v2) < abs(r_17) else 'SAME'})")
print(f"\nDone.")

244
nautilus_dolphin/dvae/resolution_alignment_check.py Executable file
View File

@@ -0,0 +1,244 @@
"""
Resolution Alignment Check
===========================
GATE TEST: Before training Titan VAE on 1m klines, verify that T0-T4 feature
distributions at 1m resolution are sufficiently aligned with 5s production data.
If they diverge, a 1m-trained model cannot be used at 5s inference — the latent
dims would represent different physics (50-min vs 4.2-min eigenvalue dynamics).
Tests:
1. Per-dim KS test (distribution shape match)
2. Per-dim mean/std ratio (scale match)
3. PCA cosine similarity (covariance structure match)
4. T1 feature correlation matrix alignment (Frobenius distance)
Pass criteria (all must hold):
- KS p > 0.05 for >= 70% of T1+T2 dims (distributions compatible)
- mean ratio within [0.1, 10x] and std ratio within [0.1, 10x] for >= 70% of dims
- PCA cosine(PC1, PC2) >= 0.80 (shared dominant variance axes)
Output: resolution_alignment_report.json
"""
import sys
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import json
import numpy as np
import pandas as pd
from scipy import stats
ROOT = Path(__file__).parent.parent
VBT5s = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
VBT1m = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines")
OUT = Path(__file__).parent / "resolution_alignment_report.json"
sys.path.insert(0, str(ROOT))
from dvae.titan_sensor import build_feature_vector
META_COLS = {'timestamp','scan_number','v50_lambda_max_velocity','v150_lambda_max_velocity',
'v300_lambda_max_velocity','v750_lambda_max_velocity','vel_div',
'instability_50','instability_150'}
N_5S_DAYS = 10 # use first N days from vbt_cache (5s data)
N_1M_DAYS = 60 # use N days from klines (1m data); pick older period if available
def extract_features(parquet_dir, n_days, label, date_before=None):
"""Build feature matrix from parquet files."""
files = sorted(parquet_dir.glob("*.parquet"))
files = [f for f in files if 'catalog' not in str(f)]
if date_before:
files = [f for f in files if f.stem < date_before]
files = files[:n_days]
if not files:
print(f" [{label}] No files found in {parquet_dir}")
return None, []
print(f" [{label}] {len(files)} days ({files[0].stem} to {files[-1].stem})")
rows = []
for pf in files:
df = pd.read_parquet(pf)
assets = [c for c in df.columns if c not in META_COLS]
for ri in range(50, len(df), 10): # stride=10 to keep runtime low
feat = build_feature_vector(df, ri, assets)
rows.append(feat)
X = np.array(rows, dtype=np.float64)
X = np.nan_to_num(X, nan=0.0, posinf=0.0, neginf=0.0)
print(f" [{label}] Feature matrix: {X.shape}")
return X, [f.stem for f in files]
def ks_test_per_dim(X5, X1, dims):
"""KS test for each dim. Returns fraction with p>0.05."""
results = {}
for d in dims:
a, b = X5[:, d], X1[:, d]
# skip if all-zero in either
if np.std(a) < 1e-10 or np.std(b) < 1e-10:
results[d] = {"p": 1.0, "stat": 0.0, "trivial": True}
continue
stat, p = stats.ks_2samp(a, b)
results[d] = {"p": float(p), "stat": float(stat), "trivial": False}
return results
def scale_check(X5, X1, dims):
"""Check mean/std ratios."""
results = {}
for d in dims:
m5, s5 = float(np.mean(X5[:, d])), float(np.std(X5[:, d]))
m1, s1 = float(np.mean(X1[:, d])), float(np.std(X1[:, d]))
mean_ratio = abs(m5) / (abs(m1) + 1e-10) if abs(m1) > 1e-10 else float('inf')
std_ratio = s5 / (s1 + 1e-10) if s1 > 1e-10 else float('inf')
results[d] = {
"mean_5s": m5, "mean_1m": m1, "mean_ratio": mean_ratio,
"std_5s": s5, "std_1m": s1, "std_ratio": std_ratio,
}
return results
def pca_cosine(X5, X1, n_components=4):
"""PCA on both matrices, return cosine sim of top components."""
from numpy.linalg import svd
def top_pcs(X, k):
Xc = X - X.mean(axis=0)
_, _, Vt = svd(Xc, full_matrices=False)
return Vt[:k] # (k, d)
pcs5 = top_pcs(X5, n_components)
pcs1 = top_pcs(X1, n_components)
cosines = []
for i in range(n_components):
c = abs(float(np.dot(pcs5[i], pcs1[i]) /
(np.linalg.norm(pcs5[i]) * np.linalg.norm(pcs1[i]) + 1e-12)))
cosines.append(c)
return cosines
def corr_frobenius(X5, X1, dims):
"""Frobenius distance between T1 correlation matrices."""
A = np.corrcoef(X5[:, dims].T)
B = np.corrcoef(X1[:, dims].T)
A = np.nan_to_num(A); B = np.nan_to_num(B)
frob = float(np.linalg.norm(A - B, 'fro'))
max_frob = float(np.sqrt(2 * len(dims))) # upper bound (orthogonal corr matrices)
return frob, frob / max_frob # raw and normalised [0,1]
def main():
print("=== Resolution Alignment Check ===")
print(f"5s source: {VBT5s}")
print(f"1m source: {VBT1m}")
# Pick 1m days from OLDER period if possible (true OOS); otherwise use what's there
klines_files = sorted(VBT1m.glob("*.parquet"))
klines_files = [f for f in klines_files if 'catalog' not in str(f)]
# try to use oldest N_1M_DAYS
date_cut = klines_files[0].stem if klines_files else None
print(f" 1m klines available: {len(klines_files)} days ({klines_files[0].stem if klines_files else '?'} to {klines_files[-1].stem if klines_files else '?'})")
print("\nBuilding feature matrices...")
X5, days5 = extract_features(VBT5s, N_5S_DAYS, "5s-prod")
X1, days1 = extract_features(VBT1m, N_1M_DAYS, "1m-klines")
if X5 is None or X1 is None:
print("ERROR: could not load data from one or both sources.")
return
# T1 dims: 8-27 (eigenvalue velocity features)
# T2 dims: 28-77 (per-asset z-scores)
T1_DIMS = list(range(8, 28))
T2_DIMS = list(range(28, 78))
report = {
"n_5s_samples": len(X5),
"n_1m_samples": len(X1),
"5s_days": days5,
"1m_days": days1,
}
print("\n--- T1 eigenvalue velocity features (dims 8-27) ---")
ks_t1 = ks_test_per_dim(X5, X1, T1_DIMS)
sc_t1 = scale_check(X5, X1, T1_DIMS)
frob_t1, frob_t1_norm = corr_frobenius(X5, X1, T1_DIMS)
pval_pass_t1 = sum(1 for v in ks_t1.values() if v['p'] > 0.05 or v.get('trivial'))
scale_pass_t1 = sum(1 for v in sc_t1.values()
if 0.1 <= v['std_ratio'] <= 10 and v['std_ratio'] != float('inf'))
print(f" KS p>0.05: {pval_pass_t1}/{len(T1_DIMS)} dims")
print(f" std_ratio in [0.1,10]: {scale_pass_t1}/{len(T1_DIMS)} dims")
print(f" Correlation matrix Frobenius dist: {frob_t1:.3f} (normalised: {frob_t1_norm:.3f})")
print(" Per-dim std ratios (5s/1m):")
for d in T1_DIMS:
r = sc_t1[d]['std_ratio']
flag = "OK" if 0.1 <= r <= 10 else "MISMATCH"
print(f" dim{d:3d}: std_5s={sc_t1[d]['std_5s']:.4f} std_1m={sc_t1[d]['std_1m']:.4f} ratio={r:.2f} {flag}")
print("\n--- T2 per-asset z-score features (dims 28-77) ---")
ks_t2 = ks_test_per_dim(X5, X1, T2_DIMS)
sc_t2 = scale_check(X5, X1, T2_DIMS)
pval_pass_t2 = sum(1 for v in ks_t2.values() if v['p'] > 0.05 or v.get('trivial'))
scale_pass_t2 = sum(1 for v in sc_t2.values()
if 0.1 <= v['std_ratio'] <= 10 and v['std_ratio'] != float('inf'))
print(f" KS p>0.05: {pval_pass_t2}/{len(T2_DIMS)} dims")
print(f" std_ratio in [0.1,10]: {scale_pass_t2}/{len(T2_DIMS)} dims")
print("\n--- PCA cosine similarity (T1+T2 joint, top-4 components) ---")
joint_dims = T1_DIMS + T2_DIMS
pca_cos = pca_cosine(X5[:, joint_dims], X1[:, joint_dims], n_components=4)
for i, c in enumerate(pca_cos):
flag = "OK" if c >= 0.80 else "DIVERGED"
print(f" PC{i+1}: cosine={c:.3f} {flag}")
# Pass/fail verdict
ks_ok = (pval_pass_t1 / len(T1_DIMS)) >= 0.70
scale_ok = (scale_pass_t1 / len(T1_DIMS)) >= 0.70
pca_ok = pca_cos[0] >= 0.80 and pca_cos[1] >= 0.80
frob_ok = frob_t1_norm < 0.40
verdict = {
"ks_pass": bool(ks_ok),
"scale_pass": bool(scale_ok),
"pca_pass": bool(pca_ok),
"frob_pass": bool(frob_ok),
"PROCEED": bool(ks_ok and scale_ok and pca_ok and frob_ok),
}
print("\n=== VERDICT ===")
print(f" KS distribution match: {'PASS' if ks_ok else 'FAIL'} ({pval_pass_t1}/{len(T1_DIMS)} dims pass)")
print(f" Scale match: {'PASS' if scale_ok else 'FAIL'} ({scale_pass_t1}/{len(T1_DIMS)} dims pass)")
print(f" PCA structure match: {'PASS' if pca_ok else 'FAIL'} (PC1={pca_cos[0]:.3f} PC2={pca_cos[1]:.3f})")
print(f" Corr matrix alignment: {'PASS' if frob_ok else 'FAIL'} (normalised Frobenius={frob_t1_norm:.3f} < 0.40)")
print(f"\n PROCEED with 1m training: {'YES' if verdict['PROCEED'] else 'NO'}")
if not verdict['PROCEED']:
print(" --> Feature distributions diverge across resolutions.")
print(" --> A 1m-trained model would learn different physics than 5s production.")
print(" --> Training on 1m would NOT yield useful representations at 5s inference.")
print(" --> Alternative: use time-normalised features (fixed-time windows, not bar-count windows)")
else:
print(" --> Feature distributions are compatible across resolutions.")
print(" --> 1m training on 2021-2024 data is a valid OOS approach.")
report.update({
"t1_ks": {str(k): v for k,v in ks_t1.items()},
"t1_scale": {str(k): v for k,v in sc_t1.items()},
"t2_ks": {str(k): v for k,v in ks_t2.items()},
"t2_scale": {str(k): v for k,v in sc_t2.items()},
"pca_cosines": pca_cos,
"t1_frob_norm": frob_t1_norm,
"verdict": verdict,
})
with open(OUT, 'w') as f:
json.dump(report, f, indent=2)
print(f"\nReport: {OUT}")
if __name__ == '__main__':
main()

1010
nautilus_dolphin/dvae/resolution_alignment_report.json Executable file
View File

File diff suppressed because it is too large Load Diff

287
nautilus_dolphin/dvae/run_gold_with_flint_gate.py Executable file
View File

@@ -0,0 +1,287 @@
"""
Task 5: Backtest with proxy_B Flint Gate vs Gold Standard.
Tests three gate configs against the gold baseline:
Gold: ROI~+44.89%, PF~1.123, DD~14.95%, Sharpe~2.50, Trades~2128
(same 55-day NG3 5s dataset, same engine stack)
Gate variants:
A. No gate (baseline reproduction — sanity check)
B. Fixed threshold=0.0 (allow when proxy_B > 0)
C. Adaptive p50 (allow when proxy_B > rolling median)
D. Adaptive p75 (allow when proxy_B > rolling p75)
Measures: ROI, PF, DD, WR, Sharpe, Trades, gate suppression rate.
DOES NOT MODIFY ANY PRODUCTION CODE.
"""
import sys, time, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
from datetime import datetime
import numpy as np
import pandas as pd
_HERE = Path(__file__).resolve().parent
_ND_ROOT = _HERE.parent # nautilus_dolphin/ outer dir — contains nautilus_dolphin pkg + mc/
# Insert ND_ROOT at index 0 so it takes priority over any stub nautilus_dolphin at project root
sys.path.insert(0, str(_ND_ROOT))
print("Compiling numba kernels...")
t0c = time.time()
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from nautilus_dolphin.nautilus.ob_features import (
OBFeatureEngine, compute_imbalance_nb, compute_depth_1pct_nb,
compute_depth_quality_nb, compute_fill_probability_nb, compute_spread_proxy_nb,
compute_depth_asymmetry_nb, compute_imbalance_persistence_nb,
compute_withdrawal_velocity_nb, compute_market_agreement_nb, compute_cascade_signal_nb,
)
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
# DolphinForewarner skipped — pickle hangs on sklearn 1.8 vs 1.7.1 mismatch; 0 interventions anyway
from mc.mc_ml import DolphinForewarner
from alpha_signal_generator_flint_gate import FlintGatedEngine
_p = np.array([1.0, 2.0, 3.0], dtype=np.float64)
compute_irp_nb(_p, -1); compute_ars_nb(1.0, 0.5, 0.01)
rank_assets_irp_nb(np.ones((10, 2), dtype=np.float64), 8, -1, 5, 500.0, 20, 0.20)
compute_sizing_nb(-0.03, -0.02, -0.05, 3.0, 0.5, 5.0, 0.20, True, True, 0.0,
np.zeros(4, dtype=np.int64), np.zeros(4, dtype=np.int64),
np.zeros(5, dtype=np.float64), 0, -1, 0.01, 0.04)
check_dc_nb(_p, 3, 1, 0.75)
_b = np.array([100.0, 200.0, 300.0, 400.0, 500.0], dtype=np.float64)
_a = np.array([110.0, 190.0, 310.0, 390.0, 510.0], dtype=np.float64)
compute_imbalance_nb(_b, _a); compute_depth_1pct_nb(_b, _a)
compute_market_agreement_nb(np.array([0.1, -0.05], dtype=np.float64), 2)
compute_cascade_signal_nb(np.array([-0.05, -0.15], dtype=np.float64), 2, -0.10)
print(f" JIT: {time.time() - t0c:.1f}s")
VBT_DIR = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
META_COLS = {'timestamp', 'scan_number', 'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity', 'vel_div',
'instability_50', 'instability_150'}
ENGINE_KWARGS = dict(
initial_capital=25000.0, vel_div_threshold=-0.02, vel_div_extreme=-0.05,
min_leverage=0.5, max_leverage=5.0, leverage_convexity=3.0,
fraction=0.20, fixed_tp_pct=0.0095, stop_pct=1.0, max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=0.75,
dc_skip_contradicts=True, dc_leverage_boost=1.0, dc_leverage_reduce=0.5,
use_asset_selection=True, min_irp_alignment=0.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=0.62, sp_maker_exit_rate=0.50,
use_ob_edge=True, ob_edge_bps=5.0, ob_confirm_rate=0.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
MC_MODELS_DIR = str(Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\nautilus_dolphin\mc_results\models"))
MC_BASE_CFG = {
'trial_id': 0,
'vel_div_threshold': -0.020, 'vel_div_extreme': -0.050,
'use_direction_confirm': True, 'dc_lookback_bars': 7,
'dc_min_magnitude_bps': 0.75, 'dc_skip_contradicts': True,
'dc_leverage_boost': 1.00, 'dc_leverage_reduce': 0.50,
'vd_trend_lookback': 10, 'min_leverage': 0.50,
'max_leverage': 5.00, 'leverage_convexity': 3.00, 'fraction': 0.20,
'use_alpha_layers': True, 'use_dynamic_leverage': True,
'fixed_tp_pct': 0.0095, 'stop_pct': 1.00, 'max_hold_bars': 120,
'use_sp_fees': True, 'use_sp_slippage': True,
'sp_maker_entry_rate': 0.62, 'sp_maker_exit_rate': 0.50,
'use_ob_edge': True, 'ob_edge_bps': 5.00, 'ob_confirm_rate': 0.40,
'ob_imbalance_bias': -0.09, 'ob_depth_scale': 1.00,
'use_asset_selection': True, 'min_irp_alignment': 0.45, 'lookback': 100,
'acb_beta_high': 0.80, 'acb_beta_low': 0.20, 'acb_w750_threshold_pct': 60,
}
print("\nLoading MC-Forewarner...")
try:
forewarner = DolphinForewarner(models_dir=MC_MODELS_DIR)
print(" MC-Forewarner ready")
except Exception as e:
print(f" [WARN] MC-Forewarner failed to load: {e} — running without it")
forewarner = None
# ── Load data ─────────────────────────────────────────────────────
parquet_files = sorted(VBT_DIR.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
acb_master = AdaptiveCircuitBreaker()
date_strings = [pf.stem for pf in parquet_files]
acb_master.preload_w750(date_strings)
all_vols = []
for pf in parquet_files[:2]:
df = pd.read_parquet(pf)
if 'BTCUSDT' not in df.columns: continue
pr = df['BTCUSDT'].values
for i in range(60, len(pr)):
seg = pr[max(0,i-50):i]
if len(seg)<10: continue
v = float(np.std(np.diff(seg)/seg[:-1]))
if v > 0: all_vols.append(v)
vol_p60 = float(np.percentile(all_vols, 60))
pq_data = {}
all_assets = set()
for pf in parquet_files:
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
all_assets.update(ac)
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0,i-50):i]
if len(seg)<10: continue
dv[i] = float(np.std(np.diff(seg)/seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
OB_ASSETS = sorted(list(all_assets))
_mock_ob = MockOBProvider(
imbalance_bias=-0.09, depth_scale=1.0, assets=OB_ASSETS,
imbalance_biases={"BTCUSDT": -0.086, "ETHUSDT": -0.092,
"BNBUSDT": +0.05, "SOLUSDT": +0.05},
)
ob_eng = OBFeatureEngine(_mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
# ── Helpers ───────────────────────────────────────────────────────
def run_backtest(engine_cls, engine_kwargs, name, gate_kwargs=None):
"""Run full 55-day backtest. Returns metrics dict.
Metrics use the SAME methodology as test_pf_dynamic_beta_validate.py:
- PF: pnl_absolute (dollar-weighted, matches gold script)
- DD: day-end capital snapshots (not trade-level curve)
- Sharpe: daily P&L, annualized with sqrt(365)
"""
gate_kwargs = gate_kwargs or {}
acb = AdaptiveCircuitBreaker()
acb.preload_w750(date_strings)
if engine_cls is NDAlphaEngine:
eng = NDAlphaEngine(**engine_kwargs)
else:
eng = FlintGatedEngine(**engine_kwargs, **gate_kwargs)
eng.set_ob_engine(ob_eng)
eng.set_acb(acb)
if forewarner is not None:
eng.set_mc_forewarner(forewarner, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
daily_caps = []
daily_pnls = []
for pf in parquet_files:
ds = pf.stem
df, acols, dvol = pq_data[ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
stats = eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
if n == 0:
return {'name': name, 'roi': 0, 'pf': 0, 'dd': 0, 'wr': 0, 'sharpe': 0,
'trades': 0, 'suppressed': 0, 'suppression_rate': 0}
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
# PF: dollar-weighted (matches gold script which uses pnl_absolute)
def _abs(t):
return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0
gross_profit = sum(_abs(t) for t in wins)
gross_loss = abs(sum(_abs(t) for t in losses))
pf = gross_profit / max(gross_loss, 1e-9)
# DD: day-end capital snapshots (matches gold script)
peak_cap = 25000.0
max_dd = 0.0
for cap in daily_caps:
peak_cap = max(peak_cap, cap)
dd = (peak_cap - cap) / peak_cap * 100.0
max_dd = max(max_dd, dd)
# Sharpe: daily P&L annualized (matches gold script)
dr = np.array([p / 25000.0 * 100.0 for p in daily_pnls])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
suppressed = getattr(eng, 'gate_suppressed', 0)
allowed = getattr(eng, 'gate_allowed', 0)
sup_rate = suppressed / max(1, suppressed + allowed) * 100.0
return {
'name': name,
'roi': roi, 'pf': pf, 'dd': max_dd, 'wr': wr, 'sharpe': sharpe,
'trades': n, 'suppressed': suppressed, 'suppression_rate': sup_rate,
}
# ── Run all configs ───────────────────────────────────────────────
GOLD = {'name': 'GOLD REFERENCE', 'roi': 88.55, 'pf': 1.215, 'dd': 15.05,
'wr': 50.5, 'sharpe': 4.38, 'trades': 2155}
configs = [
(NDAlphaEngine, {}, 'A. Baseline (no gate)', {}),
(FlintGatedEngine, {}, 'B. Gate: proxy_B > 0.00 (fixed)', {'proxy_b_threshold': 0.00}),
]
results = []
for engine_cls, _, name, gate_kw in configs:
print(f"\n{'='*55}")
print(f"Running: {name}")
t0 = time.time()
r = run_backtest(engine_cls, ENGINE_KWARGS.copy(), name, gate_kw)
r['elapsed'] = time.time() - t0
results.append(r)
print(f" Done in {r['elapsed']:.1f}s Trades={r['trades']} ROI={r['roi']:.2f}% PF={r['pf']:.4f}")
# ── Final comparison ──────────────────────────────────────────────
print("\n" + "="*75)
print("FLINT GATE vs GOLD STANDARD — FINAL COMPARISON")
print("="*75)
hdr = f"{'Config':<35} {'ROI%':>7} {'PF':>6} {'DD%':>6} {'WR%':>6} {'Sharpe':>7} {'Trades':>7} {'Supp%':>7}"
print(hdr)
print("-"*75)
g = GOLD
print(f"{'*** GOLD REFERENCE ***':<35} {g['roi']:>7.2f} {g['pf']:>6.4f} {g['dd']:>6.2f} {'N/A':>6} {g['sharpe']:>7.2f} {g['trades']:>7d} {'N/A':>7}")
print("-"*75)
for r in results:
print(f"{r['name']:<35} {r['roi']:>7.2f} {r['pf']:>6.4f} {r['dd']:>6.2f} "
f"{r['wr']:>6.2f} {r['sharpe']:>7.3f} {r['trades']:>7d} {r['suppression_rate']:>7.1f}%")
print("="*75)
print("\nLEGEND:")
print(" ROI: Return on Initial Capital ($25k)")
print(" PF : Profit Factor (gross profit / gross loss)")
print(" DD : Max Drawdown from equity peak")
print(" WR : Win Rate")
print(" Sharpe: trade-based Sharpe")
print(" Supp%: % of entry attempts suppressed by gate")
print("\n VERDICT:")
base = results[0] if results else None
best_gated = max(results[1:], key=lambda r: r['pf']) if len(results) > 1 else None
if base and best_gated:
pf_delta = best_gated['pf'] - base['pf']
roi_delta = best_gated['roi'] - base['roi']
dd_delta = best_gated['dd'] - base['dd']
print(f" Best gate ({best_gated['name']}):")
print(f" PF: {base['pf']:.4f} → {best_gated['pf']:.4f} ({pf_delta:+.4f})")
print(f" ROI: {base['roi']:.2f}% → {best_gated['roi']:.2f}% ({roi_delta:+.2f}pp)")
print(f" DD: {base['dd']:.2f}% → {best_gated['dd']:.2f}% ({dd_delta:+.2f}pp)")
print(f" Trades: {base['trades']} → {best_gated['trades']} ({best_gated['suppression_rate']:.1f}% suppressed)")
if best_gated['pf'] > base['pf'] * 1.01:
print(" → GATE IS BENEFICIAL: PF improved >1%")
elif best_gated['pf'] > base['pf']:
print(" → GATE SHOWS MARGINAL IMPROVEMENT")
else:
print(" → GATE IS NEUTRAL/NEGATIVE: no improvement over baseline")

468
nautilus_dolphin/dvae/run_trace_backtest.py Executable file
View File

@@ -0,0 +1,468 @@
"""
run_trace_backtest.py — D_LIQ_GOLD full 56-day backtest with painstaking per-tick
and per-trade state logging. Streams to CSV; never accumulates more than one day
of ticks in RAM (~1.2 MB). No PyTorch, no dvae/__init__, no ensemble warmup.
Outputs (dvae/trace/):
tick_trace.csv — one row per valid bar (346k+ rows)
trade_trace.csv — one row per closed trade (~2155 rows)
daily_trace.csv — one row per day (56 rows)
summary.json — ROI / DD / T / Calmar at end
Tick CSV columns:
date, bar_ri, global_bar, btc_price, vel_div, vol_ok,
proxy_b, base_max_lev, sizer_max_lev, regime_size_mult,
capital, pos_open, pos_dir, pos_entry_price, pos_entry_bar,
pos_entry_lev, acb_boost, day_beta, mc_status, dd_halt, event
Trade CSV columns:
trade_id, asset, date_exit, direction, entry_price, exit_price,
entry_bar, exit_bar, notional, leverage, pnl_pct, pnl_abs,
exit_reason, bars_held, proxy_b_at_entry, proxy_b_at_exit,
capital_before_exit, capital_after_exit
Daily CSV columns:
date, cap_start, cap_end, pnl, n_trades, acb_boost, day_beta,
mc_status, max_dd_intraday
Run after freeing RAM. Prints progress every 5 days.
"""
import sys, os, csv, json, gc, math, time
from pathlib import Path
import numpy as np
import pandas as pd
# ------ Path setup (MUST come before any nautilus_dolphin import) ----------
_HERE = Path(__file__).resolve().parent # dvae/
_ND_ROOT = _HERE.parent # nautilus_dolphin/
sys.path.insert(0, str(_ND_ROOT))
# Add dvae/ directly so we can import exp_shared without triggering dvae/__init__
sys.path.insert(0, str(_HERE))
# ------ Imports (NO dvae package, NO torch) --------------------------------
from exp_shared import ENGINE_KWARGS, MC_BASE_CFG, VBT_DIR, MC_MODELS_DIR, META_COLS
from nautilus_dolphin.nautilus.proxy_boost_engine import (
create_d_liq_engine, LiquidationGuardEngine,
)
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.ob_features import OBFeatureEngine
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
# ------ Output directory ---------------------------------------------------
TRACE_DIR = _HERE / 'trace'
TRACE_DIR.mkdir(exist_ok=True)
TICK_CSV = TRACE_DIR / 'tick_trace.csv'
TRADE_CSV = TRACE_DIR / 'trade_trace.csv'
DAILY_CSV = TRACE_DIR / 'daily_trace.csv'
SUMMARY = TRACE_DIR / 'summary.json'
TICK_HEADER = [
'date', 'bar_ri', 'global_bar', 'btc_price', 'vel_div', 'vol_ok',
'proxy_b', 'base_max_lev', 'sizer_max_lev', 'regime_size_mult',
'capital', 'pos_open', 'pos_dir', 'pos_entry_price', 'pos_entry_bar',
'pos_entry_lev', 'acb_boost', 'day_beta', 'mc_status', 'dd_halt', 'event',
]
TRADE_HEADER = [
'trade_id', 'asset', 'date_exit', 'direction', 'entry_price', 'exit_price',
'entry_bar', 'exit_bar', 'notional', 'leverage', 'pnl_pct', 'pnl_abs',
'exit_reason', 'bars_held',
'proxy_b_at_entry', 'proxy_b_at_exit',
'capital_before_exit', 'capital_after_exit',
]
DAILY_HEADER = [
'date', 'cap_start', 'cap_end', 'pnl', 'n_trades',
'acb_boost', 'day_beta', 'mc_status', 'max_dd_intraday',
]
# ===========================================================================
# TracingLiquidationEngine — thin subclass that instruments step_bar
# ===========================================================================
class TracingLiquidationEngine(LiquidationGuardEngine):
"""
Wraps LiquidationGuardEngine with per-bar state capture.
Overrides step_bar() to record tick + trade events.
All trace data streamed to CSV via external writers (no RAM accumulation).
"""
def __init__(self, tick_writer, trade_writer, **kwargs):
super().__init__(**kwargs)
self._tw = tick_writer
self._trd = trade_writer
self._cur_date = ''
self._entry_pb_cache: dict = {} # entry_bar → proxy_b at entry
self._tick_buf: list = [] # per-day buffer, flushed at end_day
self._trade_buf: list = [] # per-day buffer, flushed at end_day
self._day_cap_hi = 0.0
self._day_cap_lo = float('inf')
# ── Override begin_day to track date and intraday extremes ───────────────
def begin_day(self, date_str, posture='APEX', direction=None):
super().begin_day(date_str, posture=posture, direction=direction)
self._cur_date = date_str
self._tick_buf = []
self._trade_buf = []
self._day_cap_hi = self.capital
self._day_cap_lo = self.capital
# ── Override step_bar to capture tick state ──────────────────────────────
def step_bar(self, bar_idx, vel_div, prices, vol_regime_ok=True,
price_histories=None, v50_vel=0.0, v750_vel=0.0):
# --- Pre-call state snapshot ---
proxy_b_now = float(self._current_proxy_b)
n_before = len(self.trade_history)
pos_before = self.position # None or NDPosition
cap_before = self.capital
# --- Actual engine work ---
result = super().step_bar(
bar_idx=bar_idx, vel_div=vel_div, prices=prices,
vol_regime_ok=vol_regime_ok, price_histories=price_histories,
v50_vel=v50_vel, v750_vel=v750_vel,
)
# --- Post-call state snapshot ---
n_after = len(self.trade_history)
pos_after = self.position
cap_after = self.capital
# Track intraday capital range
if cap_after > self._day_cap_hi: self._day_cap_hi = cap_after
if cap_after < self._day_cap_lo: self._day_cap_lo = cap_after
# Determine event
new_entry = (pos_before is None and pos_after is not None)
new_exit = (n_after > n_before)
event = 'NONE'
if new_entry and new_exit:
event = 'ENTRY+EXIT' # same-bar close+reopen (rare)
elif new_entry:
event = 'ENTRY'
elif new_exit:
event = 'EXIT'
# Cache proxy_B at entry time
if new_entry and pos_after is not None:
self._entry_pb_cache[getattr(pos_after, 'entry_bar', bar_idx)] = proxy_b_now
# Capture trade rows for completed trades
if new_exit:
for t in self.trade_history[n_before:]:
entry_pb = self._entry_pb_cache.pop(getattr(t, 'entry_bar', bar_idx), proxy_b_now)
self._trade_buf.append((
t.trade_id,
t.asset,
self._cur_date,
t.direction,
round(t.entry_price, 4),
round(t.exit_price, 4),
t.entry_bar,
t.exit_bar,
round(t.notional, 2),
round(t.leverage, 4),
round(t.pnl_pct, 6),
round(t.pnl_absolute,4),
t.exit_reason,
t.bars_held,
round(entry_pb, 6),
round(proxy_b_now, 6),
round(cap_before, 4),
round(cap_after, 4),
))
# BTC close price (primary signal asset)
btc_price = prices.get('BTCUSDT', 0.0)
# Position fields
pos_dir = pos_entry_price = pos_entry_lev = 0.0
pos_entry_bar_i = -1
pos_open = 0
if pos_after is not None:
pos_open = 1
pos_dir = int(getattr(pos_after, 'direction', 0))
pos_entry_price = round(float(getattr(pos_after, 'entry_price', 0.0)), 4)
pos_entry_bar_i = int(getattr(pos_after, 'entry_bar', -1))
pos_entry_lev = round(float(getattr(pos_after, 'leverage', 0.0)), 4)
# global_bar was incremented INSIDE super().step_bar, so -1 gives current
global_bar = self._global_bar_idx - 1
self._tick_buf.append((
self._cur_date,
bar_idx,
global_bar,
round(btc_price, 4),
round(vel_div, 6),
int(vol_regime_ok),
round(proxy_b_now, 6),
round(self.base_max_leverage, 2),
round(self.bet_sizer.max_leverage, 2),
round(getattr(self, 'regime_size_mult', 1.0), 4),
round(cap_after, 4),
pos_open,
pos_dir,
pos_entry_price,
pos_entry_bar_i,
pos_entry_lev,
round(getattr(self, '_day_base_boost', 1.0), 4),
round(getattr(self, '_day_beta', 0.0), 4),
getattr(self, '_day_mc_status', 'OK'),
int(getattr(self, 'regime_dd_halt', False)),
event,
))
return result
# ── Override end_day to flush buffers ────────────────────────────────────
def end_day(self):
result = super().end_day()
# Flush tick buffer
if self._tick_buf:
self._tw.writerows(self._tick_buf)
# Flush trade buffer
if self._trade_buf:
self._trd.writerows(self._trade_buf)
self._tick_buf = []
self._trade_buf = []
return result
# ===========================================================================
# Main
# ===========================================================================
def main():
t_start = time.time()
print("=== D_LIQ_GOLD Trace Backtest ===")
print(f" Outputs → {TRACE_DIR}")
# -- Load data (GOLD method: float64 pq_data, seg-based vol_p60) -----------
print(" Loading data (gold method)...")
parquet_files = sorted(VBT_DIR.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
date_strings = [p.stem for p in parquet_files]
print(f" {len(parquet_files)} parquet files")
# Gold vol_p60: 2 files, range(60), seg-based, v>0 filter
all_vols = []
for pf in parquet_files[:2]:
df = pd.read_parquet(pf)
if 'BTCUSDT' in df.columns:
pr = df['BTCUSDT'].values
for i in range(60, len(pr)):
seg = pr[max(0, i-50):i]
if len(seg) < 10: continue
v = float(np.std(np.diff(seg) / seg[:-1]))
if v > 0: all_vols.append(v)
del df
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.0002
print(f" vol_p60 (gold) = {vol_p60:.8f}")
# Get asset list from first file
df0 = pd.read_parquet(parquet_files[0])
OB_ASSETS = sorted([c for c in df0.columns if c not in META_COLS])
del df0
print(f" OB assets: {len(OB_ASSETS)}")
mock_ob = MockOBProvider(
imbalance_bias=-0.09, depth_scale=1.0, assets=OB_ASSETS,
imbalance_biases={"BTCUSDT": -0.086, "ETHUSDT": -0.092,
"BNBUSDT": +0.05, "SOLUSDT": +0.05},
)
ob_eng = OBFeatureEngine(mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
# -- Load MC-Forewarner (optional) ----------------------------------------
fw = None
try:
from mc.mc_ml import DolphinForewarner
fw = DolphinForewarner(models_dir=MC_MODELS_DIR)
print(" MC-Forewarner: loaded")
except Exception as e:
print(f" MC-Forewarner: unavailable ({e})")
# -- Open output CSVs (streaming) -----------------------------------------
tick_fh = open(TICK_CSV, 'w', newline='', encoding='utf-8')
trade_fh = open(TRADE_CSV, 'w', newline='', encoding='utf-8')
daily_fh = open(DAILY_CSV, 'w', newline='', encoding='utf-8')
tick_w = csv.writer(tick_fh)
trade_w = csv.writer(trade_fh)
daily_w = csv.writer(daily_fh)
tick_w.writerow(TICK_HEADER)
trade_w.writerow(TRADE_HEADER)
daily_w.writerow(DAILY_HEADER)
# -- Build engine ---------------------------------------------------------
print(" Building TracingLiquidationEngine...")
acb = AdaptiveCircuitBreaker()
acb.preload_w750(date_strings)
eng = TracingLiquidationEngine(
tick_writer=tick_w, trade_writer=trade_w,
**ENGINE_KWARGS,
)
eng.set_ob_engine(ob_eng)
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0) # fixed function: respects _extended_soft_cap=8.0
print(f" base_max_lev={eng.base_max_leverage} abs_max_lev={eng.abs_max_leverage} "
f"sizer_max_lev={eng.bet_sizer.max_leverage}")
# -- Run 56-day backtest (lazy loading, float32, gc per day) --------------
print(" Running backtest (lazy load, float32 per day)...")
print()
daily_caps = []
running_vols = []
peak_cap = 25000.0
max_dd = 0.0
for i, pf in enumerate(parquet_files):
ds = pf.stem
# Lazy load + cast to float32 to save RAM
df = pd.read_parquet(pf)
for c in df.columns:
if df[c].dtype == 'float64':
df[c] = df[c].astype('float32')
acols = [c for c in df.columns if c not in META_COLS]
# Per-day OB preload
ob_eng.preload_date(ds, OB_ASSETS)
# Compute dvol for this day
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dvol = np.zeros(len(df), dtype=np.float32)
if bp is not None:
rets = np.diff(bp.astype('float64')) / (bp[:-1].astype('float64') + 1e-9)
for j in range(50, len(rets)):
v = np.std(rets[j-50:j])
dvol[j+1] = v
if v > 0:
running_vols.append(v)
vp60 = np.percentile(running_vols, 60) if len(running_vols) > 1000 else vol_p60
vol_ok = np.where(dvol > 0, dvol > vp60, False)
cap_before = eng.capital
n_before = len(eng.trade_history)
# process_day (which internally calls begin_day → step_bar loop → end_day)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
cap_after = eng.capital
n_after = len(eng.trade_history)
daily_caps.append(cap_after)
# Compute intraday DD (rough)
peak_cap = max(peak_cap, cap_after)
max_dd = max(max_dd, (peak_cap - cap_after) / peak_cap * 100.0)
intra_dd_pct = (eng._day_cap_hi - eng._day_cap_lo) / max(eng._day_cap_hi, 1e-9) * 100.0
# Daily row
daily_w.writerow((
ds,
round(cap_before, 4),
round(cap_after, 4),
round(cap_after - cap_before, 4),
n_after - n_before,
round(getattr(eng, '_day_base_boost', 1.0), 4),
round(getattr(eng, '_day_beta', 0.0), 4),
getattr(eng, '_day_mc_status', 'OK'),
round(intra_dd_pct, 4),
))
# Progress every 5 days
if (i + 1) % 5 == 0 or i == len(parquet_files) - 1:
roi_now = (cap_after - 25000.0) / 25000.0 * 100.0
elapsed = time.time() - t_start
print(f" Day {i+1:3d}/{len(parquet_files)} {ds} "
f"cap={cap_after:.0f} ROI={roi_now:+.2f}% T={n_after} "
f"DD={max_dd:.2f}% ({elapsed:.0f}s)")
# Clear OB cache + GC
if eng.ob_engine is not None:
for attr in ('_preloaded_placement', '_preloaded_signal',
'_preloaded_market', '_ts_to_idx'):
try: getattr(eng.ob_engine, attr).clear()
except Exception: pass
del df
gc.collect()
# -- Close CSV files -------------------------------------------------------
tick_fh.flush(); tick_fh.close()
trade_fh.flush(); trade_fh.close()
daily_fh.flush(); daily_fh.close()
# -- Final stats -----------------------------------------------------------
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
def _abs(t):
return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
wr = len(wins) / n * 100.0 if n > 0 else 0.0
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
dr = np.array([(c - 25000.0 if i == 0 else daily_caps[i] - daily_caps[i-1]) / 25000.0 * 100.0
for i, c in enumerate(daily_caps)])
sharpe = float(dr.mean() / (dr.std() + 1e-9) * math.sqrt(365)) if len(dr) > 1 else 0.0
calmar = roi / max(max_dd, 1e-9)
liq_stops = getattr(eng, 'liquidation_stops', 0)
summary = dict(
roi=round(roi, 4),
dd=round(max_dd, 4),
calmar=round(calmar, 4),
pf=round(pf, 4),
wr=round(wr, 4),
sharpe=round(sharpe, 4),
trades=n,
liq_stops=liq_stops,
capital_final=round(eng.capital, 4),
elapsed_s=round(time.time() - t_start, 1),
tick_csv=str(TICK_CSV),
trade_csv=str(TRADE_CSV),
daily_csv=str(DAILY_CSV),
gold_roi=181.81,
gold_dd=17.65,
gold_trades=2155,
)
with open(SUMMARY, 'w', encoding='utf-8') as f:
json.dump(summary, f, indent=2)
print()
print("=== RESULTS ===")
print(f" ROI = {roi:+.2f}% (gold 181.81%)")
print(f" DD = {max_dd:.2f}% (gold 17.65%)")
print(f" Calmar = {calmar:.2f} (gold 10.30)")
print(f" PF = {pf:.4f} (gold ~1.55)")
print(f" WR = {wr:.2f}%")
print(f" T = {n} (gold 2155)")
print(f" liq_stops = {liq_stops}")
print(f" Time = {time.time()-t_start:.0f}s")
print()
roi_ok = abs(roi - 181.81) < 1.0
dd_ok = abs(max_dd - 17.65) < 0.5
t_ok = n == 2155
print(f" ROI match: {'✓ PASS' if roi_ok else '✗ FAIL'} (diff={roi-181.81:+.2f}pp)")
print(f" DD match: {'✓ PASS' if dd_ok else '✗ FAIL'} (diff={max_dd-17.65:+.2f}pp)")
print(f" T match: {'✓ PASS' if t_ok else '✗ FAIL'} (got {n})")
print()
print(f" Tick trace → {TICK_CSV} ({TICK_CSV.stat().st_size//1024}KB)")
print(f" Trade trace → {TRADE_CSV}")
print(f" Daily trace → {DAILY_CSV}")
print(f" Summary → {SUMMARY}")
if __name__ == '__main__':
main()

104
nautilus_dolphin/dvae/targeted_data_archaeology.py Executable file
View File

@@ -0,0 +1,104 @@
import json
import re
import os
from pathlib import Path
from datetime import datetime
BASE = Path(r"C:\Users\Lenovo\Documents")
DIRS = {
"NG1": BASE / "- Dolphin NG",
"NG2": BASE / "- Dolphin NG2",
"NG4": BASE / "- DOLPHIN NG4" / "- Results",
"NG5": BASE / "- Dolphin NG5",
"NG3": BASE / "- Dolphin NG HD (NG3)" / "correlation_arb512" / "eigenvalues"
}
def parse_ts(s):
for fmt in ("%Y-%m-%dT%H:%M:%S.%f", "%Y-%m-%dT%H:%M:%S",
"%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S",
"%Y-%m-%dT%H:%M:%SZ"):
try:
return datetime.strptime(str(s)[:26].replace('Z', '').replace('T', ' '), fmt)
except ValueError:
continue
return None
def get_boundary_json(d, pattern):
if not d.exists(): return None, None, 0
files = sorted(list(d.glob(pattern)))
if not files: return None, None, 0
def extract_ts(f):
try:
with open(f, 'r', encoding='utf-8', errors='replace') as fb:
data = json.load(fb)
return parse_ts(data.get('timestamp'))
except: return None
ts_start = None
for f in files:
ts_start = extract_ts(f)
if ts_start: break
ts_end = None
for f in reversed(files):
ts_end = extract_ts(f)
if ts_end: break
return ts_start, ts_end, len(files)
def get_boundary_ng4(d):
if not d.exists(): return None, None, 0
files = sorted(list(d.glob('*.txt')))
if not files: return None, None, 0
log_re = re.compile(r'(\d{4}-\d{2}-\d{2}T[\d:.]+Z)')
def extract_first_last_ts(f):
first = None
last = None
try:
with open(f, 'r', encoding='utf-8', errors='replace') as fb:
for line in fb:
m = log_re.search(line)
if m:
ts = parse_ts(m.group(1))
if not first: first = ts
last = ts
except: pass
return first, last
ts_min = None
ts_max = None
for f in files:
f_min, f_max = extract_first_last_ts(f)
if not ts_min: ts_min = f_min
if f_max: ts_max = f_max
return ts_min, ts_max, len(files)
def get_boundary_ng3(d):
if not d.exists(): return None, None, 0
subdirs = sorted([s for s in d.iterdir() if s.is_dir() and not s.name.endswith('_SKIP')])
if not subdirs: return None, None, 0
ts_min, _, _ = get_boundary_json(subdirs[0], 'scan_*.json')
_, ts_max, _ = get_boundary_json(subdirs[-1], 'scan_*.json')
total_files = sum(len(list(s.glob('scan_*.json'))) for s in subdirs)
return ts_min, ts_max, total_files
print("--- Targeted Data Archaeology Result ---")
for name, d in DIRS.items():
print(f"Checking {name}...")
if name in ["NG1", "NG2", "NG5"]:
ts_start, ts_end, count = get_boundary_json(d, 'regime_result_*.json')
elif name == "NG4":
ts_start, ts_end, count = get_boundary_ng4(d)
elif name == "NG3":
ts_start, ts_end, count = get_boundary_ng3(d)
if ts_start:
print(f" {name}: {ts_start} to {ts_end} ({count} files)")
else:
print(f" {name}: No data found.")

71
nautilus_dolphin/dvae/test_bronze_daily_dd.py Executable file
View File

@@ -0,0 +1,71 @@
"""BRONZE baseline with DAILY equity curve DD (same method as test_dliq_fix_verify.py).
Expected: ROI=88.55%, DD=15.05%, T=2155
"""
import sys, time, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import ENGINE_KWARGS, MC_BASE_CFG, load_data, load_forewarner
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
def main():
print("Loading data (gold method: float64, 2-file seg-based vol_p60)...")
d = load_data()
fw = load_forewarner()
print()
kw = ENGINE_KWARGS.copy()
eng = NDAlphaEngine(**kw)
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
print(f" base_max_leverage = {eng.base_max_leverage}")
t0 = time.time()
daily_caps = []
for pf_file in d['parquet_files']:
ds = pf_file.stem
df, acols, dvol = d['pq_data'][ds]
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
peak, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak = max(peak, cap)
max_dd = max(max_dd, (peak - cap) / peak * 100.0)
def _abs(t):
return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
wr = len(wins) / n * 100.0 if n > 0 else 0.0
print(f" ROI={roi:+.2f}% T={n} DD={max_dd:.2f}% PF={pf:.4f} WR={wr:.1f}% ({time.time()-t0:.0f}s)")
print()
print(f"BRONZE TARGET: ROI=88.55% T=2155 DD=15.05%")
roi_ok = abs(roi - 88.55) < 1.0
dd_ok = abs(max_dd - 15.05) < 0.5
t_ok = n == 2155
print(f"ROI match: {'PASS' if roi_ok else 'FAIL'} (diff={roi-88.55:+.2f}pp)")
print(f"DD match: {'PASS' if dd_ok else 'FAIL'} (diff={max_dd-15.05:+.2f}pp)")
print(f"T match: {'PASS' if t_ok else 'FAIL'} (got {n})")
if __name__ == '__main__':
main()

332
nautilus_dolphin/dvae/test_convnext_dvae.py Executable file
View File

@@ -0,0 +1,332 @@
"""
Unit tests for ConvNeXt-1D β-TCVAE (pure numpy).
Tests
-----
T1 DWConv1d forward shape correct
T2 DWConv1d backward numerical gradient check
T3 LayerNorm forward/backward numerical gradient check
T4 ConvNeXtBlock1D output shape == input shape (skip preserved)
T5 ConvNeXtBlock1D backward numerical gradient check
T6 ConvNeXtVAE forward: output shapes correct
T7 ConvNeXtVAE forward/backward: loss is finite, grads finite
T8 Loss decreases over 20 steps on small batch (gradient descent works)
T9 Save/load round-trip: weights preserved
T10 β-TCVAE loss backward: numerical gradient check on z_mu/z_logvar
Run: python test_convnext_dvae.py
"""
import sys
import os
import tempfile
import numpy as np
# ensure local import
sys.path.insert(0, os.path.dirname(__file__))
from convnext_dvae import (
ConvNeXtVAE, DWConv1d, LayerNorm, ConvNeXtBlock1D,
btcvae_loss, btcvae_loss_backward
)
RNG = np.random.RandomState(0)
PASS = []
FAIL = []
def check(name: str, cond: bool, detail: str = ''):
if cond:
print(f' PASS {name}')
PASS.append(name)
else:
print(f' FAIL {name} {detail}')
FAIL.append(name)
def num_grad(f, x, eps=1e-5):
"""Numeric gradient of scalar f at x."""
g = np.zeros_like(x)
it = np.nditer(x, flags=['multi_index'])
while not it.finished:
idx = it.multi_index
orig = x[idx]
x[idx] = orig + eps
fp = f(x)
x[idx] = orig - eps
fm = f(x)
x[idx] = orig
g[idx] = (fp - fm) / (2 * eps)
it.iternext()
return g
# ---------------------------------------------------------------------------
def test_t1_dwconv_shape():
print('\nT1: DWConv1d forward shape')
layer = DWConv1d(8, 7, RNG)
x = RNG.randn(4, 8, 32)
y = layer.forward(x)
check('output_shape', y.shape == (4, 8, 32), str(y.shape))
def test_t2_dwconv_grad():
print('\nT2: DWConv1d backward numerical check')
rng2 = np.random.RandomState(1)
layer = DWConv1d(4, 3, rng2)
x = rng2.randn(2, 4, 8).astype(np.float64) * 0.5
def fwd(w_flat):
layer.w.data[:] = w_flat.reshape(layer.w.data.shape)
out = layer.forward(x)
return float(out.sum())
layer.forward(x)
layer.backward(np.ones((2, 4, 8)))
w0 = layer.w.data.copy()
ng = num_grad(lambda w: fwd(w.ravel()), w0.ravel())
ag = layer.w.grad.ravel()
rel = np.abs(ag - ng) / (np.abs(ng) + 1e-8)
check('dw_max_rel_err', rel.max() < 0.01, f'max_rel={rel.max():.4f}')
# also check dx
def fwd_x(xf):
return float(layer.forward(xf.reshape(2, 4, 8)).sum())
ng_x = num_grad(lambda xf: fwd_x(xf), x.ravel().copy())
layer.forward(x)
gx = layer.backward(np.ones((2, 4, 8))).ravel()
rel_x = np.abs(gx - ng_x) / (np.abs(ng_x) + 1e-8)
check('dx_max_rel_err', rel_x.max() < 0.01, f'max_rel_x={rel_x.max():.4f}')
def test_t3_layernorm_grad():
print('\nT3: LayerNorm backward numerical check')
rng2 = np.random.RandomState(2)
layer = LayerNorm(8)
x = rng2.randn(4, 12, 8).astype(np.float64)
def fwd_gamma(gf):
layer.gamma.data[:] = gf
return float(layer.forward(x).sum())
layer.forward(x)
layer.backward(np.ones((4, 12, 8)))
ag = layer.gamma.grad.copy()
ng = num_grad(lambda gf: fwd_gamma(gf), layer.gamma.data.copy())
rel = np.abs(ag - ng) / (np.abs(ng) + 1e-8)
check('dgamma_max_rel_err', rel.max() < 0.01, f'{rel.max():.4f}')
def fwd_x(xf):
return float(layer.forward(xf.reshape(4, 12, 8)).sum())
dx = layer.backward(np.ones((4, 12, 8))).ravel()
ng_x = num_grad(lambda xf: fwd_x(xf), x.ravel().copy())
rel_x = np.abs(dx - ng_x) / (np.abs(ng_x) + 1e-8)
check('dx_max_rel_err', rel_x.max() < 0.01, f'{rel_x.max():.4f}')
def test_t4_block_shape():
print('\nT4: ConvNeXtBlock1D shape preserved')
rng2 = np.random.RandomState(3)
blk = ConvNeXtBlock1D(16, 7, rng2)
x = rng2.randn(4, 16, 32)
y = blk.forward(x)
check('shape_unchanged', y.shape == x.shape, str(y.shape))
check('skip_active', not np.allclose(y, x)) # block changes values
def test_t5_block_grad():
print('\nT5: ConvNeXtBlock1D backward numerical check (small block)')
rng2 = np.random.RandomState(4)
blk = ConvNeXtBlock1D(4, 3, rng2)
x = rng2.randn(2, 4, 8).astype(np.float64) * 0.3
def fwd_x(xf):
return float(blk.forward(xf.reshape(2, 4, 8)).sum())
blk.forward(x)
dx_an = blk.backward(np.ones((2, 4, 8))).ravel()
ng_x = num_grad(lambda xf: fwd_x(xf), x.ravel().copy())
rel = np.abs(dx_an - ng_x) / (np.abs(ng_x) + 1e-8)
check('dx_max_rel_err', rel.max() < 0.02, f'{rel.max():.4f}')
def test_t6_vae_shapes():
print('\nT6: ConvNeXtVAE forward shapes')
rng2 = np.random.RandomState(5)
model = ConvNeXtVAE(C_in=8, T_in=32, z_dim=16, base_ch=16, n_blocks=2, seed=5)
x = rng2.randn(4, 8, 32)
x_recon, z_mu, z_logvar, z = model.forward(x)
check('x_recon_shape', x_recon.shape == (4, 8, 32), str(x_recon.shape))
check('z_mu_shape', z_mu.shape == (4, 16), str(z_mu.shape))
check('z_logvar_shape',z_logvar.shape == (4, 16), str(z_logvar.shape))
check('x_recon_finite',np.all(np.isfinite(x_recon)))
check('z_mu_finite', np.all(np.isfinite(z_mu)))
def test_t7_vae_backward():
print('\nT7: ConvNeXtVAE backward: grads finite, loss finite')
rng2 = np.random.RandomState(6)
model = ConvNeXtVAE(C_in=8, T_in=32, z_dim=16, base_ch=16, n_blocks=2, seed=6)
x = rng2.randn(8, 8, 32)
eps_noise = rng2.randn(8, 16)
# manual forward with stored eps
z_mu, z_logvar = model.encode(x)
z = z_mu + eps_noise * np.exp(0.5 * z_logvar)
x_recon = model.decode(z)
loss, info = btcvae_loss(x, x_recon, z_mu, z_logvar, z, beta_tc=2.0)
check('loss_finite', np.isfinite(loss), f'loss={loss:.4f}')
for k, v in info.items():
check(f'{k}_finite', np.isfinite(v), f'{k}={v:.4f}')
d_recon, d_z_mu, d_z_logvar = btcvae_loss_backward(
x, x_recon, z_mu, z_logvar, z, eps_noise, beta_tc=2.0
)
model.zero_grad()
dz = model.backward_decode(d_recon)
model.backward_encode(d_z_mu + dz, d_z_logvar)
all_grads = [p.grad for p in model.all_params()]
any_nan = any(np.any(np.isnan(g)) for g in all_grads)
any_inf = any(np.any(np.isinf(g)) for g in all_grads)
all_zero = all(np.all(g == 0) for g in all_grads)
check('grads_no_nan', not any_nan)
check('grads_no_inf', not any_inf)
check('grads_not_all_zero', not all_zero)
def test_t8_loss_decreases():
print('\nT8: Loss decreases over 20 gradient steps')
rng2 = np.random.RandomState(7)
model = ConvNeXtVAE(C_in=8, T_in=32, z_dim=16, base_ch=16, n_blocks=2, seed=7)
x = rng2.randn(16, 8, 32).astype(np.float64)
losses = []
lr = 1e-3
for step in range(20):
rng2_eps = np.random.RandomState(step)
eps_noise = rng2_eps.randn(16, 16)
z_mu, z_logvar = model.encode(x)
z = z_mu + eps_noise * np.exp(0.5 * z_logvar)
x_recon = model.decode(z)
loss, _ = btcvae_loss(x, x_recon, z_mu, z_logvar, z, beta_tc=1.0, alpha_mi=1.0)
losses.append(loss)
d_recon, d_z_mu, d_z_logvar = btcvae_loss_backward(
x, x_recon, z_mu, z_logvar, z, eps_noise, beta_tc=1.0, alpha_mi=1.0
)
model.zero_grad()
dz = model.backward_decode(d_recon)
model.backward_encode(d_z_mu + dz, d_z_logvar)
model.adam_step(lr)
first5 = np.mean(losses[:5])
last5 = np.mean(losses[-5:])
print(f' loss[0:5]={first5:.4f} loss[15:20]={last5:.4f}')
check('loss_decreasing', last5 < first5, f'{last5:.4f} vs {first5:.4f}')
def test_t9_save_load():
print('\nT9: Save/load round-trip')
rng2 = np.random.RandomState(8)
model = ConvNeXtVAE(C_in=8, T_in=32, z_dim=16, base_ch=16, n_blocks=2, seed=8)
x = rng2.randn(4, 8, 32)
# deterministic forward: encode then decode with fixed z_mu (no sampling)
z_mu1, _ = model.encode(x)
x_recon1 = model.decode(z_mu1)
nm = rng2.randn(8).astype(np.float64)
ns = np.abs(rng2.randn(8)).astype(np.float64) + 0.1
with tempfile.NamedTemporaryFile(suffix='.json', delete=False) as f:
path = f.name
try:
model.save(path, norm_mean=nm, norm_std=ns)
model2 = ConvNeXtVAE(C_in=8, T_in=32, z_dim=16, base_ch=16, n_blocks=2, seed=0)
extras = model2.load(path)
check('norm_mean_saved', 'norm_mean' in extras)
check('norm_std_saved', 'norm_std' in extras)
z_mu2, _ = model2.encode(x)
x_recon2 = model2.decode(z_mu2)
check('recon_reproduced', np.allclose(x_recon1, x_recon2, atol=1e-10))
check('z_mu_reproduced', np.allclose(z_mu1, z_mu2, atol=1e-10))
finally:
os.unlink(path)
def test_t10_btcvae_grad():
print('\nT10: beta-TCVAE backward numerical gradient check (z_mu, z_logvar)')
rng2 = np.random.RandomState(9)
B, D = 8, 6
C, T = 4, 8
x = rng2.randn(B, C, T).astype(np.float64)
z_mu = rng2.randn(B, D).astype(np.float64) * 0.5
z_lv = rng2.randn(B, D).astype(np.float64) * 0.3 - 1.0 # negative → small var
eps = rng2.randn(B, D).astype(np.float64)
z = z_mu + eps * np.exp(0.5 * z_lv)
# Use a fixed simple decoder output to avoid recomputing model
x_recon = rng2.randn(B, C, T).astype(np.float64) * 0.5
def loss_fn_mu(mu_flat):
mu = mu_flat.reshape(B, D)
z_ = mu + eps * np.exp(0.5 * z_lv)
l, _ = btcvae_loss(x, x_recon, mu, z_lv, z_, beta_tc=2.0, alpha_mi=1.0)
return l
def loss_fn_lv(lv_flat):
lv = lv_flat.reshape(B, D)
z_ = z_mu + eps * np.exp(0.5 * lv)
l, _ = btcvae_loss(x, x_recon, z_mu, lv, z_, beta_tc=2.0, alpha_mi=1.0)
return l
_, d_z_mu, d_z_lv = btcvae_loss_backward(
x, x_recon, z_mu, z_lv, z, eps, beta_tc=2.0, alpha_mi=1.0
)
ng_mu = num_grad(loss_fn_mu, z_mu.ravel().copy()).reshape(B, D)
ng_lv = num_grad(loss_fn_lv, z_lv.ravel().copy()).reshape(B, D)
rel_mu = np.abs(d_z_mu - ng_mu) / (np.abs(ng_mu) + 1e-6)
rel_lv = np.abs(d_z_lv - ng_lv) / (np.abs(ng_lv) + 1e-6)
print(f' d_z_mu max_rel={rel_mu.max():.4f} mean_rel={rel_mu.mean():.4f}')
print(f' d_z_lv max_rel={rel_lv.max():.4f} mean_rel={rel_lv.mean():.4f}')
check('d_z_mu_rel_err', rel_mu.max() < 0.05, f'{rel_mu.max():.4f}')
check('d_z_lv_rel_err', rel_lv.max() < 0.05, f'{rel_lv.max():.4f}')
# ---------------------------------------------------------------------------
if __name__ == '__main__':
print('=' * 60)
print('ConvNeXt-1D beta-TCVAE unit tests')
print('=' * 60)
test_t1_dwconv_shape()
test_t2_dwconv_grad()
test_t3_layernorm_grad()
test_t4_block_shape()
test_t5_block_grad()
test_t6_vae_shapes()
test_t7_vae_backward()
test_t8_loss_decreases()
test_t9_save_load()
test_t10_btcvae_grad()
print()
print('=' * 60)
print(f'Results: {len(PASS)} PASS / {len(FAIL)} FAIL')
if FAIL:
print('FAILED:', FAIL)
sys.exit(1)
else:
print('All tests passed.')

86
nautilus_dolphin/dvae/test_dliq_fix_verify.py Executable file
View File

@@ -0,0 +1,86 @@
"""Verify D_LIQ_GOLD ROI=181.81% using exact gold _run_engine() path (float64, gold vol_p60)."""
import sys, time, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent)) # nautilus_dolphin root
sys.path.insert(0, str(_HERE)) # dvae/ dir — avoid dvae/__init__ PyTorch load
from exp_shared import ENGINE_KWARGS, MC_BASE_CFG, load_data, load_forewarner
from nautilus_dolphin.nautilus.proxy_boost_engine import create_d_liq_engine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
def _run_gold(eng, d, fw):
"""Exact gold _run_engine() path: float64 pq_data, gold vol_p60, np.isfinite gate."""
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
eng.set_esoteric_hazard_multiplier(0.0)
print(f" base_max_leverage = {eng.base_max_leverage} (expect 8.0 — fix check)")
print(f" abs_max_leverage = {eng.abs_max_leverage}")
print(f" sizer.max_leverage= {eng.bet_sizer.max_leverage}")
assert eng.base_max_leverage == 8.0, f"FIX NOT APPLIED: base_max={eng.base_max_leverage}"
t0 = time.time()
daily_caps = []
for pf_file in d['parquet_files']:
ds = pf_file.stem
df, acols, dvol = d['pq_data'][ds] # float64, pre-computed dvol
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
peak, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak = max(peak, cap)
max_dd = max(max_dd, (peak - cap) / peak * 100.0)
def _abs(t):
return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
liq_stops = getattr(eng, 'liquidation_stops', 0)
print(f" ROI={roi:+.2f}% T={n} DD={max_dd:.2f}% PF={pf:.4f} "
f"liq_stops={liq_stops} ({time.time()-t0:.0f}s)")
return roi, n, max_dd, pf
def main():
print("Loading data (gold method: float64, 2-file seg-based vol_p60)...")
d = load_data()
fw = load_forewarner()
print()
print("=== D_LIQ_GOLD with fixed set_esoteric_hazard_multiplier ===")
eng = create_d_liq_engine(**ENGINE_KWARGS)
roi, n, dd, pf = _run_gold(eng, d, fw)
print()
print(f"GOLD TARGET: ROI=181.81% T=2155 DD=17.65%")
roi_ok = abs(roi - 181.81) < 1.0
dd_ok = abs(dd - 17.65) < 0.5
t_ok = n == 2155
print(f"ROI match: {'PASS' if roi_ok else 'FAIL'} (diff={roi-181.81:+.2f}pp)")
print(f"DD match: {'PASS' if dd_ok else 'FAIL'} (diff={dd-17.65:+.2f}pp)")
print(f"T match: {'PASS' if t_ok else 'FAIL'} (got {n})")
if roi_ok and dd_ok and t_ok:
print("\n=== D_LIQ_GOLD RESTORED ===")
else:
print("\n!!! MISMATCH — investigate further !!!")
if __name__ == '__main__':
main()

84
nautilus_dolphin/dvae/test_dliq_nostomp.py Executable file
View File

@@ -0,0 +1,84 @@
"""Quick test: D_LIQ with and without set_esoteric_hazard_multiplier(0.0)
to confirm that's the root cause of 181.81% → 145.84% regression.
"""
import sys, time, math
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
_HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(_HERE.parent))
from exp_shared import ensure_jit, ENGINE_KWARGS, load_data, load_forewarner, MC_BASE_CFG
from nautilus_dolphin.nautilus.proxy_boost_engine import create_d_liq_engine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
def _run(eng, d, fw, call_esof):
acb = AdaptiveCircuitBreaker()
acb.preload_w750(d['date_strings'])
eng.set_ob_engine(d['ob_eng'])
eng.set_acb(acb)
if fw is not None:
eng.set_mc_forewarner(fw, MC_BASE_CFG)
if call_esof:
eng.set_esoteric_hazard_multiplier(0.0)
# Print leverage state BEFORE trading
print(f" base_max_leverage = {eng.base_max_leverage}")
print(f" abs_max_leverage = {eng.abs_max_leverage}")
print(f" bet_sizer.max_lev = {eng.bet_sizer.max_leverage}")
t0 = time.time()
daily_caps, daily_pnls = [], []
for pf_file in d['parquet_files']:
ds = pf_file.stem
df, acols, dvol = d['pq_data'][ds]
cap_before = eng.capital
vol_ok = np.where(np.isfinite(dvol), dvol > d['vol_p60'], False)
eng.process_day(ds, df, acols, vol_regime_ok=vol_ok)
daily_caps.append(eng.capital)
daily_pnls.append(eng.capital - cap_before)
tr = eng.trade_history
n = len(tr)
roi = (eng.capital - 25000.0) / 25000.0 * 100.0
peak, max_dd = 25000.0, 0.0
for cap in daily_caps:
peak = max(peak, cap)
max_dd = max(max_dd, (peak - cap) / peak * 100.0)
def _abs(t):
return t.pnl_absolute if hasattr(t, 'pnl_absolute') else t.pnl_pct * 250.0
wins = [t for t in tr if _abs(t) > 0]
losses = [t for t in tr if _abs(t) <= 0]
pf = sum(_abs(t) for t in wins) / max(abs(sum(_abs(t) for t in losses)), 1e-9)
lev_vals = [t.leverage for t in tr if hasattr(t, 'leverage') and t.leverage > 0]
avg_lev = float(np.mean(lev_vals)) if lev_vals else 0.0
liq_stops = getattr(eng, 'liquidation_stops', 0)
print(f" ROI={roi:+.2f}% T={n} DD={max_dd:.2f}% PF={pf:.4f} avg_lev={avg_lev:.2f}x liq_stops={liq_stops} ({time.time()-t0:.0f}s)")
return roi, n, max_dd, avg_lev
def main():
ensure_jit()
d = load_data()
fw = load_forewarner()
print("\n=== WITH set_esoteric_hazard_multiplier(0.0) ===")
eng_a = create_d_liq_engine(**ENGINE_KWARGS)
roi_a, n_a, dd_a, lev_a = _run(eng_a, d, fw, call_esof=True)
print("\n=== WITHOUT set_esoteric_hazard_multiplier(0.0) ===")
eng_b = create_d_liq_engine(**ENGINE_KWARGS)
roi_b, n_b, dd_b, lev_b = _run(eng_b, d, fw, call_esof=False)
print(f"\nGOLD TARGET: ROI=181.81%, T=2155, DD=17.65%, avg_lev=4.09x")
print(f"WITH stomp: ROI={roi_a:+.2f}% T={n_a} DD={dd_a:.2f}% avg_lev={lev_a:.2f}x")
print(f"WITHOUT stomp: ROI={roi_b:+.2f}% T={n_b} DD={dd_b:.2f}% avg_lev={lev_b:.2f}x")
if __name__ == '__main__':
main()

110
nautilus_dolphin/dvae/test_flint_hd_vae.py Executable file
View File

@@ -0,0 +1,110 @@
"""
Unit test for FlintHDVAE inverse projection decoder.
Criteria:
1. No NaN during training
2. z_var per dim > 0.01 after 20 epochs (no posterior collapse)
3. Reconstruction MSE < 1.0 on held-out 20%
"""
import sys, os
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict")
import numpy as np
from pathlib import Path
HERE = Path(__file__).parent
print("=" * 65)
print("UNIT TEST: FlintHDVAE Inverse Projection Decoder")
print("=" * 65)
# ── Load T1 corpus ──────────────────────────────────────────────
print("\nLoading 16K eigen corpus...")
from corpus_builder import DolphinCorpus, OFF, T1 as T1_DIM
corpus = DolphinCorpus.load(str(HERE / 'corpus_cache.npz'))
idx_mask = corpus.mask[:, 1]
X_e = corpus.X[idx_mask]
T1 = X_e[:, OFF[1]:OFF[1] + T1_DIM].copy() # (16607, 20)
N = len(T1)
print(f" T1 shape: {T1.shape} dtype: {T1.dtype}")
print(f" Any NaN in T1: {np.isnan(T1).any()}")
print(f" T1 stats: min={T1.min():.4f} max={T1.max():.4f} std={T1.std():.4f}")
# ── Train/val split (random 80/20 — avoids regime distribution shift) ────
rng_split = np.random.RandomState(42)
idx_all = rng_split.permutation(N)
n_train = int(N * 0.8)
X_train = T1[idx_all[:n_train]]
X_val = T1[idx_all[n_train:]]
print(f"\nTrain: {len(X_train)} Val: {len(X_val)}")
# ── Instantiate model ────────────────────────────────────────────
from flint_hd_vae import FlintHDVAE
print("\nInstantiating FlintHDVAE(beta=0.1, use_flint_norm=False — plain z-score for decoder test)...")
model = FlintHDVAE(input_dim=20, hd_dim=512, latent_dim=8, beta=0.1, seed=42,
use_flint_norm=False)
print(" OK")
# ── Train 40 epochs ──────────────────────────────────────────────
print("\nTraining 40 epochs on 80% T1 data (warmup first 30%)...")
PASS = True
try:
model.fit(X_train, epochs=40, lr=1e-3, batch_size=256, verbose=True, warmup_frac=0.3)
except Exception as ex:
import traceback
traceback.print_exc()
PASS = False
if PASS:
# Check 1: No NaN in losses
losses_arr = np.array(model.train_losses)
nan_losses = np.isnan(losses_arr).sum()
print(f"\n[CHECK 1] NaN in training losses: {nan_losses}")
if nan_losses == 0:
print(" PASS: No NaN detected")
else:
print(" FAIL: NaN losses detected!")
PASS = False
# Check 2: z_var per dim > 0.01 (use encode() which applies global MCDAIN)
mu_s = model.encode(X_train[:1000]) # (1000, 8)
var_per_dim = mu_s.var(axis=0) # (8,)
min_var = var_per_dim.min()
active_dims = (var_per_dim > 0.01).sum()
print(f"\n[CHECK 2] z_var per dim: {var_per_dim.round(4)}")
print(f" Min var: {min_var:.6f} Active dims (>0.01): {active_dims}/8")
if active_dims >= 4: # at least half active
print(f" PASS: {active_dims}/8 dims active (no posterior collapse)")
elif active_dims >= 1:
print(f" PARTIAL: Only {active_dims}/8 dims active — weak but not fully collapsed")
else:
print(f" FAIL: All dims collapsed (posterior collapse)")
PASS = False
# Check 3: Reconstruction MSE < 1.0 on val
# reconstruct() returns (T1_hat, X_val_norm) — both in same normalised space
T1_hat, X_val_norm = model.reconstruct(X_val)
recon_mse = float(np.mean((T1_hat - X_val_norm) ** 2))
print(f"\n[CHECK 3] Val reconstruction MSE: {recon_mse:.4f}")
if recon_mse < 1.0:
print(f" PASS: MSE={recon_mse:.4f} < 1.0")
else:
print(f" FAIL: MSE={recon_mse:.4f} >= 1.0 (decoder not learning)")
PASS = False
# Bonus: check encode output shape and values
z_all = model.encode(X_val[:100])
print(f"\n[BONUS] encode() output shape: {z_all.shape}")
print(f" z range: [{z_all.min():.4f}, {z_all.max():.4f}] std: {z_all.std():.4f}")
if np.isnan(z_all).any():
print(" WARNING: NaN in encode() output!")
print("\n" + "=" * 65)
if PASS:
print("OVERALL: PASS — FlintHDVAE inverse projection decoder functional")
else:
print("OVERALL: FAIL — see issues above")
print("=" * 65)

394
nautilus_dolphin/dvae/test_lstm_weight_fix.py Executable file
View File

@@ -0,0 +1,394 @@
"""
Unit tests: LSTM weight save/load fix
======================================
Tests that DisentangledVAEGenerator.save_model() correctly persists W_ih/W_hh/b_h
and that TitanSensor loads them instead of random re-initialising.
Run BEFORE and AFTER retrain to catch regressions.
Usage:
cd nautilus_dolphin
python dvae/test_lstm_weight_fix.py
"""
import sys, json, os, tempfile
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import numpy as np
ROOT = Path(__file__).parent.parent # nautilus_dolphin/
PROJECT = ROOT.parent # project root (disentangled_vae_joint_generator.py lives here)
sys.path.insert(0, str(ROOT))
sys.path.insert(0, str(PROJECT))
MODEL_PATH = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict"
r"\dvae_regime_model_TITAN_ULTRA_250_ULTRA261_MCDAIN.json")
MODEL_PATH_GD = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict"
r"\dvae_regime_model_TITAN_ULTRA_GD.json")
_PASS = "[PASS]"
_FAIL = "[FAIL]"
# ── Helpers ───────────────────────────────────────────────────────────────────
def _make_dummy_generator():
"""
Build a minimal DisentangledVAEGenerator-like object with known numpy weights,
bypassing FLINT _init_weights(). Used to test save/load roundtrip without
requiring FLINT/arb to be present.
"""
import types, importlib
# Import the class but intercept _init_weights so it doesn't call crypto_random_arb
mod = importlib.import_module('disentangled_vae_joint_generator')
cls = mod.DisentangledVAEGenerator
obj = cls.__new__(cls)
obj.input_dim = 8
obj.hidden_dim = 4
obj.latent_dim = 2
obj.regime_dim = 2
obj.prec = 64
obj.beta = 1.0
obj.is_trained = True
obj.edain = None
obj.latent_names = {0: "A", 1: "B"}
rng = np.random.RandomState(7777)
obj.W_ih = rng.randn(8, 16).astype(np.float64) # (input_dim, hidden*4)
obj.W_hh = rng.randn(4, 16).astype(np.float64) # (hidden_dim, hidden*4)
obj.b_h = rng.randn(16).astype(np.float64)
obj.W_mu = rng.randn(4, 2).astype(np.float64)
obj.W_logvar = rng.randn(4, 2).astype(np.float64)
obj.b_mu = rng.randn(2).astype(np.float64)
obj.b_logvar = rng.randn(2).astype(np.float64)
obj.W_dec = rng.randn(2, 4).astype(np.float64)
obj.W_out = rng.randn(4, 8).astype(np.float64)
obj.b_dec = rng.randn(4).astype(np.float64)
obj.b_out = rng.randn(8).astype(np.float64)
return obj
def _load_sensor():
from dvae.titan_sensor import TitanSensor
return TitanSensor(str(MODEL_PATH))
# ── Test 1: save_model() roundtrip (no FLINT needed) ─────────────────────────
def test_save_model_includes_lstm():
print("\n[T1] save_model() includes W_ih / W_hh / b_h ...")
gen = _make_dummy_generator()
with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
tmp = f.name
try:
gen.save_model(tmp)
with open(tmp) as f:
m = json.load(f)
for key in ('W_ih', 'W_hh', 'b_h', 'W_mu', 'W_logvar', 'W_dec', 'W_out',
'b_mu', 'b_logvar', 'b_dec', 'b_out'):
assert key in m, f"{_FAIL}: key '{key}' missing from saved JSON"
W_ih_rt = np.array(m['W_ih'])
assert W_ih_rt.shape == gen.W_ih.shape, \
f"{_FAIL}: W_ih shape mismatch {W_ih_rt.shape} vs {gen.W_ih.shape}"
assert np.allclose(W_ih_rt, gen.W_ih, atol=1e-15), \
f"{_FAIL}: W_ih values differ after roundtrip (max={np.max(np.abs(W_ih_rt - gen.W_ih)):.2e})"
W_hh_rt = np.array(m['W_hh'])
assert np.allclose(W_hh_rt, gen.W_hh, atol=1e-15), \
f"{_FAIL}: W_hh values differ after roundtrip"
b_h_rt = np.array(m['b_h'])
assert np.allclose(b_h_rt, gen.b_h, atol=1e-15), \
f"{_FAIL}: b_h values differ after roundtrip"
print(f" {_PASS} W_ih {gen.W_ih.shape} roundtrip exact")
print(f" {_PASS} W_hh {gen.W_hh.shape} roundtrip exact")
print(f" {_PASS} b_h {gen.b_h.shape} roundtrip exact")
print(f" {_PASS} all 11 weight keys present")
finally:
os.unlink(tmp)
# ── Test 2: TitanSensor loads W_ih from JSON, not random seed=42 ─────────────
def test_sensor_loads_from_json_not_random():
print("\n[T2] TitanSensor loads LSTM weights from JSON (not seed=42 random) ...")
assert MODEL_PATH.exists(), f"{_FAIL}: model not found at {MODEL_PATH}"
with open(MODEL_PATH) as f:
m = json.load(f)
assert 'W_ih' in m, \
f"{_FAIL}: W_ih missing from model JSON — model was saved before the fix. Retrain first."
sensor = _load_sensor()
assert sensor.lstm_weights_valid, \
f"{_FAIL}: sensor.lstm_weights_valid=False — W_ih missing from JSON"
W_ih_json = np.array(m['W_ih'], dtype=np.float64)
max_diff = np.max(np.abs(sensor.W_ih - W_ih_json))
assert max_diff < 1e-12, \
f"{_FAIL}: sensor.W_ih != JSON W_ih (max_diff={max_diff:.3e})"
print(f" {_PASS} sensor.W_ih == JSON W_ih (max_diff={max_diff:.2e})")
W_hh_json = np.array(m['W_hh'], dtype=np.float64)
max_diff_hh = np.max(np.abs(sensor.W_hh - W_hh_json))
assert max_diff_hh < 1e-12, \
f"{_FAIL}: sensor.W_hh != JSON W_hh (max_diff={max_diff_hh:.3e})"
print(f" {_PASS} sensor.W_hh == JSON W_hh (max_diff={max_diff_hh:.2e})")
# Confirm it is NOT the seed=42 random initialisation
rng_42 = np.random.RandomState(42)
W_ih_42 = rng_42.randn(261, 512) * 0.1
diff_vs_42 = np.max(np.abs(sensor.W_ih - W_ih_42))
assert diff_vs_42 > 0.01, \
f"{_FAIL}: sensor.W_ih matches seed=42 random (diff={diff_vs_42:.3e}) — LSTM still wrong"
print(f" {_PASS} sensor.W_ih is NOT seed=42 random (diff_vs_42={diff_vs_42:.3f})")
# ── Test 3: recon_err is finite and in plausible range ────────────────────────
def test_recon_err_plausible():
print("\n[T3] recon_err is finite and << 10^6 (was ~10^14 pre-fix) ...")
sensor = _load_sensor()
from dvae.titan_sensor import build_feature_vector
rng = np.random.RandomState(42)
results = {}
for label, x in [
("zeros", np.zeros(261)),
("ones", np.ones(261) * 0.01),
("random_s", rng.randn(261) * 0.05),
("random_l", rng.randn(261) * 2.0),
]:
z_mu, recon_err, z_logvar = sensor.encode(x)
results[label] = recon_err
assert np.isfinite(recon_err), \
f"{_FAIL}: recon_err not finite for '{label}' input: {recon_err}"
# Pre-fix: ~10^14. Post-fix: should be O(1) or O(100) at worst.
assert recon_err < 1e8, \
f"{_FAIL}: recon_err={recon_err:.3e} suspiciously large for '{label}'"
print(f" {_PASS} [{label:10s}] recon_err={recon_err:.4e} z_mu[0:4]={z_mu[:4].round(4)}")
# Distribution check: recon_err should vary with input (not uniform noise)
errs = list(results.values())
assert max(errs) / (min(errs) + 1e-12) > 2.0, \
f"{_FAIL}: recon_err suspiciously uniform across inputs ({errs}) — LSTM may still be wrong"
print(f" {_PASS} recon_err varies meaningfully across inputs (ratio={max(errs)/(min(errs)+1e-12):.1f}x)")
# ── Test 4: Encoding is deterministic ─────────────────────────────────────────
def test_encoding_deterministic():
print("\n[T4] encode() is deterministic across two sensor instances ...")
sensor1 = _load_sensor()
sensor2 = _load_sensor()
x = np.random.RandomState(99).randn(261) * 0.1
z1, e1, _ = sensor1.encode(x)
z2, e2, _ = sensor2.encode(x)
assert np.allclose(z1, z2, atol=1e-14), \
f"{_FAIL}: z_mu differs between two sensors (max={np.max(np.abs(z1-z2)):.2e})"
assert abs(e1 - e2) < 1e-10, \
f"{_FAIL}: recon_err differs between two sensors ({e1:.6e} vs {e2:.6e})"
print(f" {_PASS} z_mu identical (max_diff={np.max(np.abs(z1-z2)):.2e})")
print(f" {_PASS} recon_err identical ({e1:.6e})")
# ── Test 5: Legacy model emits RuntimeWarning, sensor.lstm_weights_valid=False ─
def test_legacy_model_warns():
print("\n[T5] Legacy model (no W_ih in JSON) emits RuntimeWarning ...")
from dvae.titan_sensor import TitanSensor
import warnings
# Build a minimal legacy-style JSON (no W_ih/W_hh)
legacy = {
"W_mu": np.zeros((128, 32)).tolist(),
"W_logvar": np.zeros((128, 32)).tolist(),
"W_dec": np.zeros((32, 128)).tolist(),
"W_out": np.zeros((128, 261)).tolist(),
"latent_names": {},
"precision_bits": 512,
}
with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
json.dump(legacy, f)
tmp = f.name
try:
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
s = TitanSensor(tmp)
assert not s.lstm_weights_valid, \
f"{_FAIL}: lstm_weights_valid should be False for legacy model"
assert any(issubclass(w.category, RuntimeWarning) for w in caught), \
f"{_FAIL}: no RuntimeWarning emitted for legacy model"
print(f" {_PASS} lstm_weights_valid=False")
print(f" {_PASS} RuntimeWarning emitted")
finally:
os.unlink(tmp)
# ── Test 6: z_mu dimensionality and range ─────────────────────────────────────
def test_latent_dimensionality():
print("\n[T6] z_mu has correct dimensionality and plausible range ...")
sensor = _load_sensor()
x = np.random.RandomState(55).randn(261) * 0.1
z_mu, recon_err, z_logvar = sensor.encode(x)
assert z_mu.shape == (32,), f"{_FAIL}: z_mu shape {z_mu.shape} != (32,)"
assert z_logvar.shape == (32,), f"{_FAIL}: z_logvar shape {z_logvar.shape} != (32,)"
assert np.all(np.isfinite(z_mu)), f"{_FAIL}: z_mu contains non-finite values"
assert np.all(np.isfinite(z_logvar)), f"{_FAIL}: z_logvar contains non-finite values"
assert np.abs(z_mu).max() < 1e3, \
f"{_FAIL}: z_mu values suspiciously large (max={np.abs(z_mu).max():.2e})"
print(f" {_PASS} z_mu.shape=(32,) z_mu range=[{z_mu.min():.3f}, {z_mu.max():.3f}]")
print(f" {_PASS} z_logvar.shape=(32,) range=[{z_logvar.min():.3f}, {z_logvar.max():.3f}]")
# ── Test 7: GD-trained model has correct LSTM weights and sane recon_err ──────
def test_gd_model():
print("\n[T7] GD-trained model: W_ih present, recon_err << MCDAIN model ...")
assert MODEL_PATH_GD.exists(), f"{_FAIL}: GD model not found at {MODEL_PATH_GD}"
from dvae.titan_sensor import TitanSensor
sensor_gd = TitanSensor(str(MODEL_PATH_GD))
assert sensor_gd.lstm_weights_valid, f"{_FAIL}: GD sensor.lstm_weights_valid=False"
print(f" {_PASS} GD model: lstm_weights_valid=True")
# Verify W_ih from GD model matches W_ih from MCDAIN model
# (they should be THE SAME — GD model was initialized from MCDAIN model's W_ih)
sensor_mc = _load_sensor()
diff_wih = np.max(np.abs(sensor_gd.W_ih - sensor_mc.W_ih))
assert diff_wih < 1e-12, \
f"{_FAIL}: GD model W_ih != MCDAIN model W_ih (diff={diff_wih:.3e}) — should be same LSTM basis"
print(f" {_PASS} GD model W_ih == MCDAIN model W_ih (same LSTM basis, diff={diff_wih:.2e})")
# Reconstruction error: generate inputs GUARANTEED to be in-distribution by
# sampling x = norm_mean + norm_std * randn (within ±2σ of training corpus).
# After encode()'s normalization step: v_norm = clip(randn, -2, 2) → perfectly O(1).
# recon_err should be close to the training-set value (p50=0.59).
with open(MODEL_PATH_GD) as fj:
gd_json = json.load(fj)
nm_gd = np.array(gd_json['norm_mean']); ns_gd = np.array(gd_json['norm_std'])
rng = np.random.RandomState(42)
errs_gd = []; errs_mc = []
for _ in range(20):
# Guaranteed in-distribution: raw input = corpus_mean + corpus_std * noise
x_raw = nm_gd + ns_gd * np.clip(rng.randn(261), -2, 2)
x_raw[78:] = 0.0 # T3/T4/T5 are always 0 in corpus
_, e_gd, _ = sensor_gd.encode(x_raw)
_, e_mc, _ = sensor_mc.encode(x_raw)
errs_gd.append(e_gd); errs_mc.append(e_mc)
med_gd = float(np.median(errs_gd)); med_mc = float(np.median(errs_mc))
print(f" GD median recon_err={med_gd:.4e} (in-distribution) MCDAIN median recon_err={med_mc:.4e}")
# GD trained with proper GD — should reconstruct in-distribution inputs well
assert med_gd < 10.0, f"{_FAIL}: GD recon_err too large ({med_gd:.4e}) — model didn't learn"
print(f" {_PASS} GD recon_err < 10.0 for in-distribution inputs (model learned)")
# ── Test 8: GD-v2 normalization stored and applied at inference ───────────────
def test_normalization_stored_and_applied():
print("\n[T8] GD-v2 model: norm_mean/norm_std present and applied by TitanSensor ...")
assert MODEL_PATH_GD.exists(), f"{_FAIL}: GD model not found at {MODEL_PATH_GD}"
from dvae.titan_sensor import TitanSensor
# 8a: JSON must contain norm_mean / norm_std
with open(MODEL_PATH_GD) as f:
m = json.load(f)
assert 'norm_mean' in m, f"{_FAIL}: GD model JSON missing 'norm_mean'"
assert 'norm_std' in m, f"{_FAIL}: GD model JSON missing 'norm_std'"
nm = np.array(m['norm_mean']); ns = np.array(m['norm_std'])
assert nm.shape == (261,), f"{_FAIL}: norm_mean shape {nm.shape} != (261,)"
assert ns.shape == (261,), f"{_FAIL}: norm_std shape {ns.shape} != (261,)"
assert np.all(ns > 0), f"{_FAIL}: norm_std has zero or negative entries"
print(f" {_PASS} norm_mean/norm_std present, shape=(261,), all std>0")
# 8b: TitanSensor must load them (not None)
sensor = TitanSensor(str(MODEL_PATH_GD))
assert sensor.norm_mean is not None, f"{_FAIL}: sensor.norm_mean is None — not loaded"
assert sensor.norm_std is not None, f"{_FAIL}: sensor.norm_std is None — not loaded"
assert np.allclose(sensor.norm_mean, nm, atol=1e-12), \
f"{_FAIL}: sensor.norm_mean != JSON norm_mean"
print(f" {_PASS} sensor.norm_mean loaded from JSON")
# 8c: recon_err on realistic inputs (matching build_feature_vector() output structure)
# Before fix (MCDAIN-trained, no normalization stored): encode(raw) → huge recon_err
# After fix (GD-v2, normalization stored and applied): encode(raw) → O(1-50)
rng = np.random.RandomState(42)
errs = []
for _ in range(30):
x = np.zeros(261)
x[0:8] = rng.randn(8) * 0.7 # T0: time encoding
x[8:28] = rng.randn(20) * 0.02 # T1: eigenvalue velocity scale
x[28:78] = np.clip(rng.randn(50), -5, 5) # T2: return z-scores [-5,5]
# T3/T4/T5 = 0 (Stage 1 contract — matches training corpus)
_, e, _ = sensor.encode(x)
errs.append(e)
med_err = float(np.median(errs))
max_err = float(np.max(errs))
print(f" GD-v2 realistic-input recon_err: median={med_err:.4e} max={max_err:.4e}")
assert med_err < 100.0, \
f"{_FAIL}: recon_err too large ({med_err:.4e}) — normalization may not be applied"
print(f" {_PASS} recon_err O(1-100) for realistic-scale inputs")
# 8d: T5 dims are zero after normalization (non-zero norm_mean would shift them)
x_zeros = np.zeros(261)
_, e_z, _ = sensor.encode(x_zeros)
# The sensor zeros T5 after normalization — this just checks it doesn't crash
assert np.isfinite(e_z), f"{_FAIL}: encode(zeros) returned non-finite recon_err={e_z}"
print(f" {_PASS} encode(zeros) is finite after normalization: recon_err={e_z:.4e}")
# ── Runner ────────────────────────────────────────────────────────────────────
if __name__ == '__main__':
print("=" * 60)
print("TitanSensor LSTM weight fix — unit tests")
print("=" * 60)
n_pass = 0
n_fail = 0
tests = [
("T1: save_model roundtrip", test_save_model_includes_lstm),
("T2: sensor loads JSON weights", test_sensor_loads_from_json_not_random),
("T3: recon_err plausible", test_recon_err_plausible),
("T4: encoding deterministic", test_encoding_deterministic),
("T5: legacy model warns", test_legacy_model_warns),
("T6: latent dimensionality", test_latent_dimensionality),
("T7: GD model quality", test_gd_model),
("T8: normalization stored and applied", test_normalization_stored_and_applied),
]
for name, fn in tests:
try:
fn()
n_pass += 1
except AssertionError as e:
print(f" {_FAIL} ASSERTION: {e}")
n_fail += 1
except Exception as e:
print(f" {_FAIL} EXCEPTION in {name}: {type(e).__name__}: {e}")
n_fail += 1
print()
print("=" * 60)
print(f"Results: {n_pass}/{n_pass+n_fail} PASSED {n_fail} FAILED")
print("=" * 60)
if n_fail > 0:
sys.exit(1)

233
nautilus_dolphin/dvae/titan_sensor.py Executable file
View File

@@ -0,0 +1,233 @@
"""
TitanSensor — Non-invasive latent state extractor (Stage 1, read-only).
Architecture notes
------------------
The LSTM weights (W_ih, W_hh) are fixed random projections used as the
encoder basis during training. W_mu/W_logvar/W_dec/W_out were trained to
map the output of THESE specific W_ih/W_hh matrices. Loading the correct
W_ih/W_hh is therefore mandatory for valid reconstruction errors.
Models saved before 2026-03-15 did not persist W_ih/W_hh — those fall back
to legacy seed=42 re-init (meaningless recon_err ~10^14) and are flagged.
T5 oracle-leakage fix
---------------------
Training included T5 dims [111:261] = spectral path coefficients derived from
FUTURE prices [t+1, t+51]. At inference we zero those dims unconditionally.
Normalisation (GD-v2 models, training >= 2026-03-15)
------------------------------------------------------
GD-v2 models store per-feature z-score stats (norm_mean, norm_std) computed
from the raw training corpus. encode() applies v = (v - norm_mean)/norm_std
before the LSTM step so that training and inference see the same distribution.
Models without norm_mean (legacy) skip this step — recon_err is in raw-feature
space and remains valid for relative comparison within that model.
"""
import json
import numpy as np
class TitanSensor:
INPUT_DIM = 261
HIDDEN_DIM = 128
LATENT_DIM = 32
T5_START = 111 # dims [111:261] = spectral (oracle) → always zero
def __init__(self, model_path: str, lstm_seed: int = 42):
with open(model_path) as f:
m = json.load(f)
def _arr(key, default_shape):
return np.array(m[key]) if key in m else np.zeros(default_shape)
self.W_mu = _arr('W_mu', (self.HIDDEN_DIM, self.LATENT_DIM))
self.W_logvar = _arr('W_logvar', (self.HIDDEN_DIM, self.LATENT_DIM))
self.W_dec = _arr('W_dec', (self.LATENT_DIM, self.HIDDEN_DIM))
self.W_out = _arr('W_out', (self.HIDDEN_DIM, self.INPUT_DIM))
self.b_mu = _arr('b_mu', (self.LATENT_DIM,))
self.b_logvar = _arr('b_logvar', (self.LATENT_DIM,))
self.b_dec = _arr('b_dec', (self.HIDDEN_DIM,))
self.b_out = _arr('b_out', (self.INPUT_DIM,))
# LSTM weights: load from JSON if present (fixed since 2026-03-15).
# Legacy models (no W_ih in JSON) fall back to seed=42 re-init —
# recon_err will be ~10^14 for those; a warning is emitted.
if 'W_ih' in m:
self.W_ih = np.array(m['W_ih'], dtype=np.float64)
self.W_hh = np.array(m['W_hh'], dtype=np.float64)
self.b_h = np.array(m['b_h'], dtype=np.float64)
self.lstm_weights_valid = True
else:
import warnings
warnings.warn(
f"TitanSensor: model at '{model_path}' does not contain LSTM weights "
"(W_ih/W_hh/b_h). Falling back to seed=42 re-init. "
"recon_err will be ~10^14 (meaningless). Retrain with fixed save_model().",
RuntimeWarning, stacklevel=2,
)
rng = np.random.RandomState(lstm_seed)
s = 0.1
self.W_ih = rng.randn(self.INPUT_DIM, self.HIDDEN_DIM * 4) * s
self.W_hh = rng.randn(self.HIDDEN_DIM, self.HIDDEN_DIM * 4) * s
self.b_h = np.zeros(self.HIDDEN_DIM * 4)
self.lstm_weights_valid = False
self.latent_names = {int(k): v for k, v in m.get('latent_names', {}).items()}
# Per-feature normalisation stats (present in GD-v2 models trained 2026-03-15+).
# If absent (legacy models): no normalization applied — recon_err will be in
# raw-feature space and is still valid for relative comparison within that model.
if 'norm_mean' in m:
self.norm_mean = np.array(m['norm_mean'], dtype=np.float64)
self.norm_std = np.array(m['norm_std'], dtype=np.float64)
else:
self.norm_mean = None
self.norm_std = None
# ------------------------------------------------------------------
def _sigmoid(self, x):
return 1.0 / (1.0 + np.exp(-np.clip(x, -500, 500)))
def _lstm_step(self, x, h, c):
g = x @ self.W_ih + h @ self.W_hh + self.b_h
i_, f_, g_, o_ = np.split(g, 4, axis=-1)
i_ = self._sigmoid(i_); f_ = self._sigmoid(f_); o_ = self._sigmoid(o_)
g_ = np.tanh(g_)
c2 = f_ * c + i_ * g_
h2 = o_ * np.tanh(c2)
return h2, c2
# ------------------------------------------------------------------
def encode(self, x: np.ndarray):
"""
Encode one 261-dim feature row.
Returns
-------
z_mu : np.ndarray (32,) — latent mean
recon_err : float — MSE on T0-T4 reconstruction (in normalised space)
z_logvar : np.ndarray (32,) — per-dim log-variance
"""
v = np.array(x, dtype=np.float64).ravel()
v = np.resize(v, self.INPUT_DIM) # pad/truncate
# Stage-1 zero: T3 (ExF, 78-102), T4 (esoteric, 103-110) and T5 (oracle, 111+)
# are always zero in the training corpus — enforce this here so that
# callers not using build_feature_vector() don't blow up normalization.
v[78:] = 0.0
v[self.T5_START:] = 0.0 # T5 oracle fix (redundant but explicit)
v = np.nan_to_num(v, nan=0.0, posinf=0.0, neginf=0.0)
# Apply stored per-feature normalisation (GD-v2 models only).
# This reproduces the same transform applied during training so that
# the LSTM sees the same input distribution and recon_err is O(1).
if self.norm_mean is not None:
v = (v - self.norm_mean) / self.norm_std
v[78:] = 0.0 # re-zero Stage-1 dims after norm
# Clip to [-50, 50] for safety — training post-norm max was ~37
v = np.clip(v, -50.0, 50.0)
h = np.zeros((1, self.HIDDEN_DIM))
c = np.zeros((1, self.HIDDEN_DIM))
h, c = self._lstm_step(v.reshape(1, -1), h, c)
z_mu = (h @ self.W_mu + self.b_mu)[0]
z_logvar = (h @ self.W_logvar + self.b_logvar)[0]
h_dec = np.tanh(z_mu @ self.W_dec + self.b_dec)
recon = h_dec @ self.W_out + self.b_out
recon_err = float(np.mean((recon[:self.T5_START] - v[:self.T5_START]) ** 2))
return z_mu, recon_err, z_logvar
# ------------------------------------------------------------------
# Feature builder — constructs a 261-dim vector from a parquet row.
# T3 (ExF, 78-102), T4 (esoteric, 103-110), T5 (111-260) are zeroed.
# ------------------------------------------------------------------
_ASSET_CACHE = {} # parquet-path → ordered asset list
def build_feature_vector(df, ri: int, assets: list) -> np.ndarray:
"""
Build 261-dim T0-T4 feature vector (T5 always zero).
T0 (0-7) : time encoding + rolling breadth
T1 (8-27) : eigenvalue velocity features (4 windows × 5 dims)
T2 (28-77) : per-asset return z-scores (up to 50 assets, rolling 50 bars)
T3 (78-102): ExF macro — zeroed (Stage 1)
T4 (103-110): esoteric — zeroed (Stage 1)
T5 (111-260): spectral — zeroed (oracle fix)
"""
x = np.zeros(261, dtype=np.float64)
row = df.iloc[ri]
# --- T0: time encoding ---
try:
ts = row['timestamp']
if hasattr(ts, 'hour'):
h = ts.hour + ts.minute / 60.0
d = ts.dayofweek
else:
import pandas as pd
ts = pd.Timestamp(ts)
h = ts.hour + ts.minute / 60.0
d = ts.dayofweek
x[0] = np.sin(2 * np.pi * h / 24)
x[1] = np.cos(2 * np.pi * h / 24)
x[2] = np.sin(2 * np.pi * d / 7)
x[3] = np.cos(2 * np.pi * d / 7)
except Exception:
pass
x[4] = 1.0 # has_eigen always true for NG3
# rolling BTC breadth (T0 dims 5-7)
if ri >= 20 and 'BTCUSDT' in df.columns:
diffs = np.diff(df['BTCUSDT'].values[ri - 20:ri + 1])
x[5] = float(np.mean(diffs > 0))
x[6] = float(np.mean(diffs < 0))
x[7] = float(np.mean(diffs == 0))
# --- T1: eigenvalue velocity features ---
def _g(col):
v = row.get(col, 0.0)
return float(v) if v is not None and v == v else 0.0
v50 = _g('v50_lambda_max_velocity')
v150 = _g('v150_lambda_max_velocity')
v300 = _g('v300_lambda_max_velocity')
v750 = _g('v750_lambda_max_velocity')
i50 = _g('instability_50')
i150 = _g('instability_150')
vd = _g('vel_div')
# window-0 (8-12): v50 group
x[8] = v50; x[9] = i50; x[10] = v50 - v150
x[11] = v50 - v300; x[12] = v50 - v750
# window-1 (13-17): v150 group
x[13] = v150; x[14] = i150; x[15] = v150 - v300
x[16] = v150 - v750; x[17] = abs(v150)
# window-2 (18-22): v300 group
x[18] = v300; x[19] = abs(v300); x[20] = v300 - v750
x[21] = v50 * v300; x[22] = i50 - v750 # proxy_B equivalent
# window-3 (23-27): v750 + composite
x[23] = v750; x[24] = abs(v750)
x[25] = v50 / (abs(v750) + 1e-8)
x[26] = i50 - i150; x[27] = vd
# --- T2: per-asset return z-scores ---
if ri >= 50:
prices_cache = {a: df[a].values for a in assets if a in df.columns}
for j, asset in enumerate(assets[:50]):
if asset not in prices_cache:
continue
seg = prices_cache[asset][ri - 50:ri + 1]
if np.any(seg <= 0) or np.any(~np.isfinite(seg)):
continue
rets = np.diff(seg) / seg[:-1]
std = np.std(rets)
if std > 0:
x[28 + j] = float(np.clip((rets[-1] - np.mean(rets)) / std, -5.0, 5.0))
return x

319
nautilus_dolphin/dvae/titan_stage1_run.py Executable file
View File

@@ -0,0 +1,319 @@
"""
Titan Stage 1 — Silent Sensor Run
==================================
Two-pass approach:
Pass 1: run D_LIQ_GOLD engine (56 days) → assert T=2155 unchanged.
Pass 2: re-scan each parquet, build 261-dim feature vectors, run TitanSensor,
annotate each bar with trade state from pass-1 trade history.
Output: dvae/titan_latent_trail.parquet (all bars × z_mu[0:32] + meta columns)
Zero changes to any production code. Imports from production modules are read-only.
"""
import sys
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
import time
import json
from pathlib import Path
import numpy as np
import pandas as pd
ROOT = Path(__file__).parent.parent
sys.path.insert(0, str(ROOT))
# ── production imports (read-only) ──────────────────────────────────────────
from nautilus_dolphin.nautilus.alpha_asset_selector import compute_irp_nb, compute_ars_nb, rank_assets_irp_nb
from nautilus_dolphin.nautilus.alpha_bet_sizer import compute_sizing_nb
from nautilus_dolphin.nautilus.alpha_signal_generator import check_dc_nb
from nautilus_dolphin.nautilus.ob_features import (
OBFeatureEngine, compute_imbalance_nb, compute_depth_1pct_nb,
compute_depth_quality_nb, compute_fill_probability_nb, compute_spread_proxy_nb,
compute_depth_asymmetry_nb, compute_imbalance_persistence_nb,
compute_withdrawal_velocity_nb, compute_market_agreement_nb, compute_cascade_signal_nb,
)
from nautilus_dolphin.nautilus.ob_provider import MockOBProvider
from nautilus_dolphin.nautilus.esf_alpha_orchestrator import NDAlphaEngine
from nautilus_dolphin.nautilus.adaptive_circuit_breaker import AdaptiveCircuitBreaker
from nautilus_dolphin.nautilus.proxy_boost_engine import create_d_liq_engine
from mc.mc_ml import DolphinForewarner
from dvae.titan_sensor import TitanSensor, build_feature_vector
# ── JIT warmup ───────────────────────────────────────────────────────────────
print("Warming up JIT kernels...")
_p = np.array([1.0, 2.0, 3.0], dtype=np.float64)
compute_irp_nb(_p, -1); compute_ars_nb(1.0, 0.5, 0.01)
rank_assets_irp_nb(np.ones((10, 2), dtype=np.float64), 8, -1, 5, 500.0, 20, 0.20)
compute_sizing_nb(-0.03, -0.02, -0.05, 3.0, 0.5, 5.0, 0.20, True, True, 0.0,
np.zeros(4, dtype=np.int64), np.zeros(4, dtype=np.int64),
np.zeros(5, dtype=np.float64), 0, -1, 0.01, 0.04)
check_dc_nb(_p, 3, 1, 0.75)
_b = np.array([100.0, 200.0, 300.0, 400.0, 500.0], dtype=np.float64)
_a = np.array([110.0, 190.0, 310.0, 390.0, 510.0], dtype=np.float64)
compute_imbalance_nb(_b, _a); compute_depth_1pct_nb(_b, _a)
compute_depth_quality_nb(210.0, 200.0); compute_fill_probability_nb(1.0)
compute_spread_proxy_nb(_b, _a); compute_depth_asymmetry_nb(_b, _a)
compute_imbalance_persistence_nb(np.array([0.1, -0.1], dtype=np.float64), 2)
compute_withdrawal_velocity_nb(np.array([100.0, 110.0], dtype=np.float64), 1)
compute_market_agreement_nb(np.array([0.1, -0.05], dtype=np.float64), 2)
compute_cascade_signal_nb(np.array([-0.05, -0.15], dtype=np.float64), 2, -0.10)
print(" JIT ready.")
# ── paths / config ────────────────────────────────────────────────────────────
VBT_DIR = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\vbt_cache")
MODEL_PATH = Path(r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict\dvae_regime_model_TITAN_ULTRA_250_ULTRA261_MCDAIN.json")
MC_MODELS = str(ROOT / "mc_results" / "models")
OUT_FILE = Path(__file__).parent / "titan_latent_trail.parquet"
META_COLS = {'timestamp', 'scan_number', 'v50_lambda_max_velocity', 'v150_lambda_max_velocity',
'v300_lambda_max_velocity', 'v750_lambda_max_velocity', 'vel_div',
'instability_50', 'instability_150'}
ENGINE_KWARGS = dict(
initial_capital=25000.0, vel_div_threshold=-0.02, vel_div_extreme=-0.05,
min_leverage=0.5, max_leverage=5.0, leverage_convexity=3.0,
fraction=0.20, fixed_tp_pct=0.0095, stop_pct=1.0, max_hold_bars=120,
use_direction_confirm=True, dc_lookback_bars=7, dc_min_magnitude_bps=0.75,
dc_skip_contradicts=True, dc_leverage_boost=1.0, dc_leverage_reduce=0.5,
use_asset_selection=True, min_irp_alignment=0.45,
use_sp_fees=True, use_sp_slippage=True,
sp_maker_entry_rate=0.62, sp_maker_exit_rate=0.50,
use_ob_edge=True, ob_edge_bps=5.0, ob_confirm_rate=0.40,
lookback=100, use_alpha_layers=True, use_dynamic_leverage=True, seed=42,
)
MC_BASE_CFG = {
'trial_id': 0, 'vel_div_threshold': -0.020, 'vel_div_extreme': -0.050,
'use_direction_confirm': True, 'dc_lookback_bars': 7,
'dc_min_magnitude_bps': 0.75, 'dc_skip_contradicts': True,
'dc_leverage_boost': 1.00, 'dc_leverage_reduce': 0.50,
'vd_trend_lookback': 10, 'min_leverage': 0.50, 'max_leverage': 5.00,
'leverage_convexity': 3.00, 'fraction': 0.20,
'use_alpha_layers': True, 'use_dynamic_leverage': True,
'fixed_tp_pct': 0.0095, 'stop_pct': 1.00, 'max_hold_bars': 120,
'use_sp_fees': True, 'use_sp_slippage': True,
'sp_maker_entry_rate': 0.62, 'sp_maker_exit_rate': 0.50,
'use_ob_edge': True, 'ob_edge_bps': 5.00, 'ob_confirm_rate': 0.40,
'ob_imbalance_bias': -0.09, 'ob_depth_scale': 1.00,
'use_asset_selection': True, 'min_irp_alignment': 0.45, 'lookback': 100,
'acb_beta_high': 0.80, 'acb_beta_low': 0.20, 'acb_w750_threshold_pct': 60,
}
EXPECTED_TRADES = 2155
# ═══════════════════════════════════════════════════════════════════════════════
# PASS 1 — run D_LIQ_GOLD engine, collect trade timeline + bar log
# ═══════════════════════════════════════════════════════════════════════════════
def run_pass1(parquet_files, pq_data, all_assets):
print("\n=== PASS 1: D_LIQ_GOLD backtest ===")
# OB
OB_ASSETS = sorted(all_assets)
_mock_ob = MockOBProvider(
imbalance_bias=-0.09, depth_scale=1.0, assets=OB_ASSETS,
imbalance_biases={"BTCUSDT": -0.086, "ETHUSDT": -0.092,
"BNBUSDT": +0.05, "SOLUSDT": +0.05},
)
ob_eng = OBFeatureEngine(_mock_ob)
ob_eng.preload_date("mock", OB_ASSETS)
forewarner = DolphinForewarner(models_dir=MC_MODELS)
acb = AdaptiveCircuitBreaker()
acb.preload_w750([pf.stem for pf in parquet_files])
engine = create_d_liq_engine(**ENGINE_KWARGS)
engine.set_ob_engine(ob_eng)
engine.set_acb(acb)
engine.set_mc_forewarner(forewarner, MC_BASE_CFG)
engine.set_esoteric_hazard_multiplier(0.0)
engine._bar_log_enabled = True
# vol threshold (p60, first 2 days)
all_vols = []
for pf in parquet_files[:2]:
df = pd.read_parquet(pf)
if 'BTCUSDT' not in df.columns:
continue
pr = df['BTCUSDT'].values
for i in range(60, len(pr)):
seg = pr[max(0, i-50):i]
if len(seg) >= 10:
v = float(np.std(np.diff(seg) / seg[:-1]))
if v > 0:
all_vols.append(v)
vol_p60 = float(np.percentile(all_vols, 60)) if all_vols else 0.0
bar_logs = {} # date → list[dict]
t0 = time.time()
for pf in parquet_files:
ds = pf.stem
df, acols, dvol = pq_data[ds]
vol_ok = np.where(np.isfinite(dvol), dvol > vol_p60, False)
engine.process_day(ds, df, acols, vol_regime_ok=vol_ok)
bar_logs[ds] = list(engine._bar_log)
elapsed = time.time() - t0
trades = engine.trade_history
roi = (engine.capital - 25000) / 25000 * 100
print(f" Trades: {len(trades)} ROI: {roi:+.2f}% ({elapsed:.0f}s)")
assert len(trades) == EXPECTED_TRADES, \
f"REGRESSION: T={len(trades)} ≠ {EXPECTED_TRADES}. Aborting."
print(f" T={len(trades)} ✓ (matches gold standard)")
return trades, bar_logs, vol_p60
# ═══════════════════════════════════════════════════════════════════════════════
# PASS 2 — build feature vectors + run sensor, annotate with trade state
# ═══════════════════════════════════════════════════════════════════════════════
def run_pass2(parquet_files, pq_data, trades, bar_logs, sensor):
print("\n=== PASS 2: TitanSensor latent capture ===")
# Build trade timeline: global_bar_idx → trade metadata
# Each trade record has entry_bar / exit_bar (global_bar_idx values)
active_at = {} # global_bar_idx → {'trade_id', 'asset', 'pnl_pct', 'bars_held', 'outcome'}
for t in trades:
eb, xb = getattr(t, 'entry_bar', None), getattr(t, 'exit_bar', None)
if eb is None or xb is None:
continue
final_pnl = getattr(t, 'pnl_pct', 0.0)
outcome = 'win' if getattr(t, 'pnl_absolute', 0.0) > 0 else 'loss'
for b in range(int(eb), int(xb) + 1):
active_at[b] = {
'trade_id': getattr(t, 'trade_id', ''),
'asset': getattr(t, 'asset', ''),
'trade_outcome': outcome,
'trade_pnl_pct': float(final_pnl),
'bars_in_trade': b - int(eb),
'exit_reason': getattr(t, 'exit_reason', ''),
}
records = []
global_bar = 0
for pf in parquet_files:
ds = pf.stem
df, acols, _ = pq_data[ds]
assets = acols
print(f" {ds} ({len(df)} bars) ...", end=' ', flush=True)
t0 = time.time()
for ri in range(len(df)):
row = df.iloc[ri]
# build feature vector
feat = build_feature_vector(df, ri, assets)
z_mu, recon_err, z_logvar = sensor.encode(feat)
trade_info = active_at.get(global_bar, {})
rec = {
'date': ds,
'bar_idx': global_bar,
'row_idx': ri,
'vel_div': float(row.get('vel_div', 0.0) or 0.0),
'v50': float(row.get('v50_lambda_max_velocity', 0.0) or 0.0),
'v750': float(row.get('v750_lambda_max_velocity', 0.0) or 0.0),
'instab_50': float(row.get('instability_50', 0.0) or 0.0),
'recon_err': recon_err,
'is_in_trade': bool(trade_info),
**{f'z{i}': float(z_mu[i]) for i in range(32)},
**{f'lv{i}': float(z_logvar[i]) for i in range(32)},
**trade_info,
}
records.append(rec)
global_bar += 1
print(f"{time.time() - t0:.1f}s")
return pd.DataFrame(records)
# ═══════════════════════════════════════════════════════════════════════════════
# SUMMARY — print key correlations for immediate insight
# ═══════════════════════════════════════════════════════════════════════════════
def print_summary(df_out):
print("\n=== Stage 1 Summary ===")
trade_rows = df_out[df_out['is_in_trade']].copy()
print(f" Total bars: {len(df_out)} | In-trade bars: {len(trade_rows)}")
if trade_rows.empty:
print(" (no in-trade data)")
return
# recon_err: in-trade vs not
oob = df_out[~df_out['is_in_trade']]['recon_err']
print(f" recon_err — in-trade: {trade_rows['recon_err'].mean():.4f} "
f"out-of-trade: {oob.mean():.4f}")
# per z-dim correlation with recon_err
z_cols = [f'z{i}' for i in range(32)]
corrs = trade_rows[z_cols + ['recon_err']].corr()['recon_err'].drop('recon_err')
top3 = corrs.abs().nlargest(3)
print(f" Top z-dims by |corr| with recon_err: {top3.index.tolist()}")
for c in top3.index:
print(f" {c}: r={corrs[c]:+.3f} (name: {sensor.latent_names.get(int(c[1:]), '?')})")
# win vs loss recon_err (entry bar only)
entry_rows = trade_rows[trade_rows['bars_in_trade'] == 0] if 'bars_in_trade' in trade_rows.columns else trade_rows
if not entry_rows.empty and 'trade_outcome' in entry_rows.columns:
wins = entry_rows[entry_rows['trade_outcome'] == 'win']['recon_err'].mean()
losses= entry_rows[entry_rows['trade_outcome'] == 'loss']['recon_err'].mean()
print(f" recon_err at entry — wins: {wins:.4f} losses: {losses:.4f}")
# ═══════════════════════════════════════════════════════════════════════════════
# MAIN
# ═══════════════════════════════════════════════════════════════════════════════
def main():
assert MODEL_PATH.exists(), f"Model not found: {MODEL_PATH}"
parquet_files = sorted(VBT_DIR.glob("*.parquet"))
parquet_files = [p for p in parquet_files if 'catalog' not in str(p)]
print(f"Dataset: {len(parquet_files)} days ({parquet_files[0].stem} to {parquet_files[-1].stem})")
# preload parquet
print("Loading parquet data...")
pq_data = {}
all_assets = set()
for pf in parquet_files:
df = pd.read_parquet(pf)
ac = [c for c in df.columns if c not in META_COLS]
all_assets.update(ac)
bp = df['BTCUSDT'].values if 'BTCUSDT' in df.columns else None
dv = np.full(len(df), np.nan)
if bp is not None:
for i in range(50, len(bp)):
seg = bp[max(0, i-50):i]
if len(seg) >= 10:
dv[i] = float(np.std(np.diff(seg) / seg[:-1]))
pq_data[pf.stem] = (df, ac, dv)
# load sensor
print(f"\nLoading TitanSensor from {MODEL_PATH.name}...")
sensor = TitanSensor(str(MODEL_PATH))
print(f" Latent dim: {sensor.LATENT_DIM} Hidden: {sensor.HIDDEN_DIM} T5 zeroed: {sensor.T5_START}:261")
# pass 1
trades, bar_logs, vol_p60 = run_pass1(parquet_files, pq_data, all_assets)
# pass 2
df_out = run_pass2(parquet_files, pq_data, trades, bar_logs, sensor)
# save
df_out.to_parquet(str(OUT_FILE), index=False)
print(f"\nLatent trail saved: {OUT_FILE} ({len(df_out):,} rows × {len(df_out.columns)} cols)")
print_summary(df_out)
print("\n=== DONE ===")
print("Next: load titan_latent_trail.parquet and run Stage 2 characterization.")
print(" pd.read_parquet('dvae/titan_latent_trail.parquet')")
if __name__ == '__main__':
main()

328
nautilus_dolphin/dvae/titan_stage2_characterize.py Executable file
View File

@@ -0,0 +1,328 @@
"""
Titan Stage 2 — Latent Trail Characterization
===============================================
Loads titan_latent_trail.parquet and answers:
Q1. Does recon_err discriminate win vs loss at entry bar?
Q2. Which z-dims correlate with trade outcome / recon_err / proxy_B?
Q3. Are DD-driving days in a distinct latent cluster?
Q4. Is z1 (Eigenspace Stress) orthogonal to proxy_B?
Q5. Is recon_err elevated in the N bars before large drawdown events?
Results saved to dvae/titan_stage2_report.json (machine-readable for Stage 3 decisions).
"""
import sys
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
from pathlib import Path
import json
import numpy as np
import pandas as pd
from scipy import stats
TRAIL = Path(__file__).parent / "titan_latent_trail.parquet"
OUT = Path(__file__).parent / "titan_stage2_report.json"
Z_COLS = [f'z{i}' for i in range(32)]
LV_COLS = [f'lv{i}' for i in range(32)]
LATENT_NAMES = {
0: "Trend Consistency",
1: "Eigenspace Stress",
2: "Market Compression",
3: "Volatility Clustering",
}
# ── helpers ────────────────────────────────────────────────────────────────────
def corr_with_pval(a, b):
mask = np.isfinite(a) & np.isfinite(b)
if mask.sum() < 10:
return 0.0, 1.0
r, p = stats.pearsonr(a[mask], b[mask])
return float(r), float(p)
def cohen_d(g1, g2):
n1, n2 = len(g1), len(g2)
if n1 < 2 or n2 < 2:
return 0.0
pooled = np.sqrt(((n1 - 1) * np.var(g1, ddof=1) + (n2 - 1) * np.var(g2, ddof=1)) / (n1 + n2 - 2))
return float((np.mean(g1) - np.mean(g2)) / pooled) if pooled > 0 else 0.0
def mannwhitney(g1, g2):
if len(g1) < 5 or len(g2) < 5:
return 1.0
_, p = stats.mannwhitneyu(g1, g2, alternative='two-sided')
return float(p)
# ── load ───────────────────────────────────────────────────────────────────────
def load_trail():
assert TRAIL.exists(), f"Not found: {TRAIL}\nRun titan_stage1_run.py first."
df = pd.read_parquet(str(TRAIL))
# derive proxy_B (instab_50 - v750)
df['proxy_B'] = df['instab_50'] - df['v750']
return df
# ── Q1: recon_err — win vs loss at entry bar ───────────────────────────────────
def q1_recon_err_win_loss(df):
entry = df[(df['is_in_trade']) & (df['bars_in_trade'] == 0)].copy()
if entry.empty or 'trade_outcome' not in entry.columns:
return {"status": "no_entry_data"}
wins = entry[entry['trade_outcome'] == 'win']['recon_err'].values
losses= entry[entry['trade_outcome'] == 'loss']['recon_err'].values
p_mwu = mannwhitney(wins, losses)
d = cohen_d(wins, losses)
return {
"n_win": int(len(wins)),
"n_loss": int(len(losses)),
"recon_win_mean": float(np.mean(wins)) if len(wins) else None,
"recon_loss_mean": float(np.mean(losses)) if len(losses) else None,
"cohen_d": d,
"p_mannwhitney": p_mwu,
"significant_p05": bool(p_mwu < 0.05),
}
# ── Q2: z-dim correlations (trade outcome, recon_err, proxy_B) ─────────────────
def q2_zdim_correlations(df):
entry = df[(df['is_in_trade']) & (df['bars_in_trade'] == 0)].copy()
if entry.empty:
return {"status": "no_entry_data"}
# outcome as binary
if 'trade_outcome' in entry.columns:
entry['outcome_bin'] = (entry['trade_outcome'] == 'win').astype(float)
else:
entry['outcome_bin'] = np.nan
results = {}
for z in Z_COLS:
r_out, p_out = corr_with_pval(entry[z].values, entry['outcome_bin'].values)
r_rec, p_rec = corr_with_pval(entry[z].values, entry['recon_err'].values)
r_pb, p_pb = corr_with_pval(entry[z].values, entry['proxy_B'].values)
results[z] = {
"r_outcome": r_out, "p_outcome": p_out,
"r_recon": r_rec, "p_recon": p_rec,
"r_proxy_B": r_pb, "p_proxy_B": p_pb,
"name": LATENT_NAMES.get(int(z[1:]), ""),
}
# rank by |r_outcome|
ranked_outcome = sorted(results.items(), key=lambda kv: abs(kv[1]['r_outcome']), reverse=True)
ranked_recon = sorted(results.items(), key=lambda kv: abs(kv[1]['r_recon']), reverse=True)
return {
"top5_by_outcome": [(k, v) for k, v in ranked_outcome[:5]],
"top5_by_recon": [(k, v) for k, v in ranked_recon[:5]],
"z1_eigenspace": results.get("z1", {}),
"all": results,
}
# ── Q3: DD-day cluster analysis ────────────────────────────────────────────────
def q3_dd_cluster(df):
"""
Identify days where in-trade PnL was most negative (bottom 10% days by
mean trade_pnl_pct). Compare z_mu centroids of those days vs the rest.
"""
if 'trade_pnl_pct' not in df.columns:
return {"status": "no_pnl_data"}
# per-day mean pnl among in-trade bars
day_pnl = df[df['is_in_trade']].groupby('date')['trade_pnl_pct'].mean()
if len(day_pnl) < 10:
return {"status": "insufficient_days"}
thr = float(day_pnl.quantile(0.10))
bad_days = set(day_pnl[day_pnl <= thr].index)
good_days = set(day_pnl[day_pnl > thr].index)
bad_df = df[df['date'].isin(bad_days)]
good_df = df[df['date'].isin(good_days)]
diffs = {}
for z in Z_COLS:
g1 = bad_df[z].values
g2 = good_df[z].values
d = cohen_d(g1, g2)
p = mannwhitney(g1, g2)
diffs[z] = {"cohen_d": d, "p_mwu": p}
top5 = sorted(diffs.items(), key=lambda kv: abs(kv[1]['cohen_d']), reverse=True)[:5]
return {
"n_bad_days": len(bad_days),
"n_good_days": len(good_days),
"pnl_thr": thr,
"top5_by_cohend": [(k, v) for k, v in top5],
"all": diffs,
}
# ── Q4: z1 orthogonality to proxy_B ────────────────────────────────────────────
def q4_z1_proxyB(df):
r, p = corr_with_pval(df['z1'].values, df['proxy_B'].values)
return {"r_z1_proxyB": r, "p": p, "orthogonal_p05": bool(p >= 0.05)}
# ── Q5: recon_err pre-drawdown warning ─────────────────────────────────────────
def q5_recon_precursor(df, n_look_ahead=10, dd_thr_pct=-0.015):
"""
For each bar not in a trade, check if any of the next n_look_ahead bars
starts a trade that loses > dd_thr_pct. Does recon_err now predict that?
"""
df2 = df.copy().reset_index(drop=True)
df2['next_loss_flag'] = False
# mark bars N steps before a loss-entry
loss_entry_idxs = set(
df2[(df2['is_in_trade']) &
(df2['bars_in_trade'] == 0) &
(df2['trade_outcome'] == 'loss')].index.tolist()
) if 'trade_outcome' in df2.columns else set()
for idx in loss_entry_idxs:
for lag in range(1, n_look_ahead + 1):
prev = idx - lag
if prev >= 0:
df2.loc[prev, 'next_loss_flag'] = True
flag1 = df2[df2['next_loss_flag']]['recon_err'].values
flag0 = df2[~df2['next_loss_flag'] & ~df2['is_in_trade']]['recon_err'].values
if len(flag1) < 5 or len(flag0) < 5:
return {"status": "insufficient_data"}
p_mwu = mannwhitney(flag1, flag0)
d = cohen_d(flag1, flag0)
return {
"n_pre_loss": int(len(flag1)),
"n_baseline": int(len(flag0)),
"recon_pre_loss_mean": float(np.mean(flag1)),
"recon_baseline_mean": float(np.mean(flag0)),
"cohen_d": d,
"p_mwu": p_mwu,
"significant_p05": bool(p_mwu < 0.05),
}
# ── active z-dim count ─────────────────────────────────────────────────────────
def count_active_dims(df):
variances = {z: float(df[z].var()) for z in Z_COLS}
active = {k: v for k, v in variances.items() if v > 0.01}
return {"n_active": len(active), "top10_var": sorted(variances.items(), key=lambda kv: -kv[1])[:10]}
# ── in-trade vs out-of-trade distributions ────────────────────────────────────
def in_vs_out(df):
it = df[df['is_in_trade']]['recon_err'].values
oot = df[~df['is_in_trade']]['recon_err'].values
return {
"recon_intrade_mean": float(np.mean(it)) if len(it) else None,
"recon_outtrade_mean": float(np.mean(oot)) if len(oot) else None,
"p_mwu": mannwhitney(it, oot),
"cohen_d": cohen_d(it, oot),
}
# ── main ───────────────────────────────────────────────────────────────────────
def main():
print(f"Loading {TRAIL.name}...")
df = load_trail()
print(f" {len(df):,} rows, {len(df.columns)} cols, {df['date'].nunique()} days")
print(f" In-trade bars: {df['is_in_trade'].sum():,} ({df['is_in_trade'].mean()*100:.1f}%)")
report = {}
print("\n[Q0] Active latent dims...")
report['q0_active_dims'] = count_active_dims(df)
r = report['q0_active_dims']
print(f" Active (var>0.01): {r['n_active']}/32")
print(f" Top-5 variance: {r['top10_var'][:5]}")
print("\n[Qin] recon_err in-trade vs out-of-trade...")
report['q_inout'] = in_vs_out(df)
r = report['q_inout']
print(f" in-trade: {r['recon_intrade_mean']:.4f} out-of-trade: {r['recon_outtrade_mean']:.4f}")
print(f" p={r['p_mwu']:.4f} d={r['cohen_d']:.3f}")
print("\n[Q1] recon_err: win vs loss at entry bar...")
report['q1_recon_win_loss'] = q1_recon_err_win_loss(df)
r = report['q1_recon_win_loss']
if 'status' not in r:
print(f" wins: {r['recon_win_mean']:.4f} losses: {r['recon_loss_mean']:.4f}")
print(f" cohen_d={r['cohen_d']:.3f} p={r['p_mannwhitney']:.4f} sig={r['significant_p05']}")
else:
print(f" {r['status']}")
print("\n[Q2] z-dim correlations with outcome/recon/proxy_B...")
report['q2_zdim_corr'] = q2_zdim_correlations(df)
r = report['q2_zdim_corr']
if 'status' not in r:
print(" Top-5 by |r_outcome|:")
for z, v in r['top5_by_outcome']:
nm = LATENT_NAMES.get(int(z[1:]), "")
print(f" {z} {nm:25s}: r_outcome={v['r_outcome']:+.3f}(p={v['p_outcome']:.3f}) "
f"r_proxy_B={v['r_proxy_B']:+.3f}(p={v['p_proxy_B']:.3f})")
print("\n[Q3] DD-day latent cluster...")
report['q3_dd_cluster'] = q3_dd_cluster(df)
r = report['q3_dd_cluster']
if 'status' not in r:
print(f" Bad days (bot-10%): {r['n_bad_days']} threshold pnl: {r['pnl_thr']:.4f}")
print(" Top-5 z-dims separating bad vs good days:")
for z, v in r['top5_by_cohend']:
print(f" {z}: d={v['cohen_d']:+.3f} p={v['p_mwu']:.4f}")
print("\n[Q4] z1 orthogonality to proxy_B...")
report['q4_z1_proxyB'] = q4_z1_proxyB(df)
r = report['q4_z1_proxyB']
print(f" r(z1, proxy_B)={r['r_z1_proxyB']:+.4f} p={r['p']:.4f} orthogonal={r['orthogonal_p05']}")
print("\n[Q5] recon_err pre-drawdown precursor test...")
report['q5_precursor'] = q5_recon_precursor(df)
r = report['q5_precursor']
if 'status' not in r:
print(f" pre-loss: {r['recon_pre_loss_mean']:.4f} baseline: {r['recon_baseline_mean']:.4f}")
print(f" d={r['cohen_d']:.3f} p={r['p_mwu']:.4f} sig={r['significant_p05']}")
else:
print(f" {r['status']}")
# stage 3 verdict
q1 = report['q1_recon_win_loss']
q4 = report['q4_z1_proxyB']
q5 = report['q5_precursor']
q2_top = report['q2_zdim_corr']
verdict = {
"proceed_stage3": False,
"reason": [],
}
if 'cohen_d' in q1 and abs(q1['cohen_d']) > 0.15:
verdict['proceed_stage3'] = True
verdict['reason'].append(f"recon_err win/loss effect d={q1['cohen_d']:.3f}")
if 'status' not in q5 and abs(q5['cohen_d']) > 0.10:
verdict['proceed_stage3'] = True
verdict['reason'].append(f"recon_err pre-drawdown effect d={q5['cohen_d']:.3f}")
if 'top5_by_outcome' in q2_top:
top_r = abs(q2_top['top5_by_outcome'][0][1]['r_outcome']) if q2_top['top5_by_outcome'] else 0
if top_r > 0.10:
verdict['proceed_stage3'] = True
verdict['reason'].append(f"z-dim outcome correlation r={top_r:.3f}")
if not verdict['proceed_stage3']:
verdict['reason'].append("No significant signal found — sensor appears noise-level")
report['verdict'] = verdict
print(f"\n=== VERDICT ===")
print(f" Proceed to Stage 3: {verdict['proceed_stage3']}")
for r_ in verdict['reason']:
print(f" - {r_}")
with open(OUT, 'w') as f:
json.dump(report, f, indent=2, default=str)
print(f"\nReport saved: {OUT}")
if __name__ == '__main__':
main()

679
nautilus_dolphin/dvae/titan_stage2_report.json Executable file
View File

@@ -0,0 +1,679 @@
{
"q0_active_dims": {
"n_active": 32,
"top10_var": [
[
"z27",
0.32883926063986446
],
[
"z4",
0.26002923909456854
],
[
"z31",
0.24365596198872513
],
[
"z24",
0.1882054355293537
],
[
"z23",
0.17481377505536136
],
[
"z17",
0.14738766979069604
],
[
"z8",
0.13379813503359
],
[
"z18",
0.12957219683420088
],
[
"z26",
0.12909863581883954
],
[
"z25",
0.12011033328745513
]
]
},
"q_inout": {
"recon_intrade_mean": 490445256765757.3,
"recon_outtrade_mean": 173221457132288.0,
"p_mwu": 4.4477839828301855e-24,
"cohen_d": 0.0016015105028981686
},
"q1_recon_win_loss": {
"n_win": 1092,
"n_loss": 1063,
"recon_win_mean": 21336023959.1057,
"recon_loss_mean": 31830850.76159581,
"cohen_d": 0.047660513750403324,
"p_mannwhitney": 0.5269125857701439,
"significant_p05": false
},
"q2_zdim_corr": {
"top5_by_outcome": [
[
"z17",
{
"r_outcome": -0.06451684559169232,
"p_outcome": 0.002732023167158115,
"r_recon": 0.013235587955279805,
"p_recon": 0.5391541543625472,
"r_proxy_B": 0.04608323921167554,
"p_proxy_B": 0.03282677816142878,
"name": ""
}
],
[
"z2",
{
"r_outcome": -0.056617333675015556,
"p_outcome": 0.008566848922239571,
"r_recon": 0.028238939264785885,
"p_recon": 0.19005780554206572,
"r_proxy_B": 0.09758896193484892,
"p_proxy_B": 5.9565495599157014e-06,
"name": "Market Compression"
}
],
[
"z4",
{
"r_outcome": -0.05277580509930862,
"p_outcome": 0.014275968216344357,
"r_recon": 0.0017309164898371189,
"p_recon": 0.9359938816007572,
"r_proxy_B": 0.055004334527692975,
"p_proxy_B": 0.010836647564480276,
"name": ""
}
],
[
"z16",
{
"r_outcome": 0.050463851611632825,
"p_outcome": 0.01914142102075866,
"r_recon": 0.01739882363324734,
"p_recon": 0.41950502199970113,
"r_proxy_B": 0.033100281203778045,
"p_proxy_B": 0.12538984265986572,
"name": ""
}
],
[
"z27",
{
"r_outcome": -0.049315788487197844,
"p_outcome": 0.02205605389308747,
"r_recon": 0.015075880242927385,
"p_recon": 0.4842484413973795,
"r_proxy_B": 0.03961647894013642,
"p_proxy_B": 0.06658663681187363,
"name": ""
}
]
],
"top5_by_recon": [
[
"z26",
{
"r_outcome": -0.02113199385842844,
"p_outcome": 0.32682493430511894,
"r_recon": 0.04153185508864403,
"p_recon": 0.05389196119312605,
"r_proxy_B": 0.0102788819075515,
"p_proxy_B": 0.6342220744774475,
"name": ""
}
],
[
"z14",
{
"r_outcome": 0.014166671416253942,
"p_outcome": 0.5109904033766511,
"r_recon": -0.038199295429193494,
"p_recon": 0.07624444639169763,
"r_proxy_B": 0.004202317402982868,
"p_proxy_B": 0.8457733204796783,
"name": ""
}
],
[
"z25",
{
"r_outcome": 0.021375298569722354,
"p_outcome": 0.3212847456492331,
"r_recon": -0.03525294504264228,
"p_recon": 0.10182484667695804,
"r_proxy_B": -0.11572879280444771,
"p_proxy_B": 7.665764334161322e-08,
"name": ""
}
],
[
"z29",
{
"r_outcome": 0.01361839088948349,
"p_outcome": 0.5274815415993123,
"r_recon": -0.03387374513006921,
"p_recon": 0.1159447549599887,
"r_proxy_B": -0.15832646672069148,
"p_proxy_B": 1.642920633664296e-13,
"name": ""
}
],
[
"z3",
{
"r_outcome": 0.015700169427212078,
"p_outcome": 0.4663351435517929,
"r_recon": -0.0327767622712368,
"p_recon": 0.1282377588247886,
"r_proxy_B": -0.0021874890904733254,
"p_proxy_B": 0.9193498216976024,
"name": "Volatility Clustering"
}
]
],
"z1_eigenspace": {
"r_outcome": -0.0247992085145205,
"p_outcome": 0.24984031750028884,
"r_recon": -0.0162103194192992,
"p_recon": 0.4519741533201718,
"r_proxy_B": -0.08765969419087043,
"p_proxy_B": 4.796148952864113e-05,
"name": "Eigenspace Stress"
},
"all": {
"z0": {
"r_outcome": -0.0010568750982667657,
"p_outcome": 0.9608923759085856,
"r_recon": 0.009598146203416313,
"p_recon": 0.6560909247329123,
"r_proxy_B": 0.1535392845201724,
"p_proxy_B": 8.720516330731598e-13,
"name": "Trend Consistency"
},
"z1": {
"r_outcome": -0.0247992085145205,
"p_outcome": 0.24984031750028884,
"r_recon": -0.0162103194192992,
"p_recon": 0.4519741533201718,
"r_proxy_B": -0.08765969419087043,
"p_proxy_B": 4.796148952864113e-05,
"name": "Eigenspace Stress"
},
"z2": {
"r_outcome": -0.056617333675015556,
"p_outcome": 0.008566848922239571,
"r_recon": 0.028238939264785885,
"p_recon": 0.19005780554206572,
"r_proxy_B": 0.09758896193484892,
"p_proxy_B": 5.9565495599157014e-06,
"name": "Market Compression"
},
"z3": {
"r_outcome": 0.015700169427212078,
"p_outcome": 0.4663351435517929,
"r_recon": -0.0327767622712368,
"p_recon": 0.1282377588247886,
"r_proxy_B": -0.0021874890904733254,
"p_proxy_B": 0.9193498216976024,
"name": "Volatility Clustering"
},
"z4": {
"r_outcome": -0.05277580509930862,
"p_outcome": 0.014275968216344357,
"r_recon": 0.0017309164898371189,
"p_recon": 0.9359938816007572,
"r_proxy_B": 0.055004334527692975,
"p_proxy_B": 0.010836647564480276,
"name": ""
},
"z5": {
"r_outcome": 0.020000026866032066,
"p_outcome": 0.35341019960525544,
"r_recon": -0.009469967227304968,
"p_recon": 0.6603939006428033,
"r_proxy_B": -0.10834271232265892,
"p_proxy_B": 4.912827716297637e-07,
"name": ""
},
"z6": {
"r_outcome": -0.019569952694745922,
"p_outcome": 0.36385932892636746,
"r_recon": -0.0074057061713787235,
"p_recon": 0.7311525619916945,
"r_proxy_B": -0.020528881091233183,
"p_proxy_B": 0.34194790377129375,
"name": ""
},
"z7": {
"r_outcome": -0.04033988594722973,
"p_outcome": 0.06116056624508808,
"r_recon": 0.0052153687291698414,
"p_recon": 0.8088042580905699,
"r_proxy_B": 0.10085333136306103,
"p_proxy_B": 2.866651889916251e-06,
"name": ""
},
"z8": {
"r_outcome": -0.022628247113126432,
"p_outcome": 0.29373055414136506,
"r_recon": -0.0006728061311814461,
"p_recon": 0.9750981785707314,
"r_proxy_B": -0.0826384980858231,
"p_proxy_B": 0.0001272839253111422,
"name": ""
},
"z9": {
"r_outcome": 0.01690926987281038,
"p_outcome": 0.4327101833651,
"r_recon": 0.02400736945750492,
"p_recon": 0.26528626750917156,
"r_proxy_B": 0.0567844781969885,
"p_proxy_B": 0.008525582602088367,
"name": ""
},
"z10": {
"r_outcome": -0.04334251912075127,
"p_outcome": 0.0442385507477984,
"r_recon": 0.025178248383767365,
"p_recon": 0.24267322161835128,
"r_proxy_B": -0.048852104224130645,
"p_proxy_B": 0.023661870411551635,
"name": ""
},
"z11": {
"r_outcome": -0.0012230351015354048,
"p_outcome": 0.9547500890762326,
"r_recon": 0.019743119140964778,
"p_recon": 0.35962905206258083,
"r_proxy_B": -0.03055337170168354,
"p_proxy_B": 0.15719793615969194,
"name": ""
},
"z12": {
"r_outcome": 0.0041618036839659416,
"p_outcome": 0.8468901295694582,
"r_recon": 0.01520450651259507,
"p_recon": 0.4805274279049127,
"r_proxy_B": 0.041888743860036146,
"p_proxy_B": 0.052408637888979474,
"name": ""
},
"z13": {
"r_outcome": 0.036536901772677266,
"p_outcome": 0.08994321937470057,
"r_recon": -0.014487387452323983,
"p_recon": 0.5014702295391479,
"r_proxy_B": -0.1475724788444472,
"p_proxy_B": 6.495098046309721e-12,
"name": ""
},
"z14": {
"r_outcome": 0.014166671416253942,
"p_outcome": 0.5109904033766511,
"r_recon": -0.038199295429193494,
"p_recon": 0.07624444639169763,
"r_proxy_B": 0.004202317402982868,
"p_proxy_B": 0.8457733204796783,
"name": ""
},
"z15": {
"r_outcome": 0.02499948887983221,
"p_outcome": 0.2460351015023294,
"r_recon": -0.03218413809052064,
"p_recon": 0.13528768330745722,
"r_proxy_B": -0.08823042177601736,
"p_proxy_B": 4.278226406002478e-05,
"name": ""
},
"z16": {
"r_outcome": 0.050463851611632825,
"p_outcome": 0.01914142102075866,
"r_recon": 0.01739882363324734,
"p_recon": 0.41950502199970113,
"r_proxy_B": 0.033100281203778045,
"p_proxy_B": 0.12538984265986572,
"name": ""
},
"z17": {
"r_outcome": -0.06451684559169232,
"p_outcome": 0.002732023167158115,
"r_recon": 0.013235587955279805,
"p_recon": 0.5391541543625472,
"r_proxy_B": 0.04608323921167554,
"p_proxy_B": 0.03282677816142878,
"name": ""
},
"z18": {
"r_outcome": -0.010419742856575296,
"p_outcome": 0.6287850948537157,
"r_recon": -0.003605377207356623,
"p_recon": 0.8671558961288544,
"r_proxy_B": 0.14216278466870444,
"p_proxy_B": 3.742548790512605e-11,
"name": ""
},
"z19": {
"r_outcome": -0.03250126501522874,
"p_outcome": 0.13147877202501235,
"r_recon": 0.021216153102931148,
"p_recon": 0.32490160545451197,
"r_proxy_B": 0.0009477258934566436,
"p_proxy_B": 0.9650099909640183,
"name": ""
},
"z20": {
"r_outcome": -0.03777134930577331,
"p_outcome": 0.07959704172265401,
"r_recon": 0.028740518832845723,
"p_recon": 0.182304453456116,
"r_proxy_B": 0.05137003128880263,
"p_proxy_B": 0.01734351966005348,
"name": ""
},
"z21": {
"r_outcome": 0.013234681648433487,
"p_outcome": 0.5391819426159447,
"r_recon": -0.003171829009950398,
"p_recon": 0.8830076936291017,
"r_proxy_B": -0.10679829002254787,
"p_proxy_B": 7.137941914537842e-07,
"name": ""
},
"z22": {
"r_outcome": -0.02550599748413965,
"p_outcome": 0.2365934622841013,
"r_recon": 0.020307675235927632,
"p_recon": 0.3460530724297239,
"r_proxy_B": 0.10042782658884095,
"p_proxy_B": 3.1574341248678866e-06,
"name": ""
},
"z23": {
"r_outcome": 0.014935198525591557,
"p_outcome": 0.48833601511223057,
"r_recon": 0.027872647621363763,
"p_recon": 0.19587137746125674,
"r_proxy_B": 0.301977799387075,
"p_proxy_B": 1.811461686140159e-46,
"name": ""
},
"z24": {
"r_outcome": -0.0360981091555881,
"p_outcome": 0.09387159323975619,
"r_recon": 0.02857139473657159,
"p_recon": 0.18489211908852077,
"r_proxy_B": 0.06433572953545957,
"p_proxy_B": 0.002873032209764728,
"name": ""
},
"z25": {
"r_outcome": 0.021375298569722354,
"p_outcome": 0.3212847456492331,
"r_recon": -0.03525294504264228,
"p_recon": 0.10182484667695804,
"r_proxy_B": -0.11572879280444771,
"p_proxy_B": 7.665764334161322e-08,
"name": ""
},
"z26": {
"r_outcome": -0.02113199385842844,
"p_outcome": 0.32682493430511894,
"r_recon": 0.04153185508864403,
"p_recon": 0.05389196119312605,
"r_proxy_B": 0.0102788819075515,
"p_proxy_B": 0.6342220744774475,
"name": ""
},
"z27": {
"r_outcome": -0.049315788487197844,
"p_outcome": 0.02205605389308747,
"r_recon": 0.015075880242927385,
"p_recon": 0.4842484413973795,
"r_proxy_B": 0.03961647894013642,
"p_proxy_B": 0.06658663681187363,
"name": ""
},
"z28": {
"r_outcome": 0.0223352138005965,
"p_outcome": 0.3000283991479089,
"r_recon": -0.009249018350925543,
"p_recon": 0.6678375092136699,
"r_proxy_B": -0.20421421742886217,
"p_proxy_B": 1.2558249326308314e-21,
"name": ""
},
"z29": {
"r_outcome": 0.01361839088948349,
"p_outcome": 0.5274815415993123,
"r_recon": -0.03387374513006921,
"p_recon": 0.1159447549599887,
"r_proxy_B": -0.15832646672069148,
"p_proxy_B": 1.642920633664296e-13,
"name": ""
},
"z30": {
"r_outcome": 0.023239956878488108,
"p_outcome": 0.28087114844614264,
"r_recon": -0.03268331324183672,
"p_recon": 0.12933008601923748,
"r_proxy_B": -0.1835909740980365,
"p_proxy_B": 1.0272396398012494e-17,
"name": ""
},
"z31": {
"r_outcome": 0.04289343307123413,
"p_outcome": 0.046485396698545296,
"r_recon": 0.006306882337453737,
"p_recon": 0.7698195124016781,
"r_proxy_B": -0.02802640429729864,
"p_proxy_B": 0.19445329724411342,
"name": ""
}
}
},
"q3_dd_cluster": {
"n_bad_days": 6,
"n_good_days": 50,
"pnl_thr": -0.0018667932119076722,
"top5_by_cohend": [
[
"z2",
{
"cohen_d": 0.1702382007291983,
"p_mwu": 3.0736777951998913e-176
}
],
[
"z30",
{
"cohen_d": -0.13699319599298396,
"p_mwu": 1.6958984078998588e-137
}
],
[
"z14",
{
"cohen_d": -0.13589791488813383,
"p_mwu": 4.785233693695352e-114
}
],
[
"z24",
{
"cohen_d": 0.13343797760574308,
"p_mwu": 1.2927044220421628e-112
}
],
[
"z28",
{
"cohen_d": -0.13101473669555413,
"p_mwu": 2.266661710705797e-108
}
]
],
"all": {
"z0": {
"cohen_d": -0.003607296612985334,
"p_mwu": 0.05448282039381696
},
"z1": {
"cohen_d": 0.002270852314641101,
"p_mwu": 0.18464589699399714
},
"z2": {
"cohen_d": 0.1702382007291983,
"p_mwu": 3.0736777951998913e-176
},
"z3": {
"cohen_d": -0.08606377852354527,
"p_mwu": 3.858966160620741e-31
},
"z4": {
"cohen_d": -0.03930732284705677,
"p_mwu": 3.570418056957964e-08
},
"z5": {
"cohen_d": 0.03767457899823889,
"p_mwu": 0.008304914770217817
},
"z6": {
"cohen_d": -0.0038666876112777304,
"p_mwu": 0.6618622244751171
},
"z7": {
"cohen_d": 0.01694936976956897,
"p_mwu": 0.0008244556186001203
},
"z8": {
"cohen_d": -0.060054157590918734,
"p_mwu": 1.8655724326671896e-11
},
"z9": {
"cohen_d": -0.07312008305740242,
"p_mwu": 1.1770831324668023e-43
},
"z10": {
"cohen_d": 0.10041389587252378,
"p_mwu": 3.0238566750068834e-59
},
"z11": {
"cohen_d": 0.11843170197666235,
"p_mwu": 6.412698538396475e-88
},
"z12": {
"cohen_d": 0.03249783237190171,
"p_mwu": 3.917135177969126e-07
},
"z13": {
"cohen_d": -0.07090955502752205,
"p_mwu": 2.4168037389481122e-24
},
"z14": {
"cohen_d": -0.13589791488813383,
"p_mwu": 4.785233693695352e-114
},
"z15": {
"cohen_d": -0.09101483704799612,
"p_mwu": 4.857812290022197e-53
},
"z16": {
"cohen_d": -0.08306701720666351,
"p_mwu": 1.5693325289612218e-48
},
"z17": {
"cohen_d": 0.03351832104783407,
"p_mwu": 3.418058656690057e-05
},
"z18": {
"cohen_d": -0.0724821119024743,
"p_mwu": 1.841744446899598e-18
},
"z19": {
"cohen_d": 0.09819374372971937,
"p_mwu": 5.5906661745128066e-73
},
"z20": {
"cohen_d": 0.027696992079276395,
"p_mwu": 0.0004392631470794174
},
"z21": {
"cohen_d": -0.09719456361183183,
"p_mwu": 4.2805622117725385e-61
},
"z22": {
"cohen_d": -0.0016563123173948867,
"p_mwu": 0.17419143874597287
},
"z23": {
"cohen_d": 0.07151359964500655,
"p_mwu": 2.385055840415299e-37
},
"z24": {
"cohen_d": 0.13343797760574308,
"p_mwu": 1.2927044220421628e-112
},
"z25": {
"cohen_d": -0.04241011895618605,
"p_mwu": 1.5634881824466994e-06
},
"z26": {
"cohen_d": -0.02937131756252973,
"p_mwu": 0.017666616149218564
},
"z27": {
"cohen_d": 0.0008605492125759038,
"p_mwu": 0.4167828096406764
},
"z28": {
"cohen_d": -0.13101473669555413,
"p_mwu": 2.266661710705797e-108
},
"z29": {
"cohen_d": -0.11529483621765778,
"p_mwu": 4.93695857580849e-86
},
"z30": {
"cohen_d": -0.13699319599298396,
"p_mwu": 1.6958984078998588e-137
},
"z31": {
"cohen_d": 0.04758746969298401,
"p_mwu": 8.924920058240176e-23
}
}
},
"q4_z1_proxyB": {
"r_z1_proxyB": -0.03690845832536047,
"p": 2.0738200311673893e-104,
"orthogonal_p05": false
},
"q5_precursor": {
"n_pre_loss": 10630,
"n_baseline": 99929,
"recon_pre_loss_mean": 1.0888473331169846e+16,
"recon_baseline_mean": 186243987055429.94,
"cohen_d": 0.03051106041997001,
"p_mwu": 4.685046720661133e-06,
"significant_p05": true
},
"verdict": {
"proceed_stage3": false,
"reason": [
"No significant signal found \u2014 sensor appears noise-level"
]
}
}

314
nautilus_dolphin/dvae/train_corpus_dvae.py Executable file
View File

@@ -0,0 +1,314 @@
"""
DOLPHIN Hierarchical D-VAE — Curriculum Training
==================================================
Run this script to train the H-D-VAE over the full multi-generation corpus.
Usage:
cd nautilus_dolphin/dvae
python train_corpus_dvae.py [--phases 0123] [--epochs 100] [--lr 3e-4]
Training curriculum (4 phases):
Phase 0: Macro encoder only — full 500K+ corpus (breadth signal from ALL gens)
Phase 1: + Eigen encoder — NG3/NG5 subset with eigenvalue data
Phase 2: + Cross-section — NG3/NG5 subset with pricing data
Phase 3: Joint fine-tune — full eigen+pricing corpus together
The "512-bit advantage" is injected via log-ratio features in the corpus builder:
log(lambda_max) — captures the order-of-magnitude accurately
gap_ratio — scale-free condition number proxy
vel_norm — velocity normalised by magnitude
These preserve the mathematical structure even within float64 storage.
Expected discoveries (based on D-VAE theory + DOLPHIN domain knowledge):
1. z0 latent space shows MORE THAN 3 regimes — the simple BULL/BEAR/SIDEWAYS
label is lossy; the VAE will reveal 5-7 meta-states.
2. Regime transition events (high dTC/dt) precede breadth changes by N scans
— eigenvalue instability is LEADING, not coincident.
3. z1_eigen predicts FUTURE z0_macro delta — structural change comes first.
4. z2_xsection reveals which assets (BTC dominance, BNB, stablecoins) are
the primary eigenvectors of regime transitions.
"""
import sys
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
sys.stderr.reconfigure(encoding='utf-8', errors='replace')
import argparse
import json
import time
import numpy as np
from pathlib import Path
from datetime import datetime
from corpus_builder import DolphinCorpus
from hierarchical_dvae import HierarchicalDVAE
# ── Paths ──────────────────────────────────────────────────────────────────
HERE = Path(__file__).parent
CACHE = HERE / 'corpus_cache.npz'
MODEL = HERE / 'hdvae_checkpoint'
LOGS = HERE / 'training_log.jsonl'
# ── Curriculum config ──────────────────────────────────────────────────────
PHASE_CONFIG = {
0: dict(epochs=60, lr=3e-4, batch=256, name="T0: Macro-only (ALL gens, pre-eigen included)"),
1: dict(epochs=80, lr=2e-4, batch=128, name="T1: + Eigenvalue tier (NG3+)"),
2: dict(epochs=80, lr=1e-4, batch=128, name="T2: + Cross-section tier (pricing)"),
3: dict(epochs=60, lr=1e-4, batch=128, name="T3: + ExF macro tier"),
4: dict(epochs=100, lr=5e-5, batch=256, name="T4: Joint fine-tune all tiers"),
}
PRINT_EVERY = 10 # epochs
def build_or_load_corpus(force_rebuild=False) -> DolphinCorpus:
if CACHE.exists() and not force_rebuild:
print(f"Loading corpus from cache: {CACHE}")
return DolphinCorpus.load(str(CACHE))
corpus = DolphinCorpus().build(
ng3_date_from='2025-12-31', # full NG3 history from Dec-31
max_scans_per_day=300, # 300 scans/day × ~70 days = ~21K NG3 rows
max_per_source=50_000, # cap NG1/NG2/NG4 at 50K each (~RAM safe)
max_ng5=2_000, # NG5-local files read slowly (~26/s), cap low
verbose=True,
)
corpus.save(str(HERE / 'corpus_cache'))
return corpus
def get_phase_data(corpus: DolphinCorpus, phase: int):
"""Return (X, mask) appropriate for this training phase."""
if phase == 0:
return corpus.X, corpus.mask # ALL (even pre-eigen)
elif phase == 1:
idx = corpus.mask[:, 1] # has eigenvalues
return corpus.X[idx], corpus.mask[idx]
elif phase == 2:
idx = corpus.mask[:, 1] & corpus.mask[:, 2] # eigen + pricing
return corpus.X[idx], corpus.mask[idx]
elif phase == 3:
idx = corpus.mask[:, 3] # has ExF data
return corpus.X[idx], corpus.mask[idx]
else: # phase 4 joint
idx = corpus.mask[:, 1] # eigen subset, all tiers
return corpus.X[idx], corpus.mask[idx]
def log_stats(epoch: int, phase: int, stats: dict, elapsed: float):
record = dict(epoch=epoch, phase=phase, elapsed_s=round(elapsed, 2), **{k: round(float(v), 6) for k, v in stats.items()})
with open(LOGS, 'a') as f:
f.write(json.dumps(record) + '\n')
return record
def train(phases=(0, 1, 2, 3), force_rebuild=False):
rng = np.random.RandomState(42)
print("=" * 60)
print("DOLPHIN Hierarchical D-VAE — Curriculum Training")
print("=" * 60)
corpus = build_or_load_corpus(force_rebuild)
model = HierarchicalDVAE(hidden=128, beta=0.5, gamma=1.0, lam=1.0, seed=42)
model.fit_normaliser(corpus.X, corpus.mask)
print(f"\nModel: z0=4 z1=8 z2=8 z3=4 total_latent=24 hidden=128 input=111")
print(f"beta-TCVAE: beta=0.5 gamma=1.0 lam=1.0\n")
total_start = time.time()
for phase in phases:
cfg = PHASE_CONFIG[phase]
X, mask = get_phase_data(corpus, phase)
N = len(X)
print(f"\n{'─'*60}")
print(f"Phase {phase}: {cfg['name']} N={N:,} epochs={cfg['epochs']} lr={cfg['lr']}")
print(f"{'─'*60}")
best_loss = float('inf')
patience_cnt = 0
# KL warm-up: ramp ALL KL coefficients (gamma, beta, lam) from 0 to target
# over first 30% of epochs. With all=0, pure reconstruction training first.
target_beta = model.beta
target_gamma = model.gamma
target_lam = model.lam
warmup_epochs = max(1, cfg['epochs'] // 3)
for epoch in range(1, cfg['epochs'] + 1):
if epoch <= warmup_epochs:
frac = epoch / warmup_epochs
model.beta = target_beta * frac
model.gamma = target_gamma * frac
model.lam = target_lam * frac
else:
model.beta = target_beta
model.gamma = target_gamma
model.lam = target_lam
t0 = time.time()
stats = model.train_epoch(X, mask, lr=cfg['lr'],
batch_size=cfg['batch'],
phase=phase, rng=rng)
elapsed = time.time() - t0
record = log_stats(epoch + sum(PHASE_CONFIG[p]['epochs'] for p in range(phase)),
phase, stats, elapsed)
if epoch % PRINT_EVERY == 0 or epoch == 1:
print(f" ep {epoch:4d}/{cfg['epochs']} "
f"loss={stats['total']:.4f} "
f"recon=[{stats['recon0']:.4f},{stats['recon1']:.4f},{stats['recon2']:.4f}] "
f"KL=[{stats['kl0']:.4f},{stats['kl1']:.4f},{stats['kl2']:.4f}] "
f"({elapsed:.1f}s/ep)")
# Early stopping
if stats['total'] < best_loss - 1e-5:
best_loss = stats['total']
patience_cnt = 0
model.save(str(MODEL))
else:
patience_cnt += 1
if patience_cnt >= 20 and epoch > cfg['epochs'] // 3:
print(f" Early stop at epoch {epoch} (patience=20)")
break
model.beta = target_beta # restore after warm-up
model.gamma = target_gamma
model.lam = target_lam
print(f" Phase {phase} done. Best loss: {best_loss:.4f}")
total_elapsed = time.time() - total_start
print(f"\n{'='*60}")
print(f"Training complete in {total_elapsed/60:.1f} min")
print(f"Model saved: {MODEL}.npz")
print(f"Log: {LOGS}")
# ── Latent space analysis ──────────────────────────────────────────────
analyse(model, corpus)
return model
def analyse(model: HierarchicalDVAE, corpus: DolphinCorpus):
"""
Post-training analysis — the SURPRISES live here.
Writes results to analysis_results.json.
"""
print("\n" + "=" * 60)
print("LATENT SPACE ANALYSIS")
print("=" * 60)
# Use eigen subset for meaningful analysis
idx_eigen = corpus.mask[:, 1]
X_e = corpus.X[idx_eigen]
mask_e = corpus.mask[idx_eigen]
src_e = corpus.sources[idx_eigen]
latents = model.get_latents(X_e, mask_e)
z0 = latents['z0_macro'] # (N, 4)
z1 = latents['z1_eigen'] # (N, 8)
z2 = latents['z2_xsection'] # (N, 8)
results = {}
# 1. Cluster z0 — how many macro regimes does the model find?
from scipy.cluster.vq import kmeans2
n_clusters_try = [3, 4, 5, 6, 7, 8]
best_sil, best_k = -1, 3
sil_scores = {}
for k in n_clusters_try:
try:
centroids, labels = kmeans2(z0, k, seed=42, minit='points')
# Silhouette (approximate, fast)
intra = np.mean([np.mean(np.linalg.norm(z0[labels == i] - centroids[i], axis=1))
for i in range(k) if (labels == i).sum() > 0])
inter_dists = [np.linalg.norm(centroids[i] - centroids[j])
for i in range(k) for j in range(k) if i != j]
sil = min(inter_dists) / max(intra, 1e-6)
sil_scores[k] = round(float(sil), 4)
if sil > best_sil:
best_sil, best_k = sil, k
except Exception:
pass
print(f"\n1. Macro regime clusters (z0):")
print(f" Silhouette scores: {sil_scores}")
print(f" OPTIMAL k = {best_k} (expected: more than 3 if hypothesis holds)")
results['macro_regime_k'] = best_k
results['macro_regime_silhouettes'] = sil_scores
# 2. Predictive power: does z1 (eigen) predict FUTURE z0?
if len(z0) > 100:
LAGS = [1, 3, 5, 10, 20]
z1_to_z0_pred = {}
for lag in LAGS:
z1_now = z1[:-lag]
z0_future = z0[lag:]
# Linear regression: z1 → z0_future[dim 0] (proxy for regime score)
# Simple correlation
z0_f_norm = z0_future[:, 0]
z1_flat = z1_now.mean(axis=1) # scalar summary
corr = np.corrcoef(z1_flat, z0_f_norm)[0, 1]
z1_to_z0_pred[lag] = round(float(corr), 4)
print(f"\n2. z1_eigen -> future z0_macro correlation:")
for lag, c in z1_to_z0_pred.items():
print(f" lag={lag:2d}: r={c:.4f}")
peak_lag = max(z1_to_z0_pred, key=lambda l: abs(z1_to_z0_pred[l]))
print(f" PEAK at lag={peak_lag} — z1 leads z0 by {peak_lag} scans (~{peak_lag*12}s)")
results['z1_leads_z0'] = z1_to_z0_pred
results['z1_peak_lead_lag'] = peak_lag
# 3. Variance captured per latent dimension
z_all = latents['z_all']
var_per_dim = np.var(z_all, axis=0)
active_dims = (var_per_dim > 0.1).sum()
print(f"\n3. Active latent dimensions (var>0.1): {active_dims}/{z_all.shape[1]}")
print(f" Variance per dim: {np.round(var_per_dim, 3)}")
results['active_dims'] = int(active_dims)
results['var_per_dim'] = var_per_dim.tolist()
# 4. Per-source separation — do NG1 vs NG3 cluster differently in z0?
print(f"\n4. z0 centroids per data generation:")
src_all = corpus.sources[idx_eigen]
src_names = {0: 'NG1/2', 1: 'NG4', 2: 'NG5-local', 3: 'NG3-scan'}
for sid, sname in src_names.items():
sidx = src_all == sid
if sidx.sum() > 5:
c = z0[sidx].mean(axis=0)
print(f" {sname:12s} N={sidx.sum():6,} z0_mean={np.round(c,3)}")
# 5. Reconstruction error as OOD detector
rng_inf = np.random.RandomState(0)
enc_all = model.encode(X_e[:1000], mask_e[:1000], rng_inf)
dec_all = model.decode(enc_all)
t0_recon = dec_all['x0_hat']
recon_err = np.mean((t0_recon - enc_all['t0'])**2, axis=1)
print(f"\n5. Reconstruction error (tier0):")
print(f" Mean: {recon_err.mean():.4f} Std: {recon_err.std():.4f}")
print(f" Top-5% OOD threshold: {np.percentile(recon_err, 95):.4f}")
results['recon_err_mean'] = float(recon_err.mean())
results['recon_err_p95'] = float(np.percentile(recon_err, 95))
# Save
out = HERE / 'analysis_results.json'
with open(out, 'w') as f:
json.dump(results, f, indent=2)
print(f"\nAnalysis saved: {out}")
return results
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--phases', type=str, default='0123',
help='Which phases to run e.g. "0123" or "3"')
parser.add_argument('--rebuild', action='store_true', help='Force corpus rebuild')
parser.add_argument('--analyse-only', action='store_true', help='Skip training, just analyse saved model')
args = parser.parse_args()
phases = [int(p) for p in args.phases]
if args.analyse_only:
corpus = build_or_load_corpus(False)
model = HierarchicalDVAE(hidden=128, beta=6.0, gamma=1.0, lam=1.0)
d = np.load(str(MODEL) + '.npz', allow_pickle=True)
# (load weights — simplified; full load_weights() would restore all layers)
print("Note: implement full weight restore for production use.")
analyse(model, corpus)
else:
train(phases=phases, force_rebuild=args.rebuild)

309
nautilus_dolphin/dvae/training_log.jsonl Executable file
View File

@@ -0,0 +1,309 @@
{"epoch": 1, "phase": 0, "elapsed_s": 22.03, "total": 0.563407, "recon0": 0.821045, "recon1": 0.0, "recon2": 0.0, "kl0": -0.257638, "kl1": 0.0, "kl2": 0.0}
{"epoch": 2, "phase": 0, "elapsed_s": 23.21, "total": 0.011092, "recon0": 0.488985, "recon1": 0.0, "recon2": 0.0, "kl0": -0.477893, "kl1": 0.0, "kl2": 0.0}
{"epoch": 3, "phase": 0, "elapsed_s": 21.05, "total": -0.11968, "recon0": 0.526534, "recon1": 0.0, "recon2": 0.0, "kl0": -0.646214, "kl1": 0.0, "kl2": 0.0}
{"epoch": 4, "phase": 0, "elapsed_s": 19.68, "total": -0.302427, "recon0": 0.566573, "recon1": 0.0, "recon2": 0.0, "kl0": -0.869, "kl1": 0.0, "kl2": 0.0}
{"epoch": 5, "phase": 0, "elapsed_s": 20.36, "total": -0.603467, "recon0": 0.61756, "recon1": 0.0, "recon2": 0.0, "kl0": -1.221026, "kl1": 0.0, "kl2": 0.0}
{"epoch": 6, "phase": 0, "elapsed_s": 19.5, "total": -0.984, "recon0": 0.658019, "recon1": 0.0, "recon2": 0.0, "kl0": -1.642019, "kl1": 0.0, "kl2": 0.0}
{"epoch": 7, "phase": 0, "elapsed_s": 19.66, "total": -1.319487, "recon0": 0.702564, "recon1": 0.0, "recon2": 0.0, "kl0": -2.02205, "kl1": 0.0, "kl2": 0.0}
{"epoch": 8, "phase": 0, "elapsed_s": 18.56, "total": -1.716922, "recon0": 0.752583, "recon1": 0.0, "recon2": 0.0, "kl0": -2.469504, "kl1": 0.0, "kl2": 0.0}
{"epoch": 9, "phase": 0, "elapsed_s": 19.71, "total": -2.202457, "recon0": 0.797613, "recon1": 0.0, "recon2": 0.0, "kl0": -3.000069, "kl1": 0.0, "kl2": 0.0}
{"epoch": 10, "phase": 0, "elapsed_s": 18.81, "total": -2.831215, "recon0": 0.853824, "recon1": 0.0, "recon2": 0.0, "kl0": -3.685039, "kl1": 0.0, "kl2": 0.0}
{"epoch": 11, "phase": 0, "elapsed_s": 18.51, "total": -3.424253, "recon0": 0.90775, "recon1": 0.0, "recon2": 0.0, "kl0": -4.332004, "kl1": 0.0, "kl2": 0.0}
{"epoch": 12, "phase": 0, "elapsed_s": 18.58, "total": -3.981677, "recon0": 0.937774, "recon1": 0.0, "recon2": 0.0, "kl0": -4.919451, "kl1": 0.0, "kl2": 0.0}
{"epoch": 13, "phase": 0, "elapsed_s": 20.87, "total": -4.469276, "recon0": 0.963814, "recon1": 0.0, "recon2": 0.0, "kl0": -5.43309, "kl1": 0.0, "kl2": 0.0}
{"epoch": 14, "phase": 0, "elapsed_s": 20.08, "total": -4.909504, "recon0": 0.977914, "recon1": 0.0, "recon2": 0.0, "kl0": -5.887418, "kl1": 0.0, "kl2": 0.0}
{"epoch": 15, "phase": 0, "elapsed_s": 20.36, "total": -5.342119, "recon0": 0.985359, "recon1": 0.0, "recon2": 0.0, "kl0": -6.327477, "kl1": 0.0, "kl2": 0.0}
{"epoch": 16, "phase": 0, "elapsed_s": 18.58, "total": -5.773167, "recon0": 0.990528, "recon1": 0.0, "recon2": 0.0, "kl0": -6.763696, "kl1": 0.0, "kl2": 0.0}
{"epoch": 17, "phase": 0, "elapsed_s": 18.21, "total": -6.202573, "recon0": 1.001185, "recon1": 0.0, "recon2": 0.0, "kl0": -7.203758, "kl1": 0.0, "kl2": 0.0}
{"epoch": 18, "phase": 0, "elapsed_s": 17.96, "total": -6.631239, "recon0": 1.00317, "recon1": 0.0, "recon2": 0.0, "kl0": -7.63441, "kl1": 0.0, "kl2": 0.0}
{"epoch": 19, "phase": 0, "elapsed_s": 18.56, "total": -7.057077, "recon0": 1.002305, "recon1": 0.0, "recon2": 0.0, "kl0": -8.059382, "kl1": 0.0, "kl2": 0.0}
{"epoch": 20, "phase": 0, "elapsed_s": 18.79, "total": -7.487053, "recon0": 1.002168, "recon1": 0.0, "recon2": 0.0, "kl0": -8.489221, "kl1": 0.0, "kl2": 0.0}
{"epoch": 21, "phase": 0, "elapsed_s": 18.96, "total": -7.486468, "recon0": 1.004711, "recon1": 0.0, "recon2": 0.0, "kl0": -8.49118, "kl1": 0.0, "kl2": 0.0}
{"epoch": 22, "phase": 0, "elapsed_s": 18.62, "total": -7.488365, "recon0": 1.004622, "recon1": 0.0, "recon2": 0.0, "kl0": -8.492987, "kl1": 0.0, "kl2": 0.0}
{"epoch": 23, "phase": 0, "elapsed_s": 18.42, "total": -7.490015, "recon0": 1.00438, "recon1": 0.0, "recon2": 0.0, "kl0": -8.494395, "kl1": 0.0, "kl2": 0.0}
{"epoch": 24, "phase": 0, "elapsed_s": 18.2, "total": -7.489238, "recon0": 1.003929, "recon1": 0.0, "recon2": 0.0, "kl0": -8.493167, "kl1": 0.0, "kl2": 0.0}
{"epoch": 25, "phase": 0, "elapsed_s": 17.27, "total": -7.490874, "recon0": 1.003384, "recon1": 0.0, "recon2": 0.0, "kl0": -8.494258, "kl1": 0.0, "kl2": 0.0}
{"epoch": 26, "phase": 0, "elapsed_s": 17.8, "total": -7.493285, "recon0": 1.003424, "recon1": 0.0, "recon2": 0.0, "kl0": -8.496709, "kl1": 0.0, "kl2": 0.0}
{"epoch": 27, "phase": 0, "elapsed_s": 18.42, "total": -7.492076, "recon0": 1.003325, "recon1": 0.0, "recon2": 0.0, "kl0": -8.495401, "kl1": 0.0, "kl2": 0.0}
{"epoch": 28, "phase": 0, "elapsed_s": 18.65, "total": -7.492305, "recon0": 1.002934, "recon1": 0.0, "recon2": 0.0, "kl0": -8.495239, "kl1": 0.0, "kl2": 0.0}
{"epoch": 29, "phase": 0, "elapsed_s": 20.79, "total": -7.493778, "recon0": 1.003455, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497234, "kl1": 0.0, "kl2": 0.0}
{"epoch": 30, "phase": 0, "elapsed_s": 20.76, "total": -7.492053, "recon0": 1.002286, "recon1": 0.0, "recon2": 0.0, "kl0": -8.494339, "kl1": 0.0, "kl2": 0.0}
{"epoch": 31, "phase": 0, "elapsed_s": 19.65, "total": -7.494672, "recon0": 1.002418, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497089, "kl1": 0.0, "kl2": 0.0}
{"epoch": 32, "phase": 0, "elapsed_s": 21.82, "total": -7.494612, "recon0": 1.002406, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497017, "kl1": 0.0, "kl2": 0.0}
{"epoch": 33, "phase": 0, "elapsed_s": 18.7, "total": -7.494464, "recon0": 1.001947, "recon1": 0.0, "recon2": 0.0, "kl0": -8.496411, "kl1": 0.0, "kl2": 0.0}
{"epoch": 34, "phase": 0, "elapsed_s": 17.75, "total": -7.495878, "recon0": 1.002308, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498185, "kl1": 0.0, "kl2": 0.0}
{"epoch": 35, "phase": 0, "elapsed_s": 18.27, "total": -7.495623, "recon0": 1.002302, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497925, "kl1": 0.0, "kl2": 0.0}
{"epoch": 36, "phase": 0, "elapsed_s": 18.71, "total": -7.495609, "recon0": 1.002202, "recon1": 0.0, "recon2": 0.0, "kl0": -8.49781, "kl1": 0.0, "kl2": 0.0}
{"epoch": 37, "phase": 0, "elapsed_s": 18.01, "total": -7.495415, "recon0": 1.00316, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498574, "kl1": 0.0, "kl2": 0.0}
{"epoch": 38, "phase": 0, "elapsed_s": 18.29, "total": -7.496314, "recon0": 1.002531, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498845, "kl1": 0.0, "kl2": 0.0}
{"epoch": 39, "phase": 0, "elapsed_s": 18.14, "total": -7.497587, "recon0": 1.002706, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500294, "kl1": 0.0, "kl2": 0.0}
{"epoch": 40, "phase": 0, "elapsed_s": 18.0, "total": -7.496429, "recon0": 1.002001, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498431, "kl1": 0.0, "kl2": 0.0}
{"epoch": 41, "phase": 0, "elapsed_s": 17.28, "total": -7.496715, "recon0": 1.001397, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498113, "kl1": 0.0, "kl2": 0.0}
{"epoch": 42, "phase": 0, "elapsed_s": 18.34, "total": -7.496506, "recon0": 1.00252, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499026, "kl1": 0.0, "kl2": 0.0}
{"epoch": 43, "phase": 0, "elapsed_s": 19.07, "total": -7.497662, "recon0": 1.001901, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499563, "kl1": 0.0, "kl2": 0.0}
{"epoch": 44, "phase": 0, "elapsed_s": 19.05, "total": -7.4971, "recon0": 1.002153, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499253, "kl1": 0.0, "kl2": 0.0}
{"epoch": 45, "phase": 0, "elapsed_s": 18.82, "total": -7.497464, "recon0": 1.002783, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500247, "kl1": 0.0, "kl2": 0.0}
{"epoch": 46, "phase": 0, "elapsed_s": 18.34, "total": -7.497747, "recon0": 1.002593, "recon1": 0.0, "recon2": 0.0, "kl0": -8.50034, "kl1": 0.0, "kl2": 0.0}
{"epoch": 47, "phase": 0, "elapsed_s": 17.75, "total": -7.497964, "recon0": 1.002181, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500145, "kl1": 0.0, "kl2": 0.0}
{"epoch": 48, "phase": 0, "elapsed_s": 17.41, "total": -7.498381, "recon0": 1.00227, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500651, "kl1": 0.0, "kl2": 0.0}
{"epoch": 49, "phase": 0, "elapsed_s": 17.94, "total": -7.498628, "recon0": 1.002246, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500874, "kl1": 0.0, "kl2": 0.0}
{"epoch": 50, "phase": 0, "elapsed_s": 18.0, "total": -7.499048, "recon0": 1.002463, "recon1": 0.0, "recon2": 0.0, "kl0": -8.50151, "kl1": 0.0, "kl2": 0.0}
{"epoch": 51, "phase": 0, "elapsed_s": 18.27, "total": -7.498272, "recon0": 1.001012, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499284, "kl1": 0.0, "kl2": 0.0}
{"epoch": 52, "phase": 0, "elapsed_s": 17.93, "total": -7.499382, "recon0": 1.002023, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501405, "kl1": 0.0, "kl2": 0.0}
{"epoch": 53, "phase": 0, "elapsed_s": 17.71, "total": -7.498781, "recon0": 1.001925, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500706, "kl1": 0.0, "kl2": 0.0}
{"epoch": 54, "phase": 0, "elapsed_s": 17.7, "total": -7.499369, "recon0": 1.001958, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501327, "kl1": 0.0, "kl2": 0.0}
{"epoch": 55, "phase": 0, "elapsed_s": 18.18, "total": -7.499192, "recon0": 1.001922, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501114, "kl1": 0.0, "kl2": 0.0}
{"epoch": 56, "phase": 0, "elapsed_s": 17.79, "total": -7.499207, "recon0": 1.001835, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501042, "kl1": 0.0, "kl2": 0.0}
{"epoch": 57, "phase": 0, "elapsed_s": 18.16, "total": -7.499649, "recon0": 1.001424, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501072, "kl1": 0.0, "kl2": 0.0}
{"epoch": 58, "phase": 0, "elapsed_s": 17.85, "total": -7.499302, "recon0": 1.001748, "recon1": 0.0, "recon2": 0.0, "kl0": -8.50105, "kl1": 0.0, "kl2": 0.0}
{"epoch": 59, "phase": 0, "elapsed_s": 17.36, "total": -7.499962, "recon0": 1.002023, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501985, "kl1": 0.0, "kl2": 0.0}
{"epoch": 60, "phase": 0, "elapsed_s": 17.5, "total": -7.499943, "recon0": 1.002261, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502203, "kl1": 0.0, "kl2": 0.0}
{"epoch": 61, "phase": 1, "elapsed_s": 2.68, "total": 1.443433, "recon0": 0.640125, "recon1": 1.404729, "recon2": 0.0, "kl0": -0.112623, "kl1": -0.488798, "kl2": 0.0}
{"epoch": 62, "phase": 1, "elapsed_s": 2.6, "total": 0.124844, "recon0": 0.374956, "recon1": 0.918771, "recon2": 0.0, "kl0": -0.078024, "kl1": -1.09086, "kl2": 0.0}
{"epoch": 63, "phase": 1, "elapsed_s": 2.89, "total": -0.636463, "recon0": 0.369197, "recon1": 0.846841, "recon2": 0.0, "kl0": -0.154775, "kl1": -1.697727, "kl2": 0.0}
{"epoch": 64, "phase": 1, "elapsed_s": 2.7, "total": -1.415433, "recon0": 0.417681, "recon1": 0.834207, "recon2": 0.0, "kl0": -0.34309, "kl1": -2.324232, "kl2": 0.0}
{"epoch": 65, "phase": 1, "elapsed_s": 2.76, "total": -2.300887, "recon0": 0.480138, "recon1": 0.8387, "recon2": 0.0, "kl0": -0.656977, "kl1": -2.962748, "kl2": 0.0}
{"epoch": 66, "phase": 1, "elapsed_s": 2.53, "total": -3.361812, "recon0": 0.541731, "recon1": 0.85261, "recon2": 0.0, "kl0": -1.142483, "kl1": -3.61367, "kl2": 0.0}
{"epoch": 67, "phase": 1, "elapsed_s": 2.75, "total": -4.482302, "recon0": 0.587974, "recon1": 0.869021, "recon2": 0.0, "kl0": -1.646516, "kl1": -4.29278, "kl2": 0.0}
{"epoch": 68, "phase": 1, "elapsed_s": 2.55, "total": -5.526833, "recon0": 0.622729, "recon1": 0.890526, "recon2": 0.0, "kl0": -2.071015, "kl1": -4.969072, "kl2": 0.0}
{"epoch": 69, "phase": 1, "elapsed_s": 3.0, "total": -6.507378, "recon0": 0.645903, "recon1": 0.90855, "recon2": 0.0, "kl0": -2.415256, "kl1": -5.646574, "kl2": 0.0}
{"epoch": 70, "phase": 1, "elapsed_s": 2.86, "total": -7.490256, "recon0": 0.666269, "recon1": 0.920717, "recon2": 0.0, "kl0": -2.738483, "kl1": -6.338758, "kl2": 0.0}
{"epoch": 71, "phase": 1, "elapsed_s": 2.66, "total": -8.44585, "recon0": 0.68369, "recon1": 0.933173, "recon2": 0.0, "kl0": -3.046624, "kl1": -7.016088, "kl2": 0.0}
{"epoch": 72, "phase": 1, "elapsed_s": 2.79, "total": -9.414274, "recon0": 0.690818, "recon1": 0.935083, "recon2": 0.0, "kl0": -3.344225, "kl1": -7.69595, "kl2": 0.0}
{"epoch": 73, "phase": 1, "elapsed_s": 2.51, "total": -10.365102, "recon0": 0.695199, "recon1": 0.936843, "recon2": 0.0, "kl0": -3.63201, "kl1": -8.365133, "kl2": 0.0}
{"epoch": 74, "phase": 1, "elapsed_s": 2.74, "total": -11.306765, "recon0": 0.696917, "recon1": 0.936801, "recon2": 0.0, "kl0": -3.919634, "kl1": -9.02085, "kl2": 0.0}
{"epoch": 75, "phase": 1, "elapsed_s": 2.56, "total": -12.251445, "recon0": 0.699356, "recon1": 0.93311, "recon2": 0.0, "kl0": -4.201334, "kl1": -9.682578, "kl2": 0.0}
{"epoch": 76, "phase": 1, "elapsed_s": 2.68, "total": -13.180547, "recon0": 0.699638, "recon1": 0.933671, "recon2": 0.0, "kl0": -4.485024, "kl1": -10.328832, "kl2": 0.0}
{"epoch": 77, "phase": 1, "elapsed_s": 2.48, "total": -14.12101, "recon0": 0.700406, "recon1": 0.930246, "recon2": 0.0, "kl0": -4.766513, "kl1": -10.985149, "kl2": 0.0}
{"epoch": 78, "phase": 1, "elapsed_s": 2.86, "total": -15.052918, "recon0": 0.699714, "recon1": 0.933221, "recon2": 0.0, "kl0": -5.048321, "kl1": -11.637532, "kl2": 0.0}
{"epoch": 79, "phase": 1, "elapsed_s": 2.48, "total": -15.982993, "recon0": 0.699179, "recon1": 0.932173, "recon2": 0.0, "kl0": -5.329547, "kl1": -12.284798, "kl2": 0.0}
{"epoch": 80, "phase": 1, "elapsed_s": 2.69, "total": -16.89966, "recon0": 0.700024, "recon1": 0.938398, "recon2": 0.0, "kl0": -5.609677, "kl1": -12.928404, "kl2": 0.0}
{"epoch": 81, "phase": 1, "elapsed_s": 2.51, "total": -17.836191, "recon0": 0.699784, "recon1": 0.940595, "recon2": 0.0, "kl0": -5.891752, "kl1": -13.584817, "kl2": 0.0}
{"epoch": 82, "phase": 1, "elapsed_s": 2.77, "total": -18.779568, "recon0": 0.699181, "recon1": 0.939045, "recon2": 0.0, "kl0": -6.172101, "kl1": -14.245693, "kl2": 0.0}
{"epoch": 83, "phase": 1, "elapsed_s": 2.45, "total": -19.710516, "recon0": 0.69958, "recon1": 0.937045, "recon2": 0.0, "kl0": -6.452611, "kl1": -14.89453, "kl2": 0.0}
{"epoch": 84, "phase": 1, "elapsed_s": 2.68, "total": -20.634763, "recon0": 0.699543, "recon1": 0.941639, "recon2": 0.0, "kl0": -6.733226, "kl1": -15.54272, "kl2": 0.0}
{"epoch": 85, "phase": 1, "elapsed_s": 2.61, "total": -21.572125, "recon0": 0.699099, "recon1": 0.941238, "recon2": 0.0, "kl0": -7.015014, "kl1": -16.197448, "kl2": 0.0}
{"epoch": 86, "phase": 1, "elapsed_s": 2.72, "total": -22.516763, "recon0": 0.69965, "recon1": 0.928537, "recon2": 0.0, "kl0": -7.294792, "kl1": -16.850158, "kl2": 0.0}
{"epoch": 87, "phase": 1, "elapsed_s": 2.48, "total": -22.512785, "recon0": 0.699659, "recon1": 0.94048, "recon2": 0.0, "kl0": -7.294768, "kl1": -16.858158, "kl2": 0.0}
{"epoch": 88, "phase": 1, "elapsed_s": 2.71, "total": -22.502439, "recon0": 0.699073, "recon1": 0.924565, "recon2": 0.0, "kl0": -7.295304, "kl1": -16.830774, "kl2": 0.0}
{"epoch": 89, "phase": 1, "elapsed_s": 2.61, "total": -22.495103, "recon0": 0.699292, "recon1": 0.965425, "recon2": 0.0, "kl0": -7.295694, "kl1": -16.864126, "kl2": 0.0}
{"epoch": 90, "phase": 1, "elapsed_s": 2.67, "total": -22.515237, "recon0": 0.699038, "recon1": 0.934505, "recon2": 0.0, "kl0": -7.295051, "kl1": -16.85373, "kl2": 0.0}
{"epoch": 91, "phase": 1, "elapsed_s": 2.62, "total": -22.521241, "recon0": 0.69934, "recon1": 0.947994, "recon2": 0.0, "kl0": -7.295187, "kl1": -16.873387, "kl2": 0.0}
{"epoch": 92, "phase": 1, "elapsed_s": 2.59, "total": -22.508773, "recon0": 0.699033, "recon1": 0.939016, "recon2": 0.0, "kl0": -7.295233, "kl1": -16.85159, "kl2": 0.0}
{"epoch": 93, "phase": 1, "elapsed_s": 2.83, "total": -22.509582, "recon0": 0.698741, "recon1": 0.952616, "recon2": 0.0, "kl0": -7.296545, "kl1": -16.864394, "kl2": 0.0}
{"epoch": 94, "phase": 1, "elapsed_s": 2.68, "total": -22.543631, "recon0": 0.698903, "recon1": 0.934102, "recon2": 0.0, "kl0": -7.295235, "kl1": -16.881401, "kl2": 0.0}
{"epoch": 95, "phase": 1, "elapsed_s": 2.66, "total": -22.528661, "recon0": 0.699042, "recon1": 0.952859, "recon2": 0.0, "kl0": -7.2964, "kl1": -16.884162, "kl2": 0.0}
{"epoch": 96, "phase": 1, "elapsed_s": 2.85, "total": -22.549058, "recon0": 0.69911, "recon1": 0.945919, "recon2": 0.0, "kl0": -7.296002, "kl1": -16.898085, "kl2": 0.0}
{"epoch": 97, "phase": 1, "elapsed_s": 2.79, "total": -22.535128, "recon0": 0.698933, "recon1": 0.947021, "recon2": 0.0, "kl0": -7.295293, "kl1": -16.885789, "kl2": 0.0}
{"epoch": 98, "phase": 1, "elapsed_s": 2.59, "total": -22.564667, "recon0": 0.698953, "recon1": 0.94668, "recon2": 0.0, "kl0": -7.296599, "kl1": -16.9137, "kl2": 0.0}
{"epoch": 99, "phase": 1, "elapsed_s": 2.8, "total": -22.549364, "recon0": 0.698697, "recon1": 0.933321, "recon2": 0.0, "kl0": -7.297326, "kl1": -16.884056, "kl2": 0.0}
{"epoch": 100, "phase": 1, "elapsed_s": 2.77, "total": -22.534828, "recon0": 0.698886, "recon1": 0.935686, "recon2": 0.0, "kl0": -7.294871, "kl1": -16.874529, "kl2": 0.0}
{"epoch": 101, "phase": 1, "elapsed_s": 2.85, "total": -22.517648, "recon0": 0.699037, "recon1": 0.938377, "recon2": 0.0, "kl0": -7.295284, "kl1": -16.859778, "kl2": 0.0}
{"epoch": 102, "phase": 1, "elapsed_s": 2.62, "total": -22.544663, "recon0": 0.698109, "recon1": 0.948144, "recon2": 0.0, "kl0": -7.29522, "kl1": -16.895696, "kl2": 0.0}
{"epoch": 103, "phase": 1, "elapsed_s": 2.93, "total": -22.53423, "recon0": 0.698451, "recon1": 0.973207, "recon2": 0.0, "kl0": -7.295893, "kl1": -16.909996, "kl2": 0.0}
{"epoch": 104, "phase": 1, "elapsed_s": 2.8, "total": -22.547493, "recon0": 0.698021, "recon1": 0.944516, "recon2": 0.0, "kl0": -7.296163, "kl1": -16.893866, "kl2": 0.0}
{"epoch": 105, "phase": 1, "elapsed_s": 2.68, "total": -22.5344, "recon0": 0.69875, "recon1": 0.963687, "recon2": 0.0, "kl0": -7.297439, "kl1": -16.899398, "kl2": 0.0}
{"epoch": 106, "phase": 1, "elapsed_s": 2.58, "total": -22.552142, "recon0": 0.698737, "recon1": 0.930331, "recon2": 0.0, "kl0": -7.295822, "kl1": -16.885388, "kl2": 0.0}
{"epoch": 107, "phase": 1, "elapsed_s": 2.89, "total": -22.576885, "recon0": 0.698854, "recon1": 0.963703, "recon2": 0.0, "kl0": -7.297778, "kl1": -16.941663, "kl2": 0.0}
{"epoch": 108, "phase": 1, "elapsed_s": 2.71, "total": -22.563186, "recon0": 0.698425, "recon1": 0.942986, "recon2": 0.0, "kl0": -7.297396, "kl1": -16.907202, "kl2": 0.0}
{"epoch": 109, "phase": 1, "elapsed_s": 2.64, "total": -22.565159, "recon0": 0.698114, "recon1": 0.951183, "recon2": 0.0, "kl0": -7.297466, "kl1": -16.91699, "kl2": 0.0}
{"epoch": 110, "phase": 1, "elapsed_s": 2.64, "total": -22.536757, "recon0": 0.698828, "recon1": 0.9537, "recon2": 0.0, "kl0": -7.297516, "kl1": -16.891769, "kl2": 0.0}
{"epoch": 111, "phase": 1, "elapsed_s": 2.83, "total": -22.571948, "recon0": 0.697979, "recon1": 0.974188, "recon2": 0.0, "kl0": -7.297035, "kl1": -16.94708, "kl2": 0.0}
{"epoch": 112, "phase": 1, "elapsed_s": 2.76, "total": -22.561356, "recon0": 0.699151, "recon1": 0.949085, "recon2": 0.0, "kl0": -7.297806, "kl1": -16.911785, "kl2": 0.0}
{"epoch": 113, "phase": 1, "elapsed_s": 2.66, "total": -22.551509, "recon0": 0.698769, "recon1": 0.946191, "recon2": 0.0, "kl0": -7.297203, "kl1": -16.899266, "kl2": 0.0}
{"epoch": 114, "phase": 1, "elapsed_s": 2.94, "total": -22.58796, "recon0": 0.698725, "recon1": 0.944108, "recon2": 0.0, "kl0": -7.298027, "kl1": -16.932765, "kl2": 0.0}
{"epoch": 115, "phase": 1, "elapsed_s": 2.69, "total": -22.568928, "recon0": 0.698135, "recon1": 0.94247, "recon2": 0.0, "kl0": -7.296428, "kl1": -16.913105, "kl2": 0.0}
{"epoch": 116, "phase": 1, "elapsed_s": 2.7, "total": -22.577273, "recon0": 0.698659, "recon1": 0.971864, "recon2": 0.0, "kl0": -7.297817, "kl1": -16.949978, "kl2": 0.0}
{"epoch": 117, "phase": 1, "elapsed_s": 2.82, "total": -22.57663, "recon0": 0.697958, "recon1": 0.945848, "recon2": 0.0, "kl0": -7.297284, "kl1": -16.923152, "kl2": 0.0}
{"epoch": 118, "phase": 1, "elapsed_s": 3.16, "total": -22.585756, "recon0": 0.698455, "recon1": 0.962533, "recon2": 0.0, "kl0": -7.297411, "kl1": -16.949333, "kl2": 0.0}
{"epoch": 119, "phase": 1, "elapsed_s": 2.58, "total": -22.541102, "recon0": 0.698211, "recon1": 0.956221, "recon2": 0.0, "kl0": -7.29763, "kl1": -16.897905, "kl2": 0.0}
{"epoch": 120, "phase": 1, "elapsed_s": 2.63, "total": -22.600273, "recon0": 0.698056, "recon1": 0.933736, "recon2": 0.0, "kl0": -7.296003, "kl1": -16.936062, "kl2": 0.0}
{"epoch": 121, "phase": 1, "elapsed_s": 2.54, "total": -22.557611, "recon0": 0.698177, "recon1": 0.921447, "recon2": 0.0, "kl0": -7.294715, "kl1": -16.882519, "kl2": 0.0}
{"epoch": 122, "phase": 1, "elapsed_s": 3.0, "total": -22.554339, "recon0": 0.698562, "recon1": 0.928976, "recon2": 0.0, "kl0": -7.295357, "kl1": -16.886521, "kl2": 0.0}
{"epoch": 123, "phase": 1, "elapsed_s": 2.57, "total": -22.568189, "recon0": 0.698348, "recon1": 0.953124, "recon2": 0.0, "kl0": -7.295016, "kl1": -16.924644, "kl2": 0.0}
{"epoch": 124, "phase": 1, "elapsed_s": 2.63, "total": -22.58484, "recon0": 0.698826, "recon1": 0.940943, "recon2": 0.0, "kl0": -7.296033, "kl1": -16.928577, "kl2": 0.0}
{"epoch": 125, "phase": 1, "elapsed_s": 2.57, "total": -22.575661, "recon0": 0.698054, "recon1": 0.940276, "recon2": 0.0, "kl0": -7.297038, "kl1": -16.916952, "kl2": 0.0}
{"epoch": 126, "phase": 1, "elapsed_s": 2.81, "total": -22.515017, "recon0": 0.698141, "recon1": 0.933903, "recon2": 0.0, "kl0": -7.296651, "kl1": -16.85041, "kl2": 0.0}
{"epoch": 127, "phase": 1, "elapsed_s": 2.53, "total": -22.510401, "recon0": 0.698175, "recon1": 0.941155, "recon2": 0.0, "kl0": -7.296211, "kl1": -16.853521, "kl2": 0.0}
{"epoch": 128, "phase": 1, "elapsed_s": 2.63, "total": -22.539165, "recon0": 0.698273, "recon1": 0.92619, "recon2": 0.0, "kl0": -7.296936, "kl1": -16.866692, "kl2": 0.0}
{"epoch": 129, "phase": 1, "elapsed_s": 2.63, "total": -22.52548, "recon0": 0.697989, "recon1": 0.943638, "recon2": 0.0, "kl0": -7.295674, "kl1": -16.871433, "kl2": 0.0}
{"epoch": 130, "phase": 1, "elapsed_s": 2.6, "total": -22.576777, "recon0": 0.69813, "recon1": 0.96702, "recon2": 0.0, "kl0": -7.296697, "kl1": -16.94523, "kl2": 0.0}
{"epoch": 131, "phase": 1, "elapsed_s": 2.54, "total": -22.542455, "recon0": 0.698431, "recon1": 0.949088, "recon2": 0.0, "kl0": -7.296767, "kl1": -16.893207, "kl2": 0.0}
{"epoch": 132, "phase": 1, "elapsed_s": 2.72, "total": -22.561647, "recon0": 0.698674, "recon1": 0.939834, "recon2": 0.0, "kl0": -7.298199, "kl1": -16.901956, "kl2": 0.0}
{"epoch": 133, "phase": 1, "elapsed_s": 2.69, "total": -22.590657, "recon0": 0.698102, "recon1": 0.973343, "recon2": 0.0, "kl0": -7.297769, "kl1": -16.964333, "kl2": 0.0}
{"epoch": 134, "phase": 1, "elapsed_s": 2.74, "total": -22.595478, "recon0": 0.698237, "recon1": 0.940727, "recon2": 0.0, "kl0": -7.298696, "kl1": -16.935745, "kl2": 0.0}
{"epoch": 135, "phase": 1, "elapsed_s": 2.72, "total": -22.556733, "recon0": 0.698305, "recon1": 0.941065, "recon2": 0.0, "kl0": -7.297486, "kl1": -16.898616, "kl2": 0.0}
{"epoch": 136, "phase": 1, "elapsed_s": 2.5, "total": -22.547829, "recon0": 0.698234, "recon1": 0.980018, "recon2": 0.0, "kl0": -7.297943, "kl1": -16.928137, "kl2": 0.0}
{"epoch": 137, "phase": 1, "elapsed_s": 2.83, "total": -22.540444, "recon0": 0.697993, "recon1": 0.967976, "recon2": 0.0, "kl0": -7.298649, "kl1": -16.907764, "kl2": 0.0}
{"epoch": 138, "phase": 1, "elapsed_s": 2.62, "total": -22.581756, "recon0": 0.698202, "recon1": 0.987401, "recon2": 0.0, "kl0": -7.298696, "kl1": -16.968663, "kl2": 0.0}
{"epoch": 139, "phase": 1, "elapsed_s": 2.64, "total": -22.600094, "recon0": 0.697864, "recon1": 0.97283, "recon2": 0.0, "kl0": -7.297174, "kl1": -16.973614, "kl2": 0.0}
{"epoch": 140, "phase": 1, "elapsed_s": 2.9, "total": -22.611942, "recon0": 0.697902, "recon1": 0.970237, "recon2": 0.0, "kl0": -7.297302, "kl1": -16.982778, "kl2": 0.0}
{"epoch": 141, "phase": 2, "elapsed_s": 3.21, "total": 1.671626, "recon0": 0.655255, "recon1": 0.969567, "recon2": 1.488075, "kl0": -0.258749, "kl1": -0.652416, "kl2": -0.530106}
{"epoch": 142, "phase": 2, "elapsed_s": 2.95, "total": -0.361147, "recon0": 0.540897, "recon1": 0.969224, "recon2": 1.053244, "kl0": -0.45438, "kl1": -1.304523, "kl2": -1.165608}
{"epoch": 143, "phase": 2, "elapsed_s": 3.0, "total": -1.974868, "recon0": 0.500912, "recon1": 0.960665, "recon2": 0.999908, "kl0": -0.65655, "kl1": -1.956123, "kl2": -1.823679}
{"epoch": 144, "phase": 2, "elapsed_s": 3.11, "total": -3.570859, "recon0": 0.50496, "recon1": 0.948123, "recon2": 0.979814, "kl0": -0.90709, "kl1": -2.605298, "kl2": -2.491369}
{"epoch": 145, "phase": 2, "elapsed_s": 2.86, "total": -5.153226, "recon0": 0.527968, "recon1": 0.94134, "recon2": 0.973528, "kl0": -1.186914, "kl1": -3.251588, "kl2": -3.15756}
{"epoch": 146, "phase": 2, "elapsed_s": 2.9, "total": -6.762002, "recon0": 0.556558, "recon1": 0.927196, "recon2": 0.974243, "kl0": -1.486237, "kl1": -3.907897, "kl2": -3.825865}
{"epoch": 147, "phase": 2, "elapsed_s": 2.81, "total": -8.359443, "recon0": 0.58316, "recon1": 0.933016, "recon2": 0.972692, "kl0": -1.802416, "kl1": -4.55222, "kl2": -4.493676}
{"epoch": 148, "phase": 2, "elapsed_s": 3.0, "total": -9.976358, "recon0": 0.608996, "recon1": 0.918249, "recon2": 0.975678, "kl0": -2.115078, "kl1": -5.205395, "kl2": -5.158808}
{"epoch": 149, "phase": 2, "elapsed_s": 2.7, "total": -11.572782, "recon0": 0.630656, "recon1": 0.918623, "recon2": 0.983954, "kl0": -2.425082, "kl1": -5.855749, "kl2": -5.825183}
{"epoch": 150, "phase": 2, "elapsed_s": 2.92, "total": -13.183249, "recon0": 0.651847, "recon1": 0.920311, "recon2": 0.987223, "kl0": -2.735013, "kl1": -6.515043, "kl2": -6.492573}
{"epoch": 151, "phase": 2, "elapsed_s": 2.97, "total": -14.77976, "recon0": 0.669025, "recon1": 0.920059, "recon2": 0.992476, "kl0": -3.04134, "kl1": -7.1624, "kl2": -7.157579}
{"epoch": 152, "phase": 2, "elapsed_s": 2.83, "total": -16.379973, "recon0": 0.681205, "recon1": 0.921924, "recon2": 0.996351, "kl0": -3.339428, "kl1": -7.815764, "kl2": -7.824262}
{"epoch": 153, "phase": 2, "elapsed_s": 2.89, "total": -17.941027, "recon0": 0.688905, "recon1": 0.941461, "recon2": 1.000925, "kl0": -3.633597, "kl1": -8.450716, "kl2": -8.488006}
{"epoch": 154, "phase": 2, "elapsed_s": 3.01, "total": -19.546547, "recon0": 0.692575, "recon1": 0.957137, "recon2": 1.002725, "kl0": -3.919726, "kl1": -9.129161, "kl2": -9.150097}
{"epoch": 155, "phase": 2, "elapsed_s": 3.06, "total": -21.147993, "recon0": 0.697584, "recon1": 0.953546, "recon2": 1.003742, "kl0": -4.204823, "kl1": -9.786724, "kl2": -9.811318}
{"epoch": 156, "phase": 2, "elapsed_s": 2.9, "total": -22.743297, "recon0": 0.698872, "recon1": 0.931535, "recon2": 1.003867, "kl0": -4.490099, "kl1": -10.417056, "kl2": -10.470416}
{"epoch": 157, "phase": 2, "elapsed_s": 3.03, "total": -24.32319, "recon0": 0.699201, "recon1": 0.928163, "recon2": 1.002873, "kl0": -4.77274, "kl1": -11.051614, "kl2": -11.129073}
{"epoch": 158, "phase": 2, "elapsed_s": 2.99, "total": -25.923716, "recon0": 0.698328, "recon1": 0.933273, "recon2": 1.002446, "kl0": -5.051649, "kl1": -11.718859, "kl2": -11.787255}
{"epoch": 159, "phase": 2, "elapsed_s": 2.92, "total": -27.520958, "recon0": 0.698588, "recon1": 0.933459, "recon2": 1.00235, "kl0": -5.33344, "kl1": -12.377499, "kl2": -12.444415}
{"epoch": 160, "phase": 2, "elapsed_s": 3.1, "total": -29.113961, "recon0": 0.697903, "recon1": 0.933099, "recon2": 1.002023, "kl0": -5.613713, "kl1": -13.031933, "kl2": -13.10134}
{"epoch": 161, "phase": 2, "elapsed_s": 2.97, "total": -30.681476, "recon0": 0.697678, "recon1": 0.944307, "recon2": 1.001723, "kl0": -5.895099, "kl1": -13.672867, "kl2": -13.757219}
{"epoch": 162, "phase": 2, "elapsed_s": 3.05, "total": -32.289622, "recon0": 0.697672, "recon1": 0.939692, "recon2": 1.001248, "kl0": -6.17542, "kl1": -14.339734, "kl2": -14.413079}
{"epoch": 163, "phase": 2, "elapsed_s": 3.01, "total": -33.884651, "recon0": 0.697172, "recon1": 0.931534, "recon2": 1.000937, "kl0": -6.455853, "kl1": -14.989775, "kl2": -15.068667}
{"epoch": 164, "phase": 2, "elapsed_s": 3.24, "total": -35.469328, "recon0": 0.698167, "recon1": 0.939489, "recon2": 1.000846, "kl0": -6.737119, "kl1": -15.646385, "kl2": -15.724326}
{"epoch": 165, "phase": 2, "elapsed_s": 3.08, "total": -37.068317, "recon0": 0.697612, "recon1": 0.931558, "recon2": 1.000934, "kl0": -7.017703, "kl1": -16.301263, "kl2": -16.379456}
{"epoch": 166, "phase": 2, "elapsed_s": 2.91, "total": -38.647744, "recon0": 0.697477, "recon1": 0.926427, "recon2": 1.000821, "kl0": -7.29881, "kl1": -16.938834, "kl2": -17.034826}
{"epoch": 167, "phase": 2, "elapsed_s": 3.16, "total": -38.644345, "recon0": 0.697563, "recon1": 0.931897, "recon2": 1.000705, "kl0": -7.298513, "kl1": -16.941034, "kl2": -17.034963}
{"epoch": 168, "phase": 2, "elapsed_s": 2.86, "total": -38.624038, "recon0": 0.697431, "recon1": 0.948439, "recon2": 1.000515, "kl0": -7.298657, "kl1": -16.93685, "kl2": -17.034914}
{"epoch": 169, "phase": 2, "elapsed_s": 2.97, "total": -38.642562, "recon0": 0.697557, "recon1": 0.965059, "recon2": 1.000678, "kl0": -7.298401, "kl1": -16.972417, "kl2": -17.035037}
{"epoch": 170, "phase": 2, "elapsed_s": 3.08, "total": -38.655313, "recon0": 0.697337, "recon1": 0.958772, "recon2": 1.000451, "kl0": -7.298486, "kl1": -16.978215, "kl2": -17.035171}
{"epoch": 171, "phase": 2, "elapsed_s": 2.88, "total": -38.655851, "recon0": 0.697304, "recon1": 0.960978, "recon2": 1.000439, "kl0": -7.29878, "kl1": -16.980742, "kl2": -17.03505}
{"epoch": 172, "phase": 2, "elapsed_s": 3.0, "total": -38.657648, "recon0": 0.697529, "recon1": 0.95035, "recon2": 1.000391, "kl0": -7.299944, "kl1": -16.970768, "kl2": -17.035206}
{"epoch": 173, "phase": 2, "elapsed_s": 2.88, "total": -38.64251, "recon0": 0.697515, "recon1": 0.921252, "recon2": 1.000318, "kl0": -7.299102, "kl1": -16.927297, "kl2": -17.035196}
{"epoch": 174, "phase": 2, "elapsed_s": 3.12, "total": -38.633697, "recon0": 0.697302, "recon1": 0.969825, "recon2": 1.000331, "kl0": -7.299259, "kl1": -16.966744, "kl2": -17.035152}
{"epoch": 175, "phase": 2, "elapsed_s": 2.97, "total": -38.652655, "recon0": 0.697659, "recon1": 0.957246, "recon2": 1.000196, "kl0": -7.29984, "kl1": -16.972754, "kl2": -17.035161}
{"epoch": 176, "phase": 2, "elapsed_s": 2.98, "total": -38.62436, "recon0": 0.697466, "recon1": 0.944832, "recon2": 1.00026, "kl0": -7.29716, "kl1": -16.934562, "kl2": -17.035196}
{"epoch": 177, "phase": 2, "elapsed_s": 3.22, "total": -38.623011, "recon0": 0.697428, "recon1": 0.958133, "recon2": 1.000169, "kl0": -7.298883, "kl1": -16.944562, "kl2": -17.035296}
{"epoch": 178, "phase": 2, "elapsed_s": 2.83, "total": -38.656975, "recon0": 0.697295, "recon1": 0.958404, "recon2": 1.000174, "kl0": -7.299449, "kl1": -16.978192, "kl2": -17.035208}
{"epoch": 179, "phase": 2, "elapsed_s": 3.1, "total": -38.661275, "recon0": 0.697583, "recon1": 0.953954, "recon2": 1.00011, "kl0": -7.298378, "kl1": -16.979299, "kl2": -17.035246}
{"epoch": 180, "phase": 2, "elapsed_s": 2.97, "total": -38.655649, "recon0": 0.697721, "recon1": 0.963923, "recon2": 1.000216, "kl0": -7.299709, "kl1": -16.98253, "kl2": -17.035269}
{"epoch": 181, "phase": 2, "elapsed_s": 2.98, "total": -38.658292, "recon0": 0.697412, "recon1": 0.961175, "recon2": 1.000187, "kl0": -7.299374, "kl1": -16.982415, "kl2": -17.035277}
{"epoch": 182, "phase": 2, "elapsed_s": 2.91, "total": -38.657289, "recon0": 0.697303, "recon1": 0.961094, "recon2": 1.00015, "kl0": -7.299029, "kl1": -16.981467, "kl2": -17.035341}
{"epoch": 183, "phase": 2, "elapsed_s": 3.03, "total": -38.661516, "recon0": 0.697461, "recon1": 0.955227, "recon2": 1.000157, "kl0": -7.298931, "kl1": -16.980124, "kl2": -17.035306}
{"epoch": 184, "phase": 2, "elapsed_s": 3.18, "total": -38.660817, "recon0": 0.697459, "recon1": 0.958033, "recon2": 1.000107, "kl0": -7.299294, "kl1": -16.981724, "kl2": -17.035398}
{"epoch": 185, "phase": 2, "elapsed_s": 2.89, "total": -38.664146, "recon0": 0.6972, "recon1": 0.956741, "recon2": 1.000129, "kl0": -7.298147, "kl1": -16.98482, "kl2": -17.035249}
{"epoch": 186, "phase": 2, "elapsed_s": 3.1, "total": -38.659967, "recon0": 0.69756, "recon1": 0.960808, "recon2": 1.000099, "kl0": -7.299086, "kl1": -16.983979, "kl2": -17.035368}
{"epoch": 187, "phase": 2, "elapsed_s": 3.19, "total": -38.6526, "recon0": 0.697522, "recon1": 0.954463, "recon2": 1.000043, "kl0": -7.297341, "kl1": -16.971918, "kl2": -17.035369}
{"epoch": 188, "phase": 2, "elapsed_s": 2.9, "total": -38.625063, "recon0": 0.697277, "recon1": 0.937186, "recon2": 1.000102, "kl0": -7.299049, "kl1": -16.92523, "kl2": -17.035349}
{"epoch": 189, "phase": 2, "elapsed_s": 3.01, "total": -38.649774, "recon0": 0.697384, "recon1": 0.938209, "recon2": 1.000066, "kl0": -7.299461, "kl1": -16.950623, "kl2": -17.035348}
{"epoch": 190, "phase": 2, "elapsed_s": 3.03, "total": -38.639762, "recon0": 0.697157, "recon1": 0.932993, "recon2": 1.000061, "kl0": -7.298328, "kl1": -16.936229, "kl2": -17.035417}
{"epoch": 191, "phase": 2, "elapsed_s": 3.1, "total": -38.64432, "recon0": 0.697487, "recon1": 0.931817, "recon2": 1.000077, "kl0": -7.297698, "kl1": -16.940711, "kl2": -17.035293}
{"epoch": 192, "phase": 2, "elapsed_s": 2.88, "total": -38.659689, "recon0": 0.696985, "recon1": 0.944537, "recon2": 1.000114, "kl0": -7.299986, "kl1": -16.966024, "kl2": -17.035315}
{"epoch": 193, "phase": 2, "elapsed_s": 2.99, "total": -38.662268, "recon0": 0.696877, "recon1": 0.959826, "recon2": 1.000068, "kl0": -7.298814, "kl1": -16.984894, "kl2": -17.035331}
{"epoch": 194, "phase": 2, "elapsed_s": 3.16, "total": -38.621081, "recon0": 0.697222, "recon1": 0.941506, "recon2": 1.000058, "kl0": -7.297955, "kl1": -16.926558, "kl2": -17.035353}
{"epoch": 195, "phase": 2, "elapsed_s": 3.02, "total": -38.629305, "recon0": 0.69726, "recon1": 0.962823, "recon2": 1.000025, "kl0": -7.298839, "kl1": -16.955208, "kl2": -17.035366}
{"epoch": 196, "phase": 2, "elapsed_s": 3.03, "total": -38.657136, "recon0": 0.697302, "recon1": 0.963072, "recon2": 1.000066, "kl0": -7.298546, "kl1": -16.983712, "kl2": -17.035318}
{"epoch": 197, "phase": 2, "elapsed_s": 3.06, "total": -38.654723, "recon0": 0.697361, "recon1": 0.96786, "recon2": 1.000069, "kl0": -7.298994, "kl1": -16.985678, "kl2": -17.035341}
{"epoch": 198, "phase": 2, "elapsed_s": 2.92, "total": -38.642206, "recon0": 0.697132, "recon1": 0.98112, "recon2": 1.000035, "kl0": -7.299799, "kl1": -16.985293, "kl2": -17.035401}
{"epoch": 199, "phase": 2, "elapsed_s": 2.94, "total": -38.64055, "recon0": 0.697355, "recon1": 0.960747, "recon2": 1.000023, "kl0": -7.299771, "kl1": -16.963534, "kl2": -17.035371}
{"epoch": 200, "phase": 2, "elapsed_s": 3.09, "total": -38.638645, "recon0": 0.697387, "recon1": 0.940058, "recon2": 1.000024, "kl0": -7.298305, "kl1": -16.94246, "kl2": -17.035349}
{"epoch": 201, "phase": 2, "elapsed_s": 3.16, "total": -38.645281, "recon0": 0.697635, "recon1": 0.963598, "recon2": 1.000056, "kl0": -7.299814, "kl1": -16.971408, "kl2": -17.035349}
{"epoch": 202, "phase": 2, "elapsed_s": 2.82, "total": -38.66118, "recon0": 0.697299, "recon1": 0.96029, "recon2": 1.00002, "kl0": -7.29934, "kl1": -16.98409, "kl2": -17.035358}
{"epoch": 203, "phase": 2, "elapsed_s": 3.15, "total": -38.642728, "recon0": 0.696992, "recon1": 0.970511, "recon2": 1.000026, "kl0": -7.297931, "kl1": -16.97695, "kl2": -17.035375}
{"epoch": 204, "phase": 2, "elapsed_s": 2.91, "total": -38.653967, "recon0": 0.69747, "recon1": 0.966204, "recon2": 1.000016, "kl0": -7.298452, "kl1": -16.983853, "kl2": -17.035353}
{"epoch": 205, "phase": 2, "elapsed_s": 2.92, "total": -38.645944, "recon0": 0.697341, "recon1": 0.971349, "recon2": 1.00003, "kl0": -7.299046, "kl1": -16.980246, "kl2": -17.035372}
{"epoch": 221, "phase": 3, "elapsed_s": 2.9, "total": 0.578004, "recon0": 0.683215, "recon1": 0.953823, "recon2": 1.000064, "kl0": -0.358509, "kl1": -0.848828, "kl2": -0.85176}
{"epoch": 222, "phase": 3, "elapsed_s": 3.15, "total": -1.511669, "recon0": 0.583109, "recon1": 0.954665, "recon2": 1.000032, "kl0": -0.650985, "kl1": -1.694999, "kl2": -1.703491}
{"epoch": 223, "phase": 3, "elapsed_s": 3.09, "total": -3.555577, "recon0": 0.543935, "recon1": 0.926833, "recon2": 1.000025, "kl0": -0.94134, "kl1": -2.529749, "kl2": -2.555281}
{"epoch": 224, "phase": 3, "elapsed_s": 2.88, "total": -5.602556, "recon0": 0.558553, "recon1": 0.91122, "recon2": 1.000019, "kl0": -1.282878, "kl1": -3.382386, "kl2": -3.407084}
{"epoch": 225, "phase": 3, "elapsed_s": 3.07, "total": -7.673471, "recon0": 0.587636, "recon1": 0.911463, "recon2": 1.000011, "kl0": -1.67974, "kl1": -4.233979, "kl2": -4.258862}
{"epoch": 226, "phase": 3, "elapsed_s": 2.89, "total": -9.737106, "recon0": 0.619345, "recon1": 0.908999, "recon2": 1.000035, "kl0": -2.082411, "kl1": -5.072475, "kl2": -5.110599}
{"epoch": 227, "phase": 3, "elapsed_s": 2.85, "total": -11.786613, "recon0": 0.64671, "recon1": 0.926756, "recon2": 1.000035, "kl0": -2.484572, "kl1": -5.913173, "kl2": -5.962369}
{"epoch": 228, "phase": 3, "elapsed_s": 2.95, "total": -13.85122, "recon0": 0.667713, "recon1": 0.945754, "recon2": 1.000023, "kl0": -2.873778, "kl1": -6.77681, "kl2": -6.814123}
{"epoch": 229, "phase": 3, "elapsed_s": 2.97, "total": -15.926163, "recon0": 0.678209, "recon1": 0.9446, "recon2": 1.000025, "kl0": -3.255653, "kl1": -7.627426, "kl2": -7.665919}
{"epoch": 230, "phase": 3, "elapsed_s": 3.38, "total": -17.990394, "recon0": 0.688599, "recon1": 0.922017, "recon2": 1.000013, "kl0": -3.631722, "kl1": -8.451639, "kl2": -8.517662}
{"epoch": 231, "phase": 3, "elapsed_s": 2.97, "total": -20.05363, "recon0": 0.692545, "recon1": 0.929011, "recon2": 1.00003, "kl0": -4.004439, "kl1": -9.301344, "kl2": -9.369433}
{"epoch": 232, "phase": 3, "elapsed_s": 3.19, "total": -22.139363, "recon0": 0.696991, "recon1": 0.914097, "recon2": 1.000022, "kl0": -4.375071, "kl1": -10.154198, "kl2": -10.221204}
{"epoch": 233, "phase": 3, "elapsed_s": 2.74, "total": -24.166406, "recon0": 0.697202, "recon1": 0.918354, "recon2": 1.000018, "kl0": -4.742771, "kl1": -10.966177, "kl2": -11.073032}
{"epoch": 234, "phase": 3, "elapsed_s": 2.89, "total": -26.257676, "recon0": 0.697616, "recon1": 0.94674, "recon2": 1.000008, "kl0": -5.108427, "kl1": -11.86887, "kl2": -11.924743}
{"epoch": 235, "phase": 3, "elapsed_s": 3.21, "total": -28.325158, "recon0": 0.697958, "recon1": 0.951478, "recon2": 0.999996, "kl0": -5.473576, "kl1": -12.724432, "kl2": -12.776582}
{"epoch": 236, "phase": 3, "elapsed_s": 2.96, "total": -30.387442, "recon0": 0.697798, "recon1": 0.953353, "recon2": 1.000031, "kl0": -5.838439, "kl1": -13.57188, "kl2": -13.628305}
{"epoch": 237, "phase": 3, "elapsed_s": 2.86, "total": -32.45829, "recon0": 0.697736, "recon1": 0.947966, "recon2": 1.000027, "kl0": -6.203299, "kl1": -14.420664, "kl2": -14.480056}
{"epoch": 238, "phase": 3, "elapsed_s": 3.04, "total": -34.509856, "recon0": 0.69741, "recon1": 0.95943, "recon2": 1.000022, "kl0": -6.567882, "kl1": -15.266967, "kl2": -15.33187}
{"epoch": 239, "phase": 3, "elapsed_s": 2.91, "total": -36.590951, "recon0": 0.697447, "recon1": 0.947785, "recon2": 1.000037, "kl0": -6.934759, "kl1": -16.117864, "kl2": -16.183598}
{"epoch": 240, "phase": 3, "elapsed_s": 2.78, "total": -38.654125, "recon0": 0.697201, "recon1": 0.955148, "recon2": 1.000019, "kl0": -7.298895, "kl1": -16.972213, "kl2": -17.035384}
{"epoch": 241, "phase": 3, "elapsed_s": 2.79, "total": -38.658259, "recon0": 0.697061, "recon1": 0.94497, "recon2": 1.000026, "kl0": -7.298201, "kl1": -16.966735, "kl2": -17.03538}
{"epoch": 242, "phase": 3, "elapsed_s": 3.06, "total": -38.659279, "recon0": 0.697367, "recon1": 0.948459, "recon2": 1.000018, "kl0": -7.299701, "kl1": -16.97002, "kl2": -17.035402}
{"epoch": 243, "phase": 3, "elapsed_s": 2.73, "total": -38.654524, "recon0": 0.697322, "recon1": 0.949876, "recon2": 1.000022, "kl0": -7.299821, "kl1": -16.96653, "kl2": -17.035392}
{"epoch": 244, "phase": 3, "elapsed_s": 2.93, "total": -38.643778, "recon0": 0.697237, "recon1": 0.953351, "recon2": 1.000024, "kl0": -7.299924, "kl1": -16.959078, "kl2": -17.035387}
{"epoch": 245, "phase": 3, "elapsed_s": 3.01, "total": -38.618723, "recon0": 0.697386, "recon1": 0.977086, "recon2": 1.000035, "kl0": -7.299544, "kl1": -16.958299, "kl2": -17.035387}
{"epoch": 246, "phase": 3, "elapsed_s": 3.02, "total": -38.625369, "recon0": 0.697323, "recon1": 0.979142, "recon2": 1.000002, "kl0": -7.298533, "kl1": -16.967917, "kl2": -17.035385}
{"epoch": 247, "phase": 3, "elapsed_s": 2.74, "total": -38.622153, "recon0": 0.697508, "recon1": 0.978482, "recon2": 1.000004, "kl0": -7.299051, "kl1": -16.963708, "kl2": -17.035387}
{"epoch": 248, "phase": 3, "elapsed_s": 2.84, "total": -38.62692, "recon0": 0.697102, "recon1": 0.986603, "recon2": 1.000024, "kl0": -7.299653, "kl1": -16.975606, "kl2": -17.03539}
{"epoch": 249, "phase": 3, "elapsed_s": 2.96, "total": -38.636396, "recon0": 0.697507, "recon1": 0.985464, "recon2": 1.000026, "kl0": -7.299698, "kl1": -16.984327, "kl2": -17.035367}
{"epoch": 250, "phase": 3, "elapsed_s": 2.89, "total": -38.624573, "recon0": 0.697258, "recon1": 0.984266, "recon2": 1.000017, "kl0": -7.299695, "kl1": -16.97103, "kl2": -17.035389}
{"epoch": 251, "phase": 3, "elapsed_s": 2.89, "total": -38.624566, "recon0": 0.697247, "recon1": 0.98811, "recon2": 1.000013, "kl0": -7.299876, "kl1": -16.974685, "kl2": -17.035376}
{"epoch": 252, "phase": 3, "elapsed_s": 3.0, "total": -38.635577, "recon0": 0.697467, "recon1": 0.984038, "recon2": 1.000005, "kl0": -7.299523, "kl1": -16.982165, "kl2": -17.035399}
{"epoch": 253, "phase": 3, "elapsed_s": 2.82, "total": -38.640725, "recon0": 0.697414, "recon1": 0.985166, "recon2": 1.000009, "kl0": -7.299364, "kl1": -16.988548, "kl2": -17.035401}
{"epoch": 254, "phase": 3, "elapsed_s": 2.83, "total": -38.642238, "recon0": 0.697188, "recon1": 0.98575, "recon2": 1.000019, "kl0": -7.299807, "kl1": -16.990005, "kl2": -17.035384}
{"epoch": 255, "phase": 3, "elapsed_s": 2.95, "total": -38.641054, "recon0": 0.697259, "recon1": 0.983436, "recon2": 1.000017, "kl0": -7.299016, "kl1": -16.98737, "kl2": -17.035381}
{"epoch": 256, "phase": 3, "elapsed_s": 2.89, "total": -38.637622, "recon0": 0.697184, "recon1": 0.983699, "recon2": 1.00003, "kl0": -7.299709, "kl1": -16.983436, "kl2": -17.03539}
{"epoch": 257, "phase": 3, "elapsed_s": 2.98, "total": -38.64058, "recon0": 0.697072, "recon1": 0.983793, "recon2": 1.000024, "kl0": -7.299431, "kl1": -16.986666, "kl2": -17.035373}
{"epoch": 258, "phase": 3, "elapsed_s": 3.14, "total": -38.650012, "recon0": 0.697092, "recon1": 0.980382, "recon2": 1.000003, "kl0": -7.299314, "kl1": -16.992774, "kl2": -17.035401}
{"epoch": 259, "phase": 3, "elapsed_s": 3.04, "total": -38.658965, "recon0": 0.697251, "recon1": 0.980628, "recon2": 0.999999, "kl0": -7.300117, "kl1": -17.001324, "kl2": -17.035402}
{"epoch": 260, "phase": 3, "elapsed_s": 2.9, "total": -38.648161, "recon0": 0.697144, "recon1": 0.981451, "recon2": 1.000031, "kl0": -7.299365, "kl1": -16.992029, "kl2": -17.035392}
{"epoch": 261, "phase": 3, "elapsed_s": 2.9, "total": -38.653211, "recon0": 0.697179, "recon1": 0.966256, "recon2": 1.000005, "kl0": -7.299455, "kl1": -16.981796, "kl2": -17.035401}
{"epoch": 262, "phase": 3, "elapsed_s": 2.95, "total": -38.626724, "recon0": 0.697207, "recon1": 0.962443, "recon2": 1.000015, "kl0": -7.299714, "kl1": -16.951284, "kl2": -17.035391}
{"epoch": 281, "phase": 4, "elapsed_s": 3.89, "total": 1.568994, "recon0": 0.693499, "recon1": 0.947326, "recon2": 1.000015, "kl0": -0.188474, "kl1": -0.440658, "kl2": -0.442713}
{"epoch": 282, "phase": 4, "elapsed_s": 3.71, "total": 0.491498, "recon0": 0.658598, "recon1": 0.954929, "recon2": 1.000009, "kl0": -0.356572, "kl1": -0.880035, "kl2": -0.885431}
{"epoch": 283, "phase": 4, "elapsed_s": 4.0, "total": -0.592675, "recon0": 0.594064, "recon1": 0.942773, "recon2": 1.000011, "kl0": -0.481986, "kl1": -1.319369, "kl2": -1.328168}
{"epoch": 284, "phase": 4, "elapsed_s": 4.1, "total": -1.648798, "recon0": 0.581671, "recon1": 0.943071, "recon2": 0.999993, "kl0": -0.642452, "kl1": -1.760196, "kl2": -1.770886}
{"epoch": 285, "phase": 4, "elapsed_s": 4.14, "total": -2.699906, "recon0": 0.588747, "recon1": 0.940156, "recon2": 1.000009, "kl0": -0.81323, "kl1": -2.201979, "kl2": -2.213609}
{"epoch": 286, "phase": 4, "elapsed_s": 3.87, "total": -3.75732, "recon0": 0.593007, "recon1": 0.939351, "recon2": 1.000007, "kl0": -0.990753, "kl1": -2.642587, "kl2": -2.656345}
{"epoch": 287, "phase": 4, "elapsed_s": 3.91, "total": -4.808199, "recon0": 0.602536, "recon1": 0.941, "recon2": 1.000007, "kl0": -1.167965, "kl1": -3.084708, "kl2": -3.099069}
{"epoch": 288, "phase": 4, "elapsed_s": 4.06, "total": -5.87697, "recon0": 0.607726, "recon1": 0.939643, "recon2": 1.000008, "kl0": -1.355443, "kl1": -3.527134, "kl2": -3.541771}
{"epoch": 289, "phase": 4, "elapsed_s": 3.66, "total": -6.947839, "recon0": 0.618321, "recon1": 0.938686, "recon2": 1.000009, "kl0": -1.552108, "kl1": -3.96823, "kl2": -3.984516}
{"epoch": 290, "phase": 4, "elapsed_s": 3.65, "total": -8.02493, "recon0": 0.631577, "recon1": 0.940126, "recon2": 0.999996, "kl0": -1.760842, "kl1": -4.408556, "kl2": -4.42723}
{"epoch": 291, "phase": 4, "elapsed_s": 3.72, "total": -9.111374, "recon0": 0.641746, "recon1": 0.939954, "recon2": 1.000004, "kl0": -1.9735, "kl1": -4.84961, "kl2": -4.869968}
{"epoch": 292, "phase": 4, "elapsed_s": 3.6, "total": -10.19833, "recon0": 0.651906, "recon1": 0.939247, "recon2": 1.000006, "kl0": -2.185887, "kl1": -5.290919, "kl2": -5.312684}
{"epoch": 293, "phase": 4, "elapsed_s": 3.77, "total": -11.275475, "recon0": 0.662185, "recon1": 0.941353, "recon2": 1.000011, "kl0": -2.393798, "kl1": -5.729819, "kl2": -5.755407}
{"epoch": 294, "phase": 4, "elapsed_s": 3.54, "total": -12.366956, "recon0": 0.671937, "recon1": 0.937899, "recon2": 0.999992, "kl0": -2.604576, "kl1": -6.174081, "kl2": -6.198127}
{"epoch": 295, "phase": 4, "elapsed_s": 3.55, "total": -13.466951, "recon0": 0.677389, "recon1": 0.920398, "recon2": 1.00002, "kl0": -2.811491, "kl1": -6.612414, "kl2": -6.640854}
{"epoch": 296, "phase": 4, "elapsed_s": 3.8, "total": -14.558737, "recon0": 0.682467, "recon1": 0.90166, "recon2": 1.000012, "kl0": -3.011368, "kl1": -7.047936, "kl2": -7.083573}
{"epoch": 297, "phase": 4, "elapsed_s": 3.52, "total": -15.627728, "recon0": 0.686995, "recon1": 0.909203, "recon2": 1.000002, "kl0": -3.207996, "kl1": -7.48962, "kl2": -7.526311}
{"epoch": 298, "phase": 4, "elapsed_s": 3.61, "total": -16.699343, "recon0": 0.691447, "recon1": 0.909158, "recon2": 1.000008, "kl0": -3.402371, "kl1": -7.928561, "kl2": -7.969025}
{"epoch": 299, "phase": 4, "elapsed_s": 3.66, "total": -17.775625, "recon0": 0.693403, "recon1": 0.914712, "recon2": 1.00001, "kl0": -3.596156, "kl1": -8.375855, "kl2": -8.41174}
{"epoch": 300, "phase": 4, "elapsed_s": 3.45, "total": -18.843805, "recon0": 0.695437, "recon1": 0.914954, "recon2": 1.000012, "kl0": -3.789835, "kl1": -8.80989, "kl2": -8.854483}
{"epoch": 301, "phase": 4, "elapsed_s": 3.84, "total": -19.923397, "recon0": 0.696971, "recon1": 0.906998, "recon2": 1.000001, "kl0": -3.979459, "kl1": -9.250697, "kl2": -9.297211}
{"epoch": 302, "phase": 4, "elapsed_s": 3.84, "total": -21.001681, "recon0": 0.696564, "recon1": 0.907054, "recon2": 1.000007, "kl0": -4.171324, "kl1": -9.694054, "kl2": -9.739928}
{"epoch": 303, "phase": 4, "elapsed_s": 3.78, "total": -22.071152, "recon0": 0.697238, "recon1": 0.912101, "recon2": 1.000014, "kl0": -4.362586, "kl1": -10.135274, "kl2": -10.182645}
{"epoch": 304, "phase": 4, "elapsed_s": 3.72, "total": -23.136086, "recon0": 0.697386, "recon1": 0.913262, "recon2": 1.000004, "kl0": -4.553108, "kl1": -10.568258, "kl2": -10.625372}
{"epoch": 305, "phase": 4, "elapsed_s": 3.56, "total": -24.214757, "recon0": 0.697235, "recon1": 0.91282, "recon2": 1.000008, "kl0": -4.740853, "kl1": -11.015869, "kl2": -11.068098}
{"epoch": 306, "phase": 4, "elapsed_s": 3.75, "total": -25.294484, "recon0": 0.697401, "recon1": 0.912665, "recon2": 1.000012, "kl0": -4.931934, "kl1": -11.461807, "kl2": -11.510822}
{"epoch": 307, "phase": 4, "elapsed_s": 3.89, "total": -26.362134, "recon0": 0.697572, "recon1": 0.911216, "recon2": 1.000001, "kl0": -5.122238, "kl1": -11.895136, "kl2": -11.953549}
{"epoch": 308, "phase": 4, "elapsed_s": 3.73, "total": -27.440353, "recon0": 0.697543, "recon1": 0.912553, "recon2": 1.000008, "kl0": -5.31231, "kl1": -12.341876, "kl2": -12.396271}
{"epoch": 309, "phase": 4, "elapsed_s": 4.23, "total": -28.511587, "recon0": 0.697549, "recon1": 0.907015, "recon2": 1.000013, "kl0": -5.501276, "kl1": -12.775915, "kl2": -12.838974}
{"epoch": 310, "phase": 4, "elapsed_s": 3.74, "total": -29.581934, "recon0": 0.696742, "recon1": 0.91166, "recon2": 1.000006, "kl0": -5.691119, "kl1": -13.217509, "kl2": -13.281713}
{"epoch": 311, "phase": 4, "elapsed_s": 3.9, "total": -30.657105, "recon0": 0.697226, "recon1": 0.924003, "recon2": 1.000009, "kl0": -5.880981, "kl1": -13.672913, "kl2": -13.724449}
{"epoch": 312, "phase": 4, "elapsed_s": 3.87, "total": -31.724706, "recon0": 0.697059, "recon1": 0.910476, "recon2": 1.000008, "kl0": -6.070221, "kl1": -14.094861, "kl2": -14.167167}
{"epoch": 313, "phase": 4, "elapsed_s": 3.56, "total": -32.801155, "recon0": 0.696998, "recon1": 0.911396, "recon2": 0.999999, "kl0": -6.260247, "kl1": -14.539419, "kl2": -14.609882}
{"epoch": 314, "phase": 4, "elapsed_s": 3.79, "total": -32.79754, "recon0": 0.69681, "recon1": 0.931113, "recon2": 1.000006, "kl0": -6.259699, "kl1": -14.555883, "kl2": -14.609886}
{"epoch": 315, "phase": 4, "elapsed_s": 3.68, "total": -32.789688, "recon0": 0.69697, "recon1": 0.91289, "recon2": 0.999996, "kl0": -6.259897, "kl1": -14.529742, "kl2": -14.609906}
{"epoch": 316, "phase": 4, "elapsed_s": 3.69, "total": -32.799928, "recon0": 0.696995, "recon1": 0.920038, "recon2": 0.999998, "kl0": -6.260204, "kl1": -14.546849, "kl2": -14.609907}
{"epoch": 317, "phase": 4, "elapsed_s": 3.97, "total": -32.808189, "recon0": 0.696813, "recon1": 0.909605, "recon2": 1.000001, "kl0": -6.260183, "kl1": -14.544527, "kl2": -14.609898}
{"epoch": 318, "phase": 4, "elapsed_s": 3.93, "total": -32.809561, "recon0": 0.696884, "recon1": 0.910203, "recon2": 1.000008, "kl0": -6.260279, "kl1": -14.546486, "kl2": -14.609891}
{"epoch": 319, "phase": 4, "elapsed_s": 4.03, "total": -32.806915, "recon0": 0.696695, "recon1": 0.915331, "recon2": 1.000006, "kl0": -6.26038, "kl1": -14.548677, "kl2": -14.609889}
{"epoch": 320, "phase": 4, "elapsed_s": 4.03, "total": -32.809224, "recon0": 0.696971, "recon1": 0.913454, "recon2": 0.999994, "kl0": -6.26115, "kl1": -14.548583, "kl2": -14.609909}
{"epoch": 321, "phase": 4, "elapsed_s": 3.79, "total": -32.80658, "recon0": 0.696921, "recon1": 0.912158, "recon2": 1.000014, "kl0": -6.259892, "kl1": -14.545911, "kl2": -14.60987}
{"epoch": 322, "phase": 4, "elapsed_s": 4.02, "total": -32.816063, "recon0": 0.696816, "recon1": 0.909218, "recon2": 1.000008, "kl0": -6.260476, "kl1": -14.55174, "kl2": -14.609889}
{"epoch": 323, "phase": 4, "elapsed_s": 3.72, "total": -32.796806, "recon0": 0.696905, "recon1": 0.916061, "recon2": 1.000013, "kl0": -6.260144, "kl1": -14.53976, "kl2": -14.609881}
{"epoch": 324, "phase": 4, "elapsed_s": 3.6, "total": -32.791959, "recon0": 0.697228, "recon1": 0.917183, "recon2": 1.000012, "kl0": -6.259073, "kl1": -14.537418, "kl2": -14.609892}
{"epoch": 325, "phase": 4, "elapsed_s": 3.84, "total": -32.801608, "recon0": 0.696749, "recon1": 0.914895, "recon2": 1.000006, "kl0": -6.260871, "kl1": -14.542501, "kl2": -14.609887}
{"epoch": 326, "phase": 4, "elapsed_s": 3.62, "total": -32.80991, "recon0": 0.696849, "recon1": 0.906385, "recon2": 1.000009, "kl0": -6.260348, "kl1": -14.542919, "kl2": -14.609886}
{"epoch": 327, "phase": 4, "elapsed_s": 3.75, "total": -32.783185, "recon0": 0.696993, "recon1": 0.920439, "recon2": 1.000011, "kl0": -6.26005, "kl1": -14.530707, "kl2": -14.609871}
{"epoch": 328, "phase": 4, "elapsed_s": 3.86, "total": -32.780511, "recon0": 0.69678, "recon1": 0.929524, "recon2": 0.999999, "kl0": -6.260888, "kl1": -14.536023, "kl2": -14.609902}
{"epoch": 329, "phase": 4, "elapsed_s": 3.76, "total": -32.800836, "recon0": 0.696711, "recon1": 0.915503, "recon2": 0.999994, "kl0": -6.260517, "kl1": -14.54262, "kl2": -14.609907}
{"epoch": 330, "phase": 4, "elapsed_s": 3.82, "total": -32.800309, "recon0": 0.696824, "recon1": 0.919015, "recon2": 1.000001, "kl0": -6.260781, "kl1": -14.54548, "kl2": -14.609888}
{"epoch": 331, "phase": 4, "elapsed_s": 3.76, "total": -32.80928, "recon0": 0.696806, "recon1": 0.91412, "recon2": 1.000007, "kl0": -6.260476, "kl1": -14.549832, "kl2": -14.609904}
{"epoch": 332, "phase": 4, "elapsed_s": 3.86, "total": -32.800028, "recon0": 0.697042, "recon1": 0.917003, "recon2": 1.000004, "kl0": -6.26014, "kl1": -14.544058, "kl2": -14.60988}
{"epoch": 333, "phase": 4, "elapsed_s": 4.09, "total": -32.805645, "recon0": 0.69698, "recon1": 0.914057, "recon2": 1.000003, "kl0": -6.259438, "kl1": -14.547351, "kl2": -14.609897}
{"epoch": 334, "phase": 4, "elapsed_s": 4.11, "total": -32.803752, "recon0": 0.696743, "recon1": 0.911666, "recon2": 1.000004, "kl0": -6.260028, "kl1": -14.542248, "kl2": -14.609889}
{"epoch": 335, "phase": 4, "elapsed_s": 4.06, "total": -32.79918, "recon0": 0.696952, "recon1": 0.922422, "recon2": 1.000003, "kl0": -6.260643, "kl1": -14.548012, "kl2": -14.609902}
{"epoch": 336, "phase": 4, "elapsed_s": 3.92, "total": -32.806642, "recon0": 0.697, "recon1": 0.910521, "recon2": 1.000005, "kl0": -6.26087, "kl1": -14.543395, "kl2": -14.609903}
{"epoch": 337, "phase": 4, "elapsed_s": 3.79, "total": -32.802425, "recon0": 0.696715, "recon1": 0.911598, "recon2": 0.999997, "kl0": -6.260366, "kl1": -14.540477, "kl2": -14.609892}
{"epoch": 338, "phase": 4, "elapsed_s": 3.86, "total": -32.799268, "recon0": 0.696866, "recon1": 0.911219, "recon2": 1.000009, "kl0": -6.260086, "kl1": -14.53738, "kl2": -14.609895}
{"epoch": 339, "phase": 4, "elapsed_s": 3.64, "total": -32.80133, "recon0": 0.696953, "recon1": 0.914761, "recon2": 1.000007, "kl0": -6.260135, "kl1": -14.543011, "kl2": -14.609903}
{"epoch": 340, "phase": 4, "elapsed_s": 3.81, "total": -32.780337, "recon0": 0.696945, "recon1": 0.924979, "recon2": 1.000001, "kl0": -6.260224, "kl1": -14.532153, "kl2": -14.609886}
{"epoch": 341, "phase": 4, "elapsed_s": 3.89, "total": -32.776223, "recon0": 0.696934, "recon1": 0.945074, "recon2": 1.000013, "kl0": -6.260531, "kl1": -14.547831, "kl2": -14.609882}
{"epoch": 342, "phase": 4, "elapsed_s": 3.48, "total": -32.781524, "recon0": 0.69691, "recon1": 0.946269, "recon2": 1.000009, "kl0": -6.259565, "kl1": -14.555248, "kl2": -14.609898}

278
nautilus_dolphin/dvae/training_log_beta05_nocollapse.jsonl Executable file
View File

@@ -0,0 +1,278 @@
{"epoch": 1, "phase": 0, "elapsed_s": 23.69, "total": -6.79361, "recon0": 1.197567, "recon1": 0.0, "recon2": 0.0, "kl0": -7.991177, "kl1": 0.0, "kl2": 0.0}
{"epoch": 2, "phase": 0, "elapsed_s": 22.68, "total": -7.435481, "recon0": 1.022993, "recon1": 0.0, "recon2": 0.0, "kl0": -8.458473, "kl1": 0.0, "kl2": 0.0}
{"epoch": 3, "phase": 0, "elapsed_s": 22.94, "total": -7.463537, "recon0": 1.014745, "recon1": 0.0, "recon2": 0.0, "kl0": -8.478282, "kl1": 0.0, "kl2": 0.0}
{"epoch": 4, "phase": 0, "elapsed_s": 22.26, "total": -7.474298, "recon0": 1.010887, "recon1": 0.0, "recon2": 0.0, "kl0": -8.485185, "kl1": 0.0, "kl2": 0.0}
{"epoch": 5, "phase": 0, "elapsed_s": 21.82, "total": -7.480975, "recon0": 1.008915, "recon1": 0.0, "recon2": 0.0, "kl0": -8.48989, "kl1": 0.0, "kl2": 0.0}
{"epoch": 6, "phase": 0, "elapsed_s": 21.15, "total": -7.485128, "recon0": 1.006619, "recon1": 0.0, "recon2": 0.0, "kl0": -8.491746, "kl1": 0.0, "kl2": 0.0}
{"epoch": 7, "phase": 0, "elapsed_s": 20.99, "total": -7.487274, "recon0": 1.006728, "recon1": 0.0, "recon2": 0.0, "kl0": -8.494002, "kl1": 0.0, "kl2": 0.0}
{"epoch": 8, "phase": 0, "elapsed_s": 21.6, "total": -7.488429, "recon0": 1.006059, "recon1": 0.0, "recon2": 0.0, "kl0": -8.494488, "kl1": 0.0, "kl2": 0.0}
{"epoch": 9, "phase": 0, "elapsed_s": 21.54, "total": -7.48845, "recon0": 1.00426, "recon1": 0.0, "recon2": 0.0, "kl0": -8.49271, "kl1": 0.0, "kl2": 0.0}
{"epoch": 10, "phase": 0, "elapsed_s": 21.21, "total": -7.491117, "recon0": 1.005311, "recon1": 0.0, "recon2": 0.0, "kl0": -8.496428, "kl1": 0.0, "kl2": 0.0}
{"epoch": 11, "phase": 0, "elapsed_s": 20.93, "total": -7.490992, "recon0": 1.004811, "recon1": 0.0, "recon2": 0.0, "kl0": -8.495802, "kl1": 0.0, "kl2": 0.0}
{"epoch": 12, "phase": 0, "elapsed_s": 20.59, "total": -7.491286, "recon0": 1.005156, "recon1": 0.0, "recon2": 0.0, "kl0": -8.496442, "kl1": 0.0, "kl2": 0.0}
{"epoch": 13, "phase": 0, "elapsed_s": 21.11, "total": -7.492625, "recon0": 1.004928, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497553, "kl1": 0.0, "kl2": 0.0}
{"epoch": 14, "phase": 0, "elapsed_s": 21.7, "total": -7.493261, "recon0": 1.00473, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497991, "kl1": 0.0, "kl2": 0.0}
{"epoch": 15, "phase": 0, "elapsed_s": 20.89, "total": -7.493425, "recon0": 1.004742, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498167, "kl1": 0.0, "kl2": 0.0}
{"epoch": 16, "phase": 0, "elapsed_s": 21.25, "total": -7.493951, "recon0": 1.003882, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497833, "kl1": 0.0, "kl2": 0.0}
{"epoch": 17, "phase": 0, "elapsed_s": 20.54, "total": -7.493448, "recon0": 1.00418, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497628, "kl1": 0.0, "kl2": 0.0}
{"epoch": 18, "phase": 0, "elapsed_s": 21.97, "total": -7.494606, "recon0": 1.003521, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498127, "kl1": 0.0, "kl2": 0.0}
{"epoch": 19, "phase": 0, "elapsed_s": 21.5, "total": -7.495215, "recon0": 1.002139, "recon1": 0.0, "recon2": 0.0, "kl0": -8.497354, "kl1": 0.0, "kl2": 0.0}
{"epoch": 20, "phase": 0, "elapsed_s": 20.57, "total": -7.495079, "recon0": 1.001522, "recon1": 0.0, "recon2": 0.0, "kl0": -8.496601, "kl1": 0.0, "kl2": 0.0}
{"epoch": 21, "phase": 0, "elapsed_s": 20.54, "total": -7.494494, "recon0": 1.00388, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498374, "kl1": 0.0, "kl2": 0.0}
{"epoch": 22, "phase": 0, "elapsed_s": 20.43, "total": -7.495232, "recon0": 1.00332, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498552, "kl1": 0.0, "kl2": 0.0}
{"epoch": 23, "phase": 0, "elapsed_s": 22.07, "total": -7.496413, "recon0": 1.003497, "recon1": 0.0, "recon2": 0.0, "kl0": -8.49991, "kl1": 0.0, "kl2": 0.0}
{"epoch": 24, "phase": 0, "elapsed_s": 21.9, "total": -7.495635, "recon0": 1.003216, "recon1": 0.0, "recon2": 0.0, "kl0": -8.49885, "kl1": 0.0, "kl2": 0.0}
{"epoch": 25, "phase": 0, "elapsed_s": 20.4, "total": -7.496807, "recon0": 1.002496, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499303, "kl1": 0.0, "kl2": 0.0}
{"epoch": 26, "phase": 0, "elapsed_s": 20.37, "total": -7.496199, "recon0": 1.002774, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498973, "kl1": 0.0, "kl2": 0.0}
{"epoch": 27, "phase": 0, "elapsed_s": 21.33, "total": -7.497299, "recon0": 1.002738, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500037, "kl1": 0.0, "kl2": 0.0}
{"epoch": 28, "phase": 0, "elapsed_s": 21.02, "total": -7.496811, "recon0": 1.002596, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499407, "kl1": 0.0, "kl2": 0.0}
{"epoch": 29, "phase": 0, "elapsed_s": 21.11, "total": -7.496809, "recon0": 1.002868, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499677, "kl1": 0.0, "kl2": 0.0}
{"epoch": 30, "phase": 0, "elapsed_s": 22.11, "total": -7.496375, "recon0": 1.001943, "recon1": 0.0, "recon2": 0.0, "kl0": -8.498319, "kl1": 0.0, "kl2": 0.0}
{"epoch": 31, "phase": 0, "elapsed_s": 21.97, "total": -7.49738, "recon0": 1.002466, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499846, "kl1": 0.0, "kl2": 0.0}
{"epoch": 32, "phase": 0, "elapsed_s": 21.22, "total": -7.497343, "recon0": 1.002087, "recon1": 0.0, "recon2": 0.0, "kl0": -8.49943, "kl1": 0.0, "kl2": 0.0}
{"epoch": 33, "phase": 0, "elapsed_s": 20.6, "total": -7.497509, "recon0": 1.001614, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499123, "kl1": 0.0, "kl2": 0.0}
{"epoch": 34, "phase": 0, "elapsed_s": 20.61, "total": -7.498047, "recon0": 1.002, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500047, "kl1": 0.0, "kl2": 0.0}
{"epoch": 35, "phase": 0, "elapsed_s": 20.65, "total": -7.498059, "recon0": 1.001702, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499761, "kl1": 0.0, "kl2": 0.0}
{"epoch": 36, "phase": 0, "elapsed_s": 20.98, "total": -7.49774, "recon0": 1.002183, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499922, "kl1": 0.0, "kl2": 0.0}
{"epoch": 37, "phase": 0, "elapsed_s": 21.7, "total": -7.498474, "recon0": 1.002877, "recon1": 0.0, "recon2": 0.0, "kl0": -8.50135, "kl1": 0.0, "kl2": 0.0}
{"epoch": 38, "phase": 0, "elapsed_s": 21.07, "total": -7.498586, "recon0": 1.002665, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501251, "kl1": 0.0, "kl2": 0.0}
{"epoch": 39, "phase": 0, "elapsed_s": 19.99, "total": -7.498336, "recon0": 1.002378, "recon1": 0.0, "recon2": 0.0, "kl0": -8.500714, "kl1": 0.0, "kl2": 0.0}
{"epoch": 40, "phase": 0, "elapsed_s": 20.77, "total": -7.499933, "recon0": 1.001744, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501677, "kl1": 0.0, "kl2": 0.0}
{"epoch": 41, "phase": 0, "elapsed_s": 20.31, "total": -7.498233, "recon0": 1.00124, "recon1": 0.0, "recon2": 0.0, "kl0": -8.499473, "kl1": 0.0, "kl2": 0.0}
{"epoch": 42, "phase": 0, "elapsed_s": 20.58, "total": -7.499287, "recon0": 1.002397, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501684, "kl1": 0.0, "kl2": 0.0}
{"epoch": 43, "phase": 0, "elapsed_s": 20.63, "total": -7.499364, "recon0": 1.001727, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501091, "kl1": 0.0, "kl2": 0.0}
{"epoch": 44, "phase": 0, "elapsed_s": 21.2, "total": -7.499365, "recon0": 1.002052, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501417, "kl1": 0.0, "kl2": 0.0}
{"epoch": 45, "phase": 0, "elapsed_s": 21.0, "total": -7.499189, "recon0": 1.002568, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501757, "kl1": 0.0, "kl2": 0.0}
{"epoch": 46, "phase": 0, "elapsed_s": 19.55, "total": -7.499561, "recon0": 1.002305, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501866, "kl1": 0.0, "kl2": 0.0}
{"epoch": 47, "phase": 0, "elapsed_s": 19.27, "total": -7.499606, "recon0": 1.002019, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501625, "kl1": 0.0, "kl2": 0.0}
{"epoch": 48, "phase": 0, "elapsed_s": 19.39, "total": -7.50029, "recon0": 1.002188, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502478, "kl1": 0.0, "kl2": 0.0}
{"epoch": 49, "phase": 0, "elapsed_s": 19.47, "total": -7.499886, "recon0": 1.001982, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501869, "kl1": 0.0, "kl2": 0.0}
{"epoch": 50, "phase": 0, "elapsed_s": 19.66, "total": -7.500348, "recon0": 1.002294, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502642, "kl1": 0.0, "kl2": 0.0}
{"epoch": 51, "phase": 0, "elapsed_s": 18.56, "total": -7.500153, "recon0": 1.001241, "recon1": 0.0, "recon2": 0.0, "kl0": -8.501394, "kl1": 0.0, "kl2": 0.0}
{"epoch": 52, "phase": 0, "elapsed_s": 18.16, "total": -7.500405, "recon0": 1.001804, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502209, "kl1": 0.0, "kl2": 0.0}
{"epoch": 53, "phase": 0, "elapsed_s": 18.31, "total": -7.500355, "recon0": 1.002024, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502379, "kl1": 0.0, "kl2": 0.0}
{"epoch": 54, "phase": 0, "elapsed_s": 18.42, "total": -7.500735, "recon0": 1.001755, "recon1": 0.0, "recon2": 0.0, "kl0": -8.50249, "kl1": 0.0, "kl2": 0.0}
{"epoch": 55, "phase": 0, "elapsed_s": 19.32, "total": -7.500336, "recon0": 1.001857, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502193, "kl1": 0.0, "kl2": 0.0}
{"epoch": 56, "phase": 0, "elapsed_s": 18.52, "total": -7.500588, "recon0": 1.001679, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502267, "kl1": 0.0, "kl2": 0.0}
{"epoch": 57, "phase": 0, "elapsed_s": 18.41, "total": -7.500834, "recon0": 1.001549, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502383, "kl1": 0.0, "kl2": 0.0}
{"epoch": 58, "phase": 0, "elapsed_s": 17.97, "total": -7.500644, "recon0": 1.001568, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502212, "kl1": 0.0, "kl2": 0.0}
{"epoch": 59, "phase": 0, "elapsed_s": 18.14, "total": -7.500778, "recon0": 1.001908, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502686, "kl1": 0.0, "kl2": 0.0}
{"epoch": 60, "phase": 0, "elapsed_s": 19.03, "total": -7.500311, "recon0": 1.002093, "recon1": 0.0, "recon2": 0.0, "kl0": -8.502404, "kl1": 0.0, "kl2": 0.0}
{"epoch": 61, "phase": 1, "elapsed_s": 3.01, "total": -20.364208, "recon0": 0.839574, "recon1": 1.469743, "recon2": 0.0, "kl0": -7.247929, "kl1": -15.425595, "kl2": 0.0}
{"epoch": 62, "phase": 1, "elapsed_s": 2.8, "total": -22.054864, "recon0": 0.772972, "recon1": 1.105512, "recon2": 0.0, "kl0": -7.295641, "kl1": -16.637708, "kl2": 0.0}
{"epoch": 63, "phase": 1, "elapsed_s": 3.02, "total": -22.242755, "recon0": 0.756764, "recon1": 1.072717, "recon2": 0.0, "kl0": -7.297455, "kl1": -16.774781, "kl2": 0.0}
{"epoch": 64, "phase": 1, "elapsed_s": 2.98, "total": -22.349453, "recon0": 0.745225, "recon1": 1.054189, "recon2": 0.0, "kl0": -7.296726, "kl1": -16.852141, "kl2": 0.0}
{"epoch": 65, "phase": 1, "elapsed_s": 2.7, "total": -22.41031, "recon0": 0.731803, "recon1": 1.040969, "recon2": 0.0, "kl0": -7.298445, "kl1": -16.884637, "kl2": 0.0}
{"epoch": 66, "phase": 1, "elapsed_s": 2.81, "total": -22.45678, "recon0": 0.718527, "recon1": 1.033292, "recon2": 0.0, "kl0": -7.297256, "kl1": -16.911344, "kl2": 0.0}
{"epoch": 67, "phase": 1, "elapsed_s": 2.89, "total": -22.508489, "recon0": 0.709603, "recon1": 1.025378, "recon2": 0.0, "kl0": -7.298379, "kl1": -16.94509, "kl2": 0.0}
{"epoch": 68, "phase": 1, "elapsed_s": 2.95, "total": -22.52633, "recon0": 0.703889, "recon1": 1.020364, "recon2": 0.0, "kl0": -7.298726, "kl1": -16.951857, "kl2": 0.0}
{"epoch": 69, "phase": 1, "elapsed_s": 2.68, "total": -22.550039, "recon0": 0.701698, "recon1": 1.016157, "recon2": 0.0, "kl0": -7.298223, "kl1": -16.969671, "kl2": 0.0}
{"epoch": 70, "phase": 1, "elapsed_s": 3.01, "total": -22.560013, "recon0": 0.700218, "recon1": 1.012634, "recon2": 0.0, "kl0": -7.299313, "kl1": -16.973552, "kl2": 0.0}
{"epoch": 71, "phase": 1, "elapsed_s": 2.72, "total": -22.575161, "recon0": 0.699513, "recon1": 1.010541, "recon2": 0.0, "kl0": -7.298883, "kl1": -16.986333, "kl2": 0.0}
{"epoch": 72, "phase": 1, "elapsed_s": 2.77, "total": -22.582816, "recon0": 0.69972, "recon1": 1.007565, "recon2": 0.0, "kl0": -7.298496, "kl1": -16.991605, "kl2": 0.0}
{"epoch": 73, "phase": 1, "elapsed_s": 2.84, "total": -22.594111, "recon0": 0.699111, "recon1": 1.006152, "recon2": 0.0, "kl0": -7.299403, "kl1": -16.999971, "kl2": 0.0}
{"epoch": 74, "phase": 1, "elapsed_s": 2.93, "total": -22.600109, "recon0": 0.699142, "recon1": 1.005034, "recon2": 0.0, "kl0": -7.299963, "kl1": -17.004321, "kl2": 0.0}
{"epoch": 75, "phase": 1, "elapsed_s": 2.81, "total": -22.607139, "recon0": 0.699078, "recon1": 1.003796, "recon2": 0.0, "kl0": -7.299051, "kl1": -17.010961, "kl2": 0.0}
{"epoch": 76, "phase": 1, "elapsed_s": 2.7, "total": -22.611322, "recon0": 0.699158, "recon1": 1.002687, "recon2": 0.0, "kl0": -7.299572, "kl1": -17.013593, "kl2": 0.0}
{"epoch": 77, "phase": 1, "elapsed_s": 3.02, "total": -22.616044, "recon0": 0.698892, "recon1": 1.002257, "recon2": 0.0, "kl0": -7.299126, "kl1": -17.018066, "kl2": 0.0}
{"epoch": 78, "phase": 1, "elapsed_s": 2.7, "total": -22.620544, "recon0": 0.69888, "recon1": 1.002004, "recon2": 0.0, "kl0": -7.299446, "kl1": -17.021981, "kl2": 0.0}
{"epoch": 79, "phase": 1, "elapsed_s": 2.78, "total": -22.62354, "recon0": 0.698359, "recon1": 1.00144, "recon2": 0.0, "kl0": -7.299434, "kl1": -17.023905, "kl2": 0.0}
{"epoch": 80, "phase": 1, "elapsed_s": 2.87, "total": -22.624031, "recon0": 0.699373, "recon1": 1.001252, "recon2": 0.0, "kl0": -7.299434, "kl1": -17.025222, "kl2": 0.0}
{"epoch": 81, "phase": 1, "elapsed_s": 3.04, "total": -22.626789, "recon0": 0.698548, "recon1": 1.00094, "recon2": 0.0, "kl0": -7.299155, "kl1": -17.027122, "kl2": 0.0}
{"epoch": 82, "phase": 1, "elapsed_s": 2.79, "total": -22.627569, "recon0": 0.698576, "recon1": 1.001119, "recon2": 0.0, "kl0": -7.299284, "kl1": -17.02798, "kl2": 0.0}
{"epoch": 83, "phase": 1, "elapsed_s": 2.63, "total": -22.630271, "recon0": 0.698666, "recon1": 1.000909, "recon2": 0.0, "kl0": -7.299682, "kl1": -17.030164, "kl2": 0.0}
{"epoch": 84, "phase": 1, "elapsed_s": 3.07, "total": -22.630003, "recon0": 0.698579, "recon1": 1.000564, "recon2": 0.0, "kl0": -7.298856, "kl1": -17.03029, "kl2": 0.0}
{"epoch": 85, "phase": 1, "elapsed_s": 2.74, "total": -22.631745, "recon0": 0.698582, "recon1": 1.000493, "recon2": 0.0, "kl0": -7.299743, "kl1": -17.031077, "kl2": 0.0}
{"epoch": 86, "phase": 1, "elapsed_s": 2.74, "total": -22.630837, "recon0": 0.698722, "recon1": 1.000524, "recon2": 0.0, "kl0": -7.298933, "kl1": -17.031149, "kl2": 0.0}
{"epoch": 87, "phase": 1, "elapsed_s": 2.56, "total": -22.632505, "recon0": 0.698635, "recon1": 1.000349, "recon2": 0.0, "kl0": -7.299598, "kl1": -17.031891, "kl2": 0.0}
{"epoch": 88, "phase": 1, "elapsed_s": 2.99, "total": -22.633299, "recon0": 0.698251, "recon1": 1.000311, "recon2": 0.0, "kl0": -7.300114, "kl1": -17.031746, "kl2": 0.0}
{"epoch": 89, "phase": 1, "elapsed_s": 2.75, "total": -22.63266, "recon0": 0.698295, "recon1": 1.000314, "recon2": 0.0, "kl0": -7.300485, "kl1": -17.030784, "kl2": 0.0}
{"epoch": 90, "phase": 1, "elapsed_s": 2.8, "total": -22.633377, "recon0": 0.698443, "recon1": 1.000196, "recon2": 0.0, "kl0": -7.300037, "kl1": -17.031979, "kl2": 0.0}
{"epoch": 91, "phase": 1, "elapsed_s": 2.85, "total": -22.632813, "recon0": 0.698687, "recon1": 1.000214, "recon2": 0.0, "kl0": -7.299274, "kl1": -17.03244, "kl2": 0.0}
{"epoch": 92, "phase": 1, "elapsed_s": 2.79, "total": -22.634314, "recon0": 0.698674, "recon1": 1.000139, "recon2": 0.0, "kl0": -7.299809, "kl1": -17.033319, "kl2": 0.0}
{"epoch": 93, "phase": 1, "elapsed_s": 2.63, "total": -22.634267, "recon0": 0.698108, "recon1": 1.000236, "recon2": 0.0, "kl0": -7.299279, "kl1": -17.033333, "kl2": 0.0}
{"epoch": 94, "phase": 1, "elapsed_s": 2.84, "total": -22.634642, "recon0": 0.698151, "recon1": 1.000229, "recon2": 0.0, "kl0": -7.299203, "kl1": -17.033819, "kl2": 0.0}
{"epoch": 95, "phase": 1, "elapsed_s": 3.05, "total": -22.635051, "recon0": 0.698184, "recon1": 1.000185, "recon2": 0.0, "kl0": -7.299584, "kl1": -17.033836, "kl2": 0.0}
{"epoch": 96, "phase": 1, "elapsed_s": 2.67, "total": -22.634399, "recon0": 0.6987, "recon1": 1.000158, "recon2": 0.0, "kl0": -7.299652, "kl1": -17.033604, "kl2": 0.0}
{"epoch": 97, "phase": 1, "elapsed_s": 2.77, "total": -22.634712, "recon0": 0.698169, "recon1": 1.000176, "recon2": 0.0, "kl0": -7.299342, "kl1": -17.033715, "kl2": 0.0}
{"epoch": 98, "phase": 1, "elapsed_s": 2.69, "total": -22.635535, "recon0": 0.698447, "recon1": 1.000067, "recon2": 0.0, "kl0": -7.300064, "kl1": -17.033984, "kl2": 0.0}
{"epoch": 99, "phase": 1, "elapsed_s": 2.85, "total": -22.635565, "recon0": 0.698065, "recon1": 1.000134, "recon2": 0.0, "kl0": -7.299802, "kl1": -17.033961, "kl2": 0.0}
{"epoch": 100, "phase": 1, "elapsed_s": 2.53, "total": -22.635214, "recon0": 0.698039, "recon1": 1.000108, "recon2": 0.0, "kl0": -7.298939, "kl1": -17.034422, "kl2": 0.0}
{"epoch": 101, "phase": 1, "elapsed_s": 2.73, "total": -22.635313, "recon0": 0.698443, "recon1": 1.000076, "recon2": 0.0, "kl0": -7.299895, "kl1": -17.033937, "kl2": 0.0}
{"epoch": 102, "phase": 1, "elapsed_s": 2.8, "total": -22.635445, "recon0": 0.697905, "recon1": 1.000099, "recon2": 0.0, "kl0": -7.299004, "kl1": -17.034445, "kl2": 0.0}
{"epoch": 103, "phase": 1, "elapsed_s": 2.74, "total": -22.635967, "recon0": 0.697776, "recon1": 1.000115, "recon2": 0.0, "kl0": -7.299384, "kl1": -17.034474, "kl2": 0.0}
{"epoch": 104, "phase": 1, "elapsed_s": 2.61, "total": -22.635917, "recon0": 0.697951, "recon1": 1.00005, "recon2": 0.0, "kl0": -7.299553, "kl1": -17.034365, "kl2": 0.0}
{"epoch": 105, "phase": 1, "elapsed_s": 2.78, "total": -22.635594, "recon0": 0.698469, "recon1": 1.000082, "recon2": 0.0, "kl0": -7.299519, "kl1": -17.034627, "kl2": 0.0}
{"epoch": 106, "phase": 1, "elapsed_s": 2.8, "total": -22.635632, "recon0": 0.698377, "recon1": 1.000028, "recon2": 0.0, "kl0": -7.299618, "kl1": -17.034419, "kl2": 0.0}
{"epoch": 107, "phase": 1, "elapsed_s": 3.02, "total": -22.635042, "recon0": 0.698203, "recon1": 1.000083, "recon2": 0.0, "kl0": -7.299416, "kl1": -17.033912, "kl2": 0.0}
{"epoch": 108, "phase": 1, "elapsed_s": 2.88, "total": -22.63652, "recon0": 0.697917, "recon1": 1.000033, "recon2": 0.0, "kl0": -7.300002, "kl1": -17.03447, "kl2": 0.0}
{"epoch": 109, "phase": 1, "elapsed_s": 3.01, "total": -22.63586, "recon0": 0.697939, "recon1": 1.00005, "recon2": 0.0, "kl0": -7.299314, "kl1": -17.034535, "kl2": 0.0}
{"epoch": 110, "phase": 1, "elapsed_s": 2.95, "total": -22.637156, "recon0": 0.698122, "recon1": 1.000072, "recon2": 0.0, "kl0": -7.300497, "kl1": -17.034854, "kl2": 0.0}
{"epoch": 111, "phase": 1, "elapsed_s": 2.68, "total": -22.636421, "recon0": 0.697427, "recon1": 1.00006, "recon2": 0.0, "kl0": -7.299167, "kl1": -17.034742, "kl2": 0.0}
{"epoch": 112, "phase": 1, "elapsed_s": 2.93, "total": -22.635576, "recon0": 0.698373, "recon1": 1.000076, "recon2": 0.0, "kl0": -7.299761, "kl1": -17.034264, "kl2": 0.0}
{"epoch": 113, "phase": 1, "elapsed_s": 2.79, "total": -22.635881, "recon0": 0.697973, "recon1": 1.000037, "recon2": 0.0, "kl0": -7.299921, "kl1": -17.03397, "kl2": 0.0}
{"epoch": 114, "phase": 1, "elapsed_s": 2.82, "total": -22.636044, "recon0": 0.698568, "recon1": 1.000058, "recon2": 0.0, "kl0": -7.300304, "kl1": -17.034365, "kl2": 0.0}
{"epoch": 115, "phase": 1, "elapsed_s": 2.73, "total": -22.636887, "recon0": 0.697651, "recon1": 1.00005, "recon2": 0.0, "kl0": -7.300079, "kl1": -17.03451, "kl2": 0.0}
{"epoch": 116, "phase": 1, "elapsed_s": 2.96, "total": -22.635989, "recon0": 0.698411, "recon1": 1.000027, "recon2": 0.0, "kl0": -7.299376, "kl1": -17.03505, "kl2": 0.0}
{"epoch": 117, "phase": 1, "elapsed_s": 2.77, "total": -22.637205, "recon0": 0.697562, "recon1": 1.00003, "recon2": 0.0, "kl0": -7.299761, "kl1": -17.035036, "kl2": 0.0}
{"epoch": 118, "phase": 1, "elapsed_s": 2.66, "total": -22.636829, "recon0": 0.697986, "recon1": 1.000054, "recon2": 0.0, "kl0": -7.299838, "kl1": -17.035032, "kl2": 0.0}
{"epoch": 119, "phase": 1, "elapsed_s": 2.91, "total": -22.637082, "recon0": 0.697886, "recon1": 1.000052, "recon2": 0.0, "kl0": -7.300088, "kl1": -17.034933, "kl2": 0.0}
{"epoch": 120, "phase": 1, "elapsed_s": 2.66, "total": -22.637646, "recon0": 0.697742, "recon1": 1.000016, "recon2": 0.0, "kl0": -7.300269, "kl1": -17.035135, "kl2": 0.0}
{"epoch": 121, "phase": 1, "elapsed_s": 2.89, "total": -22.636979, "recon0": 0.697794, "recon1": 1.00004, "recon2": 0.0, "kl0": -7.299744, "kl1": -17.035068, "kl2": 0.0}
{"epoch": 122, "phase": 1, "elapsed_s": 2.63, "total": -22.636144, "recon0": 0.69824, "recon1": 1.000039, "recon2": 0.0, "kl0": -7.299312, "kl1": -17.035112, "kl2": 0.0}
{"epoch": 123, "phase": 1, "elapsed_s": 2.99, "total": -22.636977, "recon0": 0.697841, "recon1": 1.000013, "recon2": 0.0, "kl0": -7.300311, "kl1": -17.03452, "kl2": 0.0}
{"epoch": 124, "phase": 1, "elapsed_s": 2.67, "total": -22.636172, "recon0": 0.698229, "recon1": 1.000019, "recon2": 0.0, "kl0": -7.300149, "kl1": -17.034271, "kl2": 0.0}
{"epoch": 125, "phase": 1, "elapsed_s": 2.96, "total": -22.637311, "recon0": 0.697641, "recon1": 1.000059, "recon2": 0.0, "kl0": -7.299968, "kl1": -17.035044, "kl2": 0.0}
{"epoch": 126, "phase": 1, "elapsed_s": 2.84, "total": -22.637082, "recon0": 0.697456, "recon1": 1.000035, "recon2": 0.0, "kl0": -7.299392, "kl1": -17.035181, "kl2": 0.0}
{"epoch": 127, "phase": 1, "elapsed_s": 2.9, "total": -22.637048, "recon0": 0.697687, "recon1": 1.000017, "recon2": 0.0, "kl0": -7.299676, "kl1": -17.035076, "kl2": 0.0}
{"epoch": 128, "phase": 1, "elapsed_s": 2.99, "total": -22.637483, "recon0": 0.69754, "recon1": 1.000037, "recon2": 0.0, "kl0": -7.300015, "kl1": -17.035045, "kl2": 0.0}
{"epoch": 129, "phase": 1, "elapsed_s": 2.8, "total": -22.636571, "recon0": 0.697612, "recon1": 1.000051, "recon2": 0.0, "kl0": -7.299664, "kl1": -17.03457, "kl2": 0.0}
{"epoch": 130, "phase": 1, "elapsed_s": 3.01, "total": -22.63704, "recon0": 0.697804, "recon1": 1.000032, "recon2": 0.0, "kl0": -7.300511, "kl1": -17.034366, "kl2": 0.0}
{"epoch": 131, "phase": 1, "elapsed_s": 2.91, "total": -22.63745, "recon0": 0.697984, "recon1": 1.000031, "recon2": 0.0, "kl0": -7.300438, "kl1": -17.035027, "kl2": 0.0}
{"epoch": 132, "phase": 1, "elapsed_s": 3.0, "total": -22.636911, "recon0": 0.698006, "recon1": 1.000025, "recon2": 0.0, "kl0": -7.29979, "kl1": -17.035152, "kl2": 0.0}
{"epoch": 133, "phase": 1, "elapsed_s": 2.76, "total": -22.637178, "recon0": 0.697528, "recon1": 1.000023, "recon2": 0.0, "kl0": -7.299749, "kl1": -17.034979, "kl2": 0.0}
{"epoch": 134, "phase": 1, "elapsed_s": 2.85, "total": -22.636969, "recon0": 0.697772, "recon1": 1.000029, "recon2": 0.0, "kl0": -7.299589, "kl1": -17.035181, "kl2": 0.0}
{"epoch": 135, "phase": 1, "elapsed_s": 2.7, "total": -22.637286, "recon0": 0.697901, "recon1": 1.000023, "recon2": 0.0, "kl0": -7.299821, "kl1": -17.035388, "kl2": 0.0}
{"epoch": 136, "phase": 1, "elapsed_s": 2.73, "total": -22.636587, "recon0": 0.697988, "recon1": 1.000011, "recon2": 0.0, "kl0": -7.299313, "kl1": -17.035272, "kl2": 0.0}
{"epoch": 137, "phase": 1, "elapsed_s": 2.91, "total": -22.636527, "recon0": 0.697762, "recon1": 1.000026, "recon2": 0.0, "kl0": -7.299657, "kl1": -17.034658, "kl2": 0.0}
{"epoch": 138, "phase": 1, "elapsed_s": 2.75, "total": -22.637654, "recon0": 0.697566, "recon1": 1.000035, "recon2": 0.0, "kl0": -7.300295, "kl1": -17.03496, "kl2": 0.0}
{"epoch": 139, "phase": 1, "elapsed_s": 2.82, "total": -22.637698, "recon0": 0.697525, "recon1": 1.000031, "recon2": 0.0, "kl0": -7.299967, "kl1": -17.035286, "kl2": 0.0}
{"epoch": 140, "phase": 1, "elapsed_s": 2.84, "total": -22.636905, "recon0": 0.697643, "recon1": 1.000026, "recon2": 0.0, "kl0": -7.299335, "kl1": -17.035239, "kl2": 0.0}
{"epoch": 141, "phase": 2, "elapsed_s": 3.14, "total": -36.543898, "recon0": 0.697326, "recon1": 1.000006, "recon2": 1.538737, "kl0": -7.274257, "kl1": -17.034862, "kl2": -15.470847}
{"epoch": 142, "phase": 2, "elapsed_s": 2.87, "total": -38.121203, "recon0": 0.697128, "recon1": 1.000005, "recon2": 1.114735, "kl0": -7.29925, "kl1": -17.034726, "kl2": -16.599094}
{"epoch": 143, "phase": 2, "elapsed_s": 2.98, "total": -38.330139, "recon0": 0.697428, "recon1": 1.00002, "recon2": 1.057766, "kl0": -7.299825, "kl1": -17.034867, "kl2": -16.75066}
{"epoch": 144, "phase": 2, "elapsed_s": 3.03, "total": -38.42925, "recon0": 0.696898, "recon1": 1.000011, "recon2": 1.034188, "kl0": -7.299702, "kl1": -17.034977, "kl2": -16.825668}
{"epoch": 145, "phase": 2, "elapsed_s": 2.91, "total": -38.489503, "recon0": 0.697294, "recon1": 1.000026, "recon2": 1.022072, "kl0": -7.300096, "kl1": -17.035174, "kl2": -16.873624}
{"epoch": 146, "phase": 2, "elapsed_s": 3.01, "total": -38.529628, "recon0": 0.697166, "recon1": 1.000016, "recon2": 1.015221, "kl0": -7.300534, "kl1": -17.035242, "kl2": -16.906254}
{"epoch": 147, "phase": 2, "elapsed_s": 2.98, "total": -38.56078, "recon0": 0.69716, "recon1": 1.000011, "recon2": 1.010571, "kl0": -7.299922, "kl1": -17.035287, "kl2": -16.933313}
{"epoch": 148, "phase": 2, "elapsed_s": 3.23, "total": -38.582983, "recon0": 0.697393, "recon1": 1.000031, "recon2": 1.008188, "kl0": -7.300956, "kl1": -17.035278, "kl2": -16.952361}
{"epoch": 149, "phase": 2, "elapsed_s": 2.94, "total": -38.599553, "recon0": 0.697179, "recon1": 1.000018, "recon2": 1.006368, "kl0": -7.300628, "kl1": -17.035154, "kl2": -16.967335}
{"epoch": 150, "phase": 2, "elapsed_s": 3.13, "total": -38.611389, "recon0": 0.697263, "recon1": 1.000012, "recon2": 1.00541, "kl0": -7.299743, "kl1": -17.035303, "kl2": -16.979029}
{"epoch": 151, "phase": 2, "elapsed_s": 3.1, "total": -38.622582, "recon0": 0.697172, "recon1": 1.000025, "recon2": 1.00427, "kl0": -7.300101, "kl1": -17.035318, "kl2": -16.988629}
{"epoch": 152, "phase": 2, "elapsed_s": 2.86, "total": -38.630832, "recon0": 0.696889, "recon1": 1.000012, "recon2": 1.003719, "kl0": -7.300594, "kl1": -17.035286, "kl2": -16.995573}
{"epoch": 153, "phase": 2, "elapsed_s": 3.16, "total": -38.637556, "recon0": 0.697256, "recon1": 1.000025, "recon2": 1.002683, "kl0": -7.300415, "kl1": -17.03535, "kl2": -17.001756}
{"epoch": 154, "phase": 2, "elapsed_s": 3.28, "total": -38.644226, "recon0": 0.697238, "recon1": 1.000012, "recon2": 1.002383, "kl0": -7.300698, "kl1": -17.035387, "kl2": -17.007774}
{"epoch": 155, "phase": 2, "elapsed_s": 2.96, "total": -38.64895, "recon0": 0.697123, "recon1": 1.000015, "recon2": 1.002042, "kl0": -7.299572, "kl1": -17.035322, "kl2": -17.013235}
{"epoch": 156, "phase": 2, "elapsed_s": 3.05, "total": -38.653724, "recon0": 0.697254, "recon1": 1.00002, "recon2": 1.001875, "kl0": -7.300361, "kl1": -17.035297, "kl2": -17.017215}
{"epoch": 157, "phase": 2, "elapsed_s": 3.23, "total": -38.657449, "recon0": 0.697429, "recon1": 1.000023, "recon2": 1.001569, "kl0": -7.300517, "kl1": -17.035301, "kl2": -17.020652}
{"epoch": 158, "phase": 2, "elapsed_s": 3.08, "total": -38.660405, "recon0": 0.697205, "recon1": 1.000013, "recon2": 1.001302, "kl0": -7.300434, "kl1": -17.035034, "kl2": -17.023455}
{"epoch": 159, "phase": 2, "elapsed_s": 2.95, "total": -38.663237, "recon0": 0.697336, "recon1": 1.000001, "recon2": 1.001159, "kl0": -7.300628, "kl1": -17.035176, "kl2": -17.025929}
{"epoch": 160, "phase": 2, "elapsed_s": 3.18, "total": -38.666256, "recon0": 0.696984, "recon1": 1.000031, "recon2": 1.001064, "kl0": -7.300848, "kl1": -17.03539, "kl2": -17.028097}
{"epoch": 161, "phase": 2, "elapsed_s": 2.9, "total": -38.667149, "recon0": 0.697078, "recon1": 1.000021, "recon2": 1.001022, "kl0": -7.30032, "kl1": -17.035269, "kl2": -17.02968}
{"epoch": 162, "phase": 2, "elapsed_s": 2.99, "total": -38.668836, "recon0": 0.697178, "recon1": 1.000018, "recon2": 1.000794, "kl0": -7.300551, "kl1": -17.03533, "kl2": -17.030945}
{"epoch": 163, "phase": 2, "elapsed_s": 3.13, "total": -38.669644, "recon0": 0.696944, "recon1": 1.000013, "recon2": 1.000702, "kl0": -7.300252, "kl1": -17.035204, "kl2": -17.031848}
{"epoch": 164, "phase": 2, "elapsed_s": 2.95, "total": -38.670117, "recon0": 0.697391, "recon1": 1.000008, "recon2": 1.000728, "kl0": -7.300598, "kl1": -17.035308, "kl2": -17.032339}
{"epoch": 165, "phase": 2, "elapsed_s": 2.87, "total": -38.671019, "recon0": 0.697021, "recon1": 1.000028, "recon2": 1.00051, "kl0": -7.299978, "kl1": -17.035307, "kl2": -17.033293}
{"epoch": 166, "phase": 2, "elapsed_s": 2.96, "total": -38.671544, "recon0": 0.697056, "recon1": 1.000018, "recon2": 1.000546, "kl0": -7.3004, "kl1": -17.035193, "kl2": -17.03357}
{"epoch": 167, "phase": 2, "elapsed_s": 3.08, "total": -38.671944, "recon0": 0.697125, "recon1": 1.000012, "recon2": 1.000494, "kl0": -7.300205, "kl1": -17.03526, "kl2": -17.034111}
{"epoch": 168, "phase": 2, "elapsed_s": 2.92, "total": -38.672183, "recon0": 0.697072, "recon1": 1.000009, "recon2": 1.000487, "kl0": -7.300289, "kl1": -17.035323, "kl2": -17.03414}
{"epoch": 169, "phase": 2, "elapsed_s": 2.78, "total": -38.672464, "recon0": 0.697112, "recon1": 1.000018, "recon2": 1.000409, "kl0": -7.300436, "kl1": -17.035296, "kl2": -17.034272}
{"epoch": 170, "phase": 2, "elapsed_s": 3.27, "total": -38.673116, "recon0": 0.696789, "recon1": 1.000007, "recon2": 1.000262, "kl0": -7.300169, "kl1": -17.035254, "kl2": -17.034751}
{"epoch": 171, "phase": 2, "elapsed_s": 3.23, "total": -38.672011, "recon0": 0.697376, "recon1": 1.000023, "recon2": 1.000341, "kl0": -7.299837, "kl1": -17.035228, "kl2": -17.034687}
{"epoch": 172, "phase": 2, "elapsed_s": 3.19, "total": -38.673127, "recon0": 0.69725, "recon1": 1.000022, "recon2": 1.000308, "kl0": -7.300622, "kl1": -17.035194, "kl2": -17.034891}
{"epoch": 173, "phase": 2, "elapsed_s": 3.44, "total": -38.673147, "recon0": 0.697329, "recon1": 1.000019, "recon2": 1.000196, "kl0": -7.300523, "kl1": -17.035184, "kl2": -17.034984}
{"epoch": 174, "phase": 2, "elapsed_s": 3.63, "total": -38.673431, "recon0": 0.697008, "recon1": 1.000017, "recon2": 1.000219, "kl0": -7.300426, "kl1": -17.035198, "kl2": -17.035051}
{"epoch": 175, "phase": 2, "elapsed_s": 3.48, "total": -38.672982, "recon0": 0.697466, "recon1": 1.000003, "recon2": 1.000132, "kl0": -7.300218, "kl1": -17.035357, "kl2": -17.035009}
{"epoch": 176, "phase": 2, "elapsed_s": 3.66, "total": -38.672757, "recon0": 0.697163, "recon1": 1.000016, "recon2": 1.000188, "kl0": -7.299676, "kl1": -17.035396, "kl2": -17.035052}
{"epoch": 177, "phase": 2, "elapsed_s": 3.23, "total": -38.673414, "recon0": 0.696996, "recon1": 1.000013, "recon2": 1.000146, "kl0": -7.300013, "kl1": -17.035373, "kl2": -17.035184}
{"epoch": 178, "phase": 2, "elapsed_s": 3.23, "total": -38.673857, "recon0": 0.697094, "recon1": 1.000008, "recon2": 1.000187, "kl0": -7.300785, "kl1": -17.035217, "kl2": -17.035144}
{"epoch": 179, "phase": 2, "elapsed_s": 3.64, "total": -38.673444, "recon0": 0.697177, "recon1": 1.000019, "recon2": 1.000113, "kl0": -7.300384, "kl1": -17.035256, "kl2": -17.035113}
{"epoch": 180, "phase": 2, "elapsed_s": 3.15, "total": -38.67367, "recon0": 0.697297, "recon1": 1.000018, "recon2": 1.000141, "kl0": -7.30057, "kl1": -17.035379, "kl2": -17.035177}
{"epoch": 181, "phase": 2, "elapsed_s": 3.15, "total": -38.674174, "recon0": 0.697022, "recon1": 1.000015, "recon2": 1.000146, "kl0": -7.300739, "kl1": -17.035345, "kl2": -17.035273}
{"epoch": 182, "phase": 2, "elapsed_s": 3.41, "total": -38.673985, "recon0": 0.696937, "recon1": 1.000015, "recon2": 1.000107, "kl0": -7.300524, "kl1": -17.03535, "kl2": -17.035169}
{"epoch": 183, "phase": 2, "elapsed_s": 3.18, "total": -38.673675, "recon0": 0.697047, "recon1": 1.000015, "recon2": 1.000129, "kl0": -7.300511, "kl1": -17.035123, "kl2": -17.035232}
{"epoch": 184, "phase": 2, "elapsed_s": 3.23, "total": -38.673547, "recon0": 0.69722, "recon1": 1.000013, "recon2": 1.000102, "kl0": -7.300361, "kl1": -17.035239, "kl2": -17.035282}
{"epoch": 185, "phase": 2, "elapsed_s": 3.3, "total": -38.673591, "recon0": 0.697015, "recon1": 1.000016, "recon2": 1.000089, "kl0": -7.300181, "kl1": -17.035323, "kl2": -17.035208}
{"epoch": 186, "phase": 2, "elapsed_s": 3.25, "total": -38.673869, "recon0": 0.697255, "recon1": 1.000012, "recon2": 1.000111, "kl0": -7.3006, "kl1": -17.035351, "kl2": -17.035296}
{"epoch": 187, "phase": 2, "elapsed_s": 3.25, "total": -38.673348, "recon0": 0.697022, "recon1": 1.000008, "recon2": 1.000031, "kl0": -7.299881, "kl1": -17.035226, "kl2": -17.035302}
{"epoch": 188, "phase": 2, "elapsed_s": 3.66, "total": -38.674456, "recon0": 0.696935, "recon1": 1.000013, "recon2": 1.000062, "kl0": -7.300929, "kl1": -17.035238, "kl2": -17.035298}
{"epoch": 189, "phase": 2, "elapsed_s": 3.33, "total": -38.673811, "recon0": 0.697123, "recon1": 1.0, "recon2": 1.000058, "kl0": -7.300363, "kl1": -17.035331, "kl2": -17.035298}
{"epoch": 190, "phase": 2, "elapsed_s": 3.27, "total": -38.674591, "recon0": 0.696925, "recon1": 0.999996, "recon2": 1.000039, "kl0": -7.30086, "kl1": -17.035326, "kl2": -17.035366}
{"epoch": 191, "phase": 2, "elapsed_s": 3.46, "total": -38.673795, "recon0": 0.697175, "recon1": 1.000005, "recon2": 1.000073, "kl0": -7.300497, "kl1": -17.035261, "kl2": -17.03529}
{"epoch": 192, "phase": 2, "elapsed_s": 3.2, "total": -38.674241, "recon0": 0.696667, "recon1": 1.000013, "recon2": 1.000085, "kl0": -7.300423, "kl1": -17.035303, "kl2": -17.035281}
{"epoch": 193, "phase": 2, "elapsed_s": 3.19, "total": -38.674158, "recon0": 0.696903, "recon1": 1.000024, "recon2": 1.000051, "kl0": -7.300454, "kl1": -17.035317, "kl2": -17.035365}
{"epoch": 194, "phase": 2, "elapsed_s": 3.57, "total": -38.673654, "recon0": 0.697021, "recon1": 1.000006, "recon2": 1.000038, "kl0": -7.300003, "kl1": -17.035367, "kl2": -17.035347}
{"epoch": 195, "phase": 2, "elapsed_s": 3.4, "total": -38.674211, "recon0": 0.697071, "recon1": 1.000008, "recon2": 1.000035, "kl0": -7.300688, "kl1": -17.035285, "kl2": -17.035351}
{"epoch": 196, "phase": 2, "elapsed_s": 3.33, "total": -38.673949, "recon0": 0.696984, "recon1": 1.000014, "recon2": 1.000032, "kl0": -7.300345, "kl1": -17.035297, "kl2": -17.035336}
{"epoch": 197, "phase": 2, "elapsed_s": 3.52, "total": -38.673956, "recon0": 0.697176, "recon1": 1.000013, "recon2": 1.000062, "kl0": -7.300623, "kl1": -17.035281, "kl2": -17.035302}
{"epoch": 198, "phase": 2, "elapsed_s": 3.32, "total": -38.674521, "recon0": 0.696722, "recon1": 1.000015, "recon2": 1.00004, "kl0": -7.300587, "kl1": -17.035355, "kl2": -17.035356}
{"epoch": 199, "phase": 2, "elapsed_s": 3.26, "total": -38.674214, "recon0": 0.697188, "recon1": 1.000018, "recon2": 1.000027, "kl0": -7.300772, "kl1": -17.035315, "kl2": -17.03536}
{"epoch": 200, "phase": 2, "elapsed_s": 3.49, "total": -38.674405, "recon0": 0.696944, "recon1": 0.999997, "recon2": 1.000021, "kl0": -7.300623, "kl1": -17.035378, "kl2": -17.035367}
{"epoch": 201, "phase": 2, "elapsed_s": 3.37, "total": -38.67411, "recon0": 0.697145, "recon1": 1.000022, "recon2": 1.000038, "kl0": -7.300582, "kl1": -17.035393, "kl2": -17.03534}
{"epoch": 202, "phase": 2, "elapsed_s": 3.58, "total": -38.673823, "recon0": 0.697276, "recon1": 1.000005, "recon2": 1.000013, "kl0": -7.300408, "kl1": -17.035353, "kl2": -17.035356}
{"epoch": 203, "phase": 2, "elapsed_s": 3.43, "total": -38.673178, "recon0": 0.69705, "recon1": 1.000018, "recon2": 1.000027, "kl0": -7.299643, "kl1": -17.035277, "kl2": -17.035354}
{"epoch": 204, "phase": 2, "elapsed_s": 3.24, "total": -38.673959, "recon0": 0.697046, "recon1": 1.000011, "recon2": 1.000026, "kl0": -7.300329, "kl1": -17.035341, "kl2": -17.035372}
{"epoch": 205, "phase": 2, "elapsed_s": 3.22, "total": -38.674472, "recon0": 0.697073, "recon1": 1.000005, "recon2": 1.000021, "kl0": -7.300873, "kl1": -17.035304, "kl2": -17.035395}
{"epoch": 206, "phase": 2, "elapsed_s": 3.46, "total": -38.673754, "recon0": 0.696857, "recon1": 1.000017, "recon2": 1.000033, "kl0": -7.300015, "kl1": -17.035309, "kl2": -17.035337}
{"epoch": 207, "phase": 2, "elapsed_s": 3.19, "total": -38.673467, "recon0": 0.696934, "recon1": 1.000021, "recon2": 1.000024, "kl0": -7.299862, "kl1": -17.035235, "kl2": -17.035349}
{"epoch": 208, "phase": 2, "elapsed_s": 3.49, "total": -38.673798, "recon0": 0.696967, "recon1": 1.00001, "recon2": 1.000033, "kl0": -7.300083, "kl1": -17.035337, "kl2": -17.035387}
{"epoch": 209, "phase": 2, "elapsed_s": 3.61, "total": -38.673413, "recon0": 0.697096, "recon1": 1.000012, "recon2": 1.000014, "kl0": -7.299872, "kl1": -17.0353, "kl2": -17.035362}
{"epoch": 210, "phase": 2, "elapsed_s": 3.29, "total": -38.674057, "recon0": 0.697171, "recon1": 1.000004, "recon2": 1.000022, "kl0": -7.30069, "kl1": -17.035196, "kl2": -17.035368}
{"epoch": 221, "phase": 3, "elapsed_s": 3.57, "total": -38.674347, "recon0": 0.696889, "recon1": 1.000008, "recon2": 1.00003, "kl0": -7.300621, "kl1": -17.035275, "kl2": -17.035378}
{"epoch": 222, "phase": 3, "elapsed_s": 3.37, "total": -38.673588, "recon0": 0.696957, "recon1": 1.000026, "recon2": 1.000036, "kl0": -7.300135, "kl1": -17.035109, "kl2": -17.035363}
{"epoch": 223, "phase": 3, "elapsed_s": 3.38, "total": -38.673232, "recon0": 0.697183, "recon1": 1.000014, "recon2": 1.00002, "kl0": -7.299703, "kl1": -17.035367, "kl2": -17.035378}
{"epoch": 224, "phase": 3, "elapsed_s": 3.99, "total": -38.674388, "recon0": 0.696861, "recon1": 1.000012, "recon2": 1.000024, "kl0": -7.300596, "kl1": -17.035344, "kl2": -17.035346}
{"epoch": 225, "phase": 3, "elapsed_s": 3.34, "total": -38.673928, "recon0": 0.696897, "recon1": 1.000013, "recon2": 1.000012, "kl0": -7.300149, "kl1": -17.035321, "kl2": -17.035381}
{"epoch": 226, "phase": 3, "elapsed_s": 3.18, "total": -38.67419, "recon0": 0.697145, "recon1": 1.000011, "recon2": 0.999995, "kl0": -7.300583, "kl1": -17.035369, "kl2": -17.035388}
{"epoch": 227, "phase": 3, "elapsed_s": 3.49, "total": -38.674614, "recon0": 0.696862, "recon1": 1.000002, "recon2": 1.000013, "kl0": -7.300753, "kl1": -17.035337, "kl2": -17.035401}
{"epoch": 228, "phase": 3, "elapsed_s": 3.29, "total": -38.674064, "recon0": 0.696961, "recon1": 1.000015, "recon2": 1.000014, "kl0": -7.300496, "kl1": -17.035192, "kl2": -17.035366}
{"epoch": 229, "phase": 3, "elapsed_s": 3.27, "total": -38.674023, "recon0": 0.697038, "recon1": 1.000017, "recon2": 1.000019, "kl0": -7.30042, "kl1": -17.035294, "kl2": -17.035383}
{"epoch": 230, "phase": 3, "elapsed_s": 3.6, "total": -38.674371, "recon0": 0.696943, "recon1": 1.00002, "recon2": 1.000013, "kl0": -7.300633, "kl1": -17.035338, "kl2": -17.035376}
{"epoch": 231, "phase": 3, "elapsed_s": 3.39, "total": -38.673599, "recon0": 0.696955, "recon1": 1.000025, "recon2": 1.000018, "kl0": -7.300192, "kl1": -17.035008, "kl2": -17.035396}
{"epoch": 232, "phase": 3, "elapsed_s": 3.3, "total": -38.674617, "recon0": 0.697004, "recon1": 1.000013, "recon2": 1.000018, "kl0": -7.300836, "kl1": -17.035435, "kl2": -17.035381}
{"epoch": 233, "phase": 3, "elapsed_s": 3.76, "total": -38.674141, "recon0": 0.697063, "recon1": 1.000016, "recon2": 1.000027, "kl0": -7.300584, "kl1": -17.035294, "kl2": -17.035367}
{"epoch": 234, "phase": 3, "elapsed_s": 3.56, "total": -38.674748, "recon0": 0.69684, "recon1": 1.000005, "recon2": 1.000011, "kl0": -7.300946, "kl1": -17.035286, "kl2": -17.035372}
{"epoch": 235, "phase": 3, "elapsed_s": 3.27, "total": -38.673934, "recon0": 0.697008, "recon1": 1.000014, "recon2": 1.000014, "kl0": -7.300278, "kl1": -17.035307, "kl2": -17.035386}
{"epoch": 236, "phase": 3, "elapsed_s": 3.58, "total": -38.674221, "recon0": 0.696883, "recon1": 1.000052, "recon2": 1.000013, "kl0": -7.300362, "kl1": -17.035421, "kl2": -17.035386}
{"epoch": 237, "phase": 3, "elapsed_s": 3.21, "total": -38.674566, "recon0": 0.696997, "recon1": 1.000013, "recon2": 1.000013, "kl0": -7.300865, "kl1": -17.035333, "kl2": -17.03539}
{"epoch": 238, "phase": 3, "elapsed_s": 3.15, "total": -38.674322, "recon0": 0.696882, "recon1": 1.00002, "recon2": 1.000012, "kl0": -7.300493, "kl1": -17.035364, "kl2": -17.035379}
{"epoch": 239, "phase": 3, "elapsed_s": 3.67, "total": -38.673952, "recon0": 0.697018, "recon1": 1.000018, "recon2": 1.000018, "kl0": -7.300445, "kl1": -17.035178, "kl2": -17.035383}
{"epoch": 240, "phase": 3, "elapsed_s": 3.26, "total": -38.673952, "recon0": 0.697255, "recon1": 1.000012, "recon2": 1.000016, "kl0": -7.300654, "kl1": -17.03519, "kl2": -17.035392}
{"epoch": 241, "phase": 3, "elapsed_s": 3.16, "total": -38.674146, "recon0": 0.696803, "recon1": 1.000027, "recon2": 1.000009, "kl0": -7.300309, "kl1": -17.035284, "kl2": -17.035392}
{"epoch": 242, "phase": 3, "elapsed_s": 3.62, "total": -38.674018, "recon0": 0.697059, "recon1": 1.000013, "recon2": 1.000008, "kl0": -7.300407, "kl1": -17.035297, "kl2": -17.035394}
{"epoch": 243, "phase": 3, "elapsed_s": 3.33, "total": -38.673983, "recon0": 0.696909, "recon1": 1.000013, "recon2": 1.000014, "kl0": -7.300219, "kl1": -17.035303, "kl2": -17.035397}
{"epoch": 244, "phase": 3, "elapsed_s": 3.41, "total": -38.674155, "recon0": 0.697312, "recon1": 1.000014, "recon2": 1.000023, "kl0": -7.30077, "kl1": -17.035355, "kl2": -17.035379}
{"epoch": 245, "phase": 3, "elapsed_s": 3.77, "total": -38.674479, "recon0": 0.696863, "recon1": 1.000016, "recon2": 1.000013, "kl0": -7.30063, "kl1": -17.035349, "kl2": -17.035391}
{"epoch": 246, "phase": 3, "elapsed_s": 3.3, "total": -38.674718, "recon0": 0.69698, "recon1": 1.000009, "recon2": 1.000013, "kl0": -7.300975, "kl1": -17.035353, "kl2": -17.035392}
{"epoch": 247, "phase": 3, "elapsed_s": 3.28, "total": -38.673872, "recon0": 0.697185, "recon1": 1.000004, "recon2": 1.000011, "kl0": -7.300353, "kl1": -17.035323, "kl2": -17.035396}
{"epoch": 248, "phase": 3, "elapsed_s": 3.62, "total": -38.673598, "recon0": 0.697002, "recon1": 1.000013, "recon2": 1.000009, "kl0": -7.299961, "kl1": -17.035275, "kl2": -17.035386}
{"epoch": 249, "phase": 3, "elapsed_s": 3.17, "total": -38.674207, "recon0": 0.697069, "recon1": 1.000006, "recon2": 1.000009, "kl0": -7.300616, "kl1": -17.035293, "kl2": -17.035382}
{"epoch": 250, "phase": 3, "elapsed_s": 3.2, "total": -38.673972, "recon0": 0.697071, "recon1": 1.000012, "recon2": 1.00002, "kl0": -7.300334, "kl1": -17.035353, "kl2": -17.035388}
{"epoch": 251, "phase": 3, "elapsed_s": 3.62, "total": -38.673765, "recon0": 0.696969, "recon1": 1.000014, "recon2": 1.000019, "kl0": -7.300072, "kl1": -17.035293, "kl2": -17.035402}
{"epoch": 252, "phase": 3, "elapsed_s": 3.19, "total": -38.674367, "recon0": 0.696939, "recon1": 1.000013, "recon2": 1.000021, "kl0": -7.30061, "kl1": -17.03536, "kl2": -17.035371}
{"epoch": 253, "phase": 3, "elapsed_s": 3.38, "total": -38.67343, "recon0": 0.696841, "recon1": 1.000009, "recon2": 1.000011, "kl0": -7.29953, "kl1": -17.035372, "kl2": -17.03539}
{"epoch": 254, "phase": 3, "elapsed_s": 3.67, "total": -38.674479, "recon0": 0.697112, "recon1": 1.000012, "recon2": 1.000007, "kl0": -7.300824, "kl1": -17.035387, "kl2": -17.035399}
{"epoch": 281, "phase": 4, "elapsed_s": 4.41, "total": -32.784086, "recon0": 0.696698, "recon1": 1.000011, "recon2": 1.000011, "kl0": -6.261057, "kl1": -14.609855, "kl2": -14.609893}
{"epoch": 282, "phase": 4, "elapsed_s": 4.6, "total": -32.784133, "recon0": 0.696612, "recon1": 0.999996, "recon2": 1.000003, "kl0": -6.260942, "kl1": -14.609906, "kl2": -14.609897}
{"epoch": 283, "phase": 4, "elapsed_s": 4.32, "total": -32.784067, "recon0": 0.696555, "recon1": 1.000008, "recon2": 1.000004, "kl0": -6.260874, "kl1": -14.609879, "kl2": -14.609881}
{"epoch": 284, "phase": 4, "elapsed_s": 4.64, "total": -32.784189, "recon0": 0.696596, "recon1": 1.000005, "recon2": 1.000009, "kl0": -6.261046, "kl1": -14.60988, "kl2": -14.609873}
{"epoch": 285, "phase": 4, "elapsed_s": 4.29, "total": -32.78447, "recon0": 0.696698, "recon1": 1.000001, "recon2": 1.00001, "kl0": -6.261363, "kl1": -14.609925, "kl2": -14.60989}
{"epoch": 286, "phase": 4, "elapsed_s": 4.19, "total": -32.78394, "recon0": 0.696777, "recon1": 1.0, "recon2": 1.000008, "kl0": -6.260959, "kl1": -14.609867, "kl2": -14.609899}
{"epoch": 287, "phase": 4, "elapsed_s": 4.43, "total": -32.784069, "recon0": 0.696767, "recon1": 1.000003, "recon2": 1.000002, "kl0": -6.261059, "kl1": -14.609882, "kl2": -14.609899}
{"epoch": 288, "phase": 4, "elapsed_s": 4.5, "total": -32.784277, "recon0": 0.696757, "recon1": 1.000002, "recon2": 1.0, "kl0": -6.261247, "kl1": -14.6099, "kl2": -14.609889}
{"epoch": 289, "phase": 4, "elapsed_s": 4.46, "total": -32.784803, "recon0": 0.696826, "recon1": 1.000001, "recon2": 1.000002, "kl0": -6.261861, "kl1": -14.609879, "kl2": -14.609892}
{"epoch": 290, "phase": 4, "elapsed_s": 4.06, "total": -32.784142, "recon0": 0.696768, "recon1": 1.000006, "recon2": 1.000012, "kl0": -6.26114, "kl1": -14.609896, "kl2": -14.609891}
{"epoch": 291, "phase": 4, "elapsed_s": 4.18, "total": -32.784593, "recon0": 0.696533, "recon1": 0.999998, "recon2": 1.000003, "kl0": -6.261317, "kl1": -14.609915, "kl2": -14.609894}
{"epoch": 292, "phase": 4, "elapsed_s": 4.13, "total": -32.78401, "recon0": 0.69683, "recon1": 1.000003, "recon2": 1.000006, "kl0": -6.261047, "kl1": -14.609903, "kl2": -14.609899}
{"epoch": 293, "phase": 4, "elapsed_s": 5.08, "total": -32.784827, "recon0": 0.696699, "recon1": 1.000007, "recon2": 1.000002, "kl0": -6.261779, "kl1": -14.609864, "kl2": -14.609892}
{"epoch": 294, "phase": 4, "elapsed_s": 4.55, "total": -32.784634, "recon0": 0.696498, "recon1": 0.999999, "recon2": 1.000005, "kl0": -6.261382, "kl1": -14.609863, "kl2": -14.609891}
{"epoch": 295, "phase": 4, "elapsed_s": 4.04, "total": -32.784336, "recon0": 0.696603, "recon1": 1.000008, "recon2": 1.0, "kl0": -6.261158, "kl1": -14.6099, "kl2": -14.609889}
{"epoch": 296, "phase": 4, "elapsed_s": 4.42, "total": -32.783908, "recon0": 0.696655, "recon1": 1.000002, "recon2": 1.000002, "kl0": -6.260758, "kl1": -14.609919, "kl2": -14.60989}
{"epoch": 297, "phase": 4, "elapsed_s": 4.12, "total": -32.784142, "recon0": 0.696555, "recon1": 1.000002, "recon2": 0.999999, "kl0": -6.260925, "kl1": -14.609884, "kl2": -14.609889}
{"epoch": 298, "phase": 4, "elapsed_s": 4.64, "total": -32.783796, "recon0": 0.696658, "recon1": 1.000011, "recon2": 1.000011, "kl0": -6.26067, "kl1": -14.609912, "kl2": -14.609894}
{"epoch": 299, "phase": 4, "elapsed_s": 4.16, "total": -32.784169, "recon0": 0.696622, "recon1": 1.000004, "recon2": 1.000009, "kl0": -6.261036, "kl1": -14.609885, "kl2": -14.609883}
{"epoch": 300, "phase": 4, "elapsed_s": 4.27, "total": -32.784773, "recon0": 0.696595, "recon1": 1.000005, "recon2": 1.000009, "kl0": -6.261594, "kl1": -14.609898, "kl2": -14.60989}
{"epoch": 301, "phase": 4, "elapsed_s": 4.17, "total": -32.784503, "recon0": 0.696744, "recon1": 1.000001, "recon2": 1.000015, "kl0": -6.261502, "kl1": -14.609851, "kl2": -14.60991}
{"epoch": 302, "phase": 4, "elapsed_s": 4.11, "total": -32.784517, "recon0": 0.696613, "recon1": 1.000008, "recon2": 1.000005, "kl0": -6.261355, "kl1": -14.609908, "kl2": -14.609881}
{"epoch": 303, "phase": 4, "elapsed_s": 4.6, "total": -32.783549, "recon0": 0.696622, "recon1": 1.000009, "recon2": 1.000004, "kl0": -6.260391, "kl1": -14.609898, "kl2": -14.609895}
{"epoch": 304, "phase": 4, "elapsed_s": 4.31, "total": -32.784125, "recon0": 0.696576, "recon1": 1.000001, "recon2": 1.000005, "kl0": -6.260926, "kl1": -14.609889, "kl2": -14.609892}
{"epoch": 305, "phase": 4, "elapsed_s": 4.65, "total": -32.784356, "recon0": 0.696656, "recon1": 1.000006, "recon2": 1.000005, "kl0": -6.261234, "kl1": -14.609904, "kl2": -14.609885}
{"epoch": 306, "phase": 4, "elapsed_s": 4.39, "total": -32.784211, "recon0": 0.696681, "recon1": 1.000004, "recon2": 1.000014, "kl0": -6.261133, "kl1": -14.609884, "kl2": -14.609892}
{"epoch": 307, "phase": 4, "elapsed_s": 4.52, "total": -32.783983, "recon0": 0.69654, "recon1": 1.000009, "recon2": 1.000007, "kl0": -6.26077, "kl1": -14.609886, "kl2": -14.609883}
{"epoch": 308, "phase": 4, "elapsed_s": 4.41, "total": -32.784768, "recon0": 0.69664, "recon1": 1.000008, "recon2": 1.000003, "kl0": -6.261649, "kl1": -14.609871, "kl2": -14.609898}
{"epoch": 309, "phase": 4, "elapsed_s": 4.61, "total": -32.784202, "recon0": 0.696524, "recon1": 1.000002, "recon2": 1.000007, "kl0": -6.260946, "kl1": -14.609902, "kl2": -14.609887}
{"epoch": 310, "phase": 4, "elapsed_s": 4.44, "total": -32.784055, "recon0": 0.696745, "recon1": 1.00001, "recon2": 1.000002, "kl0": -6.261045, "kl1": -14.609878, "kl2": -14.609888}
{"epoch": 311, "phase": 4, "elapsed_s": 4.18, "total": -32.784221, "recon0": 0.696624, "recon1": 1.000004, "recon2": 1.000009, "kl0": -6.261076, "kl1": -14.609894, "kl2": -14.609887}
{"epoch": 312, "phase": 4, "elapsed_s": 4.54, "total": -32.784136, "recon0": 0.696652, "recon1": 1.000003, "recon2": 1.000001, "kl0": -6.261022, "kl1": -14.609891, "kl2": -14.609878}
{"epoch": 313, "phase": 4, "elapsed_s": 4.01, "total": -32.784591, "recon0": 0.696464, "recon1": 1.000008, "recon2": 1.000001, "kl0": -6.261299, "kl1": -14.609883, "kl2": -14.609882}
{"epoch": 314, "phase": 4, "elapsed_s": 4.26, "total": -32.78352, "recon0": 0.696619, "recon1": 1.000007, "recon2": 1.000006, "kl0": -6.260384, "kl1": -14.609879, "kl2": -14.609888}

310
nautilus_dolphin/dvae/training_log_beta6.jsonl Executable file
View File

@@ -0,0 +1,310 @@
{"epoch": 1, "phase": 0, "elapsed_s": 21.37, "total": 86.883101, "recon0": 1.209082, "recon1": 0.0, "recon2": 0.0, "kl0": 85.674019, "kl1": 0.0, "kl2": 0.0}
{"epoch": 2, "phase": 0, "elapsed_s": 22.65, "total": 86.105073, "recon0": 1.023753, "recon1": 0.0, "recon2": 0.0, "kl0": 85.08132, "kl1": 0.0, "kl2": 0.0}
{"epoch": 3, "phase": 0, "elapsed_s": 21.13, "total": 86.082626, "recon0": 1.016161, "recon1": 0.0, "recon2": 0.0, "kl0": 85.066465, "kl1": 0.0, "kl2": 0.0}
{"epoch": 4, "phase": 0, "elapsed_s": 20.71, "total": 86.07332, "recon0": 1.012764, "recon1": 0.0, "recon2": 0.0, "kl0": 85.060556, "kl1": 0.0, "kl2": 0.0}
{"epoch": 5, "phase": 0, "elapsed_s": 20.67, "total": 86.067549, "recon0": 1.010427, "recon1": 0.0, "recon2": 0.0, "kl0": 85.057122, "kl1": 0.0, "kl2": 0.0}
{"epoch": 6, "phase": 0, "elapsed_s": 21.83, "total": 86.063378, "recon0": 1.008629, "recon1": 0.0, "recon2": 0.0, "kl0": 85.054749, "kl1": 0.0, "kl2": 0.0}
{"epoch": 7, "phase": 0, "elapsed_s": 20.94, "total": 86.061734, "recon0": 1.008111, "recon1": 0.0, "recon2": 0.0, "kl0": 85.053624, "kl1": 0.0, "kl2": 0.0}
{"epoch": 8, "phase": 0, "elapsed_s": 21.69, "total": 86.060172, "recon0": 1.007566, "recon1": 0.0, "recon2": 0.0, "kl0": 85.052606, "kl1": 0.0, "kl2": 0.0}
{"epoch": 9, "phase": 0, "elapsed_s": 24.39, "total": 86.059015, "recon0": 1.00676, "recon1": 0.0, "recon2": 0.0, "kl0": 85.052255, "kl1": 0.0, "kl2": 0.0}
{"epoch": 10, "phase": 0, "elapsed_s": 21.72, "total": 86.058169, "recon0": 1.006557, "recon1": 0.0, "recon2": 0.0, "kl0": 85.051611, "kl1": 0.0, "kl2": 0.0}
{"epoch": 11, "phase": 0, "elapsed_s": 21.5, "total": 86.057308, "recon0": 1.006546, "recon1": 0.0, "recon2": 0.0, "kl0": 85.050762, "kl1": 0.0, "kl2": 0.0}
{"epoch": 12, "phase": 0, "elapsed_s": 21.15, "total": 86.057449, "recon0": 1.006532, "recon1": 0.0, "recon2": 0.0, "kl0": 85.050917, "kl1": 0.0, "kl2": 0.0}
{"epoch": 13, "phase": 0, "elapsed_s": 20.98, "total": 86.056621, "recon0": 1.006051, "recon1": 0.0, "recon2": 0.0, "kl0": 85.05057, "kl1": 0.0, "kl2": 0.0}
{"epoch": 14, "phase": 0, "elapsed_s": 21.19, "total": 86.056466, "recon0": 1.005925, "recon1": 0.0, "recon2": 0.0, "kl0": 85.050541, "kl1": 0.0, "kl2": 0.0}
{"epoch": 15, "phase": 0, "elapsed_s": 21.56, "total": 86.056324, "recon0": 1.005965, "recon1": 0.0, "recon2": 0.0, "kl0": 85.050359, "kl1": 0.0, "kl2": 0.0}
{"epoch": 16, "phase": 0, "elapsed_s": 21.07, "total": 86.05557, "recon0": 1.005388, "recon1": 0.0, "recon2": 0.0, "kl0": 85.050182, "kl1": 0.0, "kl2": 0.0}
{"epoch": 17, "phase": 0, "elapsed_s": 20.74, "total": 86.055729, "recon0": 1.005701, "recon1": 0.0, "recon2": 0.0, "kl0": 85.050027, "kl1": 0.0, "kl2": 0.0}
{"epoch": 18, "phase": 0, "elapsed_s": 20.58, "total": 86.055101, "recon0": 1.005328, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049773, "kl1": 0.0, "kl2": 0.0}
{"epoch": 19, "phase": 0, "elapsed_s": 21.51, "total": 86.054515, "recon0": 1.004854, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049661, "kl1": 0.0, "kl2": 0.0}
{"epoch": 20, "phase": 0, "elapsed_s": 21.5, "total": 86.054634, "recon0": 1.005132, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049502, "kl1": 0.0, "kl2": 0.0}
{"epoch": 21, "phase": 0, "elapsed_s": 20.8, "total": 86.054984, "recon0": 1.00535, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049634, "kl1": 0.0, "kl2": 0.0}
{"epoch": 22, "phase": 0, "elapsed_s": 21.44, "total": 86.054563, "recon0": 1.004981, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049582, "kl1": 0.0, "kl2": 0.0}
{"epoch": 23, "phase": 0, "elapsed_s": 20.82, "total": 86.054125, "recon0": 1.004879, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049246, "kl1": 0.0, "kl2": 0.0}
{"epoch": 24, "phase": 0, "elapsed_s": 20.26, "total": 86.054195, "recon0": 1.004682, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049513, "kl1": 0.0, "kl2": 0.0}
{"epoch": 25, "phase": 0, "elapsed_s": 21.69, "total": 86.053705, "recon0": 1.00435, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049355, "kl1": 0.0, "kl2": 0.0}
{"epoch": 26, "phase": 0, "elapsed_s": 21.12, "total": 86.053788, "recon0": 1.004346, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049442, "kl1": 0.0, "kl2": 0.0}
{"epoch": 27, "phase": 0, "elapsed_s": 20.49, "total": 86.053552, "recon0": 1.004326, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049226, "kl1": 0.0, "kl2": 0.0}
{"epoch": 28, "phase": 0, "elapsed_s": 21.34, "total": 86.05329, "recon0": 1.004203, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049086, "kl1": 0.0, "kl2": 0.0}
{"epoch": 29, "phase": 0, "elapsed_s": 20.16, "total": 86.053247, "recon0": 1.004118, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049129, "kl1": 0.0, "kl2": 0.0}
{"epoch": 30, "phase": 0, "elapsed_s": 20.36, "total": 86.053359, "recon0": 1.004064, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049294, "kl1": 0.0, "kl2": 0.0}
{"epoch": 31, "phase": 0, "elapsed_s": 20.51, "total": 86.05295, "recon0": 1.004003, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048947, "kl1": 0.0, "kl2": 0.0}
{"epoch": 32, "phase": 0, "elapsed_s": 19.24, "total": 86.052861, "recon0": 1.003819, "recon1": 0.0, "recon2": 0.0, "kl0": 85.049042, "kl1": 0.0, "kl2": 0.0}
{"epoch": 33, "phase": 0, "elapsed_s": 20.48, "total": 86.05269, "recon0": 1.003701, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048989, "kl1": 0.0, "kl2": 0.0}
{"epoch": 34, "phase": 0, "elapsed_s": 21.69, "total": 86.052425, "recon0": 1.003493, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048932, "kl1": 0.0, "kl2": 0.0}
{"epoch": 35, "phase": 0, "elapsed_s": 20.83, "total": 86.052509, "recon0": 1.003599, "recon1": 0.0, "recon2": 0.0, "kl0": 85.04891, "kl1": 0.0, "kl2": 0.0}
{"epoch": 36, "phase": 0, "elapsed_s": 20.88, "total": 86.052773, "recon0": 1.003713, "recon1": 0.0, "recon2": 0.0, "kl0": 85.04906, "kl1": 0.0, "kl2": 0.0}
{"epoch": 37, "phase": 0, "elapsed_s": 21.83, "total": 86.052541, "recon0": 1.003609, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048933, "kl1": 0.0, "kl2": 0.0}
{"epoch": 38, "phase": 0, "elapsed_s": 22.12, "total": 86.052515, "recon0": 1.003521, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048994, "kl1": 0.0, "kl2": 0.0}
{"epoch": 39, "phase": 0, "elapsed_s": 21.51, "total": 86.052543, "recon0": 1.003562, "recon1": 0.0, "recon2": 0.0, "kl0": 85.04898, "kl1": 0.0, "kl2": 0.0}
{"epoch": 40, "phase": 0, "elapsed_s": 21.4, "total": 86.051954, "recon0": 1.003322, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048632, "kl1": 0.0, "kl2": 0.0}
{"epoch": 41, "phase": 0, "elapsed_s": 20.97, "total": 86.05217, "recon0": 1.003313, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048856, "kl1": 0.0, "kl2": 0.0}
{"epoch": 42, "phase": 0, "elapsed_s": 20.93, "total": 86.052204, "recon0": 1.003459, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048745, "kl1": 0.0, "kl2": 0.0}
{"epoch": 43, "phase": 0, "elapsed_s": 21.29, "total": 86.051885, "recon0": 1.003173, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048712, "kl1": 0.0, "kl2": 0.0}
{"epoch": 44, "phase": 0, "elapsed_s": 22.04, "total": 86.051957, "recon0": 1.003255, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048702, "kl1": 0.0, "kl2": 0.0}
{"epoch": 45, "phase": 0, "elapsed_s": 21.58, "total": 86.051914, "recon0": 1.003216, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048699, "kl1": 0.0, "kl2": 0.0}
{"epoch": 46, "phase": 0, "elapsed_s": 20.39, "total": 86.051632, "recon0": 1.002953, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048679, "kl1": 0.0, "kl2": 0.0}
{"epoch": 47, "phase": 0, "elapsed_s": 20.78, "total": 86.051631, "recon0": 1.002924, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048707, "kl1": 0.0, "kl2": 0.0}
{"epoch": 48, "phase": 0, "elapsed_s": 20.23, "total": 86.05159, "recon0": 1.002916, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048674, "kl1": 0.0, "kl2": 0.0}
{"epoch": 49, "phase": 0, "elapsed_s": 21.3, "total": 86.051583, "recon0": 1.002921, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048662, "kl1": 0.0, "kl2": 0.0}
{"epoch": 50, "phase": 0, "elapsed_s": 21.37, "total": 86.051379, "recon0": 1.002779, "recon1": 0.0, "recon2": 0.0, "kl0": 85.0486, "kl1": 0.0, "kl2": 0.0}
{"epoch": 51, "phase": 0, "elapsed_s": 21.07, "total": 86.051596, "recon0": 1.002727, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048869, "kl1": 0.0, "kl2": 0.0}
{"epoch": 52, "phase": 0, "elapsed_s": 19.93, "total": 86.051354, "recon0": 1.002855, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048499, "kl1": 0.0, "kl2": 0.0}
{"epoch": 53, "phase": 0, "elapsed_s": 20.48, "total": 86.051221, "recon0": 1.002693, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048528, "kl1": 0.0, "kl2": 0.0}
{"epoch": 54, "phase": 0, "elapsed_s": 20.25, "total": 86.051298, "recon0": 1.002587, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048711, "kl1": 0.0, "kl2": 0.0}
{"epoch": 55, "phase": 0, "elapsed_s": 20.56, "total": 86.051269, "recon0": 1.002714, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048555, "kl1": 0.0, "kl2": 0.0}
{"epoch": 56, "phase": 0, "elapsed_s": 20.46, "total": 86.051004, "recon0": 1.002401, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048603, "kl1": 0.0, "kl2": 0.0}
{"epoch": 57, "phase": 0, "elapsed_s": 21.53, "total": 86.050967, "recon0": 1.002443, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048524, "kl1": 0.0, "kl2": 0.0}
{"epoch": 58, "phase": 0, "elapsed_s": 19.84, "total": 86.051041, "recon0": 1.002475, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048566, "kl1": 0.0, "kl2": 0.0}
{"epoch": 59, "phase": 0, "elapsed_s": 23.26, "total": 86.050996, "recon0": 1.002409, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048587, "kl1": 0.0, "kl2": 0.0}
{"epoch": 60, "phase": 0, "elapsed_s": 23.53, "total": 86.051132, "recon0": 1.002487, "recon1": 0.0, "recon2": 0.0, "kl0": 85.048645, "kl1": 0.0, "kl2": 0.0}
{"epoch": 61, "phase": 1, "elapsed_s": 3.02, "total": 248.66253, "recon0": 0.83884, "recon1": 1.481662, "recon2": 0.0, "kl0": 73.055095, "kl1": 173.286933, "kl2": 0.0}
{"epoch": 62, "phase": 1, "elapsed_s": 3.03, "total": 245.782191, "recon0": 0.774208, "recon1": 1.108398, "recon2": 0.0, "kl0": 73.011229, "kl1": 170.888356, "kl2": 0.0}
{"epoch": 63, "phase": 1, "elapsed_s": 3.17, "total": 245.5468, "recon0": 0.758408, "recon1": 1.074198, "recon2": 0.0, "kl0": 73.010282, "kl1": 170.703912, "kl2": 0.0}
{"epoch": 64, "phase": 1, "elapsed_s": 3.01, "total": 245.415567, "recon0": 0.747258, "recon1": 1.055708, "recon2": 0.0, "kl0": 73.010096, "kl1": 170.602505, "kl2": 0.0}
{"epoch": 65, "phase": 1, "elapsed_s": 2.89, "total": 245.333834, "recon0": 0.734428, "recon1": 1.041786, "recon2": 0.0, "kl0": 73.010233, "kl1": 170.547386, "kl2": 0.0}
{"epoch": 66, "phase": 1, "elapsed_s": 3.13, "total": 245.279483, "recon0": 0.721529, "recon1": 1.033593, "recon2": 0.0, "kl0": 73.010222, "kl1": 170.51414, "kl2": 0.0}
{"epoch": 67, "phase": 1, "elapsed_s": 2.93, "total": 245.226985, "recon0": 0.712027, "recon1": 1.026275, "recon2": 0.0, "kl0": 73.010072, "kl1": 170.47861, "kl2": 0.0}
{"epoch": 68, "phase": 1, "elapsed_s": 3.06, "total": 245.197208, "recon0": 0.705299, "recon1": 1.021018, "recon2": 0.0, "kl0": 73.009803, "kl1": 170.461088, "kl2": 0.0}
{"epoch": 69, "phase": 1, "elapsed_s": 3.24, "total": 245.170304, "recon0": 0.702581, "recon1": 1.016726, "recon2": 0.0, "kl0": 73.00976, "kl1": 170.441237, "kl2": 0.0}
{"epoch": 70, "phase": 1, "elapsed_s": 2.88, "total": 245.152694, "recon0": 0.700606, "recon1": 1.01305, "recon2": 0.0, "kl0": 73.009622, "kl1": 170.429416, "kl2": 0.0}
{"epoch": 71, "phase": 1, "elapsed_s": 3.38, "total": 245.137568, "recon0": 0.700029, "recon1": 1.010956, "recon2": 0.0, "kl0": 73.009604, "kl1": 170.416978, "kl2": 0.0}
{"epoch": 72, "phase": 1, "elapsed_s": 3.38, "total": 245.123892, "recon0": 0.700032, "recon1": 1.008112, "recon2": 0.0, "kl0": 73.009585, "kl1": 170.406163, "kl2": 0.0}
{"epoch": 73, "phase": 1, "elapsed_s": 3.04, "total": 245.111695, "recon0": 0.699438, "recon1": 1.006333, "recon2": 0.0, "kl0": 73.009235, "kl1": 170.396689, "kl2": 0.0}
{"epoch": 74, "phase": 1, "elapsed_s": 3.08, "total": 245.101468, "recon0": 0.699319, "recon1": 1.005132, "recon2": 0.0, "kl0": 73.009247, "kl1": 170.38777, "kl2": 0.0}
{"epoch": 75, "phase": 1, "elapsed_s": 3.06, "total": 245.09494, "recon0": 0.699216, "recon1": 1.003885, "recon2": 0.0, "kl0": 73.009414, "kl1": 170.382425, "kl2": 0.0}
{"epoch": 76, "phase": 1, "elapsed_s": 3.31, "total": 245.088454, "recon0": 0.699423, "recon1": 1.002792, "recon2": 0.0, "kl0": 73.009426, "kl1": 170.376813, "kl2": 0.0}
{"epoch": 77, "phase": 1, "elapsed_s": 3.41, "total": 245.082945, "recon0": 0.699254, "recon1": 1.002356, "recon2": 0.0, "kl0": 73.009426, "kl1": 170.37191, "kl2": 0.0}
{"epoch": 78, "phase": 1, "elapsed_s": 3.25, "total": 245.078702, "recon0": 0.699014, "recon1": 1.002072, "recon2": 0.0, "kl0": 73.009204, "kl1": 170.368413, "kl2": 0.0}
{"epoch": 79, "phase": 1, "elapsed_s": 3.27, "total": 245.077534, "recon0": 0.69883, "recon1": 1.001512, "recon2": 0.0, "kl0": 73.009275, "kl1": 170.367917, "kl2": 0.0}
{"epoch": 80, "phase": 1, "elapsed_s": 3.27, "total": 245.075582, "recon0": 0.699553, "recon1": 1.001304, "recon2": 0.0, "kl0": 73.009385, "kl1": 170.36534, "kl2": 0.0}
{"epoch": 81, "phase": 1, "elapsed_s": 3.35, "total": 245.074042, "recon0": 0.698861, "recon1": 1.000917, "recon2": 0.0, "kl0": 73.009291, "kl1": 170.364973, "kl2": 0.0}
{"epoch": 82, "phase": 1, "elapsed_s": 3.08, "total": 245.071491, "recon0": 0.698884, "recon1": 1.001143, "recon2": 0.0, "kl0": 73.009318, "kl1": 170.362145, "kl2": 0.0}
{"epoch": 83, "phase": 1, "elapsed_s": 2.89, "total": 245.070205, "recon0": 0.698939, "recon1": 1.000905, "recon2": 0.0, "kl0": 73.009463, "kl1": 170.360898, "kl2": 0.0}
{"epoch": 84, "phase": 1, "elapsed_s": 3.37, "total": 245.069235, "recon0": 0.698986, "recon1": 1.000566, "recon2": 0.0, "kl0": 73.009307, "kl1": 170.360375, "kl2": 0.0}
{"epoch": 85, "phase": 1, "elapsed_s": 3.06, "total": 245.066959, "recon0": 0.698673, "recon1": 1.00048, "recon2": 0.0, "kl0": 73.009114, "kl1": 170.358693, "kl2": 0.0}
{"epoch": 86, "phase": 1, "elapsed_s": 3.16, "total": 245.067654, "recon0": 0.699086, "recon1": 1.000543, "recon2": 0.0, "kl0": 73.009304, "kl1": 170.358721, "kl2": 0.0}
{"epoch": 87, "phase": 1, "elapsed_s": 3.47, "total": 245.066236, "recon0": 0.698813, "recon1": 1.000332, "recon2": 0.0, "kl0": 73.009418, "kl1": 170.357673, "kl2": 0.0}
{"epoch": 88, "phase": 1, "elapsed_s": 3.13, "total": 245.068739, "recon0": 0.698586, "recon1": 1.000304, "recon2": 0.0, "kl0": 73.009374, "kl1": 170.360475, "kl2": 0.0}
{"epoch": 89, "phase": 1, "elapsed_s": 3.17, "total": 245.069731, "recon0": 0.698304, "recon1": 1.000321, "recon2": 0.0, "kl0": 73.009159, "kl1": 170.361947, "kl2": 0.0}
{"epoch": 90, "phase": 1, "elapsed_s": 3.25, "total": 245.065129, "recon0": 0.698564, "recon1": 1.000205, "recon2": 0.0, "kl0": 73.009039, "kl1": 170.357321, "kl2": 0.0}
{"epoch": 91, "phase": 1, "elapsed_s": 2.94, "total": 245.065436, "recon0": 0.69884, "recon1": 1.00025, "recon2": 0.0, "kl0": 73.009381, "kl1": 170.356965, "kl2": 0.0}
{"epoch": 92, "phase": 1, "elapsed_s": 3.08, "total": 245.065126, "recon0": 0.698734, "recon1": 1.00013, "recon2": 0.0, "kl0": 73.009328, "kl1": 170.356934, "kl2": 0.0}
{"epoch": 93, "phase": 1, "elapsed_s": 3.26, "total": 245.065383, "recon0": 0.698294, "recon1": 1.000228, "recon2": 0.0, "kl0": 73.009093, "kl1": 170.357768, "kl2": 0.0}
{"epoch": 94, "phase": 1, "elapsed_s": 3.21, "total": 245.064074, "recon0": 0.69861, "recon1": 1.000234, "recon2": 0.0, "kl0": 73.009337, "kl1": 170.355894, "kl2": 0.0}
{"epoch": 95, "phase": 1, "elapsed_s": 3.41, "total": 245.063219, "recon0": 0.698392, "recon1": 1.000168, "recon2": 0.0, "kl0": 73.009203, "kl1": 170.355455, "kl2": 0.0}
{"epoch": 96, "phase": 1, "elapsed_s": 3.21, "total": 245.064144, "recon0": 0.698896, "recon1": 1.00015, "recon2": 0.0, "kl0": 73.009183, "kl1": 170.355915, "kl2": 0.0}
{"epoch": 97, "phase": 1, "elapsed_s": 3.41, "total": 245.063184, "recon0": 0.698235, "recon1": 1.000166, "recon2": 0.0, "kl0": 73.009361, "kl1": 170.355422, "kl2": 0.0}
{"epoch": 98, "phase": 1, "elapsed_s": 3.46, "total": 245.062962, "recon0": 0.698507, "recon1": 1.000071, "recon2": 0.0, "kl0": 73.009056, "kl1": 170.355328, "kl2": 0.0}
{"epoch": 99, "phase": 1, "elapsed_s": 2.94, "total": 245.063443, "recon0": 0.698436, "recon1": 1.000129, "recon2": 0.0, "kl0": 73.009337, "kl1": 170.355542, "kl2": 0.0}
{"epoch": 100, "phase": 1, "elapsed_s": 3.15, "total": 245.064084, "recon0": 0.698279, "recon1": 1.000102, "recon2": 0.0, "kl0": 73.009177, "kl1": 170.356527, "kl2": 0.0}
{"epoch": 101, "phase": 1, "elapsed_s": 2.87, "total": 245.065917, "recon0": 0.698605, "recon1": 1.000079, "recon2": 0.0, "kl0": 73.009277, "kl1": 170.357956, "kl2": 0.0}
{"epoch": 102, "phase": 1, "elapsed_s": 2.97, "total": 245.062561, "recon0": 0.698225, "recon1": 1.000098, "recon2": 0.0, "kl0": 73.009139, "kl1": 170.3551, "kl2": 0.0}
{"epoch": 103, "phase": 1, "elapsed_s": 3.27, "total": 245.062707, "recon0": 0.698316, "recon1": 1.000145, "recon2": 0.0, "kl0": 73.009187, "kl1": 170.355059, "kl2": 0.0}
{"epoch": 104, "phase": 1, "elapsed_s": 2.99, "total": 245.062482, "recon0": 0.698222, "recon1": 1.000031, "recon2": 0.0, "kl0": 73.009093, "kl1": 170.355135, "kl2": 0.0}
{"epoch": 105, "phase": 1, "elapsed_s": 3.08, "total": 245.063394, "recon0": 0.698644, "recon1": 1.000078, "recon2": 0.0, "kl0": 73.009329, "kl1": 170.355344, "kl2": 0.0}
{"epoch": 106, "phase": 1, "elapsed_s": 3.29, "total": 245.066175, "recon0": 0.698594, "recon1": 1.000032, "recon2": 0.0, "kl0": 73.009279, "kl1": 170.35827, "kl2": 0.0}
{"epoch": 107, "phase": 1, "elapsed_s": 3.14, "total": 245.064538, "recon0": 0.698568, "recon1": 1.000093, "recon2": 0.0, "kl0": 73.009305, "kl1": 170.356572, "kl2": 0.0}
{"epoch": 108, "phase": 1, "elapsed_s": 3.1, "total": 245.062603, "recon0": 0.698303, "recon1": 1.00004, "recon2": 0.0, "kl0": 73.009107, "kl1": 170.355153, "kl2": 0.0}
{"epoch": 109, "phase": 1, "elapsed_s": 3.14, "total": 245.062861, "recon0": 0.698076, "recon1": 1.000048, "recon2": 0.0, "kl0": 73.009279, "kl1": 170.355458, "kl2": 0.0}
{"epoch": 110, "phase": 1, "elapsed_s": 3.17, "total": 245.062319, "recon0": 0.698363, "recon1": 1.000067, "recon2": 0.0, "kl0": 73.008946, "kl1": 170.354943, "kl2": 0.0}
{"epoch": 111, "phase": 1, "elapsed_s": 2.86, "total": 245.06317, "recon0": 0.697733, "recon1": 1.000048, "recon2": 0.0, "kl0": 73.00928, "kl1": 170.356109, "kl2": 0.0}
{"epoch": 112, "phase": 1, "elapsed_s": 3.1, "total": 245.063972, "recon0": 0.698594, "recon1": 1.000056, "recon2": 0.0, "kl0": 73.009176, "kl1": 170.356146, "kl2": 0.0}
{"epoch": 113, "phase": 1, "elapsed_s": 3.08, "total": 245.062496, "recon0": 0.698175, "recon1": 1.000032, "recon2": 0.0, "kl0": 73.009197, "kl1": 170.355092, "kl2": 0.0}
{"epoch": 114, "phase": 1, "elapsed_s": 2.75, "total": 245.062393, "recon0": 0.698546, "recon1": 1.000057, "recon2": 0.0, "kl0": 73.008903, "kl1": 170.354888, "kl2": 0.0}
{"epoch": 115, "phase": 1, "elapsed_s": 2.97, "total": 245.062562, "recon0": 0.69792, "recon1": 1.00003, "recon2": 0.0, "kl0": 73.009382, "kl1": 170.35523, "kl2": 0.0}
{"epoch": 116, "phase": 1, "elapsed_s": 2.99, "total": 245.062535, "recon0": 0.698396, "recon1": 1.00003, "recon2": 0.0, "kl0": 73.009251, "kl1": 170.354857, "kl2": 0.0}
{"epoch": 117, "phase": 1, "elapsed_s": 3.0, "total": 245.06206, "recon0": 0.697923, "recon1": 1.000024, "recon2": 0.0, "kl0": 73.008958, "kl1": 170.355155, "kl2": 0.0}
{"epoch": 118, "phase": 1, "elapsed_s": 2.95, "total": 245.062732, "recon0": 0.698185, "recon1": 1.000067, "recon2": 0.0, "kl0": 73.009075, "kl1": 170.355406, "kl2": 0.0}
{"epoch": 119, "phase": 1, "elapsed_s": 3.37, "total": 245.063905, "recon0": 0.698133, "recon1": 1.000064, "recon2": 0.0, "kl0": 73.009053, "kl1": 170.356654, "kl2": 0.0}
{"epoch": 120, "phase": 1, "elapsed_s": 3.07, "total": 245.061189, "recon0": 0.697946, "recon1": 1.000034, "recon2": 0.0, "kl0": 73.009073, "kl1": 170.354136, "kl2": 0.0}
{"epoch": 121, "phase": 1, "elapsed_s": 2.79, "total": 245.06233, "recon0": 0.697913, "recon1": 1.000029, "recon2": 0.0, "kl0": 73.009293, "kl1": 170.355095, "kl2": 0.0}
{"epoch": 122, "phase": 1, "elapsed_s": 3.18, "total": 245.061638, "recon0": 0.698137, "recon1": 1.000039, "recon2": 0.0, "kl0": 73.009173, "kl1": 170.354288, "kl2": 0.0}
{"epoch": 123, "phase": 1, "elapsed_s": 2.94, "total": 245.062985, "recon0": 0.698249, "recon1": 1.000035, "recon2": 0.0, "kl0": 73.009122, "kl1": 170.355579, "kl2": 0.0}
{"epoch": 124, "phase": 1, "elapsed_s": 3.34, "total": 245.065013, "recon0": 0.698215, "recon1": 1.000013, "recon2": 0.0, "kl0": 73.008975, "kl1": 170.357809, "kl2": 0.0}
{"epoch": 125, "phase": 1, "elapsed_s": 3.12, "total": 245.061624, "recon0": 0.697956, "recon1": 1.000065, "recon2": 0.0, "kl0": 73.009121, "kl1": 170.354482, "kl2": 0.0}
{"epoch": 126, "phase": 1, "elapsed_s": 3.01, "total": 245.060936, "recon0": 0.69769, "recon1": 1.000027, "recon2": 0.0, "kl0": 73.009057, "kl1": 170.354162, "kl2": 0.0}
{"epoch": 127, "phase": 1, "elapsed_s": 2.91, "total": 245.061801, "recon0": 0.698012, "recon1": 1.000037, "recon2": 0.0, "kl0": 73.00914, "kl1": 170.354613, "kl2": 0.0}
{"epoch": 128, "phase": 1, "elapsed_s": 2.81, "total": 245.063489, "recon0": 0.697772, "recon1": 1.00003, "recon2": 0.0, "kl0": 73.00912, "kl1": 170.356566, "kl2": 0.0}
{"epoch": 129, "phase": 1, "elapsed_s": 3.18, "total": 245.064945, "recon0": 0.697916, "recon1": 1.00004, "recon2": 0.0, "kl0": 73.009092, "kl1": 170.357898, "kl2": 0.0}
{"epoch": 130, "phase": 1, "elapsed_s": 2.9, "total": 245.061695, "recon0": 0.697916, "recon1": 1.000021, "recon2": 0.0, "kl0": 73.009018, "kl1": 170.35474, "kl2": 0.0}
{"epoch": 131, "phase": 1, "elapsed_s": 2.95, "total": 245.061505, "recon0": 0.698165, "recon1": 1.000035, "recon2": 0.0, "kl0": 73.008899, "kl1": 170.354405, "kl2": 0.0}
{"epoch": 132, "phase": 1, "elapsed_s": 3.13, "total": 245.061918, "recon0": 0.69824, "recon1": 1.000023, "recon2": 0.0, "kl0": 73.009368, "kl1": 170.354287, "kl2": 0.0}
{"epoch": 133, "phase": 1, "elapsed_s": 3.13, "total": 245.061918, "recon0": 0.697828, "recon1": 1.000028, "recon2": 0.0, "kl0": 73.009069, "kl1": 170.354992, "kl2": 0.0}
{"epoch": 134, "phase": 1, "elapsed_s": 3.01, "total": 245.064172, "recon0": 0.698057, "recon1": 1.00003, "recon2": 0.0, "kl0": 73.009184, "kl1": 170.3569, "kl2": 0.0}
{"epoch": 135, "phase": 1, "elapsed_s": 3.07, "total": 245.061791, "recon0": 0.698137, "recon1": 1.000016, "recon2": 0.0, "kl0": 73.009052, "kl1": 170.354585, "kl2": 0.0}
{"epoch": 136, "phase": 1, "elapsed_s": 3.09, "total": 245.06198, "recon0": 0.69812, "recon1": 0.999998, "recon2": 0.0, "kl0": 73.009132, "kl1": 170.35473, "kl2": 0.0}
{"epoch": 137, "phase": 1, "elapsed_s": 2.86, "total": 245.064893, "recon0": 0.697755, "recon1": 1.000018, "recon2": 0.0, "kl0": 73.009252, "kl1": 170.357868, "kl2": 0.0}
{"epoch": 138, "phase": 1, "elapsed_s": 3.2, "total": 245.062171, "recon0": 0.697811, "recon1": 1.000029, "recon2": 0.0, "kl0": 73.008991, "kl1": 170.355339, "kl2": 0.0}
{"epoch": 139, "phase": 1, "elapsed_s": 3.31, "total": 245.061638, "recon0": 0.697711, "recon1": 1.000027, "recon2": 0.0, "kl0": 73.00917, "kl1": 170.35473, "kl2": 0.0}
{"epoch": 140, "phase": 1, "elapsed_s": 3.0, "total": 245.061362, "recon0": 0.697878, "recon1": 1.000018, "recon2": 0.0, "kl0": 73.00913, "kl1": 170.354335, "kl2": 0.0}
{"epoch": 141, "phase": 2, "elapsed_s": 3.5, "total": 419.459798, "recon0": 0.697499, "recon1": 1.000007, "recon2": 1.547828, "kl0": 73.038778, "kl1": 170.354506, "kl2": 172.821181}
{"epoch": 142, "phase": 2, "elapsed_s": 3.55, "total": 416.948685, "recon0": 0.697238, "recon1": 1.00002, "recon2": 1.117019, "kl0": 73.009815, "kl1": 170.354417, "kl2": 170.770176}
{"epoch": 143, "phase": 2, "elapsed_s": 3.3, "total": 416.73089, "recon0": 0.697466, "recon1": 1.000016, "recon2": 1.058987, "kl0": 73.009417, "kl1": 170.35426, "kl2": 170.610743}
{"epoch": 144, "phase": 2, "elapsed_s": 3.35, "total": 416.6285, "recon0": 0.697056, "recon1": 1.000013, "recon2": 1.034927, "kl0": 73.009292, "kl1": 170.354132, "kl2": 170.533081}
{"epoch": 145, "phase": 2, "elapsed_s": 3.32, "total": 416.57063, "recon0": 0.69739, "recon1": 1.00002, "recon2": 1.022492, "kl0": 73.009285, "kl1": 170.354096, "kl2": 170.487348}
{"epoch": 146, "phase": 2, "elapsed_s": 3.24, "total": 416.529586, "recon0": 0.697334, "recon1": 1.000006, "recon2": 1.015536, "kl0": 73.008931, "kl1": 170.354082, "kl2": 170.453697}
{"epoch": 147, "phase": 2, "elapsed_s": 3.42, "total": 416.503925, "recon0": 0.697302, "recon1": 1.000012, "recon2": 1.010783, "kl0": 73.009043, "kl1": 170.353999, "kl2": 170.432786}
{"epoch": 148, "phase": 2, "elapsed_s": 3.35, "total": 416.483341, "recon0": 0.697367, "recon1": 1.000033, "recon2": 1.008334, "kl0": 73.009005, "kl1": 170.353998, "kl2": 170.414606}
{"epoch": 149, "phase": 2, "elapsed_s": 3.38, "total": 416.470644, "recon0": 0.697335, "recon1": 1.00001, "recon2": 1.006469, "kl0": 73.008867, "kl1": 170.354011, "kl2": 170.403952}
{"epoch": 150, "phase": 2, "elapsed_s": 3.54, "total": 416.46048, "recon0": 0.697264, "recon1": 1.000025, "recon2": 1.005497, "kl0": 73.009136, "kl1": 170.353995, "kl2": 170.394564}
{"epoch": 151, "phase": 2, "elapsed_s": 3.31, "total": 416.452242, "recon0": 0.697281, "recon1": 1.000033, "recon2": 1.004312, "kl0": 73.009159, "kl1": 170.35396, "kl2": 170.387496}
{"epoch": 152, "phase": 2, "elapsed_s": 3.31, "total": 416.44725, "recon0": 0.696984, "recon1": 1.000015, "recon2": 1.003759, "kl0": 73.008808, "kl1": 170.353965, "kl2": 170.383718}
{"epoch": 153, "phase": 2, "elapsed_s": 3.52, "total": 416.442094, "recon0": 0.697271, "recon1": 1.000024, "recon2": 1.00271, "kl0": 73.0089, "kl1": 170.353963, "kl2": 170.379226}
{"epoch": 154, "phase": 2, "elapsed_s": 3.19, "total": 416.437258, "recon0": 0.697454, "recon1": 1.00002, "recon2": 1.002411, "kl0": 73.008803, "kl1": 170.35394, "kl2": 170.374629}
{"epoch": 155, "phase": 2, "elapsed_s": 3.26, "total": 416.433166, "recon0": 0.697317, "recon1": 1.000021, "recon2": 1.00206, "kl0": 73.009312, "kl1": 170.353974, "kl2": 170.370483}
{"epoch": 156, "phase": 2, "elapsed_s": 3.46, "total": 416.430844, "recon0": 0.697491, "recon1": 1.000016, "recon2": 1.001901, "kl0": 73.008935, "kl1": 170.354298, "kl2": 170.368202}
{"epoch": 157, "phase": 2, "elapsed_s": 3.27, "total": 416.42795, "recon0": 0.697417, "recon1": 1.000021, "recon2": 1.001588, "kl0": 73.0089, "kl1": 170.354127, "kl2": 170.365896}
{"epoch": 158, "phase": 2, "elapsed_s": 3.23, "total": 416.425389, "recon0": 0.697349, "recon1": 1.000001, "recon2": 1.001314, "kl0": 73.008961, "kl1": 170.354008, "kl2": 170.363755}
{"epoch": 159, "phase": 2, "elapsed_s": 3.48, "total": 416.423755, "recon0": 0.697305, "recon1": 1.000027, "recon2": 1.001166, "kl0": 73.008999, "kl1": 170.354388, "kl2": 170.361869}
{"epoch": 160, "phase": 2, "elapsed_s": 3.49, "total": 416.421273, "recon0": 0.697173, "recon1": 1.000042, "recon2": 1.001066, "kl0": 73.008869, "kl1": 170.353958, "kl2": 170.360165}
{"epoch": 161, "phase": 2, "elapsed_s": 3.52, "total": 416.419849, "recon0": 0.697089, "recon1": 1.000016, "recon2": 1.001035, "kl0": 73.008964, "kl1": 170.354018, "kl2": 170.358727}
{"epoch": 162, "phase": 2, "elapsed_s": 3.5, "total": 416.418892, "recon0": 0.697406, "recon1": 1.000005, "recon2": 1.000808, "kl0": 73.009006, "kl1": 170.354058, "kl2": 170.35761}
{"epoch": 163, "phase": 2, "elapsed_s": 3.28, "total": 416.417835, "recon0": 0.697041, "recon1": 1.000005, "recon2": 1.000716, "kl0": 73.008931, "kl1": 170.354206, "kl2": 170.356935}
{"epoch": 164, "phase": 2, "elapsed_s": 3.19, "total": 416.417671, "recon0": 0.697478, "recon1": 1.000002, "recon2": 1.00073, "kl0": 73.008778, "kl1": 170.354246, "kl2": 170.356437}
{"epoch": 165, "phase": 2, "elapsed_s": 3.4, "total": 416.416473, "recon0": 0.697042, "recon1": 1.000007, "recon2": 1.000516, "kl0": 73.009237, "kl1": 170.354048, "kl2": 170.355622}
{"epoch": 166, "phase": 2, "elapsed_s": 3.28, "total": 416.416383, "recon0": 0.697208, "recon1": 1.000015, "recon2": 1.000557, "kl0": 73.008953, "kl1": 170.35419, "kl2": 170.35546}
{"epoch": 167, "phase": 2, "elapsed_s": 3.24, "total": 416.415974, "recon0": 0.697206, "recon1": 1.000015, "recon2": 1.000499, "kl0": 73.009172, "kl1": 170.354136, "kl2": 170.354946}
{"epoch": 168, "phase": 2, "elapsed_s": 3.36, "total": 416.415526, "recon0": 0.697085, "recon1": 1.000009, "recon2": 1.000492, "kl0": 73.008963, "kl1": 170.354082, "kl2": 170.354896}
{"epoch": 169, "phase": 2, "elapsed_s": 3.26, "total": 416.415312, "recon0": 0.697175, "recon1": 1.000023, "recon2": 1.000413, "kl0": 73.008891, "kl1": 170.354034, "kl2": 170.354775}
{"epoch": 170, "phase": 2, "elapsed_s": 3.24, "total": 416.414749, "recon0": 0.696923, "recon1": 1.000024, "recon2": 1.000263, "kl0": 73.008857, "kl1": 170.354189, "kl2": 170.354494}
{"epoch": 171, "phase": 2, "elapsed_s": 3.42, "total": 416.415318, "recon0": 0.697313, "recon1": 1.000026, "recon2": 1.000343, "kl0": 73.009014, "kl1": 170.354149, "kl2": 170.354473}
{"epoch": 172, "phase": 2, "elapsed_s": 3.25, "total": 416.414913, "recon0": 0.697233, "recon1": 1.000018, "recon2": 1.000312, "kl0": 73.008845, "kl1": 170.354152, "kl2": 170.354352}
{"epoch": 173, "phase": 2, "elapsed_s": 3.29, "total": 416.414671, "recon0": 0.697382, "recon1": 1.000016, "recon2": 1.000194, "kl0": 73.008938, "kl1": 170.353937, "kl2": 170.354205}
{"epoch": 174, "phase": 2, "elapsed_s": 3.41, "total": 416.414863, "recon0": 0.697109, "recon1": 1.000009, "recon2": 1.000218, "kl0": 73.009087, "kl1": 170.354278, "kl2": 170.354162}
{"epoch": 175, "phase": 2, "elapsed_s": 3.79, "total": 416.41501, "recon0": 0.697472, "recon1": 1.000009, "recon2": 1.000132, "kl0": 73.009061, "kl1": 170.354123, "kl2": 170.354214}
{"epoch": 176, "phase": 2, "elapsed_s": 3.28, "total": 416.41478, "recon0": 0.697134, "recon1": 1.000013, "recon2": 1.000191, "kl0": 73.009267, "kl1": 170.353994, "kl2": 170.35418}
{"epoch": 177, "phase": 2, "elapsed_s": 3.39, "total": 416.414477, "recon0": 0.697107, "recon1": 1.000014, "recon2": 1.000152, "kl0": 73.009069, "kl1": 170.354052, "kl2": 170.354084}
{"epoch": 178, "phase": 2, "elapsed_s": 3.32, "total": 416.41435, "recon0": 0.697227, "recon1": 1.000001, "recon2": 1.000192, "kl0": 73.00884, "kl1": 170.353986, "kl2": 170.354104}
{"epoch": 179, "phase": 2, "elapsed_s": 3.67, "total": 416.41469, "recon0": 0.697377, "recon1": 1.000027, "recon2": 1.000115, "kl0": 73.008979, "kl1": 170.35407, "kl2": 170.354124}
{"epoch": 180, "phase": 2, "elapsed_s": 3.82, "total": 416.414384, "recon0": 0.697445, "recon1": 1.00001, "recon2": 1.000138, "kl0": 73.008802, "kl1": 170.353927, "kl2": 170.354061}
{"epoch": 181, "phase": 2, "elapsed_s": 3.88, "total": 416.414117, "recon0": 0.697102, "recon1": 1.000013, "recon2": 1.000153, "kl0": 73.008846, "kl1": 170.354001, "kl2": 170.354001}
{"epoch": 182, "phase": 2, "elapsed_s": 3.54, "total": 416.414253, "recon0": 0.697038, "recon1": 1.000001, "recon2": 1.000107, "kl0": 73.009018, "kl1": 170.354002, "kl2": 170.354086}
{"epoch": 183, "phase": 2, "elapsed_s": 3.37, "total": 416.415049, "recon0": 0.69711, "recon1": 1.000017, "recon2": 1.000132, "kl0": 73.008877, "kl1": 170.354889, "kl2": 170.354024}
{"epoch": 184, "phase": 2, "elapsed_s": 3.46, "total": 416.41447, "recon0": 0.697414, "recon1": 1.000005, "recon2": 1.000106, "kl0": 73.008895, "kl1": 170.354049, "kl2": 170.354}
{"epoch": 185, "phase": 2, "elapsed_s": 3.5, "total": 416.414607, "recon0": 0.697232, "recon1": 1.000013, "recon2": 1.000091, "kl0": 73.009127, "kl1": 170.354125, "kl2": 170.354019}
{"epoch": 186, "phase": 2, "elapsed_s": 3.34, "total": 416.414269, "recon0": 0.697225, "recon1": 1.000005, "recon2": 1.000117, "kl0": 73.008837, "kl1": 170.354094, "kl2": 170.353991}
{"epoch": 187, "phase": 2, "elapsed_s": 3.38, "total": 416.414399, "recon0": 0.697099, "recon1": 1.000014, "recon2": 1.000033, "kl0": 73.009163, "kl1": 170.354099, "kl2": 170.35399}
{"epoch": 188, "phase": 2, "elapsed_s": 3.63, "total": 416.413924, "recon0": 0.696979, "recon1": 1.000019, "recon2": 1.00006, "kl0": 73.008896, "kl1": 170.353977, "kl2": 170.353992}
{"epoch": 189, "phase": 2, "elapsed_s": 3.32, "total": 416.414135, "recon0": 0.69729, "recon1": 1.000007, "recon2": 1.000056, "kl0": 73.00886, "kl1": 170.353932, "kl2": 170.35399}
{"epoch": 190, "phase": 2, "elapsed_s": 3.4, "total": 416.413896, "recon0": 0.697003, "recon1": 1.000006, "recon2": 1.00004, "kl0": 73.008973, "kl1": 170.353942, "kl2": 170.353932}
{"epoch": 191, "phase": 2, "elapsed_s": 3.63, "total": 416.414689, "recon0": 0.697422, "recon1": 1.000004, "recon2": 1.000073, "kl0": 73.009187, "kl1": 170.354008, "kl2": 170.353996}
{"epoch": 192, "phase": 2, "elapsed_s": 3.26, "total": 416.414086, "recon0": 0.696804, "recon1": 1.000003, "recon2": 1.000084, "kl0": 73.008889, "kl1": 170.354316, "kl2": 170.353991}
{"epoch": 193, "phase": 2, "elapsed_s": 3.22, "total": 416.414284, "recon0": 0.697061, "recon1": 1.000018, "recon2": 1.00005, "kl0": 73.009093, "kl1": 170.354127, "kl2": 170.353937}
{"epoch": 194, "phase": 2, "elapsed_s": 3.5, "total": 416.414123, "recon0": 0.697008, "recon1": 1.000012, "recon2": 1.000041, "kl0": 73.009092, "kl1": 170.354018, "kl2": 170.353953}
{"epoch": 195, "phase": 2, "elapsed_s": 3.37, "total": 416.414318, "recon0": 0.69711, "recon1": 1.000011, "recon2": 1.000039, "kl0": 73.009076, "kl1": 170.354119, "kl2": 170.353962}
{"epoch": 196, "phase": 2, "elapsed_s": 3.29, "total": 416.414093, "recon0": 0.697159, "recon1": 1.0, "recon2": 1.00003, "kl0": 73.009028, "kl1": 170.353924, "kl2": 170.353954}
{"epoch": 197, "phase": 2, "elapsed_s": 3.47, "total": 416.414059, "recon0": 0.697126, "recon1": 1.000015, "recon2": 1.000063, "kl0": 73.00895, "kl1": 170.35394, "kl2": 170.353965}
{"epoch": 198, "phase": 2, "elapsed_s": 3.41, "total": 416.413674, "recon0": 0.696809, "recon1": 1.000007, "recon2": 1.00004, "kl0": 73.008957, "kl1": 170.353923, "kl2": 170.353938}
{"epoch": 199, "phase": 2, "elapsed_s": 3.46, "total": 416.414146, "recon0": 0.697251, "recon1": 1.000027, "recon2": 1.000029, "kl0": 73.008881, "kl1": 170.354015, "kl2": 170.353943}
{"epoch": 200, "phase": 2, "elapsed_s": 3.97, "total": 416.414026, "recon0": 0.697094, "recon1": 1.000003, "recon2": 1.000024, "kl0": 73.008901, "kl1": 170.354077, "kl2": 170.353927}
{"epoch": 201, "phase": 2, "elapsed_s": 3.37, "total": 416.414181, "recon0": 0.69716, "recon1": 1.000028, "recon2": 1.000039, "kl0": 73.008937, "kl1": 170.354063, "kl2": 170.353953}
{"epoch": 202, "phase": 2, "elapsed_s": 3.31, "total": 416.413994, "recon0": 0.6972, "recon1": 1.000007, "recon2": 1.000014, "kl0": 73.0089, "kl1": 170.353935, "kl2": 170.353939}
{"epoch": 203, "phase": 2, "elapsed_s": 3.41, "total": 416.414287, "recon0": 0.697101, "recon1": 1.000004, "recon2": 1.000027, "kl0": 73.009286, "kl1": 170.353925, "kl2": 170.353944}
{"epoch": 204, "phase": 2, "elapsed_s": 3.18, "total": 416.414221, "recon0": 0.697214, "recon1": 1.00001, "recon2": 1.000026, "kl0": 73.009061, "kl1": 170.353978, "kl2": 170.353931}
{"epoch": 205, "phase": 2, "elapsed_s": 3.25, "total": 416.414066, "recon0": 0.697245, "recon1": 1.000014, "recon2": 1.000019, "kl0": 73.008892, "kl1": 170.353966, "kl2": 170.35393}
{"epoch": 206, "phase": 2, "elapsed_s": 3.99, "total": 416.414052, "recon0": 0.696878, "recon1": 1.000029, "recon2": 1.000036, "kl0": 73.009079, "kl1": 170.354075, "kl2": 170.353955}
{"epoch": 207, "phase": 2, "elapsed_s": 3.42, "total": 416.414233, "recon0": 0.697036, "recon1": 1.000011, "recon2": 1.000025, "kl0": 73.009149, "kl1": 170.354065, "kl2": 170.353948}
{"epoch": 208, "phase": 2, "elapsed_s": 3.6, "total": 416.414038, "recon0": 0.697133, "recon1": 1.000009, "recon2": 1.000037, "kl0": 73.0089, "kl1": 170.354036, "kl2": 170.353923}
{"epoch": 209, "phase": 2, "elapsed_s": 3.41, "total": 416.414179, "recon0": 0.69714, "recon1": 1.000015, "recon2": 1.000017, "kl0": 73.00907, "kl1": 170.353994, "kl2": 170.353943}
{"epoch": 210, "phase": 2, "elapsed_s": 3.41, "total": 416.413942, "recon0": 0.697198, "recon1": 1.000003, "recon2": 1.000024, "kl0": 73.009031, "kl1": 170.353753, "kl2": 170.353932}
{"epoch": 211, "phase": 2, "elapsed_s": 3.63, "total": 416.414291, "recon0": 0.697067, "recon1": 1.000007, "recon2": 1.000032, "kl0": 73.008993, "kl1": 170.354262, "kl2": 170.353929}
{"epoch": 212, "phase": 2, "elapsed_s": 3.45, "total": 416.414439, "recon0": 0.697169, "recon1": 1.000017, "recon2": 1.000037, "kl0": 73.009101, "kl1": 170.354181, "kl2": 170.353933}
{"epoch": 213, "phase": 2, "elapsed_s": 3.38, "total": 416.414182, "recon0": 0.697123, "recon1": 1.000021, "recon2": 1.000023, "kl0": 73.009107, "kl1": 170.35398, "kl2": 170.353928}
{"epoch": 214, "phase": 2, "elapsed_s": 3.4, "total": 416.413733, "recon0": 0.696946, "recon1": 1.000013, "recon2": 1.000027, "kl0": 73.008813, "kl1": 170.35399, "kl2": 170.353943}
{"epoch": 215, "phase": 2, "elapsed_s": 3.26, "total": 416.4137, "recon0": 0.696891, "recon1": 1.000012, "recon2": 1.000015, "kl0": 73.008898, "kl1": 170.353954, "kl2": 170.35393}
{"epoch": 216, "phase": 2, "elapsed_s": 3.38, "total": 416.413964, "recon0": 0.697152, "recon1": 0.999999, "recon2": 0.999996, "kl0": 73.008948, "kl1": 170.353943, "kl2": 170.353926}
{"epoch": 217, "phase": 2, "elapsed_s": 3.54, "total": 416.413823, "recon0": 0.697037, "recon1": 1.000016, "recon2": 1.000016, "kl0": 73.008807, "kl1": 170.354025, "kl2": 170.353922}
{"epoch": 218, "phase": 2, "elapsed_s": 3.38, "total": 416.414087, "recon0": 0.69706, "recon1": 1.00002, "recon2": 1.000017, "kl0": 73.008978, "kl1": 170.354074, "kl2": 170.353938}
{"epoch": 221, "phase": 3, "elapsed_s": 3.58, "total": 416.414106, "recon0": 0.697099, "recon1": 1.000004, "recon2": 1.000019, "kl0": 73.008981, "kl1": 170.354079, "kl2": 170.353925}
{"epoch": 222, "phase": 3, "elapsed_s": 3.63, "total": 416.413927, "recon0": 0.697022, "recon1": 1.000023, "recon2": 1.000016, "kl0": 73.008971, "kl1": 170.353967, "kl2": 170.353928}
{"epoch": 223, "phase": 3, "elapsed_s": 3.26, "total": 416.41401, "recon0": 0.697006, "recon1": 1.000004, "recon2": 1.000018, "kl0": 73.008785, "kl1": 170.35427, "kl2": 170.353928}
{"epoch": 224, "phase": 3, "elapsed_s": 3.29, "total": 416.414362, "recon0": 0.697088, "recon1": 1.000009, "recon2": 1.00002, "kl0": 73.009043, "kl1": 170.354283, "kl2": 170.35392}
{"epoch": 225, "phase": 3, "elapsed_s": 3.39, "total": 416.41407, "recon0": 0.697127, "recon1": 1.000017, "recon2": 1.000026, "kl0": 73.009031, "kl1": 170.35394, "kl2": 170.353929}
{"epoch": 226, "phase": 3, "elapsed_s": 3.21, "total": 416.41373, "recon0": 0.696975, "recon1": 1.000002, "recon2": 1.000011, "kl0": 73.008757, "kl1": 170.354054, "kl2": 170.353932}
{"epoch": 227, "phase": 3, "elapsed_s": 3.25, "total": 416.414209, "recon0": 0.697186, "recon1": 1.000016, "recon2": 1.000014, "kl0": 73.009075, "kl1": 170.353997, "kl2": 170.353922}
{"epoch": 228, "phase": 3, "elapsed_s": 3.44, "total": 416.41376, "recon0": 0.696987, "recon1": 1.000008, "recon2": 1.000015, "kl0": 73.008896, "kl1": 170.353926, "kl2": 170.353927}
{"epoch": 229, "phase": 3, "elapsed_s": 3.3, "total": 416.413812, "recon0": 0.697042, "recon1": 1.000016, "recon2": 1.000009, "kl0": 73.008845, "kl1": 170.353977, "kl2": 170.353923}
{"epoch": 230, "phase": 3, "elapsed_s": 3.2, "total": 416.413824, "recon0": 0.697019, "recon1": 1.000011, "recon2": 1.000014, "kl0": 73.008926, "kl1": 170.353931, "kl2": 170.353923}
{"epoch": 231, "phase": 3, "elapsed_s": 3.31, "total": 416.413939, "recon0": 0.697144, "recon1": 1.00001, "recon2": 1.000018, "kl0": 73.008929, "kl1": 170.353913, "kl2": 170.353925}
{"epoch": 232, "phase": 3, "elapsed_s": 3.29, "total": 416.414067, "recon0": 0.697243, "recon1": 1.000013, "recon2": 1.00002, "kl0": 73.008882, "kl1": 170.35399, "kl2": 170.353919}
{"epoch": 233, "phase": 3, "elapsed_s": 3.27, "total": 416.413933, "recon0": 0.696989, "recon1": 1.000016, "recon2": 1.000005, "kl0": 73.008957, "kl1": 170.354044, "kl2": 170.353922}
{"epoch": 234, "phase": 3, "elapsed_s": 3.39, "total": 416.41405, "recon0": 0.697205, "recon1": 1.00001, "recon2": 1.000008, "kl0": 73.00889, "kl1": 170.354017, "kl2": 170.353921}
{"epoch": 235, "phase": 3, "elapsed_s": 3.31, "total": 416.414172, "recon0": 0.69699, "recon1": 1.00001, "recon2": 1.000015, "kl0": 73.009055, "kl1": 170.354183, "kl2": 170.353919}
{"epoch": 236, "phase": 3, "elapsed_s": 3.11, "total": 416.414486, "recon0": 0.69741, "recon1": 1.000009, "recon2": 1.000023, "kl0": 73.009012, "kl1": 170.354106, "kl2": 170.353925}
{"epoch": 237, "phase": 3, "elapsed_s": 3.39, "total": 416.413933, "recon0": 0.696992, "recon1": 1.000018, "recon2": 1.000014, "kl0": 73.008817, "kl1": 170.354171, "kl2": 170.35392}
{"epoch": 238, "phase": 3, "elapsed_s": 3.43, "total": 416.413949, "recon0": 0.696992, "recon1": 1.00001, "recon2": 1.00001, "kl0": 73.008966, "kl1": 170.354046, "kl2": 170.353926}
{"epoch": 239, "phase": 3, "elapsed_s": 3.44, "total": 416.414277, "recon0": 0.697343, "recon1": 1.000014, "recon2": 1.00001, "kl0": 73.008922, "kl1": 170.354068, "kl2": 170.35392}
{"epoch": 240, "phase": 3, "elapsed_s": 3.72, "total": 416.414171, "recon0": 0.697048, "recon1": 1.000011, "recon2": 1.00001, "kl0": 73.009135, "kl1": 170.354041, "kl2": 170.353925}
{"epoch": 241, "phase": 3, "elapsed_s": 3.79, "total": 416.414183, "recon0": 0.697185, "recon1": 1.000006, "recon2": 1.00001, "kl0": 73.00899, "kl1": 170.35407, "kl2": 170.353923}
{"epoch": 242, "phase": 3, "elapsed_s": 3.48, "total": 416.414319, "recon0": 0.697187, "recon1": 1.000014, "recon2": 1.000021, "kl0": 73.009097, "kl1": 170.354076, "kl2": 170.353923}
{"epoch": 243, "phase": 3, "elapsed_s": 3.45, "total": 416.413858, "recon0": 0.696978, "recon1": 1.000014, "recon2": 1.000016, "kl0": 73.00886, "kl1": 170.354074, "kl2": 170.353916}
{"epoch": 244, "phase": 3, "elapsed_s": 3.38, "total": 416.414034, "recon0": 0.697054, "recon1": 1.000009, "recon2": 1.000025, "kl0": 73.009041, "kl1": 170.353977, "kl2": 170.353929}
{"epoch": 245, "phase": 3, "elapsed_s": 3.34, "total": 416.414106, "recon0": 0.697039, "recon1": 1.000009, "recon2": 1.00001, "kl0": 73.009195, "kl1": 170.353929, "kl2": 170.353924}
{"epoch": 246, "phase": 3, "elapsed_s": 3.44, "total": 416.413991, "recon0": 0.697185, "recon1": 1.000011, "recon2": 1.000007, "kl0": 73.008929, "kl1": 170.353938, "kl2": 170.35392}
{"epoch": 281, "phase": 4, "elapsed_s": 4.44, "total": 357.508727, "recon0": 0.696738, "recon1": 1.000006, "recon2": 1.000008, "kl0": 62.614084, "kl1": 146.098958, "kl2": 146.098933}
{"epoch": 282, "phase": 4, "elapsed_s": 4.51, "total": 357.508563, "recon0": 0.696756, "recon1": 1.000005, "recon2": 1.000004, "kl0": 62.61393, "kl1": 146.098932, "kl2": 146.098936}
{"epoch": 283, "phase": 4, "elapsed_s": 4.47, "total": 357.508409, "recon0": 0.696588, "recon1": 1.0, "recon2": 1.000006, "kl0": 62.613935, "kl1": 146.098939, "kl2": 146.09894}
{"epoch": 284, "phase": 4, "elapsed_s": 4.48, "total": 357.508368, "recon0": 0.696541, "recon1": 1.000008, "recon2": 1.000014, "kl0": 62.613894, "kl1": 146.098962, "kl2": 146.098949}
{"epoch": 285, "phase": 4, "elapsed_s": 4.59, "total": 357.508472, "recon0": 0.696699, "recon1": 1.000005, "recon2": 1.00001, "kl0": 62.613899, "kl1": 146.09892, "kl2": 146.098938}
{"epoch": 286, "phase": 4, "elapsed_s": 4.27, "total": 357.50857, "recon0": 0.696757, "recon1": 1.000001, "recon2": 1.000009, "kl0": 62.613913, "kl1": 146.098956, "kl2": 146.098934}
{"epoch": 287, "phase": 4, "elapsed_s": 4.64, "total": 357.508573, "recon0": 0.69673, "recon1": 1.000006, "recon2": 1.000003, "kl0": 62.613953, "kl1": 146.098941, "kl2": 146.098939}
{"epoch": 288, "phase": 4, "elapsed_s": 4.34, "total": 357.508487, "recon0": 0.696724, "recon1": 1.000005, "recon2": 1.000002, "kl0": 62.613881, "kl1": 146.098938, "kl2": 146.098937}
{"epoch": 289, "phase": 4, "elapsed_s": 4.47, "total": 357.508513, "recon0": 0.696767, "recon1": 1.000003, "recon2": 1.000005, "kl0": 62.613871, "kl1": 146.098933, "kl2": 146.098935}
{"epoch": 290, "phase": 4, "elapsed_s": 4.15, "total": 357.508573, "recon0": 0.696788, "recon1": 1.000002, "recon2": 1.000013, "kl0": 62.613888, "kl1": 146.098946, "kl2": 146.098936}
{"epoch": 291, "phase": 4, "elapsed_s": 4.53, "total": 357.508354, "recon0": 0.696605, "recon1": 1.000005, "recon2": 1.000002, "kl0": 62.613871, "kl1": 146.098935, "kl2": 146.098935}
{"epoch": 292, "phase": 4, "elapsed_s": 4.41, "total": 357.508821, "recon0": 0.696871, "recon1": 1.000002, "recon2": 1.000004, "kl0": 62.614074, "kl1": 146.098936, "kl2": 146.098932}
{"epoch": 293, "phase": 4, "elapsed_s": 4.43, "total": 357.508301, "recon0": 0.696768, "recon1": 1.000006, "recon2": 1.000003, "kl0": 62.613648, "kl1": 146.098941, "kl2": 146.098936}
{"epoch": 294, "phase": 4, "elapsed_s": 4.39, "total": 357.508497, "recon0": 0.696589, "recon1": 1.000005, "recon2": 1.000003, "kl0": 62.613997, "kl1": 146.098967, "kl2": 146.098935}
{"epoch": 295, "phase": 4, "elapsed_s": 4.28, "total": 357.508329, "recon0": 0.696556, "recon1": 1.000004, "recon2": 1.0, "kl0": 62.613905, "kl1": 146.098929, "kl2": 146.098936}
{"epoch": 296, "phase": 4, "elapsed_s": 4.45, "total": 357.508607, "recon0": 0.696642, "recon1": 0.999999, "recon2": 1.000002, "kl0": 62.614094, "kl1": 146.098932, "kl2": 146.098937}
{"epoch": 297, "phase": 4, "elapsed_s": 4.43, "total": 357.508523, "recon0": 0.696731, "recon1": 1.000005, "recon2": 0.999999, "kl0": 62.613906, "kl1": 146.098945, "kl2": 146.098936}
{"epoch": 298, "phase": 4, "elapsed_s": 4.77, "total": 357.508516, "recon0": 0.696572, "recon1": 1.000006, "recon2": 1.000012, "kl0": 62.614058, "kl1": 146.098934, "kl2": 146.098934}
{"epoch": 299, "phase": 4, "elapsed_s": 4.45, "total": 357.508349, "recon0": 0.696625, "recon1": 1.000005, "recon2": 1.00001, "kl0": 62.613822, "kl1": 146.098951, "kl2": 146.098936}
{"epoch": 300, "phase": 4, "elapsed_s": 4.56, "total": 357.508386, "recon0": 0.696627, "recon1": 1.000006, "recon2": 1.000009, "kl0": 62.613864, "kl1": 146.098941, "kl2": 146.098939}
{"epoch": 301, "phase": 4, "elapsed_s": 4.6, "total": 357.508455, "recon0": 0.696721, "recon1": 1.000011, "recon2": 1.000016, "kl0": 62.613827, "kl1": 146.098948, "kl2": 146.098931}
{"epoch": 302, "phase": 4, "elapsed_s": 4.68, "total": 357.508406, "recon0": 0.696691, "recon1": 1.000002, "recon2": 1.000005, "kl0": 62.613836, "kl1": 146.098933, "kl2": 146.09894}
{"epoch": 303, "phase": 4, "elapsed_s": 4.67, "total": 357.508657, "recon0": 0.696732, "recon1": 0.999999, "recon2": 1.000006, "kl0": 62.61404, "kl1": 146.098945, "kl2": 146.098935}
{"epoch": 304, "phase": 4, "elapsed_s": 4.82, "total": 357.50847, "recon0": 0.69667, "recon1": 1.000002, "recon2": 1.000004, "kl0": 62.613923, "kl1": 146.098937, "kl2": 146.098933}
{"epoch": 305, "phase": 4, "elapsed_s": 4.4, "total": 357.508318, "recon0": 0.696694, "recon1": 1.000004, "recon2": 1.000003, "kl0": 62.613749, "kl1": 146.098931, "kl2": 146.098936}
{"epoch": 306, "phase": 4, "elapsed_s": 4.14, "total": 357.50834, "recon0": 0.696641, "recon1": 1.000001, "recon2": 1.000013, "kl0": 62.613814, "kl1": 146.098937, "kl2": 146.098935}
{"epoch": 307, "phase": 4, "elapsed_s": 4.87, "total": 357.508649, "recon0": 0.696624, "recon1": 1.000004, "recon2": 1.000008, "kl0": 62.614122, "kl1": 146.098955, "kl2": 146.098936}
{"epoch": 308, "phase": 4, "elapsed_s": 4.49, "total": 357.508615, "recon0": 0.696745, "recon1": 1.000002, "recon2": 1.000004, "kl0": 62.613979, "kl1": 146.09895, "kl2": 146.098935}
{"epoch": 309, "phase": 4, "elapsed_s": 4.42, "total": 357.508191, "recon0": 0.696586, "recon1": 1.000003, "recon2": 1.000005, "kl0": 62.613723, "kl1": 146.098937, "kl2": 146.098936}
{"epoch": 310, "phase": 4, "elapsed_s": 4.43, "total": 357.508561, "recon0": 0.696743, "recon1": 1.000003, "recon2": 1.000004, "kl0": 62.613923, "kl1": 146.098952, "kl2": 146.098936}
{"epoch": 311, "phase": 4, "elapsed_s": 4.49, "total": 357.508497, "recon0": 0.696693, "recon1": 1.000002, "recon2": 1.000012, "kl0": 62.613908, "kl1": 146.098945, "kl2": 146.098937}
{"epoch": 312, "phase": 4, "elapsed_s": 4.45, "total": 357.508555, "recon0": 0.696741, "recon1": 1.000002, "recon2": 1.000003, "kl0": 62.613921, "kl1": 146.098948, "kl2": 146.098939}
{"epoch": 313, "phase": 4, "elapsed_s": 4.68, "total": 357.508473, "recon0": 0.696463, "recon1": 1.000002, "recon2": 1.000002, "kl0": 62.614119, "kl1": 146.098947, "kl2": 146.09894}
{"epoch": 314, "phase": 4, "elapsed_s": 4.36, "total": 357.508525, "recon0": 0.696689, "recon1": 1.000005, "recon2": 1.000006, "kl0": 62.613928, "kl1": 146.098959, "kl2": 146.098938}
{"epoch": 315, "phase": 4, "elapsed_s": 4.2, "total": 357.508567, "recon0": 0.696621, "recon1": 0.999999, "recon2": 1.000006, "kl0": 62.613958, "kl1": 146.099045, "kl2": 146.098937}
{"epoch": 316, "phase": 4, "elapsed_s": 4.42, "total": 357.508457, "recon0": 0.696538, "recon1": 1.000003, "recon2": 1.000003, "kl0": 62.614006, "kl1": 146.098971, "kl2": 146.098936}
{"epoch": 317, "phase": 4, "elapsed_s": 4.1, "total": 357.508585, "recon0": 0.696622, "recon1": 1.000002, "recon2": 1.000005, "kl0": 62.614085, "kl1": 146.098937, "kl2": 146.098933}
{"epoch": 318, "phase": 4, "elapsed_s": 4.49, "total": 357.508534, "recon0": 0.696646, "recon1": 1.0, "recon2": 0.999997, "kl0": 62.614013, "kl1": 146.098942, "kl2": 146.098936}
{"epoch": 319, "phase": 4, "elapsed_s": 4.16, "total": 357.508534, "recon0": 0.696725, "recon1": 1.000006, "recon2": 1.000009, "kl0": 62.613885, "kl1": 146.098977, "kl2": 146.098933}
{"epoch": 320, "phase": 4, "elapsed_s": 4.46, "total": 357.508603, "recon0": 0.696733, "recon1": 1.000003, "recon2": 1.000007, "kl0": 62.613948, "kl1": 146.09898, "kl2": 146.098932}
{"epoch": 321, "phase": 4, "elapsed_s": 4.27, "total": 357.508288, "recon0": 0.696632, "recon1": 1.000006, "recon2": 1.000006, "kl0": 62.613758, "kl1": 146.098947, "kl2": 146.098939}
{"epoch": 322, "phase": 4, "elapsed_s": 4.46, "total": 357.508406, "recon0": 0.696551, "recon1": 1.000005, "recon2": 1.000007, "kl0": 62.613964, "kl1": 146.098946, "kl2": 146.098933}
{"epoch": 323, "phase": 4, "elapsed_s": 4.61, "total": 357.508458, "recon0": 0.696554, "recon1": 1.000002, "recon2": 1.000002, "kl0": 62.613944, "kl1": 146.099022, "kl2": 146.098934}
{"epoch": 324, "phase": 4, "elapsed_s": 4.24, "total": 357.50832, "recon0": 0.696757, "recon1": 1.000002, "recon2": 1.000004, "kl0": 62.613696, "kl1": 146.098924, "kl2": 146.098936}
{"epoch": 325, "phase": 4, "elapsed_s": 4.57, "total": 357.508433, "recon0": 0.696713, "recon1": 1.0, "recon2": 1.000005, "kl0": 62.613782, "kl1": 146.098999, "kl2": 146.098934}
{"epoch": 326, "phase": 4, "elapsed_s": 4.08, "total": 357.50814, "recon0": 0.696502, "recon1": 1.000002, "recon2": 1.000006, "kl0": 62.613743, "kl1": 146.098949, "kl2": 146.098938}
{"epoch": 327, "phase": 4, "elapsed_s": 4.37, "total": 357.508528, "recon0": 0.696822, "recon1": 1.000004, "recon2": 1.000014, "kl0": 62.613728, "kl1": 146.099025, "kl2": 146.098935}
{"epoch": 328, "phase": 4, "elapsed_s": 4.51, "total": 357.508499, "recon0": 0.696682, "recon1": 1.000003, "recon2": 1.000002, "kl0": 62.613899, "kl1": 146.098977, "kl2": 146.098935}
{"epoch": 329, "phase": 4, "elapsed_s": 4.56, "total": 357.508235, "recon0": 0.69663, "recon1": 1.000003, "recon2": 1.000008, "kl0": 62.613703, "kl1": 146.098952, "kl2": 146.098939}
{"epoch": 330, "phase": 4, "elapsed_s": 4.2, "total": 357.508445, "recon0": 0.696646, "recon1": 1.000003, "recon2": 1.00001, "kl0": 62.613887, "kl1": 146.098963, "kl2": 146.098936}
{"epoch": 331, "phase": 4, "elapsed_s": 3.97, "total": 357.508426, "recon0": 0.696552, "recon1": 1.000007, "recon2": 1.000009, "kl0": 62.613976, "kl1": 146.098946, "kl2": 146.098936}
{"epoch": 332, "phase": 4, "elapsed_s": 4.28, "total": 357.508478, "recon0": 0.696749, "recon1": 1.000002, "recon2": 1.000006, "kl0": 62.613837, "kl1": 146.098953, "kl2": 146.098932}
{"epoch": 333, "phase": 4, "elapsed_s": 4.15, "total": 357.508253, "recon0": 0.696554, "recon1": 1.000001, "recon2": 1.000001, "kl0": 62.613824, "kl1": 146.098937, "kl2": 146.098935}
{"epoch": 334, "phase": 4, "elapsed_s": 4.26, "total": 357.508693, "recon0": 0.696684, "recon1": 1.000001, "recon2": 1.000003, "kl0": 62.614064, "kl1": 146.099007, "kl2": 146.098934}
{"epoch": 335, "phase": 4, "elapsed_s": 4.06, "total": 357.508652, "recon0": 0.696877, "recon1": 1.000007, "recon2": 1.000006, "kl0": 62.613908, "kl1": 146.098917, "kl2": 146.098937}
{"epoch": 336, "phase": 4, "elapsed_s": 4.39, "total": 357.508355, "recon0": 0.696603, "recon1": 1.000003, "recon2": 1.0, "kl0": 62.613866, "kl1": 146.098948, "kl2": 146.098935}
{"epoch": 337, "phase": 4, "elapsed_s": 4.78, "total": 357.508362, "recon0": 0.696534, "recon1": 1.000005, "recon2": 1.000003, "kl0": 62.613953, "kl1": 146.098933, "kl2": 146.098934}
{"epoch": 338, "phase": 4, "elapsed_s": 4.55, "total": 357.508468, "recon0": 0.696718, "recon1": 1.000014, "recon2": 1.000005, "kl0": 62.613861, "kl1": 146.098934, "kl2": 146.098936}
{"epoch": 339, "phase": 4, "elapsed_s": 4.34, "total": 357.508602, "recon0": 0.696712, "recon1": 1.000005, "recon2": 1.000005, "kl0": 62.61396, "kl1": 146.098986, "kl2": 146.098935}
{"epoch": 340, "phase": 4, "elapsed_s": 3.92, "total": 357.50841, "recon0": 0.696642, "recon1": 1.000002, "recon2": 1.000002, "kl0": 62.613887, "kl1": 146.098943, "kl2": 146.098935}
{"epoch": 341, "phase": 4, "elapsed_s": 4.25, "total": 357.508368, "recon0": 0.696701, "recon1": 1.000003, "recon2": 1.000008, "kl0": 62.61379, "kl1": 146.098932, "kl2": 146.098935}
{"epoch": 342, "phase": 4, "elapsed_s": 4.57, "total": 357.508829, "recon0": 0.696823, "recon1": 0.999998, "recon2": 1.000002, "kl0": 62.614117, "kl1": 146.098948, "kl2": 146.098941}
{"epoch": 343, "phase": 4, "elapsed_s": 4.38, "total": 357.508626, "recon0": 0.69677, "recon1": 1.000005, "recon2": 1.000001, "kl0": 62.613955, "kl1": 146.098961, "kl2": 146.098934}
{"epoch": 344, "phase": 4, "elapsed_s": 4.23, "total": 357.508445, "recon0": 0.696646, "recon1": 1.000008, "recon2": 1.000008, "kl0": 62.613883, "kl1": 146.098963, "kl2": 146.098937}
{"epoch": 345, "phase": 4, "elapsed_s": 4.08, "total": 357.50854, "recon0": 0.696736, "recon1": 0.999999, "recon2": 1.0, "kl0": 62.613923, "kl1": 146.098949, "kl2": 146.098933}
{"epoch": 346, "phase": 4, "elapsed_s": 4.25, "total": 357.508199, "recon0": 0.696577, "recon1": 1.000002, "recon2": 1.000009, "kl0": 62.613729, "kl1": 146.098944, "kl2": 146.098938}

245
nautilus_dolphin/dvae/z17_signal_analysis.py Executable file
View File

@@ -0,0 +1,245 @@
"""
z1[7] Mechanical Simplification + FLINT 512-bit amplification.
z1[7] VAE finding:
w750_vel_norm r=-0.674 (long-term velocity reversing)
w300_vel_norm r=-0.357
w50_instability r=+0.421 (short-term chaos spiking)
= "eigenspace stress reversal"
Steps:
1. Mechanical proxy: linear combo from correlations
2. Different activations
3. PCA comparison
4. EDAIN-KL + HD 512-dim projection (FLINT)
"""
import sys, os
sys.stdout.reconfigure(encoding='utf-8', errors='replace')
sys.path.insert(0, os.path.dirname(__file__))
sys.path.insert(0, r"C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict")
import numpy as np
from pathlib import Path
HERE = Path(__file__).parent
WINDOWS = [50, 150, 300, 750]
T1_NAMES = []
for w in WINDOWS:
for f in ['log_lmax', 'vel_norm', 'gap_ratio', 'instability', 'rtp']:
T1_NAMES.append(f"w{w}_{f}")
T0_NAMES = ['bull_pct', 'bear_pct', 'side_pct', 'sin_hour', 'cos_hour', 'sin_day', 'cos_day', 'has_eigen']
print("Loading corpus...")
from corpus_builder import DolphinCorpus, OFF, T1 as T1_DIM
corpus = DolphinCorpus.load(str(HERE / 'corpus_cache.npz'))
idx = corpus.mask[:, 1]
X_e = corpus.X[idx]
mask_e = corpus.mask[idx]
t1 = X_e[:, OFF[1]:OFF[1]+T1_DIM].copy() # (16607, 20)
t0 = X_e[:, OFF[0]:OFF[0]+8].copy()
print(f"T1 shape: {t1.shape}")
# Feature shortcuts
vel_w50 = t1[:, 1]
vel_w150 = t1[:, 6]
vel_w300 = t1[:, 11]
vel_w750 = t1[:, 16]
inst_w50 = t1[:, 3]
inst_w300= t1[:, 13]
lmax_w50 = t1[:, 0]
lmax_w750= t1[:, 15]
# ================================================================
# 1. MECHANICAL PROXIES
# ================================================================
print("\n" + "="*65)
print("1. MECHANICAL z1[7] PROXIES")
print("="*65)
proxy_A = -0.674*vel_w750 - 0.357*vel_w300 + 0.421*inst_w50
proxy_B = inst_w50 - vel_w750
proxy_C = vel_w50 - vel_w750
proxy_D = inst_w50 * (-vel_w750)
proxy_E = (inst_w50 - inst_w300) - (vel_w50 - vel_w750) # instability delta + vel divergence
for name, p in [
('A: linear VAE weights', proxy_A),
('B: inst_w50 - vel_w750', proxy_B),
('C: vel_w50 - vel_w750 (cross-scale divergence)', proxy_C),
('D: inst_w50 * -vel_w750 (interaction)', proxy_D),
('E: dinst - dvel (delta-delta)', proxy_E),
]:
nans = np.isnan(p).sum()
pv = p[np.isfinite(p)]
print(f"\n{name}:")
print(f" std={pv.std():.4f} p5={np.percentile(pv,5):.4f} "
f"p50={np.percentile(pv,50):.4f} p95={np.percentile(pv,95):.4f} NaN={nans}")
skew = float(((pv - pv.mean())**3).mean() / (pv.std()**3 + 1e-8))
kurt = float(((pv - pv.mean())**4).mean() / (pv.std()**4 + 1e-8))
print(f" skew={skew:.3f} kurt={kurt:.2f} (>3 = heavy tails = signal potential)")
# ================================================================
# 2. WHAT DOES HIGH proxy_B LOOK LIKE IN FULL T1 SPACE?
# ================================================================
print("\n" + "="*65)
print("2. HIGH vs LOW proxy_B = inst_w50 - vel_w750")
print("="*65)
p = proxy_B
lo = p < np.percentile(p, 10)
hi = p > np.percentile(p, 90)
print(f"N low={lo.sum()} N high={hi.sum()}")
print(f"\n{'Feature':<22} {'ALL':>8} {'LOW10%':>8} {'HIGH10%':>8} {'diff':>8}")
for i, name in enumerate(T1_NAMES):
a = t1[:, i].mean()
l = t1[lo, i].mean()
h = t1[hi, i].mean()
d = h - l
if abs(d) > 0.02:
print(f" {name:<20} {a:8.4f} {l:8.4f} {h:8.4f} {d:+8.4f}")
print(f"\n T0 context:")
for i, name in enumerate(T0_NAMES[:3]):
print(f" {name:<20} {t0[:,i].mean():8.3f} {t0[lo,i].mean():8.3f} {t0[hi,i].mean():8.3f} {t0[hi,i].mean()-t0[lo,i].mean():+8.3f}")
# ================================================================
# 3. ACTIVATION FUNCTIONS
# ================================================================
print("\n" + "="*65)
print("3. ACTIVATION FUNCTIONS ON proxy_B")
print("="*65)
def softplus(x):
return np.log1p(np.exp(np.clip(x, -30, 30)))
def sigmoid(x):
return 1.0 / (1.0 + np.exp(-np.clip(x, -30, 30)))
pb_z = (proxy_B - proxy_B.mean()) / (proxy_B.std() + 1e-8)
for name, act in [
('raw z-score', pb_z),
('relu', np.maximum(0, pb_z)),
('tanh', np.tanh(pb_z)),
('softplus', softplus(pb_z)),
('sigmoid', sigmoid(pb_z)),
('sign*log1p(|x|)', np.sign(pb_z) * np.log1p(np.abs(pb_z))),
('x^3 (cubic)', pb_z**3),
]:
if np.isnan(act).any():
print(f" {name:<22} NaN!")
continue
skew = float(((act - act.mean())**3).mean() / (act.std()**3 + 1e-8))
kurt = float(((act - act.mean())**4).mean() / (act.std()**4 + 1e-8))
# High kurtosis = heavy tails = outliers preserved = signal in extremes
print(f" {name:<22} std={act.std():.4f} skew={skew:+.3f} kurt={kurt:6.1f}")
# ================================================================
# 4. PCA vs VAE
# ================================================================
print("\n" + "="*65)
print("4. PCA ON T1: linear basis vs VAE z1 basis")
print("="*65)
t1_z = (t1 - t1.mean(0)) / (t1.std(0) + 1e-8)
U, S, Vt = np.linalg.svd(t1_z, full_matrices=False)
explained = S**2 / (S**2).sum()
print(f"Explained variance: {np.round(100*explained[:8], 1)}%")
for pc_i in range(5):
loadings = Vt[pc_i]
top3 = np.argsort(np.abs(loadings))[-3:][::-1]
pc_scores = t1_z @ Vt[pc_i]
r_pb = np.corrcoef(pc_scores, proxy_B)[0,1]
print(f"PC{pc_i+1} ({100*explained[pc_i]:.1f}%) r(proxy_B)={r_pb:+.3f}: "
+ " ".join(f"{T1_NAMES[j]}={loadings[j]:+.3f}" for j in top3))
# ================================================================
# 5. FLINT EDAIN-KL + HD PROJECTION
# ================================================================
print("\n" + "="*65)
print("5. FLINT: EDAIN-KL + HD PROJECTION 20->512")
print("="*65)
try:
from SILOQY_NN_Kernel_COMPLETE6 import MCDAINLayer
FLINT_OK = True
print("FLINT OK")
except ImportError as e:
FLINT_OK = False
print(f"FLINT import failed: {e}")
if FLINT_OK:
# 5a: MCDAINLayer (no training needed, NaN-safe)
print("\n5a: MCDAINLayer normalization:")
t1_f32 = t1.astype(np.float32)
try:
import torch
t1_torch = torch.from_numpy(t1_f32)
mc = MCDAINLayer(input_dim=20)
with torch.no_grad():
t1_mc = mc(t1_torch).numpy()
mc_std = t1_mc.std(0)
print(f" Per-dim std after MCDAIN: {mc_std.round(3)}")
print(f" Max std: {mc_std.max():.4f} Min std: {mc_std.min():.4f}")
# Proxy_B in MCDAIN space
pb_mc = t1_mc[:, 3] - t1_mc[:, 16] # inst_w50 - vel_w750
print(f" proxy_B (MCDAIN): std={pb_mc.std():.4f} "
f"skew={float(((pb_mc-pb_mc.mean())**3).mean()/(pb_mc.std()**3+1e-8)):.3f} "
f"kurt={float(((pb_mc-pb_mc.mean())**4).mean()/(pb_mc.std()**4+1e-8)):.1f}")
print(f" Variance ratio MCDAIN/raw: {pb_mc.var()/proxy_B.var():.3f}x")
except Exception as ex:
print(f" MCDAINLayer: {ex}")
# 5b: HD projection 20->512 with ReLU
print("\n5b: HD projection (20->512, ReLU):")
try:
np.random.seed(42)
W_hd = np.random.randn(20, 512).astype(np.float32) * np.sqrt(2.0/20)
t1_n = t1_mc if 't1_mc' in dir() else t1_z.astype(np.float32)
H = np.maximum(0, t1_n @ W_hd) # (16607, 512)
hd_var = H.var(0)
active = (hd_var > 0.01).sum()
print(f" Active HD dims (var>0.01): {active}/512")
# Which HD dims correlate most with proxy_B?
pb_n = (proxy_B - proxy_B.mean()) / (proxy_B.std() + 1e-8)
hd_r = np.array([np.corrcoef(pb_n, H[:, d])[0, 1] for d in range(512)])
print(f" |r(HD, proxy_B)| > 0.3: {(np.abs(hd_r)>0.3).sum()} dims")
print(f" |r(HD, proxy_B)| > 0.5: {(np.abs(hd_r)>0.5).sum()} dims")
print(f" max |r|: {np.abs(hd_r).max():.4f}")
top5 = np.argsort(np.abs(hd_r))[-5:][::-1]
print(f" Top HD dims:")
for d in top5:
print(f" HD[{d:3d}]: r={hd_r[d]:+.4f} var={hd_var[d]:.4f}")
# 5c: Can HD space reconstruct proxy_B better than PCA?
print("\n5c: Ridge regression HD->proxy_B (R^2 comparison):")
from numpy.linalg import lstsq
# PCA baseline: PC1 alone
pc1_scores = (t1_z @ Vt[0]).reshape(-1, 1)
coef_pc1, _, _, _ = lstsq(pc1_scores, proxy_B, rcond=None)
pred_pc1 = pc1_scores @ coef_pc1
ss_res = ((proxy_B - pred_pc1)**2).sum()
ss_tot = ((proxy_B - proxy_B.mean())**2).sum()
r2_pc1 = 1 - ss_res/ss_tot
print(f" PC1 alone R2={r2_pc1:.4f}")
# Top 8 PCs
pc8 = (t1_z @ Vt[:8].T)
coef8, _, _, _ = lstsq(pc8, proxy_B, rcond=None)
pred8 = pc8 @ coef8
r2_pc8 = 1 - ((proxy_B - pred8)**2).sum()/ss_tot
print(f" Top 8 PCs R2={r2_pc8:.4f}")
# Top 8 HD dims
top8_hd = np.argsort(np.abs(hd_r))[-8:]
H8 = H[:, top8_hd]
coef_hd8, _, _, _ = lstsq(H8, proxy_B, rcond=None)
pred_hd8 = H8 @ coef_hd8
r2_hd8 = 1 - ((proxy_B - pred_hd8)**2).sum()/ss_tot
print(f" Top 8 HD dims R2={r2_hd8:.4f}")
print(f" HD/PCA improvement: {r2_hd8/max(r2_pc8, 1e-8):.3f}x")
except Exception as ex:
import traceback; traceback.print_exc()
print("\nDone.")