hjnormey/DOLPHIN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
exp/green-s6-esof-aem-shadow-2026-04-21
DOLPHIN/prod/docs/SYSTEM_FILE_MAP.md

616 lines
30 KiB
Markdown
Raw Permalink Normal View History

initial: import DOLPHIN baseline 2026-04-21 from dolphinng5_predict working tree Includes core prod + GREEN/BLUE subsystems: - prod/ (BLUE harness, configs, scripts, docs) - nautilus_dolphin/ (GREEN Nautilus-native impl + dvae/ preserved) - adaptive_exit/ (AEM engine + models/bucket_assignments.pkl) - Observability/ (EsoF advisor, TUI, dashboards) - external_factors/ (EsoF producer) - mc_forewarning_qlabs_fork/ (MC regime/envelope) Excludes runtime caches, logs, backups, and reproducible artifacts per .gitignore.
2026-04-21 16:58:38 +02:00
# DOLPHIN-NAUTILUS System File Map
## Authoritative location reference for all critical code, engines, and data
**Version:** 1.0
**Date:** 2026-03-22
**Scope:** All subsystems running on DOLPHIN (Linux) with cross-platform path references where shares exist
---
## TABLE OF CONTENTS
1. [Path Resolution — Cross-Platform Key](#1-path-resolution)
2. [Subsystem A — Alpha Engine Core](#2-alpha-engine-core)
3. [Subsystem B — Nautilus Trader Integration](#3-nautilus-trader-integration)
4. [Subsystem C — Order Book Features (OBF)](#4-order-book-features)
5. [Subsystem D — External Factors (ExF)](#5-external-factors)
6. [Subsystem E — Eigenvalue Scanner & ACB](#6-eigenvalue-scanner--acb)
7. [Subsystem F — Monte Carlo Forewarner](#7-monte-carlo-forewarner)
8. [Subsystem G — Prefect Orchestration](#8-prefect-orchestration)
9. [Subsystem H — Hazelcast In-Memory Grid](#9-hazelcast-in-memory-grid)
10. [Subsystem I — ML Models & DVAE](#10-ml-models--dvae)
11. [Subsystem J — Survival Stack / Watchdog](#11-survival-stack--watchdog)
12. [Data Locations — All Stores](#12-data-locations)
13. [Config & Deployment Files](#13-config--deployment-files)
14. [Test Suites](#14-test-suites)
15. [Utility & Scripts](#15-utility--scripts)
---
## 1. PATH RESOLUTION
**Authoritative resolver:** `nautilus_dolphin/dolphin_paths.py`
All code consuming shared data MUST import from this module — never hardcode paths.
```python
from dolphin_paths import (
get_arb512_storage_root, # NG3/NG6 correlation_arb512 root
get_eigenvalues_path, # per-date eigenvalue + ExF snapshots
get_project_root, # NG5 Predict root
get_vbt_cache_dir, # VBT vector parquet cache
get_klines_dir, # 5yr 5s klines parquet
get_arrow_backfill_dir, # Arrow + synthetic backfill
)
```
### Cross-Platform Path Table
| Logical Name | Linux Path | Windows Path | SMB Share |
|---|---|---|---|
| **NG3/NG6 Data Root** | `/mnt/ng6_data` | `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512` | `DolphinNG6_Data` |
| **Eigenvalues** | `/mnt/ng6_data/eigenvalues/` | `…\correlation_arb512\eigenvalues\` | DolphinNG6_Data |
| **NG5 Predict Root** | `/mnt/dolphin` | `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict` | `DolphinNG5_Predict` |
| **VBT Cache (vectors)** | `/mnt/dolphin/vbt_cache/` | `…\DOLPHIN NG HD HCM TSF Predict\vbt_cache\` | DolphinNG5_Predict |
| **Klines (5s, preferred)** | `/mnt/dolphin/vbt_cache_klines/` | `…\DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\` | DolphinNG5_Predict |
| **Arrow Backfill** | `/mnt/dolphin/arrow_backfill/` | `…\DOLPHIN NG HD HCM TSF Predict\arrow_backfill\` | DolphinNG5_Predict |
| **DOLPHIN Local Root** | `/mnt/dolphinng5_predict/` | *(local on DOLPHIN machine)* | — |
| **NG6 Sparse** | `/mnt/ng6/` | — | `DolphinNG6` |
> **SMB Server:** `100.119.158.61` (Tailscale, Windows machine)
> **Prefect Server:** `http://100.105.170.6:4200` (Tailscale)
> **Hazelcast:** `localhost:5701` (Docker on DOLPHIN)
---
## 2. ALPHA ENGINE CORE
The NDAlphaEngine is the heart of the system. All alpha logic lives here — the Nautilus layer is a thin wire.
### Primary Files
| File | Purpose |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py` | **MAIN ENGINE** — NDAlphaEngine, 7-layer alpha stack, begin_day/step_bar/end_day API |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_signal_generator.py` | Layer 1+2: vel_div gate, DC, OB-Sub2 signal confirmation |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_asset_selector.py` | Layer 5: IRP asset selection, numba kernels (compute_irp_nb, rank_assets_irp_nb) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_bet_sizer.py` | Layer 6: cubic-convex leverage sizing, numba compute_sizing_nb |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_exit_manager.py` | Layer 7: TP/SL/max_hold/OB-dynamic exit logic |
| `nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py` | ProxyBoostEngine wrapper (ACBv6 pre-compute, pre_bar_proxy_update) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator_AGENT_fork.py` | Agent-modified fork (preserve for diff/rollback) |
### Sub-Components Called by NDAlphaEngine
| File | Role |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/adaptive_circuit_breaker.py` | ACBv6 — 3-scale regime size multiplier; injected via engine.set_acb() |
| `nautilus_dolphin/nautilus_dolphin/nautilus/ob_features.py` | OBFeatureEngine — aggregates OB signals; injected via engine.set_ob_engine() |
| `nautilus_dolphin/nautilus_dolphin/nautilus/volatility_detector.py` | VolatilityRegimeDetector — 50-bar BTC return std gate |
| `nautilus_dolphin/nautilus_dolphin/nautilus/position_manager.py` | PositionManager — tracks open positions |
| `nautilus_dolphin/nautilus_dolphin/nautilus/circuit_breaker.py` | CircuitBreakerManager — per-asset trip logic |
| `nautilus_dolphin/nautilus_dolphin/nautilus/metrics_monitor.py` | MetricsMonitor — runtime performance metrics |
### Champion State
- **Config:** `prod/configs/blue.yml`
- **Performance log:** `nautilus_dolphin/run_logs/summary_20260307_163401.json`
- **Canonical backtest results:** `nautilus_dolphin/backtest_results/` (47+ JSON files)
- **VBT gold result:** `nautilus_dolphin/backtest_results/dolphin_vbt_real_champion.json` (if present)
---
## 3. NAUTILUS TRADER INTEGRATION
Nautilus Trader (v1.219.0, siloqy-env) provides the HFT execution kernel.
### Core Integration Files
| File | Purpose |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/dolphin_actor.py` | **DolphinActor(Strategy)** — Nautilus wrapper; on_start/on_bar/on_stop lifecycle; ACB pending-flag pattern; HZ integration |
| `nautilus_dolphin/nautilus_dolphin/nautilus/strategy.py` | DolphinExecutionStrategy — signal-level strategy; VolatilityRegimeDetector filter |
| `nautilus_dolphin/nautilus_dolphin/nautilus/strategy_config.py` | DolphinStrategyConfig(StrategyConfig, frozen=True) — typed champion params |
| `nautilus_dolphin/nautilus_dolphin/nautilus/launcher.py` | NautilusDolphinLauncher — TradingNode setup (live trading, future use) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/backtest_engine.py` | Nautilus BacktestEngine wrapper helpers |
| `nautilus_dolphin/nautilus_dolphin/nautilus/backtest_runner.py` | Backtest orchestration |
| `nautilus_dolphin/nautilus_dolphin/nautilus/data_adapter.py` | Data pipeline adapter (generic) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/arrow_data_adapter.py` | Arrow format data adapter |
| `nautilus_dolphin/nautilus_dolphin/nautilus/parquet_data_adapter.py` | Parquet format data adapter |
| `nautilus_dolphin/nautilus_dolphin/nautilus/arrow_parquet_catalog_builder.py` | Nautilus data catalog builder |
| `nautilus_dolphin/nautilus_dolphin/nautilus/execution_client.py` | Order execution client |
| `nautilus_dolphin/nautilus_dolphin/nautilus/trade_logger.py` | **TradeLoggerActor** — independent CSV/JSON session logger. |
| `nautilus_dolphin/nautilus_dolphin/nautilus/smart_exec_algorithm.py` | SmartExecAlgorithm — execution algo |
| `nautilus_dolphin/nautilus_dolphin/nautilus/signal_bridge.py` | SignalBridgeActor — signal distribution |
### Runners / Entry Points
| File | Purpose |
|---|---|
| `prod/run_nautilus.py` | Standalone CLI: BacktestEngine + DolphinActor, single day |
| `prod/paper_trade_flow.py` | **Primary daily flow** — NDAlphaEngine direct (00:05 UTC) |
| `prod/nautilus_prefect_flow.py` | **Nautilus supervisor flow** — BacktestEngine + DolphinActor (00:10 UTC); champion hash check; HZ heartbeats |
| `prod/ops/launch_paper_portfolio.py` | **Phoenix-01 Paper Launcher** — launches high-fidelity paper portfolio; realistic friction. |
### Nautilus Config
| File | Purpose |
|---|---|
| `nautilus_dolphin/config/config.yaml` | Nautilus runtime config — signals, strategy params, exchange, execution, Redis |
| `prod/configs/blue.yml` | **Champion SHORT config (FROZEN)** — all 15 champion params |
| `prod/configs/green.yml` | Bidirectional config (staging — pending LONG validation) |
---
## 4. ORDER BOOK FEATURES
OBF subsystem ingests Binance WS order books at ~500ms, computes 4 sub-system features, and pushes to Hazelcast.
### Source Files
| File | Purpose |
|---|---|
| `prod/obf_prefect_flow.py` | **OBF hot loop** — WS ingestion, feature computation, HZ push, file cache write |
| `prod/obf_persistence.py` | OBFPersistenceService — Parquet archival (asset=*/date=*/ partition) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/ob_provider.py` | OBProvider base class + OBSnapshot dataclass |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_ob_provider.py` | HZOBProvider — reads OB features from HZ shards; dynamic asset discovery |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_sharded_feature_store.py` | Sharded HZ feature store (SHARD_COUNT=10, routing by symbol hash) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/ob_placer.py` | OBPlacer — SmartPlacer fill probability computation |
| `scripts/verify_parquet_archive.py` | Parquet archive integrity checker (schema, gaps, corrupt files) |
### OBF Data Locations
| Store | Path / Key | Notes |
|---|---|---|
| **Live OB features** | HZ `DOLPHIN_FEATURES_SHARD_00..09` | Per-asset, pushed every ~500ms; shard = `sum(ord(c)) % 10` |
| **File fallback cache** | `/mnt/dolphinng5_predict/ob_cache/latest_ob_features.json` | Atomic tmp→rename; staleness threshold 5s |
| **Parquet archive** | `/mnt/ng6_data/ob_features/asset=*/date=*/*.parquet` | Long-term history; ~500ms resolution |
| **OB raw data** | `/mnt/dolphinng5_predict/ob_data/` | Raw WS snapshots |
### OBF Schema Reference
- `ob_cache/SCHEMA.md` — full field reference for `latest_ob_features.json`
- `prod/OBF_SUBSYSTEM.md` → now at `prod/docs/OBF_SUBSYSTEM.md`
---
## 5. EXTERNAL FACTORS (ExF)
ExF subsystem fetches macro/sentiment data (funding, dvol, fear&greed, taker ratio) and pushes to HZ for ACBv6 consumption.
### Source Files
| File | Purpose |
|---|---|
| `prod/exf_fetcher_simple.py` | **Primary ExF daemon** — live fetcher v2.1, pushes to HZ |
| `prod/exf_fetcher_flow.py` | Prefect ExF flow (main) |
| `prod/exf_fetcher_flow_fast.py` | Prefect ExF flow (fast variant) |
| `prod/exf_prefect_production.py` | ExF Prefect production runner |
| `prod/exf_persistence.py` | ExF persistence service |
| `prod/exf_integrity_monitor.py` | ExF data integrity monitoring |
| `prod/realtime_exf_service.py` | Real-time ExF service daemon |
| `prod/serve_exf.py` | ExF HTTP API server |
| `prod/deploy_exf.py` / `deploy_exf_v3.py` | Deployment scripts |
### ExF Data Locations
| Store | Path / Key | Notes |
|---|---|---|
| **Live ExF** | HZ `DOLPHIN_FEATURES["acb_boost"]` | JSON `{boost, beta}` — consumed by DolphinActor listener |
| **ExF snapshots** | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/extf_snapshot_*__Indicators.npz` | Per-scan NPZ, loaded by ACBv6 |
| **ESOF snapshots** | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/esof_snapshot_*__Indicators.npz` | Order flow NPZ |
| **Local ExF cache** | `/mnt/dolphinng5_predict/external_factors/eso_cache/` | ESO (equity short option) cache |
---
## 6. EIGENVALUE SCANNER & ACB
The NG3 correlation scanner runs on the Windows machine, producing per-scan eigenvalue data that DOLPHIN reads over SMB.
### Source Files (DOLPHIN side)
| File | Purpose |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/adaptive_circuit_breaker.py` | ACBv6 — reads ExF NPZ, computes 3-scale regime_size_mult |
| `prod/acb_processor_service.py` | ACB processor service — daily ACB boost computation + HZ write (uses HZ CP Subsystem lock) |
| `prod/esof_prefect_flow.py` | ESOF feature persistence flow |
| `prod/esof_persistence.py` | ESOF persistence service |
| `prod/esof_update_flow.py` | ESOF incremental update |
### Eigenvalue Data Locations
| Store | Linux Path | Windows Path | Content |
|---|---|---|---|
| **Daily eigenvalue scans** | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/` | `…\correlation_arb512\eigenvalues\YYYY-MM-DD\` | scan_*.npz, scan_*__Indicators.npz |
| **512-bit correlation matrices** | `/mnt/ng6_data/matrices/YYYY-MM-DD/` *(ACL restricted)* | `…\correlation_arb512\matrices\YYYY-MM-DD\` | scan_*_w50_*.arb512.pkl.zst |
| **VBT klines (5s, primary)** | `/mnt/dolphin/vbt_cache_klines/YYYY-MM-DD.parquet` | `…\vbt_cache_klines\YYYY-MM-DD.parquet` | 1439 rows × 57 cols: vel_div, v50/v150/v300/v750, instability, 48 asset prices |
| **Local klines** | `/mnt/dolphinng5_predict/vbt_cache_klines/YYYY-MM-DD.parquet` | — | Same schema; DOLPHIN-local copy used by prod flows |
| **Arrow backfill** | `/mnt/dolphin/arrow_backfill/YYYY-MM-DD/` | `…\arrow_backfill\YYYY-MM-DD\` | Arrow format, ~5yr history + synthetic |
| **VBT vectors** | `/mnt/dolphin/vbt_cache/*.parquet` | `…\vbt_cache\*.parquet` | ~1.7K files, vector format |
---
## 7. MONTE CARLO FOREWARNER
ML-based regime forewarning system that modulates ACBv6 scale (mc_scale).
### Source Files
| File | Purpose |
|---|---|
| `prod/mc_forewarner_flow.py` | **MC Forewarner Prefect flow** — prediction + HZ push |
| `nautilus_dolphin/mc_forewarning_qlabs_fork/mc/mc_ml.py` | DolphinForewarner ML model (loaded via engine.set_mc_forewarner()) |
| `prod/run_gold_monte_carlo.py` | Gold MC runner |
| `nautilus_dolphin/dvae/exp[1-15]_*.py` | Systematic DVAE experiments (sizing, exit, coupling, leverage, liquidation) |
### MC Data & Model Locations
| Store | Path | Notes |
|---|---|---|
| **MC models (prod)** | `prod/mc_results/models/` | Production-frozen model files |
| **MC models (alt)** | `nautilus_dolphin/mc_results/models/` | Subproject-local models |
| **MC manifests** | `prod/mc_results/manifests/` | Batch run manifests |
| **MC results** | `prod/mc_results/results/` | Output PnL/metrics JSON |
| **MC SQLite index** | `prod/mc_results/mc_index.sqlite` | Result index DB |
| **QLabs fork** | `nautilus_dolphin/mc_forewarning_qlabs_fork/` | Forewarner fork with benchmark results |
| **ML runs (MLflow)** | `/mnt/dolphinng5_predict/mlruns/` | MLflow experiment tracking |
---
## 8. PREFECT ORCHESTRATION
Prefect 3.6.22 (siloqy-env) orchestrates all daily flows.
### Flow Files
| File | Schedule | Purpose |
|---|---|---|
| `prod/paper_trade_flow.py` | 00:05 UTC daily | Primary paper trade — NDAlphaEngine direct; loads klines; pushes PnL + state to HZ |
| `prod/nautilus_prefect_flow.py` | 00:10 UTC daily | Nautilus BacktestEngine supervisor; champion hash check; HZ heartbeat |
| `prod/obf_prefect_flow.py` | Continuous (~500ms) | OBF hot loop — WS → compute → HZ push + JSON cache |
| `prod/exf_fetcher_flow.py` | Periodic | ExF data fetch + persistence |
| `prod/mc_forewarner_flow.py` | Daily | MC regime prediction + HZ write |
| `prod/vbt_backtest_flow.py` | On-demand | VBT backtest orchestration |
| `prod/vbt_cache_update_flow.py` | Periodic | Incremental klines cache update |
| `prod/esof_prefect_flow.py` | Periodic | Order flow feature persistence |
### Deployment Configs
| File | Deployment Name | Work Pool |
|---|---|---|
| `prod/exf_deployment.yaml` | ExF fetcher | dolphin |
| `prod/obf_deployment.yaml` | OBF hot loop | dolphin |
| `prod/esof_deployment.yaml` | ESOF features | dolphin |
### Prefect Infrastructure
| Component | Location |
|---|---|
| **Prefect Server** | Docker container, port 4200 (Tailscale: `100.105.170.6:4200`) |
| **Prefect Worker** | `prefect worker start --pool dolphin --type process` (siloqy-env) |
| **API URL** | `http://localhost:4200/api` (on DOLPHIN) |
| **Docker Compose** | `prod/docker-compose.yml` |
---
## 9. HAZELCAST IN-MEMORY GRID
Hazelcast acts as the **system memory** — real-time shared state between all subsystems.
### Infrastructure
| Component | Detail |
|---|---|
| **Version** | Hazelcast 5.3 (Docker) |
| **Address** | `localhost:5701` |
| **Cluster name** | `dolphin` |
| **CP Subsystem** | Enabled — used for atomic operations (ACB processor) |
| **Management Center** | `localhost:8080` |
| **Client (Python)** | `hazelcast-python-client 5.6.0` (siloqy-env) |
| **Docker Compose** | `prod/docker-compose.yml` |
### IMap Schema — All Named Maps
| IMap Name | Key | Value | Writer | Reader(s) |
|---|---|---|---|---|
| `DOLPHIN_SAFETY` | `"latest"` | JSON `{posture, Rm, sensors, ...}` | `system_watchdog_service.py` | `DolphinActor`, `paper_trade_flow`, `nautilus_prefect_flow` |
| `DOLPHIN_FEATURES` | `"acb_boost"` | JSON `{boost, beta}` | `acb_processor_service.py` | `DolphinActor` (HZ listener) |
| `DOLPHIN_FEATURES` | `"latest_eigen_scan"` | JSON `{vel_div, scan_number, asset_prices, timestamp_ns, ...}` | Eigenvalue scanner bridge | `DolphinActor` (live mode) |
| `DOLPHIN_PNL_BLUE` | `"YYYY-MM-DD"` | JSON daily result dict | `paper_trade_flow`, `DolphinActor._write_result_to_hz` | Analytics, monitoring |
| `DOLPHIN_PNL_GREEN` | `"YYYY-MM-DD"` | JSON daily result dict | `paper_trade_flow` (green) | Analytics |
| `DOLPHIN_STATE_BLUE` | `"latest"` | JSON `{strategy, capital, date, peak_capital, drawdown, engine_state, ...}` | `paper_trade_flow` | `paper_trade_flow` (capital restore) |
| `DOLPHIN_STATE_BLUE` | `"latest_nautilus"` | JSON `{capital, param_hash, posture, engine, ...}` | `nautilus_prefect_flow` | `nautilus_prefect_flow` (capital restore) |
| `DOLPHIN_STATE_BLUE` | `"state_{strategy}_{date}"` | JSON per-run snapshot | `paper_trade_flow` | Recovery |
| `DOLPHIN_HEARTBEAT` | `"nautilus_flow_heartbeat"` | JSON `{ts, iso, phase, run_date, ...}` | `nautilus_prefect_flow` | External monitoring |
| `DOLPHIN_HEARTBEAT` | `"probe_ts"` | Timestamp string | `nautilus_prefect_flow` (hz_probe_task) | Liveness |
| `DOLPHIN_META_HEALTH` | `"latest"` | JSON report | `meta_health_daemon.py` | Monitoring / Dashboards | L2 Health State |
| `DOLPHIN_OB` | per-asset key | JSON OB snapshot | `obf_prefect_flow` | `HZOBProvider` |
| `DOLPHIN_FEATURES_SHARD_00` | symbol | JSON OB feature dict | `obf_prefect_flow` | `HZOBProvider` |
| `DOLPHIN_FEATURES_SHARD_01..09` | symbol | JSON OB feature dict | `obf_prefect_flow` | `HZOBProvider` |
| `DOLPHIN_SIGNALS` | signal key | Signal distribution | `signal_bridge.py` | Strategy consumers |
### HZ Shard Routing (OB Features)
```python
SHARD_COUNT = 10
shard_idx = sum(ord(c) for c in symbol) % SHARD_COUNT
imap_name = f"DOLPHIN_FEATURES_SHARD_{shard_idx:02d}"
```
Routing is deterministic and requires no config. 400+ assets distribute evenly.
### HZ CP Subsystem
Used by `acb_processor_service.py` for distributed locking. CP Subsystem must be enabled in Hazelcast config (see `docker-compose.yml`).
### Key Source Files
| File | HZ Role |
|---|---|
| `prod/acb_processor_service.py` | Acquires CP lock; writes DOLPHIN_FEATURES["acb_boost"] daily |
| `prod/_hz_push.py` | Generic HZ push utility (ad-hoc writes) |
| `prod/system_watchdog_service.py` | Writes DOLPHIN_SAFETY (posture + Rm) |
| `prod/obf_prefect_flow.py` | Fire-and-forget per-asset writes to OB shards; circuit breaker on 5+ failures |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_ob_provider.py` | Reads OB shards; lazy connect; dynamic asset discovery from key_set |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_sharded_feature_store.py` | Shard routing + bulk read/write |
| `nautilus_dolphin/nautilus_dolphin/nautilus/scan_hz_bridge.py` | Scan data → HZ bridge |
| `nautilus_dolphin/nautilus_dolphin/nautilus/dolphin_actor.py` | `add_entry_listener` on DOLPHIN_FEATURES["acb_boost"]; pending-flag pattern |
---
## 10. ML MODELS & DVAE
### Model Files
| File / Directory | Purpose |
|---|---|
| `/mnt/dolphinng5_predict/models/hcm_model.json` | HCM neural network weights (12 MB) |
| `/mnt/dolphinng5_predict/models/tsf_model.json` | TSF (time-series forecaster) weights |
| `/mnt/dolphinng5_predict/models/tsf_engine.pkl` | TSF engine pickle |
| `/mnt/dolphinng5_predict/models/convnext_dvae_ML/` | ConvNeXt D-VAE ML directory |
| `/mnt/dolphinng5_predict/trained_models/` | Additional model checkpoints |
| `nautilus_dolphin/mc_forewarning_qlabs_fork/` | MC Forewarner model + benchmarks |
| `prod/mc_results/models/` | Production-frozen MC models |
### DVAE Experiments
| Location | Content |
|---|---|
| `nautilus_dolphin/dvae/` | 47+ experiment scripts |
| `nautilus_dolphin/dvae/exp[1-15]_*.py` | Systematic: sizing, exit coupling, leverage guards, liquidation, proxy |
| `nautilus_dolphin/dvae/convnext_dvae.py` | ConvNeXt model |
| `nautilus_dolphin/dvae/flint_dvae_kernel.py` | Flint-precision VAE kernel (512-bit path) |
| `nautilus_dolphin/dvae/exp_shared.py` | Shared experiment utilities |
### Training Infrastructure
| Location | Content |
|---|---|
| `/mnt/dolphin_training/` | Training data and scripts |
| `/mnt/dolphinng5_predict/training_reports/` | Training logs and metrics |
| `/mnt/dolphinng5_predict/checkpoints/` | Strategy checkpoints |
| `/mnt/dolphinng5_predict/checkpoints_10k/` | 10K-step checkpoints |
| `/mnt/dolphinng5_predict/checkpoints_production/` | Production-frozen checkpoints |
| `/mnt/dolphinng5_predict/mlruns/` | MLflow experiment tracking |
---
## 11. SURVIVAL STACK / WATCHDOG
### Source Files
| File | Purpose |
|---|---|
| `prod/system_watchdog_service.py` | **5-sensor Rm computation** → writes DOLPHIN_SAFETY to HZ |
| `prod/meta_health_daemon.py` | **Meta-System Monitoring (MHD)** — Watchdog-of-Watchdogs |
| `nautilus_dolphin/nautilus_dolphin/nautilus/survival_stack.py` | SurvivalStack class — sensor aggregation logic |
| `nautilus_dolphin/nautilus_dolphin/nautilus/macro_posture_switcher.py` | Macro regime posture switching |
### MHD (Meta) Specifics
| File | Role |
|---|---|
| `prod/meta_health_daemon.service` | Systemd unit (Linux) |
| `prod/meta_health_daemon_bsd.rc` | rc.d script (FreeBSD) |
| `run_logs/meta_health.json` | Latest MHD report |
| `run_logs/meta_health.log` | MHD persistent log |
### Posture Table
| Posture | Rm threshold | Effect |
|---|---|---|
| APEX | Rm ≥ 0.90 | Full operation |
| STALKER | Rm ≥ 0.75 | max_leverage capped to 2.0 |
| TURTLE | Rm ≥ 0.50 | abs_max_leverage × Rm |
| HIBERNATE | Rm < 0.50 | on_bar() returns immediately; no trading |
---
## 12. DATA LOCATIONS
### 12.1 Live / Real-Time Data (Hazelcast)
*See §9 IMap Schema above — Hazelcast is the primary real-time data bus.*
All live feature data (OB, ExF, ACB, scan, posture) flows through Hazelcast. Consumers must treat HZ as a data source, not a cache.
### 12.2 Scan Data (Eigenvalues)
| Dataset | Linux | Windows |
|---|---|---|
| Daily scan NPZ | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/` | `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues\YYYY-MM-DD\` |
| 512-bit matrices | `/mnt/ng6_data/matrices/YYYY-MM-DD/` *(ACL restricted)* | `…\correlation_arb512\matrices\YYYY-MM-DD\` |
### 12.3 Klines / VBT Cache (Primary Replay Source)
| Dataset | Linux | Windows |
|---|---|---|
| **5s klines (preferred)** | `/mnt/dolphin/vbt_cache_klines/YYYY-MM-DD.parquet` | `…\DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\YYYY-MM-DD.parquet` |
| VBT vector cache | `/mnt/dolphin/vbt_cache/*.parquet` | `…\vbt_cache\*.parquet` |
| Arrow backfill | `/mnt/dolphin/arrow_backfill/YYYY-MM-DD/` | `…\arrow_backfill\YYYY-MM-DD\` |
| Local klines copy | `/mnt/dolphinng5_predict/vbt_cache_klines/YYYY-MM-DD.parquet` | *(DOLPHIN-local)* |
**Klines schema:** 1439 rows × 57 cols per day. Columns: `vel_div`, `v50/v150/v300/v750_lambda_max_velocity`, `instability_50/150`, + 48 asset close prices.
### 12.4 OB Features
| Dataset | Path | Notes |
|---|---|---|
| Live OB (HZ) | `DOLPHIN_FEATURES_SHARD_00..09` | ~500ms latency |
| OB JSON fallback | `/mnt/dolphinng5_predict/ob_cache/latest_ob_features.json` | 5s staleness threshold |
| OB archive (Parquet) | `/mnt/ng6_data/ob_features/asset=*/date=*/*.parquet` | Partitioned; see SCHEMA.md |
| OB raw data | `/mnt/dolphinng5_predict/ob_data/` | Raw WS snapshots |
### 12.5 Paper Trade Logs & Results
| Dataset | Path |
|---|---|
| Blue paper logs | `prod/paper_logs/blue/paper_pnl_YYYY-MM.jsonl` |
| Green paper logs | `prod/paper_logs/green/paper_pnl_YYYY-MM.jsonl` |
| E2E trade CSVs | `prod/paper_logs/blue/E2E_trades_YYYY-MM-DD.csv` |
| E2E bar CSVs | `prod/paper_logs/blue/E2E_bars_YYYY-MM-DD.csv` |
| **Session Settings** | `logs/paper_trading/settings_*.json` |
| **Trade Audit** | `logs/paper_trading/trades_*.csv` |
| Nautilus run summary | HZ `DOLPHIN_STATE_BLUE["latest_nautilus"]` |
| Run logs JSON | `nautilus_dolphin/run_logs/summary_*.json` |
### 12.6 Backtest Results & MC
| Dataset | Path |
|---|---|
| Champion backtest | `nautilus_dolphin/backtest_results/` (47+ JSON) |
| MC results (prod) | `prod/mc_results/results/` |
| MC models | `prod/mc_results/models/` |
| MC SQLite index | `prod/mc_results/mc_index.sqlite` |
| 2-week backtest | `/mnt/dolphinng5_predict/backtest_results_2week/` |
| Paper 1-week | `/mnt/dolphinng5_predict/paper_trading_1week_results/` |
| Paper 1-month | `/mnt/dolphinng5_predict/paper_trading_1month_results/` |
| Rolling 10-week | `/mnt/dolphinng5_predict/rolling_10week_results/` |
### 12.7 Models
| Model | Path |
|---|---|
| HCM neural net | `/mnt/dolphinng5_predict/models/hcm_model.json` |
| TSF forecaster | `/mnt/dolphinng5_predict/models/tsf_model.json` |
| TSF engine | `/mnt/dolphinng5_predict/models/tsf_engine.pkl` |
| DVAE ConvNeXt | `/mnt/dolphinng5_predict/models/convnext_dvae_ML/` |
---
## 13. CONFIG & DEPLOYMENT FILES
| File | Purpose |
|---|---|
| `prod/docker-compose.yml` | Docker: Hazelcast 5.3 (port 5701), Management Center (8080), Prefect Server (4200) |
| `prod/configs/blue.yml` | **Champion SHORT — FROZEN** |
| `prod/configs/green.yml` | Bidirectional staging |
| `prod/obf_deployment.yaml` | OBF Prefect deployment (Prefect work pool: dolphin) |
| `prod/exf_deployment.yaml` | ExF Prefect deployment |
| `prod/esof_deployment.yaml` | ESOF Prefect deployment |
| `nautilus_dolphin/config/config.yaml` | Nautilus runtime config (signals, strategy, exchange, Redis `localhost:6379`) |
| `nautilus_dolphin/pyproject.toml` | Package config, pytest settings |
---
## 14. TEST SUITES
### Nautilus / Alpha Engine Tests (`nautilus_dolphin/tests/`)
| File | Tests | Coverage |
|---|---|---|
| `test_0_nautilus_bootstrap.py` | 11 | Import chain, NautilusKernelConfig, ACB, CircuitBreaker, launcher |
| `test_dolphin_actor.py` | 35 | DolphinActor lifecycle, ACB thread-safety, HIBERNATE, date change, HZ degradation |
| `test_adaptive_circuit_breaker.py` | ~10 | ACBv6 3-scale computation, cut-to-size |
| `test_circuit_breaker.py` | ~6 | CircuitBreakerManager |
| `test_volatility_detector.py` | ~6 | VolatilityRegimeDetector |
| `test_strategy.py` | ~5 | DolphinExecutionStrategy signal filters |
| `test_position_manager.py` | ~5 | PositionManager |
| `test_smart_exec_algorithm.py` | ~6 | SmartExecAlgorithm |
| `test_signal_bridge.py` | ~4 | SignalBridgeActor |
| `test_metrics_monitor.py` | ~4 | MetricsMonitor |
| `test_acb_standalone.py` | ~8 | ACB standalone (no Nautilus) |
| `test_acb_nautilus_vs_reference.py` | ~6 | ACB parity: Nautilus vs reference impl |
| `test_nd_vs_standalone_comparison.py` | ~8 | NDAlphaEngine vs standalone VBT parity |
| `test_trade_by_trade_validation.py` | ~10 | Trade-by-trade result validation |
| `test_proxy_boost_production.py` | ~8 | ProxyBoostEngine production checks |
| `test_strategy_registration.py` | ~4 | Strategy registration |
| `test_redis_integration.py` | ~4 | Redis signal integration |
### OBF Tests (`tests/`)
| File | Tests | Coverage |
|---|---|---|
| `tests/test_obf_unit.py` | ~120 | OBF subsystem: stream service, HZOBProvider, stale detection, crossed-book, buffer replay |
### CI Tests (`ci/`)
| File | Coverage |
|---|---|
| `ci/test_algo3_datasource_parity.py` | Algo3 data source parity |
| `ci/test_01_data_pipeline.py` | Data pipeline |
### Run Command
```bash
source /home/dolphin/siloqy_env/bin/activate
cd /mnt/dolphinng5_predict
python -m pytest nautilus_dolphin/tests/ -v # Nautilus/alpha suite
python -m pytest tests/test_obf_unit.py -v # OBF suite
```
---
## 15. UTILITY & SCRIPTS
| File | Purpose |
|---|---|
| `nautilus_dolphin/dolphin_paths.py` | **CRITICAL** — cross-platform path resolver |
| `scripts/verify_parquet_archive.py` | OBF Parquet archive integrity checker |
| `prod/_hz_push.py` | Ad-hoc HZ push utility |
| `prod/klines_backfill_5y_10y.py` | Historical klines builder (5yr/10yr) |
| `prod/continuous_convert.py` | Continuous Arrow→Parquet converter |
| `prod/convert_arrow_to_parquet_batch.py` | Batch Arrow→Parquet conversion |
| `prod/certify_extf_gold.py` | ExF gold certification |
| `prod/diag_5day.py` | 5-day system diagnostics |
| `prod/extract_spec.py` | Specification extractor |
---
## 16. BACKUP & FROZEN STATES
| Path | Content |
|---|---|
| `/mnt/dolphinng5_predict/FROZEN_BACKUP_20260208/` | System freeze Feb 8 2026 (alpha_engine + exit_matrix_engine) |
| `/mnt/dolphinng5_predict/alpha_engine_BACKUP_20260202_143018/` | Alpha engine pre-refactor backup |
| `/mnt/dolphinng5_predict/alpha_engine_BACKUP_20260209_203911/` | Alpha engine post-refactor backup |
| `/mnt/dolphinng5_predict/alpha_engine_BASELINE_75PCT_EDGE/` | Baseline 75% edge reference |
| `/mnt/dolphinng5_predict/backups_20260104/` | Jan 4 2026 backup |
---
## 17. DOCUMENTATION INDEX (`prod/docs/`)
| File | Content |
|---|---|
| `SYSTEM_BIBLE.md` | **THE BIBLE** — full doctrinal reference, current (v2 MIG7+Nautilus+Prefect) |
| `SYSTEM_BIBLE_v1_MIG7_20260307.md` | Bible fork — system state as of 2026-03-07, MIG7 complete |
| `NAUTILUS_DOLPHIN_SPEC.md` | Nautilus-DOLPHIN implementation spec (v1.0, 2026-03-22) |
| `SYSTEM_FILE_MAP.md` | **THIS FILE** — authoritative file/data location reference |
| `PRODUCTION_BRINGUP_MASTER_PLAN.md` | Production bringup checklist |
| `BRINGUP_GUIDE.md` | System bringup guide |
| `OBF_SUBSYSTEM.md` | OBF architecture reference (Sprint 1) |
| `NAUTILUS_INTEGRATION_ROADMAP.md` | Nautilus integration roadmap |
| `E2E_MASTER_PLAN.md` | End-to-end master plan |
| `EXTF_PROD_BRINGUP.md` | ExF production bringup |
| `EXTF_SYSTEM_PRODUCTIZATION_DETAILED_LOG.md` | ExF productization log |
| `EXF_V2_DEPLOYMENT_SUMMARY.md` | ExF v2 deployment summary |
| `LATENCY_OPTIONS.md` | Latency analysis options |
| `KLINES_5Y_10Y_DATASET_README.md` | Klines dataset readme |
| `AGENT_CHANGE_ANALYSIS_REPORT.md` | Agent change analysis |
| `AGENT_READ_CRITICAL__CHANGES_TO_ENGINE.md` | Critical engine changes |
| `NAUTILUS-DOLPHIN Prod System Spec_...FLAWS.md` | ChatGPT Survival Stack design flaws analysis |
---
*File Map version 1.0 — 2026-03-22 — DOLPHIN-NAUTILUS System*
Reference in New Issue Copy Permalink