hjnormey/DOLPHIN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
aac4484c0f184ded5c9a0a393fb4343f1ccfb33c
DOLPHIN/prod/docs/SYSTEM_FILE_MAP.md
hjnormey 01c19662cb initial: import DOLPHIN baseline 2026-04-21 from dolphinng5_predict working tree 
Includes core prod + GREEN/BLUE subsystems:
- prod/ (BLUE harness, configs, scripts, docs)
- nautilus_dolphin/ (GREEN Nautilus-native impl + dvae/ preserved)
- adaptive_exit/ (AEM engine + models/bucket_assignments.pkl)
- Observability/ (EsoF advisor, TUI, dashboards)
- external_factors/ (EsoF producer)
- mc_forewarning_qlabs_fork/ (MC regime/envelope)

Excludes runtime caches, logs, backups, and reproducible artifacts per .gitignore.
2026-04-21 16:58:38 +02:00

616 lines
30 KiB
Markdown
Executable File
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# DOLPHIN-NAUTILUS System File Map
## Authoritative location reference for all critical code, engines, and data
**Version:** 1.0
**Date:** 2026-03-22
**Scope:** All subsystems running on DOLPHIN (Linux) with cross-platform path references where shares exist
---
## TABLE OF CONTENTS
1. [Path Resolution — Cross-Platform Key](#1-path-resolution)
2. [Subsystem A — Alpha Engine Core](#2-alpha-engine-core)
3. [Subsystem B — Nautilus Trader Integration](#3-nautilus-trader-integration)
4. [Subsystem C — Order Book Features (OBF)](#4-order-book-features)
5. [Subsystem D — External Factors (ExF)](#5-external-factors)
6. [Subsystem E — Eigenvalue Scanner & ACB](#6-eigenvalue-scanner--acb)
7. [Subsystem F — Monte Carlo Forewarner](#7-monte-carlo-forewarner)
8. [Subsystem G — Prefect Orchestration](#8-prefect-orchestration)
9. [Subsystem H — Hazelcast In-Memory Grid](#9-hazelcast-in-memory-grid)
10. [Subsystem I — ML Models & DVAE](#10-ml-models--dvae)
11. [Subsystem J — Survival Stack / Watchdog](#11-survival-stack--watchdog)
12. [Data Locations — All Stores](#12-data-locations)
13. [Config & Deployment Files](#13-config--deployment-files)
14. [Test Suites](#14-test-suites)
15. [Utility & Scripts](#15-utility--scripts)
---
## 1. PATH RESOLUTION
**Authoritative resolver:** `nautilus_dolphin/dolphin_paths.py`
All code consuming shared data MUST import from this module — never hardcode paths.
```python
from dolphin_paths import (
get_arb512_storage_root, # NG3/NG6 correlation_arb512 root
get_eigenvalues_path, # per-date eigenvalue + ExF snapshots
get_project_root, # NG5 Predict root
get_vbt_cache_dir, # VBT vector parquet cache
get_klines_dir, # 5yr 5s klines parquet
get_arrow_backfill_dir, # Arrow + synthetic backfill
)
```
### Cross-Platform Path Table
| Logical Name | Linux Path | Windows Path | SMB Share |
|---|---|---|---|
| **NG3/NG6 Data Root** | `/mnt/ng6_data` | `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512` | `DolphinNG6_Data` |
| **Eigenvalues** | `/mnt/ng6_data/eigenvalues/` | `…\correlation_arb512\eigenvalues\` | DolphinNG6_Data |
| **NG5 Predict Root** | `/mnt/dolphin` | `C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict` | `DolphinNG5_Predict` |
| **VBT Cache (vectors)** | `/mnt/dolphin/vbt_cache/` | `…\DOLPHIN NG HD HCM TSF Predict\vbt_cache\` | DolphinNG5_Predict |
| **Klines (5s, preferred)** | `/mnt/dolphin/vbt_cache_klines/` | `…\DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\` | DolphinNG5_Predict |
| **Arrow Backfill** | `/mnt/dolphin/arrow_backfill/` | `…\DOLPHIN NG HD HCM TSF Predict\arrow_backfill\` | DolphinNG5_Predict |
| **DOLPHIN Local Root** | `/mnt/dolphinng5_predict/` | *(local on DOLPHIN machine)* | — |
| **NG6 Sparse** | `/mnt/ng6/` | — | `DolphinNG6` |
> **SMB Server:** `100.119.158.61` (Tailscale, Windows machine)
> **Prefect Server:** `http://100.105.170.6:4200` (Tailscale)
> **Hazelcast:** `localhost:5701` (Docker on DOLPHIN)
---
## 2. ALPHA ENGINE CORE
The NDAlphaEngine is the heart of the system. All alpha logic lives here — the Nautilus layer is a thin wire.
### Primary Files
| File | Purpose |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py` | **MAIN ENGINE** — NDAlphaEngine, 7-layer alpha stack, begin_day/step_bar/end_day API |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_signal_generator.py` | Layer 1+2: vel_div gate, DC, OB-Sub2 signal confirmation |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_asset_selector.py` | Layer 5: IRP asset selection, numba kernels (compute_irp_nb, rank_assets_irp_nb) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_bet_sizer.py` | Layer 6: cubic-convex leverage sizing, numba compute_sizing_nb |
| `nautilus_dolphin/nautilus_dolphin/nautilus/alpha_exit_manager.py` | Layer 7: TP/SL/max_hold/OB-dynamic exit logic |
| `nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py` | ProxyBoostEngine wrapper (ACBv6 pre-compute, pre_bar_proxy_update) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator_AGENT_fork.py` | Agent-modified fork (preserve for diff/rollback) |
### Sub-Components Called by NDAlphaEngine
| File | Role |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/adaptive_circuit_breaker.py` | ACBv6 — 3-scale regime size multiplier; injected via engine.set_acb() |
| `nautilus_dolphin/nautilus_dolphin/nautilus/ob_features.py` | OBFeatureEngine — aggregates OB signals; injected via engine.set_ob_engine() |
| `nautilus_dolphin/nautilus_dolphin/nautilus/volatility_detector.py` | VolatilityRegimeDetector — 50-bar BTC return std gate |
| `nautilus_dolphin/nautilus_dolphin/nautilus/position_manager.py` | PositionManager — tracks open positions |
| `nautilus_dolphin/nautilus_dolphin/nautilus/circuit_breaker.py` | CircuitBreakerManager — per-asset trip logic |
| `nautilus_dolphin/nautilus_dolphin/nautilus/metrics_monitor.py` | MetricsMonitor — runtime performance metrics |
### Champion State
- **Config:** `prod/configs/blue.yml`
- **Performance log:** `nautilus_dolphin/run_logs/summary_20260307_163401.json`
- **Canonical backtest results:** `nautilus_dolphin/backtest_results/` (47+ JSON files)
- **VBT gold result:** `nautilus_dolphin/backtest_results/dolphin_vbt_real_champion.json` (if present)
---
## 3. NAUTILUS TRADER INTEGRATION
Nautilus Trader (v1.219.0, siloqy-env) provides the HFT execution kernel.
### Core Integration Files
| File | Purpose |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/dolphin_actor.py` | **DolphinActor(Strategy)** — Nautilus wrapper; on_start/on_bar/on_stop lifecycle; ACB pending-flag pattern; HZ integration |
| `nautilus_dolphin/nautilus_dolphin/nautilus/strategy.py` | DolphinExecutionStrategy — signal-level strategy; VolatilityRegimeDetector filter |
| `nautilus_dolphin/nautilus_dolphin/nautilus/strategy_config.py` | DolphinStrategyConfig(StrategyConfig, frozen=True) — typed champion params |
| `nautilus_dolphin/nautilus_dolphin/nautilus/launcher.py` | NautilusDolphinLauncher — TradingNode setup (live trading, future use) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/backtest_engine.py` | Nautilus BacktestEngine wrapper helpers |
| `nautilus_dolphin/nautilus_dolphin/nautilus/backtest_runner.py` | Backtest orchestration |
| `nautilus_dolphin/nautilus_dolphin/nautilus/data_adapter.py` | Data pipeline adapter (generic) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/arrow_data_adapter.py` | Arrow format data adapter |
| `nautilus_dolphin/nautilus_dolphin/nautilus/parquet_data_adapter.py` | Parquet format data adapter |
| `nautilus_dolphin/nautilus_dolphin/nautilus/arrow_parquet_catalog_builder.py` | Nautilus data catalog builder |
| `nautilus_dolphin/nautilus_dolphin/nautilus/execution_client.py` | Order execution client |
| `nautilus_dolphin/nautilus_dolphin/nautilus/trade_logger.py` | **TradeLoggerActor** — independent CSV/JSON session logger. |
| `nautilus_dolphin/nautilus_dolphin/nautilus/smart_exec_algorithm.py` | SmartExecAlgorithm — execution algo |
| `nautilus_dolphin/nautilus_dolphin/nautilus/signal_bridge.py` | SignalBridgeActor — signal distribution |
### Runners / Entry Points
| File | Purpose |
|---|---|
| `prod/run_nautilus.py` | Standalone CLI: BacktestEngine + DolphinActor, single day |
| `prod/paper_trade_flow.py` | **Primary daily flow** — NDAlphaEngine direct (00:05 UTC) |
| `prod/nautilus_prefect_flow.py` | **Nautilus supervisor flow** — BacktestEngine + DolphinActor (00:10 UTC); champion hash check; HZ heartbeats |
| `prod/ops/launch_paper_portfolio.py` | **Phoenix-01 Paper Launcher** — launches high-fidelity paper portfolio; realistic friction. |
### Nautilus Config
| File | Purpose |
|---|---|
| `nautilus_dolphin/config/config.yaml` | Nautilus runtime config — signals, strategy params, exchange, execution, Redis |
| `prod/configs/blue.yml` | **Champion SHORT config (FROZEN)** — all 15 champion params |
| `prod/configs/green.yml` | Bidirectional config (staging — pending LONG validation) |
---
## 4. ORDER BOOK FEATURES
OBF subsystem ingests Binance WS order books at ~500ms, computes 4 sub-system features, and pushes to Hazelcast.
### Source Files
| File | Purpose |
|---|---|
| `prod/obf_prefect_flow.py` | **OBF hot loop** — WS ingestion, feature computation, HZ push, file cache write |
| `prod/obf_persistence.py` | OBFPersistenceService — Parquet archival (asset=*/date=*/ partition) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/ob_provider.py` | OBProvider base class + OBSnapshot dataclass |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_ob_provider.py` | HZOBProvider — reads OB features from HZ shards; dynamic asset discovery |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_sharded_feature_store.py` | Sharded HZ feature store (SHARD_COUNT=10, routing by symbol hash) |
| `nautilus_dolphin/nautilus_dolphin/nautilus/ob_placer.py` | OBPlacer — SmartPlacer fill probability computation |
| `scripts/verify_parquet_archive.py` | Parquet archive integrity checker (schema, gaps, corrupt files) |
### OBF Data Locations
| Store | Path / Key | Notes |
|---|---|---|
| **Live OB features** | HZ `DOLPHIN_FEATURES_SHARD_00..09` | Per-asset, pushed every ~500ms; shard = `sum(ord(c)) % 10` |
| **File fallback cache** | `/mnt/dolphinng5_predict/ob_cache/latest_ob_features.json` | Atomic tmp→rename; staleness threshold 5s |
| **Parquet archive** | `/mnt/ng6_data/ob_features/asset=*/date=*/*.parquet` | Long-term history; ~500ms resolution |
| **OB raw data** | `/mnt/dolphinng5_predict/ob_data/` | Raw WS snapshots |
### OBF Schema Reference
- `ob_cache/SCHEMA.md` — full field reference for `latest_ob_features.json`
- `prod/OBF_SUBSYSTEM.md` → now at `prod/docs/OBF_SUBSYSTEM.md`
---
## 5. EXTERNAL FACTORS (ExF)
ExF subsystem fetches macro/sentiment data (funding, dvol, fear&greed, taker ratio) and pushes to HZ for ACBv6 consumption.
### Source Files
| File | Purpose |
|---|---|
| `prod/exf_fetcher_simple.py` | **Primary ExF daemon** — live fetcher v2.1, pushes to HZ |
| `prod/exf_fetcher_flow.py` | Prefect ExF flow (main) |
| `prod/exf_fetcher_flow_fast.py` | Prefect ExF flow (fast variant) |
| `prod/exf_prefect_production.py` | ExF Prefect production runner |
| `prod/exf_persistence.py` | ExF persistence service |
| `prod/exf_integrity_monitor.py` | ExF data integrity monitoring |
| `prod/realtime_exf_service.py` | Real-time ExF service daemon |
| `prod/serve_exf.py` | ExF HTTP API server |
| `prod/deploy_exf.py` / `deploy_exf_v3.py` | Deployment scripts |
### ExF Data Locations
| Store | Path / Key | Notes |
|---|---|---|
| **Live ExF** | HZ `DOLPHIN_FEATURES["acb_boost"]` | JSON `{boost, beta}` — consumed by DolphinActor listener |
| **ExF snapshots** | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/extf_snapshot_*__Indicators.npz` | Per-scan NPZ, loaded by ACBv6 |
| **ESOF snapshots** | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/esof_snapshot_*__Indicators.npz` | Order flow NPZ |
| **Local ExF cache** | `/mnt/dolphinng5_predict/external_factors/eso_cache/` | ESO (equity short option) cache |
---
## 6. EIGENVALUE SCANNER & ACB
The NG3 correlation scanner runs on the Windows machine, producing per-scan eigenvalue data that DOLPHIN reads over SMB.
### Source Files (DOLPHIN side)
| File | Purpose |
|---|---|
| `nautilus_dolphin/nautilus_dolphin/nautilus/adaptive_circuit_breaker.py` | ACBv6 — reads ExF NPZ, computes 3-scale regime_size_mult |
| `prod/acb_processor_service.py` | ACB processor service — daily ACB boost computation + HZ write (uses HZ CP Subsystem lock) |
| `prod/esof_prefect_flow.py` | ESOF feature persistence flow |
| `prod/esof_persistence.py` | ESOF persistence service |
| `prod/esof_update_flow.py` | ESOF incremental update |
### Eigenvalue Data Locations
| Store | Linux Path | Windows Path | Content |
|---|---|---|---|
| **Daily eigenvalue scans** | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/` | `…\correlation_arb512\eigenvalues\YYYY-MM-DD\` | scan_*.npz, scan_*__Indicators.npz |
| **512-bit correlation matrices** | `/mnt/ng6_data/matrices/YYYY-MM-DD/` *(ACL restricted)* | `…\correlation_arb512\matrices\YYYY-MM-DD\` | scan_*_w50_*.arb512.pkl.zst |
| **VBT klines (5s, primary)** | `/mnt/dolphin/vbt_cache_klines/YYYY-MM-DD.parquet` | `…\vbt_cache_klines\YYYY-MM-DD.parquet` | 1439 rows × 57 cols: vel_div, v50/v150/v300/v750, instability, 48 asset prices |
| **Local klines** | `/mnt/dolphinng5_predict/vbt_cache_klines/YYYY-MM-DD.parquet` | — | Same schema; DOLPHIN-local copy used by prod flows |
| **Arrow backfill** | `/mnt/dolphin/arrow_backfill/YYYY-MM-DD/` | `…\arrow_backfill\YYYY-MM-DD\` | Arrow format, ~5yr history + synthetic |
| **VBT vectors** | `/mnt/dolphin/vbt_cache/*.parquet` | `…\vbt_cache\*.parquet` | ~1.7K files, vector format |
---
## 7. MONTE CARLO FOREWARNER
ML-based regime forewarning system that modulates ACBv6 scale (mc_scale).
### Source Files
| File | Purpose |
|---|---|
| `prod/mc_forewarner_flow.py` | **MC Forewarner Prefect flow** — prediction + HZ push |
| `nautilus_dolphin/mc_forewarning_qlabs_fork/mc/mc_ml.py` | DolphinForewarner ML model (loaded via engine.set_mc_forewarner()) |
| `prod/run_gold_monte_carlo.py` | Gold MC runner |
| `nautilus_dolphin/dvae/exp[1-15]_*.py` | Systematic DVAE experiments (sizing, exit, coupling, leverage, liquidation) |
### MC Data & Model Locations
| Store | Path | Notes |
|---|---|---|
| **MC models (prod)** | `prod/mc_results/models/` | Production-frozen model files |
| **MC models (alt)** | `nautilus_dolphin/mc_results/models/` | Subproject-local models |
| **MC manifests** | `prod/mc_results/manifests/` | Batch run manifests |
| **MC results** | `prod/mc_results/results/` | Output PnL/metrics JSON |
| **MC SQLite index** | `prod/mc_results/mc_index.sqlite` | Result index DB |
| **QLabs fork** | `nautilus_dolphin/mc_forewarning_qlabs_fork/` | Forewarner fork with benchmark results |
| **ML runs (MLflow)** | `/mnt/dolphinng5_predict/mlruns/` | MLflow experiment tracking |
---
## 8. PREFECT ORCHESTRATION
Prefect 3.6.22 (siloqy-env) orchestrates all daily flows.
### Flow Files
| File | Schedule | Purpose |
|---|---|---|
| `prod/paper_trade_flow.py` | 00:05 UTC daily | Primary paper trade — NDAlphaEngine direct; loads klines; pushes PnL + state to HZ |
| `prod/nautilus_prefect_flow.py` | 00:10 UTC daily | Nautilus BacktestEngine supervisor; champion hash check; HZ heartbeat |
| `prod/obf_prefect_flow.py` | Continuous (~500ms) | OBF hot loop — WS → compute → HZ push + JSON cache |
| `prod/exf_fetcher_flow.py` | Periodic | ExF data fetch + persistence |
| `prod/mc_forewarner_flow.py` | Daily | MC regime prediction + HZ write |
| `prod/vbt_backtest_flow.py` | On-demand | VBT backtest orchestration |
| `prod/vbt_cache_update_flow.py` | Periodic | Incremental klines cache update |
| `prod/esof_prefect_flow.py` | Periodic | Order flow feature persistence |
### Deployment Configs
| File | Deployment Name | Work Pool |
|---|---|---|
| `prod/exf_deployment.yaml` | ExF fetcher | dolphin |
| `prod/obf_deployment.yaml` | OBF hot loop | dolphin |
| `prod/esof_deployment.yaml` | ESOF features | dolphin |
### Prefect Infrastructure
| Component | Location |
|---|---|
| **Prefect Server** | Docker container, port 4200 (Tailscale: `100.105.170.6:4200`) |
| **Prefect Worker** | `prefect worker start --pool dolphin --type process` (siloqy-env) |
| **API URL** | `http://localhost:4200/api` (on DOLPHIN) |
| **Docker Compose** | `prod/docker-compose.yml` |
---
## 9. HAZELCAST IN-MEMORY GRID
Hazelcast acts as the **system memory** — real-time shared state between all subsystems.
### Infrastructure
| Component | Detail |
|---|---|
| **Version** | Hazelcast 5.3 (Docker) |
| **Address** | `localhost:5701` |
| **Cluster name** | `dolphin` |
| **CP Subsystem** | Enabled — used for atomic operations (ACB processor) |
| **Management Center** | `localhost:8080` |
| **Client (Python)** | `hazelcast-python-client 5.6.0` (siloqy-env) |
| **Docker Compose** | `prod/docker-compose.yml` |
### IMap Schema — All Named Maps
| IMap Name | Key | Value | Writer | Reader(s) |
|---|---|---|---|---|
| `DOLPHIN_SAFETY` | `"latest"` | JSON `{posture, Rm, sensors, ...}` | `system_watchdog_service.py` | `DolphinActor`, `paper_trade_flow`, `nautilus_prefect_flow` |
| `DOLPHIN_FEATURES` | `"acb_boost"` | JSON `{boost, beta}` | `acb_processor_service.py` | `DolphinActor` (HZ listener) |
| `DOLPHIN_FEATURES` | `"latest_eigen_scan"` | JSON `{vel_div, scan_number, asset_prices, timestamp_ns, ...}` | Eigenvalue scanner bridge | `DolphinActor` (live mode) |
| `DOLPHIN_PNL_BLUE` | `"YYYY-MM-DD"` | JSON daily result dict | `paper_trade_flow`, `DolphinActor._write_result_to_hz` | Analytics, monitoring |
| `DOLPHIN_PNL_GREEN` | `"YYYY-MM-DD"` | JSON daily result dict | `paper_trade_flow` (green) | Analytics |
| `DOLPHIN_STATE_BLUE` | `"latest"` | JSON `{strategy, capital, date, peak_capital, drawdown, engine_state, ...}` | `paper_trade_flow` | `paper_trade_flow` (capital restore) |
| `DOLPHIN_STATE_BLUE` | `"latest_nautilus"` | JSON `{capital, param_hash, posture, engine, ...}` | `nautilus_prefect_flow` | `nautilus_prefect_flow` (capital restore) |
| `DOLPHIN_STATE_BLUE` | `"state_{strategy}_{date}"` | JSON per-run snapshot | `paper_trade_flow` | Recovery |
| `DOLPHIN_HEARTBEAT` | `"nautilus_flow_heartbeat"` | JSON `{ts, iso, phase, run_date, ...}` | `nautilus_prefect_flow` | External monitoring |
| `DOLPHIN_HEARTBEAT` | `"probe_ts"` | Timestamp string | `nautilus_prefect_flow` (hz_probe_task) | Liveness |
| `DOLPHIN_META_HEALTH` | `"latest"` | JSON report | `meta_health_daemon.py` | Monitoring / Dashboards | L2 Health State |
| `DOLPHIN_OB` | per-asset key | JSON OB snapshot | `obf_prefect_flow` | `HZOBProvider` |
| `DOLPHIN_FEATURES_SHARD_00` | symbol | JSON OB feature dict | `obf_prefect_flow` | `HZOBProvider` |
| `DOLPHIN_FEATURES_SHARD_01..09` | symbol | JSON OB feature dict | `obf_prefect_flow` | `HZOBProvider` |
| `DOLPHIN_SIGNALS` | signal key | Signal distribution | `signal_bridge.py` | Strategy consumers |
### HZ Shard Routing (OB Features)
```python
SHARD_COUNT = 10
shard_idx = sum(ord(c) for c in symbol) % SHARD_COUNT
imap_name = f"DOLPHIN_FEATURES_SHARD_{shard_idx:02d}"
```
Routing is deterministic and requires no config. 400+ assets distribute evenly.
### HZ CP Subsystem
Used by `acb_processor_service.py` for distributed locking. CP Subsystem must be enabled in Hazelcast config (see `docker-compose.yml`).
### Key Source Files
| File | HZ Role |
|---|---|
| `prod/acb_processor_service.py` | Acquires CP lock; writes DOLPHIN_FEATURES["acb_boost"] daily |
| `prod/_hz_push.py` | Generic HZ push utility (ad-hoc writes) |
| `prod/system_watchdog_service.py` | Writes DOLPHIN_SAFETY (posture + Rm) |
| `prod/obf_prefect_flow.py` | Fire-and-forget per-asset writes to OB shards; circuit breaker on 5+ failures |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_ob_provider.py` | Reads OB shards; lazy connect; dynamic asset discovery from key_set |
| `nautilus_dolphin/nautilus_dolphin/nautilus/hz_sharded_feature_store.py` | Shard routing + bulk read/write |
| `nautilus_dolphin/nautilus_dolphin/nautilus/scan_hz_bridge.py` | Scan data → HZ bridge |
| `nautilus_dolphin/nautilus_dolphin/nautilus/dolphin_actor.py` | `add_entry_listener` on DOLPHIN_FEATURES["acb_boost"]; pending-flag pattern |
---
## 10. ML MODELS & DVAE
### Model Files
| File / Directory | Purpose |
|---|---|
| `/mnt/dolphinng5_predict/models/hcm_model.json` | HCM neural network weights (12 MB) |
| `/mnt/dolphinng5_predict/models/tsf_model.json` | TSF (time-series forecaster) weights |
| `/mnt/dolphinng5_predict/models/tsf_engine.pkl` | TSF engine pickle |
| `/mnt/dolphinng5_predict/models/convnext_dvae_ML/` | ConvNeXt D-VAE ML directory |
| `/mnt/dolphinng5_predict/trained_models/` | Additional model checkpoints |
| `nautilus_dolphin/mc_forewarning_qlabs_fork/` | MC Forewarner model + benchmarks |
| `prod/mc_results/models/` | Production-frozen MC models |
### DVAE Experiments
| Location | Content |
|---|---|
| `nautilus_dolphin/dvae/` | 47+ experiment scripts |
| `nautilus_dolphin/dvae/exp[1-15]_*.py` | Systematic: sizing, exit coupling, leverage guards, liquidation, proxy |
| `nautilus_dolphin/dvae/convnext_dvae.py` | ConvNeXt model |
| `nautilus_dolphin/dvae/flint_dvae_kernel.py` | Flint-precision VAE kernel (512-bit path) |
| `nautilus_dolphin/dvae/exp_shared.py` | Shared experiment utilities |
### Training Infrastructure
| Location | Content |
|---|---|
| `/mnt/dolphin_training/` | Training data and scripts |
| `/mnt/dolphinng5_predict/training_reports/` | Training logs and metrics |
| `/mnt/dolphinng5_predict/checkpoints/` | Strategy checkpoints |
| `/mnt/dolphinng5_predict/checkpoints_10k/` | 10K-step checkpoints |
| `/mnt/dolphinng5_predict/checkpoints_production/` | Production-frozen checkpoints |
| `/mnt/dolphinng5_predict/mlruns/` | MLflow experiment tracking |
---
## 11. SURVIVAL STACK / WATCHDOG
### Source Files
| File | Purpose |
|---|---|
| `prod/system_watchdog_service.py` | **5-sensor Rm computation** → writes DOLPHIN_SAFETY to HZ |
| `prod/meta_health_daemon.py` | **Meta-System Monitoring (MHD)** — Watchdog-of-Watchdogs |
| `nautilus_dolphin/nautilus_dolphin/nautilus/survival_stack.py` | SurvivalStack class — sensor aggregation logic |
| `nautilus_dolphin/nautilus_dolphin/nautilus/macro_posture_switcher.py` | Macro regime posture switching |
### MHD (Meta) Specifics
| File | Role |
|---|---|
| `prod/meta_health_daemon.service` | Systemd unit (Linux) |
| `prod/meta_health_daemon_bsd.rc` | rc.d script (FreeBSD) |
| `run_logs/meta_health.json` | Latest MHD report |
| `run_logs/meta_health.log` | MHD persistent log |
### Posture Table
| Posture | Rm threshold | Effect |
|---|---|---|
| APEX | Rm ≥ 0.90 | Full operation |
| STALKER | Rm ≥ 0.75 | max_leverage capped to 2.0 |
| TURTLE | Rm ≥ 0.50 | abs_max_leverage × Rm |
| HIBERNATE | Rm < 0.50 | on_bar() returns immediately; no trading |
---
## 12. DATA LOCATIONS
### 12.1 Live / Real-Time Data (Hazelcast)
*See §9 IMap Schema above — Hazelcast is the primary real-time data bus.*
All live feature data (OB, ExF, ACB, scan, posture) flows through Hazelcast. Consumers must treat HZ as a data source, not a cache.
### 12.2 Scan Data (Eigenvalues)
| Dataset | Linux | Windows |
|---|---|---|
| Daily scan NPZ | `/mnt/ng6_data/eigenvalues/YYYY-MM-DD/` | `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues\YYYY-MM-DD\` |
| 512-bit matrices | `/mnt/ng6_data/matrices/YYYY-MM-DD/` *(ACL restricted)* | `…\correlation_arb512\matrices\YYYY-MM-DD\` |
### 12.3 Klines / VBT Cache (Primary Replay Source)
| Dataset | Linux | Windows |
|---|---|---|
| **5s klines (preferred)** | `/mnt/dolphin/vbt_cache_klines/YYYY-MM-DD.parquet` | `…\DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\YYYY-MM-DD.parquet` |
| VBT vector cache | `/mnt/dolphin/vbt_cache/*.parquet` | `…\vbt_cache\*.parquet` |
| Arrow backfill | `/mnt/dolphin/arrow_backfill/YYYY-MM-DD/` | `…\arrow_backfill\YYYY-MM-DD\` |
| Local klines copy | `/mnt/dolphinng5_predict/vbt_cache_klines/YYYY-MM-DD.parquet` | *(DOLPHIN-local)* |
**Klines schema:** 1439 rows × 57 cols per day. Columns: `vel_div`, `v50/v150/v300/v750_lambda_max_velocity`, `instability_50/150`, + 48 asset close prices.
### 12.4 OB Features
| Dataset | Path | Notes |
|---|---|---|
| Live OB (HZ) | `DOLPHIN_FEATURES_SHARD_00..09` | ~500ms latency |
| OB JSON fallback | `/mnt/dolphinng5_predict/ob_cache/latest_ob_features.json` | 5s staleness threshold |
| OB archive (Parquet) | `/mnt/ng6_data/ob_features/asset=*/date=*/*.parquet` | Partitioned; see SCHEMA.md |
| OB raw data | `/mnt/dolphinng5_predict/ob_data/` | Raw WS snapshots |
### 12.5 Paper Trade Logs & Results
| Dataset | Path |
|---|---|
| Blue paper logs | `prod/paper_logs/blue/paper_pnl_YYYY-MM.jsonl` |
| Green paper logs | `prod/paper_logs/green/paper_pnl_YYYY-MM.jsonl` |
| E2E trade CSVs | `prod/paper_logs/blue/E2E_trades_YYYY-MM-DD.csv` |
| E2E bar CSVs | `prod/paper_logs/blue/E2E_bars_YYYY-MM-DD.csv` |
| **Session Settings** | `logs/paper_trading/settings_*.json` |
| **Trade Audit** | `logs/paper_trading/trades_*.csv` |
| Nautilus run summary | HZ `DOLPHIN_STATE_BLUE["latest_nautilus"]` |
| Run logs JSON | `nautilus_dolphin/run_logs/summary_*.json` |
### 12.6 Backtest Results & MC
| Dataset | Path |
|---|---|
| Champion backtest | `nautilus_dolphin/backtest_results/` (47+ JSON) |
| MC results (prod) | `prod/mc_results/results/` |
| MC models | `prod/mc_results/models/` |
| MC SQLite index | `prod/mc_results/mc_index.sqlite` |
| 2-week backtest | `/mnt/dolphinng5_predict/backtest_results_2week/` |
| Paper 1-week | `/mnt/dolphinng5_predict/paper_trading_1week_results/` |
| Paper 1-month | `/mnt/dolphinng5_predict/paper_trading_1month_results/` |
| Rolling 10-week | `/mnt/dolphinng5_predict/rolling_10week_results/` |
### 12.7 Models
| Model | Path |
|---|---|
| HCM neural net | `/mnt/dolphinng5_predict/models/hcm_model.json` |
| TSF forecaster | `/mnt/dolphinng5_predict/models/tsf_model.json` |
| TSF engine | `/mnt/dolphinng5_predict/models/tsf_engine.pkl` |
| DVAE ConvNeXt | `/mnt/dolphinng5_predict/models/convnext_dvae_ML/` |
---
## 13. CONFIG & DEPLOYMENT FILES
| File | Purpose |
|---|---|
| `prod/docker-compose.yml` | Docker: Hazelcast 5.3 (port 5701), Management Center (8080), Prefect Server (4200) |
| `prod/configs/blue.yml` | **Champion SHORT — FROZEN** |
| `prod/configs/green.yml` | Bidirectional staging |
| `prod/obf_deployment.yaml` | OBF Prefect deployment (Prefect work pool: dolphin) |
| `prod/exf_deployment.yaml` | ExF Prefect deployment |
| `prod/esof_deployment.yaml` | ESOF Prefect deployment |
| `nautilus_dolphin/config/config.yaml` | Nautilus runtime config (signals, strategy, exchange, Redis `localhost:6379`) |
| `nautilus_dolphin/pyproject.toml` | Package config, pytest settings |
---
## 14. TEST SUITES
### Nautilus / Alpha Engine Tests (`nautilus_dolphin/tests/`)
| File | Tests | Coverage |
|---|---|---|
| `test_0_nautilus_bootstrap.py` | 11 | Import chain, NautilusKernelConfig, ACB, CircuitBreaker, launcher |
| `test_dolphin_actor.py` | 35 | DolphinActor lifecycle, ACB thread-safety, HIBERNATE, date change, HZ degradation |
| `test_adaptive_circuit_breaker.py` | ~10 | ACBv6 3-scale computation, cut-to-size |
| `test_circuit_breaker.py` | ~6 | CircuitBreakerManager |
| `test_volatility_detector.py` | ~6 | VolatilityRegimeDetector |
| `test_strategy.py` | ~5 | DolphinExecutionStrategy signal filters |
| `test_position_manager.py` | ~5 | PositionManager |
| `test_smart_exec_algorithm.py` | ~6 | SmartExecAlgorithm |
| `test_signal_bridge.py` | ~4 | SignalBridgeActor |
| `test_metrics_monitor.py` | ~4 | MetricsMonitor |
| `test_acb_standalone.py` | ~8 | ACB standalone (no Nautilus) |
| `test_acb_nautilus_vs_reference.py` | ~6 | ACB parity: Nautilus vs reference impl |
| `test_nd_vs_standalone_comparison.py` | ~8 | NDAlphaEngine vs standalone VBT parity |
| `test_trade_by_trade_validation.py` | ~10 | Trade-by-trade result validation |
| `test_proxy_boost_production.py` | ~8 | ProxyBoostEngine production checks |
| `test_strategy_registration.py` | ~4 | Strategy registration |
| `test_redis_integration.py` | ~4 | Redis signal integration |
### OBF Tests (`tests/`)
| File | Tests | Coverage |
|---|---|---|
| `tests/test_obf_unit.py` | ~120 | OBF subsystem: stream service, HZOBProvider, stale detection, crossed-book, buffer replay |
### CI Tests (`ci/`)
| File | Coverage |
|---|---|
| `ci/test_algo3_datasource_parity.py` | Algo3 data source parity |
| `ci/test_01_data_pipeline.py` | Data pipeline |
### Run Command
```bash
source /home/dolphin/siloqy_env/bin/activate
cd /mnt/dolphinng5_predict
python -m pytest nautilus_dolphin/tests/ -v # Nautilus/alpha suite
python -m pytest tests/test_obf_unit.py -v # OBF suite
```
---
## 15. UTILITY & SCRIPTS
| File | Purpose |
|---|---|
| `nautilus_dolphin/dolphin_paths.py` | **CRITICAL** — cross-platform path resolver |
| `scripts/verify_parquet_archive.py` | OBF Parquet archive integrity checker |
| `prod/_hz_push.py` | Ad-hoc HZ push utility |
| `prod/klines_backfill_5y_10y.py` | Historical klines builder (5yr/10yr) |
| `prod/continuous_convert.py` | Continuous Arrow→Parquet converter |
| `prod/convert_arrow_to_parquet_batch.py` | Batch Arrow→Parquet conversion |
| `prod/certify_extf_gold.py` | ExF gold certification |
| `prod/diag_5day.py` | 5-day system diagnostics |
| `prod/extract_spec.py` | Specification extractor |
---
## 16. BACKUP & FROZEN STATES
| Path | Content |
|---|---|
| `/mnt/dolphinng5_predict/FROZEN_BACKUP_20260208/` | System freeze Feb 8 2026 (alpha_engine + exit_matrix_engine) |
| `/mnt/dolphinng5_predict/alpha_engine_BACKUP_20260202_143018/` | Alpha engine pre-refactor backup |
| `/mnt/dolphinng5_predict/alpha_engine_BACKUP_20260209_203911/` | Alpha engine post-refactor backup |
| `/mnt/dolphinng5_predict/alpha_engine_BASELINE_75PCT_EDGE/` | Baseline 75% edge reference |
| `/mnt/dolphinng5_predict/backups_20260104/` | Jan 4 2026 backup |
---
## 17. DOCUMENTATION INDEX (`prod/docs/`)
| File | Content |
|---|---|
| `SYSTEM_BIBLE.md` | **THE BIBLE** — full doctrinal reference, current (v2 MIG7+Nautilus+Prefect) |
| `SYSTEM_BIBLE_v1_MIG7_20260307.md` | Bible fork — system state as of 2026-03-07, MIG7 complete |
| `NAUTILUS_DOLPHIN_SPEC.md` | Nautilus-DOLPHIN implementation spec (v1.0, 2026-03-22) |
| `SYSTEM_FILE_MAP.md` | **THIS FILE** — authoritative file/data location reference |
| `PRODUCTION_BRINGUP_MASTER_PLAN.md` | Production bringup checklist |
| `BRINGUP_GUIDE.md` | System bringup guide |
| `OBF_SUBSYSTEM.md` | OBF architecture reference (Sprint 1) |
| `NAUTILUS_INTEGRATION_ROADMAP.md` | Nautilus integration roadmap |
| `E2E_MASTER_PLAN.md` | End-to-end master plan |
| `EXTF_PROD_BRINGUP.md` | ExF production bringup |
| `EXTF_SYSTEM_PRODUCTIZATION_DETAILED_LOG.md` | ExF productization log |
| `EXF_V2_DEPLOYMENT_SUMMARY.md` | ExF v2 deployment summary |
| `LATENCY_OPTIONS.md` | Latency analysis options |
| `KLINES_5Y_10Y_DATASET_README.md` | Klines dataset readme |
| `AGENT_CHANGE_ANALYSIS_REPORT.md` | Agent change analysis |
| `AGENT_READ_CRITICAL__CHANGES_TO_ENGINE.md` | Critical engine changes |
| `NAUTILUS-DOLPHIN Prod System Spec_...FLAWS.md` | ChatGPT Survival Stack design flaws analysis |
---
*File Map version 1.0 — 2026-03-22 — DOLPHIN-NAUTILUS System*
Reference in New Issue View Git Blame Copy Permalink