Files
DOLPHIN/prod/docs/SYSTEM_FILE_MAP.md
hjnormey 01c19662cb initial: import DOLPHIN baseline 2026-04-21 from dolphinng5_predict working tree
Includes core prod + GREEN/BLUE subsystems:
- prod/ (BLUE harness, configs, scripts, docs)
- nautilus_dolphin/ (GREEN Nautilus-native impl + dvae/ preserved)
- adaptive_exit/ (AEM engine + models/bucket_assignments.pkl)
- Observability/ (EsoF advisor, TUI, dashboards)
- external_factors/ (EsoF producer)
- mc_forewarning_qlabs_fork/ (MC regime/envelope)

Excludes runtime caches, logs, backups, and reproducible artifacts per .gitignore.
2026-04-21 16:58:38 +02:00

30 KiB
Executable File
Raw Blame History

DOLPHIN-NAUTILUS System File Map

Authoritative location reference for all critical code, engines, and data

Version: 1.0 Date: 2026-03-22 Scope: All subsystems running on DOLPHIN (Linux) with cross-platform path references where shares exist


TABLE OF CONTENTS

  1. Path Resolution — Cross-Platform Key
  2. Subsystem A — Alpha Engine Core
  3. Subsystem B — Nautilus Trader Integration
  4. Subsystem C — Order Book Features (OBF)
  5. Subsystem D — External Factors (ExF)
  6. Subsystem E — Eigenvalue Scanner & ACB
  7. Subsystem F — Monte Carlo Forewarner
  8. Subsystem G — Prefect Orchestration
  9. Subsystem H — Hazelcast In-Memory Grid
  10. Subsystem I — ML Models & DVAE
  11. Subsystem J — Survival Stack / Watchdog
  12. Data Locations — All Stores
  13. Config & Deployment Files
  14. Test Suites
  15. Utility & Scripts

1. PATH RESOLUTION

Authoritative resolver: nautilus_dolphin/dolphin_paths.py

All code consuming shared data MUST import from this module — never hardcode paths.

from dolphin_paths import (
    get_arb512_storage_root,   # NG3/NG6 correlation_arb512 root
    get_eigenvalues_path,      # per-date eigenvalue + ExF snapshots
    get_project_root,          # NG5 Predict root
    get_vbt_cache_dir,         # VBT vector parquet cache
    get_klines_dir,            # 5yr 5s klines parquet
    get_arrow_backfill_dir,    # Arrow + synthetic backfill
)

Cross-Platform Path Table

Logical Name Linux Path Windows Path SMB Share
NG3/NG6 Data Root /mnt/ng6_data C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512 DolphinNG6_Data
Eigenvalues /mnt/ng6_data/eigenvalues/ …\correlation_arb512\eigenvalues\ DolphinNG6_Data
NG5 Predict Root /mnt/dolphin C:\Users\Lenovo\Documents\- DOLPHIN NG HD HCM TSF Predict DolphinNG5_Predict
VBT Cache (vectors) /mnt/dolphin/vbt_cache/ …\DOLPHIN NG HD HCM TSF Predict\vbt_cache\ DolphinNG5_Predict
Klines (5s, preferred) /mnt/dolphin/vbt_cache_klines/ …\DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\ DolphinNG5_Predict
Arrow Backfill /mnt/dolphin/arrow_backfill/ …\DOLPHIN NG HD HCM TSF Predict\arrow_backfill\ DolphinNG5_Predict
DOLPHIN Local Root /mnt/dolphinng5_predict/ (local on DOLPHIN machine)
NG6 Sparse /mnt/ng6/ DolphinNG6

SMB Server: 100.119.158.61 (Tailscale, Windows machine) Prefect Server: http://100.105.170.6:4200 (Tailscale) Hazelcast: localhost:5701 (Docker on DOLPHIN)


2. ALPHA ENGINE CORE

The NDAlphaEngine is the heart of the system. All alpha logic lives here — the Nautilus layer is a thin wire.

Primary Files

File Purpose
nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator.py MAIN ENGINE — NDAlphaEngine, 7-layer alpha stack, begin_day/step_bar/end_day API
nautilus_dolphin/nautilus_dolphin/nautilus/alpha_signal_generator.py Layer 1+2: vel_div gate, DC, OB-Sub2 signal confirmation
nautilus_dolphin/nautilus_dolphin/nautilus/alpha_asset_selector.py Layer 5: IRP asset selection, numba kernels (compute_irp_nb, rank_assets_irp_nb)
nautilus_dolphin/nautilus_dolphin/nautilus/alpha_bet_sizer.py Layer 6: cubic-convex leverage sizing, numba compute_sizing_nb
nautilus_dolphin/nautilus_dolphin/nautilus/alpha_exit_manager.py Layer 7: TP/SL/max_hold/OB-dynamic exit logic
nautilus_dolphin/nautilus_dolphin/nautilus/proxy_boost_engine.py ProxyBoostEngine wrapper (ACBv6 pre-compute, pre_bar_proxy_update)
nautilus_dolphin/nautilus_dolphin/nautilus/esf_alpha_orchestrator_AGENT_fork.py Agent-modified fork (preserve for diff/rollback)

Sub-Components Called by NDAlphaEngine

File Role
nautilus_dolphin/nautilus_dolphin/nautilus/adaptive_circuit_breaker.py ACBv6 — 3-scale regime size multiplier; injected via engine.set_acb()
nautilus_dolphin/nautilus_dolphin/nautilus/ob_features.py OBFeatureEngine — aggregates OB signals; injected via engine.set_ob_engine()
nautilus_dolphin/nautilus_dolphin/nautilus/volatility_detector.py VolatilityRegimeDetector — 50-bar BTC return std gate
nautilus_dolphin/nautilus_dolphin/nautilus/position_manager.py PositionManager — tracks open positions
nautilus_dolphin/nautilus_dolphin/nautilus/circuit_breaker.py CircuitBreakerManager — per-asset trip logic
nautilus_dolphin/nautilus_dolphin/nautilus/metrics_monitor.py MetricsMonitor — runtime performance metrics

Champion State

  • Config: prod/configs/blue.yml
  • Performance log: nautilus_dolphin/run_logs/summary_20260307_163401.json
  • Canonical backtest results: nautilus_dolphin/backtest_results/ (47+ JSON files)
  • VBT gold result: nautilus_dolphin/backtest_results/dolphin_vbt_real_champion.json (if present)

3. NAUTILUS TRADER INTEGRATION

Nautilus Trader (v1.219.0, siloqy-env) provides the HFT execution kernel.

Core Integration Files

File Purpose
nautilus_dolphin/nautilus_dolphin/nautilus/dolphin_actor.py DolphinActor(Strategy) — Nautilus wrapper; on_start/on_bar/on_stop lifecycle; ACB pending-flag pattern; HZ integration
nautilus_dolphin/nautilus_dolphin/nautilus/strategy.py DolphinExecutionStrategy — signal-level strategy; VolatilityRegimeDetector filter
nautilus_dolphin/nautilus_dolphin/nautilus/strategy_config.py DolphinStrategyConfig(StrategyConfig, frozen=True) — typed champion params
nautilus_dolphin/nautilus_dolphin/nautilus/launcher.py NautilusDolphinLauncher — TradingNode setup (live trading, future use)
nautilus_dolphin/nautilus_dolphin/nautilus/backtest_engine.py Nautilus BacktestEngine wrapper helpers
nautilus_dolphin/nautilus_dolphin/nautilus/backtest_runner.py Backtest orchestration
nautilus_dolphin/nautilus_dolphin/nautilus/data_adapter.py Data pipeline adapter (generic)
nautilus_dolphin/nautilus_dolphin/nautilus/arrow_data_adapter.py Arrow format data adapter
nautilus_dolphin/nautilus_dolphin/nautilus/parquet_data_adapter.py Parquet format data adapter
nautilus_dolphin/nautilus_dolphin/nautilus/arrow_parquet_catalog_builder.py Nautilus data catalog builder
nautilus_dolphin/nautilus_dolphin/nautilus/execution_client.py Order execution client
nautilus_dolphin/nautilus_dolphin/nautilus/trade_logger.py TradeLoggerActor — independent CSV/JSON session logger.
nautilus_dolphin/nautilus_dolphin/nautilus/smart_exec_algorithm.py SmartExecAlgorithm — execution algo
nautilus_dolphin/nautilus_dolphin/nautilus/signal_bridge.py SignalBridgeActor — signal distribution

Runners / Entry Points

File Purpose
prod/run_nautilus.py Standalone CLI: BacktestEngine + DolphinActor, single day
prod/paper_trade_flow.py Primary daily flow — NDAlphaEngine direct (00:05 UTC)
prod/nautilus_prefect_flow.py Nautilus supervisor flow — BacktestEngine + DolphinActor (00:10 UTC); champion hash check; HZ heartbeats
prod/ops/launch_paper_portfolio.py Phoenix-01 Paper Launcher — launches high-fidelity paper portfolio; realistic friction.

Nautilus Config

File Purpose
nautilus_dolphin/config/config.yaml Nautilus runtime config — signals, strategy params, exchange, execution, Redis
prod/configs/blue.yml Champion SHORT config (FROZEN) — all 15 champion params
prod/configs/green.yml Bidirectional config (staging — pending LONG validation)

4. ORDER BOOK FEATURES

OBF subsystem ingests Binance WS order books at ~500ms, computes 4 sub-system features, and pushes to Hazelcast.

Source Files

File Purpose
prod/obf_prefect_flow.py OBF hot loop — WS ingestion, feature computation, HZ push, file cache write
prod/obf_persistence.py OBFPersistenceService — Parquet archival (asset=/date=/ partition)
nautilus_dolphin/nautilus_dolphin/nautilus/ob_provider.py OBProvider base class + OBSnapshot dataclass
nautilus_dolphin/nautilus_dolphin/nautilus/hz_ob_provider.py HZOBProvider — reads OB features from HZ shards; dynamic asset discovery
nautilus_dolphin/nautilus_dolphin/nautilus/hz_sharded_feature_store.py Sharded HZ feature store (SHARD_COUNT=10, routing by symbol hash)
nautilus_dolphin/nautilus_dolphin/nautilus/ob_placer.py OBPlacer — SmartPlacer fill probability computation
scripts/verify_parquet_archive.py Parquet archive integrity checker (schema, gaps, corrupt files)

OBF Data Locations

Store Path / Key Notes
Live OB features HZ DOLPHIN_FEATURES_SHARD_00..09 Per-asset, pushed every ~500ms; shard = sum(ord(c)) % 10
File fallback cache /mnt/dolphinng5_predict/ob_cache/latest_ob_features.json Atomic tmp→rename; staleness threshold 5s
Parquet archive /mnt/ng6_data/ob_features/asset=*/date=*/*.parquet Long-term history; ~500ms resolution
OB raw data /mnt/dolphinng5_predict/ob_data/ Raw WS snapshots

OBF Schema Reference

  • ob_cache/SCHEMA.md — full field reference for latest_ob_features.json
  • prod/OBF_SUBSYSTEM.md → now at prod/docs/OBF_SUBSYSTEM.md

5. EXTERNAL FACTORS (ExF)

ExF subsystem fetches macro/sentiment data (funding, dvol, fear&greed, taker ratio) and pushes to HZ for ACBv6 consumption.

Source Files

File Purpose
prod/exf_fetcher_simple.py Primary ExF daemon — live fetcher v2.1, pushes to HZ
prod/exf_fetcher_flow.py Prefect ExF flow (main)
prod/exf_fetcher_flow_fast.py Prefect ExF flow (fast variant)
prod/exf_prefect_production.py ExF Prefect production runner
prod/exf_persistence.py ExF persistence service
prod/exf_integrity_monitor.py ExF data integrity monitoring
prod/realtime_exf_service.py Real-time ExF service daemon
prod/serve_exf.py ExF HTTP API server
prod/deploy_exf.py / deploy_exf_v3.py Deployment scripts

ExF Data Locations

Store Path / Key Notes
Live ExF HZ DOLPHIN_FEATURES["acb_boost"] JSON {boost, beta} — consumed by DolphinActor listener
ExF snapshots /mnt/ng6_data/eigenvalues/YYYY-MM-DD/extf_snapshot_*__Indicators.npz Per-scan NPZ, loaded by ACBv6
ESOF snapshots /mnt/ng6_data/eigenvalues/YYYY-MM-DD/esof_snapshot_*__Indicators.npz Order flow NPZ
Local ExF cache /mnt/dolphinng5_predict/external_factors/eso_cache/ ESO (equity short option) cache

6. EIGENVALUE SCANNER & ACB

The NG3 correlation scanner runs on the Windows machine, producing per-scan eigenvalue data that DOLPHIN reads over SMB.

Source Files (DOLPHIN side)

File Purpose
nautilus_dolphin/nautilus_dolphin/nautilus/adaptive_circuit_breaker.py ACBv6 — reads ExF NPZ, computes 3-scale regime_size_mult
prod/acb_processor_service.py ACB processor service — daily ACB boost computation + HZ write (uses HZ CP Subsystem lock)
prod/esof_prefect_flow.py ESOF feature persistence flow
prod/esof_persistence.py ESOF persistence service
prod/esof_update_flow.py ESOF incremental update

Eigenvalue Data Locations

Store Linux Path Windows Path Content
Daily eigenvalue scans /mnt/ng6_data/eigenvalues/YYYY-MM-DD/ …\correlation_arb512\eigenvalues\YYYY-MM-DD\ scan_.npz, scan___Indicators.npz
512-bit correlation matrices /mnt/ng6_data/matrices/YYYY-MM-DD/ (ACL restricted) …\correlation_arb512\matrices\YYYY-MM-DD\ scan_w50.arb512.pkl.zst
VBT klines (5s, primary) /mnt/dolphin/vbt_cache_klines/YYYY-MM-DD.parquet …\vbt_cache_klines\YYYY-MM-DD.parquet 1439 rows × 57 cols: vel_div, v50/v150/v300/v750, instability, 48 asset prices
Local klines /mnt/dolphinng5_predict/vbt_cache_klines/YYYY-MM-DD.parquet Same schema; DOLPHIN-local copy used by prod flows
Arrow backfill /mnt/dolphin/arrow_backfill/YYYY-MM-DD/ …\arrow_backfill\YYYY-MM-DD\ Arrow format, ~5yr history + synthetic
VBT vectors /mnt/dolphin/vbt_cache/*.parquet …\vbt_cache\*.parquet ~1.7K files, vector format

7. MONTE CARLO FOREWARNER

ML-based regime forewarning system that modulates ACBv6 scale (mc_scale).

Source Files

File Purpose
prod/mc_forewarner_flow.py MC Forewarner Prefect flow — prediction + HZ push
nautilus_dolphin/mc_forewarning_qlabs_fork/mc/mc_ml.py DolphinForewarner ML model (loaded via engine.set_mc_forewarner())
prod/run_gold_monte_carlo.py Gold MC runner
nautilus_dolphin/dvae/exp[1-15]_*.py Systematic DVAE experiments (sizing, exit, coupling, leverage, liquidation)

MC Data & Model Locations

Store Path Notes
MC models (prod) prod/mc_results/models/ Production-frozen model files
MC models (alt) nautilus_dolphin/mc_results/models/ Subproject-local models
MC manifests prod/mc_results/manifests/ Batch run manifests
MC results prod/mc_results/results/ Output PnL/metrics JSON
MC SQLite index prod/mc_results/mc_index.sqlite Result index DB
QLabs fork nautilus_dolphin/mc_forewarning_qlabs_fork/ Forewarner fork with benchmark results
ML runs (MLflow) /mnt/dolphinng5_predict/mlruns/ MLflow experiment tracking

8. PREFECT ORCHESTRATION

Prefect 3.6.22 (siloqy-env) orchestrates all daily flows.

Flow Files

File Schedule Purpose
prod/paper_trade_flow.py 00:05 UTC daily Primary paper trade — NDAlphaEngine direct; loads klines; pushes PnL + state to HZ
prod/nautilus_prefect_flow.py 00:10 UTC daily Nautilus BacktestEngine supervisor; champion hash check; HZ heartbeat
prod/obf_prefect_flow.py Continuous (~500ms) OBF hot loop — WS → compute → HZ push + JSON cache
prod/exf_fetcher_flow.py Periodic ExF data fetch + persistence
prod/mc_forewarner_flow.py Daily MC regime prediction + HZ write
prod/vbt_backtest_flow.py On-demand VBT backtest orchestration
prod/vbt_cache_update_flow.py Periodic Incremental klines cache update
prod/esof_prefect_flow.py Periodic Order flow feature persistence

Deployment Configs

File Deployment Name Work Pool
prod/exf_deployment.yaml ExF fetcher dolphin
prod/obf_deployment.yaml OBF hot loop dolphin
prod/esof_deployment.yaml ESOF features dolphin

Prefect Infrastructure

Component Location
Prefect Server Docker container, port 4200 (Tailscale: 100.105.170.6:4200)
Prefect Worker prefect worker start --pool dolphin --type process (siloqy-env)
API URL http://localhost:4200/api (on DOLPHIN)
Docker Compose prod/docker-compose.yml

9. HAZELCAST IN-MEMORY GRID

Hazelcast acts as the system memory — real-time shared state between all subsystems.

Infrastructure

Component Detail
Version Hazelcast 5.3 (Docker)
Address localhost:5701
Cluster name dolphin
CP Subsystem Enabled — used for atomic operations (ACB processor)
Management Center localhost:8080
Client (Python) hazelcast-python-client 5.6.0 (siloqy-env)
Docker Compose prod/docker-compose.yml

IMap Schema — All Named Maps

IMap Name Key Value Writer Reader(s)
DOLPHIN_SAFETY "latest" JSON {posture, Rm, sensors, ...} system_watchdog_service.py DolphinActor, paper_trade_flow, nautilus_prefect_flow
DOLPHIN_FEATURES "acb_boost" JSON {boost, beta} acb_processor_service.py DolphinActor (HZ listener)
DOLPHIN_FEATURES "latest_eigen_scan" JSON {vel_div, scan_number, asset_prices, timestamp_ns, ...} Eigenvalue scanner bridge DolphinActor (live mode)
DOLPHIN_PNL_BLUE "YYYY-MM-DD" JSON daily result dict paper_trade_flow, DolphinActor._write_result_to_hz Analytics, monitoring
DOLPHIN_PNL_GREEN "YYYY-MM-DD" JSON daily result dict paper_trade_flow (green) Analytics
DOLPHIN_STATE_BLUE "latest" JSON {strategy, capital, date, peak_capital, drawdown, engine_state, ...} paper_trade_flow paper_trade_flow (capital restore)
DOLPHIN_STATE_BLUE "latest_nautilus" JSON {capital, param_hash, posture, engine, ...} nautilus_prefect_flow nautilus_prefect_flow (capital restore)
DOLPHIN_STATE_BLUE "state_{strategy}_{date}" JSON per-run snapshot paper_trade_flow Recovery
DOLPHIN_HEARTBEAT "nautilus_flow_heartbeat" JSON {ts, iso, phase, run_date, ...} nautilus_prefect_flow External monitoring
DOLPHIN_HEARTBEAT "probe_ts" Timestamp string nautilus_prefect_flow (hz_probe_task) Liveness
DOLPHIN_META_HEALTH "latest" JSON report meta_health_daemon.py Monitoring / Dashboards
DOLPHIN_OB per-asset key JSON OB snapshot obf_prefect_flow HZOBProvider
DOLPHIN_FEATURES_SHARD_00 symbol JSON OB feature dict obf_prefect_flow HZOBProvider
DOLPHIN_FEATURES_SHARD_01..09 symbol JSON OB feature dict obf_prefect_flow HZOBProvider
DOLPHIN_SIGNALS signal key Signal distribution signal_bridge.py Strategy consumers

HZ Shard Routing (OB Features)

SHARD_COUNT = 10
shard_idx = sum(ord(c) for c in symbol) % SHARD_COUNT
imap_name = f"DOLPHIN_FEATURES_SHARD_{shard_idx:02d}"

Routing is deterministic and requires no config. 400+ assets distribute evenly.

HZ CP Subsystem

Used by acb_processor_service.py for distributed locking. CP Subsystem must be enabled in Hazelcast config (see docker-compose.yml).

Key Source Files

File HZ Role
prod/acb_processor_service.py Acquires CP lock; writes DOLPHIN_FEATURES["acb_boost"] daily
prod/_hz_push.py Generic HZ push utility (ad-hoc writes)
prod/system_watchdog_service.py Writes DOLPHIN_SAFETY (posture + Rm)
prod/obf_prefect_flow.py Fire-and-forget per-asset writes to OB shards; circuit breaker on 5+ failures
nautilus_dolphin/nautilus_dolphin/nautilus/hz_ob_provider.py Reads OB shards; lazy connect; dynamic asset discovery from key_set
nautilus_dolphin/nautilus_dolphin/nautilus/hz_sharded_feature_store.py Shard routing + bulk read/write
nautilus_dolphin/nautilus_dolphin/nautilus/scan_hz_bridge.py Scan data → HZ bridge
nautilus_dolphin/nautilus_dolphin/nautilus/dolphin_actor.py add_entry_listener on DOLPHIN_FEATURES["acb_boost"]; pending-flag pattern

10. ML MODELS & DVAE

Model Files

File / Directory Purpose
/mnt/dolphinng5_predict/models/hcm_model.json HCM neural network weights (12 MB)
/mnt/dolphinng5_predict/models/tsf_model.json TSF (time-series forecaster) weights
/mnt/dolphinng5_predict/models/tsf_engine.pkl TSF engine pickle
/mnt/dolphinng5_predict/models/convnext_dvae_ML/ ConvNeXt D-VAE ML directory
/mnt/dolphinng5_predict/trained_models/ Additional model checkpoints
nautilus_dolphin/mc_forewarning_qlabs_fork/ MC Forewarner model + benchmarks
prod/mc_results/models/ Production-frozen MC models

DVAE Experiments

Location Content
nautilus_dolphin/dvae/ 47+ experiment scripts
nautilus_dolphin/dvae/exp[1-15]_*.py Systematic: sizing, exit coupling, leverage guards, liquidation, proxy
nautilus_dolphin/dvae/convnext_dvae.py ConvNeXt model
nautilus_dolphin/dvae/flint_dvae_kernel.py Flint-precision VAE kernel (512-bit path)
nautilus_dolphin/dvae/exp_shared.py Shared experiment utilities

Training Infrastructure

Location Content
/mnt/dolphin_training/ Training data and scripts
/mnt/dolphinng5_predict/training_reports/ Training logs and metrics
/mnt/dolphinng5_predict/checkpoints/ Strategy checkpoints
/mnt/dolphinng5_predict/checkpoints_10k/ 10K-step checkpoints
/mnt/dolphinng5_predict/checkpoints_production/ Production-frozen checkpoints
/mnt/dolphinng5_predict/mlruns/ MLflow experiment tracking

11. SURVIVAL STACK / WATCHDOG

Source Files

File Purpose
prod/system_watchdog_service.py 5-sensor Rm computation → writes DOLPHIN_SAFETY to HZ
prod/meta_health_daemon.py Meta-System Monitoring (MHD) — Watchdog-of-Watchdogs
nautilus_dolphin/nautilus_dolphin/nautilus/survival_stack.py SurvivalStack class — sensor aggregation logic
nautilus_dolphin/nautilus_dolphin/nautilus/macro_posture_switcher.py Macro regime posture switching

MHD (Meta) Specifics

File Role
prod/meta_health_daemon.service Systemd unit (Linux)
prod/meta_health_daemon_bsd.rc rc.d script (FreeBSD)
run_logs/meta_health.json Latest MHD report
run_logs/meta_health.log MHD persistent log

Posture Table

Posture Rm threshold Effect
APEX Rm ≥ 0.90 Full operation
STALKER Rm ≥ 0.75 max_leverage capped to 2.0
TURTLE Rm ≥ 0.50 abs_max_leverage × Rm
HIBERNATE Rm < 0.50 on_bar() returns immediately; no trading

12. DATA LOCATIONS

12.1 Live / Real-Time Data (Hazelcast)

See §9 IMap Schema above — Hazelcast is the primary real-time data bus.

All live feature data (OB, ExF, ACB, scan, posture) flows through Hazelcast. Consumers must treat HZ as a data source, not a cache.

12.2 Scan Data (Eigenvalues)

Dataset Linux Windows
Daily scan NPZ /mnt/ng6_data/eigenvalues/YYYY-MM-DD/ C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues\YYYY-MM-DD\
512-bit matrices /mnt/ng6_data/matrices/YYYY-MM-DD/ (ACL restricted) …\correlation_arb512\matrices\YYYY-MM-DD\

12.3 Klines / VBT Cache (Primary Replay Source)

Dataset Linux Windows
5s klines (preferred) /mnt/dolphin/vbt_cache_klines/YYYY-MM-DD.parquet …\DOLPHIN NG HD HCM TSF Predict\vbt_cache_klines\YYYY-MM-DD.parquet
VBT vector cache /mnt/dolphin/vbt_cache/*.parquet …\vbt_cache\*.parquet
Arrow backfill /mnt/dolphin/arrow_backfill/YYYY-MM-DD/ …\arrow_backfill\YYYY-MM-DD\
Local klines copy /mnt/dolphinng5_predict/vbt_cache_klines/YYYY-MM-DD.parquet (DOLPHIN-local)

Klines schema: 1439 rows × 57 cols per day. Columns: vel_div, v50/v150/v300/v750_lambda_max_velocity, instability_50/150, + 48 asset close prices.

12.4 OB Features

Dataset Path Notes
Live OB (HZ) DOLPHIN_FEATURES_SHARD_00..09 ~500ms latency
OB JSON fallback /mnt/dolphinng5_predict/ob_cache/latest_ob_features.json 5s staleness threshold
OB archive (Parquet) /mnt/ng6_data/ob_features/asset=*/date=*/*.parquet Partitioned; see SCHEMA.md
OB raw data /mnt/dolphinng5_predict/ob_data/ Raw WS snapshots

12.5 Paper Trade Logs & Results

Dataset Path
Blue paper logs prod/paper_logs/blue/paper_pnl_YYYY-MM.jsonl
Green paper logs prod/paper_logs/green/paper_pnl_YYYY-MM.jsonl
E2E trade CSVs prod/paper_logs/blue/E2E_trades_YYYY-MM-DD.csv
E2E bar CSVs prod/paper_logs/blue/E2E_bars_YYYY-MM-DD.csv
Session Settings logs/paper_trading/settings_*.json
Trade Audit logs/paper_trading/trades_*.csv
Nautilus run summary HZ DOLPHIN_STATE_BLUE["latest_nautilus"]
Run logs JSON nautilus_dolphin/run_logs/summary_*.json

12.6 Backtest Results & MC

Dataset Path
Champion backtest nautilus_dolphin/backtest_results/ (47+ JSON)
MC results (prod) prod/mc_results/results/
MC models prod/mc_results/models/
MC SQLite index prod/mc_results/mc_index.sqlite
2-week backtest /mnt/dolphinng5_predict/backtest_results_2week/
Paper 1-week /mnt/dolphinng5_predict/paper_trading_1week_results/
Paper 1-month /mnt/dolphinng5_predict/paper_trading_1month_results/
Rolling 10-week /mnt/dolphinng5_predict/rolling_10week_results/

12.7 Models

Model Path
HCM neural net /mnt/dolphinng5_predict/models/hcm_model.json
TSF forecaster /mnt/dolphinng5_predict/models/tsf_model.json
TSF engine /mnt/dolphinng5_predict/models/tsf_engine.pkl
DVAE ConvNeXt /mnt/dolphinng5_predict/models/convnext_dvae_ML/

13. CONFIG & DEPLOYMENT FILES

File Purpose
prod/docker-compose.yml Docker: Hazelcast 5.3 (port 5701), Management Center (8080), Prefect Server (4200)
prod/configs/blue.yml Champion SHORT — FROZEN
prod/configs/green.yml Bidirectional staging
prod/obf_deployment.yaml OBF Prefect deployment (Prefect work pool: dolphin)
prod/exf_deployment.yaml ExF Prefect deployment
prod/esof_deployment.yaml ESOF Prefect deployment
nautilus_dolphin/config/config.yaml Nautilus runtime config (signals, strategy, exchange, Redis localhost:6379)
nautilus_dolphin/pyproject.toml Package config, pytest settings

14. TEST SUITES

Nautilus / Alpha Engine Tests (nautilus_dolphin/tests/)

File Tests Coverage
test_0_nautilus_bootstrap.py 11 Import chain, NautilusKernelConfig, ACB, CircuitBreaker, launcher
test_dolphin_actor.py 35 DolphinActor lifecycle, ACB thread-safety, HIBERNATE, date change, HZ degradation
test_adaptive_circuit_breaker.py ~10 ACBv6 3-scale computation, cut-to-size
test_circuit_breaker.py ~6 CircuitBreakerManager
test_volatility_detector.py ~6 VolatilityRegimeDetector
test_strategy.py ~5 DolphinExecutionStrategy signal filters
test_position_manager.py ~5 PositionManager
test_smart_exec_algorithm.py ~6 SmartExecAlgorithm
test_signal_bridge.py ~4 SignalBridgeActor
test_metrics_monitor.py ~4 MetricsMonitor
test_acb_standalone.py ~8 ACB standalone (no Nautilus)
test_acb_nautilus_vs_reference.py ~6 ACB parity: Nautilus vs reference impl
test_nd_vs_standalone_comparison.py ~8 NDAlphaEngine vs standalone VBT parity
test_trade_by_trade_validation.py ~10 Trade-by-trade result validation
test_proxy_boost_production.py ~8 ProxyBoostEngine production checks
test_strategy_registration.py ~4 Strategy registration
test_redis_integration.py ~4 Redis signal integration

OBF Tests (tests/)

File Tests Coverage
tests/test_obf_unit.py ~120 OBF subsystem: stream service, HZOBProvider, stale detection, crossed-book, buffer replay

CI Tests (ci/)

File Coverage
ci/test_algo3_datasource_parity.py Algo3 data source parity
ci/test_01_data_pipeline.py Data pipeline

Run Command

source /home/dolphin/siloqy_env/bin/activate
cd /mnt/dolphinng5_predict
python -m pytest nautilus_dolphin/tests/ -v        # Nautilus/alpha suite
python -m pytest tests/test_obf_unit.py -v         # OBF suite

15. UTILITY & SCRIPTS

File Purpose
nautilus_dolphin/dolphin_paths.py CRITICAL — cross-platform path resolver
scripts/verify_parquet_archive.py OBF Parquet archive integrity checker
prod/_hz_push.py Ad-hoc HZ push utility
prod/klines_backfill_5y_10y.py Historical klines builder (5yr/10yr)
prod/continuous_convert.py Continuous Arrow→Parquet converter
prod/convert_arrow_to_parquet_batch.py Batch Arrow→Parquet conversion
prod/certify_extf_gold.py ExF gold certification
prod/diag_5day.py 5-day system diagnostics
prod/extract_spec.py Specification extractor

16. BACKUP & FROZEN STATES

Path Content
/mnt/dolphinng5_predict/FROZEN_BACKUP_20260208/ System freeze Feb 8 2026 (alpha_engine + exit_matrix_engine)
/mnt/dolphinng5_predict/alpha_engine_BACKUP_20260202_143018/ Alpha engine pre-refactor backup
/mnt/dolphinng5_predict/alpha_engine_BACKUP_20260209_203911/ Alpha engine post-refactor backup
/mnt/dolphinng5_predict/alpha_engine_BASELINE_75PCT_EDGE/ Baseline 75% edge reference
/mnt/dolphinng5_predict/backups_20260104/ Jan 4 2026 backup

17. DOCUMENTATION INDEX (prod/docs/)

File Content
SYSTEM_BIBLE.md THE BIBLE — full doctrinal reference, current (v2 MIG7+Nautilus+Prefect)
SYSTEM_BIBLE_v1_MIG7_20260307.md Bible fork — system state as of 2026-03-07, MIG7 complete
NAUTILUS_DOLPHIN_SPEC.md Nautilus-DOLPHIN implementation spec (v1.0, 2026-03-22)
SYSTEM_FILE_MAP.md THIS FILE — authoritative file/data location reference
PRODUCTION_BRINGUP_MASTER_PLAN.md Production bringup checklist
BRINGUP_GUIDE.md System bringup guide
OBF_SUBSYSTEM.md OBF architecture reference (Sprint 1)
NAUTILUS_INTEGRATION_ROADMAP.md Nautilus integration roadmap
E2E_MASTER_PLAN.md End-to-end master plan
EXTF_PROD_BRINGUP.md ExF production bringup
EXTF_SYSTEM_PRODUCTIZATION_DETAILED_LOG.md ExF productization log
EXF_V2_DEPLOYMENT_SUMMARY.md ExF v2 deployment summary
LATENCY_OPTIONS.md Latency analysis options
KLINES_5Y_10Y_DATASET_README.md Klines dataset readme
AGENT_CHANGE_ANALYSIS_REPORT.md Agent change analysis
AGENT_READ_CRITICAL__CHANGES_TO_ENGINE.md Critical engine changes
NAUTILUS-DOLPHIN Prod System Spec_...FLAWS.md ChatGPT Survival Stack design flaws analysis

File Map version 1.0 — 2026-03-22 — DOLPHIN-NAUTILUS System