initial: import DOLPHIN baseline 2026-04-21 from dolphinng5_predict working tree
Includes core prod + GREEN/BLUE subsystems: - prod/ (BLUE harness, configs, scripts, docs) - nautilus_dolphin/ (GREEN Nautilus-native impl + dvae/ preserved) - adaptive_exit/ (AEM engine + models/bucket_assignments.pkl) - Observability/ (EsoF advisor, TUI, dashboards) - external_factors/ (EsoF producer) - mc_forewarning_qlabs_fork/ (MC regime/envelope) Excludes runtime caches, logs, backups, and reproducible artifacts per .gitignore.
This commit is contained in:
hjnormey
331
prod/docs/BRINGUP_GUIDE.md
Executable file
331
prod/docs/BRINGUP_GUIDE.md
Executable file
@@ -0,0 +1,331 @@
|
||||
# DOLPHIN Paper Trading — Production Bringup Guide
|
||||
|
||||
**Purpose**: Step-by-step ops guide for standing up the Prefect + Hazelcast paper trading stack.
|
||||
**Audience**: Operations agent or junior dev. No research decisions required.
|
||||
**State as of**: 2026-03-06
|
||||
**Assumes**: Windows 11, Docker Desktop installed, Siloqy venv exists at `C:\Users\Lenovo\Documents\- Siloqy\`
|
||||
|
||||
---
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
[ARB512 Scanner] ─► eigenvalues/YYYY-MM-DD/ ─► [paper_trade_flow.py]
|
||||
|
|
||||
[NDAlphaEngine (Python)]
|
||||
|
|
||||
┌──────────────┴──────────────┐
|
||||
[Hazelcast IMap] [paper_logs/*.jsonl]
|
||||
|
|
||||
[Prefect UI :4200]
|
||||
[HZ-MC UI :8080]
|
||||
```
|
||||
|
||||
**Components:**
|
||||
- `docker-compose.yml`: Hazelcast 5.3 (port 5701) + HZ Management Center (port 8080) + Prefect Server (port 4200)
|
||||
- `paper_trade_flow.py`: Prefect flow, runs daily at 00:05 UTC
|
||||
- `configs/blue.yml`: Champion SHORT config (frozen, production)
|
||||
- `configs/green.yml`: Bidirectional config (STATUS: PENDING — LONG validation still in progress)
|
||||
- Python venv: `C:\Users\Lenovo\Documents\- Siloqy\`
|
||||
|
||||
**Data flow**: Prefect triggers daily → reads yesterday's Arrow/NPZ scans from eigenvalues dir → NDAlphaEngine processes → writes P&L to Hazelcast IMap + local JSONL log.
|
||||
|
||||
---
|
||||
|
||||
## Step 1: Prerequisites Check
|
||||
|
||||
Open a terminal (Git Bash or PowerShell).
|
||||
|
||||
```bash
|
||||
# 1a. Verify Docker Desktop is installed
|
||||
docker --version
|
||||
# Expected: Docker version 29.x.x
|
||||
|
||||
# 1b. Verify Python venv
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/python.exe" --version
|
||||
# Expected: Python 3.11.x or 3.12.x
|
||||
|
||||
# 1c. Verify working directories exist
|
||||
ls "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod/"
|
||||
# Expected: configs/ docker-compose.yml paper_trade_flow.py BRINGUP_GUIDE.md
|
||||
|
||||
ls "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod/configs/"
|
||||
# Expected: blue.yml green.yml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 2: Install Python Dependencies
|
||||
|
||||
Run once. Takes ~2-5 minutes.
|
||||
|
||||
```bash
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/pip.exe" install \
|
||||
hazelcast-python-client \
|
||||
prefect \
|
||||
pyyaml \
|
||||
pyarrow \
|
||||
numpy \
|
||||
pandas
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/python.exe" -c "import hazelcast; import prefect; import yaml; print('OK')"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 3: Start Docker Desktop
|
||||
|
||||
Docker Desktop must be running before starting containers.
|
||||
|
||||
**Option A (GUI):** Double-click Docker Desktop from Start menu. Wait for the whale icon in the system tray to stop animating (~30-60 seconds).
|
||||
|
||||
**Option B (command):**
|
||||
```powershell
|
||||
Start-Process "C:\Program Files\Docker\Docker\Docker Desktop.exe"
|
||||
# Wait ~60 seconds, then verify:
|
||||
docker ps
|
||||
```
|
||||
|
||||
**Verify Docker is ready:**
|
||||
```bash
|
||||
docker info | grep "Server Version"
|
||||
# Expected: Server Version: 27.x.x
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 4: Start the Infrastructure Stack
|
||||
|
||||
```bash
|
||||
cd "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod"
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
**Expected output:**
|
||||
```
|
||||
[+] Running 3/3
|
||||
- Container dolphin-hazelcast Started
|
||||
- Container dolphin-hazelcast-mc Started
|
||||
- Container dolphin-prefect Started
|
||||
```
|
||||
|
||||
**Verify all containers healthy:**
|
||||
```bash
|
||||
docker compose ps
|
||||
# All 3 should show "healthy" or "running"
|
||||
```
|
||||
|
||||
**Wait ~30 seconds for Hazelcast to initialize, then verify:**
|
||||
```bash
|
||||
curl http://localhost:5701/hazelcast/health/ready
|
||||
# Expected: {"message":"Hazelcast is ready!"}
|
||||
|
||||
curl http://localhost:4200/api/health
|
||||
# Expected: {"status":"healthy"}
|
||||
```
|
||||
|
||||
**UIs:**
|
||||
- Prefect UI: http://localhost:4200
|
||||
- Hazelcast MC: http://localhost:8080
|
||||
- Default cluster: `dolphin` (auto-connects to hazelcast:5701)
|
||||
|
||||
---
|
||||
|
||||
## Step 5: Register Prefect Deployments
|
||||
|
||||
Run once to register the blue and green scheduled deployments.
|
||||
|
||||
```bash
|
||||
cd "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod"
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/python.exe" paper_trade_flow.py --register
|
||||
```
|
||||
|
||||
**Expected output:**
|
||||
```
|
||||
Registered: dolphin-paper-blue
|
||||
Registered: dolphin-paper-green
|
||||
```
|
||||
|
||||
**Verify in Prefect UI:** http://localhost:4200 → Deployments → should show 2 deployments with CronSchedule "5 0 * * *".
|
||||
|
||||
---
|
||||
|
||||
## Step 6: Start the Prefect Worker
|
||||
|
||||
The Prefect worker polls for scheduled runs. Run in a separate terminal (keep it open, or run as a service).
|
||||
|
||||
```bash
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/prefect.exe" worker start --pool "dolphin"
|
||||
```
|
||||
|
||||
**OR** (if `prefect` CLI not in PATH):
|
||||
```bash
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/python.exe" -m prefect worker start --pool "dolphin"
|
||||
```
|
||||
|
||||
Leave this terminal running. It will pick up the 00:05 UTC scheduled runs.
|
||||
|
||||
---
|
||||
|
||||
## Step 7: Manual Test Run
|
||||
|
||||
Before relying on the schedule, test with a known good date (a date that has scan data).
|
||||
|
||||
```bash
|
||||
cd "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod"
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/python.exe" paper_trade_flow.py \
|
||||
--date 2026-03-05 \
|
||||
--config configs/blue.yml
|
||||
```
|
||||
|
||||
**Expected output (abbreviated):**
|
||||
```
|
||||
=== BLUE paper trade: 2026-03-05 ===
|
||||
Loaded N scans for 2026-03-05 | cols=XX
|
||||
2026-03-05: PnL=+XX.XX T=X boost=1.XXx MC=OK
|
||||
HZ write OK → DOLPHIN_PNL_BLUE[2026-03-05]
|
||||
=== DONE: blue 2026-03-05 | PnL=+XX.XX | Capital=25,XXX.XX ===
|
||||
```
|
||||
|
||||
**Verify data written to Hazelcast:**
|
||||
- Open http://localhost:8080 → Maps → DOLPHIN_PNL_BLUE → should contain entry for 2026-03-05
|
||||
|
||||
**Verify log file written:**
|
||||
```bash
|
||||
ls "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod/paper_logs/blue/"
|
||||
cat "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod/paper_logs/blue/paper_pnl_2026-03.jsonl"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Step 8: Scan Data Source Verification
|
||||
|
||||
The flow reads scan files from:
|
||||
```
|
||||
C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues\YYYY-MM-DD\
|
||||
```
|
||||
|
||||
Each date directory should contain `scan_*__Indicators.npz` or `scan_*.arrow` files.
|
||||
|
||||
```bash
|
||||
ls "/c/Users/Lenovo/Documents/- Dolphin NG HD (NG3)/correlation_arb512/eigenvalues/" | tail -5
|
||||
# Expected: recent date directories like 2026-03-05, 2026-03-04, etc.
|
||||
|
||||
ls "/c/Users/Lenovo/Documents/- Dolphin NG HD (NG3)/correlation_arb512/eigenvalues/2026-03-05/"
|
||||
# Expected: scan_NNNN__Indicators.npz files
|
||||
```
|
||||
|
||||
If a date directory is missing, the flow logs a warning and writes pnl=0 for that day (non-critical).
|
||||
|
||||
---
|
||||
|
||||
## Step 9: Daily Operations
|
||||
|
||||
**Normal daily flow (automated):**
|
||||
1. ARB512 scanner (extended_main.py) writes scans to eigenvalues/YYYY-MM-DD/ throughout the day
|
||||
2. At 00:05 UTC, Prefect triggers dolphin-paper-blue and dolphin-paper-green
|
||||
3. Each flow reads yesterday's scans, runs the engine, writes to HZ + JSONL log
|
||||
4. Monitor via Prefect UI and HZ-MC
|
||||
|
||||
**Check today's run result:**
|
||||
```bash
|
||||
# Latest P&L log entry:
|
||||
tail -1 "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod/paper_logs/blue/paper_pnl_$(date +%Y-%m).jsonl"
|
||||
```
|
||||
|
||||
**Check HZ state:**
|
||||
- http://localhost:8080 → Maps → DOLPHIN_STATE_BLUE → key "latest"
|
||||
- Should show: `{"capital": XXXXX, "strategy": "blue", "last_date": "YYYY-MM-DD", ...}`
|
||||
|
||||
---
|
||||
|
||||
## Step 10: Restart After Reboot
|
||||
|
||||
After Windows restarts:
|
||||
|
||||
```bash
|
||||
# 1. Start Docker Desktop (GUI or command — see Step 3)
|
||||
|
||||
# 2. Restart containers
|
||||
cd "/c/Users/Lenovo/Documents/- DOLPHIN NG HD HCM TSF Predict/prod"
|
||||
docker compose up -d
|
||||
|
||||
# 3. Restart Prefect worker (in a dedicated terminal)
|
||||
"/c/Users/Lenovo/Documents/- Siloqy/Scripts/python.exe" -m prefect worker start --pool "dolphin"
|
||||
```
|
||||
|
||||
Deployments and HZ data persist (docker volumes: hz_data, prefect_data).
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "No scan dir for YYYY-MM-DD"
|
||||
- The ARB512 scanner may not have run for that date
|
||||
- Check: `ls "C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues\YYYY-MM-DD\"`
|
||||
- Non-critical: flow logs pnl=0 and continues
|
||||
|
||||
### "HZ write failed (not critical)"
|
||||
- Hazelcast container not running or not yet healthy
|
||||
- Run: `docker compose ps` → check dolphin-hazelcast shows "healthy"
|
||||
- Run: `docker compose restart hazelcast`
|
||||
|
||||
### "ModuleNotFoundError: No module named 'hazelcast'"
|
||||
- Dependencies not installed in Siloqy venv
|
||||
- Rerun Step 2
|
||||
|
||||
### "error during connect: open //./pipe/dockerDesktopLinuxEngine"
|
||||
- Docker Desktop not running
|
||||
- Start Docker Desktop (see Step 3), wait 60 seconds, retry
|
||||
|
||||
### Prefect worker not picking up runs
|
||||
- Verify worker is running with `--pool "dolphin"` (matches work_queue_name in deployments)
|
||||
- Check Prefect UI → Work Pools → should show "dolphin" pool as online
|
||||
|
||||
### Green deployment errors on bidirectional config
|
||||
- Green is PENDING LONG validation. If direction: bidirectional causes engine errors,
|
||||
temporarily set green.yml direction: short_only until LONG system is validated.
|
||||
|
||||
---
|
||||
|
||||
## Key File Locations
|
||||
|
||||
| File | Path |
|
||||
|---|---|
|
||||
| Prefect flow | `prod/paper_trade_flow.py` |
|
||||
| Blue config | `prod/configs/blue.yml` |
|
||||
| Green config | `prod/configs/green.yml` |
|
||||
| Docker stack | `prod/docker-compose.yml` |
|
||||
| Blue P&L logs | `prod/paper_logs/blue/paper_pnl_YYYY-MM.jsonl` |
|
||||
| Green P&L logs | `prod/paper_logs/green/paper_pnl_YYYY-MM.jsonl` |
|
||||
| Scan data source | `C:\Users\Lenovo\Documents\- Dolphin NG HD (NG3)\correlation_arb512\eigenvalues\` |
|
||||
| NDAlphaEngine | `HCM\nautilus_dolphin\nautilus_dolphin\nautilus\esf_alpha_orchestrator.py` |
|
||||
| MC-Forewarner models | `HCM\nautilus_dolphin\mc_results\models\` |
|
||||
|
||||
---
|
||||
|
||||
## Current Status (2026-03-06)
|
||||
|
||||
| Item | Status |
|
||||
|---|---|
|
||||
| Docker stack | Built — needs Docker Desktop running |
|
||||
| Python deps (HZ + Prefect) | Installing (pip background job) |
|
||||
| Blue config | Frozen champion SHORT — ready |
|
||||
| Green config | PENDING — LONG validation running (b79rt78uv) |
|
||||
| Prefect deployments | Not yet registered (run Step 5 after deps install) |
|
||||
| Manual test run | Not yet done (run Step 7) |
|
||||
| vol_p60 calibration | Hardcoded 0.000099 (pre-calibrated from 55-day window) — acceptable |
|
||||
| Engine state persistence | Implemented — engine capital and open positions serialize to Hazelcast STATE IMap |
|
||||
|
||||
### Engine State Persistence
|
||||
|
||||
The NDAlphaEngine is instantiated fresh during each daily Prefect run, but its internal state is loaded from the Hazelcast `DOLPHIN_STATE_BLUE`/`GREEN` maps. Both `capital` and any active `position` spanning midnight are accurately tracked and restored.
|
||||
|
||||
**Impact for paper trading**: P&L and cumulative capital growth track correctly across days.
|
||||
|
||||
---
|
||||
|
||||
*Guide written 2026-03-08. Status updated.*
|
||||
Reference in New Issue
Block a user